]> source.dussan.org Git - rspamd.git/log
rspamd.git
10 years agoDetect language heuristic for text parts.
Vsevolod Stakhov [Tue, 4 Nov 2014 12:51:55 +0000 (12:51 +0000)]
Detect language heuristic for text parts.

10 years agoDetect HAN unicode script.
Vsevolod Stakhov [Tue, 4 Nov 2014 12:32:49 +0000 (12:32 +0000)]
Detect HAN unicode script.

10 years agoFix stupid error for FreeBSD build.
Vsevolod Stakhov [Mon, 3 Nov 2014 12:47:50 +0000 (12:47 +0000)]
Fix stupid error for FreeBSD build.

10 years agoRelease 0.7.3. 0.7.3
Vsevolod Stakhov [Mon, 3 Nov 2014 12:23:36 +0000 (12:23 +0000)]
Release 0.7.3.

* New upstreams code:
- simplify upstreams API;
- unify strings parsing in upstreams definition;
- add configuration options for the upstreams;
- for failed upstreams re-resolve their addresses;
- use all resolved addresses for an upstream (round-robin);
- implement stable hashing and use it by default for upstreams;
- add unit test for upstreams module.
* Rework signals processing in all rspamd workers:
- signals are now processed in the event loop;
- implement the most common signal handlers for all workers;
- add callbacks for workers specific signal handlers
*  Fix critical issue with fuzzy storage:
Fuzzy stroage could not save any hashes on termination due to bugged
signals handling
* Fix roll history IP storage
* Rework ipv4/ipv6 handling in parsing addresses:
    - turn off support of IPV6_V6ONLY socket option;
    - create ipv6 socket prior to ipv4 one to handle systems with v6/v4
    sockets enabled (Linux)
* Remove CBL as it's wholly included in Spamhaus XBL (by @fatalbanana)
* Remove nszones.com fake RBL (by @citrin)
* Fix upstreams interaction for fuzzy_check
* Verify spf PTR records (reported by @citrin)
* Fix spf MX records parsing
* Add compatibility for old libevent (by @yellowman)
* Sync bugfixes from libucl

Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
10 years agoImprove upstream tests.
Vsevolod Stakhov [Mon, 3 Nov 2014 11:41:07 +0000 (11:41 +0000)]
Improve upstream tests.

10 years agoFix tv_to_msec macro priority.
Vsevolod Stakhov [Mon, 3 Nov 2014 10:54:44 +0000 (10:54 +0000)]
Fix tv_to_msec macro priority.

10 years agoConfigure upstreams in workers.
Vsevolod Stakhov [Mon, 3 Nov 2014 10:23:32 +0000 (10:23 +0000)]
Configure upstreams in workers.

10 years agoParse unsigned integers in the configuration.
Vsevolod Stakhov [Mon, 3 Nov 2014 10:23:13 +0000 (10:23 +0000)]
Parse unsigned integers in the configuration.

10 years agoUse upstreams configuration.
Vsevolod Stakhov [Mon, 3 Nov 2014 10:19:13 +0000 (10:19 +0000)]
Use upstreams configuration.

10 years agoAdd upstreams configuration.
Vsevolod Stakhov [Mon, 3 Nov 2014 10:18:55 +0000 (10:18 +0000)]
Add upstreams configuration.

10 years agoAdd consistent hash test.
Vsevolod Stakhov [Sun, 2 Nov 2014 21:42:53 +0000 (21:42 +0000)]
Add consistent hash test.

P value is calculated as following:
when we add/remove M upstreams from the list, the probability of hash
miss should be close to the relation N / (N + M), where N is the size of
the previous upstreams list.

So far, the algorithm described in http://arxiv.org/abs/1406.2294 shows
p value more than 0.99.

10 years agoSome more fixes to the test.
Vsevolod Stakhov [Sun, 2 Nov 2014 19:03:15 +0000 (19:03 +0000)]
Some more fixes to the test.

10 years agoFix some upstream issues.
Vsevolod Stakhov [Sun, 2 Nov 2014 18:55:24 +0000 (18:55 +0000)]
Fix some upstream issues.

10 years agoAdd a simple upstreams test.
Vsevolod Stakhov [Sun, 2 Nov 2014 18:38:43 +0000 (18:38 +0000)]
Add a simple upstreams test.

10 years agoRemove old radix tests.
Vsevolod Stakhov [Sun, 2 Nov 2014 17:39:45 +0000 (17:39 +0000)]
Remove old radix tests.

10 years agoUpdate version to 0.7.3
Vsevolod Stakhov [Sun, 2 Nov 2014 17:19:00 +0000 (17:19 +0000)]
Update version to 0.7.3

10 years agoUse PTHREAD_PROCESS_SHARED mutexes if possible.
Vsevolod Stakhov [Sat, 1 Nov 2014 22:11:15 +0000 (22:11 +0000)]
Use PTHREAD_PROCESS_SHARED mutexes if possible.

10 years agoUse process-level mutex to avoid log races.
Vsevolod Stakhov [Sat, 1 Nov 2014 17:52:15 +0000 (17:52 +0000)]
Use process-level mutex to avoid log races.

10 years agoFix critical issue with fuzzy storage.
Vsevolod Stakhov [Sat, 1 Nov 2014 17:22:55 +0000 (17:22 +0000)]
Fix critical issue with fuzzy storage.

Fuzzy stroage could not save any hashes on termination due to bugged
signals handling.

Reported by: at0r at #rspamd

10 years agoFix signals initializing.
Vsevolod Stakhov [Sat, 1 Nov 2014 17:22:45 +0000 (17:22 +0000)]
Fix signals initializing.

10 years agoAdd post-signal handler.
Vsevolod Stakhov [Sat, 1 Nov 2014 17:05:35 +0000 (17:05 +0000)]
Add post-signal handler.

10 years agoRework signals processing in workers.
Vsevolod Stakhov [Sat, 1 Nov 2014 16:41:38 +0000 (16:41 +0000)]
Rework signals processing in workers.

10 years agoUse constant for hash seed for upstreams selection.
Vsevolod Stakhov [Sat, 1 Nov 2014 15:18:48 +0000 (15:18 +0000)]
Use constant for hash seed for upstreams selection.

10 years agoAdd jitter for upstream reviving.
Vsevolod Stakhov [Sat, 1 Nov 2014 15:18:33 +0000 (15:18 +0000)]
Add jitter for upstream reviving.

10 years agoFix some issues in the upstream code.
Vsevolod Stakhov [Fri, 31 Oct 2014 12:14:32 +0000 (12:14 +0000)]
Fix some issues in the upstream code.

10 years agoFix roll history IP storage.
Vsevolod Stakhov [Fri, 31 Oct 2014 11:46:48 +0000 (11:46 +0000)]
Fix roll history IP storage.

10 years agoFix upstreams in ratelimit.
Vsevolod Stakhov [Thu, 30 Oct 2014 22:29:28 +0000 (22:29 +0000)]
Fix upstreams in ratelimit.

10 years agoFuck ipv6 inventors for ipv6/v4 sockets.
Vsevolod Stakhov [Wed, 29 Oct 2014 15:40:05 +0000 (15:40 +0000)]
Fuck ipv6 inventors for ipv6/v4 sockets.

10 years agoSet length properly.
Vsevolod Stakhov [Wed, 29 Oct 2014 15:39:53 +0000 (15:39 +0000)]
Set length properly.

10 years agoFix build.
Vsevolod Stakhov [Wed, 29 Oct 2014 15:02:51 +0000 (15:02 +0000)]
Fix build.

10 years agoFix fuzzy_check.
Vsevolod Stakhov [Wed, 29 Oct 2014 14:54:28 +0000 (14:54 +0000)]
Fix fuzzy_check.

10 years agoSome more auxiliary functions.
Vsevolod Stakhov [Wed, 29 Oct 2014 14:54:18 +0000 (14:54 +0000)]
Some more auxiliary functions.

10 years agoFix surbl.
Vsevolod Stakhov [Wed, 29 Oct 2014 14:10:57 +0000 (14:10 +0000)]
Fix surbl.

10 years agoFix smtp proxy code.
Vsevolod Stakhov [Wed, 29 Oct 2014 14:02:13 +0000 (14:02 +0000)]
Fix smtp proxy code.

10 years agoAdd parsing of multiple upstreams string.
Vsevolod Stakhov [Wed, 29 Oct 2014 13:58:33 +0000 (13:58 +0000)]
Add parsing of multiple upstreams string.

10 years agoRework and simplify listen sockets creation.
Vsevolod Stakhov [Wed, 29 Oct 2014 12:18:13 +0000 (12:18 +0000)]
Rework and simplify listen sockets creation.

10 years agoAllow addresses to be allocated in rspamd_mempool.
Vsevolod Stakhov [Wed, 29 Oct 2014 11:26:08 +0000 (11:26 +0000)]
Allow addresses to be allocated in rspamd_mempool.

10 years agoFix some more old upstreams code.
Vsevolod Stakhov [Wed, 29 Oct 2014 01:38:22 +0000 (01:38 +0000)]
Fix some more old upstreams code.

10 years agoDisable binlog configuration as it is anyway broken.
Vsevolod Stakhov [Tue, 28 Oct 2014 23:24:53 +0000 (23:24 +0000)]
Disable binlog configuration as it is anyway broken.

10 years agoMerge branch 'upstream-rework'
Vsevolod Stakhov [Tue, 28 Oct 2014 17:27:59 +0000 (17:27 +0000)]
Merge branch 'upstream-rework'

10 years agoFix lua upstream code.
Vsevolod Stakhov [Tue, 28 Oct 2014 17:27:06 +0000 (17:27 +0000)]
Fix lua upstream code.

10 years agoUpstreams get implementation.
Vsevolod Stakhov [Tue, 28 Oct 2014 17:04:43 +0000 (17:04 +0000)]
Upstreams get implementation.

10 years agoAllow multiple addresses per upstream.
Vsevolod Stakhov [Tue, 28 Oct 2014 16:04:33 +0000 (16:04 +0000)]
Allow multiple addresses per upstream.

10 years agoAdd some basic logic for upstreams.
Vsevolod Stakhov [Tue, 28 Oct 2014 15:19:06 +0000 (15:19 +0000)]
Add some basic logic for upstreams.

10 years agoRework host/port/priority parsing.
Vsevolod Stakhov [Tue, 28 Oct 2014 14:03:46 +0000 (14:03 +0000)]
Rework host/port/priority parsing.

10 years agoSplit addr definition from the utils code.
Vsevolod Stakhov [Tue, 28 Oct 2014 13:36:15 +0000 (13:36 +0000)]
Split addr definition from the utils code.

10 years agoStart new upstreams implementation.
Vsevolod Stakhov [Tue, 28 Oct 2014 13:29:13 +0000 (13:29 +0000)]
Start new upstreams implementation.

10 years agoRework API completely.
Vsevolod Stakhov [Tue, 28 Oct 2014 12:18:50 +0000 (12:18 +0000)]
Rework API completely.

10 years agoUpdate readme.
Vsevolod Stakhov [Sun, 26 Oct 2014 18:07:20 +0000 (18:07 +0000)]
Update readme.

10 years agoMerge pull request #112 from fatalbanana/master
Vsevolod Stakhov [Thu, 23 Oct 2014 15:27:42 +0000 (16:27 +0100)]
Merge pull request #112 from fatalbanana/master

Remove CBL as it's wholly included in Spamhaus XBL

10 years agoRemove CBL as it's wholly included in Spamhaus XBL which we're querying already ... 112/head
Andrew Lewis [Thu, 23 Oct 2014 14:23:21 +0000 (16:23 +0200)]
Remove CBL as it's wholly included in Spamhaus XBL which we're querying already - http://cbl.abuseat.org/faq.html

10 years agoMerge pull request #111 from citrin/master
Vsevolod Stakhov [Thu, 23 Oct 2014 12:21:46 +0000 (13:21 +0100)]
Merge pull request #111 from citrin/master

Remove nszones.com fake RBL

10 years agoRemove /0 mask for SPF mx as well.
Vsevolod Stakhov [Wed, 22 Oct 2014 16:11:47 +0000 (17:11 +0100)]
Remove /0 mask for SPF mx as well.

Reported by: @citrin

10 years agoFix upstreams interaction for fuzzy_check.
Vsevolod Stakhov [Wed, 22 Oct 2014 16:07:38 +0000 (17:07 +0100)]
Fix upstreams interaction for fuzzy_check.

10 years agoRevert ucl resynchronization.
Vsevolod Stakhov [Wed, 22 Oct 2014 16:06:29 +0000 (17:06 +0100)]
Revert ucl resynchronization.

Reported by: @citrin

10 years agoProperly set address for PTR components.
Vsevolod Stakhov [Wed, 22 Oct 2014 15:52:40 +0000 (16:52 +0100)]
Properly set address for PTR components.

10 years agoVerify spf PTR records.
Vsevolod Stakhov [Wed, 22 Oct 2014 15:21:18 +0000 (16:21 +0100)]
Verify spf PTR records.

Reported by: @citrin

10 years agoRemove nszones.com fake RBL: 111/head
Anton Yuzhaninov [Tue, 21 Oct 2014 15:19:19 +0000 (19:19 +0400)]
Remove nszones.com fake RBL:
http://www.spamhaus.org/organization/statement/8/

10 years agoAdd compatibility for old libevent.
Vsevolod Stakhov [Fri, 17 Oct 2014 00:10:30 +0000 (01:10 +0100)]
Add compatibility for old libevent.

Issue: #109
Suggested by: @yellowman

10 years agoSync bugfix from libucl.
Vsevolod Stakhov [Thu, 16 Oct 2014 13:12:38 +0000 (14:12 +0100)]
Sync bugfix from libucl.

10 years agoRelease 0.7.2. 0.7.2
Vsevolod Stakhov [Wed, 15 Oct 2014 16:30:22 +0000 (17:30 +0100)]
Release 0.7.2.

* Convert all maps to the compressed radix trie
* Allow IPv6 addresses in IP maps
* Remove dynamic items support from symbols cache
* Allow hex encoded output of strings
* Fix bug with control connections count
* Process fuzzy weight correctly (reported by @fatalbanana)
* Remove extra reference retain of http connection on error
* Remove deprecated options from the default config
* Add `one_shot` attr to metric's symbols
* Doc: add documentation for metrics
* Add Upstart job to debian packaging (by @CameronNemo)
* Config: improve SURBL symbols descriptions (by @citrin)
* Config: reflect SURBL changes (by @citrin):
    - Outblaze removed, malware moved to separate list:
http://www.surbl.org/news/internal/MW-malware-sublist-added-to-multi
* Fix C modules initialization on restart
* Treat single IP as a single IP in radix lists (reported by @citrin)
* Do not touch file and core limits if not asked explicitly (reported by @citrin)
* Improve logging for fuzzy errors
* Block SIGPIPE for HTTP writing
* Doc: update manual pages
* Fix HTTP connection termination
* Reduce default number of parallel requests to 8
* Sync with libucl include features

Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
10 years agoUpdate rpm spec.
Vsevolod Stakhov [Wed, 15 Oct 2014 16:29:58 +0000 (17:29 +0100)]
Update rpm spec.

10 years agoAdded by a mistake.
Vsevolod Stakhov [Wed, 15 Oct 2014 16:29:38 +0000 (17:29 +0100)]
Added by a mistake.

10 years agoUpdate debian packaging.
Vsevolod Stakhov [Wed, 15 Oct 2014 16:29:01 +0000 (17:29 +0100)]
Update debian packaging.

10 years agoUpdate to 0.7.2.
Vsevolod Stakhov [Wed, 15 Oct 2014 16:18:46 +0000 (17:18 +0100)]
Update to 0.7.2.

10 years agoFix macro definition.
Vsevolod Stakhov [Wed, 15 Oct 2014 16:18:24 +0000 (17:18 +0100)]
Fix macro definition.

10 years agoSync libucl.
Vsevolod Stakhov [Wed, 15 Oct 2014 16:16:23 +0000 (17:16 +0100)]
Sync libucl.

10 years agoReduce default number of parallel requests to 8.
Vsevolod Stakhov [Wed, 15 Oct 2014 14:29:28 +0000 (15:29 +0100)]
Reduce default number of parallel requests to 8.

10 years agoCheck part->fuzzy before dereferencing.
Vsevolod Stakhov [Wed, 15 Oct 2014 13:37:20 +0000 (14:37 +0100)]
Check part->fuzzy before dereferencing.

10 years agoFix HTTP connection termination.
Vsevolod Stakhov [Wed, 15 Oct 2014 13:35:20 +0000 (14:35 +0100)]
Fix HTTP connection termination.

If a remote side terminates a connection, then notify and destroy
client as well.

10 years agoBlock SIGPIPE for HTTP writing.
Vsevolod Stakhov [Wed, 15 Oct 2014 13:21:57 +0000 (14:21 +0100)]
Block SIGPIPE for HTTP writing.

10 years agoUpdate manual pages.
Vsevolod Stakhov [Wed, 15 Oct 2014 13:14:07 +0000 (14:14 +0100)]
Update manual pages.

10 years agoImprove logging for fuzzy errors.
Vsevolod Stakhov [Wed, 15 Oct 2014 12:40:51 +0000 (13:40 +0100)]
Improve logging for fuzzy errors.

Suggested by: @citrin

10 years agoSend controller reply now accepts variadic args.
Vsevolod Stakhov [Wed, 15 Oct 2014 12:35:09 +0000 (13:35 +0100)]
Send controller reply now accepts variadic args.

10 years agoImplement va_list version for gstring printf.
Vsevolod Stakhov [Wed, 15 Oct 2014 12:23:39 +0000 (13:23 +0100)]
Implement va_list version for gstring printf.

10 years agoDo not touch limits if not asked explicitly.
Vsevolod Stakhov [Mon, 13 Oct 2014 16:46:46 +0000 (17:46 +0100)]
Do not touch limits if not asked explicitly.

Reported by: @citrin

10 years agoTreat single IP as a single IP.
Vsevolod Stakhov [Mon, 13 Oct 2014 16:44:11 +0000 (17:44 +0100)]
Treat single IP as a single IP.

Issue: #106
Reported by: @citrin

10 years agoRemove old controller's code.
Vsevolod Stakhov [Sun, 12 Oct 2014 21:35:36 +0000 (22:35 +0100)]
Remove old controller's code.

10 years agoFix C modules initialization on restart.
Vsevolod Stakhov [Fri, 10 Oct 2014 22:41:34 +0000 (23:41 +0100)]
Fix C modules initialization on restart.

10 years agoFix inserting to the hash.
Vsevolod Stakhov [Fri, 10 Oct 2014 21:18:37 +0000 (22:18 +0100)]
Fix inserting to the hash.

10 years agoMerge pull request #108 from citrin/master
Vsevolod Stakhov [Fri, 10 Oct 2014 19:42:02 +0000 (20:42 +0100)]
Merge pull request #108 from citrin/master

Config: SURBL update

10 years agoConfig: reflect SURBL changes. 108/head
Anton Yuzhaninov [Fri, 10 Oct 2014 15:58:25 +0000 (19:58 +0400)]
Config: reflect SURBL changes.

Outblaze removed, malware moved to separate list:
http://www.surbl.org/news/internal/MW-malware-sublist-added-to-multi

10 years agoConfig: improve SURBL symbols descriptions
Anton Yuzhaninov [Fri, 10 Oct 2014 15:51:32 +0000 (19:51 +0400)]
Config: improve SURBL symbols descriptions

10 years agoMerge pull request #84 from CameronNemo/patch-1
Vsevolod Stakhov [Fri, 10 Oct 2014 11:18:38 +0000 (12:18 +0100)]
Merge pull request #84 from CameronNemo/patch-1

Add Upstart job to debian packaging

10 years agoFix formulas.
Vsevolod Stakhov [Thu, 9 Oct 2014 00:24:14 +0000 (01:24 +0100)]
Fix formulas.

10 years agoUse math environment for formula.
Vsevolod Stakhov [Thu, 9 Oct 2014 00:18:26 +0000 (01:18 +0100)]
Use math environment for formula.

10 years agoDocument `one_shot` option.
Vsevolod Stakhov [Thu, 9 Oct 2014 00:15:06 +0000 (01:15 +0100)]
Document `one_shot` option.

10 years agoHonor one_shot setting for symbols.
Vsevolod Stakhov [Thu, 9 Oct 2014 00:11:28 +0000 (01:11 +0100)]
Honor one_shot setting for symbols.

10 years agoPlace symbol definition in metric->symbols hash.
Vsevolod Stakhov [Thu, 9 Oct 2014 00:08:58 +0000 (01:08 +0100)]
Place symbol definition in metric->symbols hash.

10 years agoAdd `one_shot` attr to metric symbols.
Vsevolod Stakhov [Wed, 8 Oct 2014 23:54:39 +0000 (00:54 +0100)]
Add `one_shot` attr to metric symbols.

10 years agoAdd metrics configuration.
Vsevolod Stakhov [Wed, 8 Oct 2014 23:50:32 +0000 (00:50 +0100)]
Add metrics configuration.

10 years agoRemove unused symbol R_IP_PBL.
Vsevolod Stakhov [Wed, 8 Oct 2014 17:26:56 +0000 (18:26 +0100)]
Remove unused symbol R_IP_PBL.

10 years agoRemove deprecated options from the default config.
Vsevolod Stakhov [Wed, 8 Oct 2014 17:25:55 +0000 (18:25 +0100)]
Remove deprecated options from the default config.

10 years agoRemove extra ref of http connection on error.
Vsevolod Stakhov [Wed, 8 Oct 2014 15:41:08 +0000 (16:41 +0100)]
Remove extra ref of http connection on error.

10 years agoPrint filename properly.
Vsevolod Stakhov [Wed, 8 Oct 2014 15:29:51 +0000 (16:29 +0100)]
Print filename properly.

10 years agoSync rdns and ucl.
Vsevolod Stakhov [Tue, 7 Oct 2014 17:05:08 +0000 (18:05 +0100)]
Sync rdns and ucl.

10 years agoProcess fuzzy weight correctly.
Vsevolod Stakhov [Tue, 7 Oct 2014 17:04:38 +0000 (18:04 +0100)]
Process fuzzy weight correctly.

10 years agoRemove unused code.
Vsevolod Stakhov [Mon, 6 Oct 2014 22:51:24 +0000 (23:51 +0100)]
Remove unused code.

10 years agoIncrease control connections counter.
Vsevolod Stakhov [Sat, 4 Oct 2014 19:44:55 +0000 (20:44 +0100)]
Increase control connections counter.

10 years agoFix some more border cases.
Vsevolod Stakhov [Thu, 2 Oct 2014 23:54:58 +0000 (00:54 +0100)]
Fix some more border cases.