]> source.dussan.org Git - rspamd.git/log
rspamd.git
9 years agoAdd preliminary version of generic LUA TCP requests.
Vsevolod Stakhov [Mon, 18 May 2015 15:09:03 +0000 (16:09 +0100)]
Add preliminary version of generic LUA TCP requests.

9 years agoFix error on spawning unique workers.
Vsevolod Stakhov [Mon, 18 May 2015 12:22:18 +0000 (13:22 +0100)]
Fix error on spawning unique workers.

Issue: #282
Reported by: @moisseev

9 years agoRelease 0.9.1. 0.9.1
Vsevolod Stakhov [Sun, 17 May 2015 15:24:08 +0000 (16:24 +0100)]
Release 0.9.1.

* Restore utf8 validation for regular expressions to avoid crashes
* Fix symbols displaying in the interface
* Add symbol groups to the interface
* Fix maps ID parsing in the controller
* Add multimap and regexp modules documentation
* Backport fixes from libucl
* Fix debian package (by @dottedmag)
* Rework XXH32 invocations

Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
9 years agoAdopt xxhash invocation.
Vsevolod Stakhov [Sun, 17 May 2015 15:11:12 +0000 (16:11 +0100)]
Adopt xxhash invocation.

9 years agoDo not free xxh32_state to work with static states.
Vsevolod Stakhov [Sun, 17 May 2015 15:10:47 +0000 (16:10 +0100)]
Do not free xxh32_state to work with static states.

9 years agoBackport fixes from libucl.
Vsevolod Stakhov [Sun, 17 May 2015 15:10:29 +0000 (16:10 +0100)]
Backport fixes from libucl.

9 years agoMerge pull request #281 from dottedmag/master
Vsevolod Stakhov [Sun, 17 May 2015 14:36:02 +0000 (15:36 +0100)]
Merge pull request #281 from dottedmag/master

Debian changes for 0.9.x

9 years agoUpdate changelog 281/head
Mikhail Gusarov [Sat, 16 May 2015 18:56:53 +0000 (18:56 +0000)]
Update changelog

9 years agoUpdate debian/copyright
Mikhail Gusarov [Sat, 16 May 2015 23:44:10 +0000 (23:44 +0000)]
Update debian/copyright

9 years agoRestart rspamd on upgrade, stop on removal
Mikhail Gusarov [Sat, 16 May 2015 19:28:57 +0000 (19:28 +0000)]
Restart rspamd on upgrade, stop on removal

9 years agoRestore utf8 validation as it causes segfaults sometimes.
Vsevolod Stakhov [Fri, 15 May 2015 15:36:15 +0000 (16:36 +0100)]
Restore utf8 validation as it causes segfaults sometimes.

9 years agoSync from interface.
Vsevolod Stakhov [Fri, 15 May 2015 14:21:04 +0000 (15:21 +0100)]
Sync from interface.

9 years agoFix symbols iteration.
Vsevolod Stakhov [Fri, 15 May 2015 14:20:34 +0000 (15:20 +0100)]
Fix symbols iteration.

9 years agoAnother sync.
Vsevolod Stakhov [Fri, 15 May 2015 14:04:03 +0000 (15:04 +0100)]
Another sync.

9 years agoSync with interface.
Vsevolod Stakhov [Fri, 15 May 2015 14:00:15 +0000 (15:00 +0100)]
Sync with interface.

9 years agoFix symbols output in the webui.
Vsevolod Stakhov [Fri, 15 May 2015 13:56:31 +0000 (14:56 +0100)]
Fix symbols output in the webui.

9 years agoFix misprint.
Vsevolod Stakhov [Fri, 15 May 2015 12:38:36 +0000 (13:38 +0100)]
Fix misprint.

9 years agoFix getting map ID.
Vsevolod Stakhov [Fri, 15 May 2015 12:24:54 +0000 (13:24 +0100)]
Fix getting map ID.

9 years agoAdd preliminary regexp module documentation.
Vsevolod Stakhov [Thu, 14 May 2015 18:08:51 +0000 (19:08 +0100)]
Add preliminary regexp module documentation.

9 years agoAdd multimap documentation.
Vsevolod Stakhov [Thu, 14 May 2015 17:33:07 +0000 (18:33 +0100)]
Add multimap documentation.

9 years agoRelease 0.9.0 0.9.0
Vsevolod Stakhov [Wed, 13 May 2015 21:04:47 +0000 (22:04 +0100)]
Release 0.9.0

* Add support of the fast and secure protocol level encryption:
- curve25519 is used for key exchange;
- chacha20/poly1305 cryptobox construction for bulk encryption;
- zero latency overhead;
- encrypting and balancing HTTP proxy worker
* Rework expressions and create new expressions library:
- aggressive optimizations based on the abstract syntax tree;
- abstract expressions support (regular expressions, functions, lua modules
composites and so on)
- New comparision and '+' operators support
- New greedy algorithm to minimize execution time of expressions and
all symbols
- Dynamic expressions benchmark and reoptimizations
* Many improvements to the LUA API:
- reworked logger module allowing to do pretty print of the most of lua
types (including tables and userdata classes)
- reworked lua redis and lua HTTP to support more features
- added opaque type for passing large text chunks without copying
- new regexp module with many auxiliary functions (e.g. `re:split`)
* LuaJIT is now the default requirement for rspamd allowing to speed up lua
execution by a large margin (however, plain lua is still supported)
* New plugins:
- spamassassin rules plugin that allows to load and re-use the most of
SA rules natively
- DMARC plugin that evaluates SPF and DKIM policies to the domain policies
- many old plugins has been reworked to implement new features and improve
stability
* New aho-corasic trie implementation from @mischasan that allows to load and
use hundreds of thousands of patterns with no influence on load
* Support of PCRE JIT and PCRE JIT fast path modes that significantly improves
the performance of regular expressions if supported by PCRE
* New URLs parser and extractor:
- removed legacy code that was useless for url finding
- reworked algorithms of URL parsing for more precise and accurate results
- added top-level-domains tree from http://publicsuffix.org
- improved emails parsing
- removed many phishing false positives due to TLD tree check
* New statistics infrastructure:
- created a separate layer of statistic library
- improved OSB-Bayes by re-weighting tokens according to the original
academic paper and `crm114` implementation, which reduced false positives
rate significantly
- created learn cache to avoid double learning of statistics and providing
an efficient way to re-learn class for a message
- created abstract layers for different statistics backends
- implemented new tokenization algorithms with fast or secure (siphash)
hashes to generate statistics features
* Reworked utf8 tokenization that previously corrupted all UTF8 words (minor
incompatibility with old fuzzy hashes with utf-8 symbols)
* SPF module has been completely rewritten to support complex cases of
`include` and `redirect` within SPF records
* DKIM module now supports multiple signatures
* Controller passwords can now be stored encrypted by `PBKDF2-HMAC` in the
configuration file
* Many hand-written HTTP clients has been replaced with the common rspamd
http module
* New test framework:
- import lua `telescope` test framework
- add unit tests for many rspamd modules and routines
- create a unit test for each possible bug found
- use luajit ffi for testing C code
- added preliminary support of functional testing by creating tasks from lua
* Randomize hash seed to avoid certain hash tables vulnerabilities
* Documentation improvements:
- added documentation for the vast majority of rspamd modules
- added documentation for rspamd protocol
- added documentation for the most of rspamd LUA extensions
* Fixed tonns of bugs and memory leaks
* Added tonns of minor features

Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
9 years agoBackport changelog entries from 0.8 branch.
Vsevolod Stakhov [Wed, 13 May 2015 20:18:22 +0000 (21:18 +0100)]
Backport changelog entries from 0.8 branch.

9 years agoMerge pull request #279 from fatalbanana/master
Vsevolod Stakhov [Wed, 13 May 2015 20:49:50 +0000 (21:49 +0100)]
Merge pull request #279 from fatalbanana/master

Update RPM BuildRequires for 0.9

9 years agoUpdate RPM BuildRequires for 0.9 279/head
Andrew Lewis [Wed, 13 May 2015 20:37:58 +0000 (22:37 +0200)]
Update RPM BuildRequires for 0.9

9 years agoFix spec for 0.9.0
Vsevolod Stakhov [Wed, 13 May 2015 20:15:02 +0000 (21:15 +0100)]
Fix spec for 0.9.0

9 years agoFix regexp splitting in lua.
Vsevolod Stakhov [Wed, 13 May 2015 16:41:50 +0000 (17:41 +0100)]
Fix regexp splitting in lua.

9 years agoPlug another memory leak.
Vsevolod Stakhov [Wed, 13 May 2015 15:41:18 +0000 (16:41 +0100)]
Plug another memory leak.

9 years agoMore workarounds for hiredis.
Vsevolod Stakhov [Wed, 13 May 2015 15:19:58 +0000 (16:19 +0100)]
More workarounds for hiredis.

9 years agoBreak deadlock in redis timeout logic.
Vsevolod Stakhov [Wed, 13 May 2015 14:30:31 +0000 (15:30 +0100)]
Break deadlock in redis timeout logic.

9 years agoDo not add invalid URLs from HTML tags.
Vsevolod Stakhov [Wed, 13 May 2015 14:10:52 +0000 (15:10 +0100)]
Do not add invalid URLs from HTML tags.

9 years agoFix method name in the documentation.
Vsevolod Stakhov [Wed, 13 May 2015 13:58:29 +0000 (14:58 +0100)]
Fix method name in the documentation.

9 years agoFix memory leak in DKIM.
Vsevolod Stakhov [Wed, 13 May 2015 13:52:18 +0000 (14:52 +0100)]
Fix memory leak in DKIM.

9 years agoOne more memory leak in lua_redis.
Vsevolod Stakhov [Wed, 13 May 2015 13:41:06 +0000 (14:41 +0100)]
One more memory leak in lua_redis.

9 years agoDo not add IP maps twice.
Vsevolod Stakhov [Wed, 13 May 2015 13:09:23 +0000 (14:09 +0100)]
Do not add IP maps twice.

9 years agoOutput some statistics about maps.
Vsevolod Stakhov [Wed, 13 May 2015 13:04:31 +0000 (14:04 +0100)]
Output some statistics about maps.

9 years agoFix critical bug with reading file maps.
Vsevolod Stakhov [Wed, 13 May 2015 13:04:11 +0000 (14:04 +0100)]
Fix critical bug with reading file maps.

9 years agoAdd routine to check radix trie size.
Vsevolod Stakhov [Wed, 13 May 2015 13:03:46 +0000 (14:03 +0100)]
Add routine to check radix trie size.

9 years agoPrint userdata in format 'class(address)'
Vsevolod Stakhov [Wed, 13 May 2015 12:44:55 +0000 (13:44 +0100)]
Print userdata in format 'class(address)'

9 years agoFix JITless world.
Vsevolod Stakhov [Wed, 13 May 2015 12:32:22 +0000 (13:32 +0100)]
Fix JITless world.

9 years agoFix another issues with raw/utf regexps.
Vsevolod Stakhov [Wed, 13 May 2015 12:00:52 +0000 (13:00 +0100)]
Fix another issues with raw/utf regexps.

9 years agoAllow to push raw pointer from the text boxed type.
Vsevolod Stakhov [Wed, 13 May 2015 10:22:03 +0000 (11:22 +0100)]
Allow to push raw pointer from the text boxed type.

9 years agoFix symbols removal in composites.
Vsevolod Stakhov [Wed, 13 May 2015 09:58:02 +0000 (10:58 +0100)]
Fix symbols removal in composites.

9 years agoFix forged recipients plugin for the modern rspamd.
Vsevolod Stakhov [Wed, 13 May 2015 09:57:14 +0000 (10:57 +0100)]
Fix forged recipients plugin for the modern rspamd.

9 years agoTry to load all lua modules ignoring failed ones.
Vsevolod Stakhov [Wed, 13 May 2015 09:53:28 +0000 (10:53 +0100)]
Try to load all lua modules ignoring failed ones.

9 years agoWhitelisted ip map should be radix, not hash.
Vsevolod Stakhov [Tue, 12 May 2015 18:19:54 +0000 (19:19 +0100)]
Whitelisted ip map should be radix, not hash.

9 years agoAdd documentation fro ratelimit module.
Vsevolod Stakhov [Tue, 12 May 2015 18:17:50 +0000 (19:17 +0100)]
Add documentation fro ratelimit module.

9 years agoRework composite symbols removal.
Vsevolod Stakhov [Tue, 12 May 2015 17:47:00 +0000 (18:47 +0100)]
Rework composite symbols removal.

Reported by: @fatalbanana

9 years agoSkip unparseable DKIM signatures.
Vsevolod Stakhov [Tue, 12 May 2015 14:40:14 +0000 (15:40 +0100)]
Skip unparseable DKIM signatures.

9 years agoFix zero-copy mode in lua_http (double free).
Vsevolod Stakhov [Tue, 12 May 2015 14:26:42 +0000 (15:26 +0100)]
Fix zero-copy mode in lua_http (double free).

9 years agoFix finding headers in HTTP messages.
Vsevolod Stakhov [Tue, 12 May 2015 14:11:42 +0000 (15:11 +0100)]
Fix finding headers in HTTP messages.

9 years agoFix coredump in a client (double free).
Vsevolod Stakhov [Tue, 12 May 2015 13:55:31 +0000 (14:55 +0100)]
Fix coredump in a client (double free).

9 years agoMore fixes to luajit detection.
Vsevolod Stakhov [Tue, 12 May 2015 12:24:03 +0000 (13:24 +0100)]
More fixes to luajit detection.

9 years agoFix converting of old storage.
Vsevolod Stakhov [Tue, 12 May 2015 11:51:36 +0000 (12:51 +0100)]
Fix converting of old storage.

9 years agoUse raw regexp instead of utf ones by default.
Vsevolod Stakhov [Tue, 12 May 2015 11:46:10 +0000 (12:46 +0100)]
Use raw regexp instead of utf ones by default.

Now to enable utf8 match, one should specify 'u' modifier, for example,
by adding '/u' to the slashed patterm.

9 years agoFix finding of luajit library.
Vsevolod Stakhov [Tue, 12 May 2015 11:10:42 +0000 (12:10 +0100)]
Fix finding of luajit library.

9 years agoRemove old radix code (no functional changes).
Vsevolod Stakhov [Tue, 12 May 2015 08:28:25 +0000 (09:28 +0100)]
Remove old radix code (no functional changes).

9 years agoAdd key rotation for http proxy.
Vsevolod Stakhov [Tue, 12 May 2015 08:16:21 +0000 (09:16 +0100)]
Add key rotation for http proxy.

9 years agoLibutil is useless everywhere but FreeBSD.
Vsevolod Stakhov [Mon, 11 May 2015 16:08:58 +0000 (17:08 +0100)]
Libutil is useless everywhere but FreeBSD.

9 years agoTry to fix libutil...
Vsevolod Stakhov [Mon, 11 May 2015 14:38:25 +0000 (15:38 +0100)]
Try to fix libutil...

9 years agoAnother workaround for pcre jit.
Vsevolod Stakhov [Mon, 11 May 2015 14:10:56 +0000 (15:10 +0100)]
Another workaround for pcre jit.

9 years agoAnother try to fixup the mess.
Vsevolod Stakhov [Mon, 11 May 2015 13:53:48 +0000 (14:53 +0100)]
Another try to fixup the mess.

9 years agoAnother iteration of libraries detection rework.
Vsevolod Stakhov [Mon, 11 May 2015 13:17:15 +0000 (14:17 +0100)]
Another iteration of libraries detection rework.

9 years agoAdd workaround for glib < 2.30.
Vsevolod Stakhov [Mon, 11 May 2015 13:17:04 +0000 (14:17 +0100)]
Add workaround for glib < 2.30.

9 years agoUnify search paths.
Vsevolod Stakhov [Mon, 11 May 2015 12:43:16 +0000 (13:43 +0100)]
Unify search paths.

9 years agoRelax glib requirements to 2.28.
Vsevolod Stakhov [Mon, 11 May 2015 12:06:42 +0000 (13:06 +0100)]
Relax glib requirements to 2.28.

9 years agoAdd hints for finding paths.
Vsevolod Stakhov [Mon, 11 May 2015 12:05:31 +0000 (13:05 +0100)]
Add hints for finding paths.

9 years agoDeal with multiple signatures in DKIM.
Vsevolod Stakhov [Mon, 11 May 2015 11:54:07 +0000 (12:54 +0100)]
Deal with multiple signatures in DKIM.

Issue: #254

9 years agoSet header value properly for decrypted headers.
Vsevolod Stakhov [Mon, 11 May 2015 09:05:16 +0000 (10:05 +0100)]
Set header value properly for decrypted headers.

9 years agoSet re limit from regexp module.
Vsevolod Stakhov [Sun, 10 May 2015 22:31:20 +0000 (23:31 +0100)]
Set re limit from regexp module.

9 years agoAdd configuration alias to avoid ambiguity.
Vsevolod Stakhov [Sat, 9 May 2015 21:55:52 +0000 (22:55 +0100)]
Add configuration alias to avoid ambiguity.

9 years agoAdd dkim module documentation.
Vsevolod Stakhov [Sat, 9 May 2015 21:53:59 +0000 (22:53 +0100)]
Add dkim module documentation.

9 years agoAdd SPF module documentation.
Vsevolod Stakhov [Sat, 9 May 2015 21:25:25 +0000 (22:25 +0100)]
Add SPF module documentation.

9 years agoAdd some documentation.
Vsevolod Stakhov [Sat, 9 May 2015 21:04:20 +0000 (22:04 +0100)]
Add some documentation.

9 years agoDie if some workers couldn't start.
Vsevolod Stakhov [Sat, 9 May 2015 20:56:35 +0000 (21:56 +0100)]
Die if some workers couldn't start.

9 years agoFix dealing with growing strings.
Vsevolod Stakhov [Sat, 9 May 2015 20:46:41 +0000 (21:46 +0100)]
Fix dealing with growing strings.

9 years agoAdd notice to fix issue with realloc.
Vsevolod Stakhov [Fri, 8 May 2015 21:00:57 +0000 (22:00 +0100)]
Add notice to fix issue with realloc.

9 years agoRestore parsing of headers.
Vsevolod Stakhov [Fri, 8 May 2015 20:53:32 +0000 (21:53 +0100)]
Restore parsing of headers.

9 years agoFix client's callback return result.
Vsevolod Stakhov [Fri, 8 May 2015 20:20:01 +0000 (21:20 +0100)]
Fix client's callback return result.

9 years agoMore fixes.
Vsevolod Stakhov [Fri, 8 May 2015 16:55:36 +0000 (17:55 +0100)]
More fixes.

9 years agoFixes to encrypted request writing.
Vsevolod Stakhov [Fri, 8 May 2015 16:07:32 +0000 (17:07 +0100)]
Fixes to encrypted request writing.

9 years agoImplement new scheme of encrypted requests reading.
Vsevolod Stakhov [Fri, 8 May 2015 15:02:14 +0000 (16:02 +0100)]
Implement new scheme of encrypted requests reading.

9 years agoMore fixes to encrypted writing.
Vsevolod Stakhov [Fri, 8 May 2015 13:42:01 +0000 (14:42 +0100)]
More fixes to encrypted writing.

9 years agoFix headers addition.
Vsevolod Stakhov [Fri, 8 May 2015 12:56:53 +0000 (13:56 +0100)]
Fix headers addition.

9 years agoCompletely rework HTTP encryption.
Vsevolod Stakhov [Fri, 8 May 2015 12:53:30 +0000 (13:53 +0100)]
Completely rework HTTP encryption.

Now we encrypt everything:
- URL
- Method
- Headers
- Body

9 years agoRework HTTP headers to avoid copying.
Vsevolod Stakhov [Fri, 8 May 2015 12:12:02 +0000 (13:12 +0100)]
Rework HTTP headers to avoid copying.

9 years agoFix documentation.
Vsevolod Stakhov [Fri, 8 May 2015 09:06:25 +0000 (10:06 +0100)]
Fix documentation.

9 years agoAdd some human readable aliases for rspamd configuration options.
Vsevolod Stakhov [Thu, 7 May 2015 17:40:50 +0000 (18:40 +0100)]
Add some human readable aliases for rspamd configuration options.

9 years agoProperly deal with negative scores and nice symbols.
Vsevolod Stakhov [Thu, 7 May 2015 17:34:36 +0000 (18:34 +0100)]
Properly deal with negative scores and nice symbols.

9 years agoDocument new option.
Vsevolod Stakhov [Thu, 7 May 2015 17:25:43 +0000 (18:25 +0100)]
Document new option.

9 years agoImplement match_limit in spamassassin plugin.
Vsevolod Stakhov [Thu, 7 May 2015 17:24:14 +0000 (18:24 +0100)]
Implement match_limit in spamassassin plugin.

9 years agoAdd function regexp:set_limit
Vsevolod Stakhov [Thu, 7 May 2015 17:16:34 +0000 (18:16 +0100)]
Add function regexp:set_limit

With this function it will be possible to limit the maximum size of text
to search using this regexp.

9 years agoOne more case to check.
Vsevolod Stakhov [Thu, 7 May 2015 15:09:09 +0000 (16:09 +0100)]
One more case to check.

9 years agoAdd fuzz tests and other split cases.
Vsevolod Stakhov [Thu, 7 May 2015 14:28:39 +0000 (15:28 +0100)]
Add fuzz tests and other split cases.

9 years agoFix splits.
Vsevolod Stakhov [Thu, 7 May 2015 14:28:29 +0000 (15:28 +0100)]
Fix splits.

9 years agoImprove regexp items parsing diagnostics.
Vsevolod Stakhov [Thu, 7 May 2015 13:13:32 +0000 (14:13 +0100)]
Improve regexp items parsing diagnostics.

9 years agoFix log message for unknown rules with scores.
Vsevolod Stakhov [Thu, 7 May 2015 12:04:08 +0000 (13:04 +0100)]
Fix log message for unknown rules with scores.

9 years agoCheck smtp recipients before indexing.
Vsevolod Stakhov [Thu, 7 May 2015 11:58:33 +0000 (12:58 +0100)]
Check smtp recipients before indexing.

9 years agoImplement unalgined tests.
Vsevolod Stakhov [Thu, 7 May 2015 11:55:29 +0000 (12:55 +0100)]
Implement unalgined tests.

9 years agoFix some unaligned cases.
Vsevolod Stakhov [Thu, 7 May 2015 11:55:20 +0000 (12:55 +0100)]
Fix some unaligned cases.

9 years agoAdd unit test for cryptobox.
Vsevolod Stakhov [Thu, 7 May 2015 11:25:11 +0000 (12:25 +0100)]
Add unit test for cryptobox.