aboutsummaryrefslogtreecommitdiffstats
path: root/ChangeLog
Commit message (Collapse)AuthorAgeFilesLines
* Release 1.1.31.1.3Vsevolod Stakhov2016-02-041-0/+16
| | | | | | | | | | | | | | | | | | | * Fix DSN rules when SMTP from is unavailable * Fix statconvert routine to avoid lua module usage * Set a sane quark for configtest to avoid NULL to be printed in logs * Support c11 if available * Fix parsing of ip:port strings * Add more diagnostic for lua subr errors * Fix task:set_from_ip lua method * Add basic routines for digital signatures * Add tool for digital signatures * Add plain open file API method for atomic open * Fix parsing nested braces inside logger vars * Pre filters now actually skip processing * Add pre-filter mode for multimap * Switch to apache 2 license Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.1.21.1.2Vsevolod Stakhov2016-01-291-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix stat_cache closing * Add checkpoints to sqlite3 learn cache * Do not recompile lua generated headers all the time * Increase number of messages learned * Fix issues with dual stack and hfilter * Disable MID checks for hfilter by default * Fix cache definitions in multiple classifier and no type * Don't crash if learn cache failed to initialize * Fix googlegroups support in maillist plugin * Rework flags LUA API: - Allow to check for a specific flag - Add `learn_spam`, `learn_ham` and `broken_headers` flags - Unify internal functions * Add `BROKEN_HEADERS` rule * Add support for forged confirmation headers (by @AdUser) * Allow `any`, `mime` and `smtp` for get_from/get_recipients * Add mime types checking plugin * Add rule to detect spammers attempts to cheat mime parsing * Rework parsing of IP addresses in configuration (better IPv6 support) * Add `util.parse_mail_address` function to LUA API * Add lua sqlite3 module * Implement synchronous redis call * Ratelimit: avoid possible indexing of nil value (Fixes #498) (by @fatalbanana) * Add stat_convert command to convert stats tokens from sqlite3 to redis * Implement redis advanced lua api with pipelining * Fix memory leak on redis stat (#500) * Fix user/language learn count in sqlite statistics (#496) (by @fatalbanana) * Fix build with custom pcre * Fix fuzzy relearning (#498) * Improve planning of asynchronous tasks * Show slow rules in log * Add warning for slow regexps * Add base32 decode/encode routines to lua util * Allow converting of learn cache from sqlite to redis * Add methods to check if a messages has from/rcpts * Improve and fix multimap plugin: - Restore 'header' maps - Add filters for headers - Add 'email:addr', 'email:user', 'email:domain' and 'email:name' filters - Add generic regexp filters * Disable reload command in rc scripts * Improve runtime CPU dispatcher for libcryptobox * Add preliminary support of digital signatures via ed25519 * Add detection for RDRAND support * Print configuration of crypto on start * A in SPF presumes AAAA lookup as well Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.1.11.1.1Vsevolod Stakhov2016-01-211-0/+15
| | | | | | | | | | | | | | | | | | * Fix duplicated XBL symbol * Reduce log severity for ratelimit missing servers * Fix XBL composite to avoid duplicate symbols * Reduce weight of URL_ONLY rule due to FP rate * Disable fuzzy hashes from the metadata for now * Fix processing of empty messages (#486) * Always treat DNS timeouts as temporary fail for SPF * Fix issue with SPF double IP stack (#483) * Use X-Forwarded-For when checking secure_ip (#488) * Fix hash calculation for sqlite stats * Fix memory corruption on punycode * Fix strings allocation in punycode * Fix error message (#491) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.1.01.1.0Vsevolod Stakhov2016-01-181-0/+118
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Incompatible change: sqlite3 and per_user behaviour: Now both redis and sqlite3 follows the common principles for per-user statistics: 1) If per-user statistics is enabled check per-user tokens ONLY 2) If per-user statistics is not enabled then check common tokens ONLY If you need old behaviour, then you'd need to use separate classifier for per-user statistics. * Implement redis statistics backend and cache * Implement autolearning for statistics * Reworked statistics architecture from scratch * Add hyperscan (https://github.com/01org/hyperscan) engine for regular expressions: - add lazy loader for hyperscan databases - rework regexp cache to have joint pcre/hyperscan scanning - implement hyperscan pre-filter support - add compilation guards for bad expressions - implement `rspamadm control recompile` command - implement hyperscan cache monitoring - slides: <https://highsecure.ru/rspamd-hyperscan.pdf> * Implement flexible task logging * Rework fuzzy worker: - it is now possible to run multiple fuzzy workers; - implement lazy writing as sqlite3 is bad at concurrent writing; - add retries for simple sql commands in fuzzy backend; - use fine-grained transactions for fuzzy; - implement new multi-pubkeys mode; - allow encrypted only storages; - rework statistics for fuzzy; - add `rspamadm control fuzzystat` command for extended statistics; - implement human readable output for the previous command; - add condition script for learning fuzzy storage; * Various fixes to SPF: - fix `redirect` records; - fix domains when parsing mx/ptr/a records in includes/redirects; - fix issues with multiple addresses in SPF records; - ignore SPF results in case of DNS failure; - adjust TTL of records when resolving subelements of SPF records; - always select `v=spf1` line if it is available - do not cache records with DNS failure in subrequests; - ignore records with temporary fails during subrequests resolving; - fix `RDNS_RC_NOREC` support; * Add clang plugin for static analysis: - implement static checks for `rspamd_printf` format strings; * Add 'allow_raw_input' option for non-mime messages * Recognize types using libmagic * Fix parsing of IPv6 received headers * Add new interface of communication between workers in rspamd * Add support for named socketpairs * Don't write URLs by default as it is too verbose * Set status for HTTP replies * Try load `rspamd.conf.override` * Implement words decaying for text parts to limit many checks * Improve support of SA rules and plugins: - add check_for_shifted_date and check_for_missing_to_header eval rules; - add 'check_relays_unparseable' support; - add `check_for_mime('mime_attachement')` function; - use new re_cache interface for all SA rules; - add support for `Mail::SpamAssassin::Plugin::MIMEHeader`; - add support of 'special' SA headers to `exists` function; - fix issue when SA metas contain other metas; - fix freemail rules; * Many fixes to the URL parser * Match any newline character in regexps * Fix resolving of upstreams and detection of poor IPv6 configurations * Parse upstreams selection algorithm from the configuration line * Add `reresolve` command to the control interface * Generate fuzzy hashes from task metadata (URLs and headers) * Add method to check if IP is local and `local_addrs` option * Implement forced timeout for delayed filters * Disable fast path of pcre-jit as it seems to be broken * Bayes fixes: - new normalizer function; - really use weights of tokens from the OSB algorithm; - restore multiple classifiers support; * Rules changes: - add `R_SUSPICIOUS_URL` rule that detects obfuscated URL's; - improve empty image rule; - rework `FORGED_RECIPIENTS` rule; - reduce weight of `SUSPICIOUS_RECIPS`; - fix `*_NORESOLVE_MX` symbols in hfilter; - add `SUBJ_ALL_CAPS` rule with support of UTF8 - add spamhaus SBL to uribl - fix `SUSPICIOUS_RECIPS` and `SORTED_RECIPS` rules - remove `R_TO_SEEMS_AUTO` as it generates a lot of FP; - add new Message-ID regexp for Thunderbird (by @moisseev); * Plugins changes: - allow ratelimit plugin to set symbol instead of pre-result - support IP DNS black lists for URIBL (e.g spamhaus SBL); - drop deprecated SURBL bits (by @fatalbanana) - rename `JP_SURBL_MULTI` to `ABUSE_SURBL` (by @fatalbanana) - add `SURBL_BLOCKED` (by @fatalbanana) - add `CR_SURBL` - SURBL: allow fallthrough to default symbol (by @fatalbanana) - Settings: fix IP match (by @fatalbanana) - SURBL: add missing symbols to metric (by @fatalbanana) - allow processing images urls for SURBL - unconditionally disable SPF for authenticated users and local networks * Rework ratelimit plugin - switch to `rates` instead of old and stupid strings to setup; - check if a bucket is zero and disable the corresponding limits' - turn off all buckets by default; - check either `rcpt` or `user` buckets, not all together' - document new `rates` and `symbol` options; - inform user about what buckets are used in the configuration; * Add neural network **experimental** plugin * Add a sample script to learn neural network from rspamd logs * Add documentation strings support to rspamd: - add strings for the main configuration options; - document workers options; - add internal plugin options; - create `rspamadm confighelp` routine; - implement human readable output for the previous command; - add subtree search support; - add keyword search support; * Documentation improvements, tutorials section, statistics description * Many other minor and major bugfixes not noted here Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.11Vsevolod Stakhov2015-12-211-0/+13
| | | | | | | | | | | | | | | | * Fix spf redirects * Fix domains when parsing mx/ptr/a records in includes/redirects * Fix unfolded base64 encoding * Fix GError use-after-free * Do not rewrite the original url when using redirector * Fix parsing of fragment in urls * Fix processing of HTML tags * Improve empty image rule * Avoid long double type * Fix tokens weights in OSB algorithm * Improve debugging for bayes Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.10Vsevolod Stakhov2015-11-061-0/+10
| | | | | | | | | | | | | * Fix settings application (#416) * Fix another issue with fixed strings * Fix hash function invocation * Use the proper string for make_dns_request in lua_http * Fix scan time output * Update webui: - fix labels for greylisting - fix dimension of scan time Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.91.0.9Vsevolod Stakhov2015-10-271-0/+4
| | | | | | | * Emergency fix in keyed blake2 to fix fuzzy hashes and encrypted password * Support passwords longer than 64 symbols Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.81.0.8Vsevolod Stakhov2015-10-261-0/+9
| | | | | | | | | | | | * Add function to traverse AST atoms * Allow dependencies on rspamd symbols for SA metas * Fix memory corruption when timeout is removed in fuzzy check * Fix encrypted fuzzy add processing * Avoid use-after-free in controller session destructor * Use session pool instead of task pool in fuzzy check * Fix assembly in i386 mode (#413, #412) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.71.0.7Vsevolod Stakhov2015-10-261-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Plugged memory leaks in internet address object & html parser * Fixed static build * Fixed multiple sigchld processing * Fixed deletion of signal events after event processing loop * Fixed build on ARM (#404 - reported by @Gottox) * Fixed setting the default mask for SPF. * Fixed sanitisation of HTTP query values * Fixed parsing of the last header in encrypted HTTP messages * Additions and fixes for test suite & benchmarks * Added openssl aes-256-gcm support to libcryptobox & HTTP server * Implemented support for starting multiple HTTP servers * Implemented batch accept in HTTP server * Added module to get data from HTTP headers (#285 - reported by @msimerson) * Added `rspamadm control` command * Added ability to sort counters output. * Added ability to specify custom headers for rspamc client * Fix architecture detection * Converted history storage to the UCL format * Allow flexible number of rows in history * Fix action badges in WebUI * Add universal cryptobox hash API * Migrated to the optimized blake2b implementation adopted from Andrew Moon * Allow explicit loading of specific modules * Always load settings module * Allow to add symbols from settings * Fix double free in the controller fuzzy learn command * Avoid endless loop when cannot open sqlite db * Updated libucl Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Update ChangeLogAndrew Lewis2015-10-231-0/+19
|
* Release 1.0.61.0.6Vsevolod Stakhov2015-10-161-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix build on i386 * Update CentOS7 service file patch (by @fatalbanana) * Fix path to rspamadm in Debian init script (by @fatalbanana) * Fix broken '_SC_GETPW_R_SIZE_MAX' on FreeBSD * Fix portability issues * Use cryptobox chacha for libottery * Better support of 32 bit builds * Fix header name tokens setup * Fix levenstein distance method for words * Add workaround for old libevent (#400) * Fix microseconds in termination timer * Fix some more issues with fixed strings * Explicitly test CPU instructions even after CPUID call * Do not check out of boundary memory * Do not output broken emails * Fix unknown symbols registration * Handle SIGILL using longjmp * Block signals when exiting event loop * Fix incorrect allocation size * Slightly optimize alignment * Restore rspamd -t for compatibility * Add more sanity checks for emails Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.51.0.5Vsevolod Stakhov2015-10-141-0/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add rspamd control interface: - support `stat` command to get runtime stats of rspamd workers - support `reload` command to reload runtime elements (e.g. sqlite3 databases) * Rework curve25519 library for modular design: - add Sandy2x implementation by Tung Chou - fix CPU detection for variables loading assembly - add testing for curve25519 ECDH * New fixed strings library * Add `R_SUSPICIOUS_IMAGES` rule * Enable mmap in sqlite3 * Use new strings in the HTTP code * Improve google perftools invocation * Improve performance profiling in http test * Reorganize includes to reduce namespace pollution * Allow specific sections printing in configdump command * Rework workers signals handlers to be chained if needed * Update socketpair utility function * Add control_path option for rspamd control protocol * Fix ownership when listening on UNIX sockets * Rework signals processing in main * Remove extra tools from rspamd (they live in rspamadm now) * Remove global rspamd_main * Add global timeout for the overall task processing (8 seconds by default) * Sanitize NULL values for fuzzy backend * Store NM between encrypt/decrypt * Add textpart:get_words_count method * Fix generic DNS request in lua * Tune hfilter weights * Add support of IPv6 in hfilter * Fix parsing of HTTP headers with IP addresses * Sync with the recent libucl * Various minor bugfixes Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.41.0.4Vsevolod Stakhov2015-10-051-0/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add configdump routine to rspamadm * Implement retransmits for fuzzy_check plugin * Fix events processing for learning anf checking fuzzy hashes * Avoid dependency on unneeded and uncompatible glib include * Add `historyreset` command to the controller * Fix loading of tokenizer config from dump (#389) * Add sorting hints for the history * Allow custom lua scripts for users/languages extraction (#388) * Do not add FORGED_RECIPIENTS when 'To' is missing (#387) * Do not add R_UNDISC_RCPT when 'To' is missing (#387) * Add encryption to fuzzy check plugin * Add encryption for fuzzy storage * Add new epoch for encrypted fuzzy request * Add encryption for `rspamd.com` storage * Remove gmime processing for LDA mode as it is deadly broken * Add routine to find end of headers position in mime messages * Fix LDA headers folding * Init libraries in rspamc client as well to avoid locale issues * Avoid collision with locally installed includes * Allocate and free memory with the same allocator in rspamadm (#385) * Preserve expired fuzzy hashes counter * Improvements in webui: - Add favicon.ico - Rework history table - Fix sorting for the history - Migrate to bootstrap 3 and jquery 2 - Fix css bugs - Add glyphicons - Add reset history - Improve history buttons - Redraw graph to avoid display issues - Webui is now MIT licensed to match licensing policy of rspamd Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.31.0.3Vsevolod Stakhov2015-09-281-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix piechart clean slice (#380) * Fix controller crashes when GString is reallocated (#381) * Correctly set locale before start * Set C locale for numeric values * Add rspamadm routine: - add `pw` command to manage passwords - add `help` command for displaying help - add `configtest` command to check configuration files - add `keypair` command for generating encryption keys - add `fuzzy_merge` routine to merge fuzzy sqlite databases - add a simple manual page for rspamadm * Allow metric registration for composite expressions * Add strict mode for configtest * Add logger counters * Save and show learned messages count (#383) * Add `no_stat` flag * Add `task:set_flag` and `task:get_flags` (#382) * Enable foreign keys in sqlite3 * Remove orphaned shingles from fuzzy storage * Optimize synchronization steps for fuzzy storage * Allow delayed conditions registration * Add lua API for conditions registering Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.21.0.2Vsevolod Stakhov2015-09-231-0/+16
| | | | | | | | | | | | | | | | | | | * Fix critical bug in webui that prevents password from being sent * Rework webui view: - Switch to d3.js for graphs - Improve piechart look - Rework colors for piechart - Fix layout for symbols - Fix refresh button * Add descriptions for whitelist maps * Fix build on arm (#379) * Fix issue with the last element in the radix trie * Add more tests for radix trie algorithm * Allow to extract URLs from query strings of other URLs (#361) * Initialize rrd fields before writing to file * Fix double free if no password has been specified Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.11.0.1Vsevolod Stakhov2015-09-221-0/+17
| | | | | | | | | | | | | | | | | | | | * Add writing to rrd from the controller * Fixed lots of bugs in rrd code * Adopt new DNS API in hfilter plugin (by @AlexeySa) * Allow only one controller process to manage rrd file * Set event base for fuzzy calls * Improve fuzzy IO errors logging * Add rra extraction function to rrd library * Add graph handler to the controller * Cache correct passwords to avoid too high CPU usage when working with webui * Controller sockets are owned by router do not export them to task * Optimize logging by skipping hash table search if it's empty * Fix loading issue whith broken statfiles * Print assertions from glib to rspamd logger * Load legacy `lua/rspamd.local.lua` * Update webui with some fixes to learning and scanning Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.0.01.0.0Vsevolod Stakhov2015-09-171-0/+89
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Rework symbols processing: - Improve sorting logic for symbols - Organize processing into multiple stages - Added asynchronous watchers for symbols - Added ability to organize dependencies between symbols * Fixed URL redirector: - Use optimized POE loop - Organize dependencies - Fix startup * New sqlite3 backend: - Allow to have per-languages and per-user statistics - Allow sqlite3 to be used as statistics backend * Store tokenizer configuration within statfiles * Improve bayes statistics: - Use headers and images metainformation in bayes - Suggest using of pre-processed tokens for statistics - Fix tokens normalization for OSB algorithm * Rewrite url parsing: - Fix numerous issues with url extraction and normalization - Fix mailto urls * Fix settings plugin to allow custom actions scores * Improve rbl plugin * Allow capturing patterns in rspamd lua regexp library * Add GTUBE support * Fix spamc legacy support * Add DKIM support to RBL module * Fix issues with multiple DKIM signatures * Fix issue if rspamd cannot create statfiles (#331) * Rework parts and task structure: - Now text_parts, parts and received are arrays - Pre-allocate arrays with some reasonable defaults - Use arrays instead of lists in plugins and checks - Remove unused fields from task structure - Rework mime_foreach callback function - Remove deprecated scan_milliseconds field * Add ip_score plugin support (not enabled by default): - Can check for asn/country and network using DNS lookups - Can store and load reputation from redis server * Improve PARTS_DIFFER rule to count merely different words * New HTML parser: - Parses HTML parts using a set of state machines - Extracts useful data and exports it to lua functions: + Styles + Images + URLs + Colors + Structure elements - Added HTML rules for some checks * New version of LUA DNS API * Table versions of many functions in LUA API * Improve rspamc client: - Print execution time - Allow executing of external commands and passing output to them - Allow mime output mode when rspamc alters message according to rspamd checks and send it to an external command or stdout * Allow scanning of local files using HTTP requests * Rework configuration system: - Rules are now moved from the $CONFDIR to $RULESDIR to avoid ambiguity - All modules configurations are now split in $CONFDIR/modules.d/* to simplify upgrades - Move hfilter to plugins - Allow plugins and rules to define default scores to simplify metrics setup - Include overrides for all modules to honor local/automatic parameters - Tune scores for many modules * Rework and enable DMARC plugin * Add whitelist plugin for SPF/DKIM/DMARC based whitelisting * Add some common domains to whitelists shipped with rspamd * Rework logging: - Now each log entry supports module name and a `tag`. Tag is used to identify unique objects (such as tasks) when checking log files - It is possible to turn on debugging for the specific modules - Systemd logging is fixed * Improve spamassassin plugin. - Now headers are matched more like SA - Improve support of Message-ID - Add support of ToCc header type - Fix :addr and :name in headers regexps * Resurrect rrd support code * Save controller stats between restarts * Fixed tonns of bugs * Added tonns of minor improvements and features * Added more unit tests * Create functional tests framework * Added documentation for missing modules * Added rpm/deb repositories and scripts * Updated WebUI and libucl externals Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Add 0.9.10 changelog entry.Vsevolod Stakhov2015-09-171-0/+9
|
* Sync log from 0.9 branch.Vsevolod Stakhov2015-08-091-0/+74
|
* Release 0.9.40.9.4Vsevolod Stakhov2015-05-211-0/+13
| | | | | | | | | | | | | | | | * Fix critical bugs in tokenization algorithm * Write unit tests for tokenization * Add documentation for lua_tcp * Switch off legacy tokenization by default. * Fix critical bugs in words normalization * Add lua bindings to tokenizer. * Implement storing of HTTP headers inside task * Add lua API to accerss HTTP headers data * Implemented base64 encoding suitable for MIME * Use caseless hash and equal functions for HTTP request headers. * Improve debian architectures support (by @dottedmag) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.9.30.9.3Vsevolod Stakhov2015-05-191-0/+4
| | | | | | | * Revert incorrect regexp change that broke the default rules * Fix lua_tcp module Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.9.20.9.2Vsevolod Stakhov2015-05-191-0/+10
| | | | | | | | | | | | | * Fix error on spawning unique workers. * Add preliminary version of generic LUA TCP requests API. * Use lua 5.1 if luajit is not available (Arm64, PowerPC, s390x etc) * Fix fuzzy mime strings with only type. * Improve thunderbird sanity checks. * Fix critical bug on matching regular expressions. * Make hiredis optional dependency. * Fix multiple bugs in daemon reloading Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.9.1.0.9.1Vsevolod Stakhov2015-05-171-0/+10
| | | | | | | | | | | | | * Restore utf8 validation for regular expressions to avoid crashes * Fix symbols displaying in the interface * Add symbol groups to the interface * Fix maps ID parsing in the controller * Add multimap and regexp modules documentation * Backport fixes from libucl * Fix debian package (by @dottedmag) * Rework XXH32 invocations Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.9.00.9.0Vsevolod Stakhov2015-05-131-0/+71
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add support of the fast and secure protocol level encryption: - curve25519 is used for key exchange; - chacha20/poly1305 cryptobox construction for bulk encryption; - zero latency overhead; - encrypting and balancing HTTP proxy worker * Rework expressions and create new expressions library: - aggressive optimizations based on the abstract syntax tree; - abstract expressions support (regular expressions, functions, lua modules composites and so on) - New comparision and '+' operators support - New greedy algorithm to minimize execution time of expressions and all symbols - Dynamic expressions benchmark and reoptimizations * Many improvements to the LUA API: - reworked logger module allowing to do pretty print of the most of lua types (including tables and userdata classes) - reworked lua redis and lua HTTP to support more features - added opaque type for passing large text chunks without copying - new regexp module with many auxiliary functions (e.g. `re:split`) * LuaJIT is now the default requirement for rspamd allowing to speed up lua execution by a large margin (however, plain lua is still supported) * New plugins: - spamassassin rules plugin that allows to load and re-use the most of SA rules natively - DMARC plugin that evaluates SPF and DKIM policies to the domain policies - many old plugins has been reworked to implement new features and improve stability * New aho-corasic trie implementation from @mischasan that allows to load and use hundreds of thousands of patterns with no influence on load * Support of PCRE JIT and PCRE JIT fast path modes that significantly improves the performance of regular expressions if supported by PCRE * New URLs parser and extractor: - removed legacy code that was useless for url finding - reworked algorithms of URL parsing for more precise and accurate results - added top-level-domains tree from http://publicsuffix.org - improved emails parsing - removed many phishing false positives due to TLD tree check * New statistics infrastructure: - created a separate layer of statistic library - improved OSB-Bayes by re-weighting tokens according to the original academic paper and `crm114` implementation, which reduced false positives rate significantly - created learn cache to avoid double learning of statistics and providing an efficient way to re-learn class for a message - created abstract layers for different statistics backends - implemented new tokenization algorithms with fast or secure (siphash) hashes to generate statistics features * Reworked utf8 tokenization that previously corrupted all UTF8 words (minor incompatibility with old fuzzy hashes with utf-8 symbols) * SPF module has been completely rewritten to support complex cases of `include` and `redirect` within SPF records * DKIM module now supports multiple signatures * Controller passwords can now be stored encrypted by `PBKDF2-HMAC` in the configuration file * Many hand-written HTTP clients has been replaced with the common rspamd http module * New test framework: - import lua `telescope` test framework - add unit tests for many rspamd modules and routines - create a unit test for each possible bug found - use luajit ffi for testing C code - added preliminary support of functional testing by creating tasks from lua * Randomize hash seed to avoid certain hash tables vulnerabilities * Documentation improvements: - added documentation for the vast majority of rspamd modules - added documentation for rspamd protocol - added documentation for the most of rspamd LUA extensions * Fixed tonns of bugs and memory leaks * Added tonns of minor features Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Backport changelog entries from 0.8 branch.Vsevolod Stakhov2015-05-131-0/+65
|
* Release 0.8.00.8.0Vsevolod Stakhov2015-01-021-0/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * New fuzzy check logic: - use shingles algorithm for fuzzy matching - use blake2 instead of md5 for larger output space - combine fuzzy and strict matching - allow to organize private storages by means of keys - preserve compatibility with previous versions * New fuzzy storage: - use sqlite instead of own memory based hash tables - rework commands interface - add conversion from the old format - add fuzzy match by shignles - support old rspamd versions * Add lemmatizing for words used in fuzzy hashes that allows to improve match quality by using of the first forms of all words * Rework language detection * Fix several critical bugs, memory leaks and deadlocks: - memory leak in HTML nodes parsing - deadlock in logger code - deadlock in signals processing - crashes in fuzzy_storage - crashes in tokenizers if the input was empty * Import new libucl with several bugfixes and improvements * Support listening on ipv6 addresses only * Fix macro expansion in SPF module * Several bugfixes in DKIM module * Add load headers support for mime parts to the lua API * Add documentation for: - workers in general - fuzzy_storage worker - fuzzy_check plugin - mimepart and textpart lua API modules Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.7.60.7.6Vsevolod Stakhov2014-11-241-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Apply boundary fix for dkim simple canonization * Fix ping command * Return nil if header was not found in lua_task * Fix hang in upstreams revive logic * Decode entitles when normalizing HTML parts * Fix logic of finding URLs in HTML parts * Do not include \0 into length of text when performing conversion to utf8 * Fix raw vs parsed reperesentations Raw parts are now: - decoded b64/qp, but *NOT* converted to utf-8 Processed parts are now: - converted to UTF-8 - normalized if needed (e.g. HTML tags are stripped) * Rework DKIM canonization to line based * Fix fuzzy hashes addding * Use more specific hash function for fuzzy * Fix leaking of iconv descriptors * Fix PTR resolving in lua resolver * Rework spf module. - Copy data to memory pool as cached record might be destroyed causing freed memory being passed to the protocol output (use after free) - Allow SPF_NEUTRAL policy to be handled separately - Add R_SPF_NEUTRAL to the default config * Rework `register_symbols` function * Allow to disable components of hfilter Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.7.50.7.5Vsevolod Stakhov2014-11-171-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | * Fix owner when creating folder /run/rspamd (by @sfirmery) * Fix IP validity checks * Decode URLs obtained from HTML tags * Fix crash with unweighted upstreams * Stop processing headers in parts * Set sockaddr.sa_family properly when connectig to upstreams * Fix reload issues in surbl and fuzzy_check (reported by @citrin) * Fix timeouts in redirector * Improve lua errors reporting * Fix lua closures processing in libucl * Rework calling of lua functions from regexp module * Choose raw regexp for raw headers * Rework conversion to utf since glib one is broken * Ignore SGML style tags in html * Fix old bug with non-capturing https urls * Fix memory corruption on fuzzy reload (reported by @citrin) * Fix percents display in rspamc * Fix buffer update for DKIM * Do not validate utf for raw headers Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.7.40.7.4Vsevolod Stakhov2014-11-081-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | * Fix build under *BSD * Detect HAN unicode script * Implement language detection heuristic for text parts * Fix time output in history * Improve piechart coloring * Fix \r\n conversion in DKIM module (reported by @citrin) * Try to detect systems with no IPv6 support * Fix multiple/single values in use settings (reported by @citrin) * Rework IP addresses in upstreams: - Select ipv4/unix addresses if they exist and use ipv6 for ipv6 only upstreams (since the support of ipv6 is poor in many OSes and environments) - Free IP list on upstream destruction - Add test cases for addresses selection - Allow adding of free form IP addresses to upstreams * Fix endiannes in lua_radix search (reported by @citrin) * Soft shutdown should also set wanna_die flag (reported by @citrin) * Stop use-after-free in event loop termination * Fix processing of very short messages in DKIM (reported by @citrin) * Detect systems without shared mutexes * Fix issues with PTR and MX elements in SPF parser (reported by @citrin) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.7.3.0.7.3Vsevolod Stakhov2014-11-031-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * New upstreams code: - simplify upstreams API; - unify strings parsing in upstreams definition; - add configuration options for the upstreams; - for failed upstreams re-resolve their addresses; - use all resolved addresses for an upstream (round-robin); - implement stable hashing and use it by default for upstreams; - add unit test for upstreams module. * Rework signals processing in all rspamd workers: - signals are now processed in the event loop; - implement the most common signal handlers for all workers; - add callbacks for workers specific signal handlers * Fix critical issue with fuzzy storage: Fuzzy stroage could not save any hashes on termination due to bugged signals handling * Fix roll history IP storage * Rework ipv4/ipv6 handling in parsing addresses: - turn off support of IPV6_V6ONLY socket option; - create ipv6 socket prior to ipv4 one to handle systems with v6/v4 sockets enabled (Linux) * Remove CBL as it's wholly included in Spamhaus XBL (by @fatalbanana) * Remove nszones.com fake RBL (by @citrin) * Fix upstreams interaction for fuzzy_check * Verify spf PTR records (reported by @citrin) * Fix spf MX records parsing * Add compatibility for old libevent (by @yellowman) * Sync bugfixes from libucl Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.7.2.0.7.2Vsevolod Stakhov2014-10-151-0/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Convert all maps to the compressed radix trie * Allow IPv6 addresses in IP maps * Remove dynamic items support from symbols cache * Allow hex encoded output of strings * Fix bug with control connections count * Process fuzzy weight correctly (reported by @fatalbanana) * Remove extra reference retain of http connection on error * Remove deprecated options from the default config * Add `one_shot` attr to metric's symbols * Doc: add documentation for metrics * Add Upstart job to debian packaging (by @CameronNemo) * Config: improve SURBL symbols descriptions (by @citrin) * Config: reflect SURBL changes (by @citrin): - Outblaze removed, malware moved to separate list: http://www.surbl.org/news/internal/MW-malware-sublist-added-to-multi * Fix C modules initialization on restart * Treat single IP as a single IP in radix lists (reported by @citrin) * Do not touch file and core limits if not asked explicitly (reported by @citrin) * Improve logging for fuzzy errors * Block SIGPIPE for HTTP writing * Doc: update manual pages * Fix HTTP connection termination * Reduce default number of parallel requests to 8 * Sync with libucl include features Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 0.7.1.0.7.1Vsevolod Stakhov2014-09-301-0/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix typo in stat output. * Fix issues with includes crossing with the system includes * Restore testing framework * Add radix trie test suite * Implement new path-compressed radix trie. - The performance benefit over the old algorithm is about 1.5 times. - Memory usage is significantly lower as well. - Now radix trie can accept any IPv4/IPv6 values * Various improvements to the memory pools code * Fix writing reply to a client when no filters are defined * Write base32 encoded fuzzy * Fix 'soft reject' action * Fix rspamd reload and modules reconfiguration * Fix subject rewriting for the default subject * Fix states for processing task and pre-filters * Fix issues with connection closing * Fix crashes in rdns * Fix ratelimit pre-filter * Update exim patch. - Update to the recent exim version - Strip extra leading src/ from the patch - Remove sendfile since it was broken - Fix rspamd spam report for exim * Improve documentation Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Update to 0.7.0.Vsevolod Stakhov2014-09-111-0/+29
| | | | Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Update ChangeLog for 0.6.8.Vsevolod Stakhov2014-02-031-0/+16
|
* Stop for 0.6.7.0.6.7Vsevolod Stakhov2014-01-101-0/+14
|
* Update changelogs.0.6.6Vsevolod Stakhov2013-12-271-0/+12
|
* Update to 0.6.5.0.6.5Vsevolod Stakhov2013-12-201-0/+6
|
* Forgot to commit.Vsevolod Stakhov2013-12-181-0/+12
|
* 0.6.3 release.0.6.3Vsevolod Stakhov2013-12-101-0/+10
|
* Update changelog.Vsevolod Stakhov2013-12-061-0/+19
|
* Add 0.6.0 ChangeLog entry.Vsevolod Stakhov2013-11-301-0/+16
|
* Detect exp2l as well. Update to 0.5.6.Vsevolod Stakhov2013-06-101-0/+7
|
* Update ChangeLog.Vsevolod Stakhov2013-06-051-0/+7
|
* Polish debian copyright, remove unused CGI.0.5.4Vsevolod Stakhov2013-02-051-0/+8
| | | | Update ChangeLog.
* Update ChangeLogVsevolod Stakhov2012-11-211-0/+9
|
* Update to 0.5.2.Vsevolod Stakhov2012-08-231-0/+11
|
* Add changelog, update debian changelog.Vsevolod Stakhov2012-08-061-0/+274
Add tag for 0.5.1.
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-26 13:12:47 +0000