1.0.10: * Fix settings application (#416) * Fix another issue with fixed strings * Fix hash function invocation * Use the proper string for make_dns_request in lua_http * Fix scan time output * Update webui: - fix labels for greylisting - fix dimension of scan time 1.0.9: * Emergency fix in keyed blake2 to fix fuzzy hashes and encrypted password * Support passwords longer than 64 symbols 1.0.8: * Add function to traverse AST atoms * Allow dependencies on rspamd symbols for SA metas * Fix memory corruption when timeout is removed in fuzzy check * Fix encrypted fuzzy add processing * Avoid use-after-free in controller session destructor * Use session pool instead of task pool in fuzzy check * Fix assembly in i386 mode (#413, #412) 1.0.7: * Plugged memory leaks in internet address object & html parser * Fixed static build * Fixed multiple sigchld processing * Fixed deletion of signal events after event processing loop * Fixed build on ARM (#404 - reported by @Gottox) * Fixed setting the default mask for SPF. * Fixed sanitisation of HTTP query values * Fixed parsing of the last header in encrypted HTTP messages * Additions and fixes for test suite & benchmarks * Added openssl aes-256-gcm support to libcryptobox & HTTP server * Implemented support for starting multiple HTTP servers * Implemented batch accept in HTTP server * Added module to get data from HTTP headers (#285 - reported by @msimerson) * Added `rspamadm control` command * Added ability to sort counters output. * Added ability to specify custom headers for rspamc client * Fix architecture detection * Converted history storage to the UCL format * Allow flexible number of rows in history * Fix action badges in WebUI * Add universal cryptobox hash API * Migrated to the optimized blake2b implementation adopted from Andrew Moon * Allow explicit loading of specific modules * Always load settings module * Allow to add symbols from settings * Fix double free in the controller fuzzy learn command * Avoid endless loop when cannot open sqlite db * Updated libucl 1.0.6: * Fix build on i386 * Update CentOS7 service file patch (by @fatalbanana) * Fix path to rspamadm in Debian init script (by @fatalbanana) * Fix broken '_SC_GETPW_R_SIZE_MAX' on FreeBSD * Fix portability issues * Use cryptobox chacha for libottery * Better support of 32 bit builds * Fix header name tokens setup * Fix levenstein distance method for words * Add workaround for old libevent (#400) * Fix microseconds in termination timer * Fix some more issues with fixed strings * Explicitly test CPU instructions even after CPUID call * Do not check out of boundary memory * Do not output broken emails * Fix unknown symbols registration * Handle SIGILL using longjmp * Block signals when exiting event loop * Fix incorrect allocation size * Slightly optimize alignment * Restore rspamd -t for compatibility * Add more sanity checks for emails 1.0.5: * Add rspamd control interface: - support `stat` command to get runtime stats of rspamd workers - support `reload` command to reload runtime elements (e.g. sqlite3 databases) * Rework curve25519 library for modular design: - add Sandy2x implementation by Tung Chou - fix CPU detection for variables loading assembly - add testing for curve25519 ECDH * New fixed strings library * Add `R_SUSPICIOUS_IMAGES` rule * Enable mmap in sqlite3 * Use new strings in the HTTP code * Improve google perftools invocation * Improve performance profiling in http test * Reorganize includes to reduce namespace pollution * Allow specific sections printing in configdump command * Rework workers signals handlers to be chained if needed * Update socketpair utility function * Add control_path option for rspamd control protocol * Fix ownership when listening on UNIX sockets * Rework signals processing in main * Remove extra tools from rspamd (they live in rspamadm now) * Remove global rspamd_main * Add global timeout for the overall task processing (8 seconds by default) * Sanitize NULL values for fuzzy backend * Store NM between encrypt/decrypt * Add textpart:get_words_count method * Fix generic DNS request in lua * Tune hfilter weights * Add support of IPv6 in hfilter * Fix parsing of HTTP headers with IP addresses * Sync with the recent libucl * Various minor bugfixes 1.0.4: * Add configdump routine to rspamadm * Implement retransmits for fuzzy_check plugin * Fix events processing for learning anf checking fuzzy hashes * Avoid dependency on unneeded and uncompatible glib include * Add `historyreset` command to the controller * Fix loading of tokenizer config from dump (#389) * Add sorting hints for the history * Allow custom lua scripts for users/languages extraction (#388) * Do not add FORGED_RECIPIENTS when 'To' is missing (#387) * Do not add R_UNDISC_RCPT when 'To' is missing (#387) * Add encryption to fuzzy check plugin * Add encryption for fuzzy storage * Add new epoch for encrypted fuzzy request * Add encryption for `rspamd.com` storage * Remove gmime processing for LDA mode as it is deadly broken * Add routine to find end of headers position in mime messages * Fix LDA headers folding * Init libraries in rspamc client as well to avoid locale issues * Avoid collision with locally installed includes * Allocate and free memory with the same allocator in rspamadm (#385) * Preserve expired fuzzy hashes counter * Improvements in webui: - Add favicon.ico - Rework history table - Fix sorting for the history - Migrate to bootstrap 3 and jquery 2 - Fix css bugs - Add glyphicons - Add reset history - Improve history buttons - Redraw graph to avoid display issues - Webui is now MIT licensed to match licensing policy of rspamd 1.0.3: * Fix piechart clean slice (#380) * Fix controller crashes when GString is reallocated (#381) * Correctly set locale before start * Set C locale for numeric values * Add rspamadm routine: - add `pw` command to manage passwords - add `help` command for displaying help - add `configtest` command to check configuration files - add `keypair` command for generating encryption keys - add `fuzzy_merge` routine to merge fuzzy sqlite databases - add a simple manual page for rspamadm * Allow metric registration for composite expressions * Add strict mode for configtest * Add logger counters * Save and show learned messages count (#383) * Add `no_stat` flag * Add `task:set_flag` and `task:get_flags` (#382) * Enable foreign keys in sqlite3 * Remove orphaned shingles from fuzzy storage * Optimize synchronization steps for fuzzy storage * Allow delayed conditions registration * Add lua API for conditions registering 1.0.2: * Fix critical bug in webui that prevents password from being sent * Rework webui view: - Switch to d3.js for graphs - Improve piechart look - Rework colors for piechart - Fix layout for symbols - Fix refresh button * Add descriptions for whitelist maps * Fix build on arm (#379) * Fix issue with the last element in the radix trie * Add more tests for radix trie algorithm * Allow to extract URLs from query strings of other URLs (#361) * Initialize rrd fields before writing to file * Fix double free if no password has been specified 1.0.1: * Add writing to rrd from the controller * Fixed lots of bugs in rrd code * Adopt new DNS API in hfilter plugin (by @AlexeySa) * Allow only one controller process to manage rrd file * Set event base for fuzzy calls * Improve fuzzy IO errors logging * Add rra extraction function to rrd library * Add graph handler to the controller * Cache correct passwords to avoid too high CPU usage when working with webui * Controller sockets are owned by router do not export them to task * Optimize logging by skipping hash table search if it's empty * Fix loading issue whith broken statfiles * Print assertions from glib to rspamd logger * Load legacy `lua/rspamd.local.lua` * Update webui with some fixes to learning and scanning 1.0.0: * Rework symbols processing: - Improve sorting logic for symbols - Organize processing into multiple stages - Added asynchronous watchers for symbols - Added ability to organize dependencies between symbols * Fixed URL redirector: - Use optimized POE loop - Organize dependencies - Fix startup * New sqlite3 backend: - Allow to have per-languages and per-user statistics - Allow sqlite3 to be used as statistics backend * Store tokenizer configuration within statfiles * Improve bayes statistics: - Use headers and images metainformation in bayes - Suggest using of pre-processed tokens for statistics - Fix tokens normalization for OSB algorithm * Rewrite url parsing: - Fix numerous issues with url extraction and normalization - Fix mailto urls * Fix settings plugin to allow custom actions scores * Improve rbl plugin * Allow capturing patterns in rspamd lua regexp library * Add GTUBE support * Fix spamc legacy support * Add DKIM support to RBL module * Fix issues with multiple DKIM signatures * Fix issue if rspamd cannot create statfiles (#331) * Rework parts and task structure: - Now text_parts, parts and received are arrays - Pre-allocate arrays with some reasonable defaults - Use arrays instead of lists in plugins and checks - Remove unused fields from task structure - Rework mime_foreach callback function - Remove deprecated scan_milliseconds field * Add ip_score plugin support (not enabled by default): - Can check for asn/country and network using DNS lookups - Can store and load reputation from redis server * Improve PARTS_DIFFER rule to count merely different words * New HTML parser: - Parses HTML parts using a set of state machines - Extracts useful data and exports it to lua functions: + Styles + Images + URLs + Colors + Structure elements - Added HTML rules for some checks * New version of LUA DNS API * Table versions of many functions in LUA API * Improve rspamc client: - Print execution time - Allow executing of external commands and passing output to them - Allow mime output mode when rspamc alters message according to rspamd checks and send it to an external command or stdout * Allow scanning of local files using HTTP requests * Rework configuration system: - Rules are now moved from the $CONFDIR to $RULESDIR to avoid ambiguity - All modules configurations are now split in $CONFDIR/modules.d/* to simplify upgrades - Move hfilter to plugins - Allow plugins and rules to define default scores to simplify metrics setup - Include overrides for all modules to honor local/automatic parameters - Tune scores for many modules * Rework and enable DMARC plugin * Add whitelist plugin for SPF/DKIM/DMARC based whitelisting * Add some common domains to whitelists shipped with rspamd * Rework logging: - Now each log entry supports module name and a `tag`. Tag is used to identify unique objects (such as tasks) when checking log files - It is possible to turn on debugging for the specific modules - Systemd logging is fixed * Improve spamassassin plugin. - Now headers are matched more like SA - Improve support of Message-ID - Add support of ToCc header type - Fix :addr and :name in headers regexps * Resurrect rrd support code * Save controller stats between restarts * Fixed tonns of bugs * Added tonns of minor improvements and features * Added more unit tests * Create functional tests framework * Added documentation for missing modules * Added rpm/deb repositories and scripts * Updated WebUI and libucl externals 0.9.10: * Do not dereference null pointer on learning. * Fix some extreme cases in BAYES. * Add a workaround to avoid bad HTML messages breaking. * Build with -O2 flags by default. * Add constraints to limit DNS requests count per task. * Add workaround for SURBL DNS flood. * Set error if rspamd cannot learn anything. 0.9.9: * Don't use RWL_SPAMHAUS_WL (unknown result) for whitelisting (by @fatalbanana) * Import updated public suffix list (by @fatalbanana) * Remove debug message * Fix settings (by @fatalbanana) * Remove duplicated symbol registration * Use WAL for fuzzy storage * RBL fixes (by @fatalbanana): - silence errors; - yield unknown results from RBLs; - fix scoring for DNSWL; - fix use of RBL name as symbol; - ignore RBL names that would not be yielded; * Support captures in regular expressions * Add captures support to lua_regexp * Support dist on FreeBSD and Darwin * Add RCVD_IN_DNSWL_NONE as whitelisting exclusion (by @fatalbanana) * Multiple fixes to URL detection: - support port definition; - fix query and path recognition; - fix parsing of multiple slashes in URL; - fix parsing query just after port; - fix path field in `url:to_table` method; - improve support of IP based URLs. * Set ignore_whitelists = true for RECEIVED_SPAMHAUS_XBL (by @fatalbanana) * Add GTUBE support * Ignore User header in SA mode 0.9.8: * Fix critical bug in bayes classifier (#305) * Fix critical bug in RBL module (by @fatalbanana) * Fix and rework settings plugin. * Fix get_all_opts for a case of non-iterable options. * Use tld for redirector's matching. 0.9.7: * Add whitelist_exception setting to RBL module (by @fatalbanana) * Don't use RWL_MAILSPIKE_POSSIBLE or DNSWL_BLOCKED for whitelisting (by @fatalbanana) * Fix extreme cases in bayes classifier. * Fix parsing of urls with '?' at the end of hostname. * Update interface. * Fix number of issues with webui interaction. * Fix saving maps. * Allow user@ and @domain matches in multimap. * Fix issues with bounces From processing. * Fix abs/fabs misuse. * Fix builds on suse and arch linux distributions. 0.9.6: * Fix memory leak if mime cannot be parsed. * Fix dkim cache expiration. * Fix issues with redirector HTTP response. * Fix abnormal connection closing with certains messages with a high score (issue #296) * Fix redirector installation. * Use specific POE loop for some systems. * Fix number of issues in URL redirector. * Fix selecting URLs for sending to redirector. 0.9.5: * Avoid double free when extending HTTP message. * Fix double free if multiple classifiers are defined. * Fix misprint in spamassassin plugin. * Fix cpuid invocation on i386. * Fix ownership issues for zero-copy decode. * Allow __len metamethod on rspamd{text}. * Add base64 decoding lua utility. * Fix build on FreeBSD * Skip spaces at the beginning of mime messages. * DBL_ABUSE_REDIR should not have significant weight. * Allow to split by lua_regexp rspamd{text} objects. * Allow to specify custom stop pattern for lua_tcp. 0.9.4: * Fix critical bugs in tokenization algorithm * Write unit tests for tokenization * Add documentation for lua_tcp * Switch off legacy tokenization by default. * Fix critical bugs in words normalization * Add lua bindings to tokenizer. * Implement storing of HTTP headers inside task * Add lua API to accerss HTTP headers data * Implemented base64 encoding suitable for MIME * Use caseless hash and equal functions for HTTP request headers. * Improve debian architectures support (by @dottedmag) 0.9.3: * Revert incorrect regexp change that broke the default rules * Fix lua_tcp module 0.9.2: * Fix error on spawning unique workers. * Add preliminary version of generic LUA TCP requests API. * Use lua 5.1 if luajit is not available (Arm64, PowerPC, s390x etc) * Fix fuzzy mime strings with only type. * Improve thunderbird sanity checks. * Fix critical bug on matching regular expressions. * Make hiredis optional dependency. * Fix multiple bugs in daemon reloading 0.9.1: * Restore utf8 validation for regular expressions to avoid crashes * Fix symbols displaying in the interface * Add symbol groups to the interface * Fix maps ID parsing in the controller * Add multimap and regexp modules documentation * Backport fixes from libucl * Fix debian package (by @dottedmag) * Rework XXH32 invocations 0.9.0: * Add support of the fast and secure protocol level encryption: - curve25519 is used for key exchange; - chacha20/poly1305 cryptobox construction for bulk encryption; - zero latency overhead; - encrypting and balancing HTTP proxy worker * Rework expressions and create new expressions library: - aggressive optimizations based on the abstract syntax tree; - abstract expressions support (regular expressions, functions, lua modules composites and so on) - New comparision and '+' operators support - New greedy algorithm to minimize execution time of expressions and all symbols - Dynamic expressions benchmark and reoptimizations * Many improvements to the LUA API: - reworked logger module allowing to do pretty print of the most of lua types (including tables and userdata classes) - reworked lua redis and lua HTTP to support more features - added opaque type for passing large text chunks without copying - new regexp module with many auxiliary functions (e.g. `re:split`) * LuaJIT is now the default requirement for rspamd allowing to speed up lua execution by a large margin (however, plain lua is still supported) * New plugins: - spamassassin rules plugin that allows to load and re-use the most of SA rules natively - DMARC plugin that evaluates SPF and DKIM policies to the domain policies - many old plugins has been reworked to implement new features and improve stability * New aho-corasic trie implementation from @mischasan that allows to load and use hundreds of thousands of patterns with no influence on load * Support of PCRE JIT and PCRE JIT fast path modes that significantly improves the performance of regular expressions if supported by PCRE * New URLs parser and extractor: - removed legacy code that was useless for url finding - reworked algorithms of URL parsing for more precise and accurate results - added top-level-domains tree from http://publicsuffix.org - improved emails parsing - removed many phishing false positives due to TLD tree check * New statistics infrastructure: - created a separate layer of statistic library - improved OSB-Bayes by re-weighting tokens according to the original academic paper and `crm114` implementation, which reduced false positives rate significantly - created learn cache to avoid double learning of statistics and providing an efficient way to re-learn class for a message - created abstract layers for different statistics backends - implemented new tokenization algorithms with fast or secure (siphash) hashes to generate statistics features * Reworked utf8 tokenization that previously corrupted all UTF8 words (minor incompatibility with old fuzzy hashes with utf-8 symbols) * SPF module has been completely rewritten to support complex cases of `include` and `redirect` within SPF records * DKIM module now supports multiple signatures * Controller passwords can now be stored encrypted by `PBKDF2-HMAC` in the configuration file * Many hand-written HTTP clients has been replaced with the common rspamd http module * New test framework: - import lua `telescope` test framework - add unit tests for many rspamd modules and routines - create a unit test for each possible bug found - use luajit ffi for testing C code - added preliminary support of functional testing by creating tasks from lua * Randomize hash seed to avoid certain hash tables vulnerabilities * Documentation improvements: - added documentation for the vast majority of rspamd modules - added documentation for rspamd protocol - added documentation for the most of rspamd LUA extensions * Fixed tonns of bugs and memory leaks * Added tonns of minor features 0.8.3: * Various critical fixes in distribution (by @dottedmag and @fatalbanana) * Fixed bugs in url detector to parse certain patterns * Add default host and helo for a client * Some sanity checks for tokenizer and classifier * Reiterate on systemd support * Fix missing symbol registration * Add support of spamc compatible output * Filter double-dots in rbl.lua validate_dns (by @fatalbanana) * Update ucl submodule due to critical bugfix 0.8.2: * Create fuzzy db if it does not exist * Fix: Centos init script: configtest() (by @AlexeySa) * Enable one_shot for RECEIVED_SPAMHAUS_XBL - Fixes #102 (by @fatalbanana) * Update Exim patch (by @fatalbanana) * Fix processing of unix sockets. * Allow applying settings to authenticated users (by @fatalbanana) * Make settings priorities work as documented (by @fatalbanana) * Fix race condition in symbols planner * Add DNSWL_BLOCKED symbol (by @fatalbanana) * Make Exim pass usernames to rspamd (by @fatalbanana) * Update RBL module (by @fatalbanana): - fix indentation; - collapse loops; - avoid calling for un-needed information; - allow disabling RBLs for authenticated users * once_received.lua: Fix indentation & add exclusion for authenticated users (by @fatalbanana) * hfilter.lua: Add exclusion for authenticated users (by @AlexeySa) * Updates to hfilter rules (by @AlexeySa) * Set empty <> user or addr for msgs without FROM (by @eneq123) * Fix: attempt to index field '?' (a nil value) (by @eneq123) * Fix: if not exist Date-header (by @AlexeySa) * Add task:get_content() method. * rbl.lua: Ignore private IP space (by @fatalbanana) * Allow to check radix maps from lua by rspamd{ip} * Make local exclusions configurable per-RBL (by @fatalbanana) * Add rspamd_config:radix_from_config() (by @fatalbanana) * Support emails dnsbl in rbl (by @fatalbanana) * Complete rework of url extraction logic * Allow customizations for unix sockets. (fixes #182) * Set lua path according to rspamd settings. * Import lua-functional for plugins stuff. * Completely rewrite multimap plugin in functional style. * Fix FORGED_MUA_THUNDERBIRD_MSGID (fixes #186) * Check IPv6 addresses at dnswl.org and Spamhaus whitelist (by @fatalbanana) * Add lowercase utility for utf8 strings. * Various fixes to build system * Updated debian configuration infrastructure (by @dottedmag) 0.8.1: * Add sqlite and perl as dependencies for RPM/Debian packages (by @fatalbanana) * Remove whitelist.lua from RPM file list (by @fatalbanana) * Make Exim pass hostnames to rspamd (by @fatalbanana) * Fix building on Fedora (by @fatalbanana) * Add toggle for disabling installation of systemd units on Linux (by @fatalbanana) * Fix double format rounding that caused output corruption (reported by @fatalbanana) * Revert broken change for destructors ordering that led to memory corruption * Do not reset symbols case of settings if parsed from lua (reported by @andrejzverev) * Fix build on SunOS (by @wiedi) * Fix multiple crashes on broken DKIM DNS records * Fix critical issue with composites weights removing * Fix memory corruption in composites processing code * Ignore non-SPF TXT records when parsing SPF includes 0.8.0: * New fuzzy check logic: - use shingles algorithm for fuzzy matching - use blake2 instead of md5 for larger output space - combine fuzzy and strict matching - allow to organize private storages by means of keys - preserve compatibility with previous versions * New fuzzy storage: - use sqlite instead of own memory based hash tables - rework commands interface - add conversion from the old format - add fuzzy match by shignles - support old rspamd versions * Add lemmatizing for words used in fuzzy hashes that allows to improve match quality by using of the first forms of all words * Rework language detection * Fix several critical bugs, memory leaks and deadlocks: - memory leak in HTML nodes parsing - deadlock in logger code - deadlock in signals processing - crashes in fuzzy_storage - crashes in tokenizers if the input was empty * Import new libucl with several bugfixes and improvements * Support listening on ipv6 addresses only * Fix macro expansion in SPF module * Several bugfixes in DKIM module * Add load headers support for mime parts to the lua API * Add documentation for: - workers in general - fuzzy_storage worker - fuzzy_check plugin - mimepart and textpart lua API modules 0.7.6: * Apply boundary fix for dkim simple canonization * Fix ping command * Return nil if header was not found in lua_task * Fix hang in upstreams revive logic * Decode entitles when normalizing HTML parts * Fix logic of finding URLs in HTML parts * Do not include \0 into length of text when performing conversion to utf8 * Fix raw vs parsed reperesentations Raw parts are now: - decoded b64/qp, but *NOT* converted to utf-8 Processed parts are now: - converted to UTF-8 - normalized if needed (e.g. HTML tags are stripped) * Rework DKIM canonization to line based * Fix fuzzy hashes addding * Use more specific hash function for fuzzy * Fix leaking of iconv descriptors * Fix PTR resolving in lua resolver * Rework spf module. - Copy data to memory pool as cached record might be destroyed causing freed memory being passed to the protocol output (use after free) - Allow SPF_NEUTRAL policy to be handled separately - Add R_SPF_NEUTRAL to the default config * Rework `register_symbols` function * Allow to disable components of hfilter 0.7.5: * Fix owner when creating folder /run/rspamd (by @sfirmery) * Fix IP validity checks * Decode URLs obtained from HTML tags * Fix crash with unweighted upstreams * Stop processing headers in parts * Set sockaddr.sa_family properly when connectig to upstreams * Fix reload issues in surbl and fuzzy_check (reported by @citrin) * Fix timeouts in redirector * Improve lua errors reporting * Fix lua closures processing in libucl * Rework calling of lua functions from regexp module * Choose raw regexp for raw headers * Rework conversion to utf since glib one is broken * Ignore SGML style tags in html * Fix old bug with non-capturing https urls * Fix memory corruption on fuzzy reload (reported by @citrin) * Fix percents display in rspamc * Fix buffer update for DKIM * Do not validate utf for raw headers 0.7.4: * Fix build under *BSD * Detect HAN unicode script * Implement language detection heuristic for text parts * Fix time output in history * Improve piechart coloring * Fix \r\n conversion in DKIM module (reported by @citrin) * Try to detect systems with no IPv6 support * Fix multiple/single values in use settings (reported by @citrin) * Rework IP addresses in upstreams: - Select ipv4/unix addresses if they exist and use ipv6 for ipv6 only upstreams (since the support of ipv6 is poor in many OSes and environments) - Free IP list on upstream destruction - Add test cases for addresses selection - Allow adding of free form IP addresses to upstreams * Fix endiannes in lua_radix search (reported by @citrin) * Soft shutdown should also set wanna_die flag (reported by @citrin) * Stop use-after-free in event loop termination * Fix processing of very short messages in DKIM (reported by @citrin) * Detect systems without shared mutexes * Fix issues with PTR and MX elements in SPF parser (reported by @citrin) 0.7.3: * New upstreams code: - simplify upstreams API; - unify strings parsing in upstreams definition; - add configuration options for the upstreams; - for failed upstreams re-resolve their addresses; - use all resolved addresses for an upstream (round-robin); - implement stable hashing and use it by default for upstreams; - add unit test for upstreams module. * Rework signals processing in all rspamd workers: - signals are now processed in the event loop; - implement the most common signal handlers for all workers; - add callbacks for workers specific signal handlers * Fix critical issue with fuzzy storage: Fuzzy stroage could not save any hashes on termination due to bugged signals handling * Fix roll history IP storage * Rework ipv4/ipv6 handling in parsing addresses: - turn off support of IPV6_V6ONLY socket option; - create ipv6 socket prior to ipv4 one to handle systems with v6/v4 sockets enabled (Linux) * Remove CBL as it's wholly included in Spamhaus XBL (by @fatalbanana) * Remove nszones.com fake RBL (by @citrin) * Fix upstreams interaction for fuzzy_check * Verify spf PTR records (reported by @citrin) * Fix spf MX records parsing * Add compatibility for old libevent (by @yellowman) * Sync bugfixes from libucl 0.7.2: * Convert all maps to the compressed radix trie * Allow IPv6 addresses in IP maps * Remove dynamic items support from symbols cache * Allow hex encoded output of strings * Fix bug with control connections count * Process fuzzy weight correctly (reported by @fatalbanana) * Remove extra reference retain of http connection on error * Remove deprecated options from the default config * Add `one_shot` attr to metric's symbols * Doc: add documentation for metrics * Add Upstart job to debian packaging (by @CameronNemo) * Config: improve SURBL symbols descriptions (by @citrin) * Config: reflect SURBL changes (by @citrin): - Outblaze removed, malware moved to separate list: http://www.surbl.org/news/internal/MW-malware-sublist-added-to-multi * Fix C modules initialization on restart * Treat single IP as a single IP in radix lists (reported by @citrin) * Do not touch file and core limits if not asked explicitly (reported by @citrin) * Improve logging for fuzzy errors * Block SIGPIPE for HTTP writing * Doc: update manual pages * Fix HTTP connection termination * Reduce default number of parallel requests to 8 * Sync with libucl include features 0.7.1: * Fix typo in stat output. * Fix issues with includes crossing with the system includes * Restore testing framework * Add radix trie test suite * Implement new path-compressed radix trie. - The performance benefit over the old algorithm is about 1.5 times. - Memory usage is significantly lower as well. - Now radix trie can accept any IPv4/IPv6 values * Various improvements to the memory pools code * Fix writing reply to a client when no filters are defined * Write base32 encoded fuzzy * Fix 'soft reject' action * Fix rspamd reload and modules reconfiguration * Fix subject rewriting for the default subject * Fix states for processing task and pre-filters * Fix issues with connection closing * Fix crashes in rdns * Fix ratelimit pre-filter * Update exim patch. - Update to the recent exim version - Strip extra leading src/ from the patch - Remove sendfile since it was broken - Fix rspamd spam report for exim * Improve documentation 0.7.0: * Use HTTP protocol for all operatiosn * Webui worker is now removed and controller works as webui * Allow to serve static files via controller by option `static_dir` * Rspamd interface is now a part of rspamd * Rspamc client has been rewritten to use HTTP and non-blocking mode allowing to start multiple operations simultaneously (see `-n` option) * Lua API was completely reworked to satisfy modern standards of LUA: * Module `lua-message` was removed * Reduced number of superglobals registered by rspamd * Many functions has been redesigned * Symbols registration is now more convenient * Users settings has been rewritten as lua plugin * Reworked headers system as gmime's based one misses many headers and is very slow to get headers values * Reorganized code and removed embedded jannsson by using UCL for all json parsing * Migrated to `librdns` for DNS resolving that improves concurrency for DNS requests significantly * Fixed tonns of bugs in MIME processing * Improved metrcis and default symbol's weights * Added new RBL's * Fixed a number of issues in the modules * Removed several memory leaks found * Fix unicode processing * Fix fuzzy checking for unicode parts * Significantly improve documentation and especially LUA API docs * See migration notes at https://rspamd.com/doc/migration.html 0.6.8: * Controller now listen for localhost and not for 127.0.0.1 by default * Allow FCrDNS-style RBL lookups (by @fatalbanana) * Reduce threshold for parts_differ function. * Fix hostname lookup for rdns rbl (by @AlexeySa) * Fix HFILTER_URL_ONELINE to reduce false positive rate. * Fix whitelist module. * Allow override system predefined settings without touching system ones by .try_include macro (by @andrejzverev) * Check for [ip.address]-style HELO and suppress lookups. (by @fatalbanana) * Optimize hfilter (by @AlexeySa) * Fix issue with random numbers generator in dns. * Use more clever time values to setup entropy. * Synced with the recent libucl. 0.6.7: * Use ChaCha20 for DNS generator (more secure DNS id) * Unknown symbols now has zero weight and not 1.0 * Fix fuzzy hashes expire time * Fix critical issue in statfiles on FreeBSD 9 (and some other platforms) * Add .include_map macro to ucl parser * Update libucl * Fix headers end detection for DKIM module * Fix a bug in received headers parser * Validate IP addresses before pushing them to lua * Start new documentation project * Fixed tonns of other minor bugs * Start to prepare for 0.7 with HTTP protocol and new settings 0.6.6: * Removed issue with BUFSIZ limitation in the controller output * Simplify logging symbols escaping * Adjusted weights for several rules * Improve spamhaus rbl support * Removed PBL for received headers checks * Added hfilter module that performs various HELO and IP checks. * Rspamd can now be reloaded by HUP signal * Fuzzy storage should expire hashes properly * Build system has been reworked for better supportof pkg-config * Various minor bugfixes 0.6.5: * Fixed critical bug in DNS resolver, introduced in 0.6.4 * Improved multimap and rbl plugins to skip * Add dns_sockets option for tuning sockets per server in DNS resolver * Improved packages for rspamd 0.6.4: * Added io channels for DNS request to balance load and reduce id collisions chance * Fixed a bug in SPF filter that may cause core dump in specific circumstances * FIxed default config for rbl module * It is possible to get a list of rspamc commands with their descriptions * Added SORBS bl to the default config * 2tld file for surbl module has been significantly extended * Perl modules has been removed from the code. * Fixed an issue in libucl when parsing macros 0.6.3: * Fixed issues with DNS: - labels decompression algorithm was fixed; - added resolve_mx to lua API; - fixed modules that use DNS. * Lua modules once_received and emails reworked for new resolver API and UCL. * Debian package was polished. * Fixed a bug in fuzzy_check module that prevents correct processing messages without valid parts. 0.6.2: * Fuzzy check module has been reworked: - now fuzzy_check operates with a group of rules, that define which servers sre to be checked; - it is possible to specify read_only groups to avoid learning errors; - turn fuzzy_check to one_shot mode permanently; - fuzzy_check module configuration is now incompatible with the previous versions. * Imported bugfixes from libucl. * Fixed whitelist plugin. * Fixed statfiles resizing. * Improved logging initialization order. * Fixed race condition in the controller worker. 0.6.1: * Critical bugfixes: - fixed build system; - fixed in_class setting in bayes learning; 0.6.0: * Use UCL instead xml for configuration (https://github.com/vstakhov/libucl) * Fix statistics module normalization * Rework the many modules for the new configuration: - surbl has incompatible configuration; - fuzzy_check has incompatible configuration; - multimap has now new configuration; - received_rbl is removed and replaced by rbl module. * Removed deprecated options: - statfile_pool_size; - action and required/reject score for a metric. * Simplify build system and unify configuration for all platforms. * Improved URL detector (reduced false positive rate). * Lua 5.2 is now the default and fully supported version. * Tons of bugfixes and minor improvements. 0.5.6: * Fix bug with counters incrementing in rolling history * Detect expl and exp2l as some systems do not have it * Support input streams without Content-Length * Implement counters output via rspamc and controller interface * Fix bug with udp sockets in fuzzy storage 0.5.5: * New bayes normalizator based on inverse chi-square function * Various fixes to fuzzy storage * Allow update fuzzy storage only from specific IP addresses * Better support of IPv6 and address selection algorithms * Add CentOS spec file 0.5.4: * Fixed issues with diff algorithm * Added support of RRD statistics * Add webui worker for interface interaction * Fix a lot of issues with dynamic conf * Fix critical memory leak in settings code * Improve stability of the system 0.5.3: * Added dynamic options * Added advanced metaclassfication * Added RESTfull API for controller * Improved hashing algorithms * Various fixes for rspamc client: - allow interacting with unix sockets - librspamdclient major cleanup - bayes is now default classifier 0.5.2: * Added lua bindings for: - basic mime parts, that allows checkign attachements for example; - DNS resolver; * Existing lua bindings now works without task object allowing to use them in custom code. * Threads system was reworked to avoid global lua interpreter lock. * DKIM module now converts all line endings to CRLF how opendkim does. * URL detector is now more accurate for text parts. * Several critical bugs and memory leaks were fixed. 0.5.1: * Added lua worker type to handle network connections in lua. * Added lua bindings for async_session, IO dispatcher, memory_pool, and worker. * Composites can now uses other composites in expressions. * Fixes for debian package and for FreeBSD >= 9.1 * Add support of gmime-2.6 if gmime-2.4 is not found. * Improve url detection and phishing detection. * Add lua mime_part library to get an access to all message part attributes (like filename, length, type). 0.5.0: * Added SMTP lightweight balancing proxy with XCLIENT support. * Added lua bindings for upstreams objects and API. * New pre-filters are implemented to support initial checking for messages. * Added ratelimit plugin that uses redis protocol to store data. * Added ipv6 support to spf and some other modules. * Unbreak spf plugin. * Allow options with the same name be threated as list. * DKIM plugin an parsing code was added. * Separate build system to put logic in several shared libraries. * Many bugfixes. 0.3.2: * Add error handling for regexps * Fix quit command in controller interface * Write symbols weights to rspamc output * Improve logic of selecting rspamc version * Do not try to parse broken DNS replies * Add 'raw' flag to FROM_EXCESS_BASE64 rule (requested by citrin) * Output message id in rspamc reply * Fix inserting composite symbol * Fix output of log line * Document composites * Add logging for fuzzy checks * Add logging for learning * Improve logic of learning messages - do not learn more than specific threshold * Fix inserting results for symbols that were incorrectly (for example more than 1 time) defined in config file * Do not output control characters if output is not terminal * Fix some logic errors in learning * Consider lua plugins errors as fatal configuration errors * Fix wirting message id during fuzzy_add command * Display weight of symbols correctly * Fixes to winnow learning * One more try to improve accuracy of winnow algorithm * Add bayesian classifier (initial version) * Remove normalizer as it is winnow specific thing, so all statistic algorithms now returns value from 0 to 1 * Some fixes to fuzzy hashes expiration: * Fix assertion while look up value in NULL hash (found by cirtin) * Fix normalization for systems that have not tanhl function * Ignore rfc822 group addresses * Move images library to core rspamd * Add lua api to access images properties * Add post filters to lua API - filters that would be called after all message's processing * Add ability to check for specified symbol in task results from lua * Add ability to check for metric's results from lua * Add ability to learn specified statfile form lua * Add ability to extract filename and size of images from lua * Fix assertion while extracting internet address * Fix races in fuzzy storage * Make spf parser case insensitive * Add ability to check hashes of selected mime types * Add ability to set minimum size in bytes for mime types * Add ability to set minimum dimensions for images * Assume all text/* content types as text/plain * Fix getting data wrapper for gmime24 * Many fixes to statfile syncronization system * Fixed statfile pool initialization and synchronization with disk * Prepare 0.3.2 * Fix Mail::Rspamd::Config for new rspamd features * Use Mail::Rspamd::Config in rspamc client * Write user's name to rspamd log * Prepare rspamd build infrastructure for creating rpm and deb packages * Fix depends * Add start script for linux systems * Fix shared usage of statfiles * Add invalidation of statfiles in case of learning, so now statfiles * This should fix shared usage of statfile pool by several processes * Fix misprint (reported by az) * Fix stupid error when all checks can be done in a single pass * New trie based url scanner (based on libcamel) * Small fixes to rspamd perl client * Write fuzzy hashes info to log * Add trie interface to lua api * Explain sample config and cleanup it 0.3.1: * Add modules documentation * Continue implementing smtp proxy * Implement new learning system, now rspamd should be much more intelligent while learning messages * Convert statistic sums to use long double for counters * Use hyperbolic tangent for internal normalizer * In classify normalize result after comparing, not before * New symbols sorter * Fix strict aliasing while compiling with optimization * Fix tanhl detection for platforms that have not implementation of it * Remove several compile warnings * Add experimental support of dynamic rules to regexp module * Document views configuration * Several fixes to documentation * Add more logic for dynamic rules * Add documentation for dynamic rules * Add ability to make negations in networks in dynamic rules * Clean up cache items correctly * Implement basic SMTP dialog: * Implement interaction with smtp upstream (with support of XCLIENT) * Check messages received via smtp proxy * Add support for sendfile in io dispatcher * Fix issues with compatibility of worker_task and smtp proxy * Proxy DATA command * Fix SMTP * Change metric logic * Completely remove lex/yacc readers for config * Make common sense of metric/action and symbols * Sync changes with all plugins * Incorrectly removed in previous commit * Fix misprint (by Andrej Zverev) * announce the "password" keyword in usage list * Implement initial version of greylisting triplets storage * Fix issues with smtp worker * Fix QUIT command in SMTP worker * Some fixes about new metrics system (may be incomplete) * Get weights of symbol from default metric for symbols cache * Fix setting task->from/task->rctp in smtp client * Copy from and rcpt correctly * Some performance improvements to IO dispatcher (do not drain the whole buffer after a single line readed) * Fix smtp data input * Fix misprint * Add limit of maximum allowed smtp session errors * New logic of SURBL module: * Use system mkstemp(3) on systems where it is available as glib implementation * Try to fix memmove issues in io dispatcher * Remove debug from SURBL module * Rewrite buffered input for line policy (again) * Fix issue with links that are ip addresses in numeric form in surbl * On Darwin use BSD style sendfile definition * Reorganize platform specific knobs in CMakeLists * Use gettimeofday on systems that have not clock_getres * Use ftime for dns trans id generation on systems without clock_getres * Darwin sendfile(2) support * TIMEDB->TIMEB * More to previous commit * Pass env from main() arguments instead of platform specific global environ * Fix compatibility issues * Fix -lintl detection * Init some variables to avoid problems * Remove garbadge (gnome terminal sucks) * Add more information about why we drop smtp connection * Fix mkstemp call * Send to upstream QUIT command at the end of session * Check return value of each rspamd_dispatcher_write as in case of write errors sessions can be destroyed early * Fix states in smtp dialog * Use rspamd_snprintf instead of libc one * Fix URLS command * Fix reconfigure process of surbl module * Fix destroying smtp session (unmap memory and do not delete pool early) * Delete pool after using its variables * Delay timer must be registered in async session to correctly handle connection termination * Register dns requests in session too * Make session before registering events * Remove events in handlers * Add ability to set filters for smtp worker for each smtp stage * Add very initial version of DNS resolver (many things to be done) * Announce weights and sync * Fix few typo * Understand short names of facility in logging config * Add ability to make whitelist for spf checks * Misprint != -> == * Handle lua tag in way that it is not required to write additional text: * Strip all starting whitespace symbols from xml texts * Fix stupid bug in calculating buffer length while reading file maps * Add resolv.conf parsing into dns.c * Fix microseconds<->milliseconds conversions * Take callback argument in Mail::Rspamd::Client for processing files and directories * Print results if rspamc is called for a directory * Fix stupid error with surbl module reconfig (another one, blame me) * Do not show duplicate urls in url header * Fix detection of numeric urls (reported by citrin) * Write real time of message's scan to log (not only virtual) * Fix chartable module in utf mode * Fix parsing of some broken urls * Add ability to test regexp with 'T' flag * Write more code for DNS resolver: * Make DNS resolver working * Many improvements to rspamd test suite: now it CAN be used for testing rspamd functionality * Write DNS resolver tests * Fix issues with memory_pool mutexes and with creating of statfiles * Forgotten in previous commit * Add support for parsing SPF and SRV records * Fix PTR parsing * Add tests * Make SURBL module to use rspamd dns module * Several fixes to DNS logic * Remove evdns and use only rspamd resolver * Very hard to detect problem with race among error in socket and destroying task while we are writing to socket and go through a hash table * Fix resolving in smtp module * Init events before configuring resolver in smtp worker * Set resolver inside task * Fix reload signal (reported by citrin) * Some improvements to redirector * Call has_forked method to inform POE about fork * Fix lua DNS code * Decompress labels in DNS packets more strictly * Fix some problems with TXT records * Try to fix removing of DNS events * Do not insert unparsed RR's into reply * Calling callbacks may cause destroying session from which we are calling callback so we MUST call callback as the latest action * Fix check_smtp_data function * Add ability to make views by recipient * Add ability to set metric's action from config file * Fix bug with writing garbadge if message has no urls or no messages * Fix bug with incorrect behaviour of compare_parts_distance function * Add ability to assign several actions to one metric * Report action in rspamc protocol * Mail::Rspamd::Client and rspamc can now understand Action header too * Write action to log as well * Make valgrind happy about comparing symbols * Add more debug to comparing parts distance function * Write action even if message has no symbols * Make improvements to HTML entites decoder: now it replaces entities with common characters and * Add -d option to force debug output * Assume 7bit as default transfer encoding * Do not overwrite lua plugins and configs if they already exists in target directory * Improve logging * Write queue id to log * Remove test messages from cmake * Reopen log file by USR1 signal * Add reopenlog method to FreeBSD rc script * Adopt foreach for cmake 2.6 * Fix to rc script * Do not try to resolve names with several dots in a row * Fix surbl request formatting for ip addresses * Handle cases of broken requests * Fix problems with parsing compressed names * Fix TXT records parsing * Fix expanding spf macros that may fail in rare cases * Fix another error with early task destroying * Handle empty from header * Improve reopenlog command in rc script * Strip trailing whitespace characters in Mail::Rspamd::Client * Use ungreedy match to strip trailing whitespaces * Stupid error in calculation compressed label length * Some optimizations to client library * Do not compare empty parts * Empty and non-empty parts are allways different * Save in regexp cache the whole regexp with header name (if exists) and with flags * Add rspamd_log variable to lua plugins to access logging functions * Each part in rspamd task now can have parent part * Check for parts distance only for multipart/alternative subparts * Do not check attachements even if they are text (but attached as file) * Do not die if write (2) returned ENOSPACE while doing logging, turn on throttling mode instead (1 write try in a second) * Add ability to turn on debug for specific symbols * Add ability to configure dns timeouts and dns retransmits in config file * More debug * Fix extracting arguments in lua logger interface * Turn off debug_ip during reload if it was disabled by new config * Improve lua logging * Pre-init symbols cache when rereading config * Fix lua representing of invalid ip (nil, not 255.255.255.255) * Fix R_TO_SEEMS_AUTO rule (by citrin) * Add multimap lua plugin * Fix some multimap issues * Try to save images hashes to fuzzy storage to stop some annoying spammers * Allocate some more bytes for read buffer to avoid incorrect behavoiur * Add ability to check dns black lists by multimap module * Add multimap documentation * Fix labels parsing * Another try to save regexps in cache correctly * Improve test logs for regexps * Fix parsing txt records to avoid reading of uninitialized data * Fix error with writing symbols cache file * Fix error while working in utf mode when raw regexps was not created properly * Do not add extra byte while converting text to utf * Add error handling for regexps