aboutsummaryrefslogtreecommitdiffstats
path: root/ChangeLog
Commit message (Collapse)AuthorAgeFilesLines
* Release 1.8.11.8.1Vsevolod Stakhov2018-10-161-0/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix options insertion * [CritFix] Fix words decay one more time (affects long messages) * [CritFix] Increase default words_decay * [CritFix] Plug memory leak in redis pool * [Feature] Add `check_violation` feature to DKIM/ARC signing * [Feature] Add only unique elements to Clickhouse url arrays * [Feature] Allow `g+:` and `g-:` composite atoms * [Feature] Allow dkim domains check in surbl * [Feature] Allow maps with HTTP auth * [Feature] Allow to disable actions by users settings * [Feature] Extend whitelisting options * [Feature] Store url object in images * [Feature] Use verdict instead of the plain action in plugins * [Fix] Allow to call fstring append with NULL string * [Fix] DCC - luacheck * [Fix] Do not load torch on each rspamadm invocation * [Fix] Fix boundaries detection and rework stop words algorithm * [Fix] Fix dependencies for DNS_SIGNED symbol * [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit * [Fix] Fix groups mess * [Fix] Fix groups mess * [Fix] Fix parsing address with comments * [Fix] Fix resolving in DMARC reports * [Fix] Fix various issues with parsing of the received headers * [Fix] Fix watchers issue in lua_tcp when doing no resolving * [Fix] Plug memory leak in language detector (affects reloads) * [Fix] Remove one letter stop words * [Fix] Slashing: backport chunk logic from libucl * [Fix] Stop libevent from using cached time in rspamadm * [Fix] Try to fix watchers chaining * [Fix] Various fixes in redis sync interface * [Fix] ip_score - respect check_authed and check_local settings from config * [Project] Rework passthrough actions * [Project] Clustering module * [Rework] Always create result for a task * [Rework] Completely rewrite DMARC checks logic * [Rework] Rework and fix whitelist plugin * [WebUI] Add symbols sorting buttons * [WebUI] Change symbols order without updating history * [WebUI] Colorize symbols * [WebUI] Do not display password form when secure_ip is set * [WebUI] Fix symbol description tooltips display * [WebUI] History: add sorting by symbol score value Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.8.01.8.0Vsevolod Stakhov2018-09-241-0/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Feature] Add arguments schemas to processors and extractors * [Feature] Add functional selectors library * [Feature] Add generic selector to reputation module * [Feature] Add more ratelimits: by digest, by attachments data, by filenames * [Feature] Add preliminary stop words detection support * [Feature] Add pure Lua debugm function * [Feature] Add schema validation for Redis settings * [Feature] Add selectors combine function * [Feature] Add some recursion protection to lua logger * [Feature] Add support for Lua API tracing * [Feature] Allow to apply schema to arguments * [Feature] Allow to get dkim signing data directly from HTTP headers * [Feature] Allow to reuse existing authentication results * [Feature] Cache selectors results in re runtime * [Feature] Implement new text tokenizer based on libicu * [Feature] Integrate selectors framework to multimap * [Feature] Relax FORGED_RECIPIENTS * [Feature] Support (almost) all html entities * [Feature] Support adding and deletion of recipients in the milter block * [Feature] Support gathering HTTP body from fragments in lua_http * [Feature] Support multi flag in regexp and glob maps * [Feature] Support selectors in ratelimit module * [Feature] Support selectors in settings * [Feature] Use khash in HTML parser * [Feature] Use pure Lua debugm function * [Fix] Add fail-safety for destroying sessions * [Fix] Allow to add result-less fake DNS records * [Fix] Another try to fix race conditions on config unload * [Fix] Call Lua callback on DNS timeouts * [Fix] Deprecate task:inc_dns_req as it is redundant * [Fix] Do not allow events deletions on cleanup * [Fix] Do not try to process skipped messages * [Fix] Fix HTTP requests with no body * [Fix] Fix another cleanup race condition * [Fix] Fix bug in processing of pcre regexps * [Fix] Fix byte array allocation in the pool * [Fix] Fix crashes on task cleanup * [Fix] Fix dynamic buckets in ratelimits * [Fix] Fix endless loop when waiting for Rspamd to stop * [Fix] Fix lua_util.str_split in case of delimiters set * [Fix] Fix more issues with watching of async events * [Fix] Fix stop words detection and loading logic * [Fix] Fix various corner cases for language detection * [Fix] Fix watchers in lua_tcp * [Fix] Fix words decay algorithm * [Fix] Implement watchers replacement to handle nested calls * [Fix] Save faked code into fake dns record * [Fix] Show the proper frame when using lua_util.debugm * [Fix] Use fake dns records in tests * [Fix] Use unicode replacements for HTML entities * [Fix] fixed "cannot find dependency on symbol 1" issue when using replaced symbols in spamassassin rules * [Fix] partition_id is not available in old versions of CH * [Project] Add implicit conversion logic to selectors * [Project] Add initial support for selectors in regexps * [Project] Add method concept * [Project] Further changes in unicode operations * [Project] Implement Clickhouse migrations * [Project] Implement implicit conversions to userdata * [Project] Implement insert method * [Project] Implement selectors registration for regular expressions * [Project] Implement selectors support in re_cache * [Project] Improve language detector: cleanup unused files, categorize * [Project] Migrate CH data to a fat table * [Project] Rework selectors logic * [Project] Start Clickhouse utilities library * [Project] Start unicode rework * [Project] coroutine threaded model for API calls: thread pool * [Rework] Move phishtank to a DNS based service * [Rework] Rework Clickhouse plugin to use the new API * [Rework] Rework language detector * [Rework] Rework utf content processing in text parts * [WebUI] Add progress bar for AJAX requests * [WebUI] Avoid errors table reinitialization * [WebUI] Avoid history table reinitialization * [WebUI] Avoid throughput summary table reinitialization * [WebUI] Destroy summary table on disconnect * [WebUI] Fix "auth" request URL * [WebUI] Fix disabling and hiding controls on page reload * [WebUI] Fix maps loading from neighbours * [WebUI] Fix symbols sorting by score * [WebUI] Fix tables destroying * [WebUI] Fix throughput data consolidation * [WebUI] Fix upload buttons disabling * [WebUI] Notify user on module loading failure * [WebUI] Update FooTable 3.1.4 -> 3.1.6 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.91.7.9Vsevolod Stakhov2018-08-011-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix caseless comparison of equal length strings * [Feature] Add HTTP basic auth support to elastic and clickhouse plugins * [Feature] Add SPF selector to reputation * [Feature] Add support of the fallback backends for HTTP maps * [Feature] Allow to print full mime structure when extracting mime data * [Feature] Allow to split symbols in reputation plugin * [Feature] Check attachments only on AV scanners in attachments_only mode * [Feature] Disable all SSL checks if ssl_no_verify flag is set * [Feature] Implement parsing of scoped IPv6 addresses * [Feature] Improve `rspamc counters` output * [Fix] Add sanity checks when expanding SPF macros * [Fix] Allow to parse SA rules with no spaces around =~ (dirty hack) * [Fix] Avoid one extra byte writing * [Fix] Deal with direct hash table * [Fix] Detect empty text part as text, not HTML * [Fix] Do not reduce map watch timeout for mixed http/file maps * [Fix] Fix HTML part detection heuristic * [Fix] Fix double free in redirectors cleanup * [Fix] Fix legacy history handling in the controller * [Fix] Fix messages insertion * [Fix] Fix sending string method * [Fix] Fix statconver command line arguments * [Fix] Fixed argument checking for being null * [Fix] Fixed issues reported by luacheck * [Fix] Freeze updates queue when do actual storage update * [Fix] HTTP map hash is per-backend and not per-map * [Fix] Plug memory leak in fuzzy updates * [Fix] Prefer 'MTA-Name' when producing authentication results * [Fix] Replace bad unicode sequences instead of stopping on them * [Fix] Set classifier version on learning * [Project] Reworked ratelimits * [Project] Apply topological sorting for symbols in Rspamd * [Project] Remove global contexts from C modules * [Project] Move performance critical hash tables to khash * [WebUI] Avoid unused indexes * [WebUI] Do not execute `on_success` callback * [WebUI] Fix history reset for "All SERVERS" (#2346) * [WebUI] Fix query URL for selected server * [WebUI] Fix symbols display in legacy history, * [WebUI] Hide symbols order selector for legacy history * [WebUI] Refactor query functions into one * [WebUI] Remove previously-attached event handlers * [WebUI] Save symbols to the selected server * [WebUI] Unify arguments of query functions * [WebUI] Use common query functions to get graph data * [WebUI] Use common query functions to save symbols Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.81.7.8Vsevolod Stakhov2018-07-121-0/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Feature] Add more extended statistics about fuzzy updates * [Feature] Add more non-conformant Received headers support * [Feature] Add preliminary function to get fuzzy hashes from text in Lua * [Feature] Allow to configure AV module rejection message * [Feature] Implement fuzzy hashes extraction in mime tool * [Feature] Improve WHITE_ON_WHITE rule * [Feature] Improve integer -> string conversion * [Feature] Reuse maps in multimap module more aggressively * [Fix] Avoid race condition in skip map as pool lifetime is not enough * [Fix] Eliminate all specific C plugins pools * [Fix] Fix DKIM check rule if DNS is unavailable * [Fix] Fix build where ucontext is defined in ucontext.h * [Fix] Fix crash in base url handling * [Fix] Fix descriptors leak in sqlite3 locking code * [Fix] Fix messages quarantine * [Fix] Fix padded numbers printing * [Fix] Fix race condition on maps reinit * [Fix] Fix regexp functions when no data is passed * [Fix] Fix specific urls extraction * [Fix] Fix styles propagation * [Fix] Improve resetting of the limit buckets * [Fix] Initialize sqlite3 properly * [Fix] Work with broken resolvers in resolv.conf * [Project] Implement HTTP maps caching * [Project] Refresh fuzzy hashes when matched * [Project] Add logic to deduplicate fuzzy updates queue * [WebUI] Add missed declarations * [WebUI] Avoid using "undefined" property * [WebUI] Do not accept passwords containing control characters * [WebUI] Do not redeclare variables * [WebUI] Enable strict mode, * [WebUI] Fix variable assignment * [WebUI] Initialize variables at declaration * [WebUI] Remove duplicated path from RequireJS config * [WebUI] Remove unused block * [WebUI] Remove unused variable * [WebUI] Remove unused variables * [WebUI] Use self-explanatory notation * [WebUI] Use type-safe equality operators Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.71.7.7Vsevolod Stakhov2018-07-021-4151/+4194
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Check NM part of pubkey to match it with rotating keypairs * [CritFix] Do not overwrite PID of the main process * [CritFix] Fix maps after reload * [CritFix] Fix maps race conditions on reload * [CritFix] Fix shmem leak in encrypting proxy mode * [Feature] Add a concept of ignored symbols to avoid race conditions * [Feature] Add ability to print bayes tokens in rspamadm mime * [Feature] Add method to get statistical tokens in Lua API * [Feature] Add preliminary mime stat command * [Feature] Add rspamadm mime tool * [Feature] Add urls extraction tool * [Feature] Address ZeroFont exploit * [Feature] Allow rspamadm mime to process multiple files * [Feature] Allow to extract words in `rspamadm mime` * [Feature] Allow to print mime part data * [Feature] Allow to show HTML structure on extraction * [Feature] Distinguish IP failures from connection failures * [Feature] Improve output for mime command * [Feature] Improve styles propagation * [Feature] Main process crash will now cleanup all children * [Feature] Preload file and static maps in main process * [Feature] Print stack trace on crash * [Feature] Process font size in HTML parser * [Feature] Propagate content length of invisible tags * [Feature] Read ordinary file maps in chunks to be more safe on rewrites * [Feature] Support base tag in HTML * [Feature] Support more size suffixes when parsing HTML styles * [Feature] Support opacity style * [Fix] Another fix for nested composites * [Fix] Fill nm id in keypairs cache code * [Fix] Fix colors alpha channel handling * [Fix] Fix destruction logic * [Fix] Fix double free * [Fix] Fix maps preload logic * [Fix] Fix nested composites process * [Fix] Fix proxying of Exim connections * [Fix] Fix reload crash * [Fix] Fix rspamadm -l command * [Fix] Update ed25519 signing schema * [WebUI] Stop using "const" declaration * [WebUI] Update RequireJS to 2.3.5 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.61.7.6Vsevolod Stakhov2018-06-151-0/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix multiple neural networks support * [Feature] Add decryption function to keypair command * [Feature] Add gzip compression for HTTP requests in elastic module * [Feature] Add gzip methods to lua util * [Feature] Add maps based on Top Level Domains * [Feature] Add pubkey checks for dkim_signing * [Feature] Add support of fake DNS records * [Feature] Add tool to encrypt files * [Feature] Allow to add symbols using settings directly * [Feature] Allow to match private and public keys for DKIM signatures * [Feature] Allow to set task flags via settings * [Feature] Allow to specify fake DNS address from the config * [Feature] Implement signatures verification using rspamadm keypair * [Feature] Implement signing using `rspamadm keypair` * [Feature] Improve error reporting for DKIM key access issues * [Feature] Provide $HOSTNAME variable in UCL * [Feature] Rework levenshtein distance computation * [Feature] Split message parsing and processing * [Feature] Support ED25519 DKIM signatures * [Feature] Support encrypted configs in UCL * [Feature] Suppress duplicate warning on very large radix tries * [Feature] Use OSB to combine header names * [Fix] Cleanup maps data on shutdown * [Fix] Fix '~' behaviour in composites * [Fix] Fix HTTP maps updates * [Fix] Fix NIST signatures * [Fix] Fix RFC822 comments when processing a mime address * [Fix] Fix double free * [Fix] Fix dynamic settings application * [Fix] Fix for CommuniGate Pro maillist * [Fix] Fix keypair creation method to actually create keypair... * [Fix] Fix matching patterns with no paths * [Fix] Fix memory leak in parsing comments * [Fix] Fix parsing of urls with numeric password * [Fix] Fix plugins intialisation in configwizard * [Fix] Fix potential crash on reload * [Fix] Fix potential race condition for a finished HTTP connections * [Fix] Fix race-condition leak on processes reload * [Fix] Fix signing in openssl mode * [Fix] Free language detector structures * [Fix] Relax alignment requirements * [Fix] Send DMARC reports compressed * [Fix] Try to fix leak in dmarc module * [Fix] Try to plug memory leak in metric exporter * [Project] Convert rspamadm subcommands to Lua * [WebUI] Display smtp sender/recipient in history * [WebUI] Fix elements disabling in "Symbols" tab * [WebUI] Limit recipients list in history column to 3 * [WebUI] Match envelope and mime addresses following in arbitrary order * [WebUI] Update column header * [WebUI] Wrap addresses in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.51.7.5Vsevolod Stakhov2018-05-181-0/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add MSBL proposed return codes * [Conf] Add additional groups for policies * [CritFix] Do not use volatile Lua strings as UCL keys * [Feature] Add ability to add fuzzy hashes to headers * [Feature] Add function to extract most meaningful urls * [Feature] Add rule to block mixed text and encrypted parts * [Feature] Allow multiple groups for symbols * [Feature] Allow to disable lua squeezing logic * [Feature] Allow to get multipart children in Lua * [Feature] Allow to insert multiple headers from milter headers * [Feature] Allow to print scores in subject and further extensions * [Feature] Be more error-prone in squeezed rules * [Feature] Support multiple return codes in emails module * [Feature] Use EMA for calculating averages * [Feature] Use common jit cache for all regexps * [Feature] support for CommuniGate Pro self-generated messages * [Fix] Allow to have multiple values for headers as arrays * [Fix] Do not open sockets for disabled workers * [Fix] Fix AuthservId * [Fix] Fix base64 folding in Lua API * [Fix] Fix build on non-x86 platforms * [Fix] Fix cached maps logic * [Fix] Fix compatibility with old maps query logic * [Fix] Fix crash if skip_map is used * [Fix] Fix importing static maps from UCL * [Fix] Fix parsing of unix sockets * [Fix] Fix raw_mime regexp on HTML part with no text content * [Fix] Fix tables logging * [Fix] Fix vertical tab handling in libucl * [Fix] Try to fix frequency counters * [Fix] Use better sharding for ip_score * [Fix] Use multiple results from SURBL DNS reply * [Fix] When doing AV scan select a different server for retransmit Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.41.7.4Vsevolod Stakhov2018-05-011-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Major stock config updates: - Workers are now specified in a new format worker "type" { ... } - Enable fuzzy worker to simplify local fuzzy storages configuration - Bind all workers to localhost by default to avoid security flaws * [Conf] Make more sane fuzzy_check default settings * [CritFix] Fix ucl escape for bad symbols * [Feature] Add failure symbol for AV module * [Feature] Add lazy expiration mode for new classifier schema * [Feature] Add preliminary version of maps stats plugin * [Feature] Allow to block fuzzy requests from specific networks * [Feature] Allow to change `expire` of live statistics * [Feature] Distinguish AV failure from clean result * [Feature] Further improvements of language detector by using khash * [Feature] Further optimization of the lang_detection * [Feature] Implement cluster-aware bayes expiry * [Feature] Implement exclude patterns in rspamc * [Feature] Implement glob maps in addition to regexp maps * [Feature] Implement map statistics function for lua API * [Feature] Implement stop symbols for Clickhouse collection * [Feature] Support recipients separated by commas * [Feature] Try harder to upload scripts to the Redis server * [Feature] Upgrade t1ha distribution * [Feature] use_domain_sign_inbound * [Feature] Use scores from maps if `symbols_set` is not defined * [Fix] Add resolving version of radix map helper * [Fix] Check URL before adding implicit prefix * [Fix] Do not check pid/state when using PRNG * [Fix] Fix CentOS logrotate script for systemd * [Fix] Fix slash + dot in urls * [Fix] Fix systemd version of the logrotate script * [Fix] Propagate key when import implicit array from Lua * [Fix] Strip spaces from map keys and values * [Fix] Try to fix a specific case when processing milter protocol * [Fix] Try to fix crash when a tcp connection cannot be set * [Fix] Typo use_domain_local --> use_domain_sign_local * [Fix] Various fixes to once_received module * [Project] Store hits counters for map elements Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.31.7.3Vsevolod Stakhov2018-04-101-0/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Plug bad memory leak in protocol reply * [Feature] Add avx2 codec for base64 * [Feature] Add method to receive all URL flags from Lua API * [Feature] Allow to fold headers on stop characters * [Feature] Allow to set lua_cpath from options * [Feature] Allow to specify custom rejection message in milter * [Feature] Deal with unnormalised Unicode obfuscation * [Feature] Do not detect language twice for relative parts * [Feature] Implement oversigning feature * [Feature] Implement silent logging level to minimize noise in logs * [Feature] Improve URL_IN_SUBJECT rule * [Feature] Use hashing to reduce redis attack surface * [Fix] Add oversigning for the most important headers * [Fix] add 'rewrite subject' to History dropdown * [Fix] Another fix in folding algorithm * [Fix] Do not call multimap addr for parts of addr if filter is presented * [Fix] Do not clean hostname on generic reset * [Fix] Do not create pid file in no-fork mode * [Fix] Fix fold_after case to preserve multiple spaces * [Fix] Fix folding and folding tests * [Fix] Fix hostname usage in milter mode * [Fix] Fix lua RSA verify and its tests * [Fix] Fix metadata exporter send_mail backend (#2124) * [Fix] Fix processing of '\v' in libucl * [Fix] Fix shemaless URLs detection * [Fix] Fix support of multiple headers in sign_header * [Fix] Fix usage of util.parse_mail_address * [Fix] Fix weights of dynamic squeezed rules * [Fix] Leak from bucket before checking the burst * [Fix] Stop using own localtime as DST could be messy in many cases * [Fix] Treat unnormalised URLs as obscured * [Rework] Restore leaky bucket model in ratelimit plugin * [WebUI] Add messages total to throughput summary * [WebUI] Add symbols order selector to history * [WebUI] Config: Load list on demand * [WebUI] Fix modalBody for maps that appear more than once * [WebUI] History: Fix Tooltips on paging, filtering and sorting * [WebUI] Remove a previously-attached event handler * [WebUI] Update D3 to v5.0.0 and jQuery to v3.3.1 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.21.7.2Vsevolod Stakhov2018-03-231-0/+15
| | | | | | | | | | | | | | | | | | * [Feature] Store emails in Clickhouse * [Feature] Support single quotes in config * [Feature] Use templates when publishing CH schema * [Feature] Improve Docker image * [Fix] Add rounding when printing a lot of FP variables * [Fix] Allow to disable certain actions by assigning null to them * [Fix] Disable results caching * [Fix] Fix disabling of squeezed symbols * [Fix] Fix scan time set * [Fix] Rework logic of actions setting * [Fix] Try to fix various Lua stack issues * [WebUI] Add link tag for favicon.ico * [WebUI] Display hostname:port/path in the page title Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.11.7.1Vsevolod Stakhov2018-03-201-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix lowercase comparison * [CritFix] Timezone defines seconds WEST UTC not East * [Feature] Add filename to log format * [Feature] Add lua rules squeezing * [Feature] Add related symbols analysis to rspamd_stats * [Feature] Remove upstream `X-Spam: Yes` header by default * [Feature] rspamd_stats: Output progress info on STDERR * [Feature] Whitelist for emails module * [Fix] Do not allow dependencies on self * [Fix] Do not cache metric result * [Fix] Do not trust all issuers as a client certificate * [Fix] Fix dependencies in lua squeeze * [Fix] Fix enabling/disabling squeezed rules * [Fix] Fix enabling/disabling symbols * [Fix] Fix external dependencies * [Fix] Fix processing of a single compressed file * [Fix] Fix some typos * [Fix] Fix various modules in case of empty message * [Fix] Handle callbacks that returns table of options * [Fix] Improve cached action interaction * [Fix] Make dynamic conf more NaN aware * [Fix] Never hide actions from WebUI `configuration` tab * [Project] Implementation of Lua rules squeezing Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.01.7.0Vsevolod Stakhov2018-03-121-0/+427
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add bayes_expiry as explicit module * [Conf] Adjust names and weights for neural network plugin * [Conf] Change updates url * [Conf] Default statistics is stored in Redis now * [Conf] Disable fann_redis module by default * [Conf] Fix default elastic configuration * [Conf] Fix double quote position * [Conf] Massive config rework for new structure of symbols and scores * [Conf] Rename Rambler BLs as they are now Rspamd's ones * [Conf] Use dedicated rspamd.com subdomains * [Conf] Use more data from rspamd.com fuzzy storage * [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Another portion of tokenization fixes * [CritFix] Do not send reject messages after set reply * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [CritFix] Fix memory leak in spf caching logic * [CritFix] Fix milter commands pipelining * [CritFix] Fix newlines detection * [CritFix] Fix semicolons parsing in the content type * [CritFix] Plug memory leak in zstd protocol compression * [Feature] Add ability to match score in force_actions module * [Feature] Add aes-rng PRF to libottery * [Feature] Add 'composites' debug module * [Feature] Add concept of experimental modules * [Feature] Add DKIM trace symbol * [Feature] Add EBL to the default config * [Feature] Add expected ip check for emails plugin * [Feature] Add framework to manage Redis scripts * [Feature] Add framing for the new reputation generic plugin * [Feature] Add function to show plugins stat * [Feature] Add gzip compression support for clickhouse module * [Feature] Add gzip compression support for rspamd controller * [Feature] Add gzip support when sending lua http requests * [Feature] Add json output for rspamd_stats * [Feature] Add method to do a synchronous Redis connection * [Feature] Add method to get all content-type attributes in Lua * [Feature] Add `-m` flag to configdump to show modules states * [Feature] Add mime types to extensions map * [Feature] Add more features to rescore utility * [Feature] Add more gtube like patterns to test other spam actions * [Feature] Add more metafunctions, improve logging * [Feature] Add more text attributes * [Feature] Add new configwizard command to rspamadm * [Feature] Add new tooling for stats conversation * [Feature] Add old groups migration tool * [Feature] Add plugins state variable * [Feature] Add preliminary ecdsa keys support in DKIM * [Feature] Add preliminary support of idempotent symbols * [Feature] Add Redis server wizard * [Feature] Add routine to convert old style stats to a new one * [Feature] Add some sanity checks for actions and controller * [Feature] Add statistic convertation module to configwizard * [Feature] Add sugestions logic to mempool allocator * [Feature] Add support of config transform in Lua * [Feature] Add timeout to rspamc when doing corpus test * [Feature] Add tooling to convert bayes schemas * [Feature] Add torch conditional to configuration * [Feature] Add torch-decisiontree package * [Feature] Add torch-optim contrib package * [Feature] Add TTL autodetection * [Feature] Add urls reputation to the reputation framework * [Feature] Allow floating and negative values in expressions limits * [Feature] Allow multiple CTs in full extensions map * [Feature] Allow multiple fann rules * [Feature] Allow randomly select User-Agent from a list * [Feature] Allow rspamadm commands to export methods in Lua * [Feature] Allow rule specific min_bytes in fuzzy check * [Feature] Allow to adjust symbols scores from Lua * [Feature] Allow to attach stat signature to messages * [Feature] Allow to change SMTP from via milter headers * [Feature] Allow to configure monitored * [Feature] Allow to create directories in Lua API * [Feature] Allow to disable torch and skip train samples for ANN * [Feature] Allow to discard messages dynamically * [Feature] Allow to enable/disable languages from the detector * [Feature] Allow to generate DKIM keys from rspamadm API * [Feature] Allow to get CPU flags from Lua * [Feature] Allow to have high precision timestamps in logs * [Feature] Allow to insert headers into specific position * [Feature] Allow to limit redirector requests per task * [Feature] Allow to load and use dynamic ANNs with torch * [Feature] Allow to quarantine rejected messages using milter interface * [Feature] Allow to receive signing keys from mempool vars * [Feature] Allow to reserve elements in libucl * [Feature] Allow to reuse signal handlers chains * [Feature] Allow to set custom mempool variables from settings * [Feature] Allow to set headers from settings * [Feature] Allow to set Settings-Id for all connections * [Feature] Allow to skip real action and add a header instead * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Allow to spawn asynchronous processes from Lua * [Feature] Allow to specify number of threads for ANN learning * [Feature] Allow to use global lua maps in settings * [Feature] Allow to use postfilters in composites * [Feature] Allow to verify signatures from HTTP headers in maps * [Feature] Antivirus: ordered pattern matches * [Feature] Authentication-Results: support hiding usernames * [Feature] Automatically create tables in clickhouse * [Feature] Catch next-to-last bad extension * [Feature] Check cached maps more frequently * [Feature] Check groups sanity * [Feature] Deal with obscured URLs with @ symbols * [Feature] Enhance task:store_in_file method * [Feature] Export password encryption routines to Redis * [Feature] Filter nan and inf when adding scores * [Feature] Finalize 7zip files support * [Feature] Further improvements in language detection * [Feature] Further improvements in language detection algorithm * [Feature] Generic key name expansion for Redis keys * [Feature] Hash whitelist for fuzzy_check * [Feature] Implement bayes signatures storage * [Feature] Implement buckets for Redis backend * [Feature] Implement DKIM reputation adjustments * [Feature] Implement forked workers children monitoring * [Feature] Implement headers flags in mime parser * [Feature] Implement l1/l2 regularization against the current weights * [Feature] Implement manual ANN train mode * [Feature] Implement per-user ANN support * [Feature] Implement torch based ANN learning * [Feature] Implement upstreams logic for clickhouse exporter * [Feature] Import torch to Rspamd... * [Feature] Improve allocation policy when interacting with Lua * [Feature] Improve Lua/C interaction in history_redis * [Feature] Improve multiple fuzzy results combining * [Feature] Improve parsing of DKIM keys: parse algorithm * [Feature] Improve subprocesses termination handle * [Feature] Improve symbol type parsing in Lua API * [Feature] Metadata Exporter: e-Mail Alerts: support multiple recipients; alerting senders/recipients/users (#1600) * [Feature] Milter headers: support adding/removing arbitrary headers from config * [Feature] More metatokens * [Feature] Multimap: checking of symbol options * [Feature] Multimap: template URL filter * [Feature] New bayes expiry plugin * [Feature] Periodically save rspamd stats to disk * [Feature] Preliminary import of the elasticsearch module * [Feature] Ratelimit: allow full addresses in whitelisted_rcpts * [Feature] Ratelimit: support fetching limits from Redis * [Feature] RBL: received: filtering by position & flags * [Feature] Read global maps for lua * [Feature] Redis settings: support checking multiple keys * [Feature] Rework fann plugin to be a normal post-filter * [Feature] Rework logging configuration for rspamadm case * [Feature] Rework short hashes generation to avoid FP * [Feature] Save real ucl types when exporting to Lua * [Feature] Set TCP_NODELAY for milter sockets * [Feature] Setup DKIM signing from configwizard * [Feature] Skip certain symbols from ANN classify * [Feature] Store plugins state * [Feature] Support etag for HTTP maps * [Feature] Support Expires header when using HTTP maps * [Feature] Support sending given header multiple times in lua_http * [Feature] Support sha512 in DKIM signatures * [Feature] Try to detect HTML messages better * [Feature] Use array instead of queue to reduce memory fragmentation * [Feature] Use controller port by default when connecting to local IP * [Feature] Use rdtsc where possible * [Fix] Actively load skip hashes map in fuzzy storage * [Fix] Add another workaround to display history properly * [Fix] Add definition for old glib compatibility method * [Fix] Add missing rspamadm control options to help * [Fix] Add workaround for IPv6 in sendmail * [Fix] Add workaround for system with non-XSI compatible tzset * [Fix] Allow oversigning in DKIM signatures * [Fix] Allow to check negative scores in force_actions * [Fix] Allow to have negative actions limits * [Fix] Allow to set any layers number for fann rules * [Fix] Another fix for rdtcs * [Fix] Another fix to lua xmlrpc * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Another try to fix threading in torch * [Fix] Apply language detection when adding fuzzy hashes * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Authentication Results: Fix SPF smtp.mail_from * [Fix] Auth-Results: Multiple DKIM signatures * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Check for magic when checking for an archive * [Fix] Cleanup mess with groups * [Fix] Clickhouse: Insertion in the symbols table * [Fix] Crash in URL processing * [Fix] Deal with another case when processing exceptions * [Fix] Deal with deeply nested messages more aggressively * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with non-key arguments in lua_redis.exec_script * [Fix] Deal with unknown weight * [Fix] Deal with URLs with no slashes after protocol * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] Deal with zero scores symbols * [Fix] Default monitoring domain for surbl plugin * [Fix] Delay upstream re-resolving when one upstream is defined * [Fix] Detection of maillist optimized and fixed * [Fix] DKIM signing: allow for auth_only to be false * [Fix] DMARC: require report_settings for sending reports only * [Fix] Do not allow garbadge when checking url domain * [Fix] Do not cache SPF records with PTR elements * [Fix] Do not constantly re-resolve failed upstreams with a single element * [Fix] Do not crash if no words defined * [Fix] Do not crash on empty subtype * [Fix] Do not expose spamtrap messages to SMTP reply * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not ignore short words * [Fix] Do not include idempotent/nostat symbols to checksum * [Fix] Do not override groups when converting metrics * [Fix] Do not override unix socket group when group comes before owner * [Fix] Do not skip the last character * [Fix] Do not spawn too many workers by default * [Fix] Do not stop monitored on dns errors * [Fix] Do not stop parsing headers on bad IP header * [Fix] Do not strip last character in the last word * [Fix] Do not treat script content as text * [Fix] Do not try to connect to non-supported addresses * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Erase unknown HTML entities * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix a lot of FP in chartable in mixed languages * [Fix] Fix ANN checks * [Fix] Fix ANN loading logic * [Fix] Fix another tokenization issue * [Fix] Fix autolearn parameters reading * [Fix] Fix bad archive characters stripping * [Fix] Fix bad extension check * [Fix] Fix bayes schema conversion * [Fix] Fix blacklists and DMARC in whitelist * [Fix] Fix brain-damaged torch build system * [Fix] Fix build on FreeBSD * [Fix] Fix clickhouse exporter * [Fix] Fix clickhouse schema * [Fix] Fix comparision * [Fix] Fix composites processing * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix couple of warnings * [Fix] Fix crashes in the rspamd_control path * [Fix] Fix deletion from hash * [Fix] Fix DKIM forgeries via multiple headers * [Fix] FIx dynamic conf plugin * [Fix] Fix emails detection * [Fix] Fix empty headers simple canonicalization * [Fix] Fix empty threshold check in greylisting module * [Fix] Fix encrypted legacy reply in fuzzy storage * [Fix] Fix enormous scores for R_WHITE_ON_WHITE * [Fix] Fix exceptions list in surbl * [Fix] Fix *_EXCESS_BASE64 rules * [Fix] Fix expire rounding * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix format strings * [Fix] Fix get_content method * [Fix] Fix groups override when defining symbols * [Fix] Fix learned count in new schema * [Fix] Fix learn errors propagation * [Fix] Fix loading of per-user redis backend for statistics * [Fix] Fix logging buffer corruption in case of repeated messages * [Fix] Fix lua cached elements invalidation * [Fix] Fix merging of the implicit arrays * [Fix] Fix mime_types scoring * [Fix] Fix multiple headers in DKIM headers list * [Fix] Fix null callee case in clang plugin * [Fix] Fix obscured url in format user@@example.com * [Fix] Fix parsing of the per-user script * [Fix] Fix priorities in rspamd_update, disable rules execution * [Fix] Fix processing of closed tags * [Fix] Fix processing of idempotent rules when autolearn fails * [Fix] Fix processing of multipart parts with no headers * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix PTR processing in SPF * [Fix] Fix pushing country to clickhouse asn table * [Fix] Fix random forests module * [Fix] Fix real IP parsing for some strange Exim received * [Fix] Fix Redis timeout setup * [Fix] Fix reload crash when hyperscan is enabled * [Fix] Fix reusing of redis connection after exec * [Fix] Fix sanity checks on macro value * [Fix] Fix setting of path and cpath for Lua * [Fix] Fix setting of signals when spawning a thread * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix ticks processing * [Fix] Fix upstream addrs updating * [Fix] Fix urls/emails distinguishing found in queries * [Fix] Fix user settings check * [Fix] Fix variable increment * [Fix] Fix various issues in stat_convert * [Fix] F-PROT Antivirus infection string for all known occurences * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Further fixes around floating point expressions * [Fix] Further fixes to ANN module * [Fix] Further fixes to rescore tool * [Fix] Further fixes to support ES 6 * [Fix] Further tokenization fixes * [Fix] Greylisting set phase is not idempotent * [Fix] Handle proxy copy errors * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Kill spawned processes on termination * [Fix] Load skip map from all processes as shared cache is unavailable * [Fix] Lowercase HTTP headers to make them searchable from Lua * [Fix] Lowercase words * [Fix] Lua_http: freeing * [Fix] Lua: lpeg to be loaded with rspamd_lua_add_preload, to avoid "rspamd_config_read: rcl parse error: cannot init lua file […] module 'lpeg' not found" * [Fix] Map absence is not an error * [Fix] Metadata exporter: check IP sanity * [Fix] Milter headers: custom headers: removing headers * [Fix] Milter headers: skip_local / skip_authenticated settings * [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first * [Fix] mime_types: fix next-to-last extension length check * [Fix] More hacks to deal with old configs * [Fix] Move composites second pass to the dedicated stage * [Fix] Multimap: received: filtering of artificial header * [Fix] Multiple fixes in torch based ANN plugins * [Fix] Once more (#1879) fix bad extension check * [Fix] Optimize rspamd_fstring_t reallocations * [Fix] options.local_networks setting * [Fix] Parse HREF urls without explicit prefix * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Plug memory leak * [Fix] Plug memory leak in lua_tcp * [Fix] Plug memory leak when setting email addresses from Lua * [Fix] Propagate learn/stat errors more precisely * [Fix] Ratelimit: fix whitelisted_rcpts matching * [Fix] Ratelimit: lowercase email addresses * [Fix] RBL: received: deal with missing data (#1965) * [Fix] Rebalance and slightly rework MX check plugin * [Fix] Redis key expansion: EVAL: deal with strings * [Fix] Redis script loading in DMARC; URL tags; URL reputation * [Fix] Reject invalid bh for DKIM signatures earlier * [Fix] Relax pem signature detection * [Fix] Relax unicode properties requirements for chartable module * [Fix] Remove extra noise from dkim and arc signing * [Fix] Remove hop-by-hop headers in proxy * [Fix] Remove incorrect method `task:set_metric_subject` * [Fix] Replace space like characters in headers with plain space * [Fix] Restore old style ratelimits support * [Fix] Rework elasticsearch plugin * [Fix] Rewriting subjects via force actions module * [Fix] RPM postinstall * [Fix] Sanitize IP in history redis * [Fix] Select the correct signature when doing simple canon * [Fix] Set CLOEXEC flag on files opened * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Settings: header: deal with multiple settings (#1988) * [Fix] Skip checks if both extensions are not bad * [Fix] Skip nostat tokens when get number of tokens * [Fix] Some more fixes towards emails detection * [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't match * [Fix] Stop using of g_slice... * [Fix] Switch rspamadm logging to message level * [Fix] Symbol 'FANNR_SPAM' has its score defined.. * [Fix] Table parameter for rspamd_config:add_doc() * [Fix] Treat 'rewrite subject' as spam action * [Fix] Try harder in passing IPv6 addresses * [Fix] Try harder to find rfc822 notifications * [Fix] Try harder to find urls * [Fix] Use decoded values when parsing mime addresses * [Fix] Use full URL when making an HTTP request * [Fix] Use greylisting threshold in greylisting module * [Fix] Use n_words attribute from ngramms * [Fix] Use raw urls when sending requests to redirector * [Fix] Use the right boolean operator on error check * [Fix] Use weight from map for fuzzy scoring * [Fix] Various fixes to elastic plugin * [Fix] Various fixes to fann_redis instantiation * [Fix] Various improvements in language detection * [Fix] Virus infection string for F-PROT Antivirus * [Fix] Virus infetction string for F-PROT Antivirus * [Fix] WebUI: use relative path for savemap (#1943) * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [Fix] Write configuration changes as UCL config * [Project] Add detection logic for words * [Project] Add fast debug logging infrastructure * [Project] Add more flags to languages * [Project] Add n-gramms data files * [Project] Add ngramms frequencies detector * [Project] Add random words selection logic * [Project] Add unigramms to language detection as well * [Project] Convert all C modules to fast debug infrastructure * [Project] Detect some languages based on unicode script * [Project] Enable fast debug lookup for some modules * [Project] Enable language detector init in scanner workers * [Project] Further improvements to language detector * [Project] Implement logic of ngramms application * [Project] Improve weighting in lang_detection * [Project] Initialize language detector * [Project] Preliminary version of ngramms based language detector * [Project] Preliminary version of the new stat_convert * [Project] Remove old language detector * [Project] Rework language detection ngramms structure * [Project] Start language detection project * [Project] Start rework of language detection to improve quality * [Project] Use fast debug logging check * [Rework] Add frame for new reputation based IP score module * [Rework] Continue stat_convert rework task * [Rework] Implement new version of fuzzy replies * [Rework] Improve readability of xmlrpc API * [Rework] Kill metrics!11 * [Rework] Ratelimit module * [Rework] Rename fann_redis to neural plugin * [Rework] Reorganize mime_types module * [Rework] Rework rescore utility * [Rework] Rewrite model and learning logic for rescore * [Rework] Run post-loads when all initialization is completed * [Rework] Simplify lua path initialization * [Rework] Start major stat_convert rework * [Rework] Start mempool fragmentation reduce project * [Rework] Start moving of fann redis to torch * [Rework] Stop embedding rspamadm scripts into C * [Rework] Use floating point arithmetics in Rspamd expressions * [Rework] Use frequencies distribution in language detector * [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages * [WebUI] Compact graph selectors * [WebUI] Escape strings inside HTML in history * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Migrate widgets to D3 v4 * [WebUI] Restore passwordless login support (#2003) * [WebUI] Show symbol descriptions as tooltips in history * [WebUI] Stop using commas in pie chart tooltips * [WebUI] Update D3 and jQuery * [WebUI] Update D3Evolution 1.0.0 -> 1.1.0 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.6.6Vsevolod Stakhov2018-03-121-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [Feature] Allow to insert headers into specific position * [Feature] Allow to receive signing keys from mempool vars * [Feature] Authentication-Results: support hiding usernames * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] DKIM signing: allow for auth_only to be false * [Fix] Do not crash on empty subtype * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not skip the last character * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix bad archive characters stripping * [Fix] Fix comparision * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix empty headers simple canonicalization * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix parsing of the per-user script * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix Redis timeout setup * [Fix] Fix sanity checks on macro value * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix urls/emails distinguishing found in queries * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Metadata exporter: check IP sanity * [Fix] Multimap: received: filtering of artificial header * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Remove hop-by-hop headers in proxy * [Fix] Sanitize IP in history redis * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Try harder in passing IPv6 addresses * [Fix] WebUI: use relative path for savemap (#1943) * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Restore passwordless login support (#2003) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Typo: comparisionViktor Szépe2017-11-141-1/+1
|
* [Minor] Sync ChangeLogVsevolod Stakhov2017-11-051-0/+70
|
* Release 1.6.3Vsevolod Stakhov2017-09-031-0/+19
| | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix semicolons parsing in the content type * [Feature] Add EBL to the default config * [Feature] Allow to configure monitored * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Multimap: checking of symbol options * [Feature] Redis settings: support checking multiple keys * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Fix get_content method * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Lua_http: freeing * [Fix] Milter headers: custom headers: removing headers * [Fix] Parse HREF urls without explicit prefix * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [WebUI] Escape strings inside HTML in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.6.21.6.2Vsevolod Stakhov2017-07-081-0/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Remove Rambler email bl for now * [Conf] Switch RAMBLER_URIBL to a locally managed source * [CritFix] Switch from ragel to C for Content-Type parsing * [Feature] Add `-e` option for lua_repl * [Feature] Add per-domain emails normalisation rules * [Feature] Add sessions cache to debug dangling sessions * [Feature] Add short_text_direct_hash for fuzzy check module * [Feature] Add text_part:get_stats function * [Feature] Allow to add custom processing script for surbl * [Feature] Allow to check reply-to email * [Feature] Allow to customize spam header, remove existing spam headers * [Feature] Allow to disable specific workers in the config * [Feature] Allow to discard messages instead of rejection * [Feature] Allow to specify custom delimiter in emails plugin * [Feature] Allow to specify custom User-Agent for rspamc * [Feature] Allow to store symbols data in Clickhouse * [Feature] Allow to use HTTPS when connecting to Clickhouse * [Feature] Enable sessions cache tracking for milter connections * [Feature] Implement per-line mode in lua_repl (like `perl -p`) * [Feature] Implement rdns-curve plugin based on rspamd cryptobox * [Feature] Improve maps cached data lifetime * [Feature] Improve maps checking frequency * [Feature] Improve monitored timeouts logic * [Feature] milter_headers: add `extended_headers_rcpt` option * [Feature] Milter headers: Add X-Spam-Flag to rmilter-compatibility headers * [Feature] Milter headers: remove-header routine * [Feature] Multimap: received filters for extracting TLDs from hostnames * [Feature] Normalize email aliases in emails module * [Feature] Re-add rambler email bl (as hashed list) * [Feature] Reload file maps more frequently * [Feature] Rework newlines strip parser one more time * [Feature] Skip updates for messages scanned via controller * [Feature] Split long DKIM public keys * [Feature] Store more data when stripping newlines * [Feature] Support SPF macros transformations * [Feature] Support suppressing DMARC reports for some domains * [Fix] Add missing `break` statement * [Fix] Allow modifiers in SPF macros * [Fix] DKIM sign tools: edge-cases around use_esld * [Fix] Do not cache SPF records with macros * [Fix] Do not overwrite score when setting pre-action * [Fix] Fix comparision logic * [Fix] Fix DKIM base64 folding for milter flagged messages * [Fix] Fix emails module configuration * [Fix] Fix folding for arc headers when milter interface is used * [Fix] Fix gmail dots removal * [Fix] Fix rspamc detection in greylist module * [Fix] Fix some more issues with HTTP maps * [Fix] Milter sessions can live forever * [Fix] Normalize fuzzy probability better * [Fix] Plug memory leak * [Fix] RBL: Fixed hashed email address lookups * [Fix] Try to deal with brain-damaged milter behaviour * [Fix] Use `\n` to fold headers for milter * [Rework] Allow to use custom callback for monitored checks * [Rework] Further steps towards one process monitoring * [Rework] Send health checks from a single worker * [WebUI] Round-up throughput summary values Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* [Minor] Massive spelling correctionAlexander Moisseev2017-06-271-11/+11
| | | | by a bot https://github.com/ka7/misspell_fixer
* Release 1.6.11.6.1Vsevolod Stakhov2017-06-141-0/+9
| | | | | | | | | | | | * [Fix] Allow to init resolver without rspamd_config * [Fix] Do not crash when resolver failed to initialize * [Fix] Fix abstract context layout * [Fix] Fix CGP helper reply parsing * [Fix] Fix crashes when socket write errors occur * [Fix] Fix parsing IPv6 nameservers in resolv.conf * [Fix] Milter: Don't defer on "greylist" action Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.6.01.6.0Vsevolod Stakhov2017-06-121-0/+146
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add rspamd_proxy to the default configuration set * [Conf] Add sample arc module config * [Conf] Do away with systemd specifics completely * [Conf] Increase min_bytes to avoid FP * [Conf] Remove ratelimits from default configuration * [CritFix] Fix accepting on IPv6 sockets * [CritFix] Fix corruption when multiple fuzzy are defined * [CritFix] Fix learn condition in fuzzy check * [CritFix] Fix memory leak in fuzzy check * [CritFix] Fix memory leak in maps scheduling * [CritFix] Paese the last character in DKIM signature correctly * [CritFix] Zero fill sockaddr_un * [Feature] Add ability to add doc strings by example * [Feature] Add API to verify DKIM (and ARC) signatures * [Feature] Add compression/decompression to proxy * [Feature] Add count to url structure * [Feature] Add initial support of the new protocol reply * [Feature] Add Lua plugin spamtrap * [Feature] Add `monitored_address` for rbls * [Feature] Add new schema for bayes tokens * [Feature] Add preliminary ARC support to dkim code * [Feature] Add preliminary support of ARC signing * [Feature] Add rules to detect bad 8bit characters in From and To * [Feature] Add scanning support for milter protocol * [Feature] Add support for bidirectional symbols in rspamd_stats * [Feature] Add support for static maps * [Feature] Add support of maps with multiple regexps matches * [Feature] Add `text_multiplier` param * [Feature] Add the preliminary ARC plugin * [Feature] Add top redirector targets rank * [Feature] Allow async events to be registered from LUA rules * [Feature] Allow storing bayes tokens in Redis * [Feature] Allow to exclude specific domains from mx check * [Feature] Allow to have a stack of watcher finalisers * [Feature] Allow to pass hostname to `-i` flag in Rspamc * [Feature] Allow to set custom user agent in url redirector * [Feature] Allow to use custom callback when parsing resolv.conf * [Feature] Allow to use domain from authenticated user * [Feature] Bayes expiry plugin * [Feature] Check dkim sign keys for modifications * [Feature] DKIM signing: sign_networks/local address specific use_domain settings * [Feature] DMARC: Support excluding domains from sampling * [Feature] Expire processing items for URL redirector aggressively * [Feature] Fix surbl monitored for IP lists, add `monitored_domain` option * [Feature] Implement caching for dkim body hashes * [Feature] Implement milter protocol scan reply * [Feature] Improve omograph phishing detection * [Feature] Initial support of self-scan in Rspamd proxy * [Feature] Keep track of headers in milter interface * [Feature] Milter headers: better controls for local/authenticated * [Feature] Multimap: email:domain:tld filter * [Feature] Preliminary DMARC reporting implementation * [Feature] Reuse stemmers in the cache * [Feature] Rework confighelp to load Lua plugins * [Feature] Rework hfilter to use hyperscan if possible * [Feature] Rework lua RSA API * [Feature] Rmilter_headers: approximate rmilter's extended_spam_headers * [Feature] Start integration of milter support in proxy * [Feature] Store average words length and short words count * [Feature] Store hash of headers order and names * [Feature] Support MTA name header * [Feature] Support multiple types of dkim signing in Lua * [Feature] Support numeric arguments for Redis requests * [Feature] Use headers hash in bayes metatokens * [Feature] Use normal resolv.conf rules of rotation in Rspamd * [Feature] Use version 2 proto for checking messages * [Fix] Allow to follow symlinks when safe * [Fix] Append MX name for authentication results as required * [Fix] Change default text multiplier from 0.5 to 2.0 * [Fix] Check min_bytes for images as well * [Fix] Deal with 7bit charsets properly * [Fix] Deal with 8bit characters in email addresses * [Fix] Deal with unpaired <a> tags * [Fix] Detect confighelp in plugins initialisation * [Fix] Disable certain checks for utf spoof detection * [Fix] DKIM Signing: avoid nil index when From header is missing * [Fix] Do not add exact hashes from different parts * [Fix] Do not check DMARC if SPF or DKIM were not checked * [Fix] Do not check URLs that are resolved to be redirected * [Fix] Do not set bayes probability if we don't use it * [Fix] Do not stop on illegal unicode points - replace them * [Fix] Fix another race condition in arc checks * [Fix] Fix arc count logic * [Fix] Fix ARC signing * [Fix] Fix brain-damaged spamc protocol for now * [Fix] Fix calling for peak functions * [Fix] Fix couple of issues in FORWARDED rule * [Fix] Fix CTE propagation from parent containers to children parts * [Fix] Fix errors processing in the controller * [Fix] Fix format string in milter * [Fix] Fix issues in SPF macros parsing * [Fix] Fix logging format string * [Fix] Fix logic of cached passwords check * [Fix] Fix lowercasing of stemmed words * [Fix] Fix LRU elements removal * [Fix] Fix memory leak when accepting from unix sockets * [Fix] Fix milter connections persistence * [Fix] Fix objects merging in UCL * [Fix] Fix order of operations to avoid race condition * [Fix] Fix parsing of long regexp types * [Fix] Fix passing data to log helper when many symbols defined * [Fix] Fix pools management for milter session * [Fix] Fix processing of the watchers * [Fix] Fix queue id macro in milter * [Fix] Fix R_BAD_CTE_7BIT rule * [Fix] Fix Redis timeout set * [Fix] Fix REPLYTO_UNPARSEABLE rule * [Fix] Fix setting of email address * [Fix] Fix some more issues about duplicated fuzzy requests * [Fix] Fix spamc support in rspamd proxy * [Fix] Fix syntax error in spamtrap plugin * [Fix] Fix url counts for href urls * [Fix] Fix url handling in the protocol * [Fix] Multimap: Received IP filters with Redis * [Fix] Oops, fix d9d0fa5e86db2f4470d34395a233b450478b2f60 * [Fix] Parse rgb[a](x,x,x[,x]) css colors * [Fix] Phishing: strict_domains * [Fix] Reduce maps aggressiveness * [Fix] Reresolve upstreams even if there is a single server there * [Fix] Rspamadm grep: Disable Lua patterns in string search by default * [Fix] Skip text parts when checking binary parts in fuzzy check * [Fix] Support v2 checks in controller * [Fix] Treat empty address as valid * [Fix] Try harder to detect CTE * [Fix] Try to deal with v4 mapped to v6 addresses on accept * [Fix] Use dkim signing callback properly * [Fix] Use non-volatile memory for storing data * [Fix] Use static maps instead of ugly hack for radix_from_config * [Fix] Use the same pool for related sessions * [Rework] Continue modularisation for lua library * [Rework] Initial milter protocol support * [Rework] Make log pipes worker agnostic, add scanners API * [Rework] Move authentication results generation to a separate routine * [Rework] Move common DKIM functions to a separate lua module * [Rework] Move global functions to a separate directory * [Rework] Prepare dkim module for ARC checks * [Rework] Propagate ucl variables from the command line * [Rework] Remove multiple metrics support from Rspamd * [Rework] Stop using name 'rmilter' for the modern protocol * [Rework] Use LFU algorithm in LRU cache * [Rules] Fix received TLS rules * [Rules] Improve URL_COUNT_ODD rule * [WebUI] Fix add header filter in history * [WebUI] Use modern protocol for checking messages Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* [Minor] Update ChangeLog from 1.5 branchVsevolod Stakhov2017-06-041-0/+74
|
* [Minor] debian: Update BD and refresh packagingSebastien Badia2017-05-131-1/+1
|
* Release 1.5.61.5.6Vsevolod Stakhov2017-04-191-0/+15
| | | | | | | | | | | | | | | | | * [Feature] Add unigramms support in bayes * [Feature] Allow configurable sign headers for DKIM * [Feature] Allow to add unigramm metatokens from Lua * [Feature] DKIM Signing: envelope match exception for local IPs * [Feature] UCL: register parser variables from Lua * [Fix] Always try to adjust filename * [Fix] Do extra copy to ensure that original content is never touched * [Fix] Fix SPOOF_REPLYTO rule * [Fix] Ignore Rmilter added Received * [Fix] More fixes for hashed email dnsbls * [Fix] Plug memory leak in chartable module * [WebUI] Display multiple alerts at once Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.5.5Vsevolod Stakhov2017-04-101-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix classifier learning with Redis backend * [CritFix] Fix issue when parsing encoded rfc822/messages * [Feature] Add escaped version of lua_ucl import * [Feature] Add task:headers_foreach function * [Feature] Allow to process filenames from content type * [Feature] Allow to query hashed emails * [Feature] Ignore bayes with mostly metatokens or with too few text * [Feature] Probabilistically skip metatokens * [Feature] Retrieve all virus names from SAVAPI * [Feature] Rework classifiers lua metatokens * [Feature] Store headers order * [Feature] Store text tokens inside bayes tokens * [Feature] Use cached shingles keys * [Fix] Add missing score normalisation for HFILTER_URL_ONLY * [Fix] Avoid lookup in absent hash * [Fix] Check return values from Lua functions called from C * [Fix] Do not count sending and loading time in rspamc * [Fix] Escape json strings for controller rejplies from Lua * [Fix] Fix archive scans for savapi * [Fix] Fix domain_only emails RBL * [Fix] Fix ip_score map configuration * [Fix] Fix JSON output for history_redis * [Fix] Fix one character length substrings search * [Fix] Fix parsing of non-RFC compatible Exim received * [Fix] Fix parsing of options for workers with the same type * [Fix] Fix processing of small tokens vectors * [Fix] Fix rfc2047 tokenization * [Fix] Fix typo * [Fix] More fixes for inplace decoding * [Fix] Try to avoid modifications of the original data * [Fix] URL redirector: Fix call to is_redirector * [Rework] Set token data as uint64_t instead of chars array * [WebUI] Check if neighbours' history backend versions match * [WebUI] Disable phrase connectors replacement in history filtering * [WebUI] Disable phrase connectors replacement in symbols filtering * [WebUI] Do not hide messages with bad subject, just replace it with '???' * [WebUI] Fix error message * [WebUI] Fix history v2 display * [WebUI] Fix legacy history * [WebUI] history: break To address lists on commas * [WebUI] Increase default timeout to 20 seconds * [WebUI] Save some history table space Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.5.41.5.4Vsevolod Stakhov2017-03-281-0/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add history_redis default configuration * [Feature] Add spoofed rules * [Feature] Add URL_IN_SUBJECT rule * [Feature] Allow to get task's subject * [Feature] Allow to specify maximum number of shots for symbols * [Feature] Distinguish URLs found in Subject * [Feature] Memoize LPEG grammars * [Feature] Parse else parts in SA rules * [Feature] Process subject for mixed characters * [Feature] Resolve url chains in url_redirector module * [Feature] Stat greylisted messages as greylisted not soft-rejected * [Feature] Support checking for redirector in Lua SURBL * [Feature] Support tag_exists SA function * [Feature] Work with broken rfc2047 tokens * [Fix] Check all watcher's dependencies * [Fix] Do not compile hyperscan with no SSSE3 support * [Fix] Do not crash if cannot decode qp encoded part * [Fix] Fix dependencies of DKIM when multiple signatures are found * [Fix] Fix lists in whitelist plugin * [Fix] Fix one-shot symbols weight calculations * [Fix] Fix options and shots match * [Fix] Fix order of symbol options * [Fix] Fix parsing of dot at the end of the address * [Fix] Fix parsing of lua table arguments * [Fix] Fix processing of subject words * [Fix] Fix string split memoization * [Fix] Fix templates grammar usage * [Fix] Fix various issues related to Lua stack manipulation * [Fix] Force actions: Use postfilter if we have honor_action / require_action * [Fix] Further fixes to avoid PHISHING FP * [Fix] Preserve order of options in symbols * [Fix] Rspamadm grep: deal with unusually-formatted logs * [Fix] Use hostname suffix when dealing with history * [Rework] Remove outdated SA rules * [WebUI] Add flexible columns * [WebUI] Add footable * [WebUI] Add sender, recipients and subject columns * [WebUI] Allow message-id break * [WebUI] Fix history clustering * [WebUI] Fix history display * [WebUI] Fix sorting * [WebUI] Humanize sizes * [WebUI] Initial move towards footable * [WebUI] Remove datatables * [WebUI] Replace `.values` method with `.map` * [WebUI] Rework v2 symbols display * [WebUI] Try to normalize frequencies * [WebUI] Unbreak WebUI * [WebUI] Use Footable to draw Throughput summary table Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.5.31.5.3Vsevolod Stakhov2017-03-171-0/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add composite for hacked wordpress phishing * [CritFix] Fix base64 decoding when there are unparseable characters * [Feature] Additional symbol metadata in metadata exporter * [Feature] Add method to get protocol reply from Lua * [Feature] Add symbols when tagged rcpt/sender are normalised * [Feature] Add task:get_symbols_all() function * [Feature] Allow multiple formats of DKIM signing key * [Feature] Allow to cache and use flexible protocol reply * [Feature] Allow to set one_shot flag from register_symbol * [Feature] Allow to skip certain types of hashes when learning fuzzy * [Feature] Cache and insert scan time into the protocol * [Feature] Detect newlines in rspamc --mime * [Feature] DKIM signing: support use of maps * [Feature] Greylist: Support excluding low-scoring messages from greylisting * [Feature] Implement lua history in controller * [Feature] Implement redis history querying * [Feature] Preliminary implementation of redis history plugin * [Feature] Support using request headers in settings * [Fix] Change default template to deal with non-ASCII characters * [Fix] Deal with lists of maps in whitelist module * [Fix] DKIM signing: use domain-specific signing key * [Fix] Do not reallocate completed zstd buffer * [Fix] Do not use local_addrs in proxy * [Fix] Fix crash when resolver is undefined * [Fix] Fix double free when closing lua_tcp connections * [Fix] Fix for lua 5.3 * [Fix] Fix freeing of arrays iterators * [Fix] Fix issue with task:get_symbol and symbols with no metric * [Fix] Fix log line duplication in `rspamadm grep` * [Fix] Fix memory corruption on termination * [Fix] Fix out-of-bound access in base64 decode * [Fix] Fix ratelimit + greylisting * [Fix] Fix subject rewriting * [Fix] Fix task:set_recipients function * [Fix] Fix URI_COUNT_ODD rule * [Fix] Follow the traditional symbols conventions in RCPT_COUNT rule * [Fix] Greylist: Suppress greylist action for whitelisted hosts too * [Fix] Metadata exporter: use rule-specific settings for emails * [Fix] Properly set missing fields in exporter * [Fix] Proxy: max_retries option * [Fix] RCPT_COUNT fixes * [Fix] Rework HAS_X_PRIO rule to match symbols conventions * [Fix] Update issues in ac-trie * [Fix] Use optimised base64 decoding in DKIM * [WebUI] Add preliminary v2 history parser * [WebUI] Allow different history parsers * [WebUI] Display symbols * [WebUI] Rework history v2 function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.5.21.5.2Vsevolod Stakhov2017-03-071-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add default config for spamassasssin plugin * [Conf] Add default configuration for antivirus module * [Conf] Add dkim signing docs * [Conf] Add mx_check default config * [Conf] Add replies config * [Conf] Add trie default config * [Feature] Add heuristic to find text parts in files * [Feature] Add rule to detect broken content type * [Feature] Allow to extract CTE in Lua API * [Feature] Allow to set from address for a lua_task * [Feature] Allow to set recipients of a task from Lua * [Feature] Enchance text_part:get_content method * [Feature] Remove + aliases from emails * [Feature] Support rmilter block and dkim signature in CGP helper * [Feature] Support running event loop from Lua * [Fix] Antivirus: use scanner-specific redis prefix * [Fix] Couple of fixes for DKIM signing module * [Fix] Distinguish missing and broken mandatory headers * [Fix] Do more heuristical detection for missing CTE * [Fix] Do not resort cache on each check * [Fix] Fix CGP escaping * [Fix] Fix MISSING_MIME_VERSION rule for plain messages * [Fix] Fix parsing of cte in expressions * [Fix] Fix partial matches in rspamadm grep * [Fix] Fix setting class on style field * [WebUI] Auto-switch Throughput units to `msg/min` for very low rate * [WebUI] Update D3Evolution to 0.0.2 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.5.11.5.1Vsevolod Stakhov2017-03-021-0/+9
| | | | | | | | | | | | * [CritFix] Fix processing of stop_patterns with `\0` character * [CritFix] Fix setting of raw key for signing * [Fix] Fix lua exports from plugins during reload * [Fix] Fix prefilters action scores * [Fix] Fix symbols processing order * [Minor] Help cmake find gthread * [Minor] Some cmake fixes Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.5.0Vsevolod Stakhov2017-03-011-0/+368
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add configurations for asn, clickhouse and dcc * [Conf] Add default config for url redirector plugin * [Conf] Add the default config for greylist module * [Conf] Allow to edit all local maps from WebUI by default * [CritFix] Deal with absent headers in DKIM * [CritFix] Do not trust remote shingles count * [CritFix] Fix bad memory leak in TLS certificates validation * [CritFix] Fix critical memory issues with radix maps * [CritFix] Fix descriptors leak on reload * [CritFix] Fix headers selection in DKIM verification * [CritFix] Fix parsing of boundaries that end with `--` * [CritFix] Repair PTR_ARRAY_FOREACH macro * [Feature] Add CORS support to the controller * [Feature] Add FROM_NAME_EXCESS_SPACE rule * [Feature] Add REPLYTO_EMAIL_HAS_TITLE rule * [Feature] Add `caseless_hash` method to `lua_util` * [Feature] Add `rip` keyword to ratelimit module * [Feature] Add a simple benchmark for content type parsing * [Feature] Add boundaries parsing in content type * [Feature] Add charset detection for text parts * [Feature] Add content disposition parser * [Feature] Add fallback if too many updates are failing * [Feature] Add function to convert struct tm to time using timezone * [Feature] Add function to normalize HTTP paths * [Feature] Add fuzzy collection plugin * [Feature] Add fuzzy logic for images * [Feature] Add gmime parser to mime_tool * [Feature] Add heuristic to detect broken messages * [Feature] Add heuristic to find displayed URLs * [Feature] Add heuristic to process broken email addresses * [Feature] Add images normalization * [Feature] Add mechanism for disabling composites (Fixes #1270) * [Feature] Add method to create regexp from a glob pattern * [Feature] Add mime encoding manipulation routines * [Feature] Add mime tool to explore messages * [Feature] Add more meta tokens from received headers * [Feature] Add neighbours option to support Rspamd cluster in WebUI * [Feature] Add new function to parse mime addresses * [Feature] Add new methods for lua_tcp * [Feature] Add own headers decoding routine * [Feature] Add own routine to generate a message id * [Feature] Add parser for SMTP date * [Feature] Add per-task lua cache to reuse 'heavy' objects * [Feature] Add plugins list path in WebUI * [Feature] Add preliminary multipart support * [Feature] Add preliminary version of DKIM signing module * [Feature] Add profiling support in client output * [Feature] Add rfc2047 grammar * [Feature] Add rfc2047 variant for QP decoding * [Feature] Add rmilter_headers module (Fixes #1227) * [Feature] Add sse42 version of base64 decoding * [Feature] Add ssse3 and avx2 base64 decoders * [Feature] Add support of libgd * [Feature] Add the preliminary version of redirects resolver in Lua * [Feature] Add ucl_object_iterate_full function * [Feature] Add url encoding function * [Feature] Allow SOA requests in lua dns * [Feature] Allow custom parse types in lua ucl * [Feature] Allow plugins to register webui handlers * [Feature] Allow to add options explicitly to symbols * [Feature] Allow to call a callback when symbol frequency is on peak * [Feature] Allow to call redirector script from SURBL * [Feature] Allow to create variable length dkim keys * [Feature] Allow to have module specific options for Redis in plugins * [Feature] Allow to pass sign key directly from Lua * [Feature] Allow to register configuration docs from Lua API * [Feature] Allow to return options as a table * [Feature] Allow to set peak callbacks from Lua * [Feature] Allow to specify custom method for a message * [Feature] Allow to store dkim keys in Redis * [Feature] Allow to store messages in files * [Feature] Apply DCT using AAN for fuzzy signature * [Feature] Avira SAVAPI support * [Feature] Cache and simplify DCT and jpeg decode * [Feature] Cache libicu convertors * [Feature] Detect URLs with suspicious omographs * [Feature] Do not increase score for duplicate options * [Feature] Do not trust CTE, check base64 and qp strictly * [Feature] Dynamic reputation in URL reputation plugin * [Feature] Extend redis lock when learning spawned * [Feature] Filter non-utf chars from all decoded headers * [Feature] Fix phishing detection for IDNA urls * [Feature] Ignore bad symbols on base64 decoding * [Feature] Ignore too wide elements in SPF * [Feature] Implement fuzzy collection mode * [Feature] Implement helo maps in multimap * [Feature] Implement human readable buckets configuration * [Feature] Implement min-hash shingles for DCT data from images * [Feature] Implement new algorithm for fuzzy hashes of images * [Feature] Implement new unicode normalizer * [Feature] Implement quoted printable decoding * [Feature] Implement received headers flags * [Feature] Implement rspamdgrep tool * [Feature] Implement sane checksum for config file * [Feature] Implement url tags concept * [Feature] Improve detection of omographs using libicu * [Feature] Improve url redirector module * [Feature] Multimap: Received header processing * [Feature] Multiple improvements in the maps * [Feature] New URL filters in multimap * [Feature] Plugin to force actions on selected symbols * [Feature] RBL module: support hashing for emails and helo RBL * [Feature] Reuse URL tags in SURBL module * [Feature] Rework RRD ds count, add conversion path * [Feature] Rework surbl module to avoid extra redirector calls * [Feature] Send config id to the WebUI * [Feature] Simplify HTTPCrypt client support * [Feature] Skip processing for large images * [Feature] Start collection only mode implementation for fuzzy storage * [Feature] Start import of the optimized base64 decode * [Feature] Store all received headers in lua * [Feature] Store relational order of all headers in a message * [Feature] Support DKIM signing in Lua plugins * [Feature] Support HTTPCrypt client in lua_http * [Feature] Support setting SMTP message in multimap * [Feature] Support setting metric subject from Lua * [Feature] Support setting subject in force actions module * [Feature] Treat v6 mapped addresses as v4 addresses * [Feature] URL reputation plugin * [Feature] Use Redis instead of memcached in URLs redirector * [Feature] Use Rspamd rfc2047 decoder instead of gmime one * [Feature] Use a different normalization for fuzzy images * [Feature] Use normalized images in fuzzy hashes * [Feature] Use own code for parsing of date * [Feature] Use shingles for images fuzzying * [Feature] Use t1ha for hashes, allow inlining * [Feature] Use t1ha instead of metrohash and xxhash32 * [Feature] Various new features in metadata exporter module * [Feature] rmilter_headers: authentication-results (#78) * [Fix] Add additional check to mark redis connection inactive * [Fix] Add packed attribute for protocol structure * [Fix] Adopt OMOGRAPH_URL rule * [Fix] Allow static maps * [Fix] Allow to disable classifiers checks using settings and conditions * [Fix] Another try to fix 0 length maps * [Fix] Another try to fix corruption during maps reload * [Fix] Another try to fix descriptors leak * [Fix] Another try to fix reload and logger * [Fix] Antivirus module: register virtual symbols for patterns * [Fix] Avoid extensive reallocs * [Fix] Avoid mempool leak in SA plugin on reload * [Fix] Avoid race condition on saving cache and reload * [Fix] Avoid reusing g_error (Fixes #1262) * [Fix] Break pool connection on fatal redis errors * [Fix] Check for NaN properly * [Fix] Couple of fixes for date parsing * [Fix] Date header timezone adjustments (#1279) * [Fix] Deal with EOF properly * [Fix] Decode filename in content disposition * [Fix] Disable fuzzy images by default * [Fix] Disable zero-copy mode for text parts to avoid crashes * [Fix] Do not destroy session when not all finish scripts are done * [Fix] Do not greyscale images * [Fix] Do not leave parent-less workers processes on fatal errors * [Fix] Do not lowercase Content-Disposition to perform decoding * [Fix] Do not penalize characters just after numeric prefix * [Fix] Do not refork workers that are intended to die * [Fix] Do not set pre-result and update records for no `Queue-ID` messages * [Fix] Do not skip post-filters when pre-filters have set some results * [Fix] Do not stop symbols planning if async events are pending * [Fix] Do not try to set keys for unencrypted requests in proxy * [Fix] Encode URLs according to rfc3986 * [Fix] Encode URLs before sending them to the protocol * [Fix] Filter bad characters from message id * [Fix] Fix CTE detection heuristic * [Fix] Fix Content-Type in HTTP requests * [Fix] Fix IDN eslds phishing checks * [Fix] Fix adding maps from config in Lua * [Fix] Fix another reload memory issue * [Fix] Fix argument returned on redis backend errors * [Fix] Fix assertion in graph handling * [Fix] Fix body trie matching * [Fix] Fix build * [Fix] Fix byte array expansion during toutf8 conversion * [Fix] Fix charset normalisation * [Fix] Fix checking of DKIM bodies that needs just `\n` to be added * [Fix] Fix couple of cornercases with email addresses * [Fix] Fix couple of issues * [Fix] Fix dependencies tracking for callback symbols * [Fix] Fix detection of jpeg size * [Fix] Fix errors handling in fuzzy backend initialization * [Fix] Fix fuzzy hashes count * [Fix] Fix globbing and convert lists to arrays in fuzzy_check * [Fix] Fix heuristical CTE detection for QP encoding * [Fix] Fix ignoring of bad text parts * [Fix] Fix indexes in array access, interleave loop * [Fix] Fix int64 -> double conversion * [Fix] Fix invalid memory access on reload * [Fix] Fix issues with empty updates * [Fix] Fix issues with quoted-printable encoding * [Fix] Fix keys names * [Fix] Fix lots of issues in mime parser code * [Fix] Fix lua maps load * [Fix] Fix macro name * [Fix] Fix mas group score calculations * [Fix] Fix matching of the same patterns from different tries * [Fix] Fix memory corruprtion and leak * [Fix] Fix memory leak in HTTP maps * [Fix] Fix memory leak in expression destroying * [Fix] Fix memory leak in parsing of mime names * [Fix] Fix memory leak in safe ucl iterators * [Fix] Fix memory leak on reload in plugins * [Fix] Fix modules reconfigure on reload * [Fix] Fix monitored setup fro URLBLs with IP addresses * [Fix] Fix name of var * [Fix] Fix new rrd updates * [Fix] Fix out of bounds access * [Fix] Fix parsing messages with no body * [Fix] Fix parsing of '=' character in headers * [Fix] Fix parsing of messages with no content type * [Fix] Fix plugins callbacks in webui * [Fix] Fix possible memory corruption in redis pool * [Fix] Fix probability calculations for fuzzy redis backend * [Fix] Fix processing errors in lua_tcp * [Fix] Fix processing of emails with name only * [Fix] Fix processing of non-multipart messages * [Fix] Fix processing of parts with no valid content type * [Fix] Fix race condition in SIGUSR2 handler * [Fix] Fix redis options parsing when no redis servers are defined * [Fix] Fix reload and hyperscan ready event * [Fix] Fix reload memory issue * [Fix] Fix rra_ptr conversion * [Fix] Fix rrd file conversion * [Fix] Fix setting of content-type attributes * [Fix] Fix signing headers creation in DKIM * [Fix] Fix stddev calculations * [Fix] Fix surbl plugin to work with composite maps * [Fix] Fix timezones parsing * [Fix] Fix tokens usage * [Fix] Fix urls and emails hashes * [Fix] Fix usage of unsafe ucl iterators * [Fix] Fix work with broken utf8 tokens * [Fix] Fix writing of user to roll history * [Fix] Forgotten worker * [Fix] Further memory leaks fixes * [Fix] Ignore lua metatokens in bayes for now * [Fix] Improve OMOGRAPH_URL rule * [Fix] Lua IP from string should be invalid if parsing failed * [Fix] Miltiple fixes to new lua_tcp, add debugging * [Fix] More fixes for iterators cleanup * [Fix] More fixes to logger initialization * [Fix] More heuristic fixes for phishing detection * [Fix] More leaks eliminated * [Fix] More leaks... * [Fix] More random fixes for reload... * [Fix] Multimap: Fixes for email filters * [Fix] Multiple fixes for fann module * [Fix] Multiple memory corruption fixes * [Fix] Normalize path in HTTP router * [Fix] Plug memory leak * [Fix] Plug memory leak in adding radix trees * [Fix] Plug memory leak in configuration parser * [Fix] Plug memory leak in expressions parsing during reload * [Fix] Plug memory leak in learning fuzzy storage * [Fix] Plug memory leak in lua_tcp * [Fix] Plug reload leaks * [Fix] Plug termination memory leaks * [Fix] Really increase lock lifetime * [Fix] Replies module: fix symbol weight * [Fix] Restore content type params related functions * [Fix] Set task's subject from mime subject * [Fix] Sigh, one more reload leak * [Fix] Simplify images shingles * [Fix] Some more memory issues are fixed * [Fix] Stop hardcoding of lua in C * [Fix] Stop processing of bad parts as text parts * [Fix] Strictly filter bad characters when emittin json * [Fix] Strings returned from lua are ephemeral * [Fix] Support unix sockets for lua redis * [Fix] Try to fix issues with reloading config * [Fix] Try to fix race condition in redis_pool * [Fix] Use checksum to avoid intersection between different ANNs * [Fix] Use rspamd hashes in embedded ucl * [Fix] Use sane default rewrite subject (*** SPAM *** %s) * [Fix] Various collection mode fixes * [Fix] Various fixes to mime parser * [Fix] Various reload leak fixing * [Fix] Whitelist certain extensions from archive checks * [Rework] Add preliminary implementation of the mime parser * [Rework] Adopt code for the new options * [Rework] Change logger setup interface * [Rework] Composite configuration (#1270) * [Rework] Finally remove gmime dependency from Rspamd * [Rework] Further fixes to symbols frequencies * [Rework] Implement content type parser for mime * [Rework] Kill all InternetAddressList usages * [Rework] Multiple fixes for symbols cache statistics * [Rework] Refactor struct names * [Rework] Rework images fuzzy hashes algorithm * [Rework] Rework lua_tcp to allow TCP dialog * [Rework] Start massive rework to get rid of gmime * [Rework] Start new approach for multiparts parsing * [Rework] Start rework of mime addresses * [Rework] Start rework of symbols cache updates * [Rework] Start switching to libicu * [Rework] Use a special structure for stats tokens * [Rework] Use hash tables for symbols options * [Rework] Use libicu instead of iconv for conversions * [Rework] Use new scheme to parse mime parts * [WebUI] Add Access-Control-Allow-Origin for cluster management * [WebUI] Add Throughput graph autorefreshing (#820) * [WebUI] Add Visibility.js library * [WebUI] Add basic cluster support to Throughput tab * [WebUI] Add graph legend entries for new DSes * [WebUI] Add graph tab * [WebUI] Add neighbours RRD data consolidation * [WebUI] Add preliminary save symbols clustering * [WebUI] Add server selector to navbar * [WebUI] Add soft reject to auth stats * [WebUI] Add summary to the Throughput tab * [WebUI] Allow to save maps on the cluster * [WebUI] Avoid extra graph redraw and alerts glitching * [WebUI] Be more generous with AJAX timeout * [WebUI] Disable error ring loading in `read only` mode * [WebUI] Enclose table header cells with `tr`s * [WebUI] Finish interface rework * [WebUI] Fix RRD summary pie chart position * [WebUI] Fix `All SERVERS` graph fot just one available server * [WebUI] Fix case when no cluster is defined * [WebUI] Fix compatibility with non-ES6 compliant browsers * [WebUI] Fix config ID * [WebUI] Fix configuration page partially * [WebUI] Fix disabled state * [WebUI] Fix graph dataset selector initialization * [WebUI] Fix graph selectors state resetting * [WebUI] Fix mouse events on throughput summary table area * [WebUI] Fix multiple JS issues * [WebUI] Fix pie chart displaying * [WebUI] Fix read only * [WebUI] Fix read only2 * [WebUI] Fix retarded datatables * [WebUI] Fix soft reject in pie chart * [WebUI] Fix stat widgets timers multiplication on `Refresh` click * [WebUI] Fix symbols config * [WebUI] Fix various errors with login form * [WebUI] Further fixes * [WebUI] Hide learning tab in read-only mode * [WebUI] Initial clusters support * [WebUI] Make legend entry colours more contrast * [WebUI] Move configuration tab to a separate module * [WebUI] Move history tab * [WebUI] Move symbols config as well * [WebUI] New sec to time function * [WebUI] Prevent multiple clicks on `Refresh` * [WebUI] RRD summary: Hide inner labels of tiny pie sectors * [WebUI] RRD summary: Respect undefined values * [WebUI] Reduce font size of graph's legend * [WebUI] Remove orphaned font duplicates * [WebUI] Remove unused code * [WebUI] Replace spinner with animated glyphicon * [WebUI] Reset refresh timer on server switching * [WebUI] Rework interface to use requirejs * [WebUI] Rework neighbours query function * [WebUI] Separate attributes by space * [WebUI] Set focus to password field (#1230) * [WebUI] Simplify neighbours table populating * [WebUI] Start rework of modules * [WebUI] Stop stats refreshing if the page is hidden * [WebUI] Turn d3pie's stuff into a reusable function, * [WebUI] Unify send data functions * [WebUI] Update D3Evolution to 0.0.1 * [WebUI] Update d3.js * [WebUI] Update datatables to work with the requirejs * [WebUI] Use unified tab click event handler, * [WebUI] clusters for the chart * [WebUI] fix uptime Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.4.2Vsevolod Stakhov2017-01-061-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Deal with absent headers in DKIM * [CritFix] Do not trust remote shingles count * [CritFix] Fix headers selection in DKIM verification * [Feature] Add EXT_CSS rule * [Feature] Add toggle for disabling SURBLs * [Feature] Extend redis lock when learning spawned * [Feature] Parse <link> HTML tags * [Fix] Avoid reusing g_error (Fixes #1262) * [Fix] Do not reset loaded ANN when learning is requested * [Fix] Fix another issue with external deps in SA * [Fix] Fix body trie matching * [Fix] Fix checking of DKIM bodies that needs just `\n` to be added * [Fix] Fix fuzzy hashes count * [Fix] Fix keys names * [Fix] Fix length calculations for url encoded urls * [Fix] Fix matching of the same patterns from different tries * [Fix] Fix name of var * [Fix] Fix parsing of URLs with spaces and other bad chars * [Fix] Fix probability calculations for fuzzy redis backend * [Fix] Fix signing headers creation in DKIM * [Fix] Plug memory leak * [Fix] Really fix chained SA dependencies * [Fix] Really increase lock lifetime * [Fix] Use checksum to avoid intersection between different ANNs * [Fix] Use rspamd hashes in embedded ucl * [Fix] Yet another change for testing external deps Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.4.11.4.1Vsevolod Stakhov2016-11-301-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Feature] ASN support in Clickhouse module * [Feature] Add clickhouse plugin * [Feature] Add generic tool to add universal maps for lua modules * [Feature] Add logger.debugm to debug lua modules * [Feature] Allow to register metrics symbols using register_symbol * [Feature] Allow to specify prefix for fann_redis * [Feature] Clickhouse: support different masks for IPv4/IPv6 * [Feature] Support forcing action in antivirus plugin * [Fix] Add handling of regexp maps * [Fix] Allow backslashes in http urls * [Fix] Avoid mapping of empty files * [Fix] Do not load tld file to speed up rspamadm * [Fix] Do not resolve numeric IP addresses due to ipv6 insanity * [Fix] Filter incorrect training data * [Fix] Fix Fuzzyconvert tool when password or DB is given * [Fix] Fix build with custom glib/gmime * [Fix] Fix converting of learn count from sqlite to redis * [Fix] Fix crashes with invalid received and task:set_from_ip * [Fix] Fix external dependencies for SA module * [Fix] Fix fann_redis when number of scores has been changed * [Fix] Fix hyperscan usage for non compatible platforms * [Fix] Fix loading of maps from UCL objects * [Fix] Fix memory leak for task-less redis requests * [Fix] Fix mid module with new maps syntax * [Fix] Fix parsing of URLs with username * [Fix] Fix re cache initialisation * [Fix] Fix replacements to sanitize '%' character * [Fix] Fix set and regexp like static maps * [Fix] Fix some issues in redis settings * [Fix] Fix static IP maps * [Fix] Fix total learns counter for redis stats * [Fix] Fix usage of config during reload * [Fix] Fix various warnings and issues * [Fix] Invalidate ANN if training data is incorrect * [Fix] Miltiple fixes to fann_redis module * [Fix] More fixes for URLs with backslashes * [Fix] Properly get options for ip_score module * [Fix] Relax requirements for Received as gmail cannot RFC * [Fix] Remove or fix hyperscan incompatible regexps * [Fix] Settings: correctly read redis config * [Rework] Rework lua logger interface slightly * [Rework] Use new maps add function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.4.01.4.0Vsevolod Stakhov2016-11-211-0/+291
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Add guards for inactive redis connections * [CritFix] Another fix for proxying files using rspamd_proxy * [CritFix] Cleanup inactive redis connections * [CritFix] Do not sometimes try to exec posfilters before classification * [CritFix] Fix application of IPv6 mask * [CritFix] Fix chunked encoding when reading messages * [CritFix] Fix file mode for rspamd_proxy * [CritFix] Fix hyperscan compilation on regexp change * [CritFix] Fix issue with finding of end of lines pointers * [CritFix] Fix iteration over headers array (introduced in 1.4) * [CritFix] Fix processing of learned tokens count for redis backend * [CritFix] Fix race condition in checking of cached maps * [CritFix] Fix workers scripts by sharing workers configs * [CritFix] Introduce raw content to text parts * [CritFix] Plug memory leak and potential memory corruption * [Feature] Adaptive ratelimits * [Feature] Add ASN -> rbldnsd script for asn.rspamd.com * [Feature] Add DMARC_NA symbol * [Feature] Add F-Prot support to antivirus module * [Feature] Add HTTP backend to metadata exporter * [Feature] Add Lua API module for monitored objects * [Feature] Add R_DKIM_NA / R_SPF_NA / AUTH_NA symbols * [Feature] Add R_DKIM_PERMFAIL symbol * [Feature] Add R_SPF_PERMFAIL symbol * [Feature] Add Sophos antivirus support * [Feature] Add ZSTD compression to Lua API * [Feature] Add `mid` Lua module * [Feature] Add `one_param` flag for metric symbols * [Feature] Add a generic lua classifier * [Feature] Add a very basic interface to access workers data from on_load * [Feature] Add ability to delete a hash by its data to fuzzy_check plugin * [Feature] Add ability to enable/disable symbols via dynamic_conf * [Feature] Add ability to lookup settings by key * [Feature] Add common way to disable Lua modules * [Feature] Add compression support to rspamd client * [Feature] Add condition to do antiviral check * [Feature] Add configuration for lua classifiers * [Feature] Add configuration knobs for the errors circular buffer * [Feature] Add decompression support in rspamd client * [Feature] Add errors exporter to the controller * [Feature] Add expected value for monitored DNS resources * [Feature] Add exporter from error ringbuf to ucl * [Feature] Add extended version for fann creation function * [Feature] Add ffi friendly version of process_regexp function * [Feature] Add frequency and time display to webui * [Feature] Add fuzzy_delhash command to rspamc client * [Feature] Add implementation of redis connections pool * [Feature] Add latency and offline time monitoring * [Feature] Add learning support for lua classifiers * [Feature] Add max-size and timeout options to CGP helper * [Feature] Add method to enable/disable symbols in config * [Feature] Add methods to get metric's actions and symbols from Lua * [Feature] Add mmap support to lua_text * [Feature] Add monitored object for surbl plugin * [Feature] Add more exceptions to surbl whitelist * [Feature] Add more meta-tokens to bayes * [Feature] Add neural net classifier to fann_scores module * [Feature] Add neural net serialization/deserialization * [Feature] Add new dynamic conf module * [Feature] Add periodic events support for lua_config * [Feature] Add plugin to check MX'es for the sender's domain * [Feature] Add preliminary monitored module * [Feature] Add preliminary support of dynamic conf updates in Redis * [Feature] Add preliminary version of clamav plugin * [Feature] Add redis cache to asn module * [Feature] Add replies compression * [Feature] Add spamhaus DROP dnsbl * [Feature] Add support for dictionary in client compression * [Feature] Add support for fuzzy learn and unlearn from lua * [Feature] Add support for input encryption * [Feature] Add support of min_learns to neural net classifier * [Feature] Add termination callbacks for workers * [Feature] Add user-agent for rspamc * [Feature] Add utility to perform classifier tests * [Feature] Add zstd compression library * [Feature] Allow HTTPS requests in lua_http * [Feature] Allow conditions for pre and postfilters * [Feature] Allow custom functions for ratelimits * [Feature] Allow for excluding messages from AV scanning based on size * [Feature] Allow for getting worker stats from Lua * [Feature] Allow getting task UID from Lua * [Feature] Allow parsing of mailbox messages from the commandline * [Feature] Allow plugins to publish their lua API via rspamd_plugins * [Feature] Allow to compare other systems with Rspamd * [Feature] Allow to execute Lua scripts by controller * [Feature] Allow to have a function to set custom greylist message * [Feature] Allow to iterate over multiple tags * [Feature] Allow to pass extra data from plugins to log helper * [Feature] Allow to plan new periodics at different time * [Feature] Allow to reset hashes * [Feature] Allow to run rspamadm lua just as a lua interpreter * [Feature] Allow to store settings in redis * [Feature] Allow to update dynamic conf in Redis * [Feature] Allow to use dictionaries for compression * [Feature] Allow to use md5, sha1, sha256, sha384 and sha512 hashes in Lua * [Feature] Allow whitelisting by IP for greylisting plugin * [Feature] Antivirus: Support whitelists & pattern-matching sig names * [Feature] Backport pack/unpack routines from Lua 5.3 * [Feature] Check settings with equal priopities in alphabetical order * [Feature] Compress neural net in redis * [Feature] Consider more tags when doing WHITE_ON_WHITE rule * [Feature] Descriptive options for DMARC failure symbols * [Feature] Descriptive options for RBL symbols * [Feature] Enable configuration for monitored objects * [Feature] Execute on_load scripts with ev_base ready * [Feature] Fann scores now uses metadata from a message * [Feature] Implement FANN threaded learning * [Feature] Implement classifying for lua classifiers * [Feature] Implement finish scripts for worker processes * [Feature] Implement monitoring for DNS resources * [Feature] Implement real priorities for pre and post filters * [Feature] Insert two symbols: FANN_HAM and FANN_SPAM instead of one * [Feature] Module to push metadata/messages to redis pubsub * [Feature] Monitor RBL records * [Feature] Move fann_classifier to a separate plugin * [Feature] Normalize all ANN inputs * [Feature] Preliminary version of metric exporter module * [Feature] Preserve decompression context between tasks * [Feature] Ratelimit: Support dynamic bucket size/leak rate * [Feature] Relax FORGED_RECIPIENTS: allow senders to BCC themselves * [Feature] Remove symbols weights on composites processing * [Feature] Return symbol scores when getting resulting symbols * [Feature] Rework lua tcp module * [Feature] Rule to detect some obvious X-PHP-Originating-Script forgeries * [Feature] Rule to identify some X-PHP-Script forgeries * [Feature] Rules for scoring Google Message-ID fixes * [Feature] Send hashes values to reply * [Feature] Set expire for dmarc reports * [Feature] Stop using cymru zone as it is unstable * [Feature] Stop using of GLists for headers, improve performance * [Feature] Store `for` in task:get_received_headers * [Feature] Store `for` part in received headers * [Feature] Store enabled flag for webui session * [Feature] Store error messages in ring buffer * [Feature] Support compressed maps * [Feature] Support excluding selected users from ratelimits * [Feature] Support looking up NS records in lua_dns * [Feature] Support modern style SURBL configuration * [Feature] Support multiple hashes in delhash path * [Feature] Support new messages in rspamc * [Feature] Support requests without reads in lua_tcp * [Feature] Support setting task message from Lua * [Feature] Track visibility of HTML elements * [Feature] Try to add CRLF when checking DKIM * [Feature] Try to guess line endings when folding headers * [Feature] Try to improve normalization function for bayes * [Feature] Use FFI to optimize SA module * [Feature] Use length based arguments for redis, allow lua_text as arg * [Feature] Use more layers for fann and another normalization * [Feature] User-defined ratelimits * [Feature] Utility to convert fuzzy storage from sqlite to redis * [Feature] Yield DMARC_DNSFAIL on lookup failure * [Fix] Adopt fuzzy storage for flexible backends * [Fix] Allow plain IP addresses in Rspamd maps * [Fix] Another fix for brain-damaged hiredis * [Fix] Another fix for rdns write errors * [Fix] Another fix for rdns_make_request_full invocation * [Fix] Another fix in DKIM canonicalization * [Fix] Another memory leak plugged * [Fix] Another try to deal with posix idiotizm * [Fix] Another try to fix RDNS events processing logic * [Fix] Avoid double frees in HEAD requests * [Fix] Avoid extra symbols for RBLs * [Fix] Banish table.maxn from Lua parts * [Fix] Check for socket error before connection in lua_tcp * [Fix] Correctly propagate redis timeouts to Lua * [Fix] Do not add extra newline in MIME mode * [Fix] Do not be cheated by system hiredis * [Fix] Do not classify when a message has not enough tokens * [Fix] Do not crash on redis errors * [Fix] Do not distinguish NXDOMAIN and NOREC for monitored * [Fix] Do not replan retransmits if merely one server is defined * [Fix] Do not use headers to calculate messages digests * [Fix] Don't force action in replies module for authenticated users/local networks * [Fix] Explicitly ban default passwords in webui * [Fix] Finally fix ambiguity between parsed and resolved spf elts * [Fix] Fix 'decoded' value in task:get_header_full() * [Fix] Fix DKIM calculations * [Fix] Fix DKIM signing for messages with no newline at the end * [Fix] Fix DNS request in monitored * [Fix] Fix DNS write errors processing * [Fix] Fix HTTP methods other than GET and POST * [Fix] Fix PERMFAIL for v6/v4 ambiguities * [Fix] Fix absurdic scores for HFILTER_URL_ONLY * [Fix] Fix actions in rolling history * [Fix] Fix actrie patterns * [Fix] Fix applying of lua dynamic confg * [Fix] Fix autolearning errors and redis cache * [Fix] Fix bayes learn_condition * [Fix] Fix build with the recent OpenSSL * [Fix] Fix caching and compressed maps * [Fix] Fix check plain text part * [Fix] Fix crash on OpenBSD in `url_email_start` * [Fix] Fix double free in SPF * [Fix] Fix extraction of shingles from redis fuzzy storage * [Fix] Fix false sharing for symbols in the cache * [Fix] Fix float usage in util:get_time * [Fix] Fix folding algorithm to deal with empty tokens * [Fix] Fix format string * [Fix] Fix format string usage in controller errors handling * [Fix] Fix handling of '\0' in lua_tcp * [Fix] Fix handling of HTTP HEAD methods * [Fix] Fix hash creation * [Fix] Fix hiredis stupidity * [Fix] Fix implicit settings module settingsup * [Fix] Fix interaction with lua GC to avoid craches * [Fix] Fix ip_score module registration * [Fix] Fix issue with empty messages and dkim * [Fix] Fix issues with CGP helper * [Fix] Fix issues with the recent SPF changes * [Fix] Fix key name to load ANN correctly * [Fix] Fix lua tcp module by saving `do_read` in callback data * [Fix] Fix memory leak in client when using compression * [Fix] Fix min_learns option * [Fix] Fix on_finish scripts and async handlers * [Fix] Fix options for SPF dnsfail symbol * [Fix] Fix parsing includes and redirects in SPF * [Fix] Fix parsing of lua comments with empty lines * [Fix] Fix parsing of unquoted HTML attributes * [Fix] Fix periodic events and redis * [Fix] Fix processing of fuzzy learns from Lua * [Fix] Fix processing of redirect in SPF includes * [Fix] Fix processing of symbols when reject limit is reached * [Fix] Fix refcounts when map is specified by IP * [Fix] Fix rspamd{session} class in Lua API * [Fix] Fix setting ratelimit key for 'ip' bucket * [Fix] Fix some cases of TLD urls detector * [Fix] Fix statconvert tool * [Fix] Fix stats for backend-less classifiers * [Fix] Fix training script for fann_redis * [Fix] Fix variable in ann module * [Fix] Fix various errors in lua dynamic conf plugin * [Fix] Forget old ANN when max_usages is reached to avoid overtrain * [Fix] Further canonicalization fixes * [Fix] Further fixes for fann_redis prefixes * [Fix] Handle failures for inactive pooled connections * [Fix] Improve multimap info message * [Fix] More fixes in ANN loading * [Fix] More fixes to fann_redis * [Fix] More issues in fann_redis * [Fix] More spaces fix in DKIM signature * [Fix] Multiple fixes to asn script, add IPv6 support * [Fix] Multiple issues in fann_redis * [Fix] No greylist rejected messages * [Fix] One more attempt to fix lua_redis * [Fix] One more check for readdir... * [Fix] Params should be treated as a hash * [Fix] Plug memory leak in regexp desctructor * [Fix] Process headers only once * [Fix] Properly handle nil values in ratelimit plugin * [Fix] Really fix redis shingles check * [Fix] Remove fann with incorrect layers count * [Fix] Remove mentions of deleted include * [Fix] Remove some incompatible functions * [Fix] Settings: fix `authenticated` parameter (#886) * [Fix] Skip MX check for authenticated users and local networks * [Fix] Slightly fix ANN routines * [Fix] Stop caching records with DNS failures * [Fix] Treat all errors in redis_pool as fatal errors for a connection * [Fix] Try avoid false-positives in HEADER_FORGED_MDN rule * [Fix] Try to avoid race condition when using rrd * [Fix] Try to reload redis scripts if they are missing * [Fix] Unbreak once_received skipping for local networks * [Fix] Unlock ANN on error * [Fix] Use memmove for overlapping regions * [Fix] Use real size instead of displayed for core limits * [Fix] Use the correct macro to get the size of control * [Fix] Various fixes for errors ringbuffer * [Fix] Yield R_SPF_DNSFAIL if lookup of included record fails * [Fix] mid: fix map initialization * [Fix] mid: handle incorrect rgexps in the map * [Rework] Add extract training data function to fann_redis * [Rework] Add preliminary train tests * [Rework] Add redis storage feature to fann_redis * [Rework] Adopt fuzzy storage for abstract backend * [Rework] Adopt plugins * [Rework] First reiteration on fann scores * [Rework] Implement loading/invalidating * [Rework] Make lua_redis task agnostic * [Rework] Make rspamd protocol messages useful * [Rework] Massive removal of legacy code * [Rework] More cleanup actions * [Rework] Remove legacy code never used for classifiers * [Rework] Remove outdated and unused lua_session module * [Rework] Reorganize fuzzy backend structure * [Rework] Reorganize the internal backend structure * [Rework] Restore old fann_scores, move common parts * [Rework] Rework and simplify rbl plugin * [Rework] Rework parsing of DMARC records Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.3.41.3.4Vsevolod Stakhov2016-08-231-0/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Feature] ASN module; support matching ASN/country in multimap * [Feature] Add SPF method in spf return result * [Feature] Add Yandex and Mail.ru forwarding rules * [Feature] Add mempool maps in multimap * [Feature] Add rule for identifying mail sent by eval()'d PHP code * [Feature] Add support of stub DNSSEC resolver to rdns * [Feature] Add task:get_digest method * [Feature] Allow for more fine-grained scoring for ip_score * [Feature] Allow to get digest of a mime part from lua * [Feature] Allow to print message digest in logs * [Feature] Fold DKIM-Signature header * [Feature] Implement encrypted logs * [Feature] Log URLs encrypted if we have log encryption pubkey * [Feature] Pass authenticated bit to lua * [Feature] Read redis backend statistics configuration from global section * [Feature] Show the exact value matched for multima symbols * [Feature] Store task checksum * [Fix] Avoid setting limits when required elements are missing * [Fix] DMARC: Fix alignment checking for subdomains * [Fix] DMARC: deal with missing and spurious spaces * [Fix] Defer insertion of results in ip_score to avoid skewing stats * [Fix] Disable DMARC for local/authorized mail * [Fix] Fix handling of proxied headers in controller * [Fix] Fix hex printing of strings * [Fix] Fix issue with spaces in maps * [Fix] Fix parsing of forwarded IP * [Fix] Fix reload in some plugins and workers * [Fix] Fix reloading on SIGHUP * [Fix] Fix some border cases for DKIM canonicalization * [Fix] Fix url maps * [Fix] Make dnssec configurable option disabled by default for now * [Fix] rspamadm statconvert: force db to be a string * [Fix] rspamadm statconvert: use db/password for learn cache * [Rework] Rework flags in rspamd logger Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.3.3Vsevolod Stakhov2016-08-151-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Check hyperscan cache sanity before loading * [CritFix] Fix setting of fuzzy keys (completely breaks fuzzy storage) * [Feature] Add SARBL (sarbl.org) uribl * [Feature] Add `--search-pattern` option to rspamd_stats * [Feature] Add some sanity check for very long from/to log elements * [Feature] Allow to create hashes from string in a single step * [Feature] Fix order of pre and postfilters * [Feature] Improve lua URLs API * [Feature] Improve message about fuzzy rules * [Feature] Pre-calculate blake2 digest for all parts * [Feature] Print radix duplicate keys as IP addresses * [Feature] Simple mechanism for disabling RBLs in local.d/rbl.conf * [Feature] Use faster hash function for fuzzy storage * [Feature] rspamd_stats: support log directory reading * [Fix] Add sanity check for url filters * [Fix] Do not show rmilter section as a fake metric in rspamc * [Fix] Fix URL filters * [Fix] Fix a stupid mistake in util.strequal_caseless * [Fix] Fix blake2b hash of the string "rspamd" * [Fix] Fix filename maps filter * [Fix] Fix finding tld in util.get_tld * [Fix] Fix multimap content filters * [Fix] Fix returning boolean from Lua * [Fix] Fix returning of REDIS_NIL * [Fix] Try to deal with multiple workers terminated * [Fix] Use forced DNS request when calling for lua_http * [Rework] Rework multimap filters, add redis maps Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.3.21.3.2Vsevolod Stakhov2016-08-081-0/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Feature] Add a special symbol for SPF DNS errors: R_SPF_DNSFAIL * [Feature] Add correlations report in fuzzy stats * [Feature] Add experimental CGP integration * [Feature] Add method to get urls length in a text part * [Feature] Add new methods to lua_html to access HTML tags * [Feature] Allow all types of symbols to be added via __newindex method * [Feature] Allow to create settings for authenticated users * [Feature] Allow to get block content for HTML tags * [Feature] Improve DNS failures when dealing with SPF * [Feature] Properly implement R_WHITE_ON_WHITE rule * [Feature] Remove old ugly rules * [Feature] Rspamc can now add dkim signature in mime mode * [Feature] Store content length for HTML tags * [Feature] Support reacher set of HTML colors * [Feature] Try to avoid FP for low contrast fonts detection * [Fix] Add missing HTML colors * [Fix] Add spaces to dkim signature to allow folding * [Fix] Avoid returning NaN as score on scan * [Fix] Decode entitles in href parts * [Fix] Do not cache SPF records with DNS errors * [Fix] Do not crash on cyclic depends * [Fix] Do not insert HELO/HOSTNAME unknown when they are not passed * [Fix] Do not set absent hostname to "unknown" * [Fix] Do not stress redis with KEYS command (#791) * [Fix] Fix DMARC_BAD_POLICY symbol * [Fix] Fix HFILTER_URL module * [Fix] Fix HFILTER_URL_ONELINE rule * [Fix] Fix buffering in CGP integration * [Fix] Fix colors propagation from parent nodes * [Fix] Fix confusing OpenSSL API usage of i2d_RSAPublicKey * [Fix] Fix dependencies id sanity check * [Fix] Fix folding for semicolon separated tokens * [Fix] Fix largest possible TLD behaviour * [Fix] Fix last token folding * [Fix] Fix length calculations in white on white rule * [Fix] Fix multiple request headers structure * [Fix] Fix multiple values headers freeing * [Fix] Fix parsing of background color * [Fix] Fix printing from field in log_urls * [Fix] Fix processing of last element of DMARC policies * [Fix] Further fixes for HTML colors * [Fix] Further fixes for multiple values headers * [Fix] Further fixes for white on white rule * [Fix] Further fixes in HTML tags parsing * [Fix] Ignore content type/subtype case * [Fix] Increase score of R_WHITE_ON_WHITE * [Fix] Parse CGP envelope data * [Fix] Propagate colors in HTML * [Fix] Restore multiple values headers in protocol * [Fix] Restore multiple values in headers processing * [Fix] Some more changes to tag's content length calculations * [Fix] Some more fixes for low contrast fonts detector * [Fix] SpamAssassin plugin: support check_freemail_header('EnvelopeFrom', [..]) * [Fix] Trigger HTML_SHORT_LINK_IMG on any external image * [Fix] rspamd_stats: remove deprecated defined(@array) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.3.11.3.1Vsevolod Stakhov2016-08-011-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix catena passwords validation * [CritFix] Fix crash when the first received is faked * [Feature] Add DMARC_BAD_POLICY symbol when DMARC policy was invalid * [Feature] Allow for matching hostnames in multimap (#773) * [Feature] Allow for setting action based on DMARC disposition * [Feature] Allow limiting of the inbound message size * [Feature] Allow maps with multiple symbols and scores * [Feature] Allow regexps in the emails maps * [Feature] Allow to register metric symbols from multimap * [Feature] Allow to reset redis tokens instead of appendig values * [Feature] Allow to store strings in radix maps * [Feature] Check UTF validity when there are utf regexps in a map * [Feature] Correctly work when there is no hard reject action * [Feature] Implement dependencies for maps * [Fix] Another effort to unbreak sqlite locking * [Fix] Avoid crash when closing mmapped file * [Fix] Do not break history on NaN in required score * [Fix] Ensure that hyperscan cache written is written properly * [Fix] Filter NaN from scores in history * [Fix] Fix DNSBL maps * [Fix] Fix another locking issue in sqlite * [Fix] Fix another locking issue with mapped files * [Fix] Fix deadlock in mmaped file stats * [Fix] Fix dependencies in multimap plugin * [Fix] Fix emails module configuration * [Fix] Fix greylist plugin (#755) * [Fix] Fix greylisting plugin variable usage * [Fix] Fix installed permissions for rspamd_stats * [Fix] Fix locking in mmapped statistics * [Fix] Fix paths in tests * [Fix] Fix prefilter mode for multimap * [Fix] Forgot to commit leftover changes * [Fix] Really fix local.d includes * [Fix] Restore selective greylisting behaviour * [Fix] Set max size on per connection basis * [Fix] Use temporary storage for hyperscan cache * [Rework] Remove systemd socket activation Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.3.01.3.0Vsevolod Stakhov2016-07-251-0/+539
| | | | | | https://rspamd.com/announce/2016/07/25/rspamd-1.3.0.html Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.8Vsevolod Stakhov2016-06-201-0/+12
| | | | | | | | | | | | | | | * Another fix for exim workaround (#637) * Fix unencrypted passwords processing in the controller * Fix setting path for lua (#652) * Fix usage of rdns reply structure (#654) * Use file lock in logger to avoid deadlocks * Add `application/octet-stream` mime type for `pdf` extension (by @moisseev) * Implement new automata to skip empty lines for dkim signing (#651) * Fix parsing of missing classes * Clarify some rspamc arguments (by @fatalbanana) * Correct suppress spelling Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.7Vsevolod Stakhov2016-05-261-0/+20
| | | | | | | | | | | | | | | | | | | | | | | * Slightly reduce weights of rules with high FP rate * Add workround for rspamd-1.3 * Fix possible FP in TRACKER_ID rule * Simplify MISSING_MIMEOLE rule * Add workaround for gmime CTE stupidity * Fix mime headers processing * Fix false positive URL detections in text parts * Fix Exim shutdown patch * Enable workaround for exim mailbox format * Backport shingles static test * Fix levenshtein distance calculations * Fix max_train setup in ANN module * Fix redis structure by adding {NULL, NULL} member * Fix build with unmodified LibreSSL opensslv.h * Repair optional dependencies * Really skip filters in case of pre-result set * Restore the intended pre-filters behaviour * Fix ipv6 mask application Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.6Vsevolod Stakhov2016-04-291-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix parsing of URLs in texts * Fix creating of URLs from LUA * Fix some more URL detector issues * Fix unit tests * Fix JIT compilation for PCRE2 expressions * Fix JIT usage for PCRE2 * Fix UTF8 mode in PCRE2 * Add workaround for pre-historic compilers (#605) * Fix and rescore R_PARTS_DIFFER logic * Properly set lua paths for tests * Fix SA rawbody processing - exclude top part * Store text parts content with newlines stripped * Properly support SA body regexps * Fix body rules in SA plugin * Fix setting of score for parts differ * More fixes to parts distance calculations - Use hashed words instead of full words for speed - Improve levenstein distance calculations and penalise replaces - Always return number from 0 to 1 - Use g_malloc instead of alloca * Fix percents output in R_PARTS_DIFFER * Plug memory leak in dkim module * Plug minor memory leak in regexps creation Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.5Vsevolod Stakhov2016-04-221-0/+4
| | | | | | | * Plug an important memory leak in headers getting code * Remove some bad domains from whitelists Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.41.2.4Vsevolod Stakhov2016-04-201-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Implement new multipattern matcher that uses hyperscan if possible * Use mutlipattern for lua_trie code * Add utility methods for multipattern * Use multipattern in url matcher * Add escape functions for hyperscan * Allow to optimize lua -> C transition by flattening table args * Optimize hot paths in SA plugin * Optimize rspamd_re_cache_type_from_string * Allow empty tries * Fix extraction of URLs from Subject * Allow to have different flags for different patterns in multipattern * Add common directory for hyperscan cache to config * Implement caching for hyperscan multipattern * Attach domain part to `R_SUSPICIOUS_URL` (by @moisseev) * Allow multipattern scans to be nested for the case of hyperscan * Simplify SURBL redirector search code and avoid ac_trie * Add two way substring search algorithm * Avoid acism usage to find gtube pattern * Fix processing of empty headers * Allow to disable pthread mutexes on broken platforms * Make web interface not send password in query strings (#585) by @fatalbanana * Add maximum delay to ratelimit module * Backport fix for empty files inclusion from libucl * Fix settings id setup * Add min_learns option to classifiers * Use more clever to utf8 conversion strategy * Fix disabling of virtual symbols in the settings * Rework settings to work properly in metric-less configuration * Set the default limit for classifier * Fix ttl based expiration from LRU cache * Rework DKIM module to use OpenSSL for digests * Fix mailto urls parsing with hyperscan * Do not set obscured flag for urls starting with spaces * Fix crash on redis learn * Fix ratelimit ctime setting Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.31.2.3Vsevolod Stakhov2016-04-121-0/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * New DCC module (by @smfreegard) * Rework whitelist module: - Now we check different elements for different checks - MIME from for DMARC - DKIM signature domain for DKIM - SMTP from or HELO for SPF * Fix regexps results combination (*critical*) * Fix issue with expressions processing (*critical*) * Optimize strlcpy for aligned input * Add support of half-closed connection in lua_tcp * Allow to print compact json in client * Save required score in history (#581) * Allow to attach file descriptors to control commands * Allow to send descriptors from workers to main * Allow to attach fd when broadcasting to workers * Implement log pipe feature for rspamd logs analysis * Add `log_helper` worker * Add `URIBL_SBL_CSS` (by @smfreegard) * Add worker scripts functionality * Add on load hooks for rspamd_config * Add lua scripts for log_helper worker * Add generic maillist detector (#584) * Implement FANN autolearn using log_helper worker * Rework metrics configuration to allow includes * Change default value of forced removal in composite rules * Allow to use assembly version of blake2b on x86 cpu * Use less precise (but faster) clock if possible * Insert redirected URL to the urls list * Allow to get and set callback data for rspamd symbols * Add binary heap implementation * Use binary heap for expire algorithms in the hash * Use `least frequent used` expiration strategy * Allow to get mime headers from a task * Add support for mime headers in `regexp` module * Update Exim patches (by @fatalbanana) * Allow building rspamd with jemalloc * Save multipart boundaries * SA plugin changes: - Properly handle MIME headers - Fix eval:check_for_missing_to_header rule - Implement SA compatible body regexps - Use sabody rules in SA plugin * LUA API changes: - Add util.get_ticks function - Add util.stat function - Add task:get_symbols_numeric method - Add method to get number of symbols in the cache - Add lua methods to get redirected urls - Allow to get callbacks for lua symbols - Add config:set_symbol_callback function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.21.2.2Vsevolod Stakhov2016-04-011-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | * Use HTTP Content-Type on non mime input if possible * Save log level when compressing log messages * Further rework of composite rules (add '^' prefix) * Add tracking for rspamd expressions * Store actions limits in metric result * Fix parsing of include/redirect with many records in SPF * Add method to disable symbols execution in the cache * Allow to disable checks from settings * Allow to select settings by id in HTTP query * Find URLs with '\r' and '\n' inside href attribute * Implement vectored mode for hyperscan (experimental) * Improve client connection errors diagnostics * Allow to edit new files with signtool * Improve hashes performance on 32 bit platforms * Fix sorting of limits * Remove slow and unused rules `INVALID_EXIM_RECEIVED*` * Add expression:process_traced lua method * Allow tables in task:insert_result * Save trace for SA metas * Do not parse broken TLD parts in URLs * Investigate many border cases in URLs parser Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.11.2.1Vsevolod Stakhov2016-03-251-0/+12
| | | | | | | | | | | | | | | * Add list support to `mime types` module configuration (by @moisseev) * Allow symbols params to be printed in logs * Fix `MIME_BAD_ATTACHMENT` false positives for MDN/DSN * Fix crashes on arm32 * Do not classify message if some class is missing * Fix cryptobox cleanup * Remove multipart/report from bad mime types (#569) * Improve logging for fuzzy hashes * Show map URLs in webui * Sort symbols in webui Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.2.01.2.0Vsevolod Stakhov2016-03-211-3/+83
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * New dynamic updates plugin * Regular expressions map support * Faster radix trie algorithm * Faster siphash for AVX2 supporing CPUs (used in fuzzy hashes) * PCRE2 support * Allow quoted and slashed keys in map * Add proper support of DNS resolvers balancing (#552) * Rework includes and configuration system for better local changes support * New keypairs framework for signing and encryption * Added support for dynamic modules and workers * Allow to dump configuration with help comments * Rework once_received module - Fix priority for `good_hosts` - If a good host has been found do not add once_received symbols - Fix priorities for strict once_received - Add ability to whitelist IP addresses * Implement support of signed maps for HTTP and file maps * Add command to sync fuzzy storage (#533) * Rework system of symbols and actions registration It is possible now to use priorities when adding symbols to metrics and override scores for symbols with lower priority with the scores with high priority. * Add auth support and db selection for redis stats * Improve composite rules application * Add ignore_received option * Fix critical issue with inconsistent resorting * Fix `all` in spf redirects * Add punycoded versions for IDN domains (#554) * Improve sorting order for symbols cache * Add lockless logging for processes management * Allow to specify flags for metric symbols * Load images height and width from style attribute (#538) * Override DNS requests limits for SPF and DKIM * Fix resetting symbols to their default values in WebUI * Improve configuration agility for redis stats * Allow to set db and password for redis in stat_convert * Import the latest libucl * LUA API changes: - Add rspamd_version function to LUA API - Add lua_cryptobox module - Add lua_map module - Add task:set_metric_action lua API method - Fix race condition in lua_tcp module - Fix a lot of issues in lua_redis module - Rework and abstract lua maps API - Add util.strlen_utf8 lua function - Add lua functions for caseless comparison - Allow optional symbols registration - Add config:add_map table form method, add regexp maps - Add task:has_urls method - Add task:has_flag method - Add html tags methods to lua_html - Add task:get_dns_req * Plugins changes: - Add support for WLBLEval SA plugin - Use caseless comparison in SA and DMARC plugins - Allow SA plugin to set scores for rspamd symbols - Add regexp maps support to multimap - Allow filenames match in multimap - Add more filters for the existing map types - Fix html images rules to reduce FP rates * New rules: - LONG_SUBJ - too long subject - MIME_BAD_ATTACHMENT - bad attachement type - RDNS_NONE - no reverse DNS record for sender's IP - Fix MISSING_MIMEOLE rule for modern OE * Many other bugfixes, memory leaks plugs thanks to: - Coverity scan - New gcc-6 warnings - valgrind manual iterations * Documentation improvements: - FAQ list: https://rspamd.com/doc/faq.html - Reworked quick start guide - Added documentation for all active modules * Other changes: - Dropped Ubuntu Vivid support - Added Ubuntu Xenial support - Rework build system for rspamd and rmilter Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.1.4Vsevolod Stakhov2016-02-221-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Print traceback on lua errors in lua config * Fix leaks in lua error paths * Improve 'R_EMPTY_IMAGE' rule * Fix metas memoization in SA plugin * Properly set `flag` in fuzzy replies * Fix arguments order * Fix issue with out-of-boundary reading * Fix issues found by coverity * Same result checking error found by coverity * Fix varargs processing (found by coverity) * Fix error in printing hex * Reduce weights for some hfilter patterns * Add aliases for task:get_from_ip: - task:get_addr - task:get_from_addr - task:get_ip * Rework once_received module - Fix priority for `good_hosts` - If a good host has been found do not add once_received symbols - Fix priorities for strict once_received - Add ability to whitelist IP addresses * Fix `MISSING_MIMEOLE` rule for modern OE * Treat meta tags as embedded tags (#501) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.1.31.1.3Vsevolod Stakhov2016-02-041-0/+16
| | | | | | | | | | | | | | | | | | | * Fix DSN rules when SMTP from is unavailable * Fix statconvert routine to avoid lua module usage * Set a sane quark for configtest to avoid NULL to be printed in logs * Support c11 if available * Fix parsing of ip:port strings * Add more diagnostic for lua subr errors * Fix task:set_from_ip lua method * Add basic routines for digital signatures * Add tool for digital signatures * Add plain open file API method for atomic open * Fix parsing nested braces inside logger vars * Pre filters now actually skip processing * Add pre-filter mode for multimap * Switch to apache 2 license Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.1.21.1.2Vsevolod Stakhov2016-01-291-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix stat_cache closing * Add checkpoints to sqlite3 learn cache * Do not recompile lua generated headers all the time * Increase number of messages learned * Fix issues with dual stack and hfilter * Disable MID checks for hfilter by default * Fix cache definitions in multiple classifier and no type * Don't crash if learn cache failed to initialize * Fix googlegroups support in maillist plugin * Rework flags LUA API: - Allow to check for a specific flag - Add `learn_spam`, `learn_ham` and `broken_headers` flags - Unify internal functions * Add `BROKEN_HEADERS` rule * Add support for forged confirmation headers (by @AdUser) * Allow `any`, `mime` and `smtp` for get_from/get_recipients * Add mime types checking plugin * Add rule to detect spammers attempts to cheat mime parsing * Rework parsing of IP addresses in configuration (better IPv6 support) * Add `util.parse_mail_address` function to LUA API * Add lua sqlite3 module * Implement synchronous redis call * Ratelimit: avoid possible indexing of nil value (Fixes #498) (by @fatalbanana) * Add stat_convert command to convert stats tokens from sqlite3 to redis * Implement redis advanced lua api with pipelining * Fix memory leak on redis stat (#500) * Fix user/language learn count in sqlite statistics (#496) (by @fatalbanana) * Fix build with custom pcre * Fix fuzzy relearning (#498) * Improve planning of asynchronous tasks * Show slow rules in log * Add warning for slow regexps * Add base32 decode/encode routines to lua util * Allow converting of learn cache from sqlite to redis * Add methods to check if a messages has from/rcpts * Improve and fix multimap plugin: - Restore 'header' maps - Add filters for headers - Add 'email:addr', 'email:user', 'email:domain' and 'email:name' filters - Add generic regexp filters * Disable reload command in rc scripts * Improve runtime CPU dispatcher for libcryptobox * Add preliminary support of digital signatures via ed25519 * Add detection for RDRAND support * Print configuration of crypto on start * A in SPF presumes AAAA lookup as well Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.1.11.1.1Vsevolod Stakhov2016-01-211-0/+15
| | | | | | | | | | | | | | | | | | * Fix duplicated XBL symbol * Reduce log severity for ratelimit missing servers * Fix XBL composite to avoid duplicate symbols * Reduce weight of URL_ONLY rule due to FP rate * Disable fuzzy hashes from the metadata for now * Fix processing of empty messages (#486) * Always treat DNS timeouts as temporary fail for SPF * Fix issue with SPF double IP stack (#483) * Use X-Forwarded-For when checking secure_ip (#488) * Fix hash calculation for sqlite stats * Fix memory corruption on punycode * Fix strings allocation in punycode * Fix error message (#491) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-25 17:32:29 +0000