aboutsummaryrefslogtreecommitdiffstats
path: root/ChangeLog
Commit message (Collapse)AuthorAgeFilesLines
* [Project] Keep Changelog updatedChris Funderburg2024-10-241-0/+14
|
* Release 3.10.03.10.0Vsevolod Stakhov2024-09-301-0/+63
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add SenderScore RPBL return codes * [Conf] Add SenderScore Reputationlist RBL * [Conf] Increase scores for strange things in the archives * [CritFix] The max size for signing key is actually 65 bytes for p256 * [Feature] Add rspamadm secretbox command * [Feature] Add rspamd_cxx_unit_cryptobox for unit testing * [Feature] Add support for OpenSSL 3.0 * [Feature] Add support for OpenSSL 3.0 for DKIM * [Feature] Add tests for rspamd_cryptobox * [Feature] Add tooling to encrypt strings in Lua * [Feature] Allow differen modes for fuzzy rules * [Feature] Allow to set negative group score limit via `min_score` * [Feature] DMARC: Implement reporting.only_domains setting * [Fix] Add workaround for the bug in OpenSSL < 3.0.8 * [Fix] Another fix for Redis schema * [Fix] Another safe-guards for learning/classifying an empty message * [Fix] Apply the same workaround for signing keys * [Fix] Check message before trying to dereference pointer * [Fix] Do not install doctest stuff * [Fix] Encode headers that we send to milter add header * [Fix] Fix DCC `rep` handling * [Fix] Fix Redis scripts uploading when Redis is not ready * [Fix] Fix and rework various parts * [Fix] Fix compatibility with OSSL 1.0 * [Fix] Fix glib internals by setting locale in `rspamc` * [Fix] GPT: Fix bug in condition check * [Fix] Get rid of EVP_PKEY_CTX_set1_rsa_keygen_pubexp * [Fix] Iterate over dynamic keys in fuzzy storage * [Fix] Make tostring in UCL a bit less brain-damaged * [Fix] More bogus sizes fix * [Fix] Preserve the previous behaviour of RDNS_* checks * [Fix] Rework DMARC to correctly handle spaces in DMARC records Issue: #4906 * [Fix] Sign key != encryption key, omg * [Fix] Unify lua symbols registration * [Fix] Use proper keys when doing asymmetric encryption * [Project] Add API method to push unwrapped UCL object * [Project] Add more stuff to transparent UCL * [Project] Add parsing of key limits and expire date * [Project] Add ratelimit parsing for fuzzy keys * [Project] Allow manipulations with opaque UCL objects * [Project] Allow to change log tag from HTTP request * [Project] Implement expiration * [Project] Implement per-key ratelimit * [Project] Move ratelimit parsing stuff to a common library * [Project] Remove NIST (OpenSSL) mode from cryptobox * [Project] Remove NIST mode from everywhere * [Project] Rework ratelimits check * [Project] Some more fixes * [Project] Start support of MIME UTF8 * [Project] Try to allow more transparent access of ucl elements * [Rework] Allow more flexible keypair encoding * [Rework] Breaking: Rewrite cfg transform and remove legacy * [Rework] Change fuzzy error symbols * [Rework] Change the logic of skipping symbols * [Rework] Clean up legacy code * [Rework] Implement new replies logic on the server's side * [Rework] Remove control block support * [Rework] Resolve rdns in a separate function * [Rework] Use __builtin_cpu_supports where possible * [Rework] Use a more straight structure for DKIM keys * [Rules] Fix some old rules
* Release 3.9.13.9.1Vsevolod Stakhov2024-07-231-0/+10
| | | | | | | | | | | * [Conf] Spf: Add R_SPF_PLUSALL symbol with some score * [Feature] Spf: Treat SPF +all in a special way * [Minor] Ensure some safety when checking weights * [Minor] Fix several issues with flag propagation * [Minor] Gpt: Improve prompt and add some conversion heursitics * [Minor] Gpt: Remove top_p reduce temperature to 0 * [Minor] Gpt: Set response_format * [Minor] Gpt: Use gpt-4o-mini by default
* Release 3.9.03.9.0Vsevolod Stakhov2024-07-121-0/+63
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Protect regexp matcher from regexps with empty patterns * [Feature] Allow adding X-CMAE-Score header * [Feature] Allow custom milter quarantine and tempfail messages * [Feature] Allow to specify minimum weight in GPT plugin * [Feature] Cloudmark: Add scores_symbols setting * [Feature] Further optimization to the hot path * [Feature] Make min_resolve_interval configurable * [Feature] Report slow synchronous rules * [Feature] Rspamc: add `files-list` option * [Feature] Support metrics command in normal/proxy workers * [Feature] Support reply in message pack format * [Feature] Use msgpack in the client * [Feature] Verify sanity of l= tag in DKIM * [Feature] milter_headers: support not removing headers * [Fix] Allow autolearn for the controller worker * [Fix] Allow spaces in DMARC records * [Fix] Allow to set 0 as number of rows to disable roll history * [Fix] Always set the unlearn flag when relearning * [Fix] Apply detection phase if fasttext could not detect language * [Fix] Backport fixes from libucl * [Fix] Backport multiple fixes from libucl * [Fix] Change expiration logic for redis_cache * [Fix] Check nconns when firing the final termination event * [Fix] Do not apply tableshape for known senders * [Fix] Do not crash if symbol is missing in the metric * [Fix] Do not read out-of-boundary when doing base64 encoding * [Fix] Do not save multipatterns to FS in certain cases * [Fix] Fix another corner case that allows candidates to be freed without init * [Fix] Fix documentation nesting * [Fix] Fix dynamic_symbols in the multimap plugin * [Fix] Fix inconsistent nesting in mime parts * [Fix] Fix parsing of maps definitions * [Fix] Fix shared memory proxying when compression is set * [Fix] Fuzzy add/delete handlers are badly broken for compression * [Fix] Honor dynamic thresholds for greylisting module * [Fix] If we have one statfile disabled we need to disable all * [Fix] Increase/decrease hash_key value for tokens depending on is_unlearn * [Fix] Libucl: Fix messagepack autodetection * [Fix] Resgore the collaterally removed line * [Fix] Unset autolearn flag if we learn explicitly * [Fix] Update `nconns` in proxy * [Fix] Update greylisting threshold as it can be easily dynamic * [Fix] Update is_spam in learned_ids when relearning * [Fix] Use explicit timeouts in the metadata exporter * [Fix] metric_exporter: avoid sending `nil` in metric values * [Project] Add GPT plugin * [Project] Add a tool to perform statistical analysis of classifiers * [Project] Finally fix slow timer behaviour * [Project] Implement fuzzy check retransmits backpressure * [Project] Improve stats processing * [Project] Reduce default window size of OSB tokenizer to 2 * [Project] Rework grow factor * [Project] Start optimization of userdata hashing * [Project] Use libarchive for troublesome archives * [Rework] Allow metrics endpoint to be enabled for each scanner * [Rework] Allow multiple conditions for dkim signing when using vault/redis * [Rework] Introduce dynamic_rate_limit for ratelimits * [Rework] Serialize control commands * [Rework] Update fmt to version 11 * [Rework] Use direct hashing instead of string hashing * [Rules] Added rules for detecting likely malware
* [Minor] Update version to 3.9.0Vsevolod Stakhov2024-02-201-0/+15
|
* Release 3.8.13.8.1Vsevolod Stakhov2024-01-251-0/+8
| | | | | | | | | * [Fix] Fix headers insertion in the ordered list * [Fix] Fix learn error propagation * [Fix] Fix learning with long prefixes * [Fix] Fix potential double free by expclicitly zeroing pointers * [Fix] Fix storing of the bayes tokens * [Fix] Use short comments rather than long one
* Release 3.8.03.8.0Vsevolod Stakhov2024-01-191-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Project] Rspamadm fuzzyping command * [Project] Use Redis scripts for Bayes statistics and cache * [Project] Support JSON logging * [CritFix] - Fix reported length of logging structure * [Feature] Allow to set `max_users` for Redis bayes backend * [Feature] Escape JSON when needed * [Feature] Proxy: Allow `encrypted_only` option * [Feature] Reiterate on gtube patterns * [Feature] Support ping command in fuzzy storage * [Feature] Support suppressing DMARC reporting for particular recipients * [Feature] rbl: support disabling or replacing url_whitelist per RBL * [Fix] Another try to fix setproctitle * [Fix] Cleanup session on exit * [Fix] Do not cleanup hyperscan files unless new ones are loaded * [Fix] Fix bad access when printing pending events on failure path * [Fix] Fix issues with the raw header on header insertion/modification * [Fix] Fix overflow in fuzzy_stats command * [Fix] Fix some corner cases when parsing maps with no newlines * [Fix] Fix various issues with canonicalisation of the paths * [Fix] Fix wrong raw_len usage * [Fix] Make words selection random deterministic upon content * [Fix] Properly set config field when creating tasks from Lua * [Fix] Really fix the language detector statistical heuristic * [Fix] Set loaded variable explicitly * [Fix] dkim_signing: siging_table: lowercase before lookup * [Fix] known_senders: fix config handling * [Fix] rbl: fix `exclude_local`
* Release 3.7.4Vsevolod Stakhov2023-11-141-0/+22
| | | | | | | | | | | | | | | | | | | | | | | * [Enhancement] Add composite rule for suspicious URLs in suspicious messages * [Fix] Another try to fix setproctitle * [Fix] Do not cleanup hyperscan files unless new ones are loaded * [Fix] Fix various issues with canonicalisation of the paths * [Fix] Properly set config field when creating tasks from Lua * [Fix] Set loaded variable explicitly * [Fix] known_senders: fix config handling * [Fix] rbl: fix `exclude_local` * [Minor] Add missing <algorithm> include for gcc 14 * [Minor] Add rule for messages missing both X-Mailer and User-Agent header * [Minor] Bundle fasttext as linux distributives are just useless * [Minor] Enable fasttext on RPM based linux * [Minor] Forgot it in one more place * [Minor] Improve FREEMAIL_AFF capture rates * [Minor] Limit fasttext to amd64 only * [Minor] Properly search for fasttext include * [Minor] Treat *.zpaq attachments as archives and harmful * [Minor] Update to 3.7.4 * [Minor] force_actions: set a group for symbols * [WebUI] Update map editor
* Release 3.7.33.7.3Vsevolod Stakhov2023-10-271-0/+3
| | | | * [Fix] Emergency fix for the hyperscan path error
* Release 3.7.23.7.2Vsevolod Stakhov2023-10-261-0/+9
| | | | | | | | | | * [Feature] rbl: support checking returncodes by CIDR * [Feature] rbl: support checking returncodes by regex * [Feature] rbl: support globbed return codes * [Fix] DMARC reporting: fix reporting for subdomains * [Fix] Deal with fmtlib exceptions properly * [Fix] backport fix for dlfcn.h from backward-cpp * [Rules] Blank spam detection
* Release 3.7.13.7.1Vsevolod Stakhov2023-10-071-0/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix leak in `gzip` function * [Feature] Add ICAP Content-Type and Filename * [Feature] Add `logging`->`task_max_elts` option * [Feature] Add utility to split string like stuff for C++ code * [Feature] Allow to set HTTP auth parameters for the maps * [Feature] Check for plugin configuration errors on `configtest` * [Feature] `known_senders` plugin * [Feature] Use backward-cpp instead of manual libunwind stuff * [Feature] rbl: support checking numeric URLs in isolation * [Fix] CMakeLists.txt remove whitespace added by linter as it makes tests fail * [Fix] Change Date: header location to conform with RFC * [Fix] Correct format pattern for RE tree tempfile name * [Fix] Correct format string for unw_word_t * [Fix] Do not accept invalid ucl object types * [Fix] Do not pollute public headers with libev internals * [Fix] Do not set output type if list application failed * [Fix] Fix `url:set_redirected` method * [Fix] Fix format string and some length issues * [Fix] Fix grammar definition for content-disposition attributes * [Fix] Fix lua schema enrichment logic for Redis params * [Fix] Fix lua stack corruption when logging large tables * [Fix] Fix merge table utility * [Fix] Fix output of non-RSA DKIM keys * [Fix] Fix some corner cases of single-host urls parsing * [Fix] Fix various issues in the `url_redirector` plugin * [Fix] MISSING_MIMEOLE: avoid matching messages from Android GMail app (#4561) * [Fix] Prevent DNSWL sabotage * [Fix] Try to fix unzip function * [Fix] rbl: really fix dependency registration when symbols_prefixes is used * [Fix] rspamadm mime: arguments beginning with letter `t` * [Rework] Breaking: return back to semver * [Rework] Move rcl logic to C++
* Release 3.63.6Vsevolod Stakhov2023-08-031-0/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add `one_shot` to some specific multimap rules * [Conf] Add language detection configuration * [Conf] Add missing attributes for the language detection configuration * [Conf] Remove outdated composite rules * [Feature] Add `sentinel_password` option * [Feature] Add ability to deny specific fuzzy flags by default * [Feature] Add controller endpoint to get fuzzy hashes from messages * [Feature] Add extra symbol when URL redirector reaches nested limit * [Feature] Add function to transliterate utf8 to ascii with some normalisation * [Feature] Add html parsing limit * [Feature] Add order to urls structure * [Feature] Add some missing functions to `lua_rsa` library * [Feature] Allow fuzzy workers to exchange blocked information * [Feature] Allow to have weak flags in fuzzy storage * [Feature] Allow to read options from maps in the multimap plugin * [Feature] Allow to use other methods when fasttext detection is enabled * [Feature] Count stats per key per flag * [Feature] Finish all features of dkim_keygen in Lua * [Feature] Khash: Allow static initialisation * [Feature] Maps: Add on_load support * [Feature] Preliminary implementation of dynamic composites * [Feature] Process HTML parts before text ones * [Feature] Reorganise struct rspamd_url to be 64 bytes size * [Feature] Save fuzzy ratelimit buckets * [Feature] Use in-place deflation for strings * [Feature] external_relay: add ip_map strategy * [Fix] Avoid race between config new/free by using a counter * [Fix] Do not use `rspamadm.dkim_keygen` * [Fix] Feed fasttext language model with the pre-tokenized words * [Fix] Fix `rspamd_has_only_html_part` * [Fix] Fix an old issue with order of destruction race between redis pool and lua * [Fix] Fix format string usage * [Fix] Fix parsing due to old bug revealed * [Fix] Fix parsing of the mask values that are invalid * [Fix] Ignore non-unique stop words * [Fix] Include the last character when parsing the last header with no value * [Fix] More fixes to fuzzystat * [Fix] Set proper counter * [Fix] Try harder to clean pending bucket * [Fix] Try harder to remove bad hyperscan files * [Fix] Update stats before encryption... * [Fix] dmarc gramar - allow spaces before ";" * [Fix] rbl: fix dependency registration when symbols_prefixes is used * [Fix] remove obsolete rspamd-redirector files * [Fix] test external_relay: count should always be the last rule, as it have no matching condition * [Project] Allow to register multimap symbols dynamically * [Project] Implement fasttext language detection * [Rework] Default max shots must not influence options * [Rework] Write dkim keygen tool in lua * [Rules] Add thread hijacking composite rule
* Release 3.53.5Vsevolod Stakhov2023-03-171-0/+47
|
* Release 3.43.4Vsevolod Stakhov2022-11-011-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Restore compatibility with the integrations and headers alterations * [Feature] Milter_headers: Add `x-rspamd-action` routine * [Feature] Share hyperscan database among processes * [Fix] Another corner case in url parsing * [Fix] Another fix for the enable password * [Fix] Another try to fix close method in lua_tcp * [Fix] Fix additional fields in the Redis schema * [Fix] Fix emoji joiner FP * [Fix] Fix favicon.ico Content-Type header * [Fix] Fix hang when close is used * [Fix] Lua_tcp: Sigh, another try to fix `close` invocation * [Fix] Mx_check: Cache the fact of a missing MX record * [Fix] Try to fix parsing of the unencoded `>` characters in html attributes * [Fix] Try to fix the case where password == enable_password * [Project] (Re)implement hyperscan caching * [Project] Rework cleanup * [Project] Synchronize hyperscan caches via the main process * [Rework] Convert multipattern to use hyperscan tools * [Rework] Make http normalize path function a generic function * [Rework] Split locked and unlocked files, as mmap does not need flock normally * [Rework] Start movement of the hyperscan related routines into a single unit * [Rework] Store the current worker, so other libraries could use this information * [Rework] Use blocking socket for IPC between main and workers * [Rework] Use more predictable size for commands buffers * [Rules] Do not insert ONCE_RECEIVED_STRICT on RDNS missing * [Rules] Reduce score of HTTP_TO_HTTPS - subject to remove completely
* Release 3.33.3Vsevolod Stakhov2022-10-011-0/+123
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add missing groups for whitelist module symbols * [CritFix] Neural: Fix keys regression after #3968 * [Feature] Accept upstream in lua_tcp * [Feature] Add ability to statically maintain disabled/enabled patterns * [Feature] Add function to store upstreams for HTTP urls * [Feature] Allow augmentations set in Lua API * [Feature] Allow lua_http module to accept upstreams * [Feature] Allow to limit write access to fuzzy storage by key * [Feature] Allow to sort symbols output * [Feature] Check content for binary stuff before dumping it to Lua * [Feature] Implement symbols augmentations * [Fix] Add missing flags * [Fix] Add more sanity checks for rua in dmarc_report * [Fix] Adjust length of the fuzzy checks for short text parts * [Fix] Another try to fix add headers compatibility logic * [Fix] Another try to fix race condition in the runtime destruction * [Fix] Avoid cyclic references in symcache and fix memory leaks * [Fix] Avoid overriding IP with Sender IP * [Fix] BAD_REP_POLICIES did not trigger when message was classified as spam by Bayes * [Fix] Bind AF_UNIX DGRAM client connection to annonymous address * [Fix] Disable IPv6 lookups for Blocklist.de RBL * [Fix] Distinguish dynamic and static items * [Fix] Dkim: Ignore unknown DKIM kv pairs as stated in RFC * [Fix] Dmarc report: Use local timezone instead of GMT * [Fix] Do not exclude authenticated users from URIBL lookups * [Fix] Empty envelopes should not be emitted as arrays (json+messagepack) when populated envelopes are objects. This greatly complicates decoding in strictly typed languages. * [Fix] External_relay: Restore the originating hostname check * [Fix] Fix DKIM keys with spaces still allowing errors on invalid base64 * [Fix] Fix copying of sockaddr_un addresses * [Fix] Fix crash with cname replies * [Fix] Fix dependencies propagation * [Fix] Fix iteration over milter headers * [Fix] Fix ordering when sorting symcache * [Fix] Fix reading of the cached maps * [Fix] Fix several issues with the HTTP keepalive parsing * [Fix] Fix stack smashing * [Fix] Fix synchronous auth/select in lua_redis * [Fix] Fix various symcache issues * [Fix] Ignore all (I hope) unknown DKIM signature KV pairs * [Fix] Ignore directories in RarV5 archives * [Fix] Libucl: avoid memory leak on objects merging * [Fix] Lua_tcp: Another try to fix closing logic * [Fix] Mempool: Fix alloc_array function to actually multiply nmembers by size * [Fix] Only check allowed fuzzy worker update ips for non-unix sockets * [Fix] Plug memory leak in regexp destruction with pcre2 * [Fix] Properly check the original email flag * [Fix] Properly deal with `get_symbol/get_metric_symbol` ambiguity * [Fix] Properly parse expressions atoms * [Fix] Properly set `Host` in rspamd_proxy * [Fix] Rbl: Fix received positioned checks * [Fix] Remove check for a score with no symbol being registered * [Fix] Same fix for lua_tcp * [Fix] Skip cname records when processing SPF records * [Fix] Skip sending dmarc reports in no-opt mode fixes https://github.com/rspamd/rspamd/issues/4241 * [Fix] Stop slow timer on task destruction * [Fix] Symcache: Do not use C style comparators in C++ sorts * [Fix] Try to avoid a corner case for `@` pattern * [Fix] Try to fix dkim reputation adjustements * [Fix] Try to fix passthrough results processing logic * [Fix] Try to fix the mess with read only flag * [Fix] Upstreams: Don't ignore revive_time config option * [Fix] Use proper format string, sigh... * [Fix] Use space category in ragel automata to resolve space characters * [Fix] Zstd: Fix compression with the new Zstd API * [Fix] milter_headers: Header fields may be inserted at wrong position. * [Project] Add experimental HTTP statistics backend * [Project] Add more methods for symbols addition * [Project] Add raii_sink file helper * [Project] Add some more methods * [Project] Add symbols processing methods * [Project] Allow `=` separated augmentations to be treated as kv pairs * [Project] Allow to extract augmentation values * [Project] Few more methods * [Project] Fix on conditions * [Project] Further efforts to make a more consistent architecture * [Project] Further rework * [Project] Further rework tracking * [Project] Further split of the code * [Project] Get rid of C style ctors/dtors * [Project] Http_stat: Notice statfiles when creating runtime * [Project] Implement dynamic items lookup and processing * [Project] Implement item finalization * [Project] Implement more methods * [Project] Implement runtime creation * [Project] Implement settings processing + some neats * [Project] Implement some conditions checks * [Project] Implement validation logic * [Project] More methods * [Project] Move runtime cache part to a separate unit * [Project] Move some more methods * [Project] Re-implement counters method * [Project] Reimplement dependencies processing * [Project] Remove obsoleted methods * [Project] Remove old code (finally) * [Project] Rework symbols execution * [Project] Some more adjustments in symbols registration * [Project] Start rewrite symcache in c++ * [Project] Support augmentations with values * [Project] Symcache: Use ordered filters to avoid extra lookups * [Rework] Another movement * [Rework] Augmentations can now imply flags * [Rework] Further steps * [Rework] Further work on deps processing * [Rework] Implement cache resorting * [Rework] Isolate disable/enable logic for the configuration ucl objects * [Rework] Move item implementation to a separate header * [Rework] Multimap: Avoid prefilters usage where augmentations can be used * [Rework] Pass upstream when sending TCP requests * [Rework] Re-implement cache sorting * [Rework] Reimplement saving/loading the cache items * [Rework] Reiterate on priorities * [Rework] Rework files structure * [Rework] Rewrite rspamc in C++ * [Rework] Simplify scores check and extend it to pre/post filters * [Rework] Switch minimum C++ standard version to C++20 * [Rework] Try to fix the mess with types & flags * [Rework] Use another version of hash table from the same author * [Rework] Use dynamic items for calling callbacks * [Rework] Use dynamic items in the callbacks * [Rework] Use hash map for id->symbol mappings * [Rework] Use khash instead of uthash in rdns compression logic
* Release 3.23.2Vsevolod Stakhov2022-03-261-0/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Score MIME_OBFUSCATED_ARCHIVE to 8 points * [Conf] Set one_shot for URIBL rules by default * [CritFix] Fix upstreams name resolution when there is also a port * [Feature] Add ROC feature to neural network plugin * [Feature] Add public suffic compilation utility * [Feature] Add support of Cloudmark * [Feature] Allow hyperscan for ppc64, as vectorscan now suports it. * [Feature] Allow to skip DNS resolution for keep-alive connections * [Feature] Aws_s3: Allow to store large parts separately * [Feature] BIMI: Add preliminary version of the BIMI plugin * [Feature] JSON endpoint for querying maps * [Feature] Lua_magic: Add a sane CSV heuristic * [Feature] Lua_mime: Add schema for message transfer * [Feature] Output average scan time in /stat endpoint * [Feature] Show average scan time in `rspamc stat` output * [Fix] Add guards to avoid race condition on TCP connection * [Fix] Allow spaces in DKIM key records * [Fix] Apply the similar fix to the url_reputation * [Fix] Avoid overwriting whitelisted_signers_map * [Fix] Backport PR from libucl * [Fix] Clear SSL errors * [Fix] ClickHouse cleanup of old partitions * [Fix] Do not double call error handler on ssl errors in the timeout path * [Fix] Do not forget to clear pointers on IOC reset * [Fix] External_relay: Remove useless check of the map value * [Fix] Find suspicious url encodings that could break url extraction * [Fix] Fix HTTP(s) client timeout * [Fix] Fix exclude flags setting * [Fix] Fix expanding of the variables * [Fix] Fix host header usage in lua_http * [Fix] Fix http maps shared memory cache cleanup * [Fix] Fix logic in HTML processing FSM * [Fix] Fix parsing of the compound mailto urls * [Fix] Fix processing captures from pcre2 * [Fix] Fix removing from khash * [Fix] Fix stuctured headers pushing * [Fix] Further fix for i386 compilation * [Fix] Improve duplicate settings error reporting * [Fix] Lua: task:remove_result didn't work in some cases * [Fix] Output service parts as well * [Fix] Phishing: Deal with phishing + redirected URL * [Fix] Phishing: Fix finding domains in the phishing map * [Fix] Plug memory leak by using mempool for a copied address * [Fix] Properly find the request and the number of requested entries * [Fix] Rbl: Fix inversed logic of the url_full_hostname * [Fix] Read file maps if they were not pre-read during preload * [Fix] Restrict x86_64 assembly to x86_64 * [Fix] Return a real number of recipients when dealing with aliases * [Fix] Rework unshedule DNS request function * [Fix] Support definition of ungrouped symbol in conf file, use group info from lua or other conf file * [Fix] Unschedule DNS request when clearing IO channel * [Fix] When checking for phishing, we need to convert punicode -> UTF8, not vice versa * [Fix] lua_cfg_transform - actions without score (discard) * [Fix] lua_cfg_transform - silly break break actions * [Fix] ratelimit - symbol per bucket * [Project] BIMI: Fix helper integration issues * [Project] Further DNS over TCP architecturing * [Project] Rdns: Add more functions for TCP based requests * [Project] Rdns: Add preliminary reading logic for TCP channels * [Project] Rdns: Add reaper for inactive TCP connections * [Project] Rdns: Add timeout logic for TCP requests * [Project] Rdns: Do not treat TCP channels failure as fatal * [Project] Rdns: Fix TCP connection mess * [Project] Rdns: Fix TCP stuff cleanup * [Project] Rdns: Fix various ownership issues * [Project] Rdns: Implement TCP writing logic * [Project] Rdns: Initial support of TCP IO channels * [Project] Rdns: More fixes in TCP handling * [Project] Rdns: Restore the previous EDNS0 size * [Project] Rdns: Send truncated replies via TCP * [Project] Rdns: Unregister TCP requests * [Rework] Allow to restore SSL handlers after keepalive pooling * [Rework] Allow to set a different behaviour for actions from settings * [Rework] Include SSL flag into keepalive hash * [Rework] Make `rspamadm dmarc_report` default behaviour more sane * [Rework] Mempool: Use explicit alignment * [Rework] Rdns: Use faster and more compact hash table for DNS requests * [Rework] Rework SSL flag operations * [Rework] Take disabled flag into account * [Rework] Timeouts are now global per event and not reseted by IO activity * [Rework] Use xxh3 as a default hash and fix memory/alignment issues * [Rules] Fix old rules to stop global functions usage * [Rules] Fix symbol for DKIM temporary failure * [Rules] Remove ancient and inefficient rules * [Rules] Slightly reduce MULTIPLE_FROM score
* Release 3.13.1Vsevolod Stakhov2021-11-011-0/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Feature] Add junk_threshold for autolearn * [Feature] Add neural test command * [Feature] Antivirus: Allow to set fake eicar patterns for testing AV engines * [Feature] Lua_cdb: Add cdb building interface * [Feature] Ratelimit: Add per bucket configurations * [Feature] S3: Allow to store structured data in messagepack * [Fix] Add concept of uncancellable events to prevent use-after-free * [Fix] Add temporary guard to prevent linked list exploitation * [Fix] Another rework of the ucl hashing * [Fix] Another try to fix references safety * [Fix] Another try to fix rspamd_text passing in the selectors * [Fix] Avoid copy for received structure as it has raw C pointers * [Fix] Avoid dangling reference * [Fix] Correctly check numeric URLs in URL DNS lists * [Fix] Delete the correct pointer type * [Fix] Dmarc: Always lowercase domain * [Fix] Fix compilation of the hyperscan databases with errors * [Fix] Fix hash table lookup * [Fix] Fix http message flag shift * [Fix] Fix parsing of the from_hostname when it is an IP address * [Fix] Fix parsing of the unquoted attributes in HTML * [Fix] Fix passing of rspamd_text in selectors pipelines * [Fix] Fix rubbish QP sequences decoding * [Fix] Fix some complicated case with the closing tags parsing * [Fix] Fix the case when l tag is too small * [Fix] Html: Fix the case where only bgcolor is explicitly set * [Fix] Libucl: Fix deletion from ucl objects * [Fix] Namespace and add metadata for OpenMetrics, fix interleaving * [Fix] Plug memory leak in http settings reload * [Fix] Preserve SPF top record in the mempool variable * [Fix] Remove aarch64 GC64 workaround * [Fix] Remove bogus G_LIKELY * [Fix] Spf: Do not parse non TXT DNS replies as TXT replies * [Fix] Try to use on_connect/on_disconnect callbacks to handle internal Redis failures * [Fix] buffer overflow in rspamc counters * [Fix] fix static building * [Fix] lua_scanners - message_min_words logic * [Fix] src/lua/lua_mimepart.c: fix null dereference * [Project] Add constant iterators * [Project] Add helper library to handle mime strings in a more safe matter * [Project] Add preliminary support of CDB bayes dump * [Project] Add trim operations * [Project] Allow mempool allocated mime strings * [Project] Cdb: Finish backend implementation * [Project] Cdb: Fix configuration load * [Project] Cdb: Use shared data between cdb statfiles * [Project] Cdb: continue statistics backend implementation * [Project] Finish received headers rework part * [Project] Move C++ specific declarations to C++ header * [Project] Rework received headers parsing to C++ * [Project] Start using of the new received structure * [Project] Start work on cdb backend * [Rework] Further rework of the redis pool * [Rework] Redis_pool: fix issues found * [Rework] Rework learn and add classify condition * [Rework] Save invisible content to a separate buffer * [Rework] Start rewriting of the redis pool logic * [Rules] Improve zero font rule Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 3.03.0Vsevolod Stakhov2021-08-191-0/+273
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Align ARC scores with DKIM scores * [CritFix] Neural: Fix sorting application * [Feature] Add a simple dumper for bayes tokens * [Feature] Add lua_maps.fill_config_maps function * [Feature] Add preliminary exporter to AWS S3 * [Feature] Add preliminary restore bayes support * [Feature] Add race condition protection against hs_helper restarts * [Feature] Add rspamd_utf8_strcmp utility * [Feature] Add zstd streaming API * [Feature] Allow to log severity level explicitly * [Feature] Allow to save and show attachment name when inserting AV scan results * [Feature] Allow to sort urls for Lua * [Feature] Allow to specify different timeouts/retransmits for fuzzy rules * [Feature] Aws_s3: Allow to compress data stored * [Feature] CMakeLists.txt: Change check and run-test to use rspamd-test-cxx * fixes #3807 * [Feature] Dmarc_report: allow sending reports in batches * [Feature] Fuzzy_check: Allow to disable subject when making short text hash * [Feature] Lua_cryptobox: Add keyed ssl hash functions via HMAC * [Feature] Lua_task: Add get_urls_filtered method * [Feature] Make monitored checks less frequent * [Feature] Milter_headers: Add x-rspamd-pre-result header * [Feature] Neural: Allow to balance FP/FN for the network * [Feature] Ppopagate monitored errors from rbl module * [Feature] Pyzor calculate score dynamically Count - WL-Count of default_score in percent * [Feature] Rbl: Distinguish flattened and non-flattened selectors in RBL requests * [Feature] Re-add pyzor support * [Feature] Settings: add ip_map check and rework structure slightly * [Feature] Spamassassin: Allow to set the default priority for SA scores * [Feature] Strip smtp comments from message id * [Feature] add SYSTEM_ZSTD cmake option To use the system zstd instead on the bundled version * [Feature] external_relay plugin * [Feature] rspamadm clickhouse neural_train subcommand * [Fix] #3400 milter_headers: fix inverted logic for extended_headers_rcpt * [Fix] ASN: fix _FAIL symbol for when main symbol is disabled * [Fix] Add a special logic for text part with no text extraction * [Fix] Add diacritics flag for several eu languages * [Fix] Another FSM fix to accomodate possibility of multiple consequent ? * [Fix] Avoid curse of dynamic array referencing * [Fix] Avoid reinitialising neural settings * [Fix] Check remain before processing TXT records * [Fix] Enable error multiplier on http errors * [Fix] Finally rework parsing entities logic * [Fix] Fix '==' parsing in the content type attributes parser * [Fix] Fix IPv6 expansion for SPF macros * [Fix] Fix Mozilla Message-ID detection * [Fix] Fix an edge case in BITCOIN_ADDR rule * [Fix] Fix brain-damaged behaviour when http request has a custom Host header * [Fix] Fix check of limits in email address parsing * [Fix] Fix copy&paste error and rework * [Fix] Fix expressions logic for and/or and float values * [Fix] Fix fuzzy retransmits * [Fix] Fix http maps with no or invalid expires data * [Fix] Fix last quote character parsing in the content-type state machine * [Fix] Fix normalisation flags propagation * [Fix] Fix overflow when appending many broken tags * [Fix] Fix parsing of rfc2047 tokens with '?' inside * [Fix] Fix phishing flag set * [Fix] Fix rfc2047 embedded into rfc2231 pieces in special headers * [Fix] Fix round-robin rotation * [Fix] Fix searching for symbols * [Fix] Fix storing of the regexps inside variant * [Fix] Fix tokenization near exceptions * [Fix] Fix visibility calculations * [Fix] Html: Attach inline tags to the structure * [Fix] Html: Do not treat empty tags as block tags * [Fix] Ical: Do not extract urls from all flags using merely specific ones * [Fix] Initialise symcache even if it cannot be loaded properly * [Fix] Lua_fuzzy: Remove text parts check when checking image dimensions * [Fix] Lua_maps: Fix adjustments for the map type in the complex map definitions * [Fix] Lua_task: Fix deleted symbols in has_symbol/get_symbol * [Fix] Move metric and symcache link from validation to the init stage * [Fix] Oletools: Another try to fix table sorting * [Fix] One more default behaviour fix * [Fix] Phishing: Rework urls processing * [Fix] RBL: was missing some config schema * [Fix] Replies: Fix 'Reply-To' handling in task:get_reply_sender * [Fix] Rework metrics handling * [Fix] Save symcache on exit * [Fix] Selectors: Filter nil elements in lists * [Fix] Selectors: Properly fix implicit tostring for nils * [Fix] Try to fix some broken code in DMARC reporting plugin * [Fix] Urls: Fix processing of html urls when it comes to the flags * [Fix] Use proper buffer length * [Fix] Various visibility fixes * [Fix]: ASN: dns cb func should also return in case of an error * [Project] Add a simple css rule definition * [Project] Add css style skeleton * [Project] Add css syntax (adopted from ebnf) * [Project] Add css_selectors * [Project] Add doctest unit testing library * [Project] Add expected library * [Project] Add fmt library for simple string ops * [Project] Add fu2 library to better functions abstractions * [Project] Add hashing method * [Project] Add parsers skeleton * [Project] Add preliminary support of vcard parser * [Project] Add process exceptions for invisible text * [Project] Add some methods for css parser * [Project] Allow static libstdc++ * [Project] Another whitespace hack * [Project] CSS: Various fixes in the declarations and values parsing * [Project] Cpp: Add robin-hood hash map library * [Project] Css: Add AST debug * [Project] Css: Add colors conversion functions * [Project] Css: Add dimensions handling * [Project] Css: Add display value support * [Project] Css: Add frozen library from https://github.com/serge-sans-paille/frozen/ * [Project] Css: Add opacity support * [Project] Css: Add parser helpers to simplify debugging * [Project] Css: Add preliminary stylesheet support * [Project] Css: Add rules processing functions and tests * [Project] Css: Add simple selectors unit tests * [Project] Css: Add some c++ unit tests * [Project] Css: Add some debug methods * [Project] Css: Add some debug statements for the css parser * [Project] Css: Add some logical skeleton for declarations parser * [Project] Css: Add url/function tokens * [Project] Css: Allow at rules parsing * [Project] Css: Declarations parsing logic skeleton * [Project] Css: Enable conditional css parsing support from the HTML parser * [Project] Css: Finish generic lexer cases * [Project] Css: Fix HSL conversion * [Project] Css: Fix minus parsing * [Project] Css: Fix parser consumers nesting * [Project] Css: Fix parsing of the qualified rules * [Project] Css: Fix rules merging * [Project] Css: Further fixes to lexer * [Project] Css: Further steps to parse css colors + rework * [Project] Css: Further work on parser's methods * [Project] Css: Implement backlog of css tokens * [Project] Css: Implement numbers and ident parsers * [Project] Css: Implement simple css selectors lookup * [Project] Css: Implement styles merging * [Project] Css: Make debug strings json like to simplify tests * [Project] Css: Minor adjustments * [Project] Css: More meat to the lexer * [Project] Css: Move some of the tests to the doctest * [Project] Css: Projected a parser * [Project] Css: Properties attachment logic * [Project] Css: Remove ragel from build targets (maybe keep for reference) * [Project] Css: Rework css block structure * [Project] Css: Rework flags of css properties * [Project] Css: Rework tokens structure * [Project] Css: Several fixes + tests * [Project] Css: Simplify checks * [Project] Css: Simplify debug code * [Project] Css: Start css selectors parsing logic * [Project] Css: Start semantic parsing for rules * [Project] Css: Start stylesheet implementation * [Project] Css: Tidy up lambdas * [Project] Css: rework tokeniser * [Project] Dmarc: Add dmarc report tool (WIP) * [Project] Dmarc: Add munging configuration * [Project] Dmarc: Add preliminary munging logic * [Project] Dmarc: Fix header removal * [Project] Dmarc: Fix munging logic * [Project] Dmarc: Use full recipient address instead of a domain map * [Project] Dmarc: Use zlists for dmarc reports * [Project] Dmarc_report: Add message generation logic * [Project] Dmarc_report: Add preliminary sending support * [Project] Fix lua bindings * [Project] Fix xml/sgml tags processing * [Project] Handle new modification * [Project] Html/CSS: Add transform from a CSS rule to html block * [Project] Html/CSS: Link html and css styles * [Project] Html/CSS: Switch styles parsing to css parser * [Project] Html/Css: Fix some issues found * [Project] Html/Css: Implement visibility rules for a block * [Project] Html: Add more tests cases and fix some more corner issues * [Project] Html: Add rows display type support * [Project] Html: Allow decode entities function to normalise spaces + unit tests * [Project] Html: Another rework of the tags structure * [Project] Html: Another try to fix unbalanced cases * [Project] Html: Fix crossing spans * [Project] Html: Fix parent propagation * [Project] Html: Further rework of the html parsing stuff * [Project] Html: Implement logic for tags pairing * [Project] Html: Implement rawtext state machine * [Project] Html: Insert closing tags as well :( * [Project] Html: More fixes * [Project] Html: More fixes * [Project] Html: More spaces logic fixes * [Project] Html: One more attempt to write text content * [Project] Html: Replace \0 in html content * [Project] Html: Rework img/a tags handling * [Project] Html: Rework propagation method * [Project] Html: Rework tags placement * [Project] Html: Rework transparency logic * [Project] Html: Support 'hidden' attribute * [Project] Html: Try another approach to append tags content * [Project] Html: Try to deal with bad unknown tags properly * [Project] Lua_aws: Add canonicalisation utility * [Project] Lua_aws: Add function to produce AWS Authorisation header * [Project] Lua_aws: Implement request signing * [Project] Lua_mime: Add lua_mime.modify_headers routine * [Project] Lua_task: Add modify_header method * [Project] Lua_task: Allow to extract modified headers * [Project] Make unescape code public for unit testing * [Project] More fixes for closed tags * [Project] More fixes to calculations * [Project] Rework API for the modified headers * [Project] Rework html visibility rule * [Project] Skeleton of the css library * [Project] Start headers modification API structure * [Project] Start working on AWS Lua API * [Project] Use lua_mime to modify headers * [Project] Use modified headers on dkim signing * [Project] Use string_view to constexpr variant unpacking * [Rework] Add composites manager concept * [Rework] Add tags definitions * [Rework] Allow C code to be compiled with C++ compiler * [Rework] Clickhouse: Store url flags * [Rework] Composites: Rewrite the composites logic * [Rework] Composites: Start rework of the composites framework * [Rework] Dmarc: Move check policy function to the common utils * [Rework] Dmarc: Rework reports keys structure * [Rework] Further work to make html content private * [Rework] Html/CSS: Remove css C bindings as they are useless now * [Rework] Html/CSS: Rework Lua bindings * [Rework] Html/Css: Start rework of the html blocks * [Rework] Html: Add images processing logic * [Rework] Html: Add traverse function * [Rework] Html: Another steps to get rid of gnode * [Rework] Html: Convert to variant * [Rework] Html: Deal with the utf_content part * [Rework] Html: Final rework part for the html processing code * [Rework] Html: Fix Lua bindings * [Rework] Html: Forgot to add the internal include * [Rework] Html: Further html urls rework * [Rework] Html: Further rework of the tags content extraction * [Rework] Html: Make parameters as a vector again * [Rework] Html: Move blocks part * [Rework] Html: Move images processing stuff * [Rework] Html: Rework lua bindings * [Rework] Html: Start html text extraction rework * [Rework] Html: Start refactoring of the html tags handling * [Rework] Html: Start removing of GNode stuff * [Rework] Html: Start rework of the html content structure * [Rework] Lua_magic: Try to detect text parts with 8bit characters for non-utf8 encodings * [Rework] Move HTML url functions and rework them * [Rework] Move and adopt entities handling logic * [Rework] Move common and rarely used dmarc code to the library * [Rework] Move compression routines outside of rspamd_util library * [Rework] Move entities/tags handling * [Rework] Phishing: Split from redirectors usage * [Rework] Redesign html blocks propagation logic * [Rework] Remove tag name string * [Rework] Rename phished url to a linked url * [Rework] Reorganize dmarc plugin and remove unsupported reporting code * [Rework] Reputation: Use more flexible types in get/set functions * [Rework] Require proper C++ environment for Rspamd build * [Rework] Rework extended urls output * [Rework] Rework tags parsing machine * [Rework] Slightly improve old regexp API * [Rework] Start conversion of the redis pool code to c++ * [Rework] Try to resolve failed upstreams more agressively * [Rework] Use C++ utf8 library with unit tests to trim whitespaces * [Rework] Use C++ version for unicode normalisation * [Rework] Use C++ version of the lua threads pool * [Rules] Add raw addresses to MULTIPLE_FROM options * [Rules] Another fix to HTTP_TO_HTTPS rule * [Rules] Do not trigger HTML_SHORT_LINK_IMG on external images * [Rules] Extend FORGED_X_MAILER * [Rules] Extend OLD_X_MAILER * [Rules] Fix CTYPE_MIXED_BOGUS for text attachments * [Rules] Fix FPs for CTYPE_MIXED_BOGUS * [Rules] Fix HTTP_TO_HTTPS rule * [Rules] Fix HTTP_TO_HTTPS rule * [Rules] Fix zerofont rule (partially) * [Rules] Micro-optimize X_PHP_EVAL * [Rules] Reduce default weight for R_MISSING_CHARSET Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 2.72.7Vsevolod Stakhov2021-01-081-0/+93
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add R_DKIM_PERMFAIL to the metric * [CritFix] Dkim: Fix simple canonicalisation if multiple signatures are presented * [CritFix] Fix controller paths normalisation * [Feature] Add INVALID_DATE rule * [Feature] Add controller endpoint for training neural * [Feature] Add sanity checks for actions thresholds * [Feature] Add support of '==' and '!=' in Rspamd expressions * [Feature] Composites: Improve composite atoms parser * [Feature] Docker: use Debian slim variant * [Feature] Elastic: Add some missing fields * [Feature] Extract text from img alt attributes * [Feature] Improve charset detection logic * [Feature] Lua_clickhouse: Add optional row callback for large selections * [Feature] Lua_dns_resolver: Add idna_convert_utf8 method * [Feature] Lua_mime: Add ability to do multipattern replacement * [Feature] Lua_trie: Allow to report start of the match * [Feature] Multimap: support adding map values as extra options * [Feature] Neural: Move PCA learning to a subprocess * [Feature] RBL: support matching content/image URLs only * [Feature] RBL: support use of multiple selectors * [Feature] Reputation: Allow to specify ip masks * [Feature] Support SMIME signed messages container * [Feature] Support multiple conditions for symbols * [Feature] Support ping in milter mode * [Feature] Support rspamd_text in selector regexps * [Feature] Use own daemonization routine * [Feature] Vadesecure: Implement settings_outbound feature as recommended by Vade * [Feature] `rspamadm clickhouse` command * [Feature] allow hyperscan for aarch64 * [Fix] Allow to set priorities between post init scripts * [Fix] Allow to use maps for strings that are not zero terminated * [Fix] Apply max_lua_urls limit for emails as well * [Fix] Arc: Fix CV check on signing * [Fix] Arc: Fix signing of the broken ARC chains * [Fix] Clickhouse: escape carriage return * [Fix] Composites: Allow partial match * [Fix] Deduct type of a table methods * [Fix] Do not load errored hyperscan database * [Fix] Do not process links in ignored html tags * [Fix] Fix ClamAV result for cached encrypted file (#3395) * [Fix] Fix canonicalisation when l= tag is presented * [Fix] Fix flag shift * [Fix] Fix handling of skip/skip_process http flags * [Fix] Fix html attachments checks * [Fix] Fix issue with pushing binary formats to Lua strings * [Fix] Fix logging for rspamadm * [Fix] Fix off-by-one with init check * [Fix] Fix parsing of escape characters in quoted pairs * [Fix] Fix pushing ucl strings with \0 inside * [Fix] Fix quoted-printable soft newlines bugged case * [Fix] Fix settings in case actions are set to null (#3415) * [Fix] Fix several issues with auth results producing * [Fix] Fix smtp comments exclusion * [Fix] Fix smtp date syntax definition * [Fix] Fix substring search in case if srchlen == inlen * [Fix] Fix text selectors * [Fix] Honour `systemd` setting when logging to console (#3514) * [Fix] Html: Add entities collisions prevention logic (e.g. for mathml entities) * [Fix] Lua_auth_results: Quote potentially bad values in AR header * [Fix] Multimap: Fix flags usage * [Fix] Multimap: Fix scoring for combined maps * [Fix] Plug GList * leak in redis pool * [Fix] RBL: allow for multiple matches of the same label if types are different * [Fix] Rely on libev checks for file maps * [Fix] Restore simple dkim canonicalisation mode * [Fix] Return MimeCharset as we work with emails... * [Fix] Spamassassin: Fix pcre_only flags * [Fix] Spamassassin: Preserve 'pcre_only' flag when dealing with regexp replacements * [Fix] Try to fix GError leak * [Fix] Try to fix a mess with settings loading by adding priorities * [Fix] Try to move setings initialisation to a later stage * [Fix] Use dup fd in milter handler to avoid races with the proxy * [Fix] Use message pointer to avoid obsolete data to be cached * [Project] Rbl: Migrate to `checks` * [Project] Rbl: Move config code outside of the plugin * [Project] Ressurect empty prefilters as connection filters * [Project] Support connection filters registration from Lua * [Rework] Add final cleanup logic * [Rework] Add preliminary support of hyperscan caching for re maps * [Rework] Add stale cache removal * [Rework] Clickhouse: Improve performance * [Rework] Distinguish between strict config test mode * [Rework] Furhter logging improvements * [Rework] Milter_headers: improve extended_headers_rcpt support * [Rework] Move parsers to a separate lua library * [Rework] Neural: Skip composite symbols * [Rework] Rbl: Rework defaults logic * [Rework] Some tunes to cache saving * [Rework] Track maps origins * [Rework] Use full crypto hash for regexp maps * [Rules] Remove broken rule Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 2.62.6Vsevolod Stakhov2020-09-301-0/+172
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add missing symbols * [Conf] Add missing symbols * [Conf] Fix fat-fingers typo * [Conf] Fix wrong comment in options.inc * [Conf] Neural: Fix the default name for max_trains * [Conf] Register a known symbol * [Conf] Spf: Add R_SPF_PERMFAIL symbol * [CritFix] Arc: Fix ARC validation for chains of signatures * [CritFix] Distinguish socketpairs between different fuzzy workers * [CritFix] Fix IDNA dots parsing * [CritFix] Fix test assertion method * [CritFix] Fix usage of crypto_sign it should be crypto_sign_detached! * [Feature] Add BOUNCE rule * [Feature] Add controller plugins support and selectors plugin * [Feature] Add maps query method * [Feature] Add minimal delay to fuzzy storage * [Feature] Add multiple base32 alphabets for decoding * [Feature] Add preliminary support of BCH addresses * [Feature] Add query_specific endpoint * [Feature] Allow multiple base32 encodings in Lua API * [Feature] Allow to specify nonces manually * [Feature] Controller: Allow to pass query arguments to the lua webui plugins * [Feature] Fuzzy_check: Add gen_hashes command * [Feature] Fuzzy_check: Add weight_threshold option for fuzzy rules * [Feature] Implement address retry on connection failure * [Feature] Improve limits in pdf scanning * [Feature] Initial support of subscribe command in lua_redis * [Feature] Lua_cryptobox: Add secretbox API * [Feature] Lua_text: Add encoding methods * [Feature] Milter_headers: Allow to activate routines via users settings * [Feature] PDF: Add timeouts for expensive operations * [Feature] Preliminary maps addon for controller * [Feature] Split pdf processing object and output object to allow GC * [Feature] Support BLIS blas library * [Feature] Support input vectorisation by recvmmsg call * [Feature] Support multiple base32 alphabets * [Feature] add queueid, uid, messageid and specific symbols to selectors [Minor] use only selectors to fill vars in force_actions message * [Feature] allow variables in force_actions messages * [Feature] extend lua api * [Fix] #3249 * [Fix] Allow to adjust neurons in the hidden layer * [Fix] Another try to fix email names parsing * [Fix] Arc: Allow to reuse authentication results when doing multi-stage signing * [Fix] Arc: Fix bug with arc chains verification where i>1 * [Fix] Arc: Sort headers by their i= value * [Fix] Change neural plugin's loss function * [Fix] Deal with double eqsigns when decoding headers * [Fix] Default ANN names in clickhouse * [Fix] Disable reuseport for TCP sockets as it causes too many troubles * [Fix] Disable text detection heuristics for encrypted parts * [Fix] Distinguish DKIM keys by md5 * [Fix] Distinguish type from flags in register_symbol * [Fix] Dmarc: Unbreak reporting after cf2ae3292ac93da8b6e0624b48a62828a51803c9 * [Fix] Do not flag pre-result of virus scanners as least if action is reject * [Fix] Do not use GC64 workaround on 32bit platforms, omg * [Fix] Exclude damaged urls from html parser * [Fix] Fix FREEMAIL_REPLYTO_NEQ_FROM_DOM * [Fix] Fix FROM_NEQ_ENVFROM * [Fix] Fix FWD_GOOGLE rule (#1815) * [Fix] Fix adding of the empty archive file for gzip * [Fix] Fix aliases in forged recipients and limit number of iterations * [Fix] Fix authentication results insertion * [Fix] Fix calling of methods in selectors * [Fix] Fix clen length for hiredis... * [Fix] Fix endless loop if broken arc chain has been found * [Fix] Fix false - operation * [Fix] Fix get_urls table invocation * [Fix] Fix group based composites * [Fix] Fix headers passing in rspamd_proxy * [Fix] Fix incomplete utf8 sequences handling * [Fix] Fix lua_next invocation * [Fix] Fix lua_parse_symbol_type function logic * [Fix] Fix multiple listen configuration * [Fix] Fix occasional encryption of the cached data * [Fix] Fix parsing boundaries with spaces * [Fix] Fix passing of methods arguments * [Fix] Fix poor man allocator algorithm * [Fix] Fix regexp selector and add flattening * [Fix] Fix rfc base32 encode ordering (skip inverse bits) * [Fix] Fix rfc based base32 decoding * [Fix] Fix sockets leak in the client * [Fix] Fix storing of the original smtp from * [Fix] Fix types check and types usage in lua_cryptobox * [Fix] Fix unused results * [Fix] Fuzzy_check: Disable shingles for short texts (really) * [Fix] Ical: Fix identation grammar * [Fix] Improve part:is_attachment logic * [Fix] Mmap return value must be checked versus MAP_FAILED * [Fix] One more fix to skip images that are not urls * [Fix] Pdf: Support some weird objects with no newline before endobj * [Fix] Rbl: Fix ignore_defaults in conjunction with ignore_whitelists * [Fix] Restore support for `for` and `id` parts in received headers * [Fix] Segmentation fault in contrib/lua-lpeg/lpvm.c on ppc64el * [Fix] Skip spaces at the boundary end * [Fix] Slashing fix: fix captures matching API * [Fix] Spamassassin: Rework metas processing * [Fix] Store reference of upstream list in upstreams objects * [Fix] Understand utf8 in content-disposition parser * [Fix] Unify selectors digest functions * [Fix] Use `abs` value when checking composites * [Fix] Use strict IDNA for utf8 DNS names + add sanity checks for DNS names * [Fix] Use unsigned char and better support of utf8 in ragel parser * [Fix] add missing selector_cache declaration * [Project] Add `L` flag for regexps to save start of the match in Hyperscan * [Project] Add `lower` method to lua_text * [Project] Add a simple matrix Lua library * [Project] Add implicit bitcoincash prefix * [Project] Add linalg ffi library for prototyping * [Project] Add methods to append data to fuzzy requests * [Project] Add routine to call a generic lua function * [Project] Add ssyev method interface * [Project] Add tensors index method * [Project] Add text:sub method * [Project] Allow rspamd_text based selectors * [Project] Allow to specify re_conditions for regular expressions * [Project] Attach extensions to the binary fuzzy commands * [Project] Bitcoin: BTC cash addresses needs some checksum validation * [Project] Cleanup the redis script * [Project] Convert bitcoin rules to the new regexp conditions feature * [Project] Detect memrchr in systems that supports it * [Project] Do not listen sockets in the main process * [Project] Implement 'probabilistic' learn mode for ANN * [Project] Implement BTC polymod in C as it requires 64 bit ops * [Project] Implement bitcoin cash validation in a proper way * [Project] Implement extensions logic for fuzzy storage * [Project] Implement symbols insertion in multiple results mode * [Project] Lua_text: Add method memchr * [Project] Neural: Add PCA loading logic * [Project] Neural: Fix PCA based learning * [Project] Neural: Fix matrix gemm * [Project] Neural: Further PCA fixes * [Project] Neural: Implement PCA in learning * [Project] Neural: Implement PCA learning * [Project] Neural: Implement PCA on ANN forward * [Project] Neural: Implement PCA serialisation * [Project] Neural: Start PCA implementation * [Project] Neural: Use C version of scatter matrix producing * [Project] Preliminary support of lua conditions for regexps * [Project] Preliminary usage of the reuseport * [Project] Process composites separately for each shadow result * [Project] Remove old code * [Project] Rework scan result functions to support shadow results * [Project] Rework some more functions to work with shadow results * [Project] Some more fixes * [Project] Start results chain implementation * [Project] Support fun iterators on rspamd_text objects * [Project] Support multiply, minus and divide operators in expressions * [Project] Tensor: Move scatter matrix calculation to C * [Rework] Allow to specify exat metric result when adding a symbol * [Rework] Change and improve openblas detection and usage * [Rework] Close listen sockets in main after fork * [Rework] Further rework of lua urls extraction API * [Rework] Lua_cryptobox: Allow to store output of the hash function * [Rework] Lua_task: Add more methods to deal with shadow results * [Rework] Modernize logging for expressions * [Rework] Remove empty prefilters feature - we are not prepared... * [Rework] Remove old FindLua module, disable lua fallback when LuaJIT is enabled * [Rework] Rework and refactor forged recipients plugin * [Rework] Rework expressions processing * [Rework] Rework fuzzy commands processing * [Rework] Rework url flags handling API * [Rework] Rework urls extraction * [Rework] Split operations processing and add more debug logs * [Rework] Update zstd to 1.4.5 * [Rework] Use google-ced instead of libicu chardet as the former sucks * [Rework] add alias util:parse_addr for util:parse_mail_address * [Rework] get rid of util:parse_addr duplicating the util:parse_mail_address, replace where used * [Rules] Allow prefix for bitcoin cash addresses * [Rules] More fixes for bitcoin cash addresses decoding * [Rules] Refactor bleach32 addresses handling Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 2.52.5Vsevolod Stakhov2020-04-011-0/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Mark Rspamd emailbl as ignore whitelist * [Conf] RBL: Add missing emails = true option * [Feature] Add support for scripts in fuzzy storage * [Feature] Arc: Add whitelisted_signers_map option * [Feature] Implement hosts file processing * [Feature] Neural: Introduce classes bias that allows non-equal classes learning * [Feature] Update libev to 4.33 * [Fix] Another brain damage html standard adoptions * [Fix] Another fix for brain damaged obs-fws state * [Fix] Fix flags that caused force_actions failure * [Fix] Fix logging issue * [Fix] Fix lua symbols scores registration when config does not define scores * [Fix] Fix opaque maps logic * [Fix] Fix parsing of the html tags with no spaces after attributes * [Fix] Fix some corner cases in urls parsing, add limits * [Fix] Fix tlds extraction if custom composition rules are used * [Fix] Fix variables replacement in mempool * [Fix] Improve base64 detection * [Fix] Normalize dynamic scores in ANN correctly * [Fix] Plug memory leak introduced by #3153 * [Fix] Stat_redis_backend: Fix memory leak and simplify learn path * [Fix] Try hard to deal with ghost workers * [Fix] metadata_exporter default formatter * [Rework] Change the way to extract URLs when dealing with alternative parts * [Rework] Fix various url extraction issues * [Rework] Re cache: Load compiled hyperscan in the main process as well * [Rework] Re cache: Load hyperscan early * [Rework] Rework URL structure: adjust tld part * [Rework] Rework URL structure: host field * [Rework] Rework URL structure: more structure optimisations * [Rework] Rework URL structure: user field * [Rework] URL: Another update for urls extraction logic * [Rework] Urls: Improve query urls handling * [Rework] Urls: adopt html related stuff * [Rework] Urls: more rework of the urls sets * [Rework] Urls: process query urls in HTML urls correctly * [Rework] Urls: rework urls hash structure * [Rework] Urls: update lua libraries * [Rework] Use multiple search tries for different url extraction types Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 2.42.4Vsevolod Stakhov2020-02-261-0/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix parsing of the content type attributes * [Feature] Clickhouse: Add extra columns support * [Feature] Rbl: Add url_compose_map option for RBL rules * [Fix] 'R' flag is for all headers regexp * [Fix] Allow to reset settings id from Lua (e.g. because of the priority) * [Fix] Avoid collisions in mempool variables by changing fuzzy caching logic * [Fix] Avoid strdup usage for symbols options * [Fix] Do not trust stat(2) it lies * [Fix] Filter all options for symbols to have sane characters * [Fix] Fix all headers iteration * [Fix] Fix allowed_settings for neural * [Fix] Fix listen socket parsing * [Fix] Fix maps expressions evaluation * [Fix] Fix sentinel connections leak by using async connections * [Fix] Fix smtp message on passthrough result * [Fix] Fix tld compositon rules * [Fix] Fuzzy_storage: Do not check for shingles if a direct hash has been found * [Fix] Lua_mime: Do not perform QP encoding for 7bit parts * [Fix] Neural: Distinguish missing symbols from symbols with low scores * [Fix] Support listening on systemd sockets by name * [Project] Add lua_urls_compose library * [Project] Allow to set a custom log function to the logger * [Project] CDB maps: Start making cdb a first class citizen * [Project] Clickhouse: Add extra columns concept * [Project] Fix urls composition rules, add unit tests * [Project] Unify cdb maps * [Rework] Logger infrastructure rework * [Rework] Refactor libraries structure * [Rework] Rework SSL caching * [Rework] Update snowball stemmer to 2.0 and remove all crap aside of UTF8 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 2.32.3Vsevolod Stakhov2020-02-041-0/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] SPF is no longer a C module * [Conf] Update spamtrap map path example * [CritFix] Fix html entities decoding * [CritFix] Fix re cache when mix of pcre and hyperscan is used * [Feature] Allow milter code to deal with multiple headers * [Feature] Antivirus: Add avast support * [Feature] Dkim_signing: Allow to sign via milter_headers * [Feature] Implement content hashes * [Feature] Lua_text: Add regexp split iterator method * [Feature] Lua_text: Implement flattening of the input tables * [Feature] Send quit command to Redis * [Feature] Speed up is_ascii function * [Feature] Spf: Add external_relay option * [Fix] Avoid double escaping * [Fix] Fix O(N^2) algorithm * [Fix] Fix arc seal validation * [Fix] Fix base tag processing according to stupid HTML renderer behaviour * [Fix] Fix dealing with `\0` in ucl strings and JSON * [Fix] Fix gpg parts misdetection * [Fix] Fix ignored symbols exporting * [Fix] Fix processing of numeric url's * [Fix] Fix processing of the closed tcp connections * [Fix] Fix regexp type check for pcre2 * [Fix] Fix urls encode function * [Fix] Fix urls shifting when doing decode to include separators * [Fix] Fix white on white rule and add is_leaf flag * [Fix] Further fixes in charset detection * [Fix] Ignore diacritics in chartable module for specific languages * [Fix] Limit size of symbols options by max_opts_len option * [Fix] More fixes in html tag content calculations * [Fix] Plug memory leak in fuzzy storage * [Fix] Process high priority settings even if settings/id has been specified * [Fix] Select a different upstream on last retransmit * [Fix] Treat soft hyphen as zero width space * [Fix] Try harder to watch the lifetime of the key_stat * [Fix] Use ipv6-mapped-ipv4 addresses in radix trie * [Project] Add logic to break execution when processing symbols* * [Project] Add methods to set specific content for mime parts from Lua * [Project] Lua_content: support PDF files * [Project] Move dns_tool to using of the rspamd_spf from FFI module * [Project] Preliminary SPF plugin in Lua * [Project] Show debug stat for memory pool * [Project] Some rework about specific data that is now tagged * [Project] Start reworking of the mempool structure * [Rework] Allow to add userdata as symbols options * [Rework] Change mime part specifics handling * [Rework] Move LRU SPF cache from spf plugin * [Rework] Rework HTML tags content attachment * [Rework] Rework options hash structure * [Rework] Start lua_content library * [Rework] Stop using of uthash for http headers * [Rework] Use faster hashing approach for memory pools variables * [Rules] Add PDF related rules Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 2.22.2Vsevolod Stakhov2019-11-191-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Antivirus: Fix the default config * [Feature] Add verdict library in lua * [Feature] Allow exception when choosing upstream * [Feature] Allow to disable symbols from the metric config * [Feature] Allow to limit maps per specific worker * [Feature] Always validate Rspamd protocol output * [Feature] Antivirus: Add preliminary virustotal support * [Feature] Clickhouse: Rework Clickhouse collection logic * [Feature] Improve base64 usage * [Feature] Shutdown timeout is now associated with task timeout * [Fix] #3129 Multiple classifiers on redis working incorrectly * [Fix] Allow real upstreams configuration * [Fix] Another try to fix slow callbacks and timers * [Fix] Check results of write message as SSL can bork them * [Fix] Clickhouse: Avoid potential races in collection * [Fix] Clickhouse: Fix periodic script * [Fix] Fail DNS upstream on each retransmit attempt * [Fix] Fix consistent hashing when upstreams are marked inactive * [Fix] Fix issues found * [Fix] Fix off-by-one in retries for the proxy * [Fix] Fix termination * [Fix] Fix upstreams exclusion logic * [Fix] Fix utf8 validation for symbols options and empty strings * [Fix] Oops, fix maps reload * [Fix] Rbl: Allow utf8 lookups for IDN domains * [Fix] Sigh, another try to fix brain-damaged openssl * [Project] Add fast utf8 validation library * [Project] Use own utf8 validation instead of glib * [Rework] Another phase of finish actions rework * [Rework] Further cmake system rework * [Rework] Further isolation of the controller's functions * [Rework] Make cmake structure more modular * [Rework] Move cmake modules to a dedicated path * [Rework] Replace controller functions by any scanner worker if needed * [Rework] Rework final scripts logic * [Rework] Rewrite rspamd_str_make_utf_valid function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 2.12.1Vsevolod Stakhov2019-10-281-0/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Update neural.conf * [CritFix] Fix dkim verification for multiple headers listed * [Feature] Add support of uudecode * [Feature] Allow to explicitly set events backend * [Feature] Implement configurable limits for SPF lookups * [Feature] Lua_scanners: Use lua magic for inclusion/exclusion logic * [Feature] Multimap: Do not check files in office archives * [Feature] Neural: Add sampling when storing training vectors * [Feature] SPF: Allow to disable AAAA checks in configuration * [Feature] Spf: Add limits configuration support * [Feature] Store etag in cached HTTP maps + better logging * [Feature] Support segwit BTC addresses, fix LTC verification * [Feature] Support uuencoding * [Fix] Add configurable number of threads for OpenBLAS * [Fix] Add workaround for ragel 7 in hyperscan related maps code * [Fix] Another fix for numeric urls parsing * [Fix] Correct EMA time calculations * [Fix] Do not treat archives as text * [Fix] Do not use strdup on data extracted from lua * [Fix] Fix a failure calcuating URL reputation. * [Fix] Fix crash due to constructors init order * [Fix] Fix crash on parts with no cd * [Fix] Fix empty prefilters that require mime structures * [Fix] Fix event loop creation * [Fix] Fix issues sending DMARC reports. * [Fix] Fix misprint * [Fix] Fix saving of the file maps * [Fix] Fix size calculations when converting from utf16 * [Fix] Fix support of disable_monitoring in rbl * [Fix] Fix use-after-free * [Fix] Fix zip files check to relax requirements * [Fix] Important hiredis fixes * [Fix] Lot's of fixes in maps check logic * [Fix] Lua_tcp: Deal with temporary fails on write * [Fix] Lua_tcp: Make write errors fatal and rework error handlers * [Fix] Meta: Filter some more values * [Fix] Neural: Add protection agains infinities * [Fix] Oops, fix math.huge invocation * [Fix] Plug memory leak * [Fix] Sigh, another email to string fix * [Fix] Try to fix another ownership race in ssl connection * [Fix] Uuencode: Fix parsing of corrupted uuencode * [Fix] lua_scanners - razor rename need_check function * [Rework] Require CMake 3.9 to work, remove manual lto crap Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 2.02.0Vsevolod Stakhov2019-10-111-0/+244
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add BROKEN_HEADERS_MAILLIST composite * [Conf] Add path to greylist-whitelist-domains.inc * [Conf] Clarify documentation in the config files * [Conf] Introduce maps.d directories * [Conf] Log settings id by default * [Conf] Make LEAKED_PASSWORD_SCAM a composite rule again * [Conf] Move all surbl/emails rules to rbl * [Conf] Register new Spamhaus codes * [Conf] Remove configs for deleted modules * [Conf] Remove surbl parts, fix hash_format attribute * [Conf] Show autolearn sample * [Conf] Slashing: Change default stats backend to Redis * [Conf] Surbl: Utilise new `check_emails` option * [Conf] Update header * [Conf] Use multi-prefixes RBLs in the default config * [CritFix] Deal with case-sensivity in Content-Disposition parser * [CritFix] Eliminate old endpoint * [CritFix] Fix case sensivity when parsing Content-Type * [CritFix] Fix loading of DKIM public keys * [CritFix] Fix procesing of urls * [CritFix] Fix whitelisting when both spf and dkim are required to be valid * [CritFix] Langdet: Fix language detection where no stop words found * [Feature] Add description to the groups * [Feature] Add limit for number of URLs in Lua * [Feature] Add logging of groups to the log_format * [Feature] Add lua_smtp library * [Feature] Add maps cache and type refinement * [Feature] Add p0f scanner * [Feature] Adopt emails module to use lua_maps * [Feature] Allow options matching in composites * [Feature] Allow selectors in rbl module * [Feature] Allow to output group results * [Feature] Asn: Allow to use bgpdump when NET::MRT is broken * [Feature] Calculate tokens occurrences distribution * [Feature] Clickhouse: Add authenticated user and settings id columns * [Feature] Clickhouse: Store groups data * [Feature] Clickhouse: Utilise LowCardinality feature * [Feature] Implement Redis prefixes registration logic * [Feature] Implement settings id propagation between deps * [Feature] Improve AV results caching * [Feature] Improve autolearning * [Feature] Improve logging locking logic (remove it actually) * [Feature] Improve settings processing * [Feature] Langdet: Limit number of stop words to be checked * [Feature] Libucl: Allow to sort keys in ucl objects * [Feature] Lua_config: Extend get symbols method * [Feature] Lua_maps: Allow static maps for key-value pairs * [Feature] Lua_mimepart: Add function filter_words * [Feature] Lua_selectors: Add `words` selector * [Feature] Lua_selectors: Add sort and uniq transform functions * [Feature] Lua_selectors: Allow table arguments for selectors * [Feature] Lua_tcp: Add preliminary support of SSL connections * [Feature] Lua_trie: More flexible API * [Feature] Lua_util: Add filter_specific_url function * [Feature] Lua_util: table_digest can now recursively traverse tables * [Feature] Maillist: Improve detection * [Feature] Maps: Allow caching for complex maps * [Feature] Monitored: Support random lookups * [Feature] Multimap: Add combined maps prototype * [Feature] Multimap: Add dependend maps via redis keys selectors * [Feature] Multimap: Allow multiple email addresses matches * [Feature] Multimap: Also check detected charset when do filename checks * [Feature] Output number of messages processed to proctitle * [Feature] Perform clean SSL shutdown * [Feature] Performance: Do not use base64 SIMD version for bad inputs * [Feature] RBL: Support bit results in replies * [Feature] RBL: Support type specific prefixes * [Feature] Ratelimit: Consider number of SMTP recipients * [Feature] Rbl: Add ability to check urls * [Feature] Rbl: Add resolve_ip based RBLs * [Feature] Rbl: Make config checks much more strict * [Feature] Rbl: Support per-rule whitelists * [Feature] Rbl: Support process script * [Feature] Rbl: Support replyto addresses * [Feature] SURBL: Allow to check email domains * [Feature] Selectors: Add `list` generator * [Feature] Selectors: Add `specific_urls` extractor * [Feature] Selectors: Add flatten function * [Feature] Selectors: Support filter_map and apply_map functions * [Feature] Store Clickhouse data outside of lua alloc * [Feature] Support caching for encrypted files and macros * [Feature] Support images when extracting urls * [Feature] Support more hyperscan flags * [Feature] Support protocol flags * [Feature] URL: Apply stringprep to hostnames to filter garbage * [Feature] Upstreams: Add lazy resolving logic to all upstreams * [Feature] Upstreams: Set noresolve flag on numeric upstreams * [Feature] Use `scores` in apply section * [Feature] Use maps logic from lua_maps for multimap * [Feature] Use random monitored in rbl module * [Feature] lua_scanners - add Razor support * [Fix] Add another safe-guard in urls processing * [Fix] Add debug to ssl, fixed write hangs * [Fix] Add missing groups to C callback symbols * [Fix] Add more checks for ghosts symbols * [Fix] Allow to enable or add new actions via settings * [Fix] Allow to set 0 size for spf/dkim caches * [Fix] Another bunch of fixes towards protocol mess * [Fix] Another fix to deal with bad URLs * [Fix] Arc: Another bunch of fixes for arc signing * [Fix] Arc: More arc signing fixes * [Fix] Avoid another overflow in fpconv * [Fix] Clickhouse: Fix quoting * [Fix] Clickhouse: Fix retention query quoting * [Fix] Distinguish empty and non-empty prefilters * [Fix] Distinguish remote and local addrs parsing * [Fix] Do not assert if length of sig is bad, just fail verification * [Fix] Do not assert if we have broken mime boundary in the headers * [Fix] Do not call implicit strlen to avoid issues * [Fix] Do not count images urls when checking url regexps for compatibility * [Fix] Do not output rbl suffix in symbol option * [Fix] Do not use config pool to avoid issues with double reload * [Fix] Do not use ephemeral string * [Fix] Do not use lightuserdata for traceback * [Fix] Do not use priority in metric registration * [Fix] Emails: Check email sanity before testing on BL * [Fix] Emails: Fix misprint in key name * [Fix] Escape utf in regexp to dodge ragel/hyperscan issue * [Fix] Extend task_timeout to postfilters stage * [Fix] Fix ARC signing after fixing another bug in it... * [Fix] Fix AV scan logic * [Fix] Fix DMARC_NA behaviour in case of no valid policies * [Fix] Fix LRU hash iteration logic * [Fix] Fix alignment mess * [Fix] Fix configuring symbols without scores * [Fix] Fix disabling of the actions * [Fix] Fix dkim signing exceptions * [Fix] Fix embedded images linking logic * [Fix] Fix events leak * [Fix] Fix eviction corner case * [Fix] Fix fuzzy image score calculation #2962 * [Fix] Fix hang in fuzzy_learn when explicit rotation is set * [Fix] Fix headers propagation logic * [Fix] Fix hearbeats restart issue * [Fix] Fix history reset * [Fix] Fix log parameter * [Fix] Fix lua_ip_equal logic * [Fix] Fix more issues with nested messages + tests * [Fix] Fix normalization of non-alphabet based languages * [Fix] Fix offsets when parsing message/rfc822 in multipart * [Fix] Fix options in rbl symbols * [Fix] Fix out of bound access in lua logger * [Fix] Fix out-of-bound read in qp decode * [Fix] Fix parent CTE propagation * [Fix] Fix parsing of the received headers with empty part * [Fix] Fix pending checks for events * [Fix] Fix printing of NULL pointer with fixed length * [Fix] Fix race condition in watcher handler * [Fix] Fix read-after-end in quoted printable decoding * [Fix] Fix redis sentinel support * [Fix] Fix registry leak in case of DNS errors * [Fix] Fix reload logic * [Fix] Fix sending of large entries via HTTPS * [Fix] Fix settings reload * [Fix] Fix some more corner cases for fpconv * [Fix] Fix trie code when there are regexps and Hyperscan is absent * [Fix] Further fixes to printing of the FP numbers * [Fix] Fuzzy_check: Fix timeouts * [Fix] Grrr, fix empty ip case * [Fix] Html: Fix processing of fjlig entity * [Fix] Lang_det: Try better to distinguish Chinese and Japanese * [Fix] Lua_mime: Fix reversed extensions map * [Fix] Lua_task: Fix message-less API * [Fix] Lua_tcp: Report connection failures * [Fix] Lua_tcp: Various fixes and debugging improvements * [Fix] Metadata_exporter: This plugin is idempotent not a postfilter * [Fix] More fixes to extract_specific_urls * [Fix] More stages fixes * [Fix] Neural: Another bunch of fixes * [Fix] Neural: use version in ANN key profile * [Fix] Postpone lua state destruction to allow lua dtors to be used * [Fix] Prefer surbl/emails rule on rbl to preserve compatibility * [Fix] RBL: Fix behaviour of emails_domainonly * [Fix] Ratelimit: Fix dynamic score * [Fix] Rbl: Fix emailbl functions * [Fix] Really fix hyperscan workaround * [Fix] Set sanity limits for pcre2 * [Fix] Settings: Fix settings check flags * [Fix] Sort keys when getting data from Lua when filling rules * [Fix] Statistics: Do not query Redis tokens when there are no learns * [Fix] Stop IO event on write finished in http connection * [Fix] Use heuristically detected text parts data * [Fix] Various fixes to QP encoding algorithm * [Fix] Various fixes to SSL state machine handler * [Fix] Various fixes to asn module * [Fix] Workaround for empty charset in rfc2231 encoding * [Project] Switch from torch to KANN * [Project] Add heartbeat events * [Project] Add preliminary support of the Kaspersky Scan Engine * [Project] Add preliminary version of maps expressions * [Project] Add preprocessed settings to the config structure * [Project] Add simple forward propagation function * [Project] Add small helpers for migration simplifications * [Project] Allow to replace body in milter * [Project] Bundle libev * [Project] First refactoring step libevent->libev * [Project] Implement syntax highlighting for Lua * [Project] Lua_magic: Adopt lua_magic stuff in mime_types * [Project] Remove libfann, gd and other unsupported stuff * [Project] Remove torch * [Project] Rework upstreams * [Rework] Allow execution of async events when hs compiles regexps * [Rework] Bayes expiry: eliminate `default` expiration mode * [Rework] Dkim: Remove signing code * [Rework] Dkim_signing: Move sign condition to dkim_signing * [Rework] Do not lowercase all data send to ClickHouse * [Rework] Drop url tags * [Rework] Eliminate lua_squeeze as it has shown no improvements * [Rework] Eliminate virtual scan time as it is useless * [Rework] Lua core: Use lightuserdata to index classes * [Rework] Lua_util: Another rework for extract_specific_urls * [Rework] Migrate from ip_score to reputation * [Rework] Move mime modification functions to lua_mime library * [Rework] Rbl: Major whitelisting logic rework * [Rework] Remove deprecated plugins * [Rework] Remove log helper worker * [Rework] Remove rspamd.classifiers.lua * [Rework] Rename filter.h to a more sane name * [Rework] Reorganise selectors implementation * [Rework] Replace linenoise with replxx * [Rework] Reputation: Remove ipnet from the ip reputation * [Rework] Reputation: Slashing - change name of symbols * [Rework] Rework children operations * [Rework] Rework config reload * [Rework] Rework expression API * [Rework] Rework image urls processing * [Rework] Rework initialisation to reduce static leaks count * [Rework] Rework request headers processing * [Rework] Slashing: Change versioning schema - move to 2.0 * [Rework] Slashing: Turn off postfilters when passthrough result is set * [Rework] Start moving to replxx * [Rework] Stop support of signed HTTP maps to simplify code * [Rework] Store ASN as UInt32 in ClickHouse * [Rework] Url_redirector: Rewrite plugin * [Rework] Use a dedicated library for autolearn * [Rework] Use libsodium instead of hand crafted crypto implementations * [Rework] Use opaque structure to store a table of mime headers * [Rules] Add dedicated bitcoin addresses filter rule * [Rules] Add more detection to LEAKED_PASSWORD_SCAM * [Rules] Catch LTC addresses * [Rules] Reduce weight of RSPAMD_EMAILBL * [Rules] Rework LEAKED_PASSWORD_SCAM rule one more time Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* [Minor] Add Changelog for 1.9.4Vsevolod Stakhov2019-05-231-0/+11
|
* Release 1.9.31.9.3Vsevolod Stakhov2019-05-131-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add IP_SCORE_FREEMAIL composite rule * [Feature] Add cryptobox method to generate dkim keypairs * [Feature] Add fast hashes to lua cryptobox hash * [Feature] Add least passthrough results * [Feature] Allow oversign if exists mode * [Feature] Clickhouse: Modernise table initial schema * [Feature] Implement IUF interface for specific fast hashes * [Feature] Lua_util: Allow to obfuscate different fields * [Feature] Tune memory management in Rspamd and Lua * [Fix] Avoid buffer overflow when printing long lua strings * [Fix] Change the default oversigning headers to a more sane list * [Fix] Clickhouse: Do not store digest as it is not needed now * [Fix] Clickhouse: Fix lots of storage issues * [Fix] Clickhouse: Support custom actions * [Fix] Deny URLs where hostname is bogus * [Fix] Do not blacklist mail by SPF/DMARC for local/authed users * [Fix] Fix DoS caused by bug in glib * [Fix] Fix UCL parsing of the multiline strings * [Fix] Fix buffer overflow when printing small floats * [Fix] Fix init code for servers keypairs cache * [Fix] Fix issue with urls with no tld (e.g. IP) * [Fix] Fix memory in arc signing logic * [Fix] Fix memory leak in language detector during reloads * [Fix] Fix mixed case content type processing * [Fix] Fix processing of the ip urls in file * [Fix] Fix use after free * [Fix] HTML: Fix `size` attribute processing * [Fix] Hum, it seems that 99ff1c8 was not correct * [Fix] Lua_task: Fix task:get_from method * [Fix] Preserve fd when mapping file to scan * [Fix] Re-use milter_headers settings when doing arc signing * [Fix] Set dmarc force action as least action * [Fix] Switch to GMT * [Fix] allow PKCS7 signatures to be text/plain, too * [Project] Add initial version of the vault management tool * [Project] Add vault support for DKIM and ARC signing * [Project] Implement keys rotation in the vault * [Project] Improve dkim keys generation for vault * [Project] Improve keys creation in rspamadm vault * [Rework] Move lua_worker to a dedicated unit * [WebUI] Add URL fragments (#) support * [WebUI] Fix AJAX request URL Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.9.21.9.2Vsevolod Stakhov2019-04-161-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Allow to load users plugins from plugins.d * [Conf] oversign openpgp and autocrypt headers * [Feature] Add SPF FFI library for Lua * [Feature] Add more verbosity for SPF caching * [Feature] Antivirus: Handle encrypted files specially * [Feature] Clickhouse: Slashing - add new fields to CH * [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table * [Feature] Dkim_signing: Allow to use new options as maps * [Feature] Import fpconv library * [Feature] Lua_maps: Allow static regexp and glob maps * [Feature] Parse ical files * [Feature] Rspamadm: Add dns_tool utility * [Feature] Store SPF records digests * [Feature] Use fpconv girsu2 implementation for printing floats * [Fix] Clickhouse: Use integer seconds when inserting rows * [Fix] Fix floating point printing * [Fix] Fix processing of embedded urls * [Fix] Lua_clickhouse: Fix CH errors processing * [Fix] Make spf digest stable * [Fix] Properly detect encrypted files in zip archives * [Fix] Slashing: Store times in GMT timezone in ClickHouse * [Rules] Add additional conditions to perform BTC checks * [Rules] Fix pay-to-hash addresses validation Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.9.11.9.1Vsevolod Stakhov2019-04-051-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add vendor groups for symbols * [Feature] Add `rspamadm template` command * [Feature] Allow to add messages from settings * [Feature] Allow unconnected DNS servers operations * [Feature] Check limits after being set, migrate to uint64 * [Feature] Greylist: Allow to disable greylisting depending on symbols * [Feature] Improve lua binary strings output * [Feature] Mime_types: Implement user configurable extension filters * [Feature] Mime_types: When no extension defined, detect it by content * [Feature] Preprocess config files using jinja templates * [Feature] Replies: Filter replies sender to limit whitelisting to direct messages * [Feature] Treat all tags with HREF as a potential hyperlinks * [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM * [Fix] Add crash safety for HTTP async routines * [Fix] Another fix for Redis sentinel * [Fix] Clickhouse: Fix table schema upload * [Fix] Core: Fix squeezed dependencies handling for virtual symbols * [Fix] Finally fix default parameters parsing in actions section * [Fix] Fix ES sending logic (restore from coroutines mess) * [Fix] Fix finishing script for clickhouse collection * [Fix] Fix priority for regexp symbols registriation * [Fix] Fix various issues found by PVS Studio * [Fix] Initialize lua debugging earlier * [Fix] Neural: Fix training * [Fix] Rework cached Redis logic to avoid sentinels breaking * [Fix] SURBL: Fix regression in surbl module * [Fix] Fix double signing in the milter * [Project] Add support of HTTP proxy in requests * [Rework] Change lua global variables registration * [Rework] Rework HTML content urls extraction * [Rework] Start rework of aliasing in Rspamd * [WebUI] Combine Scan and Learning into one tab * [WebUI] Fix symbol score input type * [WebUI] Show grayed out pie * [WebUI] Update Throughput summary values dynamically Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.9.01.9.0Vsevolod Stakhov2019-03-121-0/+139
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add missing includes * [Conf] Move to options * [Conf] Rbl: DWL is actually special whitelist * [Conf] Relax some uribl rules * [Conf] Remove abuse.ch * [CritFix] Html: Entities are not valid within tag params values * [Feature] Add `rspamadm mime sign` tool * [Feature] Add configgraph utility * [Feature] Add dedicated ZW spaces detection for URLs * [Feature] Add flag to url object when visible part is url_like * [Feature] Add method task:lookup_words * [Feature] Add pyzor support (by crosenberg) * [Feature] Allow to add upstream watchers to Lua API * [Feature] Allow to set rewrite subject pattern from settings * [Feature] Better escaping of unicode * [Feature] Clickhouse: Allow to store subject in Clickhouse * [Feature] Core: Add QP encoding utility * [Feature] Core: Add libmagic detection for all parts * [Feature] Core: Add support for gzip archives * [Feature] Core: Allow to construct scan tasks from raw data * [Feature] Core: Detect charset in archived files * [Feature] Core: Ignore and mark invisible spaces * [Feature] Core: Normalise zero-width spaces in urls * [Feature] Core: Process data urls for images * [Feature] Core: Relax quoted-printable encoding * [Feature] Core: Support RFC2231 encoding in headers * [Feature] Core: Support telephone URLs * [Feature] Core: allow to emit soft reject on task timeout * [Feature] DCC: Add bulkness and reputation checks to dcc * [Feature] Elastic: Modernize plugin * [Feature] Export visible part of url to lua * [Feature] Fuzzy_storage: add preliminary support of rate limits * [Feature] HTML: Specially treat data urls in HTML * [Feature] Implement event watchers for upstreams * [Feature] Implement includes tracing in Lua * [Feature] Improve dkim part in configwizard * [Feature] Lua_scanners: Add VadeSecure engine support * [Feature] Lua_task: Add flexible method to get specific urls * [Feature] Mime_types: Add MIME_BAD_UNICODE rule * [Feature] Mime_types: Use detected content type as well * [Feature] Plugins: Add preliminary version of the external services plugin * [Feature] Query sentinel on master errors * [Feature] Regexp: Allow local lua functions in Rspamd regexp module * [Feature] Rspamadm: Allow to append footers to plain messages * [Feature] Rspamadm: Allow to rewrite headers in messages * [Feature] Selectors: Add `ipmask` processor * [Feature] Settings: Allow hostname match * [Feature] Settings: Allow local when selecting settings * [Feature] Settings: Allow multiple selectors * [Feature] Settings: Allow to inverse conditions * [Feature] Support User-Agent in HTTP requests * [Feature] Support ed25519 dkim keys generation * [Feature] Try to filter bad unicode types during normalisation * [Feature] external_services - oletools (olefy) support * [Feature] lua_scanners - icap protocol support * [Feature] lua_scanners - spamassassin spam scanner * [Fix] Add filter for absurdic URLs * [Fix] Add some more cases for Received header * [Fix] Allow to disable/enable composite symbols * [Fix] Arc: Use a separated list of headers for arc signing * [Fix] Archive: Final fixes for 7z archives * [Fix] Clickhouse: Fix database usage * [Fix] Controller: Make save stats timer persistent * [Fix] Core: Detect encrypted rarv5 archives * [Fix] Core: Don't detect language twice * [Fix] Core: Fix address rotation bug * [Fix] Core: Fix content calculations for message parts * [Fix] Core: Fix emails comments parsing and other issues * [Fix] Core: Fix etags support * [Fix] Core: Fix headers folding on the last token * [Fix] Core: Fix iso-8859-16 encoding * [Fix] Core: Fix log_urls flag (and encrypted logging) * [Fix] Core: Fix part length when dealing with boundaries * [Fix] Core: Fix parts distance calculations * [Fix] Core: Fix processing of NDNs of certain type * [Fix] Core: Implement logic to find some bad characters in URLs * [Fix] Core: treat nodes with ttl properly in lru cache * [Fix] Fix Content-Type parsing * [Fix] Fix HTTP headers signing case * [Fix] Fix control interface * [Fix] Fix deletion of the duplicate headers * [Fix] Fix emails filtering in emails module * [Fix] Fix greylisting log message and logic * [Fix] Fix issues with storing of the accepted addr in rspamd control * [Fix] Fix maps object update race condition * [Fix] Fix memor leaks and whitespace processing * [Fix] Fix processing of null bytes in headers * [Fix] Fix rcpt_mime and from_mime in user settings * [Fix] Fix rfc2047 decoding for CD headers * [Fix] Fix rfc2231 for Content-Disposition header * [Fix] Fix setting of the subject pattern in config * [Fix] Greylist: fix records checking * [Fix] HTML: Another HTML comments exception fix * [Fix] HTML: Another entities decoding logic fix * [Fix] HTML: Fix HTML comments with many dashes * [Fix] HTML: Fix entities in HTML attributes * [Fix] HTML: Fix some more SGML tags issues * [Fix] Ignore whitespaces at the end of value in DKIM records * [Fix] MID module: Fix DKIM domain matching * [Fix] Milter_headers: Fix remove_upstream_spam_flag and modernise config * [Fix] Mime_parser: Fix issue with parsing of the trailing garbadge * [Fix] Mime_parser: Fix parsing of mime parts without closing boundary * [Fix] Multimap: Fix operating with userdata * [Fix] Process orphaned `symbols` section * [Fix] Rdns: Fix multiple replies in fake replies * [Fix] Rework groups scores definitions * [Fix] Set proper element when reading data from Sentinel * [Fix] Set rspamd user to initialise supplementary groups on reload * [Fix] Settings: Fix selectors usage * [Fix] Sort data received from Sentinel to avoid constant replacing * [Fix] groups.conf - filename typo * [Fix] lua_scanner - oletools typos, logging * [Fix] lua_scanners - actions and symbol_fail * [Fix] lua_scanners - fix luacheck * [Fix] lua_scanners - kaspersky - response with fname * [Fix] lua_scanners - savapi redis prefix * [Fix] tests - antivirus - fprot symbols * [Project] Add concept of flexible actions * [Project] Add heuristical from parser to received parser * [Project] Add new flags to clickhouse, redis and elastic exporters * [Project] Attach new received parser * [Project] Fallback to callbacks from coroutines * [Project] Implement keep-alive support in lua_http * [Project] Lua_udp: Implement fully functional client * [Project] Plug keepalive knobs into http connection handling * [Project] Rspamadm: Add `modify` tool * [Rework] Convert rspamd-server to a shared library * [Rework] Dcc: Rework DCC plugin * [Rework] Enable explicit coroutines symbols * [Rework] Rework telephone urls parsing logic * [Rework] Rewrite RBL module * [Rework] Settings: Rework settings check * [Rework] Slashing: Distinguish lualibdir, pluginsdir and sharedir * [Rework] Unify task_timeout * [Rework] Use VEX instructions in assembly, relocate * [WebUI] Notify user if uploaded data was not learned * [WebUI] Remove redundant condition Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.8.31.8.3Vsevolod Stakhov2018-12-031-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Make flags mutually exclusive for mime parts * [CritFix] Strictly deny unencoded bad utf8 sequences in headers * [Feature] Add Kaspersky antivirus support * [Feature] Add method to get dkim results * [Feature] Add more words regexp classes * [Feature] Allow to choose words format in `rspamadm mime` * [Feature] Allow to get all types of words from Lua * [Feature] Allow to get task flags in C expressions * [Feature] Allow to require encryption when accepting connections * [Feature] Ignore bogus whitespaces in the words * [Feature] Implement more strict configuration tests * [Feature] Improve SPF results in Authentication-Results * [Feature] Support ClickHouse database * [Fix] Add failsafety for utf8 regexps * [Fix] Do not trigger BROKEN_CONTENT_TYPE on innocent text parts * [Fix] Emit error if connection has been terminated with no stop pattern * [Fix] Fix boundaries checks in embedded messages * [Fix] Fix double free * [Fix] Perform policy downgrade on sample out, add tests * [Fix] Properly escape utf8 regexps in hyperscan mode * [Fix] Selectors - attachments args condition * [Fix] Some fixes for raw parts * [Fix] Treat learning errors as non-fatal * [Fix] Use tld when looking for DKIM domains * [Project] Words unicode structure rework * [Project] Add preliminary Redis Sentinel support * [Project] Improve Authentication-Results header * [Project] Rework DKIM checks results * [Project] Use more generalised API to produce meta words Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.8.21.8.2Vsevolod Stakhov2018-11-191-0/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add DWL support in the default configuration * [Conf] Disable rspamd_update by default (again) * [Conf] Fix configuration sample for ratelimit * [CritFix] Disable broken url tags by default * [CritFix] Fix \0 processing when doing RSA sign * [CritFix] Fix adding symbols to their primary groups * [Feature] Add `rspamadm cookie` utility * [Feature] Add specialised functions for generating encrypted cookies * [Feature] Add support of cookies in replies module * [Feature] Add support of words regexps * [Feature] Allow to add 3rd party clang plugins * [Feature] Allow to create lua regexps from glob or plain patterns * [Feature] Allow to set custom limits for upstream lists * [Feature] Detect orphaned parts and attach them to message * [Feature] Filter tokens in bayes * [Feature] Fold b= value when doing arc sealing * [Feature] Ignore cookies in the future and too old in the past * [Feature] Skip stop words in statistics * [Feature] Store stop words and allow to query them * [Feature] Support query arguments in controller's custom commands * [Feature] Tune upstream limits in Rspamd proxy * [Feature] Use different callback symbols for different uribls * [Feature] Write DKIM selector in dkim allow/reject symbols * [Fix] Add obs_fws state support to eoh state machine * [Fix] Add sanity check when applying mime boundaries heuristic * [Fix] Antivirus - virus names with 0 were recognized as tables * [Fix] Disable headernames in bayes temporarily * [Fix] Do not allow syntax errors in include files... * [Fix] Do not allow to merge an object with an array (or vice versa) * [Fix] Don't perform forged recipients check for missing recipients * [Fix] Fix DKIM based RBLs * [Fix] Fix actrie implementation (sync from upstream), fixed OOB read * [Fix] Fix explicit methods call in selectors * [Fix] Fix extraction of additional parts * [Fix] Fix finalization for internal plugins * [Fix] Fix override_defaults function * [Fix] Fix squeezed symbols when using settings * [Fix] Fix urls insertion in Clickhouse module * [Fix] Furhter fixes to ratelimits logic * [Fix] Ignore signatures when looking for boundaries * [Fix] Properly set learned count * [Fix] Really fix ratelimits configuration and work * [Fix] Remove ambigious format flag from printf * [Fix] Restore URLs exporting in ClickHouse plugin * [Fix] Rework bayes calculations... * [Fix] Switch from chi-square to naive for large Fisher value * [Fix] Treat normal password as enable password if there is no enable password * [Fix] Use proper syntax for making DNS requests * [Fix] Various fixes in embedded plugins * [Project] Change fuzzy check selection logic to lua_fuzzy library * [Project] Rework async events and symbols * [Project] Move all metatokens in Bayes to lua_stat from C * [WebUI] Add history rows per page control Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.8.11.8.1Vsevolod Stakhov2018-10-161-0/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix options insertion * [CritFix] Fix words decay one more time (affects long messages) * [CritFix] Increase default words_decay * [CritFix] Plug memory leak in redis pool * [Feature] Add `check_violation` feature to DKIM/ARC signing * [Feature] Add only unique elements to Clickhouse url arrays * [Feature] Allow `g+:` and `g-:` composite atoms * [Feature] Allow dkim domains check in surbl * [Feature] Allow maps with HTTP auth * [Feature] Allow to disable actions by users settings * [Feature] Extend whitelisting options * [Feature] Store url object in images * [Feature] Use verdict instead of the plain action in plugins * [Fix] Allow to call fstring append with NULL string * [Fix] DCC - luacheck * [Fix] Do not load torch on each rspamadm invocation * [Fix] Fix boundaries detection and rework stop words algorithm * [Fix] Fix dependencies for DNS_SIGNED symbol * [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit * [Fix] Fix groups mess * [Fix] Fix groups mess * [Fix] Fix parsing address with comments * [Fix] Fix resolving in DMARC reports * [Fix] Fix various issues with parsing of the received headers * [Fix] Fix watchers issue in lua_tcp when doing no resolving * [Fix] Plug memory leak in language detector (affects reloads) * [Fix] Remove one letter stop words * [Fix] Slashing: backport chunk logic from libucl * [Fix] Stop libevent from using cached time in rspamadm * [Fix] Try to fix watchers chaining * [Fix] Various fixes in redis sync interface * [Fix] ip_score - respect check_authed and check_local settings from config * [Project] Rework passthrough actions * [Project] Clustering module * [Rework] Always create result for a task * [Rework] Completely rewrite DMARC checks logic * [Rework] Rework and fix whitelist plugin * [WebUI] Add symbols sorting buttons * [WebUI] Change symbols order without updating history * [WebUI] Colorize symbols * [WebUI] Do not display password form when secure_ip is set * [WebUI] Fix symbol description tooltips display * [WebUI] History: add sorting by symbol score value Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.8.01.8.0Vsevolod Stakhov2018-09-241-0/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Feature] Add arguments schemas to processors and extractors * [Feature] Add functional selectors library * [Feature] Add generic selector to reputation module * [Feature] Add more ratelimits: by digest, by attachments data, by filenames * [Feature] Add preliminary stop words detection support * [Feature] Add pure Lua debugm function * [Feature] Add schema validation for Redis settings * [Feature] Add selectors combine function * [Feature] Add some recursion protection to lua logger * [Feature] Add support for Lua API tracing * [Feature] Allow to apply schema to arguments * [Feature] Allow to get dkim signing data directly from HTTP headers * [Feature] Allow to reuse existing authentication results * [Feature] Cache selectors results in re runtime * [Feature] Implement new text tokenizer based on libicu * [Feature] Integrate selectors framework to multimap * [Feature] Relax FORGED_RECIPIENTS * [Feature] Support (almost) all html entities * [Feature] Support adding and deletion of recipients in the milter block * [Feature] Support gathering HTTP body from fragments in lua_http * [Feature] Support multi flag in regexp and glob maps * [Feature] Support selectors in ratelimit module * [Feature] Support selectors in settings * [Feature] Use khash in HTML parser * [Feature] Use pure Lua debugm function * [Fix] Add fail-safety for destroying sessions * [Fix] Allow to add result-less fake DNS records * [Fix] Another try to fix race conditions on config unload * [Fix] Call Lua callback on DNS timeouts * [Fix] Deprecate task:inc_dns_req as it is redundant * [Fix] Do not allow events deletions on cleanup * [Fix] Do not try to process skipped messages * [Fix] Fix HTTP requests with no body * [Fix] Fix another cleanup race condition * [Fix] Fix bug in processing of pcre regexps * [Fix] Fix byte array allocation in the pool * [Fix] Fix crashes on task cleanup * [Fix] Fix dynamic buckets in ratelimits * [Fix] Fix endless loop when waiting for Rspamd to stop * [Fix] Fix lua_util.str_split in case of delimiters set * [Fix] Fix more issues with watching of async events * [Fix] Fix stop words detection and loading logic * [Fix] Fix various corner cases for language detection * [Fix] Fix watchers in lua_tcp * [Fix] Fix words decay algorithm * [Fix] Implement watchers replacement to handle nested calls * [Fix] Save faked code into fake dns record * [Fix] Show the proper frame when using lua_util.debugm * [Fix] Use fake dns records in tests * [Fix] Use unicode replacements for HTML entities * [Fix] fixed "cannot find dependency on symbol 1" issue when using replaced symbols in spamassassin rules * [Fix] partition_id is not available in old versions of CH * [Project] Add implicit conversion logic to selectors * [Project] Add initial support for selectors in regexps * [Project] Add method concept * [Project] Further changes in unicode operations * [Project] Implement Clickhouse migrations * [Project] Implement implicit conversions to userdata * [Project] Implement insert method * [Project] Implement selectors registration for regular expressions * [Project] Implement selectors support in re_cache * [Project] Improve language detector: cleanup unused files, categorize * [Project] Migrate CH data to a fat table * [Project] Rework selectors logic * [Project] Start Clickhouse utilities library * [Project] Start unicode rework * [Project] coroutine threaded model for API calls: thread pool * [Rework] Move phishtank to a DNS based service * [Rework] Rework Clickhouse plugin to use the new API * [Rework] Rework language detector * [Rework] Rework utf content processing in text parts * [WebUI] Add progress bar for AJAX requests * [WebUI] Avoid errors table reinitialization * [WebUI] Avoid history table reinitialization * [WebUI] Avoid throughput summary table reinitialization * [WebUI] Destroy summary table on disconnect * [WebUI] Fix "auth" request URL * [WebUI] Fix disabling and hiding controls on page reload * [WebUI] Fix maps loading from neighbours * [WebUI] Fix symbols sorting by score * [WebUI] Fix tables destroying * [WebUI] Fix throughput data consolidation * [WebUI] Fix upload buttons disabling * [WebUI] Notify user on module loading failure * [WebUI] Update FooTable 3.1.4 -> 3.1.6 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.91.7.9Vsevolod Stakhov2018-08-011-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix caseless comparison of equal length strings * [Feature] Add HTTP basic auth support to elastic and clickhouse plugins * [Feature] Add SPF selector to reputation * [Feature] Add support of the fallback backends for HTTP maps * [Feature] Allow to print full mime structure when extracting mime data * [Feature] Allow to split symbols in reputation plugin * [Feature] Check attachments only on AV scanners in attachments_only mode * [Feature] Disable all SSL checks if ssl_no_verify flag is set * [Feature] Implement parsing of scoped IPv6 addresses * [Feature] Improve `rspamc counters` output * [Fix] Add sanity checks when expanding SPF macros * [Fix] Allow to parse SA rules with no spaces around =~ (dirty hack) * [Fix] Avoid one extra byte writing * [Fix] Deal with direct hash table * [Fix] Detect empty text part as text, not HTML * [Fix] Do not reduce map watch timeout for mixed http/file maps * [Fix] Fix HTML part detection heuristic * [Fix] Fix double free in redirectors cleanup * [Fix] Fix legacy history handling in the controller * [Fix] Fix messages insertion * [Fix] Fix sending string method * [Fix] Fix statconver command line arguments * [Fix] Fixed argument checking for being null * [Fix] Fixed issues reported by luacheck * [Fix] Freeze updates queue when do actual storage update * [Fix] HTTP map hash is per-backend and not per-map * [Fix] Plug memory leak in fuzzy updates * [Fix] Prefer 'MTA-Name' when producing authentication results * [Fix] Replace bad unicode sequences instead of stopping on them * [Fix] Set classifier version on learning * [Project] Reworked ratelimits * [Project] Apply topological sorting for symbols in Rspamd * [Project] Remove global contexts from C modules * [Project] Move performance critical hash tables to khash * [WebUI] Avoid unused indexes * [WebUI] Do not execute `on_success` callback * [WebUI] Fix history reset for "All SERVERS" (#2346) * [WebUI] Fix query URL for selected server * [WebUI] Fix symbols display in legacy history, * [WebUI] Hide symbols order selector for legacy history * [WebUI] Refactor query functions into one * [WebUI] Remove previously-attached event handlers * [WebUI] Save symbols to the selected server * [WebUI] Unify arguments of query functions * [WebUI] Use common query functions to get graph data * [WebUI] Use common query functions to save symbols Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.81.7.8Vsevolod Stakhov2018-07-121-0/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Feature] Add more extended statistics about fuzzy updates * [Feature] Add more non-conformant Received headers support * [Feature] Add preliminary function to get fuzzy hashes from text in Lua * [Feature] Allow to configure AV module rejection message * [Feature] Implement fuzzy hashes extraction in mime tool * [Feature] Improve WHITE_ON_WHITE rule * [Feature] Improve integer -> string conversion * [Feature] Reuse maps in multimap module more aggressively * [Fix] Avoid race condition in skip map as pool lifetime is not enough * [Fix] Eliminate all specific C plugins pools * [Fix] Fix DKIM check rule if DNS is unavailable * [Fix] Fix build where ucontext is defined in ucontext.h * [Fix] Fix crash in base url handling * [Fix] Fix descriptors leak in sqlite3 locking code * [Fix] Fix messages quarantine * [Fix] Fix padded numbers printing * [Fix] Fix race condition on maps reinit * [Fix] Fix regexp functions when no data is passed * [Fix] Fix specific urls extraction * [Fix] Fix styles propagation * [Fix] Improve resetting of the limit buckets * [Fix] Initialize sqlite3 properly * [Fix] Work with broken resolvers in resolv.conf * [Project] Implement HTTP maps caching * [Project] Refresh fuzzy hashes when matched * [Project] Add logic to deduplicate fuzzy updates queue * [WebUI] Add missed declarations * [WebUI] Avoid using "undefined" property * [WebUI] Do not accept passwords containing control characters * [WebUI] Do not redeclare variables * [WebUI] Enable strict mode, * [WebUI] Fix variable assignment * [WebUI] Initialize variables at declaration * [WebUI] Remove duplicated path from RequireJS config * [WebUI] Remove unused block * [WebUI] Remove unused variable * [WebUI] Remove unused variables * [WebUI] Use self-explanatory notation * [WebUI] Use type-safe equality operators Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.71.7.7Vsevolod Stakhov2018-07-021-4151/+4194
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Check NM part of pubkey to match it with rotating keypairs * [CritFix] Do not overwrite PID of the main process * [CritFix] Fix maps after reload * [CritFix] Fix maps race conditions on reload * [CritFix] Fix shmem leak in encrypting proxy mode * [Feature] Add a concept of ignored symbols to avoid race conditions * [Feature] Add ability to print bayes tokens in rspamadm mime * [Feature] Add method to get statistical tokens in Lua API * [Feature] Add preliminary mime stat command * [Feature] Add rspamadm mime tool * [Feature] Add urls extraction tool * [Feature] Address ZeroFont exploit * [Feature] Allow rspamadm mime to process multiple files * [Feature] Allow to extract words in `rspamadm mime` * [Feature] Allow to print mime part data * [Feature] Allow to show HTML structure on extraction * [Feature] Distinguish IP failures from connection failures * [Feature] Improve output for mime command * [Feature] Improve styles propagation * [Feature] Main process crash will now cleanup all children * [Feature] Preload file and static maps in main process * [Feature] Print stack trace on crash * [Feature] Process font size in HTML parser * [Feature] Propagate content length of invisible tags * [Feature] Read ordinary file maps in chunks to be more safe on rewrites * [Feature] Support base tag in HTML * [Feature] Support more size suffixes when parsing HTML styles * [Feature] Support opacity style * [Fix] Another fix for nested composites * [Fix] Fill nm id in keypairs cache code * [Fix] Fix colors alpha channel handling * [Fix] Fix destruction logic * [Fix] Fix double free * [Fix] Fix maps preload logic * [Fix] Fix nested composites process * [Fix] Fix proxying of Exim connections * [Fix] Fix reload crash * [Fix] Fix rspamadm -l command * [Fix] Update ed25519 signing schema * [WebUI] Stop using "const" declaration * [WebUI] Update RequireJS to 2.3.5 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.61.7.6Vsevolod Stakhov2018-06-151-0/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix multiple neural networks support * [Feature] Add decryption function to keypair command * [Feature] Add gzip compression for HTTP requests in elastic module * [Feature] Add gzip methods to lua util * [Feature] Add maps based on Top Level Domains * [Feature] Add pubkey checks for dkim_signing * [Feature] Add support of fake DNS records * [Feature] Add tool to encrypt files * [Feature] Allow to add symbols using settings directly * [Feature] Allow to match private and public keys for DKIM signatures * [Feature] Allow to set task flags via settings * [Feature] Allow to specify fake DNS address from the config * [Feature] Implement signatures verification using rspamadm keypair * [Feature] Implement signing using `rspamadm keypair` * [Feature] Improve error reporting for DKIM key access issues * [Feature] Provide $HOSTNAME variable in UCL * [Feature] Rework levenshtein distance computation * [Feature] Split message parsing and processing * [Feature] Support ED25519 DKIM signatures * [Feature] Support encrypted configs in UCL * [Feature] Suppress duplicate warning on very large radix tries * [Feature] Use OSB to combine header names * [Fix] Cleanup maps data on shutdown * [Fix] Fix '~' behaviour in composites * [Fix] Fix HTTP maps updates * [Fix] Fix NIST signatures * [Fix] Fix RFC822 comments when processing a mime address * [Fix] Fix double free * [Fix] Fix dynamic settings application * [Fix] Fix for CommuniGate Pro maillist * [Fix] Fix keypair creation method to actually create keypair... * [Fix] Fix matching patterns with no paths * [Fix] Fix memory leak in parsing comments * [Fix] Fix parsing of urls with numeric password * [Fix] Fix plugins intialisation in configwizard * [Fix] Fix potential crash on reload * [Fix] Fix potential race condition for a finished HTTP connections * [Fix] Fix race-condition leak on processes reload * [Fix] Fix signing in openssl mode * [Fix] Free language detector structures * [Fix] Relax alignment requirements * [Fix] Send DMARC reports compressed * [Fix] Try to fix leak in dmarc module * [Fix] Try to plug memory leak in metric exporter * [Project] Convert rspamadm subcommands to Lua * [WebUI] Display smtp sender/recipient in history * [WebUI] Fix elements disabling in "Symbols" tab * [WebUI] Limit recipients list in history column to 3 * [WebUI] Match envelope and mime addresses following in arbitrary order * [WebUI] Update column header * [WebUI] Wrap addresses in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.51.7.5Vsevolod Stakhov2018-05-181-0/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add MSBL proposed return codes * [Conf] Add additional groups for policies * [CritFix] Do not use volatile Lua strings as UCL keys * [Feature] Add ability to add fuzzy hashes to headers * [Feature] Add function to extract most meaningful urls * [Feature] Add rule to block mixed text and encrypted parts * [Feature] Allow multiple groups for symbols * [Feature] Allow to disable lua squeezing logic * [Feature] Allow to get multipart children in Lua * [Feature] Allow to insert multiple headers from milter headers * [Feature] Allow to print scores in subject and further extensions * [Feature] Be more error-prone in squeezed rules * [Feature] Support multiple return codes in emails module * [Feature] Use EMA for calculating averages * [Feature] Use common jit cache for all regexps * [Feature] support for CommuniGate Pro self-generated messages * [Fix] Allow to have multiple values for headers as arrays * [Fix] Do not open sockets for disabled workers * [Fix] Fix AuthservId * [Fix] Fix base64 folding in Lua API * [Fix] Fix build on non-x86 platforms * [Fix] Fix cached maps logic * [Fix] Fix compatibility with old maps query logic * [Fix] Fix crash if skip_map is used * [Fix] Fix importing static maps from UCL * [Fix] Fix parsing of unix sockets * [Fix] Fix raw_mime regexp on HTML part with no text content * [Fix] Fix tables logging * [Fix] Fix vertical tab handling in libucl * [Fix] Try to fix frequency counters * [Fix] Use better sharding for ip_score * [Fix] Use multiple results from SURBL DNS reply * [Fix] When doing AV scan select a different server for retransmit Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.41.7.4Vsevolod Stakhov2018-05-011-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Major stock config updates: - Workers are now specified in a new format worker "type" { ... } - Enable fuzzy worker to simplify local fuzzy storages configuration - Bind all workers to localhost by default to avoid security flaws * [Conf] Make more sane fuzzy_check default settings * [CritFix] Fix ucl escape for bad symbols * [Feature] Add failure symbol for AV module * [Feature] Add lazy expiration mode for new classifier schema * [Feature] Add preliminary version of maps stats plugin * [Feature] Allow to block fuzzy requests from specific networks * [Feature] Allow to change `expire` of live statistics * [Feature] Distinguish AV failure from clean result * [Feature] Further improvements of language detector by using khash * [Feature] Further optimization of the lang_detection * [Feature] Implement cluster-aware bayes expiry * [Feature] Implement exclude patterns in rspamc * [Feature] Implement glob maps in addition to regexp maps * [Feature] Implement map statistics function for lua API * [Feature] Implement stop symbols for Clickhouse collection * [Feature] Support recipients separated by commas * [Feature] Try harder to upload scripts to the Redis server * [Feature] Upgrade t1ha distribution * [Feature] use_domain_sign_inbound * [Feature] Use scores from maps if `symbols_set` is not defined * [Fix] Add resolving version of radix map helper * [Fix] Check URL before adding implicit prefix * [Fix] Do not check pid/state when using PRNG * [Fix] Fix CentOS logrotate script for systemd * [Fix] Fix slash + dot in urls * [Fix] Fix systemd version of the logrotate script * [Fix] Propagate key when import implicit array from Lua * [Fix] Strip spaces from map keys and values * [Fix] Try to fix a specific case when processing milter protocol * [Fix] Try to fix crash when a tcp connection cannot be set * [Fix] Typo use_domain_local --> use_domain_sign_local * [Fix] Various fixes to once_received module * [Project] Store hits counters for map elements Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.31.7.3Vsevolod Stakhov2018-04-101-0/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Plug bad memory leak in protocol reply * [Feature] Add avx2 codec for base64 * [Feature] Add method to receive all URL flags from Lua API * [Feature] Allow to fold headers on stop characters * [Feature] Allow to set lua_cpath from options * [Feature] Allow to specify custom rejection message in milter * [Feature] Deal with unnormalised Unicode obfuscation * [Feature] Do not detect language twice for relative parts * [Feature] Implement oversigning feature * [Feature] Implement silent logging level to minimize noise in logs * [Feature] Improve URL_IN_SUBJECT rule * [Feature] Use hashing to reduce redis attack surface * [Fix] Add oversigning for the most important headers * [Fix] add 'rewrite subject' to History dropdown * [Fix] Another fix in folding algorithm * [Fix] Do not call multimap addr for parts of addr if filter is presented * [Fix] Do not clean hostname on generic reset * [Fix] Do not create pid file in no-fork mode * [Fix] Fix fold_after case to preserve multiple spaces * [Fix] Fix folding and folding tests * [Fix] Fix hostname usage in milter mode * [Fix] Fix lua RSA verify and its tests * [Fix] Fix metadata exporter send_mail backend (#2124) * [Fix] Fix processing of '\v' in libucl * [Fix] Fix shemaless URLs detection * [Fix] Fix support of multiple headers in sign_header * [Fix] Fix usage of util.parse_mail_address * [Fix] Fix weights of dynamic squeezed rules * [Fix] Leak from bucket before checking the burst * [Fix] Stop using own localtime as DST could be messy in many cases * [Fix] Treat unnormalised URLs as obscured * [Rework] Restore leaky bucket model in ratelimit plugin * [WebUI] Add messages total to throughput summary * [WebUI] Add symbols order selector to history * [WebUI] Config: Load list on demand * [WebUI] Fix modalBody for maps that appear more than once * [WebUI] History: Fix Tooltips on paging, filtering and sorting * [WebUI] Remove a previously-attached event handler * [WebUI] Update D3 to v5.0.0 and jQuery to v3.3.1 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.21.7.2Vsevolod Stakhov2018-03-231-0/+15
| | | | | | | | | | | | | | | | | | * [Feature] Store emails in Clickhouse * [Feature] Support single quotes in config * [Feature] Use templates when publishing CH schema * [Feature] Improve Docker image * [Fix] Add rounding when printing a lot of FP variables * [Fix] Allow to disable certain actions by assigning null to them * [Fix] Disable results caching * [Fix] Fix disabling of squeezed symbols * [Fix] Fix scan time set * [Fix] Rework logic of actions setting * [Fix] Try to fix various Lua stack issues * [WebUI] Add link tag for favicon.ico * [WebUI] Display hostname:port/path in the page title Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.11.7.1Vsevolod Stakhov2018-03-201-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix lowercase comparison * [CritFix] Timezone defines seconds WEST UTC not East * [Feature] Add filename to log format * [Feature] Add lua rules squeezing * [Feature] Add related symbols analysis to rspamd_stats * [Feature] Remove upstream `X-Spam: Yes` header by default * [Feature] rspamd_stats: Output progress info on STDERR * [Feature] Whitelist for emails module * [Fix] Do not allow dependencies on self * [Fix] Do not cache metric result * [Fix] Do not trust all issuers as a client certificate * [Fix] Fix dependencies in lua squeeze * [Fix] Fix enabling/disabling squeezed rules * [Fix] Fix enabling/disabling symbols * [Fix] Fix external dependencies * [Fix] Fix processing of a single compressed file * [Fix] Fix some typos * [Fix] Fix various modules in case of empty message * [Fix] Handle callbacks that returns table of options * [Fix] Improve cached action interaction * [Fix] Make dynamic conf more NaN aware * [Fix] Never hide actions from WebUI `configuration` tab * [Project] Implementation of Lua rules squeezing Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.7.01.7.0Vsevolod Stakhov2018-03-121-0/+427
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Add bayes_expiry as explicit module * [Conf] Adjust names and weights for neural network plugin * [Conf] Change updates url * [Conf] Default statistics is stored in Redis now * [Conf] Disable fann_redis module by default * [Conf] Fix default elastic configuration * [Conf] Fix double quote position * [Conf] Massive config rework for new structure of symbols and scores * [Conf] Rename Rambler BLs as they are now Rspamd's ones * [Conf] Use dedicated rspamd.com subdomains * [Conf] Use more data from rspamd.com fuzzy storage * [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Another portion of tokenization fixes * [CritFix] Do not send reject messages after set reply * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [CritFix] Fix memory leak in spf caching logic * [CritFix] Fix milter commands pipelining * [CritFix] Fix newlines detection * [CritFix] Fix semicolons parsing in the content type * [CritFix] Plug memory leak in zstd protocol compression * [Feature] Add ability to match score in force_actions module * [Feature] Add aes-rng PRF to libottery * [Feature] Add 'composites' debug module * [Feature] Add concept of experimental modules * [Feature] Add DKIM trace symbol * [Feature] Add EBL to the default config * [Feature] Add expected ip check for emails plugin * [Feature] Add framework to manage Redis scripts * [Feature] Add framing for the new reputation generic plugin * [Feature] Add function to show plugins stat * [Feature] Add gzip compression support for clickhouse module * [Feature] Add gzip compression support for rspamd controller * [Feature] Add gzip support when sending lua http requests * [Feature] Add json output for rspamd_stats * [Feature] Add method to do a synchronous Redis connection * [Feature] Add method to get all content-type attributes in Lua * [Feature] Add `-m` flag to configdump to show modules states * [Feature] Add mime types to extensions map * [Feature] Add more features to rescore utility * [Feature] Add more gtube like patterns to test other spam actions * [Feature] Add more metafunctions, improve logging * [Feature] Add more text attributes * [Feature] Add new configwizard command to rspamadm * [Feature] Add new tooling for stats conversation * [Feature] Add old groups migration tool * [Feature] Add plugins state variable * [Feature] Add preliminary ecdsa keys support in DKIM * [Feature] Add preliminary support of idempotent symbols * [Feature] Add Redis server wizard * [Feature] Add routine to convert old style stats to a new one * [Feature] Add some sanity checks for actions and controller * [Feature] Add statistic convertation module to configwizard * [Feature] Add sugestions logic to mempool allocator * [Feature] Add support of config transform in Lua * [Feature] Add timeout to rspamc when doing corpus test * [Feature] Add tooling to convert bayes schemas * [Feature] Add torch conditional to configuration * [Feature] Add torch-decisiontree package * [Feature] Add torch-optim contrib package * [Feature] Add TTL autodetection * [Feature] Add urls reputation to the reputation framework * [Feature] Allow floating and negative values in expressions limits * [Feature] Allow multiple CTs in full extensions map * [Feature] Allow multiple fann rules * [Feature] Allow randomly select User-Agent from a list * [Feature] Allow rspamadm commands to export methods in Lua * [Feature] Allow rule specific min_bytes in fuzzy check * [Feature] Allow to adjust symbols scores from Lua * [Feature] Allow to attach stat signature to messages * [Feature] Allow to change SMTP from via milter headers * [Feature] Allow to configure monitored * [Feature] Allow to create directories in Lua API * [Feature] Allow to disable torch and skip train samples for ANN * [Feature] Allow to discard messages dynamically * [Feature] Allow to enable/disable languages from the detector * [Feature] Allow to generate DKIM keys from rspamadm API * [Feature] Allow to get CPU flags from Lua * [Feature] Allow to have high precision timestamps in logs * [Feature] Allow to insert headers into specific position * [Feature] Allow to limit redirector requests per task * [Feature] Allow to load and use dynamic ANNs with torch * [Feature] Allow to quarantine rejected messages using milter interface * [Feature] Allow to receive signing keys from mempool vars * [Feature] Allow to reserve elements in libucl * [Feature] Allow to reuse signal handlers chains * [Feature] Allow to set custom mempool variables from settings * [Feature] Allow to set headers from settings * [Feature] Allow to set Settings-Id for all connections * [Feature] Allow to skip real action and add a header instead * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Allow to spawn asynchronous processes from Lua * [Feature] Allow to specify number of threads for ANN learning * [Feature] Allow to use global lua maps in settings * [Feature] Allow to use postfilters in composites * [Feature] Allow to verify signatures from HTTP headers in maps * [Feature] Antivirus: ordered pattern matches * [Feature] Authentication-Results: support hiding usernames * [Feature] Automatically create tables in clickhouse * [Feature] Catch next-to-last bad extension * [Feature] Check cached maps more frequently * [Feature] Check groups sanity * [Feature] Deal with obscured URLs with @ symbols * [Feature] Enhance task:store_in_file method * [Feature] Export password encryption routines to Redis * [Feature] Filter nan and inf when adding scores * [Feature] Finalize 7zip files support * [Feature] Further improvements in language detection * [Feature] Further improvements in language detection algorithm * [Feature] Generic key name expansion for Redis keys * [Feature] Hash whitelist for fuzzy_check * [Feature] Implement bayes signatures storage * [Feature] Implement buckets for Redis backend * [Feature] Implement DKIM reputation adjustments * [Feature] Implement forked workers children monitoring * [Feature] Implement headers flags in mime parser * [Feature] Implement l1/l2 regularization against the current weights * [Feature] Implement manual ANN train mode * [Feature] Implement per-user ANN support * [Feature] Implement torch based ANN learning * [Feature] Implement upstreams logic for clickhouse exporter * [Feature] Import torch to Rspamd... * [Feature] Improve allocation policy when interacting with Lua * [Feature] Improve Lua/C interaction in history_redis * [Feature] Improve multiple fuzzy results combining * [Feature] Improve parsing of DKIM keys: parse algorithm * [Feature] Improve subprocesses termination handle * [Feature] Improve symbol type parsing in Lua API * [Feature] Metadata Exporter: e-Mail Alerts: support multiple recipients; alerting senders/recipients/users (#1600) * [Feature] Milter headers: support adding/removing arbitrary headers from config * [Feature] More metatokens * [Feature] Multimap: checking of symbol options * [Feature] Multimap: template URL filter * [Feature] New bayes expiry plugin * [Feature] Periodically save rspamd stats to disk * [Feature] Preliminary import of the elasticsearch module * [Feature] Ratelimit: allow full addresses in whitelisted_rcpts * [Feature] Ratelimit: support fetching limits from Redis * [Feature] RBL: received: filtering by position & flags * [Feature] Read global maps for lua * [Feature] Redis settings: support checking multiple keys * [Feature] Rework fann plugin to be a normal post-filter * [Feature] Rework logging configuration for rspamadm case * [Feature] Rework short hashes generation to avoid FP * [Feature] Save real ucl types when exporting to Lua * [Feature] Set TCP_NODELAY for milter sockets * [Feature] Setup DKIM signing from configwizard * [Feature] Skip certain symbols from ANN classify * [Feature] Store plugins state * [Feature] Support etag for HTTP maps * [Feature] Support Expires header when using HTTP maps * [Feature] Support sending given header multiple times in lua_http * [Feature] Support sha512 in DKIM signatures * [Feature] Try to detect HTML messages better * [Feature] Use array instead of queue to reduce memory fragmentation * [Feature] Use controller port by default when connecting to local IP * [Feature] Use rdtsc where possible * [Fix] Actively load skip hashes map in fuzzy storage * [Fix] Add another workaround to display history properly * [Fix] Add definition for old glib compatibility method * [Fix] Add missing rspamadm control options to help * [Fix] Add workaround for IPv6 in sendmail * [Fix] Add workaround for system with non-XSI compatible tzset * [Fix] Allow oversigning in DKIM signatures * [Fix] Allow to check negative scores in force_actions * [Fix] Allow to have negative actions limits * [Fix] Allow to set any layers number for fann rules * [Fix] Another fix for rdtcs * [Fix] Another fix to lua xmlrpc * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Another try to fix threading in torch * [Fix] Apply language detection when adding fuzzy hashes * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Authentication Results: Fix SPF smtp.mail_from * [Fix] Auth-Results: Multiple DKIM signatures * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Check for magic when checking for an archive * [Fix] Cleanup mess with groups * [Fix] Clickhouse: Insertion in the symbols table * [Fix] Crash in URL processing * [Fix] Deal with another case when processing exceptions * [Fix] Deal with deeply nested messages more aggressively * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with non-key arguments in lua_redis.exec_script * [Fix] Deal with unknown weight * [Fix] Deal with URLs with no slashes after protocol * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] Deal with zero scores symbols * [Fix] Default monitoring domain for surbl plugin * [Fix] Delay upstream re-resolving when one upstream is defined * [Fix] Detection of maillist optimized and fixed * [Fix] DKIM signing: allow for auth_only to be false * [Fix] DMARC: require report_settings for sending reports only * [Fix] Do not allow garbadge when checking url domain * [Fix] Do not cache SPF records with PTR elements * [Fix] Do not constantly re-resolve failed upstreams with a single element * [Fix] Do not crash if no words defined * [Fix] Do not crash on empty subtype * [Fix] Do not expose spamtrap messages to SMTP reply * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not ignore short words * [Fix] Do not include idempotent/nostat symbols to checksum * [Fix] Do not override groups when converting metrics * [Fix] Do not override unix socket group when group comes before owner * [Fix] Do not skip the last character * [Fix] Do not spawn too many workers by default * [Fix] Do not stop monitored on dns errors * [Fix] Do not stop parsing headers on bad IP header * [Fix] Do not strip last character in the last word * [Fix] Do not treat script content as text * [Fix] Do not try to connect to non-supported addresses * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Erase unknown HTML entities * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix a lot of FP in chartable in mixed languages * [Fix] Fix ANN checks * [Fix] Fix ANN loading logic * [Fix] Fix another tokenization issue * [Fix] Fix autolearn parameters reading * [Fix] Fix bad archive characters stripping * [Fix] Fix bad extension check * [Fix] Fix bayes schema conversion * [Fix] Fix blacklists and DMARC in whitelist * [Fix] Fix brain-damaged torch build system * [Fix] Fix build on FreeBSD * [Fix] Fix clickhouse exporter * [Fix] Fix clickhouse schema * [Fix] Fix comparision * [Fix] Fix composites processing * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix couple of warnings * [Fix] Fix crashes in the rspamd_control path * [Fix] Fix deletion from hash * [Fix] Fix DKIM forgeries via multiple headers * [Fix] FIx dynamic conf plugin * [Fix] Fix emails detection * [Fix] Fix empty headers simple canonicalization * [Fix] Fix empty threshold check in greylisting module * [Fix] Fix encrypted legacy reply in fuzzy storage * [Fix] Fix enormous scores for R_WHITE_ON_WHITE * [Fix] Fix exceptions list in surbl * [Fix] Fix *_EXCESS_BASE64 rules * [Fix] Fix expire rounding * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix format strings * [Fix] Fix get_content method * [Fix] Fix groups override when defining symbols * [Fix] Fix learned count in new schema * [Fix] Fix learn errors propagation * [Fix] Fix loading of per-user redis backend for statistics * [Fix] Fix logging buffer corruption in case of repeated messages * [Fix] Fix lua cached elements invalidation * [Fix] Fix merging of the implicit arrays * [Fix] Fix mime_types scoring * [Fix] Fix multiple headers in DKIM headers list * [Fix] Fix null callee case in clang plugin * [Fix] Fix obscured url in format user@@example.com * [Fix] Fix parsing of the per-user script * [Fix] Fix priorities in rspamd_update, disable rules execution * [Fix] Fix processing of closed tags * [Fix] Fix processing of idempotent rules when autolearn fails * [Fix] Fix processing of multipart parts with no headers * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix PTR processing in SPF * [Fix] Fix pushing country to clickhouse asn table * [Fix] Fix random forests module * [Fix] Fix real IP parsing for some strange Exim received * [Fix] Fix Redis timeout setup * [Fix] Fix reload crash when hyperscan is enabled * [Fix] Fix reusing of redis connection after exec * [Fix] Fix sanity checks on macro value * [Fix] Fix setting of path and cpath for Lua * [Fix] Fix setting of signals when spawning a thread * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix ticks processing * [Fix] Fix upstream addrs updating * [Fix] Fix urls/emails distinguishing found in queries * [Fix] Fix user settings check * [Fix] Fix variable increment * [Fix] Fix various issues in stat_convert * [Fix] F-PROT Antivirus infection string for all known occurences * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Further fixes around floating point expressions * [Fix] Further fixes to ANN module * [Fix] Further fixes to rescore tool * [Fix] Further fixes to support ES 6 * [Fix] Further tokenization fixes * [Fix] Greylisting set phase is not idempotent * [Fix] Handle proxy copy errors * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Kill spawned processes on termination * [Fix] Load skip map from all processes as shared cache is unavailable * [Fix] Lowercase HTTP headers to make them searchable from Lua * [Fix] Lowercase words * [Fix] Lua_http: freeing * [Fix] Lua: lpeg to be loaded with rspamd_lua_add_preload, to avoid "rspamd_config_read: rcl parse error: cannot init lua file […] module 'lpeg' not found" * [Fix] Map absence is not an error * [Fix] Metadata exporter: check IP sanity * [Fix] Milter headers: custom headers: removing headers * [Fix] Milter headers: skip_local / skip_authenticated settings * [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first * [Fix] mime_types: fix next-to-last extension length check * [Fix] More hacks to deal with old configs * [Fix] Move composites second pass to the dedicated stage * [Fix] Multimap: received: filtering of artificial header * [Fix] Multiple fixes in torch based ANN plugins * [Fix] Once more (#1879) fix bad extension check * [Fix] Optimize rspamd_fstring_t reallocations * [Fix] options.local_networks setting * [Fix] Parse HREF urls without explicit prefix * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Plug memory leak * [Fix] Plug memory leak in lua_tcp * [Fix] Plug memory leak when setting email addresses from Lua * [Fix] Propagate learn/stat errors more precisely * [Fix] Ratelimit: fix whitelisted_rcpts matching * [Fix] Ratelimit: lowercase email addresses * [Fix] RBL: received: deal with missing data (#1965) * [Fix] Rebalance and slightly rework MX check plugin * [Fix] Redis key expansion: EVAL: deal with strings * [Fix] Redis script loading in DMARC; URL tags; URL reputation * [Fix] Reject invalid bh for DKIM signatures earlier * [Fix] Relax pem signature detection * [Fix] Relax unicode properties requirements for chartable module * [Fix] Remove extra noise from dkim and arc signing * [Fix] Remove hop-by-hop headers in proxy * [Fix] Remove incorrect method `task:set_metric_subject` * [Fix] Replace space like characters in headers with plain space * [Fix] Restore old style ratelimits support * [Fix] Rework elasticsearch plugin * [Fix] Rewriting subjects via force actions module * [Fix] RPM postinstall * [Fix] Sanitize IP in history redis * [Fix] Select the correct signature when doing simple canon * [Fix] Set CLOEXEC flag on files opened * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Settings: header: deal with multiple settings (#1988) * [Fix] Skip checks if both extensions are not bad * [Fix] Skip nostat tokens when get number of tokens * [Fix] Some more fixes towards emails detection * [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't match * [Fix] Stop using of g_slice... * [Fix] Switch rspamadm logging to message level * [Fix] Symbol 'FANNR_SPAM' has its score defined.. * [Fix] Table parameter for rspamd_config:add_doc() * [Fix] Treat 'rewrite subject' as spam action * [Fix] Try harder in passing IPv6 addresses * [Fix] Try harder to find rfc822 notifications * [Fix] Try harder to find urls * [Fix] Use decoded values when parsing mime addresses * [Fix] Use full URL when making an HTTP request * [Fix] Use greylisting threshold in greylisting module * [Fix] Use n_words attribute from ngramms * [Fix] Use raw urls when sending requests to redirector * [Fix] Use the right boolean operator on error check * [Fix] Use weight from map for fuzzy scoring * [Fix] Various fixes to elastic plugin * [Fix] Various fixes to fann_redis instantiation * [Fix] Various improvements in language detection * [Fix] Virus infection string for F-PROT Antivirus * [Fix] Virus infetction string for F-PROT Antivirus * [Fix] WebUI: use relative path for savemap (#1943) * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [Fix] Write configuration changes as UCL config * [Project] Add detection logic for words * [Project] Add fast debug logging infrastructure * [Project] Add more flags to languages * [Project] Add n-gramms data files * [Project] Add ngramms frequencies detector * [Project] Add random words selection logic * [Project] Add unigramms to language detection as well * [Project] Convert all C modules to fast debug infrastructure * [Project] Detect some languages based on unicode script * [Project] Enable fast debug lookup for some modules * [Project] Enable language detector init in scanner workers * [Project] Further improvements to language detector * [Project] Implement logic of ngramms application * [Project] Improve weighting in lang_detection * [Project] Initialize language detector * [Project] Preliminary version of ngramms based language detector * [Project] Preliminary version of the new stat_convert * [Project] Remove old language detector * [Project] Rework language detection ngramms structure * [Project] Start language detection project * [Project] Start rework of language detection to improve quality * [Project] Use fast debug logging check * [Rework] Add frame for new reputation based IP score module * [Rework] Continue stat_convert rework task * [Rework] Implement new version of fuzzy replies * [Rework] Improve readability of xmlrpc API * [Rework] Kill metrics!11 * [Rework] Ratelimit module * [Rework] Rename fann_redis to neural plugin * [Rework] Reorganize mime_types module * [Rework] Rework rescore utility * [Rework] Rewrite model and learning logic for rescore * [Rework] Run post-loads when all initialization is completed * [Rework] Simplify lua path initialization * [Rework] Start major stat_convert rework * [Rework] Start mempool fragmentation reduce project * [Rework] Start moving of fann redis to torch * [Rework] Stop embedding rspamadm scripts into C * [Rework] Use floating point arithmetics in Rspamd expressions * [Rework] Use frequencies distribution in language detector * [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages * [WebUI] Compact graph selectors * [WebUI] Escape strings inside HTML in history * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Migrate widgets to D3 v4 * [WebUI] Restore passwordless login support (#2003) * [WebUI] Show symbol descriptions as tooltips in history * [WebUI] Stop using commas in pie chart tooltips * [WebUI] Update D3 and jQuery * [WebUI] Update D3Evolution 1.0.0 -> 1.1.0 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.6.6Vsevolod Stakhov2018-03-121-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [Feature] Allow to insert headers into specific position * [Feature] Allow to receive signing keys from mempool vars * [Feature] Authentication-Results: support hiding usernames * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] DKIM signing: allow for auth_only to be false * [Fix] Do not crash on empty subtype * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not skip the last character * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix bad archive characters stripping * [Fix] Fix comparision * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix empty headers simple canonicalization * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix parsing of the per-user script * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix Redis timeout setup * [Fix] Fix sanity checks on macro value * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix urls/emails distinguishing found in queries * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Metadata exporter: check IP sanity * [Fix] Multimap: received: filtering of artificial header * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Remove hop-by-hop headers in proxy * [Fix] Sanitize IP in history redis * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Try harder in passing IPv6 addresses * [Fix] WebUI: use relative path for savemap (#1943) * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Restore passwordless login support (#2003) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Typo: comparisionViktor Szépe2017-11-141-1/+1
|
* [Minor] Sync ChangeLogVsevolod Stakhov2017-11-051-0/+70
|
* Release 1.6.3Vsevolod Stakhov2017-09-031-0/+19
| | | | | | | | | | | | | | | | | | | | | | * [CritFix] Fix semicolons parsing in the content type * [Feature] Add EBL to the default config * [Feature] Allow to configure monitored * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Multimap: checking of symbol options * [Feature] Redis settings: support checking multiple keys * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Fix get_content method * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Lua_http: freeing * [Fix] Milter headers: custom headers: removing headers * [Fix] Parse HREF urls without explicit prefix * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [WebUI] Escape strings inside HTML in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
* Release 1.6.21.6.2Vsevolod Stakhov2017-07-081-0/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [Conf] Remove Rambler email bl for now * [Conf] Switch RAMBLER_URIBL to a locally managed source * [CritFix] Switch from ragel to C for Content-Type parsing * [Feature] Add `-e` option for lua_repl * [Feature] Add per-domain emails normalisation rules * [Feature] Add sessions cache to debug dangling sessions * [Feature] Add short_text_direct_hash for fuzzy check module * [Feature] Add text_part:get_stats function * [Feature] Allow to add custom processing script for surbl * [Feature] Allow to check reply-to email * [Feature] Allow to customize spam header, remove existing spam headers * [Feature] Allow to disable specific workers in the config * [Feature] Allow to discard messages instead of rejection * [Feature] Allow to specify custom delimiter in emails plugin * [Feature] Allow to specify custom User-Agent for rspamc * [Feature] Allow to store symbols data in Clickhouse * [Feature] Allow to use HTTPS when connecting to Clickhouse * [Feature] Enable sessions cache tracking for milter connections * [Feature] Implement per-line mode in lua_repl (like `perl -p`) * [Feature] Implement rdns-curve plugin based on rspamd cryptobox * [Feature] Improve maps cached data lifetime * [Feature] Improve maps checking frequency * [Feature] Improve monitored timeouts logic * [Feature] milter_headers: add `extended_headers_rcpt` option * [Feature] Milter headers: Add X-Spam-Flag to rmilter-compatibility headers * [Feature] Milter headers: remove-header routine * [Feature] Multimap: received filters for extracting TLDs from hostnames * [Feature] Normalize email aliases in emails module * [Feature] Re-add rambler email bl (as hashed list) * [Feature] Reload file maps more frequently * [Feature] Rework newlines strip parser one more time * [Feature] Skip updates for messages scanned via controller * [Feature] Split long DKIM public keys * [Feature] Store more data when stripping newlines * [Feature] Support SPF macros transformations * [Feature] Support suppressing DMARC reports for some domains * [Fix] Add missing `break` statement * [Fix] Allow modifiers in SPF macros * [Fix] DKIM sign tools: edge-cases around use_esld * [Fix] Do not cache SPF records with macros * [Fix] Do not overwrite score when setting pre-action * [Fix] Fix comparision logic * [Fix] Fix DKIM base64 folding for milter flagged messages * [Fix] Fix emails module configuration * [Fix] Fix folding for arc headers when milter interface is used * [Fix] Fix gmail dots removal * [Fix] Fix rspamc detection in greylist module * [Fix] Fix some more issues with HTTP maps * [Fix] Milter sessions can live forever * [Fix] Normalize fuzzy probability better * [Fix] Plug memory leak * [Fix] RBL: Fixed hashed email address lookups * [Fix] Try to deal with brain-damaged milter behaviour * [Fix] Use `\n` to fold headers for milter * [Rework] Allow to use custom callback for monitored checks * [Rework] Further steps towards one process monitoring * [Rework] Send health checks from a single worker * [WebUI] Round-up throughput summary values Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-22 19:43:05 +0000