Tree:
5de6247460
master
rspamd-3.8
vstakhov-bayes-experiments
vstakhov-fuzzy-tcp
vstakhov-slow-timer
external-maps
libev-migration
log_json
mime-rework
rdns-tcp-rework
rework-symcache
rspamd-0.5
rspamd-0.6
rspamd-0.7
rspamd-0.8
rspamd-0.9
rspamd-1.0
rspamd-1.1
rspamd-1.2
rspamd-1.3
rspamd-1.4
rspamd-1.5
rspamd-1.6
rspamd-1.9
rspamd-3.7
torch-removal
vstakhov-fasttext-langdet
vstakhov-fuzzy-cxx
vstakhov-known-senders
vstakhov-stringzilla
3.8.4
3.8.3
3.8.2
3.8.1
3.8.0
3.7.5
3.7.4
3.7.3
3.7.2
3.7.1
3.6
3.5
3.4
3.3
3.2
3.1
3.0
2.7
2.6
2.5
2.4
2.3
2.2
2.1
2.0
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.3
1.8.2
1.8.1
1.8.0
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.9
1.5.8
1.5.7
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.11
1.0.10
1.0.9
1.0.8
1.0.7
1.0.6
1.0.5
1.0.4
1.0.3
1.0.2
1.0.1
1.0.0
0.9.10
0.9.9
0.9.8
0.9.7
0.9.6
0.9.5
0.9.4
0.9.3
0.9.2
0.9.1
0.9.0
0.8.3
0.8.2
0.8.1
0.8.0
0.7.6
0.7.5
0.7.4
0.7.3
0.7.2
0.7.1
0.7.0
0.6.10
0.6.9
0.6.8
0.6.7
0.6.6
0.6.5
0.6.4
0.6.3
0.6.2
0.6.1
0.6.0
0.5.6
0.5.5
0.5.4
0.5.3
0.5.2
0.5.1
0.5.0
0.4.7
0.4.6
0.4.5
0.4.4
0.4.3
0.4.2
0.4.1
0.4.0
0.3.14
0.3.13
0.3.12
0.3.11
0.3.10
0.3.9
0.3.8
0.3.7
0.3.3
0.3.2
0.3.1
0.3.0
0.2.7
123 Commits (5de62474600c4e0722241f3c31e81633c08aa85c)
123 Commits (5de62474600c4e0722241f3c31e81633c08aa85c)
| Author | SHA1 | Message | Date |
|---|---|---|---|
 Vsevolod Stakhov
|
8c8f53d783
|
[Minor] Update version to 3.9.0 | 3 months ago |
|
Vsevolod Stakhov
|
b8a2d79eea
|
Release 3.8.1
* [Fix] Fix headers insertion in the ordered list * [Fix] Fix learn error propagation * [Fix] Fix learning with long prefixes * [Fix] Fix potential double free by expclicitly zeroing pointers * [Fix] Fix storing of the bayes tokens * [Fix] Use short comments rather than long one |
4 months ago |
|
Vsevolod Stakhov
|
28391190c3
|
Release 3.8.0
* [Project] Rspamadm fuzzyping command * [Project] Use Redis scripts for Bayes statistics and cache * [Project] Support JSON logging * [CritFix] - Fix reported length of logging structure * [Feature] Allow to set `max_users` for Redis bayes backend * [Feature] Escape JSON when needed * [Feature] Proxy: Allow `encrypted_only` option * [Feature] Reiterate on gtube patterns * [Feature] Support ping command in fuzzy storage * [Feature] Support suppressing DMARC reporting for particular recipients * [Feature] rbl: support disabling or replacing url_whitelist per RBL * [Fix] Another try to fix setproctitle * [Fix] Cleanup session on exit * [Fix] Do not cleanup hyperscan files unless new ones are loaded * [Fix] Fix bad access when printing pending events on failure path * [Fix] Fix issues with the raw header on header insertion/modification * [Fix] Fix overflow in fuzzy_stats command * [Fix] Fix some corner cases when parsing maps with no newlines * [Fix] Fix various issues with canonicalisation of the paths * [Fix] Fix wrong raw_len usage * [Fix] Make words selection random deterministic upon content * [Fix] Properly set config field when creating tasks from Lua * [Fix] Really fix the language detector statistical heuristic * [Fix] Set loaded variable explicitly * [Fix] dkim_signing: siging_table: lowercase before lookup * [Fix] known_senders: fix config handling * [Fix] rbl: fix `exclude_local` |
5 months ago |
|
Vsevolod Stakhov
|
dfd698fe1c
|
Release 3.7.4
* [Enhancement] Add composite rule for suspicious URLs in suspicious messages * [Fix] Another try to fix setproctitle * [Fix] Do not cleanup hyperscan files unless new ones are loaded * [Fix] Fix various issues with canonicalisation of the paths * [Fix] Properly set config field when creating tasks from Lua * [Fix] Set loaded variable explicitly * [Fix] known_senders: fix config handling * [Fix] rbl: fix `exclude_local` * [Minor] Add missing <algorithm> include for gcc 14 * [Minor] Add rule for messages missing both X-Mailer and User-Agent header * [Minor] Bundle fasttext as linux distributives are just useless * [Minor] Enable fasttext on RPM based linux * [Minor] Forgot it in one more place * [Minor] Improve FREEMAIL_AFF capture rates * [Minor] Limit fasttext to amd64 only * [Minor] Properly search for fasttext include * [Minor] Treat *.zpaq attachments as archives and harmful * [Minor] Update to 3.7.4 * [Minor] force_actions: set a group for symbols * [WebUI] Update map editor |
7 months ago |
|
Vsevolod Stakhov
|
23ed02bcac
|
Release 3.7.3
* [Fix] Emergency fix for the hyperscan path error |
7 months ago |
|
Vsevolod Stakhov
|
b5bbeaa5d9
|
Release 3.7.2
* [Feature] rbl: support checking returncodes by CIDR * [Feature] rbl: support checking returncodes by regex * [Feature] rbl: support globbed return codes * [Fix] DMARC reporting: fix reporting for subdomains * [Fix] Deal with fmtlib exceptions properly * [Fix] backport fix for dlfcn.h from backward-cpp * [Rules] Blank spam detection |
7 months ago |
|
Vsevolod Stakhov
|
af3e90f67c
|
Release 3.7.1
* [CritFix] Fix leak in `gzip` function * [Feature] Add ICAP Content-Type and Filename * [Feature] Add `logging`->`task_max_elts` option * [Feature] Add utility to split string like stuff for C++ code * [Feature] Allow to set HTTP auth parameters for the maps * [Feature] Check for plugin configuration errors on `configtest` * [Feature] `known_senders` plugin * [Feature] Use backward-cpp instead of manual libunwind stuff * [Feature] rbl: support checking numeric URLs in isolation * [Fix] CMakeLists.txt remove whitespace added by linter as it makes tests fail * [Fix] Change Date: header location to conform with RFC * [Fix] Correct format pattern for RE tree tempfile name * [Fix] Correct format string for unw_word_t * [Fix] Do not accept invalid ucl object types * [Fix] Do not pollute public headers with libev internals * [Fix] Do not set output type if list application failed * [Fix] Fix `url:set_redirected` method * [Fix] Fix format string and some length issues * [Fix] Fix grammar definition for content-disposition attributes * [Fix] Fix lua schema enrichment logic for Redis params * [Fix] Fix lua stack corruption when logging large tables * [Fix] Fix merge table utility * [Fix] Fix output of non-RSA DKIM keys * [Fix] Fix some corner cases of single-host urls parsing * [Fix] Fix various issues in the `url_redirector` plugin * [Fix] MISSING_MIMEOLE: avoid matching messages from Android GMail app (#4561) * [Fix] Prevent DNSWL sabotage * [Fix] Try to fix unzip function * [Fix] rbl: really fix dependency registration when symbols_prefixes is used * [Fix] rspamadm mime: arguments beginning with letter `t` * [Rework] Breaking: return back to semver * [Rework] Move rcl logic to C++ |
8 months ago |
|
Vsevolod Stakhov
|
bbb8660e55
|
Release 3.6
* [Conf] Add `one_shot` to some specific multimap rules * [Conf] Add language detection configuration * [Conf] Add missing attributes for the language detection configuration * [Conf] Remove outdated composite rules * [Feature] Add `sentinel_password` option * [Feature] Add ability to deny specific fuzzy flags by default * [Feature] Add controller endpoint to get fuzzy hashes from messages * [Feature] Add extra symbol when URL redirector reaches nested limit * [Feature] Add function to transliterate utf8 to ascii with some normalisation * [Feature] Add html parsing limit * [Feature] Add order to urls structure * [Feature] Add some missing functions to `lua_rsa` library * [Feature] Allow fuzzy workers to exchange blocked information * [Feature] Allow to have weak flags in fuzzy storage * [Feature] Allow to read options from maps in the multimap plugin * [Feature] Allow to use other methods when fasttext detection is enabled * [Feature] Count stats per key per flag * [Feature] Finish all features of dkim_keygen in Lua * [Feature] Khash: Allow static initialisation * [Feature] Maps: Add on_load support * [Feature] Preliminary implementation of dynamic composites * [Feature] Process HTML parts before text ones * [Feature] Reorganise struct rspamd_url to be 64 bytes size * [Feature] Save fuzzy ratelimit buckets * [Feature] Use in-place deflation for strings * [Feature] external_relay: add ip_map strategy * [Fix] Avoid race between config new/free by using a counter * [Fix] Do not use `rspamadm.dkim_keygen` * [Fix] Feed fasttext language model with the pre-tokenized words * [Fix] Fix `rspamd_has_only_html_part` * [Fix] Fix an old issue with order of destruction race between redis pool and lua * [Fix] Fix format string usage * [Fix] Fix parsing due to old bug revealed * [Fix] Fix parsing of the mask values that are invalid * [Fix] Ignore non-unique stop words * [Fix] Include the last character when parsing the last header with no value * [Fix] More fixes to fuzzystat * [Fix] Set proper counter * [Fix] Try harder to clean pending bucket * [Fix] Try harder to remove bad hyperscan files * [Fix] Update stats before encryption... * [Fix] dmarc gramar - allow spaces before ";" * [Fix] rbl: fix dependency registration when symbols_prefixes is used * [Fix] remove obsolete rspamd-redirector files * [Fix] test external_relay: count should always be the last rule, as it have no matching condition * [Project] Allow to register multimap symbols dynamically * [Project] Implement fasttext language detection * [Rework] Default max shots must not influence options * [Rework] Write dkim keygen tool in lua * [Rules] Add thread hijacking composite rule |
10 months ago |
|
Vsevolod Stakhov
|
0c671194e8
|
Release 3.5 | 1 year ago |
|
Vsevolod Stakhov
|
05da2709be
|
Release 3.4
* [CritFix] Restore compatibility with the integrations and headers alterations * [Feature] Milter_headers: Add `x-rspamd-action` routine * [Feature] Share hyperscan database among processes * [Fix] Another corner case in url parsing * [Fix] Another fix for the enable password * [Fix] Another try to fix close method in lua_tcp * [Fix] Fix additional fields in the Redis schema * [Fix] Fix emoji joiner FP * [Fix] Fix favicon.ico Content-Type header * [Fix] Fix hang when close is used * [Fix] Lua_tcp: Sigh, another try to fix `close` invocation * [Fix] Mx_check: Cache the fact of a missing MX record * [Fix] Try to fix parsing of the unencoded `>` characters in html attributes * [Fix] Try to fix the case where password == enable_password * [Project] (Re)implement hyperscan caching * [Project] Rework cleanup * [Project] Synchronize hyperscan caches via the main process * [Rework] Convert multipattern to use hyperscan tools * [Rework] Make http normalize path function a generic function * [Rework] Split locked and unlocked files, as mmap does not need flock normally * [Rework] Start movement of the hyperscan related routines into a single unit * [Rework] Store the current worker, so other libraries could use this information * [Rework] Use blocking socket for IPC between main and workers * [Rework] Use more predictable size for commands buffers * [Rules] Do not insert ONCE_RECEIVED_STRICT on RDNS missing * [Rules] Reduce score of HTTP_TO_HTTPS - subject to remove completely |
1 year ago |
|
Vsevolod Stakhov
|
16819f6fd0
|
Release 3.3
* [Conf] Add missing groups for whitelist module symbols * [CritFix] Neural: Fix keys regression after #3968 * [Feature] Accept upstream in lua_tcp * [Feature] Add ability to statically maintain disabled/enabled patterns * [Feature] Add function to store upstreams for HTTP urls * [Feature] Allow augmentations set in Lua API * [Feature] Allow lua_http module to accept upstreams * [Feature] Allow to limit write access to fuzzy storage by key * [Feature] Allow to sort symbols output * [Feature] Check content for binary stuff before dumping it to Lua * [Feature] Implement symbols augmentations * [Fix] Add missing flags * [Fix] Add more sanity checks for rua in dmarc_report * [Fix] Adjust length of the fuzzy checks for short text parts * [Fix] Another try to fix add headers compatibility logic * [Fix] Another try to fix race condition in the runtime destruction * [Fix] Avoid cyclic references in symcache and fix memory leaks * [Fix] Avoid overriding IP with Sender IP * [Fix] BAD_REP_POLICIES did not trigger when message was classified as spam by Bayes * [Fix] Bind AF_UNIX DGRAM client connection to annonymous address * [Fix] Disable IPv6 lookups for Blocklist.de RBL * [Fix] Distinguish dynamic and static items * [Fix] Dkim: Ignore unknown DKIM kv pairs as stated in RFC * [Fix] Dmarc report: Use local timezone instead of GMT * [Fix] Do not exclude authenticated users from URIBL lookups * [Fix] Empty envelopes should not be emitted as arrays (json+messagepack) when populated envelopes are objects. This greatly complicates decoding in strictly typed languages. * [Fix] External_relay: Restore the originating hostname check * [Fix] Fix DKIM keys with spaces still allowing errors on invalid base64 * [Fix] Fix copying of sockaddr_un addresses * [Fix] Fix crash with cname replies * [Fix] Fix dependencies propagation * [Fix] Fix iteration over milter headers * [Fix] Fix ordering when sorting symcache * [Fix] Fix reading of the cached maps * [Fix] Fix several issues with the HTTP keepalive parsing * [Fix] Fix stack smashing * [Fix] Fix synchronous auth/select in lua_redis * [Fix] Fix various symcache issues * [Fix] Ignore all (I hope) unknown DKIM signature KV pairs * [Fix] Ignore directories in RarV5 archives * [Fix] Libucl: avoid memory leak on objects merging * [Fix] Lua_tcp: Another try to fix closing logic * [Fix] Mempool: Fix alloc_array function to actually multiply nmembers by size * [Fix] Only check allowed fuzzy worker update ips for non-unix sockets * [Fix] Plug memory leak in regexp destruction with pcre2 * [Fix] Properly check the original email flag * [Fix] Properly deal with `get_symbol/get_metric_symbol` ambiguity * [Fix] Properly parse expressions atoms * [Fix] Properly set `Host` in rspamd_proxy * [Fix] Rbl: Fix received positioned checks * [Fix] Remove check for a score with no symbol being registered * [Fix] Same fix for lua_tcp * [Fix] Skip cname records when processing SPF records * [Fix] Skip sending dmarc reports in no-opt mode fixes https://github.com/rspamd/rspamd/issues/4241 * [Fix] Stop slow timer on task destruction * [Fix] Symcache: Do not use C style comparators in C++ sorts * [Fix] Try to avoid a corner case for `@` pattern * [Fix] Try to fix dkim reputation adjustements * [Fix] Try to fix passthrough results processing logic * [Fix] Try to fix the mess with read only flag * [Fix] Upstreams: Don't ignore revive_time config option * [Fix] Use proper format string, sigh... * [Fix] Use space category in ragel automata to resolve space characters * [Fix] Zstd: Fix compression with the new Zstd API * [Fix] milter_headers: Header fields may be inserted at wrong position. * [Project] Add experimental HTTP statistics backend * [Project] Add more methods for symbols addition * [Project] Add raii_sink file helper * [Project] Add some more methods * [Project] Add symbols processing methods * [Project] Allow `=` separated augmentations to be treated as kv pairs * [Project] Allow to extract augmentation values * [Project] Few more methods * [Project] Fix on conditions * [Project] Further efforts to make a more consistent architecture * [Project] Further rework * [Project] Further rework tracking * [Project] Further split of the code * [Project] Get rid of C style ctors/dtors * [Project] Http_stat: Notice statfiles when creating runtime * [Project] Implement dynamic items lookup and processing * [Project] Implement item finalization * [Project] Implement more methods * [Project] Implement runtime creation * [Project] Implement settings processing + some neats * [Project] Implement some conditions checks * [Project] Implement validation logic * [Project] More methods * [Project] Move runtime cache part to a separate unit * [Project] Move some more methods * [Project] Re-implement counters method * [Project] Reimplement dependencies processing * [Project] Remove obsoleted methods * [Project] Remove old code (finally) * [Project] Rework symbols execution * [Project] Some more adjustments in symbols registration * [Project] Start rewrite symcache in c++ * [Project] Support augmentations with values * [Project] Symcache: Use ordered filters to avoid extra lookups * [Rework] Another movement * [Rework] Augmentations can now imply flags * [Rework] Further steps * [Rework] Further work on deps processing * [Rework] Implement cache resorting * [Rework] Isolate disable/enable logic for the configuration ucl objects * [Rework] Move item implementation to a separate header * [Rework] Multimap: Avoid prefilters usage where augmentations can be used * [Rework] Pass upstream when sending TCP requests * [Rework] Re-implement cache sorting * [Rework] Reimplement saving/loading the cache items * [Rework] Reiterate on priorities * [Rework] Rework files structure * [Rework] Rewrite rspamc in C++ * [Rework] Simplify scores check and extend it to pre/post filters * [Rework] Switch minimum C++ standard version to C++20 * [Rework] Try to fix the mess with types & flags * [Rework] Use another version of hash table from the same author * [Rework] Use dynamic items for calling callbacks * [Rework] Use dynamic items in the callbacks * [Rework] Use hash map for id->symbol mappings * [Rework] Use khash instead of uthash in rdns compression logic |
1 year ago |
 Vsevolod Stakhov
|
c1b767c234
|
Release 3.2
* [Conf] Score MIME_OBFUSCATED_ARCHIVE to 8 points * [Conf] Set one_shot for URIBL rules by default * [CritFix] Fix upstreams name resolution when there is also a port * [Feature] Add ROC feature to neural network plugin * [Feature] Add public suffic compilation utility * [Feature] Add support of Cloudmark * [Feature] Allow hyperscan for ppc64, as vectorscan now suports it. * [Feature] Allow to skip DNS resolution for keep-alive connections * [Feature] Aws_s3: Allow to store large parts separately * [Feature] BIMI: Add preliminary version of the BIMI plugin * [Feature] JSON endpoint for querying maps * [Feature] Lua_magic: Add a sane CSV heuristic * [Feature] Lua_mime: Add schema for message transfer * [Feature] Output average scan time in /stat endpoint * [Feature] Show average scan time in `rspamc stat` output * [Fix] Add guards to avoid race condition on TCP connection * [Fix] Allow spaces in DKIM key records * [Fix] Apply the similar fix to the url_reputation * [Fix] Avoid overwriting whitelisted_signers_map * [Fix] Backport PR from libucl * [Fix] Clear SSL errors * [Fix] ClickHouse cleanup of old partitions * [Fix] Do not double call error handler on ssl errors in the timeout path * [Fix] Do not forget to clear pointers on IOC reset * [Fix] External_relay: Remove useless check of the map value * [Fix] Find suspicious url encodings that could break url extraction * [Fix] Fix HTTP(s) client timeout * [Fix] Fix exclude flags setting * [Fix] Fix expanding of the variables * [Fix] Fix host header usage in lua_http * [Fix] Fix http maps shared memory cache cleanup * [Fix] Fix logic in HTML processing FSM * [Fix] Fix parsing of the compound mailto urls * [Fix] Fix processing captures from pcre2 * [Fix] Fix removing from khash * [Fix] Fix stuctured headers pushing * [Fix] Further fix for i386 compilation * [Fix] Improve duplicate settings error reporting * [Fix] Lua: task:remove_result didn't work in some cases * [Fix] Output service parts as well * [Fix] Phishing: Deal with phishing + redirected URL * [Fix] Phishing: Fix finding domains in the phishing map * [Fix] Plug memory leak by using mempool for a copied address * [Fix] Properly find the request and the number of requested entries * [Fix] Rbl: Fix inversed logic of the url_full_hostname * [Fix] Read file maps if they were not pre-read during preload * [Fix] Restrict x86_64 assembly to x86_64 * [Fix] Return a real number of recipients when dealing with aliases * [Fix] Rework unshedule DNS request function * [Fix] Support definition of ungrouped symbol in conf file, use group info from lua or other conf file * [Fix] Unschedule DNS request when clearing IO channel * [Fix] When checking for phishing, we need to convert punicode -> UTF8, not vice versa * [Fix] lua_cfg_transform - actions without score (discard) * [Fix] lua_cfg_transform - silly break break actions * [Fix] ratelimit - symbol per bucket * [Project] BIMI: Fix helper integration issues * [Project] Further DNS over TCP architecturing * [Project] Rdns: Add more functions for TCP based requests * [Project] Rdns: Add preliminary reading logic for TCP channels * [Project] Rdns: Add reaper for inactive TCP connections * [Project] Rdns: Add timeout logic for TCP requests * [Project] Rdns: Do not treat TCP channels failure as fatal * [Project] Rdns: Fix TCP connection mess * [Project] Rdns: Fix TCP stuff cleanup * [Project] Rdns: Fix various ownership issues * [Project] Rdns: Implement TCP writing logic * [Project] Rdns: Initial support of TCP IO channels * [Project] Rdns: More fixes in TCP handling * [Project] Rdns: Restore the previous EDNS0 size * [Project] Rdns: Send truncated replies via TCP * [Project] Rdns: Unregister TCP requests * [Rework] Allow to restore SSL handlers after keepalive pooling * [Rework] Allow to set a different behaviour for actions from settings * [Rework] Include SSL flag into keepalive hash * [Rework] Make `rspamadm dmarc_report` default behaviour more sane * [Rework] Mempool: Use explicit alignment * [Rework] Rdns: Use faster and more compact hash table for DNS requests * [Rework] Rework SSL flag operations * [Rework] Take disabled flag into account * [Rework] Timeouts are now global per event and not reseted by IO activity * [Rework] Use xxh3 as a default hash and fix memory/alignment issues * [Rules] Fix old rules to stop global functions usage * [Rules] Fix symbol for DKIM temporary failure * [Rules] Remove ancient and inefficient rules * [Rules] Slightly reduce MULTIPLE_FROM score |
2 years ago |
|
Vsevolod Stakhov
|
87a4c54609
|
Release 3.1
* [Feature] Add junk_threshold for autolearn * [Feature] Add neural test command * [Feature] Antivirus: Allow to set fake eicar patterns for testing AV engines * [Feature] Lua_cdb: Add cdb building interface * [Feature] Ratelimit: Add per bucket configurations * [Feature] S3: Allow to store structured data in messagepack * [Fix] Add concept of uncancellable events to prevent use-after-free * [Fix] Add temporary guard to prevent linked list exploitation * [Fix] Another rework of the ucl hashing * [Fix] Another try to fix references safety * [Fix] Another try to fix rspamd_text passing in the selectors * [Fix] Avoid copy for received structure as it has raw C pointers * [Fix] Avoid dangling reference * [Fix] Correctly check numeric URLs in URL DNS lists * [Fix] Delete the correct pointer type * [Fix] Dmarc: Always lowercase domain * [Fix] Fix compilation of the hyperscan databases with errors * [Fix] Fix hash table lookup * [Fix] Fix http message flag shift * [Fix] Fix parsing of the from_hostname when it is an IP address * [Fix] Fix parsing of the unquoted attributes in HTML * [Fix] Fix passing of rspamd_text in selectors pipelines * [Fix] Fix rubbish QP sequences decoding * [Fix] Fix some complicated case with the closing tags parsing * [Fix] Fix the case when l tag is too small * [Fix] Html: Fix the case where only bgcolor is explicitly set * [Fix] Libucl: Fix deletion from ucl objects * [Fix] Namespace and add metadata for OpenMetrics, fix interleaving * [Fix] Plug memory leak in http settings reload * [Fix] Preserve SPF top record in the mempool variable * [Fix] Remove aarch64 GC64 workaround * [Fix] Remove bogus G_LIKELY * [Fix] Spf: Do not parse non TXT DNS replies as TXT replies * [Fix] Try to use on_connect/on_disconnect callbacks to handle internal Redis failures * [Fix] buffer overflow in rspamc counters * [Fix] fix static building * [Fix] lua_scanners - message_min_words logic * [Fix] src/lua/lua_mimepart.c: fix null dereference * [Project] Add constant iterators * [Project] Add helper library to handle mime strings in a more safe matter * [Project] Add preliminary support of CDB bayes dump * [Project] Add trim operations * [Project] Allow mempool allocated mime strings * [Project] Cdb: Finish backend implementation * [Project] Cdb: Fix configuration load * [Project] Cdb: Use shared data between cdb statfiles * [Project] Cdb: continue statistics backend implementation * [Project] Finish received headers rework part * [Project] Move C++ specific declarations to C++ header * [Project] Rework received headers parsing to C++ * [Project] Start using of the new received structure * [Project] Start work on cdb backend * [Rework] Further rework of the redis pool * [Rework] Redis_pool: fix issues found * [Rework] Rework learn and add classify condition * [Rework] Save invisible content to a separate buffer * [Rework] Start rewriting of the redis pool logic * [Rules] Improve zero font rule Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
2 years ago |
|
Vsevolod Stakhov
|
4fc136a7f8
|
Release 3.0
* [Conf] Align ARC scores with DKIM scores * [CritFix] Neural: Fix sorting application * [Feature] Add a simple dumper for bayes tokens * [Feature] Add lua_maps.fill_config_maps function * [Feature] Add preliminary exporter to AWS S3 * [Feature] Add preliminary restore bayes support * [Feature] Add race condition protection against hs_helper restarts * [Feature] Add rspamd_utf8_strcmp utility * [Feature] Add zstd streaming API * [Feature] Allow to log severity level explicitly * [Feature] Allow to save and show attachment name when inserting AV scan results * [Feature] Allow to sort urls for Lua * [Feature] Allow to specify different timeouts/retransmits for fuzzy rules * [Feature] Aws_s3: Allow to compress data stored * [Feature] CMakeLists.txt: Change check and run-test to use rspamd-test-cxx * fixes #3807 * [Feature] Dmarc_report: allow sending reports in batches * [Feature] Fuzzy_check: Allow to disable subject when making short text hash * [Feature] Lua_cryptobox: Add keyed ssl hash functions via HMAC * [Feature] Lua_task: Add get_urls_filtered method * [Feature] Make monitored checks less frequent * [Feature] Milter_headers: Add x-rspamd-pre-result header * [Feature] Neural: Allow to balance FP/FN for the network * [Feature] Ppopagate monitored errors from rbl module * [Feature] Pyzor calculate score dynamically Count - WL-Count of default_score in percent * [Feature] Rbl: Distinguish flattened and non-flattened selectors in RBL requests * [Feature] Re-add pyzor support * [Feature] Settings: add ip_map check and rework structure slightly * [Feature] Spamassassin: Allow to set the default priority for SA scores * [Feature] Strip smtp comments from message id * [Feature] add SYSTEM_ZSTD cmake option To use the system zstd instead on the bundled version * [Feature] external_relay plugin * [Feature] rspamadm clickhouse neural_train subcommand * [Fix] #3400 milter_headers: fix inverted logic for extended_headers_rcpt * [Fix] ASN: fix _FAIL symbol for when main symbol is disabled * [Fix] Add a special logic for text part with no text extraction * [Fix] Add diacritics flag for several eu languages * [Fix] Another FSM fix to accomodate possibility of multiple consequent ? * [Fix] Avoid curse of dynamic array referencing * [Fix] Avoid reinitialising neural settings * [Fix] Check remain before processing TXT records * [Fix] Enable error multiplier on http errors * [Fix] Finally rework parsing entities logic * [Fix] Fix '==' parsing in the content type attributes parser * [Fix] Fix IPv6 expansion for SPF macros * [Fix] Fix Mozilla Message-ID detection * [Fix] Fix an edge case in BITCOIN_ADDR rule * [Fix] Fix brain-damaged behaviour when http request has a custom Host header * [Fix] Fix check of limits in email address parsing * [Fix] Fix copy&paste error and rework * [Fix] Fix expressions logic for and/or and float values * [Fix] Fix fuzzy retransmits * [Fix] Fix http maps with no or invalid expires data * [Fix] Fix last quote character parsing in the content-type state machine * [Fix] Fix normalisation flags propagation * [Fix] Fix overflow when appending many broken tags * [Fix] Fix parsing of rfc2047 tokens with '?' inside * [Fix] Fix phishing flag set * [Fix] Fix rfc2047 embedded into rfc2231 pieces in special headers * [Fix] Fix round-robin rotation * [Fix] Fix searching for symbols * [Fix] Fix storing of the regexps inside variant * [Fix] Fix tokenization near exceptions * [Fix] Fix visibility calculations * [Fix] Html: Attach inline tags to the structure * [Fix] Html: Do not treat empty tags as block tags * [Fix] Ical: Do not extract urls from all flags using merely specific ones * [Fix] Initialise symcache even if it cannot be loaded properly * [Fix] Lua_fuzzy: Remove text parts check when checking image dimensions * [Fix] Lua_maps: Fix adjustments for the map type in the complex map definitions * [Fix] Lua_task: Fix deleted symbols in has_symbol/get_symbol * [Fix] Move metric and symcache link from validation to the init stage * [Fix] Oletools: Another try to fix table sorting * [Fix] One more default behaviour fix * [Fix] Phishing: Rework urls processing * [Fix] RBL: was missing some config schema * [Fix] Replies: Fix 'Reply-To' handling in task:get_reply_sender * [Fix] Rework metrics handling * [Fix] Save symcache on exit * [Fix] Selectors: Filter nil elements in lists * [Fix] Selectors: Properly fix implicit tostring for nils * [Fix] Try to fix some broken code in DMARC reporting plugin * [Fix] Urls: Fix processing of html urls when it comes to the flags * [Fix] Use proper buffer length * [Fix] Various visibility fixes * [Fix]: ASN: dns cb func should also return in case of an error * [Project] Add a simple css rule definition * [Project] Add css style skeleton * [Project] Add css syntax (adopted from ebnf) * [Project] Add css_selectors * [Project] Add doctest unit testing library * [Project] Add expected library * [Project] Add fmt library for simple string ops * [Project] Add fu2 library to better functions abstractions * [Project] Add hashing method * [Project] Add parsers skeleton * [Project] Add preliminary support of vcard parser * [Project] Add process exceptions for invisible text * [Project] Add some methods for css parser * [Project] Allow static libstdc++ * [Project] Another whitespace hack * [Project] CSS: Various fixes in the declarations and values parsing * [Project] Cpp: Add robin-hood hash map library * [Project] Css: Add AST debug * [Project] Css: Add colors conversion functions * [Project] Css: Add dimensions handling * [Project] Css: Add display value support * [Project] Css: Add frozen library from https://github.com/serge-sans-paille/frozen/ * [Project] Css: Add opacity support * [Project] Css: Add parser helpers to simplify debugging * [Project] Css: Add preliminary stylesheet support * [Project] Css: Add rules processing functions and tests * [Project] Css: Add simple selectors unit tests * [Project] Css: Add some c++ unit tests * [Project] Css: Add some debug methods * [Project] Css: Add some debug statements for the css parser * [Project] Css: Add some logical skeleton for declarations parser * [Project] Css: Add url/function tokens * [Project] Css: Allow at rules parsing * [Project] Css: Declarations parsing logic skeleton * [Project] Css: Enable conditional css parsing support from the HTML parser * [Project] Css: Finish generic lexer cases * [Project] Css: Fix HSL conversion * [Project] Css: Fix minus parsing * [Project] Css: Fix parser consumers nesting * [Project] Css: Fix parsing of the qualified rules * [Project] Css: Fix rules merging * [Project] Css: Further fixes to lexer * [Project] Css: Further steps to parse css colors + rework * [Project] Css: Further work on parser's methods * [Project] Css: Implement backlog of css tokens * [Project] Css: Implement numbers and ident parsers * [Project] Css: Implement simple css selectors lookup * [Project] Css: Implement styles merging * [Project] Css: Make debug strings json like to simplify tests * [Project] Css: Minor adjustments * [Project] Css: More meat to the lexer * [Project] Css: Move some of the tests to the doctest * [Project] Css: Projected a parser * [Project] Css: Properties attachment logic * [Project] Css: Remove ragel from build targets (maybe keep for reference) * [Project] Css: Rework css block structure * [Project] Css: Rework flags of css properties * [Project] Css: Rework tokens structure * [Project] Css: Several fixes + tests * [Project] Css: Simplify checks * [Project] Css: Simplify debug code * [Project] Css: Start css selectors parsing logic * [Project] Css: Start semantic parsing for rules * [Project] Css: Start stylesheet implementation * [Project] Css: Tidy up lambdas * [Project] Css: rework tokeniser * [Project] Dmarc: Add dmarc report tool (WIP) * [Project] Dmarc: Add munging configuration * [Project] Dmarc: Add preliminary munging logic * [Project] Dmarc: Fix header removal * [Project] Dmarc: Fix munging logic * [Project] Dmarc: Use full recipient address instead of a domain map * [Project] Dmarc: Use zlists for dmarc reports * [Project] Dmarc_report: Add message generation logic * [Project] Dmarc_report: Add preliminary sending support * [Project] Fix lua bindings * [Project] Fix xml/sgml tags processing * [Project] Handle new modification * [Project] Html/CSS: Add transform from a CSS rule to html block * [Project] Html/CSS: Link html and css styles * [Project] Html/CSS: Switch styles parsing to css parser * [Project] Html/Css: Fix some issues found * [Project] Html/Css: Implement visibility rules for a block * [Project] Html: Add more tests cases and fix some more corner issues * [Project] Html: Add rows display type support * [Project] Html: Allow decode entities function to normalise spaces + unit tests * [Project] Html: Another rework of the tags structure * [Project] Html: Another try to fix unbalanced cases * [Project] Html: Fix crossing spans * [Project] Html: Fix parent propagation * [Project] Html: Further rework of the html parsing stuff * [Project] Html: Implement logic for tags pairing * [Project] Html: Implement rawtext state machine * [Project] Html: Insert closing tags as well :( * [Project] Html: More fixes * [Project] Html: More fixes * [Project] Html: More spaces logic fixes * [Project] Html: One more attempt to write text content * [Project] Html: Replace \0 in html content * [Project] Html: Rework img/a tags handling * [Project] Html: Rework propagation method * [Project] Html: Rework tags placement * [Project] Html: Rework transparency logic * [Project] Html: Support 'hidden' attribute * [Project] Html: Try another approach to append tags content * [Project] Html: Try to deal with bad unknown tags properly * [Project] Lua_aws: Add canonicalisation utility * [Project] Lua_aws: Add function to produce AWS Authorisation header * [Project] Lua_aws: Implement request signing * [Project] Lua_mime: Add lua_mime.modify_headers routine * [Project] Lua_task: Add modify_header method * [Project] Lua_task: Allow to extract modified headers * [Project] Make unescape code public for unit testing * [Project] More fixes for closed tags * [Project] More fixes to calculations * [Project] Rework API for the modified headers * [Project] Rework html visibility rule * [Project] Skeleton of the css library * [Project] Start headers modification API structure * [Project] Start working on AWS Lua API * [Project] Use lua_mime to modify headers * [Project] Use modified headers on dkim signing * [Project] Use string_view to constexpr variant unpacking * [Rework] Add composites manager concept * [Rework] Add tags definitions * [Rework] Allow C code to be compiled with C++ compiler * [Rework] Clickhouse: Store url flags * [Rework] Composites: Rewrite the composites logic * [Rework] Composites: Start rework of the composites framework * [Rework] Dmarc: Move check policy function to the common utils * [Rework] Dmarc: Rework reports keys structure * [Rework] Further work to make html content private * [Rework] Html/CSS: Remove css C bindings as they are useless now * [Rework] Html/CSS: Rework Lua bindings * [Rework] Html/Css: Start rework of the html blocks * [Rework] Html: Add images processing logic * [Rework] Html: Add traverse function * [Rework] Html: Another steps to get rid of gnode * [Rework] Html: Convert to variant * [Rework] Html: Deal with the utf_content part * [Rework] Html: Final rework part for the html processing code * [Rework] Html: Fix Lua bindings * [Rework] Html: Forgot to add the internal include * [Rework] Html: Further html urls rework * [Rework] Html: Further rework of the tags content extraction * [Rework] Html: Make parameters as a vector again * [Rework] Html: Move blocks part * [Rework] Html: Move images processing stuff * [Rework] Html: Rework lua bindings * [Rework] Html: Start html text extraction rework * [Rework] Html: Start refactoring of the html tags handling * [Rework] Html: Start removing of GNode stuff * [Rework] Html: Start rework of the html content structure * [Rework] Lua_magic: Try to detect text parts with 8bit characters for non-utf8 encodings * [Rework] Move HTML url functions and rework them * [Rework] Move and adopt entities handling logic * [Rework] Move common and rarely used dmarc code to the library * [Rework] Move compression routines outside of rspamd_util library * [Rework] Move entities/tags handling * [Rework] Phishing: Split from redirectors usage * [Rework] Redesign html blocks propagation logic * [Rework] Remove tag name string * [Rework] Rename phished url to a linked url * [Rework] Reorganize dmarc plugin and remove unsupported reporting code * [Rework] Reputation: Use more flexible types in get/set functions * [Rework] Require proper C++ environment for Rspamd build * [Rework] Rework extended urls output * [Rework] Rework tags parsing machine * [Rework] Slightly improve old regexp API * [Rework] Start conversion of the redis pool code to c++ * [Rework] Try to resolve failed upstreams more agressively * [Rework] Use C++ utf8 library with unit tests to trim whitespaces * [Rework] Use C++ version for unicode normalisation * [Rework] Use C++ version of the lua threads pool * [Rules] Add raw addresses to MULTIPLE_FROM options * [Rules] Another fix to HTTP_TO_HTTPS rule * [Rules] Do not trigger HTML_SHORT_LINK_IMG on external images * [Rules] Extend FORGED_X_MAILER * [Rules] Extend OLD_X_MAILER * [Rules] Fix CTYPE_MIXED_BOGUS for text attachments * [Rules] Fix FPs for CTYPE_MIXED_BOGUS * [Rules] Fix HTTP_TO_HTTPS rule * [Rules] Fix HTTP_TO_HTTPS rule * [Rules] Fix zerofont rule (partially) * [Rules] Micro-optimize X_PHP_EVAL * [Rules] Reduce default weight for R_MISSING_CHARSET Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
2 years ago |
|
Vsevolod Stakhov
|
4c56c1210d
|
Release 2.7
* [Conf] Add R_DKIM_PERMFAIL to the metric * [CritFix] Dkim: Fix simple canonicalisation if multiple signatures are presented * [CritFix] Fix controller paths normalisation * [Feature] Add INVALID_DATE rule * [Feature] Add controller endpoint for training neural * [Feature] Add sanity checks for actions thresholds * [Feature] Add support of '==' and '!=' in Rspamd expressions * [Feature] Composites: Improve composite atoms parser * [Feature] Docker: use Debian slim variant * [Feature] Elastic: Add some missing fields * [Feature] Extract text from img alt attributes * [Feature] Improve charset detection logic * [Feature] Lua_clickhouse: Add optional row callback for large selections * [Feature] Lua_dns_resolver: Add idna_convert_utf8 method * [Feature] Lua_mime: Add ability to do multipattern replacement * [Feature] Lua_trie: Allow to report start of the match * [Feature] Multimap: support adding map values as extra options * [Feature] Neural: Move PCA learning to a subprocess * [Feature] RBL: support matching content/image URLs only * [Feature] RBL: support use of multiple selectors * [Feature] Reputation: Allow to specify ip masks * [Feature] Support SMIME signed messages container * [Feature] Support multiple conditions for symbols * [Feature] Support ping in milter mode * [Feature] Support rspamd_text in selector regexps * [Feature] Use own daemonization routine * [Feature] Vadesecure: Implement settings_outbound feature as recommended by Vade * [Feature] `rspamadm clickhouse` command * [Feature] allow hyperscan for aarch64 * [Fix] Allow to set priorities between post init scripts * [Fix] Allow to use maps for strings that are not zero terminated * [Fix] Apply max_lua_urls limit for emails as well * [Fix] Arc: Fix CV check on signing * [Fix] Arc: Fix signing of the broken ARC chains * [Fix] Clickhouse: escape carriage return * [Fix] Composites: Allow partial match * [Fix] Deduct type of a table methods * [Fix] Do not load errored hyperscan database * [Fix] Do not process links in ignored html tags * [Fix] Fix ClamAV result for cached encrypted file (#3395) * [Fix] Fix canonicalisation when l= tag is presented * [Fix] Fix flag shift * [Fix] Fix handling of skip/skip_process http flags * [Fix] Fix html attachments checks * [Fix] Fix issue with pushing binary formats to Lua strings * [Fix] Fix logging for rspamadm * [Fix] Fix off-by-one with init check * [Fix] Fix parsing of escape characters in quoted pairs * [Fix] Fix pushing ucl strings with \0 inside * [Fix] Fix quoted-printable soft newlines bugged case * [Fix] Fix settings in case actions are set to null (#3415) * [Fix] Fix several issues with auth results producing * [Fix] Fix smtp comments exclusion * [Fix] Fix smtp date syntax definition * [Fix] Fix substring search in case if srchlen == inlen * [Fix] Fix text selectors * [Fix] Honour `systemd` setting when logging to console (#3514) * [Fix] Html: Add entities collisions prevention logic (e.g. for mathml entities) * [Fix] Lua_auth_results: Quote potentially bad values in AR header * [Fix] Multimap: Fix flags usage * [Fix] Multimap: Fix scoring for combined maps * [Fix] Plug GList * leak in redis pool * [Fix] RBL: allow for multiple matches of the same label if types are different * [Fix] Rely on libev checks for file maps * [Fix] Restore simple dkim canonicalisation mode * [Fix] Return MimeCharset as we work with emails... * [Fix] Spamassassin: Fix pcre_only flags * [Fix] Spamassassin: Preserve 'pcre_only' flag when dealing with regexp replacements * [Fix] Try to fix GError leak * [Fix] Try to fix a mess with settings loading by adding priorities * [Fix] Try to move setings initialisation to a later stage * [Fix] Use dup fd in milter handler to avoid races with the proxy * [Fix] Use message pointer to avoid obsolete data to be cached * [Project] Rbl: Migrate to `checks` * [Project] Rbl: Move config code outside of the plugin * [Project] Ressurect empty prefilters as connection filters * [Project] Support connection filters registration from Lua * [Rework] Add final cleanup logic * [Rework] Add preliminary support of hyperscan caching for re maps * [Rework] Add stale cache removal * [Rework] Clickhouse: Improve performance * [Rework] Distinguish between strict config test mode * [Rework] Furhter logging improvements * [Rework] Milter_headers: improve extended_headers_rcpt support * [Rework] Move parsers to a separate lua library * [Rework] Neural: Skip composite symbols * [Rework] Rbl: Rework defaults logic * [Rework] Some tunes to cache saving * [Rework] Track maps origins * [Rework] Use full crypto hash for regexp maps * [Rules] Remove broken rule Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
3 years ago |
|
Vsevolod Stakhov
|
eec4438705
|
Release 2.6
* [Conf] Add missing symbols
* [Conf] Add missing symbols
* [Conf] Fix fat-fingers typo
* [Conf] Fix wrong comment in options.inc
* [Conf] Neural: Fix the default name for max_trains
* [Conf] Register a known symbol
* [Conf] Spf: Add R_SPF_PERMFAIL symbol
* [CritFix] Arc: Fix ARC validation for chains of signatures
* [CritFix] Distinguish socketpairs between different fuzzy workers
* [CritFix] Fix IDNA dots parsing
* [CritFix] Fix test assertion method
* [CritFix] Fix usage of crypto_sign it should be crypto_sign_detached!
* [Feature] Add BOUNCE rule
* [Feature] Add controller plugins support and selectors plugin
* [Feature] Add maps query method
* [Feature] Add minimal delay to fuzzy storage
* [Feature] Add multiple base32 alphabets for decoding
* [Feature] Add preliminary support of BCH addresses
* [Feature] Add query_specific endpoint
* [Feature] Allow multiple base32 encodings in Lua API
* [Feature] Allow to specify nonces manually
* [Feature] Controller: Allow to pass query arguments to the lua webui plugins
* [Feature] Fuzzy_check: Add gen_hashes command
* [Feature] Fuzzy_check: Add weight_threshold option for fuzzy rules
* [Feature] Implement address retry on connection failure
* [Feature] Improve limits in pdf scanning
* [Feature] Initial support of subscribe command in lua_redis
* [Feature] Lua_cryptobox: Add secretbox API
* [Feature] Lua_text: Add encoding methods
* [Feature] Milter_headers: Allow to activate routines via users settings
* [Feature] PDF: Add timeouts for expensive operations
* [Feature] Preliminary maps addon for controller
* [Feature] Split pdf processing object and output object to allow GC
* [Feature] Support BLIS blas library
* [Feature] Support input vectorisation by recvmmsg call
* [Feature] Support multiple base32 alphabets
* [Feature] add queueid, uid, messageid and specific symbols to selectors [Minor] use only selectors to fill vars in force_actions message
* [Feature] allow variables in force_actions messages
* [Feature] extend lua api
* [Fix] #3249
* [Fix] Allow to adjust neurons in the hidden layer
* [Fix] Another try to fix email names parsing
* [Fix] Arc: Allow to reuse authentication results when doing multi-stage signing
* [Fix] Arc: Fix bug with arc chains verification where i>1
* [Fix] Arc: Sort headers by their i= value
* [Fix] Change neural plugin's loss function
* [Fix] Deal with double eqsigns when decoding headers
* [Fix] Default ANN names in clickhouse
* [Fix] Disable reuseport for TCP sockets as it causes too many troubles
* [Fix] Disable text detection heuristics for encrypted parts
* [Fix] Distinguish DKIM keys by md5
* [Fix] Distinguish type from flags in register_symbol
* [Fix] Dmarc: Unbreak reporting after
|
3 years ago |
|
Vsevolod Stakhov
|
61cbcd0483
|
Release 2.5
* [Conf] Mark Rspamd emailbl as ignore whitelist * [Conf] RBL: Add missing emails = true option * [Feature] Add support for scripts in fuzzy storage * [Feature] Arc: Add whitelisted_signers_map option * [Feature] Implement hosts file processing * [Feature] Neural: Introduce classes bias that allows non-equal classes learning * [Feature] Update libev to 4.33 * [Fix] Another brain damage html standard adoptions * [Fix] Another fix for brain damaged obs-fws state * [Fix] Fix flags that caused force_actions failure * [Fix] Fix logging issue * [Fix] Fix lua symbols scores registration when config does not define scores * [Fix] Fix opaque maps logic * [Fix] Fix parsing of the html tags with no spaces after attributes * [Fix] Fix some corner cases in urls parsing, add limits * [Fix] Fix tlds extraction if custom composition rules are used * [Fix] Fix variables replacement in mempool * [Fix] Improve base64 detection * [Fix] Normalize dynamic scores in ANN correctly * [Fix] Plug memory leak introduced by #3153 * [Fix] Stat_redis_backend: Fix memory leak and simplify learn path * [Fix] Try hard to deal with ghost workers * [Fix] metadata_exporter default formatter * [Rework] Change the way to extract URLs when dealing with alternative parts * [Rework] Fix various url extraction issues * [Rework] Re cache: Load compiled hyperscan in the main process as well * [Rework] Re cache: Load hyperscan early * [Rework] Rework URL structure: adjust tld part * [Rework] Rework URL structure: host field * [Rework] Rework URL structure: more structure optimisations * [Rework] Rework URL structure: user field * [Rework] URL: Another update for urls extraction logic * [Rework] Urls: Improve query urls handling * [Rework] Urls: adopt html related stuff * [Rework] Urls: more rework of the urls sets * [Rework] Urls: process query urls in HTML urls correctly * [Rework] Urls: rework urls hash structure * [Rework] Urls: update lua libraries * [Rework] Use multiple search tries for different url extraction types Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
a74e458e15 |
Release 2.4
* [CritFix] Fix parsing of the content type attributes * [Feature] Clickhouse: Add extra columns support * [Feature] Rbl: Add url_compose_map option for RBL rules * [Fix] 'R' flag is for all headers regexp * [Fix] Allow to reset settings id from Lua (e.g. because of the priority) * [Fix] Avoid collisions in mempool variables by changing fuzzy caching logic * [Fix] Avoid strdup usage for symbols options * [Fix] Do not trust stat(2) it lies * [Fix] Filter all options for symbols to have sane characters * [Fix] Fix all headers iteration * [Fix] Fix allowed_settings for neural * [Fix] Fix listen socket parsing * [Fix] Fix maps expressions evaluation * [Fix] Fix sentinel connections leak by using async connections * [Fix] Fix smtp message on passthrough result * [Fix] Fix tld compositon rules * [Fix] Fuzzy_storage: Do not check for shingles if a direct hash has been found * [Fix] Lua_mime: Do not perform QP encoding for 7bit parts * [Fix] Neural: Distinguish missing symbols from symbols with low scores * [Fix] Support listening on systemd sockets by name * [Project] Add lua_urls_compose library * [Project] Allow to set a custom log function to the logger * [Project] CDB maps: Start making cdb a first class citizen * [Project] Clickhouse: Add extra columns concept * [Project] Fix urls composition rules, add unit tests * [Project] Unify cdb maps * [Rework] Logger infrastructure rework * [Rework] Refactor libraries structure * [Rework] Rework SSL caching * [Rework] Update snowball stemmer to 2.0 and remove all crap aside of UTF8 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
11a3b09a5d
|
Release 2.3
* [Conf] SPF is no longer a C module * [Conf] Update spamtrap map path example * [CritFix] Fix html entities decoding * [CritFix] Fix re cache when mix of pcre and hyperscan is used * [Feature] Allow milter code to deal with multiple headers * [Feature] Antivirus: Add avast support * [Feature] Dkim_signing: Allow to sign via milter_headers * [Feature] Implement content hashes * [Feature] Lua_text: Add regexp split iterator method * [Feature] Lua_text: Implement flattening of the input tables * [Feature] Send quit command to Redis * [Feature] Speed up is_ascii function * [Feature] Spf: Add external_relay option * [Fix] Avoid double escaping * [Fix] Fix O(N^2) algorithm * [Fix] Fix arc seal validation * [Fix] Fix base tag processing according to stupid HTML renderer behaviour * [Fix] Fix dealing with `\0` in ucl strings and JSON * [Fix] Fix gpg parts misdetection * [Fix] Fix ignored symbols exporting * [Fix] Fix processing of numeric url's * [Fix] Fix processing of the closed tcp connections * [Fix] Fix regexp type check for pcre2 * [Fix] Fix urls encode function * [Fix] Fix urls shifting when doing decode to include separators * [Fix] Fix white on white rule and add is_leaf flag * [Fix] Further fixes in charset detection * [Fix] Ignore diacritics in chartable module for specific languages * [Fix] Limit size of symbols options by max_opts_len option * [Fix] More fixes in html tag content calculations * [Fix] Plug memory leak in fuzzy storage * [Fix] Process high priority settings even if settings/id has been specified * [Fix] Select a different upstream on last retransmit * [Fix] Treat soft hyphen as zero width space * [Fix] Try harder to watch the lifetime of the key_stat * [Fix] Use ipv6-mapped-ipv4 addresses in radix trie * [Project] Add logic to break execution when processing symbols* * [Project] Add methods to set specific content for mime parts from Lua * [Project] Lua_content: support PDF files * [Project] Move dns_tool to using of the rspamd_spf from FFI module * [Project] Preliminary SPF plugin in Lua * [Project] Show debug stat for memory pool * [Project] Some rework about specific data that is now tagged * [Project] Start reworking of the mempool structure * [Rework] Allow to add userdata as symbols options * [Rework] Change mime part specifics handling * [Rework] Move LRU SPF cache from spf plugin * [Rework] Rework HTML tags content attachment * [Rework] Rework options hash structure * [Rework] Start lua_content library * [Rework] Stop using of uthash for http headers * [Rework] Use faster hashing approach for memory pools variables * [Rules] Add PDF related rules Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
f87b6eb314 |
Release 2.2
* [Conf] Antivirus: Fix the default config * [Feature] Add verdict library in lua * [Feature] Allow exception when choosing upstream * [Feature] Allow to disable symbols from the metric config * [Feature] Allow to limit maps per specific worker * [Feature] Always validate Rspamd protocol output * [Feature] Antivirus: Add preliminary virustotal support * [Feature] Clickhouse: Rework Clickhouse collection logic * [Feature] Improve base64 usage * [Feature] Shutdown timeout is now associated with task timeout * [Fix] #3129 Multiple classifiers on redis working incorrectly * [Fix] Allow real upstreams configuration * [Fix] Another try to fix slow callbacks and timers * [Fix] Check results of write message as SSL can bork them * [Fix] Clickhouse: Avoid potential races in collection * [Fix] Clickhouse: Fix periodic script * [Fix] Fail DNS upstream on each retransmit attempt * [Fix] Fix consistent hashing when upstreams are marked inactive * [Fix] Fix issues found * [Fix] Fix off-by-one in retries for the proxy * [Fix] Fix termination * [Fix] Fix upstreams exclusion logic * [Fix] Fix utf8 validation for symbols options and empty strings * [Fix] Oops, fix maps reload * [Fix] Rbl: Allow utf8 lookups for IDN domains * [Fix] Sigh, another try to fix brain-damaged openssl * [Project] Add fast utf8 validation library * [Project] Use own utf8 validation instead of glib * [Rework] Another phase of finish actions rework * [Rework] Further cmake system rework * [Rework] Further isolation of the controller's functions * [Rework] Make cmake structure more modular * [Rework] Move cmake modules to a dedicated path * [Rework] Replace controller functions by any scanner worker if needed * [Rework] Rework final scripts logic * [Rework] Rewrite rspamd_str_make_utf_valid function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
601e28d2ea
|
Release 2.1
* [Conf] Update neural.conf * [CritFix] Fix dkim verification for multiple headers listed * [Feature] Add support of uudecode * [Feature] Allow to explicitly set events backend * [Feature] Implement configurable limits for SPF lookups * [Feature] Lua_scanners: Use lua magic for inclusion/exclusion logic * [Feature] Multimap: Do not check files in office archives * [Feature] Neural: Add sampling when storing training vectors * [Feature] SPF: Allow to disable AAAA checks in configuration * [Feature] Spf: Add limits configuration support * [Feature] Store etag in cached HTTP maps + better logging * [Feature] Support segwit BTC addresses, fix LTC verification * [Feature] Support uuencoding * [Fix] Add configurable number of threads for OpenBLAS * [Fix] Add workaround for ragel 7 in hyperscan related maps code * [Fix] Another fix for numeric urls parsing * [Fix] Correct EMA time calculations * [Fix] Do not treat archives as text * [Fix] Do not use strdup on data extracted from lua * [Fix] Fix a failure calcuating URL reputation. * [Fix] Fix crash due to constructors init order * [Fix] Fix crash on parts with no cd * [Fix] Fix empty prefilters that require mime structures * [Fix] Fix event loop creation * [Fix] Fix issues sending DMARC reports. * [Fix] Fix misprint * [Fix] Fix saving of the file maps * [Fix] Fix size calculations when converting from utf16 * [Fix] Fix support of disable_monitoring in rbl * [Fix] Fix use-after-free * [Fix] Fix zip files check to relax requirements * [Fix] Important hiredis fixes * [Fix] Lot's of fixes in maps check logic * [Fix] Lua_tcp: Deal with temporary fails on write * [Fix] Lua_tcp: Make write errors fatal and rework error handlers * [Fix] Meta: Filter some more values * [Fix] Neural: Add protection agains infinities * [Fix] Oops, fix math.huge invocation * [Fix] Plug memory leak * [Fix] Sigh, another email to string fix * [Fix] Try to fix another ownership race in ssl connection * [Fix] Uuencode: Fix parsing of corrupted uuencode * [Fix] lua_scanners - razor rename need_check function * [Rework] Require CMake 3.9 to work, remove manual lto crap Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
8d31adb5cb
|
Release 2.0
* [Conf] Add BROKEN_HEADERS_MAILLIST composite * [Conf] Add path to greylist-whitelist-domains.inc * [Conf] Clarify documentation in the config files * [Conf] Introduce maps.d directories * [Conf] Log settings id by default * [Conf] Make LEAKED_PASSWORD_SCAM a composite rule again * [Conf] Move all surbl/emails rules to rbl * [Conf] Register new Spamhaus codes * [Conf] Remove configs for deleted modules * [Conf] Remove surbl parts, fix hash_format attribute * [Conf] Show autolearn sample * [Conf] Slashing: Change default stats backend to Redis * [Conf] Surbl: Utilise new `check_emails` option * [Conf] Update header * [Conf] Use multi-prefixes RBLs in the default config * [CritFix] Deal with case-sensivity in Content-Disposition parser * [CritFix] Eliminate old endpoint * [CritFix] Fix case sensivity when parsing Content-Type * [CritFix] Fix loading of DKIM public keys * [CritFix] Fix procesing of urls * [CritFix] Fix whitelisting when both spf and dkim are required to be valid * [CritFix] Langdet: Fix language detection where no stop words found * [Feature] Add description to the groups * [Feature] Add limit for number of URLs in Lua * [Feature] Add logging of groups to the log_format * [Feature] Add lua_smtp library * [Feature] Add maps cache and type refinement * [Feature] Add p0f scanner * [Feature] Adopt emails module to use lua_maps * [Feature] Allow options matching in composites * [Feature] Allow selectors in rbl module * [Feature] Allow to output group results * [Feature] Asn: Allow to use bgpdump when NET::MRT is broken * [Feature] Calculate tokens occurrences distribution * [Feature] Clickhouse: Add authenticated user and settings id columns * [Feature] Clickhouse: Store groups data * [Feature] Clickhouse: Utilise LowCardinality feature * [Feature] Implement Redis prefixes registration logic * [Feature] Implement settings id propagation between deps * [Feature] Improve AV results caching * [Feature] Improve autolearning * [Feature] Improve logging locking logic (remove it actually) * [Feature] Improve settings processing * [Feature] Langdet: Limit number of stop words to be checked * [Feature] Libucl: Allow to sort keys in ucl objects * [Feature] Lua_config: Extend get symbols method * [Feature] Lua_maps: Allow static maps for key-value pairs * [Feature] Lua_mimepart: Add function filter_words * [Feature] Lua_selectors: Add `words` selector * [Feature] Lua_selectors: Add sort and uniq transform functions * [Feature] Lua_selectors: Allow table arguments for selectors * [Feature] Lua_tcp: Add preliminary support of SSL connections * [Feature] Lua_trie: More flexible API * [Feature] Lua_util: Add filter_specific_url function * [Feature] Lua_util: table_digest can now recursively traverse tables * [Feature] Maillist: Improve detection * [Feature] Maps: Allow caching for complex maps * [Feature] Monitored: Support random lookups * [Feature] Multimap: Add combined maps prototype * [Feature] Multimap: Add dependend maps via redis keys selectors * [Feature] Multimap: Allow multiple email addresses matches * [Feature] Multimap: Also check detected charset when do filename checks * [Feature] Output number of messages processed to proctitle * [Feature] Perform clean SSL shutdown * [Feature] Performance: Do not use base64 SIMD version for bad inputs * [Feature] RBL: Support bit results in replies * [Feature] RBL: Support type specific prefixes * [Feature] Ratelimit: Consider number of SMTP recipients * [Feature] Rbl: Add ability to check urls * [Feature] Rbl: Add resolve_ip based RBLs * [Feature] Rbl: Make config checks much more strict * [Feature] Rbl: Support per-rule whitelists * [Feature] Rbl: Support process script * [Feature] Rbl: Support replyto addresses * [Feature] SURBL: Allow to check email domains * [Feature] Selectors: Add `list` generator * [Feature] Selectors: Add `specific_urls` extractor * [Feature] Selectors: Add flatten function * [Feature] Selectors: Support filter_map and apply_map functions * [Feature] Store Clickhouse data outside of lua alloc * [Feature] Support caching for encrypted files and macros * [Feature] Support images when extracting urls * [Feature] Support more hyperscan flags * [Feature] Support protocol flags * [Feature] URL: Apply stringprep to hostnames to filter garbage * [Feature] Upstreams: Add lazy resolving logic to all upstreams * [Feature] Upstreams: Set noresolve flag on numeric upstreams * [Feature] Use `scores` in apply section * [Feature] Use maps logic from lua_maps for multimap * [Feature] Use random monitored in rbl module * [Feature] lua_scanners - add Razor support * [Fix] Add another safe-guard in urls processing * [Fix] Add debug to ssl, fixed write hangs * [Fix] Add missing groups to C callback symbols * [Fix] Add more checks for ghosts symbols * [Fix] Allow to enable or add new actions via settings * [Fix] Allow to set 0 size for spf/dkim caches * [Fix] Another bunch of fixes towards protocol mess * [Fix] Another fix to deal with bad URLs * [Fix] Arc: Another bunch of fixes for arc signing * [Fix] Arc: More arc signing fixes * [Fix] Avoid another overflow in fpconv * [Fix] Clickhouse: Fix quoting * [Fix] Clickhouse: Fix retention query quoting * [Fix] Distinguish empty and non-empty prefilters * [Fix] Distinguish remote and local addrs parsing * [Fix] Do not assert if length of sig is bad, just fail verification * [Fix] Do not assert if we have broken mime boundary in the headers * [Fix] Do not call implicit strlen to avoid issues * [Fix] Do not count images urls when checking url regexps for compatibility * [Fix] Do not output rbl suffix in symbol option * [Fix] Do not use config pool to avoid issues with double reload * [Fix] Do not use ephemeral string * [Fix] Do not use lightuserdata for traceback * [Fix] Do not use priority in metric registration * [Fix] Emails: Check email sanity before testing on BL * [Fix] Emails: Fix misprint in key name * [Fix] Escape utf in regexp to dodge ragel/hyperscan issue * [Fix] Extend task_timeout to postfilters stage * [Fix] Fix ARC signing after fixing another bug in it... * [Fix] Fix AV scan logic * [Fix] Fix DMARC_NA behaviour in case of no valid policies * [Fix] Fix LRU hash iteration logic * [Fix] Fix alignment mess * [Fix] Fix configuring symbols without scores * [Fix] Fix disabling of the actions * [Fix] Fix dkim signing exceptions * [Fix] Fix embedded images linking logic * [Fix] Fix events leak * [Fix] Fix eviction corner case * [Fix] Fix fuzzy image score calculation #2962 * [Fix] Fix hang in fuzzy_learn when explicit rotation is set * [Fix] Fix headers propagation logic * [Fix] Fix hearbeats restart issue * [Fix] Fix history reset * [Fix] Fix log parameter * [Fix] Fix lua_ip_equal logic * [Fix] Fix more issues with nested messages + tests * [Fix] Fix normalization of non-alphabet based languages * [Fix] Fix offsets when parsing message/rfc822 in multipart * [Fix] Fix options in rbl symbols * [Fix] Fix out of bound access in lua logger * [Fix] Fix out-of-bound read in qp decode * [Fix] Fix parent CTE propagation * [Fix] Fix parsing of the received headers with empty part * [Fix] Fix pending checks for events * [Fix] Fix printing of NULL pointer with fixed length * [Fix] Fix race condition in watcher handler * [Fix] Fix read-after-end in quoted printable decoding * [Fix] Fix redis sentinel support * [Fix] Fix registry leak in case of DNS errors * [Fix] Fix reload logic * [Fix] Fix sending of large entries via HTTPS * [Fix] Fix settings reload * [Fix] Fix some more corner cases for fpconv * [Fix] Fix trie code when there are regexps and Hyperscan is absent * [Fix] Further fixes to printing of the FP numbers * [Fix] Fuzzy_check: Fix timeouts * [Fix] Grrr, fix empty ip case * [Fix] Html: Fix processing of fjlig entity * [Fix] Lang_det: Try better to distinguish Chinese and Japanese * [Fix] Lua_mime: Fix reversed extensions map * [Fix] Lua_task: Fix message-less API * [Fix] Lua_tcp: Report connection failures * [Fix] Lua_tcp: Various fixes and debugging improvements * [Fix] Metadata_exporter: This plugin is idempotent not a postfilter * [Fix] More fixes to extract_specific_urls * [Fix] More stages fixes * [Fix] Neural: Another bunch of fixes * [Fix] Neural: use version in ANN key profile * [Fix] Postpone lua state destruction to allow lua dtors to be used * [Fix] Prefer surbl/emails rule on rbl to preserve compatibility * [Fix] RBL: Fix behaviour of emails_domainonly * [Fix] Ratelimit: Fix dynamic score * [Fix] Rbl: Fix emailbl functions * [Fix] Really fix hyperscan workaround * [Fix] Set sanity limits for pcre2 * [Fix] Settings: Fix settings check flags * [Fix] Sort keys when getting data from Lua when filling rules * [Fix] Statistics: Do not query Redis tokens when there are no learns * [Fix] Stop IO event on write finished in http connection * [Fix] Use heuristically detected text parts data * [Fix] Various fixes to QP encoding algorithm * [Fix] Various fixes to SSL state machine handler * [Fix] Various fixes to asn module * [Fix] Workaround for empty charset in rfc2231 encoding * [Project] Switch from torch to KANN * [Project] Add heartbeat events * [Project] Add preliminary support of the Kaspersky Scan Engine * [Project] Add preliminary version of maps expressions * [Project] Add preprocessed settings to the config structure * [Project] Add simple forward propagation function * [Project] Add small helpers for migration simplifications * [Project] Allow to replace body in milter * [Project] Bundle libev * [Project] First refactoring step libevent->libev * [Project] Implement syntax highlighting for Lua * [Project] Lua_magic: Adopt lua_magic stuff in mime_types * [Project] Remove libfann, gd and other unsupported stuff * [Project] Remove torch * [Project] Rework upstreams * [Rework] Allow execution of async events when hs compiles regexps * [Rework] Bayes expiry: eliminate `default` expiration mode * [Rework] Dkim: Remove signing code * [Rework] Dkim_signing: Move sign condition to dkim_signing * [Rework] Do not lowercase all data send to ClickHouse * [Rework] Drop url tags * [Rework] Eliminate lua_squeeze as it has shown no improvements * [Rework] Eliminate virtual scan time as it is useless * [Rework] Lua core: Use lightuserdata to index classes * [Rework] Lua_util: Another rework for extract_specific_urls * [Rework] Migrate from ip_score to reputation * [Rework] Move mime modification functions to lua_mime library * [Rework] Rbl: Major whitelisting logic rework * [Rework] Remove deprecated plugins * [Rework] Remove log helper worker * [Rework] Remove rspamd.classifiers.lua * [Rework] Rename filter.h to a more sane name * [Rework] Reorganise selectors implementation * [Rework] Replace linenoise with replxx * [Rework] Reputation: Remove ipnet from the ip reputation * [Rework] Reputation: Slashing - change name of symbols * [Rework] Rework children operations * [Rework] Rework config reload * [Rework] Rework expression API * [Rework] Rework image urls processing * [Rework] Rework initialisation to reduce static leaks count * [Rework] Rework request headers processing * [Rework] Slashing: Change versioning schema - move to 2.0 * [Rework] Slashing: Turn off postfilters when passthrough result is set * [Rework] Start moving to replxx * [Rework] Stop support of signed HTTP maps to simplify code * [Rework] Store ASN as UInt32 in ClickHouse * [Rework] Url_redirector: Rewrite plugin * [Rework] Use a dedicated library for autolearn * [Rework] Use libsodium instead of hand crafted crypto implementations * [Rework] Use opaque structure to store a table of mime headers * [Rules] Add dedicated bitcoin addresses filter rule * [Rules] Add more detection to LEAKED_PASSWORD_SCAM * [Rules] Catch LTC addresses * [Rules] Reduce weight of RSPAMD_EMAILBL * [Rules] Rework LEAKED_PASSWORD_SCAM rule one more time Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
cc1aef5ea4 | [Minor] Add Changelog for 1.9.4 | 5 years ago |
|
Vsevolod Stakhov
|
fe91396af3
|
Release 1.9.3
* [Conf] Add IP_SCORE_FREEMAIL composite rule
* [Feature] Add cryptobox method to generate dkim keypairs
* [Feature] Add fast hashes to lua cryptobox hash
* [Feature] Add least passthrough results
* [Feature] Allow oversign if exists mode
* [Feature] Clickhouse: Modernise table initial schema
* [Feature] Implement IUF interface for specific fast hashes
* [Feature] Lua_util: Allow to obfuscate different fields
* [Feature] Tune memory management in Rspamd and Lua
* [Fix] Avoid buffer overflow when printing long lua strings
* [Fix] Change the default oversigning headers to a more sane list
* [Fix] Clickhouse: Do not store digest as it is not needed now
* [Fix] Clickhouse: Fix lots of storage issues
* [Fix] Clickhouse: Support custom actions
* [Fix] Deny URLs where hostname is bogus
* [Fix] Do not blacklist mail by SPF/DMARC for local/authed users
* [Fix] Fix DoS caused by bug in glib
* [Fix] Fix UCL parsing of the multiline strings
* [Fix] Fix buffer overflow when printing small floats
* [Fix] Fix init code for servers keypairs cache
* [Fix] Fix issue with urls with no tld (e.g. IP)
* [Fix] Fix memory in arc signing logic
* [Fix] Fix memory leak in language detector during reloads
* [Fix] Fix mixed case content type processing
* [Fix] Fix processing of the ip urls in file
* [Fix] Fix use after free
* [Fix] HTML: Fix `size` attribute processing
* [Fix] Hum, it seems that
|
5 years ago |
|
Vsevolod Stakhov
|
fb577bd5a3
|
Release 1.9.2
* [Conf] Allow to load users plugins from plugins.d * [Conf] oversign openpgp and autocrypt headers * [Feature] Add SPF FFI library for Lua * [Feature] Add more verbosity for SPF caching * [Feature] Antivirus: Handle encrypted files specially * [Feature] Clickhouse: Slashing - add new fields to CH * [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table * [Feature] Dkim_signing: Allow to use new options as maps * [Feature] Import fpconv library * [Feature] Lua_maps: Allow static regexp and glob maps * [Feature] Parse ical files * [Feature] Rspamadm: Add dns_tool utility * [Feature] Store SPF records digests * [Feature] Use fpconv girsu2 implementation for printing floats * [Fix] Clickhouse: Use integer seconds when inserting rows * [Fix] Fix floating point printing * [Fix] Fix processing of embedded urls * [Fix] Lua_clickhouse: Fix CH errors processing * [Fix] Make spf digest stable * [Fix] Properly detect encrypted files in zip archives * [Fix] Slashing: Store times in GMT timezone in ClickHouse * [Rules] Add additional conditions to perform BTC checks * [Rules] Fix pay-to-hash addresses validation Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
38ca7b5e10
|
Release 1.9.1
* [Conf] Add vendor groups for symbols * [Feature] Add `rspamadm template` command * [Feature] Allow to add messages from settings * [Feature] Allow unconnected DNS servers operations * [Feature] Check limits after being set, migrate to uint64 * [Feature] Greylist: Allow to disable greylisting depending on symbols * [Feature] Improve lua binary strings output * [Feature] Mime_types: Implement user configurable extension filters * [Feature] Mime_types: When no extension defined, detect it by content * [Feature] Preprocess config files using jinja templates * [Feature] Replies: Filter replies sender to limit whitelisting to direct messages * [Feature] Treat all tags with HREF as a potential hyperlinks * [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM * [Fix] Add crash safety for HTTP async routines * [Fix] Another fix for Redis sentinel * [Fix] Clickhouse: Fix table schema upload * [Fix] Core: Fix squeezed dependencies handling for virtual symbols * [Fix] Finally fix default parameters parsing in actions section * [Fix] Fix ES sending logic (restore from coroutines mess) * [Fix] Fix finishing script for clickhouse collection * [Fix] Fix priority for regexp symbols registriation * [Fix] Fix various issues found by PVS Studio * [Fix] Initialize lua debugging earlier * [Fix] Neural: Fix training * [Fix] Rework cached Redis logic to avoid sentinels breaking * [Fix] SURBL: Fix regression in surbl module * [Fix] Fix double signing in the milter * [Project] Add support of HTTP proxy in requests * [Rework] Change lua global variables registration * [Rework] Rework HTML content urls extraction * [Rework] Start rework of aliasing in Rspamd * [WebUI] Combine Scan and Learning into one tab * [WebUI] Fix symbol score input type * [WebUI] Show grayed out pie * [WebUI] Update Throughput summary values dynamically Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
0e2437493e
|
Release 1.9.0
* [Conf] Add missing includes
* [Conf] Move to options
* [Conf] Rbl: DWL is actually special whitelist
* [Conf] Relax some uribl rules
* [Conf] Remove abuse.ch
* [CritFix] Html: Entities are not valid within tag params values
* [Feature] Add `rspamadm mime sign` tool
* [Feature] Add configgraph utility
* [Feature] Add dedicated ZW spaces detection for URLs
* [Feature] Add flag to url object when visible part is url_like
* [Feature] Add method task:lookup_words
* [Feature] Add pyzor support (by crosenberg)
* [Feature] Allow to add upstream watchers to Lua API
* [Feature] Allow to set rewrite subject pattern from settings
* [Feature] Better escaping of unicode
* [Feature] Clickhouse: Allow to store subject in Clickhouse
* [Feature] Core: Add QP encoding utility
* [Feature] Core: Add libmagic detection for all parts
* [Feature] Core: Add support for gzip archives
* [Feature] Core: Allow to construct scan tasks from raw data
* [Feature] Core: Detect charset in archived files
* [Feature] Core: Ignore and mark invisible spaces
* [Feature] Core: Normalise zero-width spaces in urls
* [Feature] Core: Process data urls for images
* [Feature] Core: Relax quoted-printable encoding
* [Feature] Core: Support RFC2231 encoding in headers
* [Feature] Core: Support telephone URLs
* [Feature] Core: allow to emit soft reject on task timeout
* [Feature] DCC: Add bulkness and reputation checks to dcc
* [Feature] Elastic: Modernize plugin
* [Feature] Export visible part of url to lua
* [Feature] Fuzzy_storage: add preliminary support of rate limits
* [Feature] HTML: Specially treat data urls in HTML
* [Feature] Implement event watchers for upstreams
* [Feature] Implement includes tracing in Lua
* [Feature] Improve dkim part in configwizard
* [Feature] Lua_scanners: Add VadeSecure engine support
* [Feature] Lua_task: Add flexible method to get specific urls
* [Feature] Mime_types: Add MIME_BAD_UNICODE rule
* [Feature] Mime_types: Use detected content type as well
* [Feature] Plugins: Add preliminary version of the external services plugin
* [Feature] Query sentinel on master errors
* [Feature] Regexp: Allow local lua functions in Rspamd regexp module
* [Feature] Rspamadm: Allow to append footers to plain messages
* [Feature] Rspamadm: Allow to rewrite headers in messages
* [Feature] Selectors: Add `ipmask` processor
* [Feature] Settings: Allow hostname match
* [Feature] Settings: Allow local when selecting settings
* [Feature] Settings: Allow multiple selectors
* [Feature] Settings: Allow to inverse conditions
* [Feature] Support User-Agent in HTTP requests
* [Feature] Support
|
5 years ago |
|
Vsevolod Stakhov
|
1b9f2f3026
|
Release 1.8.3
* [CritFix] Make flags mutually exclusive for mime parts * [CritFix] Strictly deny unencoded bad utf8 sequences in headers * [Feature] Add Kaspersky antivirus support * [Feature] Add method to get dkim results * [Feature] Add more words regexp classes * [Feature] Allow to choose words format in `rspamadm mime` * [Feature] Allow to get all types of words from Lua * [Feature] Allow to get task flags in C expressions * [Feature] Allow to require encryption when accepting connections * [Feature] Ignore bogus whitespaces in the words * [Feature] Implement more strict configuration tests * [Feature] Improve SPF results in Authentication-Results * [Feature] Support ClickHouse database * [Fix] Add failsafety for utf8 regexps * [Fix] Do not trigger BROKEN_CONTENT_TYPE on innocent text parts * [Fix] Emit error if connection has been terminated with no stop pattern * [Fix] Fix boundaries checks in embedded messages * [Fix] Fix double free * [Fix] Perform policy downgrade on sample out, add tests * [Fix] Properly escape utf8 regexps in hyperscan mode * [Fix] Selectors - attachments args condition * [Fix] Some fixes for raw parts * [Fix] Treat learning errors as non-fatal * [Fix] Use tld when looking for DKIM domains * [Project] Words unicode structure rework * [Project] Add preliminary Redis Sentinel support * [Project] Improve Authentication-Results header * [Project] Rework DKIM checks results * [Project] Use more generalised API to produce meta words Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
dcab7362e0
|
Release 1.8.2
* [Conf] Add DWL support in the default configuration * [Conf] Disable rspamd_update by default (again) * [Conf] Fix configuration sample for ratelimit * [CritFix] Disable broken url tags by default * [CritFix] Fix \0 processing when doing RSA sign * [CritFix] Fix adding symbols to their primary groups * [Feature] Add `rspamadm cookie` utility * [Feature] Add specialised functions for generating encrypted cookies * [Feature] Add support of cookies in replies module * [Feature] Add support of words regexps * [Feature] Allow to add 3rd party clang plugins * [Feature] Allow to create lua regexps from glob or plain patterns * [Feature] Allow to set custom limits for upstream lists * [Feature] Detect orphaned parts and attach them to message * [Feature] Filter tokens in bayes * [Feature] Fold b= value when doing arc sealing * [Feature] Ignore cookies in the future and too old in the past * [Feature] Skip stop words in statistics * [Feature] Store stop words and allow to query them * [Feature] Support query arguments in controller's custom commands * [Feature] Tune upstream limits in Rspamd proxy * [Feature] Use different callback symbols for different uribls * [Feature] Write DKIM selector in dkim allow/reject symbols * [Fix] Add obs_fws state support to eoh state machine * [Fix] Add sanity check when applying mime boundaries heuristic * [Fix] Antivirus - virus names with 0 were recognized as tables * [Fix] Disable headernames in bayes temporarily * [Fix] Do not allow syntax errors in include files... * [Fix] Do not allow to merge an object with an array (or vice versa) * [Fix] Don't perform forged recipients check for missing recipients * [Fix] Fix DKIM based RBLs * [Fix] Fix actrie implementation (sync from upstream), fixed OOB read * [Fix] Fix explicit methods call in selectors * [Fix] Fix extraction of additional parts * [Fix] Fix finalization for internal plugins * [Fix] Fix override_defaults function * [Fix] Fix squeezed symbols when using settings * [Fix] Fix urls insertion in Clickhouse module * [Fix] Furhter fixes to ratelimits logic * [Fix] Ignore signatures when looking for boundaries * [Fix] Properly set learned count * [Fix] Really fix ratelimits configuration and work * [Fix] Remove ambigious format flag from printf * [Fix] Restore URLs exporting in ClickHouse plugin * [Fix] Rework bayes calculations... * [Fix] Switch from chi-square to naive for large Fisher value * [Fix] Treat normal password as enable password if there is no enable password * [Fix] Use proper syntax for making DNS requests * [Fix] Various fixes in embedded plugins * [Project] Change fuzzy check selection logic to lua_fuzzy library * [Project] Rework async events and symbols * [Project] Move all metatokens in Bayes to lua_stat from C * [WebUI] Add history rows per page control Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
6fe627f31f
|
Release 1.8.1
* [CritFix] Fix options insertion * [CritFix] Fix words decay one more time (affects long messages) * [CritFix] Increase default words_decay * [CritFix] Plug memory leak in redis pool * [Feature] Add `check_violation` feature to DKIM/ARC signing * [Feature] Add only unique elements to Clickhouse url arrays * [Feature] Allow `g+:` and `g-:` composite atoms * [Feature] Allow dkim domains check in surbl * [Feature] Allow maps with HTTP auth * [Feature] Allow to disable actions by users settings * [Feature] Extend whitelisting options * [Feature] Store url object in images * [Feature] Use verdict instead of the plain action in plugins * [Fix] Allow to call fstring append with NULL string * [Fix] DCC - luacheck * [Fix] Do not load torch on each rspamadm invocation * [Fix] Fix boundaries detection and rework stop words algorithm * [Fix] Fix dependencies for DNS_SIGNED symbol * [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit * [Fix] Fix groups mess * [Fix] Fix groups mess * [Fix] Fix parsing address with comments * [Fix] Fix resolving in DMARC reports * [Fix] Fix various issues with parsing of the received headers * [Fix] Fix watchers issue in lua_tcp when doing no resolving * [Fix] Plug memory leak in language detector (affects reloads) * [Fix] Remove one letter stop words * [Fix] Slashing: backport chunk logic from libucl * [Fix] Stop libevent from using cached time in rspamadm * [Fix] Try to fix watchers chaining * [Fix] Various fixes in redis sync interface * [Fix] ip_score - respect check_authed and check_local settings from config * [Project] Rework passthrough actions * [Project] Clustering module * [Rework] Always create result for a task * [Rework] Completely rewrite DMARC checks logic * [Rework] Rework and fix whitelist plugin * [WebUI] Add symbols sorting buttons * [WebUI] Change symbols order without updating history * [WebUI] Colorize symbols * [WebUI] Do not display password form when secure_ip is set * [WebUI] Fix symbol description tooltips display * [WebUI] History: add sorting by symbol score value Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
f7f7588e22
|
Release 1.8.0
* [Feature] Add arguments schemas to processors and extractors * [Feature] Add functional selectors library * [Feature] Add generic selector to reputation module * [Feature] Add more ratelimits: by digest, by attachments data, by filenames * [Feature] Add preliminary stop words detection support * [Feature] Add pure Lua debugm function * [Feature] Add schema validation for Redis settings * [Feature] Add selectors combine function * [Feature] Add some recursion protection to lua logger * [Feature] Add support for Lua API tracing * [Feature] Allow to apply schema to arguments * [Feature] Allow to get dkim signing data directly from HTTP headers * [Feature] Allow to reuse existing authentication results * [Feature] Cache selectors results in re runtime * [Feature] Implement new text tokenizer based on libicu * [Feature] Integrate selectors framework to multimap * [Feature] Relax FORGED_RECIPIENTS * [Feature] Support (almost) all html entities * [Feature] Support adding and deletion of recipients in the milter block * [Feature] Support gathering HTTP body from fragments in lua_http * [Feature] Support multi flag in regexp and glob maps * [Feature] Support selectors in ratelimit module * [Feature] Support selectors in settings * [Feature] Use khash in HTML parser * [Feature] Use pure Lua debugm function * [Fix] Add fail-safety for destroying sessions * [Fix] Allow to add result-less fake DNS records * [Fix] Another try to fix race conditions on config unload * [Fix] Call Lua callback on DNS timeouts * [Fix] Deprecate task:inc_dns_req as it is redundant * [Fix] Do not allow events deletions on cleanup * [Fix] Do not try to process skipped messages * [Fix] Fix HTTP requests with no body * [Fix] Fix another cleanup race condition * [Fix] Fix bug in processing of pcre regexps * [Fix] Fix byte array allocation in the pool * [Fix] Fix crashes on task cleanup * [Fix] Fix dynamic buckets in ratelimits * [Fix] Fix endless loop when waiting for Rspamd to stop * [Fix] Fix lua_util.str_split in case of delimiters set * [Fix] Fix more issues with watching of async events * [Fix] Fix stop words detection and loading logic * [Fix] Fix various corner cases for language detection * [Fix] Fix watchers in lua_tcp * [Fix] Fix words decay algorithm * [Fix] Implement watchers replacement to handle nested calls * [Fix] Save faked code into fake dns record * [Fix] Show the proper frame when using lua_util.debugm * [Fix] Use fake dns records in tests * [Fix] Use unicode replacements for HTML entities * [Fix] fixed "cannot find dependency on symbol 1" issue when using replaced symbols in spamassassin rules * [Fix] partition_id is not available in old versions of CH * [Project] Add implicit conversion logic to selectors * [Project] Add initial support for selectors in regexps * [Project] Add method concept * [Project] Further changes in unicode operations * [Project] Implement Clickhouse migrations * [Project] Implement implicit conversions to userdata * [Project] Implement insert method * [Project] Implement selectors registration for regular expressions * [Project] Implement selectors support in re_cache * [Project] Improve language detector: cleanup unused files, categorize * [Project] Migrate CH data to a fat table * [Project] Rework selectors logic * [Project] Start Clickhouse utilities library * [Project] Start unicode rework * [Project] coroutine threaded model for API calls: thread pool * [Rework] Move phishtank to a DNS based service * [Rework] Rework Clickhouse plugin to use the new API * [Rework] Rework language detector * [Rework] Rework utf content processing in text parts * [WebUI] Add progress bar for AJAX requests * [WebUI] Avoid errors table reinitialization * [WebUI] Avoid history table reinitialization * [WebUI] Avoid throughput summary table reinitialization * [WebUI] Destroy summary table on disconnect * [WebUI] Fix "auth" request URL * [WebUI] Fix disabling and hiding controls on page reload * [WebUI] Fix maps loading from neighbours * [WebUI] Fix symbols sorting by score * [WebUI] Fix tables destroying * [WebUI] Fix throughput data consolidation * [WebUI] Fix upload buttons disabling * [WebUI] Notify user on module loading failure * [WebUI] Update FooTable 3.1.4 -> 3.1.6 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
f63ec0e39b
|
Release 1.7.9
* [CritFix] Fix caseless comparison of equal length strings * [Feature] Add HTTP basic auth support to elastic and clickhouse plugins * [Feature] Add SPF selector to reputation * [Feature] Add support of the fallback backends for HTTP maps * [Feature] Allow to print full mime structure when extracting mime data * [Feature] Allow to split symbols in reputation plugin * [Feature] Check attachments only on AV scanners in attachments_only mode * [Feature] Disable all SSL checks if ssl_no_verify flag is set * [Feature] Implement parsing of scoped IPv6 addresses * [Feature] Improve `rspamc counters` output * [Fix] Add sanity checks when expanding SPF macros * [Fix] Allow to parse SA rules with no spaces around =~ (dirty hack) * [Fix] Avoid one extra byte writing * [Fix] Deal with direct hash table * [Fix] Detect empty text part as text, not HTML * [Fix] Do not reduce map watch timeout for mixed http/file maps * [Fix] Fix HTML part detection heuristic * [Fix] Fix double free in redirectors cleanup * [Fix] Fix legacy history handling in the controller * [Fix] Fix messages insertion * [Fix] Fix sending string method * [Fix] Fix statconver command line arguments * [Fix] Fixed argument checking for being null * [Fix] Fixed issues reported by luacheck * [Fix] Freeze updates queue when do actual storage update * [Fix] HTTP map hash is per-backend and not per-map * [Fix] Plug memory leak in fuzzy updates * [Fix] Prefer 'MTA-Name' when producing authentication results * [Fix] Replace bad unicode sequences instead of stopping on them * [Fix] Set classifier version on learning * [Project] Reworked ratelimits * [Project] Apply topological sorting for symbols in Rspamd * [Project] Remove global contexts from C modules * [Project] Move performance critical hash tables to khash * [WebUI] Avoid unused indexes * [WebUI] Do not execute `on_success` callback * [WebUI] Fix history reset for "All SERVERS" (#2346) * [WebUI] Fix query URL for selected server * [WebUI] Fix symbols display in legacy history, * [WebUI] Hide symbols order selector for legacy history * [WebUI] Refactor query functions into one * [WebUI] Remove previously-attached event handlers * [WebUI] Save symbols to the selected server * [WebUI] Unify arguments of query functions * [WebUI] Use common query functions to get graph data * [WebUI] Use common query functions to save symbols Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
4c80975059
|
Release 1.7.8
* [Feature] Add more extended statistics about fuzzy updates * [Feature] Add more non-conformant Received headers support * [Feature] Add preliminary function to get fuzzy hashes from text in Lua * [Feature] Allow to configure AV module rejection message * [Feature] Implement fuzzy hashes extraction in mime tool * [Feature] Improve WHITE_ON_WHITE rule * [Feature] Improve integer -> string conversion * [Feature] Reuse maps in multimap module more aggressively * [Fix] Avoid race condition in skip map as pool lifetime is not enough * [Fix] Eliminate all specific C plugins pools * [Fix] Fix DKIM check rule if DNS is unavailable * [Fix] Fix build where ucontext is defined in ucontext.h * [Fix] Fix crash in base url handling * [Fix] Fix descriptors leak in sqlite3 locking code * [Fix] Fix messages quarantine * [Fix] Fix padded numbers printing * [Fix] Fix race condition on maps reinit * [Fix] Fix regexp functions when no data is passed * [Fix] Fix specific urls extraction * [Fix] Fix styles propagation * [Fix] Improve resetting of the limit buckets * [Fix] Initialize sqlite3 properly * [Fix] Work with broken resolvers in resolv.conf * [Project] Implement HTTP maps caching * [Project] Refresh fuzzy hashes when matched * [Project] Add logic to deduplicate fuzzy updates queue * [WebUI] Add missed declarations * [WebUI] Avoid using "undefined" property * [WebUI] Do not accept passwords containing control characters * [WebUI] Do not redeclare variables * [WebUI] Enable strict mode, * [WebUI] Fix variable assignment * [WebUI] Initialize variables at declaration * [WebUI] Remove duplicated path from RequireJS config * [WebUI] Remove unused block * [WebUI] Remove unused variable * [WebUI] Remove unused variables * [WebUI] Use self-explanatory notation * [WebUI] Use type-safe equality operators Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
a20e33c22d
|
Release 1.7.7
* [CritFix] Check NM part of pubkey to match it with rotating keypairs
* [CritFix] Do not overwrite PID of the main process
* [CritFix] Fix maps after reload
* [CritFix] Fix maps race conditions on reload
* [CritFix] Fix shmem leak in encrypting proxy mode
* [Feature] Add a concept of ignored symbols to avoid race conditions
* [Feature] Add ability to print bayes tokens in rspamadm mime
* [Feature] Add method to get statistical tokens in Lua API
* [Feature] Add preliminary mime stat command
* [Feature] Add rspamadm mime tool
* [Feature] Add urls extraction tool
* [Feature] Address ZeroFont exploit
* [Feature] Allow rspamadm mime to process multiple files
* [Feature] Allow to extract words in `rspamadm mime`
* [Feature] Allow to print mime part data
* [Feature] Allow to show HTML structure on extraction
* [Feature] Distinguish IP failures from connection failures
* [Feature] Improve output for mime command
* [Feature] Improve styles propagation
* [Feature] Main process crash will now cleanup all children
* [Feature] Preload file and static maps in main process
* [Feature] Print stack trace on crash
* [Feature] Process font size in HTML parser
* [Feature] Propagate content length of invisible tags
* [Feature] Read ordinary file maps in chunks to be more safe on rewrites
* [Feature] Support base tag in HTML
* [Feature] Support more size suffixes when parsing HTML styles
* [Feature] Support opacity style
* [Fix] Another fix for nested composites
* [Fix] Fill nm id in keypairs cache code
* [Fix] Fix colors alpha channel handling
* [Fix] Fix destruction logic
* [Fix] Fix double free
* [Fix] Fix maps preload logic
* [Fix] Fix nested composites process
* [Fix] Fix proxying of Exim connections
* [Fix] Fix reload crash
* [Fix] Fix rspamadm -l command
* [Fix] Update
|
6 years ago |
|
Vsevolod Stakhov
|
d9c84b4bd0
|
Release 1.7.6
* [CritFix] Fix multiple neural networks support * [Feature] Add decryption function to keypair command * [Feature] Add gzip compression for HTTP requests in elastic module * [Feature] Add gzip methods to lua util * [Feature] Add maps based on Top Level Domains * [Feature] Add pubkey checks for dkim_signing * [Feature] Add support of fake DNS records * [Feature] Add tool to encrypt files * [Feature] Allow to add symbols using settings directly * [Feature] Allow to match private and public keys for DKIM signatures * [Feature] Allow to set task flags via settings * [Feature] Allow to specify fake DNS address from the config * [Feature] Implement signatures verification using rspamadm keypair * [Feature] Implement signing using `rspamadm keypair` * [Feature] Improve error reporting for DKIM key access issues * [Feature] Provide $HOSTNAME variable in UCL * [Feature] Rework levenshtein distance computation * [Feature] Split message parsing and processing * [Feature] Support ED25519 DKIM signatures * [Feature] Support encrypted configs in UCL * [Feature] Suppress duplicate warning on very large radix tries * [Feature] Use OSB to combine header names * [Fix] Cleanup maps data on shutdown * [Fix] Fix '~' behaviour in composites * [Fix] Fix HTTP maps updates * [Fix] Fix NIST signatures * [Fix] Fix RFC822 comments when processing a mime address * [Fix] Fix double free * [Fix] Fix dynamic settings application * [Fix] Fix for CommuniGate Pro maillist * [Fix] Fix keypair creation method to actually create keypair... * [Fix] Fix matching patterns with no paths * [Fix] Fix memory leak in parsing comments * [Fix] Fix parsing of urls with numeric password * [Fix] Fix plugins intialisation in configwizard * [Fix] Fix potential crash on reload * [Fix] Fix potential race condition for a finished HTTP connections * [Fix] Fix race-condition leak on processes reload * [Fix] Fix signing in openssl mode * [Fix] Free language detector structures * [Fix] Relax alignment requirements * [Fix] Send DMARC reports compressed * [Fix] Try to fix leak in dmarc module * [Fix] Try to plug memory leak in metric exporter * [Project] Convert rspamadm subcommands to Lua * [WebUI] Display smtp sender/recipient in history * [WebUI] Fix elements disabling in "Symbols" tab * [WebUI] Limit recipients list in history column to 3 * [WebUI] Match envelope and mime addresses following in arbitrary order * [WebUI] Update column header * [WebUI] Wrap addresses in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
341222861d
|
Release 1.7.5
* [Conf] Add MSBL proposed return codes * [Conf] Add additional groups for policies * [CritFix] Do not use volatile Lua strings as UCL keys * [Feature] Add ability to add fuzzy hashes to headers * [Feature] Add function to extract most meaningful urls * [Feature] Add rule to block mixed text and encrypted parts * [Feature] Allow multiple groups for symbols * [Feature] Allow to disable lua squeezing logic * [Feature] Allow to get multipart children in Lua * [Feature] Allow to insert multiple headers from milter headers * [Feature] Allow to print scores in subject and further extensions * [Feature] Be more error-prone in squeezed rules * [Feature] Support multiple return codes in emails module * [Feature] Use EMA for calculating averages * [Feature] Use common jit cache for all regexps * [Feature] support for CommuniGate Pro self-generated messages * [Fix] Allow to have multiple values for headers as arrays * [Fix] Do not open sockets for disabled workers * [Fix] Fix AuthservId * [Fix] Fix base64 folding in Lua API * [Fix] Fix build on non-x86 platforms * [Fix] Fix cached maps logic * [Fix] Fix compatibility with old maps query logic * [Fix] Fix crash if skip_map is used * [Fix] Fix importing static maps from UCL * [Fix] Fix parsing of unix sockets * [Fix] Fix raw_mime regexp on HTML part with no text content * [Fix] Fix tables logging * [Fix] Fix vertical tab handling in libucl * [Fix] Try to fix frequency counters * [Fix] Use better sharding for ip_score * [Fix] Use multiple results from SURBL DNS reply * [Fix] When doing AV scan select a different server for retransmit Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
72b3e29432
|
Release 1.7.4
* [Conf] Major stock config updates:
- Workers are now specified in a new format worker "type" { ... }
- Enable fuzzy worker to simplify local fuzzy storages configuration
- Bind all workers to localhost by default to avoid security flaws
* [Conf] Make more sane fuzzy_check default settings
* [CritFix] Fix ucl escape for bad symbols
* [Feature] Add failure symbol for AV module
* [Feature] Add lazy expiration mode for new classifier schema
* [Feature] Add preliminary version of maps stats plugin
* [Feature] Allow to block fuzzy requests from specific networks
* [Feature] Allow to change `expire` of live statistics
* [Feature] Distinguish AV failure from clean result
* [Feature] Further improvements of language detector by using khash
* [Feature] Further optimization of the lang_detection
* [Feature] Implement cluster-aware bayes expiry
* [Feature] Implement exclude patterns in rspamc
* [Feature] Implement glob maps in addition to regexp maps
* [Feature] Implement map statistics function for lua API
* [Feature] Implement stop symbols for Clickhouse collection
* [Feature] Support recipients separated by commas
* [Feature] Try harder to upload scripts to the Redis server
* [Feature] Upgrade t1ha distribution
* [Feature] use_domain_sign_inbound
* [Feature] Use scores from maps if `symbols_set` is not defined
* [Fix] Add resolving version of radix map helper
* [Fix] Check URL before adding implicit prefix
* [Fix] Do not check pid/state when using PRNG
* [Fix] Fix CentOS logrotate script for systemd
* [Fix] Fix slash + dot in urls
* [Fix] Fix systemd version of the logrotate script
* [Fix] Propagate key when import implicit array from Lua
* [Fix] Strip spaces from map keys and values
* [Fix] Try to fix a specific case when processing milter protocol
* [Fix] Try to fix crash when a tcp connection cannot be set
* [Fix] Typo use_domain_local --> use_domain_sign_local
* [Fix] Various fixes to once_received module
* [Project] Store hits counters for map elements
Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
|
6 years ago |
|
Vsevolod Stakhov
|
4f481f93bc
|
Release 1.7.3
* [CritFix] Plug bad memory leak in protocol reply * [Feature] Add avx2 codec for base64 * [Feature] Add method to receive all URL flags from Lua API * [Feature] Allow to fold headers on stop characters * [Feature] Allow to set lua_cpath from options * [Feature] Allow to specify custom rejection message in milter * [Feature] Deal with unnormalised Unicode obfuscation * [Feature] Do not detect language twice for relative parts * [Feature] Implement oversigning feature * [Feature] Implement silent logging level to minimize noise in logs * [Feature] Improve URL_IN_SUBJECT rule * [Feature] Use hashing to reduce redis attack surface * [Fix] Add oversigning for the most important headers * [Fix] add 'rewrite subject' to History dropdown * [Fix] Another fix in folding algorithm * [Fix] Do not call multimap addr for parts of addr if filter is presented * [Fix] Do not clean hostname on generic reset * [Fix] Do not create pid file in no-fork mode * [Fix] Fix fold_after case to preserve multiple spaces * [Fix] Fix folding and folding tests * [Fix] Fix hostname usage in milter mode * [Fix] Fix lua RSA verify and its tests * [Fix] Fix metadata exporter send_mail backend (#2124) * [Fix] Fix processing of '\v' in libucl * [Fix] Fix shemaless URLs detection * [Fix] Fix support of multiple headers in sign_header * [Fix] Fix usage of util.parse_mail_address * [Fix] Fix weights of dynamic squeezed rules * [Fix] Leak from bucket before checking the burst * [Fix] Stop using own localtime as DST could be messy in many cases * [Fix] Treat unnormalised URLs as obscured * [Rework] Restore leaky bucket model in ratelimit plugin * [WebUI] Add messages total to throughput summary * [WebUI] Add symbols order selector to history * [WebUI] Config: Load list on demand * [WebUI] Fix modalBody for maps that appear more than once * [WebUI] History: Fix Tooltips on paging, filtering and sorting * [WebUI] Remove a previously-attached event handler * [WebUI] Update D3 to v5.0.0 and jQuery to v3.3.1 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
8073aece67
|
Release 1.7.2
* [Feature] Store emails in Clickhouse * [Feature] Support single quotes in config * [Feature] Use templates when publishing CH schema * [Feature] Improve Docker image * [Fix] Add rounding when printing a lot of FP variables * [Fix] Allow to disable certain actions by assigning null to them * [Fix] Disable results caching * [Fix] Fix disabling of squeezed symbols * [Fix] Fix scan time set * [Fix] Rework logic of actions setting * [Fix] Try to fix various Lua stack issues * [WebUI] Add link tag for favicon.ico * [WebUI] Display hostname:port/path in the page title Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
3ba34325ab
|
Release 1.7.1
* [CritFix] Fix lowercase comparison * [CritFix] Timezone defines seconds WEST UTC not East * [Feature] Add filename to log format * [Feature] Add lua rules squeezing * [Feature] Add related symbols analysis to rspamd_stats * [Feature] Remove upstream `X-Spam: Yes` header by default * [Feature] rspamd_stats: Output progress info on STDERR * [Feature] Whitelist for emails module * [Fix] Do not allow dependencies on self * [Fix] Do not cache metric result * [Fix] Do not trust all issuers as a client certificate * [Fix] Fix dependencies in lua squeeze * [Fix] Fix enabling/disabling squeezed rules * [Fix] Fix enabling/disabling symbols * [Fix] Fix external dependencies * [Fix] Fix processing of a single compressed file * [Fix] Fix some typos * [Fix] Fix various modules in case of empty message * [Fix] Handle callbacks that returns table of options * [Fix] Improve cached action interaction * [Fix] Make dynamic conf more NaN aware * [Fix] Never hide actions from WebUI `configuration` tab * [Project] Implementation of Lua rules squeezing Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
720ae30a31
|
Release 1.7.0
* [Conf] Add bayes_expiry as explicit module * [Conf] Adjust names and weights for neural network plugin * [Conf] Change updates url * [Conf] Default statistics is stored in Redis now * [Conf] Disable fann_redis module by default * [Conf] Fix default elastic configuration * [Conf] Fix double quote position * [Conf] Massive config rework for new structure of symbols and scores * [Conf] Rename Rambler BLs as they are now Rspamd's ones * [Conf] Use dedicated rspamd.com subdomains * [Conf] Use more data from rspamd.com fuzzy storage * [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Another portion of tokenization fixes * [CritFix] Do not send reject messages after set reply * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [CritFix] Fix memory leak in spf caching logic * [CritFix] Fix milter commands pipelining * [CritFix] Fix newlines detection * [CritFix] Fix semicolons parsing in the content type * [CritFix] Plug memory leak in zstd protocol compression * [Feature] Add ability to match score in force_actions module * [Feature] Add aes-rng PRF to libottery * [Feature] Add 'composites' debug module * [Feature] Add concept of experimental modules * [Feature] Add DKIM trace symbol * [Feature] Add EBL to the default config * [Feature] Add expected ip check for emails plugin * [Feature] Add framework to manage Redis scripts * [Feature] Add framing for the new reputation generic plugin * [Feature] Add function to show plugins stat * [Feature] Add gzip compression support for clickhouse module * [Feature] Add gzip compression support for rspamd controller * [Feature] Add gzip support when sending lua http requests * [Feature] Add json output for rspamd_stats * [Feature] Add method to do a synchronous Redis connection * [Feature] Add method to get all content-type attributes in Lua * [Feature] Add `-m` flag to configdump to show modules states * [Feature] Add mime types to extensions map * [Feature] Add more features to rescore utility * [Feature] Add more gtube like patterns to test other spam actions * [Feature] Add more metafunctions, improve logging * [Feature] Add more text attributes * [Feature] Add new configwizard command to rspamadm * [Feature] Add new tooling for stats conversation * [Feature] Add old groups migration tool * [Feature] Add plugins state variable * [Feature] Add preliminary ecdsa keys support in DKIM * [Feature] Add preliminary support of idempotent symbols * [Feature] Add Redis server wizard * [Feature] Add routine to convert old style stats to a new one * [Feature] Add some sanity checks for actions and controller * [Feature] Add statistic convertation module to configwizard * [Feature] Add sugestions logic to mempool allocator * [Feature] Add support of config transform in Lua * [Feature] Add timeout to rspamc when doing corpus test * [Feature] Add tooling to convert bayes schemas * [Feature] Add torch conditional to configuration * [Feature] Add torch-decisiontree package * [Feature] Add torch-optim contrib package * [Feature] Add TTL autodetection * [Feature] Add urls reputation to the reputation framework * [Feature] Allow floating and negative values in expressions limits * [Feature] Allow multiple CTs in full extensions map * [Feature] Allow multiple fann rules * [Feature] Allow randomly select User-Agent from a list * [Feature] Allow rspamadm commands to export methods in Lua * [Feature] Allow rule specific min_bytes in fuzzy check * [Feature] Allow to adjust symbols scores from Lua * [Feature] Allow to attach stat signature to messages * [Feature] Allow to change SMTP from via milter headers * [Feature] Allow to configure monitored * [Feature] Allow to create directories in Lua API * [Feature] Allow to disable torch and skip train samples for ANN * [Feature] Allow to discard messages dynamically * [Feature] Allow to enable/disable languages from the detector * [Feature] Allow to generate DKIM keys from rspamadm API * [Feature] Allow to get CPU flags from Lua * [Feature] Allow to have high precision timestamps in logs * [Feature] Allow to insert headers into specific position * [Feature] Allow to limit redirector requests per task * [Feature] Allow to load and use dynamic ANNs with torch * [Feature] Allow to quarantine rejected messages using milter interface * [Feature] Allow to receive signing keys from mempool vars * [Feature] Allow to reserve elements in libucl * [Feature] Allow to reuse signal handlers chains * [Feature] Allow to set custom mempool variables from settings * [Feature] Allow to set headers from settings * [Feature] Allow to set Settings-Id for all connections * [Feature] Allow to skip real action and add a header instead * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Allow to spawn asynchronous processes from Lua * [Feature] Allow to specify number of threads for ANN learning * [Feature] Allow to use global lua maps in settings * [Feature] Allow to use postfilters in composites * [Feature] Allow to verify signatures from HTTP headers in maps * [Feature] Antivirus: ordered pattern matches * [Feature] Authentication-Results: support hiding usernames * [Feature] Automatically create tables in clickhouse * [Feature] Catch next-to-last bad extension * [Feature] Check cached maps more frequently * [Feature] Check groups sanity * [Feature] Deal with obscured URLs with @ symbols * [Feature] Enhance task:store_in_file method * [Feature] Export password encryption routines to Redis * [Feature] Filter nan and inf when adding scores * [Feature] Finalize 7zip files support * [Feature] Further improvements in language detection * [Feature] Further improvements in language detection algorithm * [Feature] Generic key name expansion for Redis keys * [Feature] Hash whitelist for fuzzy_check * [Feature] Implement bayes signatures storage * [Feature] Implement buckets for Redis backend * [Feature] Implement DKIM reputation adjustments * [Feature] Implement forked workers children monitoring * [Feature] Implement headers flags in mime parser * [Feature] Implement l1/l2 regularization against the current weights * [Feature] Implement manual ANN train mode * [Feature] Implement per-user ANN support * [Feature] Implement torch based ANN learning * [Feature] Implement upstreams logic for clickhouse exporter * [Feature] Import torch to Rspamd... * [Feature] Improve allocation policy when interacting with Lua * [Feature] Improve Lua/C interaction in history_redis * [Feature] Improve multiple fuzzy results combining * [Feature] Improve parsing of DKIM keys: parse algorithm * [Feature] Improve subprocesses termination handle * [Feature] Improve symbol type parsing in Lua API * [Feature] Metadata Exporter: e-Mail Alerts: support multiple recipients; alerting senders/recipients/users (#1600) * [Feature] Milter headers: support adding/removing arbitrary headers from config * [Feature] More metatokens * [Feature] Multimap: checking of symbol options * [Feature] Multimap: template URL filter * [Feature] New bayes expiry plugin * [Feature] Periodically save rspamd stats to disk * [Feature] Preliminary import of the elasticsearch module * [Feature] Ratelimit: allow full addresses in whitelisted_rcpts * [Feature] Ratelimit: support fetching limits from Redis * [Feature] RBL: received: filtering by position & flags * [Feature] Read global maps for lua * [Feature] Redis settings: support checking multiple keys * [Feature] Rework fann plugin to be a normal post-filter * [Feature] Rework logging configuration for rspamadm case * [Feature] Rework short hashes generation to avoid FP * [Feature] Save real ucl types when exporting to Lua * [Feature] Set TCP_NODELAY for milter sockets * [Feature] Setup DKIM signing from configwizard * [Feature] Skip certain symbols from ANN classify * [Feature] Store plugins state * [Feature] Support etag for HTTP maps * [Feature] Support Expires header when using HTTP maps * [Feature] Support sending given header multiple times in lua_http * [Feature] Support sha512 in DKIM signatures * [Feature] Try to detect HTML messages better * [Feature] Use array instead of queue to reduce memory fragmentation * [Feature] Use controller port by default when connecting to local IP * [Feature] Use rdtsc where possible * [Fix] Actively load skip hashes map in fuzzy storage * [Fix] Add another workaround to display history properly * [Fix] Add definition for old glib compatibility method * [Fix] Add missing rspamadm control options to help * [Fix] Add workaround for IPv6 in sendmail * [Fix] Add workaround for system with non-XSI compatible tzset * [Fix] Allow oversigning in DKIM signatures * [Fix] Allow to check negative scores in force_actions * [Fix] Allow to have negative actions limits * [Fix] Allow to set any layers number for fann rules * [Fix] Another fix for rdtcs * [Fix] Another fix to lua xmlrpc * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Another try to fix threading in torch * [Fix] Apply language detection when adding fuzzy hashes * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Authentication Results: Fix SPF smtp.mail_from * [Fix] Auth-Results: Multiple DKIM signatures * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Check for magic when checking for an archive * [Fix] Cleanup mess with groups * [Fix] Clickhouse: Insertion in the symbols table * [Fix] Crash in URL processing * [Fix] Deal with another case when processing exceptions * [Fix] Deal with deeply nested messages more aggressively * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with non-key arguments in lua_redis.exec_script * [Fix] Deal with unknown weight * [Fix] Deal with URLs with no slashes after protocol * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] Deal with zero scores symbols * [Fix] Default monitoring domain for surbl plugin * [Fix] Delay upstream re-resolving when one upstream is defined * [Fix] Detection of maillist optimized and fixed * [Fix] DKIM signing: allow for auth_only to be false * [Fix] DMARC: require report_settings for sending reports only * [Fix] Do not allow garbadge when checking url domain * [Fix] Do not cache SPF records with PTR elements * [Fix] Do not constantly re-resolve failed upstreams with a single element * [Fix] Do not crash if no words defined * [Fix] Do not crash on empty subtype * [Fix] Do not expose spamtrap messages to SMTP reply * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not ignore short words * [Fix] Do not include idempotent/nostat symbols to checksum * [Fix] Do not override groups when converting metrics * [Fix] Do not override unix socket group when group comes before owner * [Fix] Do not skip the last character * [Fix] Do not spawn too many workers by default * [Fix] Do not stop monitored on dns errors * [Fix] Do not stop parsing headers on bad IP header * [Fix] Do not strip last character in the last word * [Fix] Do not treat script content as text * [Fix] Do not try to connect to non-supported addresses * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Erase unknown HTML entities * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix a lot of FP in chartable in mixed languages * [Fix] Fix ANN checks * [Fix] Fix ANN loading logic * [Fix] Fix another tokenization issue * [Fix] Fix autolearn parameters reading * [Fix] Fix bad archive characters stripping * [Fix] Fix bad extension check * [Fix] Fix bayes schema conversion * [Fix] Fix blacklists and DMARC in whitelist * [Fix] Fix brain-damaged torch build system * [Fix] Fix build on FreeBSD * [Fix] Fix clickhouse exporter * [Fix] Fix clickhouse schema * [Fix] Fix comparision * [Fix] Fix composites processing * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix couple of warnings * [Fix] Fix crashes in the rspamd_control path * [Fix] Fix deletion from hash * [Fix] Fix DKIM forgeries via multiple headers * [Fix] FIx dynamic conf plugin * [Fix] Fix emails detection * [Fix] Fix empty headers simple canonicalization * [Fix] Fix empty threshold check in greylisting module * [Fix] Fix encrypted legacy reply in fuzzy storage * [Fix] Fix enormous scores for R_WHITE_ON_WHITE * [Fix] Fix exceptions list in surbl * [Fix] Fix *_EXCESS_BASE64 rules * [Fix] Fix expire rounding * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix format strings * [Fix] Fix get_content method * [Fix] Fix groups override when defining symbols * [Fix] Fix learned count in new schema * [Fix] Fix learn errors propagation * [Fix] Fix loading of per-user redis backend for statistics * [Fix] Fix logging buffer corruption in case of repeated messages * [Fix] Fix lua cached elements invalidation * [Fix] Fix merging of the implicit arrays * [Fix] Fix mime_types scoring * [Fix] Fix multiple headers in DKIM headers list * [Fix] Fix null callee case in clang plugin * [Fix] Fix obscured url in format user@@example.com * [Fix] Fix parsing of the per-user script * [Fix] Fix priorities in rspamd_update, disable rules execution * [Fix] Fix processing of closed tags * [Fix] Fix processing of idempotent rules when autolearn fails * [Fix] Fix processing of multipart parts with no headers * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix PTR processing in SPF * [Fix] Fix pushing country to clickhouse asn table * [Fix] Fix random forests module * [Fix] Fix real IP parsing for some strange Exim received * [Fix] Fix Redis timeout setup * [Fix] Fix reload crash when hyperscan is enabled * [Fix] Fix reusing of redis connection after exec * [Fix] Fix sanity checks on macro value * [Fix] Fix setting of path and cpath for Lua * [Fix] Fix setting of signals when spawning a thread * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix ticks processing * [Fix] Fix upstream addrs updating * [Fix] Fix urls/emails distinguishing found in queries * [Fix] Fix user settings check * [Fix] Fix variable increment * [Fix] Fix various issues in stat_convert * [Fix] F-PROT Antivirus infection string for all known occurences * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Further fixes around floating point expressions * [Fix] Further fixes to ANN module * [Fix] Further fixes to rescore tool * [Fix] Further fixes to support ES 6 * [Fix] Further tokenization fixes * [Fix] Greylisting set phase is not idempotent * [Fix] Handle proxy copy errors * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Kill spawned processes on termination * [Fix] Load skip map from all processes as shared cache is unavailable * [Fix] Lowercase HTTP headers to make them searchable from Lua * [Fix] Lowercase words * [Fix] Lua_http: freeing * [Fix] Lua: lpeg to be loaded with rspamd_lua_add_preload, to avoid "rspamd_config_read: rcl parse error: cannot init lua file […] module 'lpeg' not found" * [Fix] Map absence is not an error * [Fix] Metadata exporter: check IP sanity * [Fix] Milter headers: custom headers: removing headers * [Fix] Milter headers: skip_local / skip_authenticated settings * [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first * [Fix] mime_types: fix next-to-last extension length check * [Fix] More hacks to deal with old configs * [Fix] Move composites second pass to the dedicated stage * [Fix] Multimap: received: filtering of artificial header * [Fix] Multiple fixes in torch based ANN plugins * [Fix] Once more (#1879) fix bad extension check * [Fix] Optimize rspamd_fstring_t reallocations * [Fix] options.local_networks setting * [Fix] Parse HREF urls without explicit prefix * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Plug memory leak * [Fix] Plug memory leak in lua_tcp * [Fix] Plug memory leak when setting email addresses from Lua * [Fix] Propagate learn/stat errors more precisely * [Fix] Ratelimit: fix whitelisted_rcpts matching * [Fix] Ratelimit: lowercase email addresses * [Fix] RBL: received: deal with missing data (#1965) * [Fix] Rebalance and slightly rework MX check plugin * [Fix] Redis key expansion: EVAL: deal with strings * [Fix] Redis script loading in DMARC; URL tags; URL reputation * [Fix] Reject invalid bh for DKIM signatures earlier * [Fix] Relax pem signature detection * [Fix] Relax unicode properties requirements for chartable module * [Fix] Remove extra noise from dkim and arc signing * [Fix] Remove hop-by-hop headers in proxy * [Fix] Remove incorrect method `task:set_metric_subject` * [Fix] Replace space like characters in headers with plain space * [Fix] Restore old style ratelimits support * [Fix] Rework elasticsearch plugin * [Fix] Rewriting subjects via force actions module * [Fix] RPM postinstall * [Fix] Sanitize IP in history redis * [Fix] Select the correct signature when doing simple canon * [Fix] Set CLOEXEC flag on files opened * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Settings: header: deal with multiple settings (#1988) * [Fix] Skip checks if both extensions are not bad * [Fix] Skip nostat tokens when get number of tokens * [Fix] Some more fixes towards emails detection * [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't match * [Fix] Stop using of g_slice... * [Fix] Switch rspamadm logging to message level * [Fix] Symbol 'FANNR_SPAM' has its score defined.. * [Fix] Table parameter for rspamd_config:add_doc() * [Fix] Treat 'rewrite subject' as spam action * [Fix] Try harder in passing IPv6 addresses * [Fix] Try harder to find rfc822 notifications * [Fix] Try harder to find urls * [Fix] Use decoded values when parsing mime addresses * [Fix] Use full URL when making an HTTP request * [Fix] Use greylisting threshold in greylisting module * [Fix] Use n_words attribute from ngramms * [Fix] Use raw urls when sending requests to redirector * [Fix] Use the right boolean operator on error check * [Fix] Use weight from map for fuzzy scoring * [Fix] Various fixes to elastic plugin * [Fix] Various fixes to fann_redis instantiation * [Fix] Various improvements in language detection * [Fix] Virus infection string for F-PROT Antivirus * [Fix] Virus infetction string for F-PROT Antivirus * [Fix] WebUI: use relative path for savemap (#1943) * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [Fix] Write configuration changes as UCL config * [Project] Add detection logic for words * [Project] Add fast debug logging infrastructure * [Project] Add more flags to languages * [Project] Add n-gramms data files * [Project] Add ngramms frequencies detector * [Project] Add random words selection logic * [Project] Add unigramms to language detection as well * [Project] Convert all C modules to fast debug infrastructure * [Project] Detect some languages based on unicode script * [Project] Enable fast debug lookup for some modules * [Project] Enable language detector init in scanner workers * [Project] Further improvements to language detector * [Project] Implement logic of ngramms application * [Project] Improve weighting in lang_detection * [Project] Initialize language detector * [Project] Preliminary version of ngramms based language detector * [Project] Preliminary version of the new stat_convert * [Project] Remove old language detector * [Project] Rework language detection ngramms structure * [Project] Start language detection project * [Project] Start rework of language detection to improve quality * [Project] Use fast debug logging check * [Rework] Add frame for new reputation based IP score module * [Rework] Continue stat_convert rework task * [Rework] Implement new version of fuzzy replies * [Rework] Improve readability of xmlrpc API * [Rework] Kill metrics!11 * [Rework] Ratelimit module * [Rework] Rename fann_redis to neural plugin * [Rework] Reorganize mime_types module * [Rework] Rework rescore utility * [Rework] Rewrite model and learning logic for rescore * [Rework] Run post-loads when all initialization is completed * [Rework] Simplify lua path initialization * [Rework] Start major stat_convert rework * [Rework] Start mempool fragmentation reduce project * [Rework] Start moving of fann redis to torch * [Rework] Stop embedding rspamadm scripts into C * [Rework] Use floating point arithmetics in Rspamd expressions * [Rework] Use frequencies distribution in language detector * [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages * [WebUI] Compact graph selectors * [WebUI] Escape strings inside HTML in history * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Migrate widgets to D3 v4 * [WebUI] Restore passwordless login support (#2003) * [WebUI] Show symbol descriptions as tooltips in history * [WebUI] Stop using commas in pie chart tooltips * [WebUI] Update D3 and jQuery * [WebUI] Update D3Evolution 1.0.0 -> 1.1.0 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
8fb0ef19f4 |
Release 1.6.6
* [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [Feature] Allow to insert headers into specific position * [Feature] Allow to receive signing keys from mempool vars * [Feature] Authentication-Results: support hiding usernames * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] DKIM signing: allow for auth_only to be false * [Fix] Do not crash on empty subtype * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not skip the last character * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix bad archive characters stripping * [Fix] Fix comparision * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix empty headers simple canonicalization * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix parsing of the per-user script * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix Redis timeout setup * [Fix] Fix sanity checks on macro value * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix urls/emails distinguishing found in queries * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Metadata exporter: check IP sanity * [Fix] Multimap: received: filtering of artificial header * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Remove hop-by-hop headers in proxy * [Fix] Sanitize IP in history redis * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Try harder in passing IPv6 addresses * [Fix] WebUI: use relative path for savemap (#1943) * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Restore passwordless login support (#2003) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
 Viktor Szépe
|
b3e626e260
|
Typo: comparision | 6 years ago |
|
Vsevolod Stakhov
|
111dfde1ad | [Minor] Sync ChangeLog | 6 years ago |
|
Vsevolod Stakhov
|
fa9b2fc2d8 |
Release 1.6.3
* [CritFix] Fix semicolons parsing in the content type * [Feature] Add EBL to the default config * [Feature] Allow to configure monitored * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Multimap: checking of symbol options * [Feature] Redis settings: support checking multiple keys * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Fix get_content method * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Lua_http: freeing * [Fix] Milter headers: custom headers: removing headers * [Fix] Parse HREF urls without explicit prefix * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [WebUI] Escape strings inside HTML in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
e5cefe177e
|
Release 1.6.2
* [Conf] Remove Rambler email bl for now * [Conf] Switch RAMBLER_URIBL to a locally managed source * [CritFix] Switch from ragel to C for Content-Type parsing * [Feature] Add `-e` option for lua_repl * [Feature] Add per-domain emails normalisation rules * [Feature] Add sessions cache to debug dangling sessions * [Feature] Add short_text_direct_hash for fuzzy check module * [Feature] Add text_part:get_stats function * [Feature] Allow to add custom processing script for surbl * [Feature] Allow to check reply-to email * [Feature] Allow to customize spam header, remove existing spam headers * [Feature] Allow to disable specific workers in the config * [Feature] Allow to discard messages instead of rejection * [Feature] Allow to specify custom delimiter in emails plugin * [Feature] Allow to specify custom User-Agent for rspamc * [Feature] Allow to store symbols data in Clickhouse * [Feature] Allow to use HTTPS when connecting to Clickhouse * [Feature] Enable sessions cache tracking for milter connections * [Feature] Implement per-line mode in lua_repl (like `perl -p`) * [Feature] Implement rdns-curve plugin based on rspamd cryptobox * [Feature] Improve maps cached data lifetime * [Feature] Improve maps checking frequency * [Feature] Improve monitored timeouts logic * [Feature] milter_headers: add `extended_headers_rcpt` option * [Feature] Milter headers: Add X-Spam-Flag to rmilter-compatibility headers * [Feature] Milter headers: remove-header routine * [Feature] Multimap: received filters for extracting TLDs from hostnames * [Feature] Normalize email aliases in emails module * [Feature] Re-add rambler email bl (as hashed list) * [Feature] Reload file maps more frequently * [Feature] Rework newlines strip parser one more time * [Feature] Skip updates for messages scanned via controller * [Feature] Split long DKIM public keys * [Feature] Store more data when stripping newlines * [Feature] Support SPF macros transformations * [Feature] Support suppressing DMARC reports for some domains * [Fix] Add missing `break` statement * [Fix] Allow modifiers in SPF macros * [Fix] DKIM sign tools: edge-cases around use_esld * [Fix] Do not cache SPF records with macros * [Fix] Do not overwrite score when setting pre-action * [Fix] Fix comparision logic * [Fix] Fix DKIM base64 folding for milter flagged messages * [Fix] Fix emails module configuration * [Fix] Fix folding for arc headers when milter interface is used * [Fix] Fix gmail dots removal * [Fix] Fix rspamc detection in greylist module * [Fix] Fix some more issues with HTTP maps * [Fix] Milter sessions can live forever * [Fix] Normalize fuzzy probability better * [Fix] Plug memory leak * [Fix] RBL: Fixed hashed email address lookups * [Fix] Try to deal with brain-damaged milter behaviour * [Fix] Use `\n` to fold headers for milter * [Rework] Allow to use custom callback for monitored checks * [Rework] Further steps towards one process monitoring * [Rework] Send health checks from a single worker * [WebUI] Round-up throughput summary values Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
 Alexander Moisseev
|
bb5a9f8e45 |
[Minor] Massive spelling correction
by a bot https://github.com/ka7/misspell_fixer |
7 years ago |
|
Vsevolod Stakhov
|
8895fc6fab
|
Release 1.6.1
* [Fix] Allow to init resolver without rspamd_config * [Fix] Do not crash when resolver failed to initialize * [Fix] Fix abstract context layout * [Fix] Fix CGP helper reply parsing * [Fix] Fix crashes when socket write errors occur * [Fix] Fix parsing IPv6 nameservers in resolv.conf * [Fix] Milter: Don't defer on "greylist" action Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
9c0fd5e439
|
Release 1.6.0
* [Conf] Add rspamd_proxy to the default configuration set
* [Conf] Add sample arc module config
* [Conf] Do away with systemd specifics completely
* [Conf] Increase min_bytes to avoid FP
* [Conf] Remove ratelimits from default configuration
* [CritFix] Fix accepting on IPv6 sockets
* [CritFix] Fix corruption when multiple fuzzy are defined
* [CritFix] Fix learn condition in fuzzy check
* [CritFix] Fix memory leak in fuzzy check
* [CritFix] Fix memory leak in maps scheduling
* [CritFix] Paese the last character in DKIM signature correctly
* [CritFix] Zero fill sockaddr_un
* [Feature] Add ability to add doc strings by example
* [Feature] Add API to verify DKIM (and ARC) signatures
* [Feature] Add compression/decompression to proxy
* [Feature] Add count to url structure
* [Feature] Add initial support of the new protocol reply
* [Feature] Add Lua plugin spamtrap
* [Feature] Add `monitored_address` for rbls
* [Feature] Add new schema for bayes tokens
* [Feature] Add preliminary ARC support to dkim code
* [Feature] Add preliminary support of ARC signing
* [Feature] Add rules to detect bad 8bit characters in From and To
* [Feature] Add scanning support for milter protocol
* [Feature] Add support for bidirectional symbols in rspamd_stats
* [Feature] Add support for static maps
* [Feature] Add support of maps with multiple regexps matches
* [Feature] Add `text_multiplier` param
* [Feature] Add the preliminary ARC plugin
* [Feature] Add top redirector targets rank
* [Feature] Allow async events to be registered from LUA rules
* [Feature] Allow storing bayes tokens in Redis
* [Feature] Allow to exclude specific domains from mx check
* [Feature] Allow to have a stack of watcher finalisers
* [Feature] Allow to pass hostname to `-i` flag in Rspamc
* [Feature] Allow to set custom user agent in url redirector
* [Feature] Allow to use custom callback when parsing resolv.conf
* [Feature] Allow to use domain from authenticated user
* [Feature] Bayes expiry plugin
* [Feature] Check dkim sign keys for modifications
* [Feature] DKIM signing: sign_networks/local address specific use_domain settings
* [Feature] DMARC: Support excluding domains from sampling
* [Feature] Expire processing items for URL redirector aggressively
* [Feature] Fix surbl monitored for IP lists, add `monitored_domain` option
* [Feature] Implement caching for dkim body hashes
* [Feature] Implement milter protocol scan reply
* [Feature] Improve omograph phishing detection
* [Feature] Initial support of self-scan in Rspamd proxy
* [Feature] Keep track of headers in milter interface
* [Feature] Milter headers: better controls for local/authenticated
* [Feature] Multimap: email:domain:tld filter
* [Feature] Preliminary DMARC reporting implementation
* [Feature] Reuse stemmers in the cache
* [Feature] Rework confighelp to load Lua plugins
* [Feature] Rework hfilter to use hyperscan if possible
* [Feature] Rework lua RSA API
* [Feature] Rmilter_headers: approximate rmilter's extended_spam_headers
* [Feature] Start integration of milter support in proxy
* [Feature] Store average words length and short words count
* [Feature] Store hash of headers order and names
* [Feature] Support MTA name header
* [Feature] Support multiple types of dkim signing in Lua
* [Feature] Support numeric arguments for Redis requests
* [Feature] Use headers hash in bayes metatokens
* [Feature] Use normal resolv.conf rules of rotation in Rspamd
* [Feature] Use version 2 proto for checking messages
* [Fix] Allow to follow symlinks when safe
* [Fix] Append MX name for authentication results as required
* [Fix] Change default text multiplier from 0.5 to 2.0
* [Fix] Check min_bytes for images as well
* [Fix] Deal with 7bit charsets properly
* [Fix] Deal with 8bit characters in email addresses
* [Fix] Deal with unpaired <a> tags
* [Fix] Detect confighelp in plugins initialisation
* [Fix] Disable certain checks for utf spoof detection
* [Fix] DKIM Signing: avoid nil index when From header is missing
* [Fix] Do not add exact hashes from different parts
* [Fix] Do not check DMARC if SPF or DKIM were not checked
* [Fix] Do not check URLs that are resolved to be redirected
* [Fix] Do not set bayes probability if we don't use it
* [Fix] Do not stop on illegal unicode points - replace them
* [Fix] Fix another race condition in arc checks
* [Fix] Fix arc count logic
* [Fix] Fix ARC signing
* [Fix] Fix brain-damaged spamc protocol for now
* [Fix] Fix calling for peak functions
* [Fix] Fix couple of issues in FORWARDED rule
* [Fix] Fix CTE propagation from parent containers to children parts
* [Fix] Fix errors processing in the controller
* [Fix] Fix format string in milter
* [Fix] Fix issues in SPF macros parsing
* [Fix] Fix logging format string
* [Fix] Fix logic of cached passwords check
* [Fix] Fix lowercasing of stemmed words
* [Fix] Fix LRU elements removal
* [Fix] Fix memory leak when accepting from unix sockets
* [Fix] Fix milter connections persistence
* [Fix] Fix objects merging in UCL
* [Fix] Fix order of operations to avoid race condition
* [Fix] Fix parsing of long regexp types
* [Fix] Fix passing data to log helper when many symbols defined
* [Fix] Fix pools management for milter session
* [Fix] Fix processing of the watchers
* [Fix] Fix queue id macro in milter
* [Fix] Fix R_BAD_CTE_7BIT rule
* [Fix] Fix Redis timeout set
* [Fix] Fix REPLYTO_UNPARSEABLE rule
* [Fix] Fix setting of email address
* [Fix] Fix some more issues about duplicated fuzzy requests
* [Fix] Fix spamc support in rspamd proxy
* [Fix] Fix syntax error in spamtrap plugin
* [Fix] Fix url counts for href urls
* [Fix] Fix url handling in the protocol
* [Fix] Multimap: Received IP filters with Redis
* [Fix] Oops, fix
|
7 years ago |
|
Vsevolod Stakhov
|
a92e55249f | [Minor] Update ChangeLog from 1.5 branch | 7 years ago |