Tree:
4d3d5401c2
master
rspamd-3.8
vstakhov-fuzzy-tcp
external-maps
libev-migration
log_json
mime-rework
rdns-tcp-rework
rework-symcache
rspamd-0.5
rspamd-0.6
rspamd-0.7
rspamd-0.8
rspamd-0.9
rspamd-1.0
rspamd-1.1
rspamd-1.2
rspamd-1.3
rspamd-1.4
rspamd-1.5
rspamd-1.6
rspamd-1.9
rspamd-3.7
torch-removal
vstakhov-fasttext-langdet
vstakhov-fuzzy-cxx
vstakhov-known-senders
vstakhov-stringzilla
3.8.4
3.8.3
3.8.2
3.8.1
3.8.0
3.7.5
3.7.4
3.7.3
3.7.2
3.7.1
3.6
3.5
3.4
3.3
3.2
3.1
3.0
2.7
2.6
2.5
2.4
2.3
2.2
2.1
2.0
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.3
1.8.2
1.8.1
1.8.0
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.9
1.5.8
1.5.7
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.11
1.0.10
1.0.9
1.0.8
1.0.7
1.0.6
1.0.5
1.0.4
1.0.3
1.0.2
1.0.1
1.0.0
0.9.10
0.9.9
0.9.8
0.9.7
0.9.6
0.9.5
0.9.4
0.9.3
0.9.2
0.9.1
0.9.0
0.8.3
0.8.2
0.8.1
0.8.0
0.7.6
0.7.5
0.7.4
0.7.3
0.7.2
0.7.1
0.7.0
0.6.10
0.6.9
0.6.8
0.6.7
0.6.6
0.6.5
0.6.4
0.6.3
0.6.2
0.6.1
0.6.0
0.5.6
0.5.5
0.5.4
0.5.3
0.5.2
0.5.1
0.5.0
0.4.7
0.4.6
0.4.5
0.4.4
0.4.3
0.4.2
0.4.1
0.4.0
0.3.14
0.3.13
0.3.12
0.3.11
0.3.10
0.3.9
0.3.8
0.3.7
0.3.3
0.3.2
0.3.1
0.3.0
0.2.7
110 Commits (4d3d5401c2aa6cc8316cacc415893ec785d3a47d)
110 Commits (4d3d5401c2aa6cc8316cacc415893ec785d3a47d)
| Author | SHA1 | Message | Date |
|---|---|---|---|
 Vsevolod Stakhov
|
4fc136a7f8
|
Release 3.0
* [Conf] Align ARC scores with DKIM scores * [CritFix] Neural: Fix sorting application * [Feature] Add a simple dumper for bayes tokens * [Feature] Add lua_maps.fill_config_maps function * [Feature] Add preliminary exporter to AWS S3 * [Feature] Add preliminary restore bayes support * [Feature] Add race condition protection against hs_helper restarts * [Feature] Add rspamd_utf8_strcmp utility * [Feature] Add zstd streaming API * [Feature] Allow to log severity level explicitly * [Feature] Allow to save and show attachment name when inserting AV scan results * [Feature] Allow to sort urls for Lua * [Feature] Allow to specify different timeouts/retransmits for fuzzy rules * [Feature] Aws_s3: Allow to compress data stored * [Feature] CMakeLists.txt: Change check and run-test to use rspamd-test-cxx * fixes #3807 * [Feature] Dmarc_report: allow sending reports in batches * [Feature] Fuzzy_check: Allow to disable subject when making short text hash * [Feature] Lua_cryptobox: Add keyed ssl hash functions via HMAC * [Feature] Lua_task: Add get_urls_filtered method * [Feature] Make monitored checks less frequent * [Feature] Milter_headers: Add x-rspamd-pre-result header * [Feature] Neural: Allow to balance FP/FN for the network * [Feature] Ppopagate monitored errors from rbl module * [Feature] Pyzor calculate score dynamically Count - WL-Count of default_score in percent * [Feature] Rbl: Distinguish flattened and non-flattened selectors in RBL requests * [Feature] Re-add pyzor support * [Feature] Settings: add ip_map check and rework structure slightly * [Feature] Spamassassin: Allow to set the default priority for SA scores * [Feature] Strip smtp comments from message id * [Feature] add SYSTEM_ZSTD cmake option To use the system zstd instead on the bundled version * [Feature] external_relay plugin * [Feature] rspamadm clickhouse neural_train subcommand * [Fix] #3400 milter_headers: fix inverted logic for extended_headers_rcpt * [Fix] ASN: fix _FAIL symbol for when main symbol is disabled * [Fix] Add a special logic for text part with no text extraction * [Fix] Add diacritics flag for several eu languages * [Fix] Another FSM fix to accomodate possibility of multiple consequent ? * [Fix] Avoid curse of dynamic array referencing * [Fix] Avoid reinitialising neural settings * [Fix] Check remain before processing TXT records * [Fix] Enable error multiplier on http errors * [Fix] Finally rework parsing entities logic * [Fix] Fix '==' parsing in the content type attributes parser * [Fix] Fix IPv6 expansion for SPF macros * [Fix] Fix Mozilla Message-ID detection * [Fix] Fix an edge case in BITCOIN_ADDR rule * [Fix] Fix brain-damaged behaviour when http request has a custom Host header * [Fix] Fix check of limits in email address parsing * [Fix] Fix copy&paste error and rework * [Fix] Fix expressions logic for and/or and float values * [Fix] Fix fuzzy retransmits * [Fix] Fix http maps with no or invalid expires data * [Fix] Fix last quote character parsing in the content-type state machine * [Fix] Fix normalisation flags propagation * [Fix] Fix overflow when appending many broken tags * [Fix] Fix parsing of rfc2047 tokens with '?' inside * [Fix] Fix phishing flag set * [Fix] Fix rfc2047 embedded into rfc2231 pieces in special headers * [Fix] Fix round-robin rotation * [Fix] Fix searching for symbols * [Fix] Fix storing of the regexps inside variant * [Fix] Fix tokenization near exceptions * [Fix] Fix visibility calculations * [Fix] Html: Attach inline tags to the structure * [Fix] Html: Do not treat empty tags as block tags * [Fix] Ical: Do not extract urls from all flags using merely specific ones * [Fix] Initialise symcache even if it cannot be loaded properly * [Fix] Lua_fuzzy: Remove text parts check when checking image dimensions * [Fix] Lua_maps: Fix adjustments for the map type in the complex map definitions * [Fix] Lua_task: Fix deleted symbols in has_symbol/get_symbol * [Fix] Move metric and symcache link from validation to the init stage * [Fix] Oletools: Another try to fix table sorting * [Fix] One more default behaviour fix * [Fix] Phishing: Rework urls processing * [Fix] RBL: was missing some config schema * [Fix] Replies: Fix 'Reply-To' handling in task:get_reply_sender * [Fix] Rework metrics handling * [Fix] Save symcache on exit * [Fix] Selectors: Filter nil elements in lists * [Fix] Selectors: Properly fix implicit tostring for nils * [Fix] Try to fix some broken code in DMARC reporting plugin * [Fix] Urls: Fix processing of html urls when it comes to the flags * [Fix] Use proper buffer length * [Fix] Various visibility fixes * [Fix]: ASN: dns cb func should also return in case of an error * [Project] Add a simple css rule definition * [Project] Add css style skeleton * [Project] Add css syntax (adopted from ebnf) * [Project] Add css_selectors * [Project] Add doctest unit testing library * [Project] Add expected library * [Project] Add fmt library for simple string ops * [Project] Add fu2 library to better functions abstractions * [Project] Add hashing method * [Project] Add parsers skeleton * [Project] Add preliminary support of vcard parser * [Project] Add process exceptions for invisible text * [Project] Add some methods for css parser * [Project] Allow static libstdc++ * [Project] Another whitespace hack * [Project] CSS: Various fixes in the declarations and values parsing * [Project] Cpp: Add robin-hood hash map library * [Project] Css: Add AST debug * [Project] Css: Add colors conversion functions * [Project] Css: Add dimensions handling * [Project] Css: Add display value support * [Project] Css: Add frozen library from https://github.com/serge-sans-paille/frozen/ * [Project] Css: Add opacity support * [Project] Css: Add parser helpers to simplify debugging * [Project] Css: Add preliminary stylesheet support * [Project] Css: Add rules processing functions and tests * [Project] Css: Add simple selectors unit tests * [Project] Css: Add some c++ unit tests * [Project] Css: Add some debug methods * [Project] Css: Add some debug statements for the css parser * [Project] Css: Add some logical skeleton for declarations parser * [Project] Css: Add url/function tokens * [Project] Css: Allow at rules parsing * [Project] Css: Declarations parsing logic skeleton * [Project] Css: Enable conditional css parsing support from the HTML parser * [Project] Css: Finish generic lexer cases * [Project] Css: Fix HSL conversion * [Project] Css: Fix minus parsing * [Project] Css: Fix parser consumers nesting * [Project] Css: Fix parsing of the qualified rules * [Project] Css: Fix rules merging * [Project] Css: Further fixes to lexer * [Project] Css: Further steps to parse css colors + rework * [Project] Css: Further work on parser's methods * [Project] Css: Implement backlog of css tokens * [Project] Css: Implement numbers and ident parsers * [Project] Css: Implement simple css selectors lookup * [Project] Css: Implement styles merging * [Project] Css: Make debug strings json like to simplify tests * [Project] Css: Minor adjustments * [Project] Css: More meat to the lexer * [Project] Css: Move some of the tests to the doctest * [Project] Css: Projected a parser * [Project] Css: Properties attachment logic * [Project] Css: Remove ragel from build targets (maybe keep for reference) * [Project] Css: Rework css block structure * [Project] Css: Rework flags of css properties * [Project] Css: Rework tokens structure * [Project] Css: Several fixes + tests * [Project] Css: Simplify checks * [Project] Css: Simplify debug code * [Project] Css: Start css selectors parsing logic * [Project] Css: Start semantic parsing for rules * [Project] Css: Start stylesheet implementation * [Project] Css: Tidy up lambdas * [Project] Css: rework tokeniser * [Project] Dmarc: Add dmarc report tool (WIP) * [Project] Dmarc: Add munging configuration * [Project] Dmarc: Add preliminary munging logic * [Project] Dmarc: Fix header removal * [Project] Dmarc: Fix munging logic * [Project] Dmarc: Use full recipient address instead of a domain map * [Project] Dmarc: Use zlists for dmarc reports * [Project] Dmarc_report: Add message generation logic * [Project] Dmarc_report: Add preliminary sending support * [Project] Fix lua bindings * [Project] Fix xml/sgml tags processing * [Project] Handle new modification * [Project] Html/CSS: Add transform from a CSS rule to html block * [Project] Html/CSS: Link html and css styles * [Project] Html/CSS: Switch styles parsing to css parser * [Project] Html/Css: Fix some issues found * [Project] Html/Css: Implement visibility rules for a block * [Project] Html: Add more tests cases and fix some more corner issues * [Project] Html: Add rows display type support * [Project] Html: Allow decode entities function to normalise spaces + unit tests * [Project] Html: Another rework of the tags structure * [Project] Html: Another try to fix unbalanced cases * [Project] Html: Fix crossing spans * [Project] Html: Fix parent propagation * [Project] Html: Further rework of the html parsing stuff * [Project] Html: Implement logic for tags pairing * [Project] Html: Implement rawtext state machine * [Project] Html: Insert closing tags as well :( * [Project] Html: More fixes * [Project] Html: More fixes * [Project] Html: More spaces logic fixes * [Project] Html: One more attempt to write text content * [Project] Html: Replace \0 in html content * [Project] Html: Rework img/a tags handling * [Project] Html: Rework propagation method * [Project] Html: Rework tags placement * [Project] Html: Rework transparency logic * [Project] Html: Support 'hidden' attribute * [Project] Html: Try another approach to append tags content * [Project] Html: Try to deal with bad unknown tags properly * [Project] Lua_aws: Add canonicalisation utility * [Project] Lua_aws: Add function to produce AWS Authorisation header * [Project] Lua_aws: Implement request signing * [Project] Lua_mime: Add lua_mime.modify_headers routine * [Project] Lua_task: Add modify_header method * [Project] Lua_task: Allow to extract modified headers * [Project] Make unescape code public for unit testing * [Project] More fixes for closed tags * [Project] More fixes to calculations * [Project] Rework API for the modified headers * [Project] Rework html visibility rule * [Project] Skeleton of the css library * [Project] Start headers modification API structure * [Project] Start working on AWS Lua API * [Project] Use lua_mime to modify headers * [Project] Use modified headers on dkim signing * [Project] Use string_view to constexpr variant unpacking * [Rework] Add composites manager concept * [Rework] Add tags definitions * [Rework] Allow C code to be compiled with C++ compiler * [Rework] Clickhouse: Store url flags * [Rework] Composites: Rewrite the composites logic * [Rework] Composites: Start rework of the composites framework * [Rework] Dmarc: Move check policy function to the common utils * [Rework] Dmarc: Rework reports keys structure * [Rework] Further work to make html content private * [Rework] Html/CSS: Remove css C bindings as they are useless now * [Rework] Html/CSS: Rework Lua bindings * [Rework] Html/Css: Start rework of the html blocks * [Rework] Html: Add images processing logic * [Rework] Html: Add traverse function * [Rework] Html: Another steps to get rid of gnode * [Rework] Html: Convert to variant * [Rework] Html: Deal with the utf_content part * [Rework] Html: Final rework part for the html processing code * [Rework] Html: Fix Lua bindings * [Rework] Html: Forgot to add the internal include * [Rework] Html: Further html urls rework * [Rework] Html: Further rework of the tags content extraction * [Rework] Html: Make parameters as a vector again * [Rework] Html: Move blocks part * [Rework] Html: Move images processing stuff * [Rework] Html: Rework lua bindings * [Rework] Html: Start html text extraction rework * [Rework] Html: Start refactoring of the html tags handling * [Rework] Html: Start removing of GNode stuff * [Rework] Html: Start rework of the html content structure * [Rework] Lua_magic: Try to detect text parts with 8bit characters for non-utf8 encodings * [Rework] Move HTML url functions and rework them * [Rework] Move and adopt entities handling logic * [Rework] Move common and rarely used dmarc code to the library * [Rework] Move compression routines outside of rspamd_util library * [Rework] Move entities/tags handling * [Rework] Phishing: Split from redirectors usage * [Rework] Redesign html blocks propagation logic * [Rework] Remove tag name string * [Rework] Rename phished url to a linked url * [Rework] Reorganize dmarc plugin and remove unsupported reporting code * [Rework] Reputation: Use more flexible types in get/set functions * [Rework] Require proper C++ environment for Rspamd build * [Rework] Rework extended urls output * [Rework] Rework tags parsing machine * [Rework] Slightly improve old regexp API * [Rework] Start conversion of the redis pool code to c++ * [Rework] Try to resolve failed upstreams more agressively * [Rework] Use C++ utf8 library with unit tests to trim whitespaces * [Rework] Use C++ version for unicode normalisation * [Rework] Use C++ version of the lua threads pool * [Rules] Add raw addresses to MULTIPLE_FROM options * [Rules] Another fix to HTTP_TO_HTTPS rule * [Rules] Do not trigger HTML_SHORT_LINK_IMG on external images * [Rules] Extend FORGED_X_MAILER * [Rules] Extend OLD_X_MAILER * [Rules] Fix CTYPE_MIXED_BOGUS for text attachments * [Rules] Fix FPs for CTYPE_MIXED_BOGUS * [Rules] Fix HTTP_TO_HTTPS rule * [Rules] Fix HTTP_TO_HTTPS rule * [Rules] Fix zerofont rule (partially) * [Rules] Micro-optimize X_PHP_EVAL * [Rules] Reduce default weight for R_MISSING_CHARSET Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
2 years ago |
|
Vsevolod Stakhov
|
4c56c1210d
|
Release 2.7
* [Conf] Add R_DKIM_PERMFAIL to the metric * [CritFix] Dkim: Fix simple canonicalisation if multiple signatures are presented * [CritFix] Fix controller paths normalisation * [Feature] Add INVALID_DATE rule * [Feature] Add controller endpoint for training neural * [Feature] Add sanity checks for actions thresholds * [Feature] Add support of '==' and '!=' in Rspamd expressions * [Feature] Composites: Improve composite atoms parser * [Feature] Docker: use Debian slim variant * [Feature] Elastic: Add some missing fields * [Feature] Extract text from img alt attributes * [Feature] Improve charset detection logic * [Feature] Lua_clickhouse: Add optional row callback for large selections * [Feature] Lua_dns_resolver: Add idna_convert_utf8 method * [Feature] Lua_mime: Add ability to do multipattern replacement * [Feature] Lua_trie: Allow to report start of the match * [Feature] Multimap: support adding map values as extra options * [Feature] Neural: Move PCA learning to a subprocess * [Feature] RBL: support matching content/image URLs only * [Feature] RBL: support use of multiple selectors * [Feature] Reputation: Allow to specify ip masks * [Feature] Support SMIME signed messages container * [Feature] Support multiple conditions for symbols * [Feature] Support ping in milter mode * [Feature] Support rspamd_text in selector regexps * [Feature] Use own daemonization routine * [Feature] Vadesecure: Implement settings_outbound feature as recommended by Vade * [Feature] `rspamadm clickhouse` command * [Feature] allow hyperscan for aarch64 * [Fix] Allow to set priorities between post init scripts * [Fix] Allow to use maps for strings that are not zero terminated * [Fix] Apply max_lua_urls limit for emails as well * [Fix] Arc: Fix CV check on signing * [Fix] Arc: Fix signing of the broken ARC chains * [Fix] Clickhouse: escape carriage return * [Fix] Composites: Allow partial match * [Fix] Deduct type of a table methods * [Fix] Do not load errored hyperscan database * [Fix] Do not process links in ignored html tags * [Fix] Fix ClamAV result for cached encrypted file (#3395) * [Fix] Fix canonicalisation when l= tag is presented * [Fix] Fix flag shift * [Fix] Fix handling of skip/skip_process http flags * [Fix] Fix html attachments checks * [Fix] Fix issue with pushing binary formats to Lua strings * [Fix] Fix logging for rspamadm * [Fix] Fix off-by-one with init check * [Fix] Fix parsing of escape characters in quoted pairs * [Fix] Fix pushing ucl strings with \0 inside * [Fix] Fix quoted-printable soft newlines bugged case * [Fix] Fix settings in case actions are set to null (#3415) * [Fix] Fix several issues with auth results producing * [Fix] Fix smtp comments exclusion * [Fix] Fix smtp date syntax definition * [Fix] Fix substring search in case if srchlen == inlen * [Fix] Fix text selectors * [Fix] Honour `systemd` setting when logging to console (#3514) * [Fix] Html: Add entities collisions prevention logic (e.g. for mathml entities) * [Fix] Lua_auth_results: Quote potentially bad values in AR header * [Fix] Multimap: Fix flags usage * [Fix] Multimap: Fix scoring for combined maps * [Fix] Plug GList * leak in redis pool * [Fix] RBL: allow for multiple matches of the same label if types are different * [Fix] Rely on libev checks for file maps * [Fix] Restore simple dkim canonicalisation mode * [Fix] Return MimeCharset as we work with emails... * [Fix] Spamassassin: Fix pcre_only flags * [Fix] Spamassassin: Preserve 'pcre_only' flag when dealing with regexp replacements * [Fix] Try to fix GError leak * [Fix] Try to fix a mess with settings loading by adding priorities * [Fix] Try to move setings initialisation to a later stage * [Fix] Use dup fd in milter handler to avoid races with the proxy * [Fix] Use message pointer to avoid obsolete data to be cached * [Project] Rbl: Migrate to `checks` * [Project] Rbl: Move config code outside of the plugin * [Project] Ressurect empty prefilters as connection filters * [Project] Support connection filters registration from Lua * [Rework] Add final cleanup logic * [Rework] Add preliminary support of hyperscan caching for re maps * [Rework] Add stale cache removal * [Rework] Clickhouse: Improve performance * [Rework] Distinguish between strict config test mode * [Rework] Furhter logging improvements * [Rework] Milter_headers: improve extended_headers_rcpt support * [Rework] Move parsers to a separate lua library * [Rework] Neural: Skip composite symbols * [Rework] Rbl: Rework defaults logic * [Rework] Some tunes to cache saving * [Rework] Track maps origins * [Rework] Use full crypto hash for regexp maps * [Rules] Remove broken rule Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
3 years ago |
|
Vsevolod Stakhov
|
eec4438705
|
Release 2.6
* [Conf] Add missing symbols
* [Conf] Add missing symbols
* [Conf] Fix fat-fingers typo
* [Conf] Fix wrong comment in options.inc
* [Conf] Neural: Fix the default name for max_trains
* [Conf] Register a known symbol
* [Conf] Spf: Add R_SPF_PERMFAIL symbol
* [CritFix] Arc: Fix ARC validation for chains of signatures
* [CritFix] Distinguish socketpairs between different fuzzy workers
* [CritFix] Fix IDNA dots parsing
* [CritFix] Fix test assertion method
* [CritFix] Fix usage of crypto_sign it should be crypto_sign_detached!
* [Feature] Add BOUNCE rule
* [Feature] Add controller plugins support and selectors plugin
* [Feature] Add maps query method
* [Feature] Add minimal delay to fuzzy storage
* [Feature] Add multiple base32 alphabets for decoding
* [Feature] Add preliminary support of BCH addresses
* [Feature] Add query_specific endpoint
* [Feature] Allow multiple base32 encodings in Lua API
* [Feature] Allow to specify nonces manually
* [Feature] Controller: Allow to pass query arguments to the lua webui plugins
* [Feature] Fuzzy_check: Add gen_hashes command
* [Feature] Fuzzy_check: Add weight_threshold option for fuzzy rules
* [Feature] Implement address retry on connection failure
* [Feature] Improve limits in pdf scanning
* [Feature] Initial support of subscribe command in lua_redis
* [Feature] Lua_cryptobox: Add secretbox API
* [Feature] Lua_text: Add encoding methods
* [Feature] Milter_headers: Allow to activate routines via users settings
* [Feature] PDF: Add timeouts for expensive operations
* [Feature] Preliminary maps addon for controller
* [Feature] Split pdf processing object and output object to allow GC
* [Feature] Support BLIS blas library
* [Feature] Support input vectorisation by recvmmsg call
* [Feature] Support multiple base32 alphabets
* [Feature] add queueid, uid, messageid and specific symbols to selectors [Minor] use only selectors to fill vars in force_actions message
* [Feature] allow variables in force_actions messages
* [Feature] extend lua api
* [Fix] #3249
* [Fix] Allow to adjust neurons in the hidden layer
* [Fix] Another try to fix email names parsing
* [Fix] Arc: Allow to reuse authentication results when doing multi-stage signing
* [Fix] Arc: Fix bug with arc chains verification where i>1
* [Fix] Arc: Sort headers by their i= value
* [Fix] Change neural plugin's loss function
* [Fix] Deal with double eqsigns when decoding headers
* [Fix] Default ANN names in clickhouse
* [Fix] Disable reuseport for TCP sockets as it causes too many troubles
* [Fix] Disable text detection heuristics for encrypted parts
* [Fix] Distinguish DKIM keys by md5
* [Fix] Distinguish type from flags in register_symbol
* [Fix] Dmarc: Unbreak reporting after
|
3 years ago |
|
Vsevolod Stakhov
|
61cbcd0483
|
Release 2.5
* [Conf] Mark Rspamd emailbl as ignore whitelist * [Conf] RBL: Add missing emails = true option * [Feature] Add support for scripts in fuzzy storage * [Feature] Arc: Add whitelisted_signers_map option * [Feature] Implement hosts file processing * [Feature] Neural: Introduce classes bias that allows non-equal classes learning * [Feature] Update libev to 4.33 * [Fix] Another brain damage html standard adoptions * [Fix] Another fix for brain damaged obs-fws state * [Fix] Fix flags that caused force_actions failure * [Fix] Fix logging issue * [Fix] Fix lua symbols scores registration when config does not define scores * [Fix] Fix opaque maps logic * [Fix] Fix parsing of the html tags with no spaces after attributes * [Fix] Fix some corner cases in urls parsing, add limits * [Fix] Fix tlds extraction if custom composition rules are used * [Fix] Fix variables replacement in mempool * [Fix] Improve base64 detection * [Fix] Normalize dynamic scores in ANN correctly * [Fix] Plug memory leak introduced by #3153 * [Fix] Stat_redis_backend: Fix memory leak and simplify learn path * [Fix] Try hard to deal with ghost workers * [Fix] metadata_exporter default formatter * [Rework] Change the way to extract URLs when dealing with alternative parts * [Rework] Fix various url extraction issues * [Rework] Re cache: Load compiled hyperscan in the main process as well * [Rework] Re cache: Load hyperscan early * [Rework] Rework URL structure: adjust tld part * [Rework] Rework URL structure: host field * [Rework] Rework URL structure: more structure optimisations * [Rework] Rework URL structure: user field * [Rework] URL: Another update for urls extraction logic * [Rework] Urls: Improve query urls handling * [Rework] Urls: adopt html related stuff * [Rework] Urls: more rework of the urls sets * [Rework] Urls: process query urls in HTML urls correctly * [Rework] Urls: rework urls hash structure * [Rework] Urls: update lua libraries * [Rework] Use multiple search tries for different url extraction types Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
a74e458e15 |
Release 2.4
* [CritFix] Fix parsing of the content type attributes * [Feature] Clickhouse: Add extra columns support * [Feature] Rbl: Add url_compose_map option for RBL rules * [Fix] 'R' flag is for all headers regexp * [Fix] Allow to reset settings id from Lua (e.g. because of the priority) * [Fix] Avoid collisions in mempool variables by changing fuzzy caching logic * [Fix] Avoid strdup usage for symbols options * [Fix] Do not trust stat(2) it lies * [Fix] Filter all options for symbols to have sane characters * [Fix] Fix all headers iteration * [Fix] Fix allowed_settings for neural * [Fix] Fix listen socket parsing * [Fix] Fix maps expressions evaluation * [Fix] Fix sentinel connections leak by using async connections * [Fix] Fix smtp message on passthrough result * [Fix] Fix tld compositon rules * [Fix] Fuzzy_storage: Do not check for shingles if a direct hash has been found * [Fix] Lua_mime: Do not perform QP encoding for 7bit parts * [Fix] Neural: Distinguish missing symbols from symbols with low scores * [Fix] Support listening on systemd sockets by name * [Project] Add lua_urls_compose library * [Project] Allow to set a custom log function to the logger * [Project] CDB maps: Start making cdb a first class citizen * [Project] Clickhouse: Add extra columns concept * [Project] Fix urls composition rules, add unit tests * [Project] Unify cdb maps * [Rework] Logger infrastructure rework * [Rework] Refactor libraries structure * [Rework] Rework SSL caching * [Rework] Update snowball stemmer to 2.0 and remove all crap aside of UTF8 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
11a3b09a5d
|
Release 2.3
* [Conf] SPF is no longer a C module * [Conf] Update spamtrap map path example * [CritFix] Fix html entities decoding * [CritFix] Fix re cache when mix of pcre and hyperscan is used * [Feature] Allow milter code to deal with multiple headers * [Feature] Antivirus: Add avast support * [Feature] Dkim_signing: Allow to sign via milter_headers * [Feature] Implement content hashes * [Feature] Lua_text: Add regexp split iterator method * [Feature] Lua_text: Implement flattening of the input tables * [Feature] Send quit command to Redis * [Feature] Speed up is_ascii function * [Feature] Spf: Add external_relay option * [Fix] Avoid double escaping * [Fix] Fix O(N^2) algorithm * [Fix] Fix arc seal validation * [Fix] Fix base tag processing according to stupid HTML renderer behaviour * [Fix] Fix dealing with `\0` in ucl strings and JSON * [Fix] Fix gpg parts misdetection * [Fix] Fix ignored symbols exporting * [Fix] Fix processing of numeric url's * [Fix] Fix processing of the closed tcp connections * [Fix] Fix regexp type check for pcre2 * [Fix] Fix urls encode function * [Fix] Fix urls shifting when doing decode to include separators * [Fix] Fix white on white rule and add is_leaf flag * [Fix] Further fixes in charset detection * [Fix] Ignore diacritics in chartable module for specific languages * [Fix] Limit size of symbols options by max_opts_len option * [Fix] More fixes in html tag content calculations * [Fix] Plug memory leak in fuzzy storage * [Fix] Process high priority settings even if settings/id has been specified * [Fix] Select a different upstream on last retransmit * [Fix] Treat soft hyphen as zero width space * [Fix] Try harder to watch the lifetime of the key_stat * [Fix] Use ipv6-mapped-ipv4 addresses in radix trie * [Project] Add logic to break execution when processing symbols* * [Project] Add methods to set specific content for mime parts from Lua * [Project] Lua_content: support PDF files * [Project] Move dns_tool to using of the rspamd_spf from FFI module * [Project] Preliminary SPF plugin in Lua * [Project] Show debug stat for memory pool * [Project] Some rework about specific data that is now tagged * [Project] Start reworking of the mempool structure * [Rework] Allow to add userdata as symbols options * [Rework] Change mime part specifics handling * [Rework] Move LRU SPF cache from spf plugin * [Rework] Rework HTML tags content attachment * [Rework] Rework options hash structure * [Rework] Start lua_content library * [Rework] Stop using of uthash for http headers * [Rework] Use faster hashing approach for memory pools variables * [Rules] Add PDF related rules Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
f87b6eb314 |
Release 2.2
* [Conf] Antivirus: Fix the default config * [Feature] Add verdict library in lua * [Feature] Allow exception when choosing upstream * [Feature] Allow to disable symbols from the metric config * [Feature] Allow to limit maps per specific worker * [Feature] Always validate Rspamd protocol output * [Feature] Antivirus: Add preliminary virustotal support * [Feature] Clickhouse: Rework Clickhouse collection logic * [Feature] Improve base64 usage * [Feature] Shutdown timeout is now associated with task timeout * [Fix] #3129 Multiple classifiers on redis working incorrectly * [Fix] Allow real upstreams configuration * [Fix] Another try to fix slow callbacks and timers * [Fix] Check results of write message as SSL can bork them * [Fix] Clickhouse: Avoid potential races in collection * [Fix] Clickhouse: Fix periodic script * [Fix] Fail DNS upstream on each retransmit attempt * [Fix] Fix consistent hashing when upstreams are marked inactive * [Fix] Fix issues found * [Fix] Fix off-by-one in retries for the proxy * [Fix] Fix termination * [Fix] Fix upstreams exclusion logic * [Fix] Fix utf8 validation for symbols options and empty strings * [Fix] Oops, fix maps reload * [Fix] Rbl: Allow utf8 lookups for IDN domains * [Fix] Sigh, another try to fix brain-damaged openssl * [Project] Add fast utf8 validation library * [Project] Use own utf8 validation instead of glib * [Rework] Another phase of finish actions rework * [Rework] Further cmake system rework * [Rework] Further isolation of the controller's functions * [Rework] Make cmake structure more modular * [Rework] Move cmake modules to a dedicated path * [Rework] Replace controller functions by any scanner worker if needed * [Rework] Rework final scripts logic * [Rework] Rewrite rspamd_str_make_utf_valid function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
601e28d2ea
|
Release 2.1
* [Conf] Update neural.conf * [CritFix] Fix dkim verification for multiple headers listed * [Feature] Add support of uudecode * [Feature] Allow to explicitly set events backend * [Feature] Implement configurable limits for SPF lookups * [Feature] Lua_scanners: Use lua magic for inclusion/exclusion logic * [Feature] Multimap: Do not check files in office archives * [Feature] Neural: Add sampling when storing training vectors * [Feature] SPF: Allow to disable AAAA checks in configuration * [Feature] Spf: Add limits configuration support * [Feature] Store etag in cached HTTP maps + better logging * [Feature] Support segwit BTC addresses, fix LTC verification * [Feature] Support uuencoding * [Fix] Add configurable number of threads for OpenBLAS * [Fix] Add workaround for ragel 7 in hyperscan related maps code * [Fix] Another fix for numeric urls parsing * [Fix] Correct EMA time calculations * [Fix] Do not treat archives as text * [Fix] Do not use strdup on data extracted from lua * [Fix] Fix a failure calcuating URL reputation. * [Fix] Fix crash due to constructors init order * [Fix] Fix crash on parts with no cd * [Fix] Fix empty prefilters that require mime structures * [Fix] Fix event loop creation * [Fix] Fix issues sending DMARC reports. * [Fix] Fix misprint * [Fix] Fix saving of the file maps * [Fix] Fix size calculations when converting from utf16 * [Fix] Fix support of disable_monitoring in rbl * [Fix] Fix use-after-free * [Fix] Fix zip files check to relax requirements * [Fix] Important hiredis fixes * [Fix] Lot's of fixes in maps check logic * [Fix] Lua_tcp: Deal with temporary fails on write * [Fix] Lua_tcp: Make write errors fatal and rework error handlers * [Fix] Meta: Filter some more values * [Fix] Neural: Add protection agains infinities * [Fix] Oops, fix math.huge invocation * [Fix] Plug memory leak * [Fix] Sigh, another email to string fix * [Fix] Try to fix another ownership race in ssl connection * [Fix] Uuencode: Fix parsing of corrupted uuencode * [Fix] lua_scanners - razor rename need_check function * [Rework] Require CMake 3.9 to work, remove manual lto crap Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
8d31adb5cb
|
Release 2.0
* [Conf] Add BROKEN_HEADERS_MAILLIST composite * [Conf] Add path to greylist-whitelist-domains.inc * [Conf] Clarify documentation in the config files * [Conf] Introduce maps.d directories * [Conf] Log settings id by default * [Conf] Make LEAKED_PASSWORD_SCAM a composite rule again * [Conf] Move all surbl/emails rules to rbl * [Conf] Register new Spamhaus codes * [Conf] Remove configs for deleted modules * [Conf] Remove surbl parts, fix hash_format attribute * [Conf] Show autolearn sample * [Conf] Slashing: Change default stats backend to Redis * [Conf] Surbl: Utilise new `check_emails` option * [Conf] Update header * [Conf] Use multi-prefixes RBLs in the default config * [CritFix] Deal with case-sensivity in Content-Disposition parser * [CritFix] Eliminate old endpoint * [CritFix] Fix case sensivity when parsing Content-Type * [CritFix] Fix loading of DKIM public keys * [CritFix] Fix procesing of urls * [CritFix] Fix whitelisting when both spf and dkim are required to be valid * [CritFix] Langdet: Fix language detection where no stop words found * [Feature] Add description to the groups * [Feature] Add limit for number of URLs in Lua * [Feature] Add logging of groups to the log_format * [Feature] Add lua_smtp library * [Feature] Add maps cache and type refinement * [Feature] Add p0f scanner * [Feature] Adopt emails module to use lua_maps * [Feature] Allow options matching in composites * [Feature] Allow selectors in rbl module * [Feature] Allow to output group results * [Feature] Asn: Allow to use bgpdump when NET::MRT is broken * [Feature] Calculate tokens occurrences distribution * [Feature] Clickhouse: Add authenticated user and settings id columns * [Feature] Clickhouse: Store groups data * [Feature] Clickhouse: Utilise LowCardinality feature * [Feature] Implement Redis prefixes registration logic * [Feature] Implement settings id propagation between deps * [Feature] Improve AV results caching * [Feature] Improve autolearning * [Feature] Improve logging locking logic (remove it actually) * [Feature] Improve settings processing * [Feature] Langdet: Limit number of stop words to be checked * [Feature] Libucl: Allow to sort keys in ucl objects * [Feature] Lua_config: Extend get symbols method * [Feature] Lua_maps: Allow static maps for key-value pairs * [Feature] Lua_mimepart: Add function filter_words * [Feature] Lua_selectors: Add `words` selector * [Feature] Lua_selectors: Add sort and uniq transform functions * [Feature] Lua_selectors: Allow table arguments for selectors * [Feature] Lua_tcp: Add preliminary support of SSL connections * [Feature] Lua_trie: More flexible API * [Feature] Lua_util: Add filter_specific_url function * [Feature] Lua_util: table_digest can now recursively traverse tables * [Feature] Maillist: Improve detection * [Feature] Maps: Allow caching for complex maps * [Feature] Monitored: Support random lookups * [Feature] Multimap: Add combined maps prototype * [Feature] Multimap: Add dependend maps via redis keys selectors * [Feature] Multimap: Allow multiple email addresses matches * [Feature] Multimap: Also check detected charset when do filename checks * [Feature] Output number of messages processed to proctitle * [Feature] Perform clean SSL shutdown * [Feature] Performance: Do not use base64 SIMD version for bad inputs * [Feature] RBL: Support bit results in replies * [Feature] RBL: Support type specific prefixes * [Feature] Ratelimit: Consider number of SMTP recipients * [Feature] Rbl: Add ability to check urls * [Feature] Rbl: Add resolve_ip based RBLs * [Feature] Rbl: Make config checks much more strict * [Feature] Rbl: Support per-rule whitelists * [Feature] Rbl: Support process script * [Feature] Rbl: Support replyto addresses * [Feature] SURBL: Allow to check email domains * [Feature] Selectors: Add `list` generator * [Feature] Selectors: Add `specific_urls` extractor * [Feature] Selectors: Add flatten function * [Feature] Selectors: Support filter_map and apply_map functions * [Feature] Store Clickhouse data outside of lua alloc * [Feature] Support caching for encrypted files and macros * [Feature] Support images when extracting urls * [Feature] Support more hyperscan flags * [Feature] Support protocol flags * [Feature] URL: Apply stringprep to hostnames to filter garbage * [Feature] Upstreams: Add lazy resolving logic to all upstreams * [Feature] Upstreams: Set noresolve flag on numeric upstreams * [Feature] Use `scores` in apply section * [Feature] Use maps logic from lua_maps for multimap * [Feature] Use random monitored in rbl module * [Feature] lua_scanners - add Razor support * [Fix] Add another safe-guard in urls processing * [Fix] Add debug to ssl, fixed write hangs * [Fix] Add missing groups to C callback symbols * [Fix] Add more checks for ghosts symbols * [Fix] Allow to enable or add new actions via settings * [Fix] Allow to set 0 size for spf/dkim caches * [Fix] Another bunch of fixes towards protocol mess * [Fix] Another fix to deal with bad URLs * [Fix] Arc: Another bunch of fixes for arc signing * [Fix] Arc: More arc signing fixes * [Fix] Avoid another overflow in fpconv * [Fix] Clickhouse: Fix quoting * [Fix] Clickhouse: Fix retention query quoting * [Fix] Distinguish empty and non-empty prefilters * [Fix] Distinguish remote and local addrs parsing * [Fix] Do not assert if length of sig is bad, just fail verification * [Fix] Do not assert if we have broken mime boundary in the headers * [Fix] Do not call implicit strlen to avoid issues * [Fix] Do not count images urls when checking url regexps for compatibility * [Fix] Do not output rbl suffix in symbol option * [Fix] Do not use config pool to avoid issues with double reload * [Fix] Do not use ephemeral string * [Fix] Do not use lightuserdata for traceback * [Fix] Do not use priority in metric registration * [Fix] Emails: Check email sanity before testing on BL * [Fix] Emails: Fix misprint in key name * [Fix] Escape utf in regexp to dodge ragel/hyperscan issue * [Fix] Extend task_timeout to postfilters stage * [Fix] Fix ARC signing after fixing another bug in it... * [Fix] Fix AV scan logic * [Fix] Fix DMARC_NA behaviour in case of no valid policies * [Fix] Fix LRU hash iteration logic * [Fix] Fix alignment mess * [Fix] Fix configuring symbols without scores * [Fix] Fix disabling of the actions * [Fix] Fix dkim signing exceptions * [Fix] Fix embedded images linking logic * [Fix] Fix events leak * [Fix] Fix eviction corner case * [Fix] Fix fuzzy image score calculation #2962 * [Fix] Fix hang in fuzzy_learn when explicit rotation is set * [Fix] Fix headers propagation logic * [Fix] Fix hearbeats restart issue * [Fix] Fix history reset * [Fix] Fix log parameter * [Fix] Fix lua_ip_equal logic * [Fix] Fix more issues with nested messages + tests * [Fix] Fix normalization of non-alphabet based languages * [Fix] Fix offsets when parsing message/rfc822 in multipart * [Fix] Fix options in rbl symbols * [Fix] Fix out of bound access in lua logger * [Fix] Fix out-of-bound read in qp decode * [Fix] Fix parent CTE propagation * [Fix] Fix parsing of the received headers with empty part * [Fix] Fix pending checks for events * [Fix] Fix printing of NULL pointer with fixed length * [Fix] Fix race condition in watcher handler * [Fix] Fix read-after-end in quoted printable decoding * [Fix] Fix redis sentinel support * [Fix] Fix registry leak in case of DNS errors * [Fix] Fix reload logic * [Fix] Fix sending of large entries via HTTPS * [Fix] Fix settings reload * [Fix] Fix some more corner cases for fpconv * [Fix] Fix trie code when there are regexps and Hyperscan is absent * [Fix] Further fixes to printing of the FP numbers * [Fix] Fuzzy_check: Fix timeouts * [Fix] Grrr, fix empty ip case * [Fix] Html: Fix processing of fjlig entity * [Fix] Lang_det: Try better to distinguish Chinese and Japanese * [Fix] Lua_mime: Fix reversed extensions map * [Fix] Lua_task: Fix message-less API * [Fix] Lua_tcp: Report connection failures * [Fix] Lua_tcp: Various fixes and debugging improvements * [Fix] Metadata_exporter: This plugin is idempotent not a postfilter * [Fix] More fixes to extract_specific_urls * [Fix] More stages fixes * [Fix] Neural: Another bunch of fixes * [Fix] Neural: use version in ANN key profile * [Fix] Postpone lua state destruction to allow lua dtors to be used * [Fix] Prefer surbl/emails rule on rbl to preserve compatibility * [Fix] RBL: Fix behaviour of emails_domainonly * [Fix] Ratelimit: Fix dynamic score * [Fix] Rbl: Fix emailbl functions * [Fix] Really fix hyperscan workaround * [Fix] Set sanity limits for pcre2 * [Fix] Settings: Fix settings check flags * [Fix] Sort keys when getting data from Lua when filling rules * [Fix] Statistics: Do not query Redis tokens when there are no learns * [Fix] Stop IO event on write finished in http connection * [Fix] Use heuristically detected text parts data * [Fix] Various fixes to QP encoding algorithm * [Fix] Various fixes to SSL state machine handler * [Fix] Various fixes to asn module * [Fix] Workaround for empty charset in rfc2231 encoding * [Project] Switch from torch to KANN * [Project] Add heartbeat events * [Project] Add preliminary support of the Kaspersky Scan Engine * [Project] Add preliminary version of maps expressions * [Project] Add preprocessed settings to the config structure * [Project] Add simple forward propagation function * [Project] Add small helpers for migration simplifications * [Project] Allow to replace body in milter * [Project] Bundle libev * [Project] First refactoring step libevent->libev * [Project] Implement syntax highlighting for Lua * [Project] Lua_magic: Adopt lua_magic stuff in mime_types * [Project] Remove libfann, gd and other unsupported stuff * [Project] Remove torch * [Project] Rework upstreams * [Rework] Allow execution of async events when hs compiles regexps * [Rework] Bayes expiry: eliminate `default` expiration mode * [Rework] Dkim: Remove signing code * [Rework] Dkim_signing: Move sign condition to dkim_signing * [Rework] Do not lowercase all data send to ClickHouse * [Rework] Drop url tags * [Rework] Eliminate lua_squeeze as it has shown no improvements * [Rework] Eliminate virtual scan time as it is useless * [Rework] Lua core: Use lightuserdata to index classes * [Rework] Lua_util: Another rework for extract_specific_urls * [Rework] Migrate from ip_score to reputation * [Rework] Move mime modification functions to lua_mime library * [Rework] Rbl: Major whitelisting logic rework * [Rework] Remove deprecated plugins * [Rework] Remove log helper worker * [Rework] Remove rspamd.classifiers.lua * [Rework] Rename filter.h to a more sane name * [Rework] Reorganise selectors implementation * [Rework] Replace linenoise with replxx * [Rework] Reputation: Remove ipnet from the ip reputation * [Rework] Reputation: Slashing - change name of symbols * [Rework] Rework children operations * [Rework] Rework config reload * [Rework] Rework expression API * [Rework] Rework image urls processing * [Rework] Rework initialisation to reduce static leaks count * [Rework] Rework request headers processing * [Rework] Slashing: Change versioning schema - move to 2.0 * [Rework] Slashing: Turn off postfilters when passthrough result is set * [Rework] Start moving to replxx * [Rework] Stop support of signed HTTP maps to simplify code * [Rework] Store ASN as UInt32 in ClickHouse * [Rework] Url_redirector: Rewrite plugin * [Rework] Use a dedicated library for autolearn * [Rework] Use libsodium instead of hand crafted crypto implementations * [Rework] Use opaque structure to store a table of mime headers * [Rules] Add dedicated bitcoin addresses filter rule * [Rules] Add more detection to LEAKED_PASSWORD_SCAM * [Rules] Catch LTC addresses * [Rules] Reduce weight of RSPAMD_EMAILBL * [Rules] Rework LEAKED_PASSWORD_SCAM rule one more time Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
4 years ago |
|
Vsevolod Stakhov
|
cc1aef5ea4 | [Minor] Add Changelog for 1.9.4 | 5 years ago |
|
Vsevolod Stakhov
|
fe91396af3
|
Release 1.9.3
* [Conf] Add IP_SCORE_FREEMAIL composite rule
* [Feature] Add cryptobox method to generate dkim keypairs
* [Feature] Add fast hashes to lua cryptobox hash
* [Feature] Add least passthrough results
* [Feature] Allow oversign if exists mode
* [Feature] Clickhouse: Modernise table initial schema
* [Feature] Implement IUF interface for specific fast hashes
* [Feature] Lua_util: Allow to obfuscate different fields
* [Feature] Tune memory management in Rspamd and Lua
* [Fix] Avoid buffer overflow when printing long lua strings
* [Fix] Change the default oversigning headers to a more sane list
* [Fix] Clickhouse: Do not store digest as it is not needed now
* [Fix] Clickhouse: Fix lots of storage issues
* [Fix] Clickhouse: Support custom actions
* [Fix] Deny URLs where hostname is bogus
* [Fix] Do not blacklist mail by SPF/DMARC for local/authed users
* [Fix] Fix DoS caused by bug in glib
* [Fix] Fix UCL parsing of the multiline strings
* [Fix] Fix buffer overflow when printing small floats
* [Fix] Fix init code for servers keypairs cache
* [Fix] Fix issue with urls with no tld (e.g. IP)
* [Fix] Fix memory in arc signing logic
* [Fix] Fix memory leak in language detector during reloads
* [Fix] Fix mixed case content type processing
* [Fix] Fix processing of the ip urls in file
* [Fix] Fix use after free
* [Fix] HTML: Fix `size` attribute processing
* [Fix] Hum, it seems that
|
5 years ago |
|
Vsevolod Stakhov
|
fb577bd5a3
|
Release 1.9.2
* [Conf] Allow to load users plugins from plugins.d * [Conf] oversign openpgp and autocrypt headers * [Feature] Add SPF FFI library for Lua * [Feature] Add more verbosity for SPF caching * [Feature] Antivirus: Handle encrypted files specially * [Feature] Clickhouse: Slashing - add new fields to CH * [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table * [Feature] Dkim_signing: Allow to use new options as maps * [Feature] Import fpconv library * [Feature] Lua_maps: Allow static regexp and glob maps * [Feature] Parse ical files * [Feature] Rspamadm: Add dns_tool utility * [Feature] Store SPF records digests * [Feature] Use fpconv girsu2 implementation for printing floats * [Fix] Clickhouse: Use integer seconds when inserting rows * [Fix] Fix floating point printing * [Fix] Fix processing of embedded urls * [Fix] Lua_clickhouse: Fix CH errors processing * [Fix] Make spf digest stable * [Fix] Properly detect encrypted files in zip archives * [Fix] Slashing: Store times in GMT timezone in ClickHouse * [Rules] Add additional conditions to perform BTC checks * [Rules] Fix pay-to-hash addresses validation Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
38ca7b5e10
|
Release 1.9.1
* [Conf] Add vendor groups for symbols * [Feature] Add `rspamadm template` command * [Feature] Allow to add messages from settings * [Feature] Allow unconnected DNS servers operations * [Feature] Check limits after being set, migrate to uint64 * [Feature] Greylist: Allow to disable greylisting depending on symbols * [Feature] Improve lua binary strings output * [Feature] Mime_types: Implement user configurable extension filters * [Feature] Mime_types: When no extension defined, detect it by content * [Feature] Preprocess config files using jinja templates * [Feature] Replies: Filter replies sender to limit whitelisting to direct messages * [Feature] Treat all tags with HREF as a potential hyperlinks * [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM * [Fix] Add crash safety for HTTP async routines * [Fix] Another fix for Redis sentinel * [Fix] Clickhouse: Fix table schema upload * [Fix] Core: Fix squeezed dependencies handling for virtual symbols * [Fix] Finally fix default parameters parsing in actions section * [Fix] Fix ES sending logic (restore from coroutines mess) * [Fix] Fix finishing script for clickhouse collection * [Fix] Fix priority for regexp symbols registriation * [Fix] Fix various issues found by PVS Studio * [Fix] Initialize lua debugging earlier * [Fix] Neural: Fix training * [Fix] Rework cached Redis logic to avoid sentinels breaking * [Fix] SURBL: Fix regression in surbl module * [Fix] Fix double signing in the milter * [Project] Add support of HTTP proxy in requests * [Rework] Change lua global variables registration * [Rework] Rework HTML content urls extraction * [Rework] Start rework of aliasing in Rspamd * [WebUI] Combine Scan and Learning into one tab * [WebUI] Fix symbol score input type * [WebUI] Show grayed out pie * [WebUI] Update Throughput summary values dynamically Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
0e2437493e
|
Release 1.9.0
* [Conf] Add missing includes
* [Conf] Move to options
* [Conf] Rbl: DWL is actually special whitelist
* [Conf] Relax some uribl rules
* [Conf] Remove abuse.ch
* [CritFix] Html: Entities are not valid within tag params values
* [Feature] Add `rspamadm mime sign` tool
* [Feature] Add configgraph utility
* [Feature] Add dedicated ZW spaces detection for URLs
* [Feature] Add flag to url object when visible part is url_like
* [Feature] Add method task:lookup_words
* [Feature] Add pyzor support (by crosenberg)
* [Feature] Allow to add upstream watchers to Lua API
* [Feature] Allow to set rewrite subject pattern from settings
* [Feature] Better escaping of unicode
* [Feature] Clickhouse: Allow to store subject in Clickhouse
* [Feature] Core: Add QP encoding utility
* [Feature] Core: Add libmagic detection for all parts
* [Feature] Core: Add support for gzip archives
* [Feature] Core: Allow to construct scan tasks from raw data
* [Feature] Core: Detect charset in archived files
* [Feature] Core: Ignore and mark invisible spaces
* [Feature] Core: Normalise zero-width spaces in urls
* [Feature] Core: Process data urls for images
* [Feature] Core: Relax quoted-printable encoding
* [Feature] Core: Support RFC2231 encoding in headers
* [Feature] Core: Support telephone URLs
* [Feature] Core: allow to emit soft reject on task timeout
* [Feature] DCC: Add bulkness and reputation checks to dcc
* [Feature] Elastic: Modernize plugin
* [Feature] Export visible part of url to lua
* [Feature] Fuzzy_storage: add preliminary support of rate limits
* [Feature] HTML: Specially treat data urls in HTML
* [Feature] Implement event watchers for upstreams
* [Feature] Implement includes tracing in Lua
* [Feature] Improve dkim part in configwizard
* [Feature] Lua_scanners: Add VadeSecure engine support
* [Feature] Lua_task: Add flexible method to get specific urls
* [Feature] Mime_types: Add MIME_BAD_UNICODE rule
* [Feature] Mime_types: Use detected content type as well
* [Feature] Plugins: Add preliminary version of the external services plugin
* [Feature] Query sentinel on master errors
* [Feature] Regexp: Allow local lua functions in Rspamd regexp module
* [Feature] Rspamadm: Allow to append footers to plain messages
* [Feature] Rspamadm: Allow to rewrite headers in messages
* [Feature] Selectors: Add `ipmask` processor
* [Feature] Settings: Allow hostname match
* [Feature] Settings: Allow local when selecting settings
* [Feature] Settings: Allow multiple selectors
* [Feature] Settings: Allow to inverse conditions
* [Feature] Support User-Agent in HTTP requests
* [Feature] Support
|
5 years ago |
|
Vsevolod Stakhov
|
1b9f2f3026
|
Release 1.8.3
* [CritFix] Make flags mutually exclusive for mime parts * [CritFix] Strictly deny unencoded bad utf8 sequences in headers * [Feature] Add Kaspersky antivirus support * [Feature] Add method to get dkim results * [Feature] Add more words regexp classes * [Feature] Allow to choose words format in `rspamadm mime` * [Feature] Allow to get all types of words from Lua * [Feature] Allow to get task flags in C expressions * [Feature] Allow to require encryption when accepting connections * [Feature] Ignore bogus whitespaces in the words * [Feature] Implement more strict configuration tests * [Feature] Improve SPF results in Authentication-Results * [Feature] Support ClickHouse database * [Fix] Add failsafety for utf8 regexps * [Fix] Do not trigger BROKEN_CONTENT_TYPE on innocent text parts * [Fix] Emit error if connection has been terminated with no stop pattern * [Fix] Fix boundaries checks in embedded messages * [Fix] Fix double free * [Fix] Perform policy downgrade on sample out, add tests * [Fix] Properly escape utf8 regexps in hyperscan mode * [Fix] Selectors - attachments args condition * [Fix] Some fixes for raw parts * [Fix] Treat learning errors as non-fatal * [Fix] Use tld when looking for DKIM domains * [Project] Words unicode structure rework * [Project] Add preliminary Redis Sentinel support * [Project] Improve Authentication-Results header * [Project] Rework DKIM checks results * [Project] Use more generalised API to produce meta words Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
dcab7362e0
|
Release 1.8.2
* [Conf] Add DWL support in the default configuration * [Conf] Disable rspamd_update by default (again) * [Conf] Fix configuration sample for ratelimit * [CritFix] Disable broken url tags by default * [CritFix] Fix \0 processing when doing RSA sign * [CritFix] Fix adding symbols to their primary groups * [Feature] Add `rspamadm cookie` utility * [Feature] Add specialised functions for generating encrypted cookies * [Feature] Add support of cookies in replies module * [Feature] Add support of words regexps * [Feature] Allow to add 3rd party clang plugins * [Feature] Allow to create lua regexps from glob or plain patterns * [Feature] Allow to set custom limits for upstream lists * [Feature] Detect orphaned parts and attach them to message * [Feature] Filter tokens in bayes * [Feature] Fold b= value when doing arc sealing * [Feature] Ignore cookies in the future and too old in the past * [Feature] Skip stop words in statistics * [Feature] Store stop words and allow to query them * [Feature] Support query arguments in controller's custom commands * [Feature] Tune upstream limits in Rspamd proxy * [Feature] Use different callback symbols for different uribls * [Feature] Write DKIM selector in dkim allow/reject symbols * [Fix] Add obs_fws state support to eoh state machine * [Fix] Add sanity check when applying mime boundaries heuristic * [Fix] Antivirus - virus names with 0 were recognized as tables * [Fix] Disable headernames in bayes temporarily * [Fix] Do not allow syntax errors in include files... * [Fix] Do not allow to merge an object with an array (or vice versa) * [Fix] Don't perform forged recipients check for missing recipients * [Fix] Fix DKIM based RBLs * [Fix] Fix actrie implementation (sync from upstream), fixed OOB read * [Fix] Fix explicit methods call in selectors * [Fix] Fix extraction of additional parts * [Fix] Fix finalization for internal plugins * [Fix] Fix override_defaults function * [Fix] Fix squeezed symbols when using settings * [Fix] Fix urls insertion in Clickhouse module * [Fix] Furhter fixes to ratelimits logic * [Fix] Ignore signatures when looking for boundaries * [Fix] Properly set learned count * [Fix] Really fix ratelimits configuration and work * [Fix] Remove ambigious format flag from printf * [Fix] Restore URLs exporting in ClickHouse plugin * [Fix] Rework bayes calculations... * [Fix] Switch from chi-square to naive for large Fisher value * [Fix] Treat normal password as enable password if there is no enable password * [Fix] Use proper syntax for making DNS requests * [Fix] Various fixes in embedded plugins * [Project] Change fuzzy check selection logic to lua_fuzzy library * [Project] Rework async events and symbols * [Project] Move all metatokens in Bayes to lua_stat from C * [WebUI] Add history rows per page control Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
6fe627f31f
|
Release 1.8.1
* [CritFix] Fix options insertion * [CritFix] Fix words decay one more time (affects long messages) * [CritFix] Increase default words_decay * [CritFix] Plug memory leak in redis pool * [Feature] Add `check_violation` feature to DKIM/ARC signing * [Feature] Add only unique elements to Clickhouse url arrays * [Feature] Allow `g+:` and `g-:` composite atoms * [Feature] Allow dkim domains check in surbl * [Feature] Allow maps with HTTP auth * [Feature] Allow to disable actions by users settings * [Feature] Extend whitelisting options * [Feature] Store url object in images * [Feature] Use verdict instead of the plain action in plugins * [Fix] Allow to call fstring append with NULL string * [Fix] DCC - luacheck * [Fix] Do not load torch on each rspamadm invocation * [Fix] Fix boundaries detection and rework stop words algorithm * [Fix] Fix dependencies for DNS_SIGNED symbol * [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit * [Fix] Fix groups mess * [Fix] Fix groups mess * [Fix] Fix parsing address with comments * [Fix] Fix resolving in DMARC reports * [Fix] Fix various issues with parsing of the received headers * [Fix] Fix watchers issue in lua_tcp when doing no resolving * [Fix] Plug memory leak in language detector (affects reloads) * [Fix] Remove one letter stop words * [Fix] Slashing: backport chunk logic from libucl * [Fix] Stop libevent from using cached time in rspamadm * [Fix] Try to fix watchers chaining * [Fix] Various fixes in redis sync interface * [Fix] ip_score - respect check_authed and check_local settings from config * [Project] Rework passthrough actions * [Project] Clustering module * [Rework] Always create result for a task * [Rework] Completely rewrite DMARC checks logic * [Rework] Rework and fix whitelist plugin * [WebUI] Add symbols sorting buttons * [WebUI] Change symbols order without updating history * [WebUI] Colorize symbols * [WebUI] Do not display password form when secure_ip is set * [WebUI] Fix symbol description tooltips display * [WebUI] History: add sorting by symbol score value Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
f7f7588e22
|
Release 1.8.0
* [Feature] Add arguments schemas to processors and extractors * [Feature] Add functional selectors library * [Feature] Add generic selector to reputation module * [Feature] Add more ratelimits: by digest, by attachments data, by filenames * [Feature] Add preliminary stop words detection support * [Feature] Add pure Lua debugm function * [Feature] Add schema validation for Redis settings * [Feature] Add selectors combine function * [Feature] Add some recursion protection to lua logger * [Feature] Add support for Lua API tracing * [Feature] Allow to apply schema to arguments * [Feature] Allow to get dkim signing data directly from HTTP headers * [Feature] Allow to reuse existing authentication results * [Feature] Cache selectors results in re runtime * [Feature] Implement new text tokenizer based on libicu * [Feature] Integrate selectors framework to multimap * [Feature] Relax FORGED_RECIPIENTS * [Feature] Support (almost) all html entities * [Feature] Support adding and deletion of recipients in the milter block * [Feature] Support gathering HTTP body from fragments in lua_http * [Feature] Support multi flag in regexp and glob maps * [Feature] Support selectors in ratelimit module * [Feature] Support selectors in settings * [Feature] Use khash in HTML parser * [Feature] Use pure Lua debugm function * [Fix] Add fail-safety for destroying sessions * [Fix] Allow to add result-less fake DNS records * [Fix] Another try to fix race conditions on config unload * [Fix] Call Lua callback on DNS timeouts * [Fix] Deprecate task:inc_dns_req as it is redundant * [Fix] Do not allow events deletions on cleanup * [Fix] Do not try to process skipped messages * [Fix] Fix HTTP requests with no body * [Fix] Fix another cleanup race condition * [Fix] Fix bug in processing of pcre regexps * [Fix] Fix byte array allocation in the pool * [Fix] Fix crashes on task cleanup * [Fix] Fix dynamic buckets in ratelimits * [Fix] Fix endless loop when waiting for Rspamd to stop * [Fix] Fix lua_util.str_split in case of delimiters set * [Fix] Fix more issues with watching of async events * [Fix] Fix stop words detection and loading logic * [Fix] Fix various corner cases for language detection * [Fix] Fix watchers in lua_tcp * [Fix] Fix words decay algorithm * [Fix] Implement watchers replacement to handle nested calls * [Fix] Save faked code into fake dns record * [Fix] Show the proper frame when using lua_util.debugm * [Fix] Use fake dns records in tests * [Fix] Use unicode replacements for HTML entities * [Fix] fixed "cannot find dependency on symbol 1" issue when using replaced symbols in spamassassin rules * [Fix] partition_id is not available in old versions of CH * [Project] Add implicit conversion logic to selectors * [Project] Add initial support for selectors in regexps * [Project] Add method concept * [Project] Further changes in unicode operations * [Project] Implement Clickhouse migrations * [Project] Implement implicit conversions to userdata * [Project] Implement insert method * [Project] Implement selectors registration for regular expressions * [Project] Implement selectors support in re_cache * [Project] Improve language detector: cleanup unused files, categorize * [Project] Migrate CH data to a fat table * [Project] Rework selectors logic * [Project] Start Clickhouse utilities library * [Project] Start unicode rework * [Project] coroutine threaded model for API calls: thread pool * [Rework] Move phishtank to a DNS based service * [Rework] Rework Clickhouse plugin to use the new API * [Rework] Rework language detector * [Rework] Rework utf content processing in text parts * [WebUI] Add progress bar for AJAX requests * [WebUI] Avoid errors table reinitialization * [WebUI] Avoid history table reinitialization * [WebUI] Avoid throughput summary table reinitialization * [WebUI] Destroy summary table on disconnect * [WebUI] Fix "auth" request URL * [WebUI] Fix disabling and hiding controls on page reload * [WebUI] Fix maps loading from neighbours * [WebUI] Fix symbols sorting by score * [WebUI] Fix tables destroying * [WebUI] Fix throughput data consolidation * [WebUI] Fix upload buttons disabling * [WebUI] Notify user on module loading failure * [WebUI] Update FooTable 3.1.4 -> 3.1.6 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
f63ec0e39b
|
Release 1.7.9
* [CritFix] Fix caseless comparison of equal length strings * [Feature] Add HTTP basic auth support to elastic and clickhouse plugins * [Feature] Add SPF selector to reputation * [Feature] Add support of the fallback backends for HTTP maps * [Feature] Allow to print full mime structure when extracting mime data * [Feature] Allow to split symbols in reputation plugin * [Feature] Check attachments only on AV scanners in attachments_only mode * [Feature] Disable all SSL checks if ssl_no_verify flag is set * [Feature] Implement parsing of scoped IPv6 addresses * [Feature] Improve `rspamc counters` output * [Fix] Add sanity checks when expanding SPF macros * [Fix] Allow to parse SA rules with no spaces around =~ (dirty hack) * [Fix] Avoid one extra byte writing * [Fix] Deal with direct hash table * [Fix] Detect empty text part as text, not HTML * [Fix] Do not reduce map watch timeout for mixed http/file maps * [Fix] Fix HTML part detection heuristic * [Fix] Fix double free in redirectors cleanup * [Fix] Fix legacy history handling in the controller * [Fix] Fix messages insertion * [Fix] Fix sending string method * [Fix] Fix statconver command line arguments * [Fix] Fixed argument checking for being null * [Fix] Fixed issues reported by luacheck * [Fix] Freeze updates queue when do actual storage update * [Fix] HTTP map hash is per-backend and not per-map * [Fix] Plug memory leak in fuzzy updates * [Fix] Prefer 'MTA-Name' when producing authentication results * [Fix] Replace bad unicode sequences instead of stopping on them * [Fix] Set classifier version on learning * [Project] Reworked ratelimits * [Project] Apply topological sorting for symbols in Rspamd * [Project] Remove global contexts from C modules * [Project] Move performance critical hash tables to khash * [WebUI] Avoid unused indexes * [WebUI] Do not execute `on_success` callback * [WebUI] Fix history reset for "All SERVERS" (#2346) * [WebUI] Fix query URL for selected server * [WebUI] Fix symbols display in legacy history, * [WebUI] Hide symbols order selector for legacy history * [WebUI] Refactor query functions into one * [WebUI] Remove previously-attached event handlers * [WebUI] Save symbols to the selected server * [WebUI] Unify arguments of query functions * [WebUI] Use common query functions to get graph data * [WebUI] Use common query functions to save symbols Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
4c80975059
|
Release 1.7.8
* [Feature] Add more extended statistics about fuzzy updates * [Feature] Add more non-conformant Received headers support * [Feature] Add preliminary function to get fuzzy hashes from text in Lua * [Feature] Allow to configure AV module rejection message * [Feature] Implement fuzzy hashes extraction in mime tool * [Feature] Improve WHITE_ON_WHITE rule * [Feature] Improve integer -> string conversion * [Feature] Reuse maps in multimap module more aggressively * [Fix] Avoid race condition in skip map as pool lifetime is not enough * [Fix] Eliminate all specific C plugins pools * [Fix] Fix DKIM check rule if DNS is unavailable * [Fix] Fix build where ucontext is defined in ucontext.h * [Fix] Fix crash in base url handling * [Fix] Fix descriptors leak in sqlite3 locking code * [Fix] Fix messages quarantine * [Fix] Fix padded numbers printing * [Fix] Fix race condition on maps reinit * [Fix] Fix regexp functions when no data is passed * [Fix] Fix specific urls extraction * [Fix] Fix styles propagation * [Fix] Improve resetting of the limit buckets * [Fix] Initialize sqlite3 properly * [Fix] Work with broken resolvers in resolv.conf * [Project] Implement HTTP maps caching * [Project] Refresh fuzzy hashes when matched * [Project] Add logic to deduplicate fuzzy updates queue * [WebUI] Add missed declarations * [WebUI] Avoid using "undefined" property * [WebUI] Do not accept passwords containing control characters * [WebUI] Do not redeclare variables * [WebUI] Enable strict mode, * [WebUI] Fix variable assignment * [WebUI] Initialize variables at declaration * [WebUI] Remove duplicated path from RequireJS config * [WebUI] Remove unused block * [WebUI] Remove unused variable * [WebUI] Remove unused variables * [WebUI] Use self-explanatory notation * [WebUI] Use type-safe equality operators Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
5 years ago |
|
Vsevolod Stakhov
|
a20e33c22d
|
Release 1.7.7
* [CritFix] Check NM part of pubkey to match it with rotating keypairs
* [CritFix] Do not overwrite PID of the main process
* [CritFix] Fix maps after reload
* [CritFix] Fix maps race conditions on reload
* [CritFix] Fix shmem leak in encrypting proxy mode
* [Feature] Add a concept of ignored symbols to avoid race conditions
* [Feature] Add ability to print bayes tokens in rspamadm mime
* [Feature] Add method to get statistical tokens in Lua API
* [Feature] Add preliminary mime stat command
* [Feature] Add rspamadm mime tool
* [Feature] Add urls extraction tool
* [Feature] Address ZeroFont exploit
* [Feature] Allow rspamadm mime to process multiple files
* [Feature] Allow to extract words in `rspamadm mime`
* [Feature] Allow to print mime part data
* [Feature] Allow to show HTML structure on extraction
* [Feature] Distinguish IP failures from connection failures
* [Feature] Improve output for mime command
* [Feature] Improve styles propagation
* [Feature] Main process crash will now cleanup all children
* [Feature] Preload file and static maps in main process
* [Feature] Print stack trace on crash
* [Feature] Process font size in HTML parser
* [Feature] Propagate content length of invisible tags
* [Feature] Read ordinary file maps in chunks to be more safe on rewrites
* [Feature] Support base tag in HTML
* [Feature] Support more size suffixes when parsing HTML styles
* [Feature] Support opacity style
* [Fix] Another fix for nested composites
* [Fix] Fill nm id in keypairs cache code
* [Fix] Fix colors alpha channel handling
* [Fix] Fix destruction logic
* [Fix] Fix double free
* [Fix] Fix maps preload logic
* [Fix] Fix nested composites process
* [Fix] Fix proxying of Exim connections
* [Fix] Fix reload crash
* [Fix] Fix rspamadm -l command
* [Fix] Update
|
5 years ago |
|
Vsevolod Stakhov
|
d9c84b4bd0
|
Release 1.7.6
* [CritFix] Fix multiple neural networks support * [Feature] Add decryption function to keypair command * [Feature] Add gzip compression for HTTP requests in elastic module * [Feature] Add gzip methods to lua util * [Feature] Add maps based on Top Level Domains * [Feature] Add pubkey checks for dkim_signing * [Feature] Add support of fake DNS records * [Feature] Add tool to encrypt files * [Feature] Allow to add symbols using settings directly * [Feature] Allow to match private and public keys for DKIM signatures * [Feature] Allow to set task flags via settings * [Feature] Allow to specify fake DNS address from the config * [Feature] Implement signatures verification using rspamadm keypair * [Feature] Implement signing using `rspamadm keypair` * [Feature] Improve error reporting for DKIM key access issues * [Feature] Provide $HOSTNAME variable in UCL * [Feature] Rework levenshtein distance computation * [Feature] Split message parsing and processing * [Feature] Support ED25519 DKIM signatures * [Feature] Support encrypted configs in UCL * [Feature] Suppress duplicate warning on very large radix tries * [Feature] Use OSB to combine header names * [Fix] Cleanup maps data on shutdown * [Fix] Fix '~' behaviour in composites * [Fix] Fix HTTP maps updates * [Fix] Fix NIST signatures * [Fix] Fix RFC822 comments when processing a mime address * [Fix] Fix double free * [Fix] Fix dynamic settings application * [Fix] Fix for CommuniGate Pro maillist * [Fix] Fix keypair creation method to actually create keypair... * [Fix] Fix matching patterns with no paths * [Fix] Fix memory leak in parsing comments * [Fix] Fix parsing of urls with numeric password * [Fix] Fix plugins intialisation in configwizard * [Fix] Fix potential crash on reload * [Fix] Fix potential race condition for a finished HTTP connections * [Fix] Fix race-condition leak on processes reload * [Fix] Fix signing in openssl mode * [Fix] Free language detector structures * [Fix] Relax alignment requirements * [Fix] Send DMARC reports compressed * [Fix] Try to fix leak in dmarc module * [Fix] Try to plug memory leak in metric exporter * [Project] Convert rspamadm subcommands to Lua * [WebUI] Display smtp sender/recipient in history * [WebUI] Fix elements disabling in "Symbols" tab * [WebUI] Limit recipients list in history column to 3 * [WebUI] Match envelope and mime addresses following in arbitrary order * [WebUI] Update column header * [WebUI] Wrap addresses in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
341222861d
|
Release 1.7.5
* [Conf] Add MSBL proposed return codes * [Conf] Add additional groups for policies * [CritFix] Do not use volatile Lua strings as UCL keys * [Feature] Add ability to add fuzzy hashes to headers * [Feature] Add function to extract most meaningful urls * [Feature] Add rule to block mixed text and encrypted parts * [Feature] Allow multiple groups for symbols * [Feature] Allow to disable lua squeezing logic * [Feature] Allow to get multipart children in Lua * [Feature] Allow to insert multiple headers from milter headers * [Feature] Allow to print scores in subject and further extensions * [Feature] Be more error-prone in squeezed rules * [Feature] Support multiple return codes in emails module * [Feature] Use EMA for calculating averages * [Feature] Use common jit cache for all regexps * [Feature] support for CommuniGate Pro self-generated messages * [Fix] Allow to have multiple values for headers as arrays * [Fix] Do not open sockets for disabled workers * [Fix] Fix AuthservId * [Fix] Fix base64 folding in Lua API * [Fix] Fix build on non-x86 platforms * [Fix] Fix cached maps logic * [Fix] Fix compatibility with old maps query logic * [Fix] Fix crash if skip_map is used * [Fix] Fix importing static maps from UCL * [Fix] Fix parsing of unix sockets * [Fix] Fix raw_mime regexp on HTML part with no text content * [Fix] Fix tables logging * [Fix] Fix vertical tab handling in libucl * [Fix] Try to fix frequency counters * [Fix] Use better sharding for ip_score * [Fix] Use multiple results from SURBL DNS reply * [Fix] When doing AV scan select a different server for retransmit Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
72b3e29432
|
Release 1.7.4
* [Conf] Major stock config updates:
- Workers are now specified in a new format worker "type" { ... }
- Enable fuzzy worker to simplify local fuzzy storages configuration
- Bind all workers to localhost by default to avoid security flaws
* [Conf] Make more sane fuzzy_check default settings
* [CritFix] Fix ucl escape for bad symbols
* [Feature] Add failure symbol for AV module
* [Feature] Add lazy expiration mode for new classifier schema
* [Feature] Add preliminary version of maps stats plugin
* [Feature] Allow to block fuzzy requests from specific networks
* [Feature] Allow to change `expire` of live statistics
* [Feature] Distinguish AV failure from clean result
* [Feature] Further improvements of language detector by using khash
* [Feature] Further optimization of the lang_detection
* [Feature] Implement cluster-aware bayes expiry
* [Feature] Implement exclude patterns in rspamc
* [Feature] Implement glob maps in addition to regexp maps
* [Feature] Implement map statistics function for lua API
* [Feature] Implement stop symbols for Clickhouse collection
* [Feature] Support recipients separated by commas
* [Feature] Try harder to upload scripts to the Redis server
* [Feature] Upgrade t1ha distribution
* [Feature] use_domain_sign_inbound
* [Feature] Use scores from maps if `symbols_set` is not defined
* [Fix] Add resolving version of radix map helper
* [Fix] Check URL before adding implicit prefix
* [Fix] Do not check pid/state when using PRNG
* [Fix] Fix CentOS logrotate script for systemd
* [Fix] Fix slash + dot in urls
* [Fix] Fix systemd version of the logrotate script
* [Fix] Propagate key when import implicit array from Lua
* [Fix] Strip spaces from map keys and values
* [Fix] Try to fix a specific case when processing milter protocol
* [Fix] Try to fix crash when a tcp connection cannot be set
* [Fix] Typo use_domain_local --> use_domain_sign_local
* [Fix] Various fixes to once_received module
* [Project] Store hits counters for map elements
Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
|
6 years ago |
|
Vsevolod Stakhov
|
4f481f93bc
|
Release 1.7.3
* [CritFix] Plug bad memory leak in protocol reply * [Feature] Add avx2 codec for base64 * [Feature] Add method to receive all URL flags from Lua API * [Feature] Allow to fold headers on stop characters * [Feature] Allow to set lua_cpath from options * [Feature] Allow to specify custom rejection message in milter * [Feature] Deal with unnormalised Unicode obfuscation * [Feature] Do not detect language twice for relative parts * [Feature] Implement oversigning feature * [Feature] Implement silent logging level to minimize noise in logs * [Feature] Improve URL_IN_SUBJECT rule * [Feature] Use hashing to reduce redis attack surface * [Fix] Add oversigning for the most important headers * [Fix] add 'rewrite subject' to History dropdown * [Fix] Another fix in folding algorithm * [Fix] Do not call multimap addr for parts of addr if filter is presented * [Fix] Do not clean hostname on generic reset * [Fix] Do not create pid file in no-fork mode * [Fix] Fix fold_after case to preserve multiple spaces * [Fix] Fix folding and folding tests * [Fix] Fix hostname usage in milter mode * [Fix] Fix lua RSA verify and its tests * [Fix] Fix metadata exporter send_mail backend (#2124) * [Fix] Fix processing of '\v' in libucl * [Fix] Fix shemaless URLs detection * [Fix] Fix support of multiple headers in sign_header * [Fix] Fix usage of util.parse_mail_address * [Fix] Fix weights of dynamic squeezed rules * [Fix] Leak from bucket before checking the burst * [Fix] Stop using own localtime as DST could be messy in many cases * [Fix] Treat unnormalised URLs as obscured * [Rework] Restore leaky bucket model in ratelimit plugin * [WebUI] Add messages total to throughput summary * [WebUI] Add symbols order selector to history * [WebUI] Config: Load list on demand * [WebUI] Fix modalBody for maps that appear more than once * [WebUI] History: Fix Tooltips on paging, filtering and sorting * [WebUI] Remove a previously-attached event handler * [WebUI] Update D3 to v5.0.0 and jQuery to v3.3.1 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
8073aece67
|
Release 1.7.2
* [Feature] Store emails in Clickhouse * [Feature] Support single quotes in config * [Feature] Use templates when publishing CH schema * [Feature] Improve Docker image * [Fix] Add rounding when printing a lot of FP variables * [Fix] Allow to disable certain actions by assigning null to them * [Fix] Disable results caching * [Fix] Fix disabling of squeezed symbols * [Fix] Fix scan time set * [Fix] Rework logic of actions setting * [Fix] Try to fix various Lua stack issues * [WebUI] Add link tag for favicon.ico * [WebUI] Display hostname:port/path in the page title Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
3ba34325ab
|
Release 1.7.1
* [CritFix] Fix lowercase comparison * [CritFix] Timezone defines seconds WEST UTC not East * [Feature] Add filename to log format * [Feature] Add lua rules squeezing * [Feature] Add related symbols analysis to rspamd_stats * [Feature] Remove upstream `X-Spam: Yes` header by default * [Feature] rspamd_stats: Output progress info on STDERR * [Feature] Whitelist for emails module * [Fix] Do not allow dependencies on self * [Fix] Do not cache metric result * [Fix] Do not trust all issuers as a client certificate * [Fix] Fix dependencies in lua squeeze * [Fix] Fix enabling/disabling squeezed rules * [Fix] Fix enabling/disabling symbols * [Fix] Fix external dependencies * [Fix] Fix processing of a single compressed file * [Fix] Fix some typos * [Fix] Fix various modules in case of empty message * [Fix] Handle callbacks that returns table of options * [Fix] Improve cached action interaction * [Fix] Make dynamic conf more NaN aware * [Fix] Never hide actions from WebUI `configuration` tab * [Project] Implementation of Lua rules squeezing Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
720ae30a31
|
Release 1.7.0
* [Conf] Add bayes_expiry as explicit module * [Conf] Adjust names and weights for neural network plugin * [Conf] Change updates url * [Conf] Default statistics is stored in Redis now * [Conf] Disable fann_redis module by default * [Conf] Fix default elastic configuration * [Conf] Fix double quote position * [Conf] Massive config rework for new structure of symbols and scores * [Conf] Rename Rambler BLs as they are now Rspamd's ones * [Conf] Use dedicated rspamd.com subdomains * [Conf] Use more data from rspamd.com fuzzy storage * [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Another portion of tokenization fixes * [CritFix] Do not send reject messages after set reply * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [CritFix] Fix memory leak in spf caching logic * [CritFix] Fix milter commands pipelining * [CritFix] Fix newlines detection * [CritFix] Fix semicolons parsing in the content type * [CritFix] Plug memory leak in zstd protocol compression * [Feature] Add ability to match score in force_actions module * [Feature] Add aes-rng PRF to libottery * [Feature] Add 'composites' debug module * [Feature] Add concept of experimental modules * [Feature] Add DKIM trace symbol * [Feature] Add EBL to the default config * [Feature] Add expected ip check for emails plugin * [Feature] Add framework to manage Redis scripts * [Feature] Add framing for the new reputation generic plugin * [Feature] Add function to show plugins stat * [Feature] Add gzip compression support for clickhouse module * [Feature] Add gzip compression support for rspamd controller * [Feature] Add gzip support when sending lua http requests * [Feature] Add json output for rspamd_stats * [Feature] Add method to do a synchronous Redis connection * [Feature] Add method to get all content-type attributes in Lua * [Feature] Add `-m` flag to configdump to show modules states * [Feature] Add mime types to extensions map * [Feature] Add more features to rescore utility * [Feature] Add more gtube like patterns to test other spam actions * [Feature] Add more metafunctions, improve logging * [Feature] Add more text attributes * [Feature] Add new configwizard command to rspamadm * [Feature] Add new tooling for stats conversation * [Feature] Add old groups migration tool * [Feature] Add plugins state variable * [Feature] Add preliminary ecdsa keys support in DKIM * [Feature] Add preliminary support of idempotent symbols * [Feature] Add Redis server wizard * [Feature] Add routine to convert old style stats to a new one * [Feature] Add some sanity checks for actions and controller * [Feature] Add statistic convertation module to configwizard * [Feature] Add sugestions logic to mempool allocator * [Feature] Add support of config transform in Lua * [Feature] Add timeout to rspamc when doing corpus test * [Feature] Add tooling to convert bayes schemas * [Feature] Add torch conditional to configuration * [Feature] Add torch-decisiontree package * [Feature] Add torch-optim contrib package * [Feature] Add TTL autodetection * [Feature] Add urls reputation to the reputation framework * [Feature] Allow floating and negative values in expressions limits * [Feature] Allow multiple CTs in full extensions map * [Feature] Allow multiple fann rules * [Feature] Allow randomly select User-Agent from a list * [Feature] Allow rspamadm commands to export methods in Lua * [Feature] Allow rule specific min_bytes in fuzzy check * [Feature] Allow to adjust symbols scores from Lua * [Feature] Allow to attach stat signature to messages * [Feature] Allow to change SMTP from via milter headers * [Feature] Allow to configure monitored * [Feature] Allow to create directories in Lua API * [Feature] Allow to disable torch and skip train samples for ANN * [Feature] Allow to discard messages dynamically * [Feature] Allow to enable/disable languages from the detector * [Feature] Allow to generate DKIM keys from rspamadm API * [Feature] Allow to get CPU flags from Lua * [Feature] Allow to have high precision timestamps in logs * [Feature] Allow to insert headers into specific position * [Feature] Allow to limit redirector requests per task * [Feature] Allow to load and use dynamic ANNs with torch * [Feature] Allow to quarantine rejected messages using milter interface * [Feature] Allow to receive signing keys from mempool vars * [Feature] Allow to reserve elements in libucl * [Feature] Allow to reuse signal handlers chains * [Feature] Allow to set custom mempool variables from settings * [Feature] Allow to set headers from settings * [Feature] Allow to set Settings-Id for all connections * [Feature] Allow to skip real action and add a header instead * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Allow to spawn asynchronous processes from Lua * [Feature] Allow to specify number of threads for ANN learning * [Feature] Allow to use global lua maps in settings * [Feature] Allow to use postfilters in composites * [Feature] Allow to verify signatures from HTTP headers in maps * [Feature] Antivirus: ordered pattern matches * [Feature] Authentication-Results: support hiding usernames * [Feature] Automatically create tables in clickhouse * [Feature] Catch next-to-last bad extension * [Feature] Check cached maps more frequently * [Feature] Check groups sanity * [Feature] Deal with obscured URLs with @ symbols * [Feature] Enhance task:store_in_file method * [Feature] Export password encryption routines to Redis * [Feature] Filter nan and inf when adding scores * [Feature] Finalize 7zip files support * [Feature] Further improvements in language detection * [Feature] Further improvements in language detection algorithm * [Feature] Generic key name expansion for Redis keys * [Feature] Hash whitelist for fuzzy_check * [Feature] Implement bayes signatures storage * [Feature] Implement buckets for Redis backend * [Feature] Implement DKIM reputation adjustments * [Feature] Implement forked workers children monitoring * [Feature] Implement headers flags in mime parser * [Feature] Implement l1/l2 regularization against the current weights * [Feature] Implement manual ANN train mode * [Feature] Implement per-user ANN support * [Feature] Implement torch based ANN learning * [Feature] Implement upstreams logic for clickhouse exporter * [Feature] Import torch to Rspamd... * [Feature] Improve allocation policy when interacting with Lua * [Feature] Improve Lua/C interaction in history_redis * [Feature] Improve multiple fuzzy results combining * [Feature] Improve parsing of DKIM keys: parse algorithm * [Feature] Improve subprocesses termination handle * [Feature] Improve symbol type parsing in Lua API * [Feature] Metadata Exporter: e-Mail Alerts: support multiple recipients; alerting senders/recipients/users (#1600) * [Feature] Milter headers: support adding/removing arbitrary headers from config * [Feature] More metatokens * [Feature] Multimap: checking of symbol options * [Feature] Multimap: template URL filter * [Feature] New bayes expiry plugin * [Feature] Periodically save rspamd stats to disk * [Feature] Preliminary import of the elasticsearch module * [Feature] Ratelimit: allow full addresses in whitelisted_rcpts * [Feature] Ratelimit: support fetching limits from Redis * [Feature] RBL: received: filtering by position & flags * [Feature] Read global maps for lua * [Feature] Redis settings: support checking multiple keys * [Feature] Rework fann plugin to be a normal post-filter * [Feature] Rework logging configuration for rspamadm case * [Feature] Rework short hashes generation to avoid FP * [Feature] Save real ucl types when exporting to Lua * [Feature] Set TCP_NODELAY for milter sockets * [Feature] Setup DKIM signing from configwizard * [Feature] Skip certain symbols from ANN classify * [Feature] Store plugins state * [Feature] Support etag for HTTP maps * [Feature] Support Expires header when using HTTP maps * [Feature] Support sending given header multiple times in lua_http * [Feature] Support sha512 in DKIM signatures * [Feature] Try to detect HTML messages better * [Feature] Use array instead of queue to reduce memory fragmentation * [Feature] Use controller port by default when connecting to local IP * [Feature] Use rdtsc where possible * [Fix] Actively load skip hashes map in fuzzy storage * [Fix] Add another workaround to display history properly * [Fix] Add definition for old glib compatibility method * [Fix] Add missing rspamadm control options to help * [Fix] Add workaround for IPv6 in sendmail * [Fix] Add workaround for system with non-XSI compatible tzset * [Fix] Allow oversigning in DKIM signatures * [Fix] Allow to check negative scores in force_actions * [Fix] Allow to have negative actions limits * [Fix] Allow to set any layers number for fann rules * [Fix] Another fix for rdtcs * [Fix] Another fix to lua xmlrpc * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Another try to fix threading in torch * [Fix] Apply language detection when adding fuzzy hashes * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Authentication Results: Fix SPF smtp.mail_from * [Fix] Auth-Results: Multiple DKIM signatures * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Check for magic when checking for an archive * [Fix] Cleanup mess with groups * [Fix] Clickhouse: Insertion in the symbols table * [Fix] Crash in URL processing * [Fix] Deal with another case when processing exceptions * [Fix] Deal with deeply nested messages more aggressively * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with non-key arguments in lua_redis.exec_script * [Fix] Deal with unknown weight * [Fix] Deal with URLs with no slashes after protocol * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] Deal with zero scores symbols * [Fix] Default monitoring domain for surbl plugin * [Fix] Delay upstream re-resolving when one upstream is defined * [Fix] Detection of maillist optimized and fixed * [Fix] DKIM signing: allow for auth_only to be false * [Fix] DMARC: require report_settings for sending reports only * [Fix] Do not allow garbadge when checking url domain * [Fix] Do not cache SPF records with PTR elements * [Fix] Do not constantly re-resolve failed upstreams with a single element * [Fix] Do not crash if no words defined * [Fix] Do not crash on empty subtype * [Fix] Do not expose spamtrap messages to SMTP reply * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not ignore short words * [Fix] Do not include idempotent/nostat symbols to checksum * [Fix] Do not override groups when converting metrics * [Fix] Do not override unix socket group when group comes before owner * [Fix] Do not skip the last character * [Fix] Do not spawn too many workers by default * [Fix] Do not stop monitored on dns errors * [Fix] Do not stop parsing headers on bad IP header * [Fix] Do not strip last character in the last word * [Fix] Do not treat script content as text * [Fix] Do not try to connect to non-supported addresses * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Erase unknown HTML entities * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix a lot of FP in chartable in mixed languages * [Fix] Fix ANN checks * [Fix] Fix ANN loading logic * [Fix] Fix another tokenization issue * [Fix] Fix autolearn parameters reading * [Fix] Fix bad archive characters stripping * [Fix] Fix bad extension check * [Fix] Fix bayes schema conversion * [Fix] Fix blacklists and DMARC in whitelist * [Fix] Fix brain-damaged torch build system * [Fix] Fix build on FreeBSD * [Fix] Fix clickhouse exporter * [Fix] Fix clickhouse schema * [Fix] Fix comparision * [Fix] Fix composites processing * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix couple of warnings * [Fix] Fix crashes in the rspamd_control path * [Fix] Fix deletion from hash * [Fix] Fix DKIM forgeries via multiple headers * [Fix] FIx dynamic conf plugin * [Fix] Fix emails detection * [Fix] Fix empty headers simple canonicalization * [Fix] Fix empty threshold check in greylisting module * [Fix] Fix encrypted legacy reply in fuzzy storage * [Fix] Fix enormous scores for R_WHITE_ON_WHITE * [Fix] Fix exceptions list in surbl * [Fix] Fix *_EXCESS_BASE64 rules * [Fix] Fix expire rounding * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix format strings * [Fix] Fix get_content method * [Fix] Fix groups override when defining symbols * [Fix] Fix learned count in new schema * [Fix] Fix learn errors propagation * [Fix] Fix loading of per-user redis backend for statistics * [Fix] Fix logging buffer corruption in case of repeated messages * [Fix] Fix lua cached elements invalidation * [Fix] Fix merging of the implicit arrays * [Fix] Fix mime_types scoring * [Fix] Fix multiple headers in DKIM headers list * [Fix] Fix null callee case in clang plugin * [Fix] Fix obscured url in format user@@example.com * [Fix] Fix parsing of the per-user script * [Fix] Fix priorities in rspamd_update, disable rules execution * [Fix] Fix processing of closed tags * [Fix] Fix processing of idempotent rules when autolearn fails * [Fix] Fix processing of multipart parts with no headers * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix PTR processing in SPF * [Fix] Fix pushing country to clickhouse asn table * [Fix] Fix random forests module * [Fix] Fix real IP parsing for some strange Exim received * [Fix] Fix Redis timeout setup * [Fix] Fix reload crash when hyperscan is enabled * [Fix] Fix reusing of redis connection after exec * [Fix] Fix sanity checks on macro value * [Fix] Fix setting of path and cpath for Lua * [Fix] Fix setting of signals when spawning a thread * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix ticks processing * [Fix] Fix upstream addrs updating * [Fix] Fix urls/emails distinguishing found in queries * [Fix] Fix user settings check * [Fix] Fix variable increment * [Fix] Fix various issues in stat_convert * [Fix] F-PROT Antivirus infection string for all known occurences * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Further fixes around floating point expressions * [Fix] Further fixes to ANN module * [Fix] Further fixes to rescore tool * [Fix] Further fixes to support ES 6 * [Fix] Further tokenization fixes * [Fix] Greylisting set phase is not idempotent * [Fix] Handle proxy copy errors * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Kill spawned processes on termination * [Fix] Load skip map from all processes as shared cache is unavailable * [Fix] Lowercase HTTP headers to make them searchable from Lua * [Fix] Lowercase words * [Fix] Lua_http: freeing * [Fix] Lua: lpeg to be loaded with rspamd_lua_add_preload, to avoid "rspamd_config_read: rcl parse error: cannot init lua file […] module 'lpeg' not found" * [Fix] Map absence is not an error * [Fix] Metadata exporter: check IP sanity * [Fix] Milter headers: custom headers: removing headers * [Fix] Milter headers: skip_local / skip_authenticated settings * [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first * [Fix] mime_types: fix next-to-last extension length check * [Fix] More hacks to deal with old configs * [Fix] Move composites second pass to the dedicated stage * [Fix] Multimap: received: filtering of artificial header * [Fix] Multiple fixes in torch based ANN plugins * [Fix] Once more (#1879) fix bad extension check * [Fix] Optimize rspamd_fstring_t reallocations * [Fix] options.local_networks setting * [Fix] Parse HREF urls without explicit prefix * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Plug memory leak * [Fix] Plug memory leak in lua_tcp * [Fix] Plug memory leak when setting email addresses from Lua * [Fix] Propagate learn/stat errors more precisely * [Fix] Ratelimit: fix whitelisted_rcpts matching * [Fix] Ratelimit: lowercase email addresses * [Fix] RBL: received: deal with missing data (#1965) * [Fix] Rebalance and slightly rework MX check plugin * [Fix] Redis key expansion: EVAL: deal with strings * [Fix] Redis script loading in DMARC; URL tags; URL reputation * [Fix] Reject invalid bh for DKIM signatures earlier * [Fix] Relax pem signature detection * [Fix] Relax unicode properties requirements for chartable module * [Fix] Remove extra noise from dkim and arc signing * [Fix] Remove hop-by-hop headers in proxy * [Fix] Remove incorrect method `task:set_metric_subject` * [Fix] Replace space like characters in headers with plain space * [Fix] Restore old style ratelimits support * [Fix] Rework elasticsearch plugin * [Fix] Rewriting subjects via force actions module * [Fix] RPM postinstall * [Fix] Sanitize IP in history redis * [Fix] Select the correct signature when doing simple canon * [Fix] Set CLOEXEC flag on files opened * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Settings: header: deal with multiple settings (#1988) * [Fix] Skip checks if both extensions are not bad * [Fix] Skip nostat tokens when get number of tokens * [Fix] Some more fixes towards emails detection * [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't match * [Fix] Stop using of g_slice... * [Fix] Switch rspamadm logging to message level * [Fix] Symbol 'FANNR_SPAM' has its score defined.. * [Fix] Table parameter for rspamd_config:add_doc() * [Fix] Treat 'rewrite subject' as spam action * [Fix] Try harder in passing IPv6 addresses * [Fix] Try harder to find rfc822 notifications * [Fix] Try harder to find urls * [Fix] Use decoded values when parsing mime addresses * [Fix] Use full URL when making an HTTP request * [Fix] Use greylisting threshold in greylisting module * [Fix] Use n_words attribute from ngramms * [Fix] Use raw urls when sending requests to redirector * [Fix] Use the right boolean operator on error check * [Fix] Use weight from map for fuzzy scoring * [Fix] Various fixes to elastic plugin * [Fix] Various fixes to fann_redis instantiation * [Fix] Various improvements in language detection * [Fix] Virus infection string for F-PROT Antivirus * [Fix] Virus infetction string for F-PROT Antivirus * [Fix] WebUI: use relative path for savemap (#1943) * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [Fix] Write configuration changes as UCL config * [Project] Add detection logic for words * [Project] Add fast debug logging infrastructure * [Project] Add more flags to languages * [Project] Add n-gramms data files * [Project] Add ngramms frequencies detector * [Project] Add random words selection logic * [Project] Add unigramms to language detection as well * [Project] Convert all C modules to fast debug infrastructure * [Project] Detect some languages based on unicode script * [Project] Enable fast debug lookup for some modules * [Project] Enable language detector init in scanner workers * [Project] Further improvements to language detector * [Project] Implement logic of ngramms application * [Project] Improve weighting in lang_detection * [Project] Initialize language detector * [Project] Preliminary version of ngramms based language detector * [Project] Preliminary version of the new stat_convert * [Project] Remove old language detector * [Project] Rework language detection ngramms structure * [Project] Start language detection project * [Project] Start rework of language detection to improve quality * [Project] Use fast debug logging check * [Rework] Add frame for new reputation based IP score module * [Rework] Continue stat_convert rework task * [Rework] Implement new version of fuzzy replies * [Rework] Improve readability of xmlrpc API * [Rework] Kill metrics!11 * [Rework] Ratelimit module * [Rework] Rename fann_redis to neural plugin * [Rework] Reorganize mime_types module * [Rework] Rework rescore utility * [Rework] Rewrite model and learning logic for rescore * [Rework] Run post-loads when all initialization is completed * [Rework] Simplify lua path initialization * [Rework] Start major stat_convert rework * [Rework] Start mempool fragmentation reduce project * [Rework] Start moving of fann redis to torch * [Rework] Stop embedding rspamadm scripts into C * [Rework] Use floating point arithmetics in Rspamd expressions * [Rework] Use frequencies distribution in language detector * [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages * [WebUI] Compact graph selectors * [WebUI] Escape strings inside HTML in history * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Migrate widgets to D3 v4 * [WebUI] Restore passwordless login support (#2003) * [WebUI] Show symbol descriptions as tooltips in history * [WebUI] Stop using commas in pie chart tooltips * [WebUI] Update D3 and jQuery * [WebUI] Update D3Evolution 1.0.0 -> 1.1.0 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
8fb0ef19f4 |
Release 1.6.6
* [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [Feature] Allow to insert headers into specific position * [Feature] Allow to receive signing keys from mempool vars * [Feature] Authentication-Results: support hiding usernames * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] DKIM signing: allow for auth_only to be false * [Fix] Do not crash on empty subtype * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not skip the last character * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix bad archive characters stripping * [Fix] Fix comparision * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix empty headers simple canonicalization * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix parsing of the per-user script * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix Redis timeout setup * [Fix] Fix sanity checks on macro value * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix urls/emails distinguishing found in queries * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Metadata exporter: check IP sanity * [Fix] Multimap: received: filtering of artificial header * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Remove hop-by-hop headers in proxy * [Fix] Sanitize IP in history redis * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Try harder in passing IPv6 addresses * [Fix] WebUI: use relative path for savemap (#1943) * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Restore passwordless login support (#2003) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
 Viktor Szépe
|
b3e626e260
|
Typo: comparision | 6 years ago |
|
Vsevolod Stakhov
|
111dfde1ad | [Minor] Sync ChangeLog | 6 years ago |
|
Vsevolod Stakhov
|
fa9b2fc2d8 |
Release 1.6.3
* [CritFix] Fix semicolons parsing in the content type * [Feature] Add EBL to the default config * [Feature] Allow to configure monitored * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Multimap: checking of symbol options * [Feature] Redis settings: support checking multiple keys * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Fix get_content method * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Lua_http: freeing * [Fix] Milter headers: custom headers: removing headers * [Fix] Parse HREF urls without explicit prefix * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [WebUI] Escape strings inside HTML in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
|
Vsevolod Stakhov
|
e5cefe177e
|
Release 1.6.2
* [Conf] Remove Rambler email bl for now * [Conf] Switch RAMBLER_URIBL to a locally managed source * [CritFix] Switch from ragel to C for Content-Type parsing * [Feature] Add `-e` option for lua_repl * [Feature] Add per-domain emails normalisation rules * [Feature] Add sessions cache to debug dangling sessions * [Feature] Add short_text_direct_hash for fuzzy check module * [Feature] Add text_part:get_stats function * [Feature] Allow to add custom processing script for surbl * [Feature] Allow to check reply-to email * [Feature] Allow to customize spam header, remove existing spam headers * [Feature] Allow to disable specific workers in the config * [Feature] Allow to discard messages instead of rejection * [Feature] Allow to specify custom delimiter in emails plugin * [Feature] Allow to specify custom User-Agent for rspamc * [Feature] Allow to store symbols data in Clickhouse * [Feature] Allow to use HTTPS when connecting to Clickhouse * [Feature] Enable sessions cache tracking for milter connections * [Feature] Implement per-line mode in lua_repl (like `perl -p`) * [Feature] Implement rdns-curve plugin based on rspamd cryptobox * [Feature] Improve maps cached data lifetime * [Feature] Improve maps checking frequency * [Feature] Improve monitored timeouts logic * [Feature] milter_headers: add `extended_headers_rcpt` option * [Feature] Milter headers: Add X-Spam-Flag to rmilter-compatibility headers * [Feature] Milter headers: remove-header routine * [Feature] Multimap: received filters for extracting TLDs from hostnames * [Feature] Normalize email aliases in emails module * [Feature] Re-add rambler email bl (as hashed list) * [Feature] Reload file maps more frequently * [Feature] Rework newlines strip parser one more time * [Feature] Skip updates for messages scanned via controller * [Feature] Split long DKIM public keys * [Feature] Store more data when stripping newlines * [Feature] Support SPF macros transformations * [Feature] Support suppressing DMARC reports for some domains * [Fix] Add missing `break` statement * [Fix] Allow modifiers in SPF macros * [Fix] DKIM sign tools: edge-cases around use_esld * [Fix] Do not cache SPF records with macros * [Fix] Do not overwrite score when setting pre-action * [Fix] Fix comparision logic * [Fix] Fix DKIM base64 folding for milter flagged messages * [Fix] Fix emails module configuration * [Fix] Fix folding for arc headers when milter interface is used * [Fix] Fix gmail dots removal * [Fix] Fix rspamc detection in greylist module * [Fix] Fix some more issues with HTTP maps * [Fix] Milter sessions can live forever * [Fix] Normalize fuzzy probability better * [Fix] Plug memory leak * [Fix] RBL: Fixed hashed email address lookups * [Fix] Try to deal with brain-damaged milter behaviour * [Fix] Use `\n` to fold headers for milter * [Rework] Allow to use custom callback for monitored checks * [Rework] Further steps towards one process monitoring * [Rework] Send health checks from a single worker * [WebUI] Round-up throughput summary values Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
6 years ago |
 Alexander Moisseev
|
bb5a9f8e45 |
[Minor] Massive spelling correction
by a bot https://github.com/ka7/misspell_fixer |
7 years ago |
|
Vsevolod Stakhov
|
8895fc6fab
|
Release 1.6.1
* [Fix] Allow to init resolver without rspamd_config * [Fix] Do not crash when resolver failed to initialize * [Fix] Fix abstract context layout * [Fix] Fix CGP helper reply parsing * [Fix] Fix crashes when socket write errors occur * [Fix] Fix parsing IPv6 nameservers in resolv.conf * [Fix] Milter: Don't defer on "greylist" action Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
9c0fd5e439
|
Release 1.6.0
* [Conf] Add rspamd_proxy to the default configuration set
* [Conf] Add sample arc module config
* [Conf] Do away with systemd specifics completely
* [Conf] Increase min_bytes to avoid FP
* [Conf] Remove ratelimits from default configuration
* [CritFix] Fix accepting on IPv6 sockets
* [CritFix] Fix corruption when multiple fuzzy are defined
* [CritFix] Fix learn condition in fuzzy check
* [CritFix] Fix memory leak in fuzzy check
* [CritFix] Fix memory leak in maps scheduling
* [CritFix] Paese the last character in DKIM signature correctly
* [CritFix] Zero fill sockaddr_un
* [Feature] Add ability to add doc strings by example
* [Feature] Add API to verify DKIM (and ARC) signatures
* [Feature] Add compression/decompression to proxy
* [Feature] Add count to url structure
* [Feature] Add initial support of the new protocol reply
* [Feature] Add Lua plugin spamtrap
* [Feature] Add `monitored_address` for rbls
* [Feature] Add new schema for bayes tokens
* [Feature] Add preliminary ARC support to dkim code
* [Feature] Add preliminary support of ARC signing
* [Feature] Add rules to detect bad 8bit characters in From and To
* [Feature] Add scanning support for milter protocol
* [Feature] Add support for bidirectional symbols in rspamd_stats
* [Feature] Add support for static maps
* [Feature] Add support of maps with multiple regexps matches
* [Feature] Add `text_multiplier` param
* [Feature] Add the preliminary ARC plugin
* [Feature] Add top redirector targets rank
* [Feature] Allow async events to be registered from LUA rules
* [Feature] Allow storing bayes tokens in Redis
* [Feature] Allow to exclude specific domains from mx check
* [Feature] Allow to have a stack of watcher finalisers
* [Feature] Allow to pass hostname to `-i` flag in Rspamc
* [Feature] Allow to set custom user agent in url redirector
* [Feature] Allow to use custom callback when parsing resolv.conf
* [Feature] Allow to use domain from authenticated user
* [Feature] Bayes expiry plugin
* [Feature] Check dkim sign keys for modifications
* [Feature] DKIM signing: sign_networks/local address specific use_domain settings
* [Feature] DMARC: Support excluding domains from sampling
* [Feature] Expire processing items for URL redirector aggressively
* [Feature] Fix surbl monitored for IP lists, add `monitored_domain` option
* [Feature] Implement caching for dkim body hashes
* [Feature] Implement milter protocol scan reply
* [Feature] Improve omograph phishing detection
* [Feature] Initial support of self-scan in Rspamd proxy
* [Feature] Keep track of headers in milter interface
* [Feature] Milter headers: better controls for local/authenticated
* [Feature] Multimap: email:domain:tld filter
* [Feature] Preliminary DMARC reporting implementation
* [Feature] Reuse stemmers in the cache
* [Feature] Rework confighelp to load Lua plugins
* [Feature] Rework hfilter to use hyperscan if possible
* [Feature] Rework lua RSA API
* [Feature] Rmilter_headers: approximate rmilter's extended_spam_headers
* [Feature] Start integration of milter support in proxy
* [Feature] Store average words length and short words count
* [Feature] Store hash of headers order and names
* [Feature] Support MTA name header
* [Feature] Support multiple types of dkim signing in Lua
* [Feature] Support numeric arguments for Redis requests
* [Feature] Use headers hash in bayes metatokens
* [Feature] Use normal resolv.conf rules of rotation in Rspamd
* [Feature] Use version 2 proto for checking messages
* [Fix] Allow to follow symlinks when safe
* [Fix] Append MX name for authentication results as required
* [Fix] Change default text multiplier from 0.5 to 2.0
* [Fix] Check min_bytes for images as well
* [Fix] Deal with 7bit charsets properly
* [Fix] Deal with 8bit characters in email addresses
* [Fix] Deal with unpaired <a> tags
* [Fix] Detect confighelp in plugins initialisation
* [Fix] Disable certain checks for utf spoof detection
* [Fix] DKIM Signing: avoid nil index when From header is missing
* [Fix] Do not add exact hashes from different parts
* [Fix] Do not check DMARC if SPF or DKIM were not checked
* [Fix] Do not check URLs that are resolved to be redirected
* [Fix] Do not set bayes probability if we don't use it
* [Fix] Do not stop on illegal unicode points - replace them
* [Fix] Fix another race condition in arc checks
* [Fix] Fix arc count logic
* [Fix] Fix ARC signing
* [Fix] Fix brain-damaged spamc protocol for now
* [Fix] Fix calling for peak functions
* [Fix] Fix couple of issues in FORWARDED rule
* [Fix] Fix CTE propagation from parent containers to children parts
* [Fix] Fix errors processing in the controller
* [Fix] Fix format string in milter
* [Fix] Fix issues in SPF macros parsing
* [Fix] Fix logging format string
* [Fix] Fix logic of cached passwords check
* [Fix] Fix lowercasing of stemmed words
* [Fix] Fix LRU elements removal
* [Fix] Fix memory leak when accepting from unix sockets
* [Fix] Fix milter connections persistence
* [Fix] Fix objects merging in UCL
* [Fix] Fix order of operations to avoid race condition
* [Fix] Fix parsing of long regexp types
* [Fix] Fix passing data to log helper when many symbols defined
* [Fix] Fix pools management for milter session
* [Fix] Fix processing of the watchers
* [Fix] Fix queue id macro in milter
* [Fix] Fix R_BAD_CTE_7BIT rule
* [Fix] Fix Redis timeout set
* [Fix] Fix REPLYTO_UNPARSEABLE rule
* [Fix] Fix setting of email address
* [Fix] Fix some more issues about duplicated fuzzy requests
* [Fix] Fix spamc support in rspamd proxy
* [Fix] Fix syntax error in spamtrap plugin
* [Fix] Fix url counts for href urls
* [Fix] Fix url handling in the protocol
* [Fix] Multimap: Received IP filters with Redis
* [Fix] Oops, fix
|
7 years ago |
|
Vsevolod Stakhov
|
a92e55249f | [Minor] Update ChangeLog from 1.5 branch | 7 years ago |
 Sebastien Badia
|
dd64721ae6
|
[Minor] debian: Update BD and refresh packaging | 7 years ago |
|
Vsevolod Stakhov
|
27ea426985
|
Release 1.5.6
* [Feature] Add unigramms support in bayes * [Feature] Allow configurable sign headers for DKIM * [Feature] Allow to add unigramm metatokens from Lua * [Feature] DKIM Signing: envelope match exception for local IPs * [Feature] UCL: register parser variables from Lua * [Fix] Always try to adjust filename * [Fix] Do extra copy to ensure that original content is never touched * [Fix] Fix SPOOF_REPLYTO rule * [Fix] Ignore Rmilter added Received * [Fix] More fixes for hashed email dnsbls * [Fix] Plug memory leak in chartable module * [WebUI] Display multiple alerts at once Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
939ab1342e |
Release 1.5.5
* [CritFix] Fix classifier learning with Redis backend * [CritFix] Fix issue when parsing encoded rfc822/messages * [Feature] Add escaped version of lua_ucl import * [Feature] Add task:headers_foreach function * [Feature] Allow to process filenames from content type * [Feature] Allow to query hashed emails * [Feature] Ignore bayes with mostly metatokens or with too few text * [Feature] Probabilistically skip metatokens * [Feature] Retrieve all virus names from SAVAPI * [Feature] Rework classifiers lua metatokens * [Feature] Store headers order * [Feature] Store text tokens inside bayes tokens * [Feature] Use cached shingles keys * [Fix] Add missing score normalisation for HFILTER_URL_ONLY * [Fix] Avoid lookup in absent hash * [Fix] Check return values from Lua functions called from C * [Fix] Do not count sending and loading time in rspamc * [Fix] Escape json strings for controller rejplies from Lua * [Fix] Fix archive scans for savapi * [Fix] Fix domain_only emails RBL * [Fix] Fix ip_score map configuration * [Fix] Fix JSON output for history_redis * [Fix] Fix one character length substrings search * [Fix] Fix parsing of non-RFC compatible Exim received * [Fix] Fix parsing of options for workers with the same type * [Fix] Fix processing of small tokens vectors * [Fix] Fix rfc2047 tokenization * [Fix] Fix typo * [Fix] More fixes for inplace decoding * [Fix] Try to avoid modifications of the original data * [Fix] URL redirector: Fix call to is_redirector * [Rework] Set token data as uint64_t instead of chars array * [WebUI] Check if neighbours' history backend versions match * [WebUI] Disable phrase connectors replacement in history filtering * [WebUI] Disable phrase connectors replacement in symbols filtering * [WebUI] Do not hide messages with bad subject, just replace it with '???' * [WebUI] Fix error message * [WebUI] Fix history v2 display * [WebUI] Fix legacy history * [WebUI] history: break To address lists on commas * [WebUI] Increase default timeout to 20 seconds * [WebUI] Save some history table space Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
0839f87a25
|
Release 1.5.4
* [Conf] Add history_redis default configuration * [Feature] Add spoofed rules * [Feature] Add URL_IN_SUBJECT rule * [Feature] Allow to get task's subject * [Feature] Allow to specify maximum number of shots for symbols * [Feature] Distinguish URLs found in Subject * [Feature] Memoize LPEG grammars * [Feature] Parse else parts in SA rules * [Feature] Process subject for mixed characters * [Feature] Resolve url chains in url_redirector module * [Feature] Stat greylisted messages as greylisted not soft-rejected * [Feature] Support checking for redirector in Lua SURBL * [Feature] Support tag_exists SA function * [Feature] Work with broken rfc2047 tokens * [Fix] Check all watcher's dependencies * [Fix] Do not compile hyperscan with no SSSE3 support * [Fix] Do not crash if cannot decode qp encoded part * [Fix] Fix dependencies of DKIM when multiple signatures are found * [Fix] Fix lists in whitelist plugin * [Fix] Fix one-shot symbols weight calculations * [Fix] Fix options and shots match * [Fix] Fix order of symbol options * [Fix] Fix parsing of dot at the end of the address * [Fix] Fix parsing of lua table arguments * [Fix] Fix processing of subject words * [Fix] Fix string split memoization * [Fix] Fix templates grammar usage * [Fix] Fix various issues related to Lua stack manipulation * [Fix] Force actions: Use postfilter if we have honor_action / require_action * [Fix] Further fixes to avoid PHISHING FP * [Fix] Preserve order of options in symbols * [Fix] Rspamadm grep: deal with unusually-formatted logs * [Fix] Use hostname suffix when dealing with history * [Rework] Remove outdated SA rules * [WebUI] Add flexible columns * [WebUI] Add footable * [WebUI] Add sender, recipients and subject columns * [WebUI] Allow message-id break * [WebUI] Fix history clustering * [WebUI] Fix history display * [WebUI] Fix sorting * [WebUI] Humanize sizes * [WebUI] Initial move towards footable * [WebUI] Remove datatables * [WebUI] Replace `.values` method with `.map` * [WebUI] Rework v2 symbols display * [WebUI] Try to normalize frequencies * [WebUI] Unbreak WebUI * [WebUI] Use Footable to draw Throughput summary table Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
ca1a968cf5
|
Release 1.5.3
* [Conf] Add composite for hacked wordpress phishing * [CritFix] Fix base64 decoding when there are unparseable characters * [Feature] Additional symbol metadata in metadata exporter * [Feature] Add method to get protocol reply from Lua * [Feature] Add symbols when tagged rcpt/sender are normalised * [Feature] Add task:get_symbols_all() function * [Feature] Allow multiple formats of DKIM signing key * [Feature] Allow to cache and use flexible protocol reply * [Feature] Allow to set one_shot flag from register_symbol * [Feature] Allow to skip certain types of hashes when learning fuzzy * [Feature] Cache and insert scan time into the protocol * [Feature] Detect newlines in rspamc --mime * [Feature] DKIM signing: support use of maps * [Feature] Greylist: Support excluding low-scoring messages from greylisting * [Feature] Implement lua history in controller * [Feature] Implement redis history querying * [Feature] Preliminary implementation of redis history plugin * [Feature] Support using request headers in settings * [Fix] Change default template to deal with non-ASCII characters * [Fix] Deal with lists of maps in whitelist module * [Fix] DKIM signing: use domain-specific signing key * [Fix] Do not reallocate completed zstd buffer * [Fix] Do not use local_addrs in proxy * [Fix] Fix crash when resolver is undefined * [Fix] Fix double free when closing lua_tcp connections * [Fix] Fix for lua 5.3 * [Fix] Fix freeing of arrays iterators * [Fix] Fix issue with task:get_symbol and symbols with no metric * [Fix] Fix log line duplication in `rspamadm grep` * [Fix] Fix memory corruption on termination * [Fix] Fix out-of-bound access in base64 decode * [Fix] Fix ratelimit + greylisting * [Fix] Fix subject rewriting * [Fix] Fix task:set_recipients function * [Fix] Fix URI_COUNT_ODD rule * [Fix] Follow the traditional symbols conventions in RCPT_COUNT rule * [Fix] Greylist: Suppress greylist action for whitelisted hosts too * [Fix] Metadata exporter: use rule-specific settings for emails * [Fix] Properly set missing fields in exporter * [Fix] Proxy: max_retries option * [Fix] RCPT_COUNT fixes * [Fix] Rework HAS_X_PRIO rule to match symbols conventions * [Fix] Update issues in ac-trie * [Fix] Use optimised base64 decoding in DKIM * [WebUI] Add preliminary v2 history parser * [WebUI] Allow different history parsers * [WebUI] Display symbols * [WebUI] Rework history v2 function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
898c9c7b5f
|
Release 1.5.2
* [Conf] Add default config for spamassasssin plugin * [Conf] Add default configuration for antivirus module * [Conf] Add dkim signing docs * [Conf] Add mx_check default config * [Conf] Add replies config * [Conf] Add trie default config * [Feature] Add heuristic to find text parts in files * [Feature] Add rule to detect broken content type * [Feature] Allow to extract CTE in Lua API * [Feature] Allow to set from address for a lua_task * [Feature] Allow to set recipients of a task from Lua * [Feature] Enchance text_part:get_content method * [Feature] Remove + aliases from emails * [Feature] Support rmilter block and dkim signature in CGP helper * [Feature] Support running event loop from Lua * [Fix] Antivirus: use scanner-specific redis prefix * [Fix] Couple of fixes for DKIM signing module * [Fix] Distinguish missing and broken mandatory headers * [Fix] Do more heuristical detection for missing CTE * [Fix] Do not resort cache on each check * [Fix] Fix CGP escaping * [Fix] Fix MISSING_MIME_VERSION rule for plain messages * [Fix] Fix parsing of cte in expressions * [Fix] Fix partial matches in rspamadm grep * [Fix] Fix setting class on style field * [WebUI] Auto-switch Throughput units to `msg/min` for very low rate * [WebUI] Update D3Evolution to 0.0.2 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
f6c7e98b74
|
Release 1.5.1
* [CritFix] Fix processing of stop_patterns with `\0` character * [CritFix] Fix setting of raw key for signing * [Fix] Fix lua exports from plugins during reload * [Fix] Fix prefilters action scores * [Fix] Fix symbols processing order * [Minor] Help cmake find gthread * [Minor] Some cmake fixes Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
9d2e7e0e2c |
Release 1.5.0
* [Conf] Add configurations for asn, clickhouse and dcc * [Conf] Add default config for url redirector plugin * [Conf] Add the default config for greylist module * [Conf] Allow to edit all local maps from WebUI by default * [CritFix] Deal with absent headers in DKIM * [CritFix] Do not trust remote shingles count * [CritFix] Fix bad memory leak in TLS certificates validation * [CritFix] Fix critical memory issues with radix maps * [CritFix] Fix descriptors leak on reload * [CritFix] Fix headers selection in DKIM verification * [CritFix] Fix parsing of boundaries that end with `--` * [CritFix] Repair PTR_ARRAY_FOREACH macro * [Feature] Add CORS support to the controller * [Feature] Add FROM_NAME_EXCESS_SPACE rule * [Feature] Add REPLYTO_EMAIL_HAS_TITLE rule * [Feature] Add `caseless_hash` method to `lua_util` * [Feature] Add `rip` keyword to ratelimit module * [Feature] Add a simple benchmark for content type parsing * [Feature] Add boundaries parsing in content type * [Feature] Add charset detection for text parts * [Feature] Add content disposition parser * [Feature] Add fallback if too many updates are failing * [Feature] Add function to convert struct tm to time using timezone * [Feature] Add function to normalize HTTP paths * [Feature] Add fuzzy collection plugin * [Feature] Add fuzzy logic for images * [Feature] Add gmime parser to mime_tool * [Feature] Add heuristic to detect broken messages * [Feature] Add heuristic to find displayed URLs * [Feature] Add heuristic to process broken email addresses * [Feature] Add images normalization * [Feature] Add mechanism for disabling composites (Fixes #1270) * [Feature] Add method to create regexp from a glob pattern * [Feature] Add mime encoding manipulation routines * [Feature] Add mime tool to explore messages * [Feature] Add more meta tokens from received headers * [Feature] Add neighbours option to support Rspamd cluster in WebUI * [Feature] Add new function to parse mime addresses * [Feature] Add new methods for lua_tcp * [Feature] Add own headers decoding routine * [Feature] Add own routine to generate a message id * [Feature] Add parser for SMTP date * [Feature] Add per-task lua cache to reuse 'heavy' objects * [Feature] Add plugins list path in WebUI * [Feature] Add preliminary multipart support * [Feature] Add preliminary version of DKIM signing module * [Feature] Add profiling support in client output * [Feature] Add rfc2047 grammar * [Feature] Add rfc2047 variant for QP decoding * [Feature] Add rmilter_headers module (Fixes #1227) * [Feature] Add sse42 version of base64 decoding * [Feature] Add ssse3 and avx2 base64 decoders * [Feature] Add support of libgd * [Feature] Add the preliminary version of redirects resolver in Lua * [Feature] Add ucl_object_iterate_full function * [Feature] Add url encoding function * [Feature] Allow SOA requests in lua dns * [Feature] Allow custom parse types in lua ucl * [Feature] Allow plugins to register webui handlers * [Feature] Allow to add options explicitly to symbols * [Feature] Allow to call a callback when symbol frequency is on peak * [Feature] Allow to call redirector script from SURBL * [Feature] Allow to create variable length dkim keys * [Feature] Allow to have module specific options for Redis in plugins * [Feature] Allow to pass sign key directly from Lua * [Feature] Allow to register configuration docs from Lua API * [Feature] Allow to return options as a table * [Feature] Allow to set peak callbacks from Lua * [Feature] Allow to specify custom method for a message * [Feature] Allow to store dkim keys in Redis * [Feature] Allow to store messages in files * [Feature] Apply DCT using AAN for fuzzy signature * [Feature] Avira SAVAPI support * [Feature] Cache and simplify DCT and jpeg decode * [Feature] Cache libicu convertors * [Feature] Detect URLs with suspicious omographs * [Feature] Do not increase score for duplicate options * [Feature] Do not trust CTE, check base64 and qp strictly * [Feature] Dynamic reputation in URL reputation plugin * [Feature] Extend redis lock when learning spawned * [Feature] Filter non-utf chars from all decoded headers * [Feature] Fix phishing detection for IDNA urls * [Feature] Ignore bad symbols on base64 decoding * [Feature] Ignore too wide elements in SPF * [Feature] Implement fuzzy collection mode * [Feature] Implement helo maps in multimap * [Feature] Implement human readable buckets configuration * [Feature] Implement min-hash shingles for DCT data from images * [Feature] Implement new algorithm for fuzzy hashes of images * [Feature] Implement new unicode normalizer * [Feature] Implement quoted printable decoding * [Feature] Implement received headers flags * [Feature] Implement rspamdgrep tool * [Feature] Implement sane checksum for config file * [Feature] Implement url tags concept * [Feature] Improve detection of omographs using libicu * [Feature] Improve url redirector module * [Feature] Multimap: Received header processing * [Feature] Multiple improvements in the maps * [Feature] New URL filters in multimap * [Feature] Plugin to force actions on selected symbols * [Feature] RBL module: support hashing for emails and helo RBL * [Feature] Reuse URL tags in SURBL module * [Feature] Rework RRD ds count, add conversion path * [Feature] Rework surbl module to avoid extra redirector calls * [Feature] Send config id to the WebUI * [Feature] Simplify HTTPCrypt client support * [Feature] Skip processing for large images * [Feature] Start collection only mode implementation for fuzzy storage * [Feature] Start import of the optimized base64 decode * [Feature] Store all received headers in lua * [Feature] Store relational order of all headers in a message * [Feature] Support DKIM signing in Lua plugins * [Feature] Support HTTPCrypt client in lua_http * [Feature] Support setting SMTP message in multimap * [Feature] Support setting metric subject from Lua * [Feature] Support setting subject in force actions module * [Feature] Treat v6 mapped addresses as v4 addresses * [Feature] URL reputation plugin * [Feature] Use Redis instead of memcached in URLs redirector * [Feature] Use Rspamd rfc2047 decoder instead of gmime one * [Feature] Use a different normalization for fuzzy images * [Feature] Use normalized images in fuzzy hashes * [Feature] Use own code for parsing of date * [Feature] Use shingles for images fuzzying * [Feature] Use t1ha for hashes, allow inlining * [Feature] Use t1ha instead of metrohash and xxhash32 * [Feature] Various new features in metadata exporter module * [Feature] rmilter_headers: authentication-results (#78) * [Fix] Add additional check to mark redis connection inactive * [Fix] Add packed attribute for protocol structure * [Fix] Adopt OMOGRAPH_URL rule * [Fix] Allow static maps * [Fix] Allow to disable classifiers checks using settings and conditions * [Fix] Another try to fix 0 length maps * [Fix] Another try to fix corruption during maps reload * [Fix] Another try to fix descriptors leak * [Fix] Another try to fix reload and logger * [Fix] Antivirus module: register virtual symbols for patterns * [Fix] Avoid extensive reallocs * [Fix] Avoid mempool leak in SA plugin on reload * [Fix] Avoid race condition on saving cache and reload * [Fix] Avoid reusing g_error (Fixes #1262) * [Fix] Break pool connection on fatal redis errors * [Fix] Check for NaN properly * [Fix] Couple of fixes for date parsing * [Fix] Date header timezone adjustments (#1279) * [Fix] Deal with EOF properly * [Fix] Decode filename in content disposition * [Fix] Disable fuzzy images by default * [Fix] Disable zero-copy mode for text parts to avoid crashes * [Fix] Do not destroy session when not all finish scripts are done * [Fix] Do not greyscale images * [Fix] Do not leave parent-less workers processes on fatal errors * [Fix] Do not lowercase Content-Disposition to perform decoding * [Fix] Do not penalize characters just after numeric prefix * [Fix] Do not refork workers that are intended to die * [Fix] Do not set pre-result and update records for no `Queue-ID` messages * [Fix] Do not skip post-filters when pre-filters have set some results * [Fix] Do not stop symbols planning if async events are pending * [Fix] Do not try to set keys for unencrypted requests in proxy * [Fix] Encode URLs according to rfc3986 * [Fix] Encode URLs before sending them to the protocol * [Fix] Filter bad characters from message id * [Fix] Fix CTE detection heuristic * [Fix] Fix Content-Type in HTTP requests * [Fix] Fix IDN eslds phishing checks * [Fix] Fix adding maps from config in Lua * [Fix] Fix another reload memory issue * [Fix] Fix argument returned on redis backend errors * [Fix] Fix assertion in graph handling * [Fix] Fix body trie matching * [Fix] Fix build * [Fix] Fix byte array expansion during toutf8 conversion * [Fix] Fix charset normalisation * [Fix] Fix checking of DKIM bodies that needs just `\n` to be added * [Fix] Fix couple of cornercases with email addresses * [Fix] Fix couple of issues * [Fix] Fix dependencies tracking for callback symbols * [Fix] Fix detection of jpeg size * [Fix] Fix errors handling in fuzzy backend initialization * [Fix] Fix fuzzy hashes count * [Fix] Fix globbing and convert lists to arrays in fuzzy_check * [Fix] Fix heuristical CTE detection for QP encoding * [Fix] Fix ignoring of bad text parts * [Fix] Fix indexes in array access, interleave loop * [Fix] Fix int64 -> double conversion * [Fix] Fix invalid memory access on reload * [Fix] Fix issues with empty updates * [Fix] Fix issues with quoted-printable encoding * [Fix] Fix keys names * [Fix] Fix lots of issues in mime parser code * [Fix] Fix lua maps load * [Fix] Fix macro name * [Fix] Fix mas group score calculations * [Fix] Fix matching of the same patterns from different tries * [Fix] Fix memory corruprtion and leak * [Fix] Fix memory leak in HTTP maps * [Fix] Fix memory leak in expression destroying * [Fix] Fix memory leak in parsing of mime names * [Fix] Fix memory leak in safe ucl iterators * [Fix] Fix memory leak on reload in plugins * [Fix] Fix modules reconfigure on reload * [Fix] Fix monitored setup fro URLBLs with IP addresses * [Fix] Fix name of var * [Fix] Fix new rrd updates * [Fix] Fix out of bounds access * [Fix] Fix parsing messages with no body * [Fix] Fix parsing of '=' character in headers * [Fix] Fix parsing of messages with no content type * [Fix] Fix plugins callbacks in webui * [Fix] Fix possible memory corruption in redis pool * [Fix] Fix probability calculations for fuzzy redis backend * [Fix] Fix processing errors in lua_tcp * [Fix] Fix processing of emails with name only * [Fix] Fix processing of non-multipart messages * [Fix] Fix processing of parts with no valid content type * [Fix] Fix race condition in SIGUSR2 handler * [Fix] Fix redis options parsing when no redis servers are defined * [Fix] Fix reload and hyperscan ready event * [Fix] Fix reload memory issue * [Fix] Fix rra_ptr conversion * [Fix] Fix rrd file conversion * [Fix] Fix setting of content-type attributes * [Fix] Fix signing headers creation in DKIM * [Fix] Fix stddev calculations * [Fix] Fix surbl plugin to work with composite maps * [Fix] Fix timezones parsing * [Fix] Fix tokens usage * [Fix] Fix urls and emails hashes * [Fix] Fix usage of unsafe ucl iterators * [Fix] Fix work with broken utf8 tokens * [Fix] Fix writing of user to roll history * [Fix] Forgotten worker * [Fix] Further memory leaks fixes * [Fix] Ignore lua metatokens in bayes for now * [Fix] Improve OMOGRAPH_URL rule * [Fix] Lua IP from string should be invalid if parsing failed * [Fix] Miltiple fixes to new lua_tcp, add debugging * [Fix] More fixes for iterators cleanup * [Fix] More fixes to logger initialization * [Fix] More heuristic fixes for phishing detection * [Fix] More leaks eliminated * [Fix] More leaks... * [Fix] More random fixes for reload... * [Fix] Multimap: Fixes for email filters * [Fix] Multiple fixes for fann module * [Fix] Multiple memory corruption fixes * [Fix] Normalize path in HTTP router * [Fix] Plug memory leak * [Fix] Plug memory leak in adding radix trees * [Fix] Plug memory leak in configuration parser * [Fix] Plug memory leak in expressions parsing during reload * [Fix] Plug memory leak in learning fuzzy storage * [Fix] Plug memory leak in lua_tcp * [Fix] Plug reload leaks * [Fix] Plug termination memory leaks * [Fix] Really increase lock lifetime * [Fix] Replies module: fix symbol weight * [Fix] Restore content type params related functions * [Fix] Set task's subject from mime subject * [Fix] Sigh, one more reload leak * [Fix] Simplify images shingles * [Fix] Some more memory issues are fixed * [Fix] Stop hardcoding of lua in C * [Fix] Stop processing of bad parts as text parts * [Fix] Strictly filter bad characters when emittin json * [Fix] Strings returned from lua are ephemeral * [Fix] Support unix sockets for lua redis * [Fix] Try to fix issues with reloading config * [Fix] Try to fix race condition in redis_pool * [Fix] Use checksum to avoid intersection between different ANNs * [Fix] Use rspamd hashes in embedded ucl * [Fix] Use sane default rewrite subject (*** SPAM *** %s) * [Fix] Various collection mode fixes * [Fix] Various fixes to mime parser * [Fix] Various reload leak fixing * [Fix] Whitelist certain extensions from archive checks * [Rework] Add preliminary implementation of the mime parser * [Rework] Adopt code for the new options * [Rework] Change logger setup interface * [Rework] Composite configuration (#1270) * [Rework] Finally remove gmime dependency from Rspamd * [Rework] Further fixes to symbols frequencies * [Rework] Implement content type parser for mime * [Rework] Kill all InternetAddressList usages * [Rework] Multiple fixes for symbols cache statistics * [Rework] Refactor struct names * [Rework] Rework images fuzzy hashes algorithm * [Rework] Rework lua_tcp to allow TCP dialog * [Rework] Start massive rework to get rid of gmime * [Rework] Start new approach for multiparts parsing * [Rework] Start rework of mime addresses * [Rework] Start rework of symbols cache updates * [Rework] Start switching to libicu * [Rework] Use a special structure for stats tokens * [Rework] Use hash tables for symbols options * [Rework] Use libicu instead of iconv for conversions * [Rework] Use new scheme to parse mime parts * [WebUI] Add Access-Control-Allow-Origin for cluster management * [WebUI] Add Throughput graph autorefreshing (#820) * [WebUI] Add Visibility.js library * [WebUI] Add basic cluster support to Throughput tab * [WebUI] Add graph legend entries for new DSes * [WebUI] Add graph tab * [WebUI] Add neighbours RRD data consolidation * [WebUI] Add preliminary save symbols clustering * [WebUI] Add server selector to navbar * [WebUI] Add soft reject to auth stats * [WebUI] Add summary to the Throughput tab * [WebUI] Allow to save maps on the cluster * [WebUI] Avoid extra graph redraw and alerts glitching * [WebUI] Be more generous with AJAX timeout * [WebUI] Disable error ring loading in `read only` mode * [WebUI] Enclose table header cells with `tr`s * [WebUI] Finish interface rework * [WebUI] Fix RRD summary pie chart position * [WebUI] Fix `All SERVERS` graph fot just one available server * [WebUI] Fix case when no cluster is defined * [WebUI] Fix compatibility with non-ES6 compliant browsers * [WebUI] Fix config ID * [WebUI] Fix configuration page partially * [WebUI] Fix disabled state * [WebUI] Fix graph dataset selector initialization * [WebUI] Fix graph selectors state resetting * [WebUI] Fix mouse events on throughput summary table area * [WebUI] Fix multiple JS issues * [WebUI] Fix pie chart displaying * [WebUI] Fix read only * [WebUI] Fix read only2 * [WebUI] Fix retarded datatables * [WebUI] Fix soft reject in pie chart * [WebUI] Fix stat widgets timers multiplication on `Refresh` click * [WebUI] Fix symbols config * [WebUI] Fix various errors with login form * [WebUI] Further fixes * [WebUI] Hide learning tab in read-only mode * [WebUI] Initial clusters support * [WebUI] Make legend entry colours more contrast * [WebUI] Move configuration tab to a separate module * [WebUI] Move history tab * [WebUI] Move symbols config as well * [WebUI] New sec to time function * [WebUI] Prevent multiple clicks on `Refresh` * [WebUI] RRD summary: Hide inner labels of tiny pie sectors * [WebUI] RRD summary: Respect undefined values * [WebUI] Reduce font size of graph's legend * [WebUI] Remove orphaned font duplicates * [WebUI] Remove unused code * [WebUI] Replace spinner with animated glyphicon * [WebUI] Reset refresh timer on server switching * [WebUI] Rework interface to use requirejs * [WebUI] Rework neighbours query function * [WebUI] Separate attributes by space * [WebUI] Set focus to password field (#1230) * [WebUI] Simplify neighbours table populating * [WebUI] Start rework of modules * [WebUI] Stop stats refreshing if the page is hidden * [WebUI] Turn d3pie's stuff into a reusable function, * [WebUI] Unify send data functions * [WebUI] Update D3Evolution to 0.0.1 * [WebUI] Update d3.js * [WebUI] Update datatables to work with the requirejs * [WebUI] Use unified tab click event handler, * [WebUI] clusters for the chart * [WebUI] fix uptime Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
37573488e2 |
Release 1.4.2
* [CritFix] Deal with absent headers in DKIM * [CritFix] Do not trust remote shingles count * [CritFix] Fix headers selection in DKIM verification * [Feature] Add EXT_CSS rule * [Feature] Add toggle for disabling SURBLs * [Feature] Extend redis lock when learning spawned * [Feature] Parse <link> HTML tags * [Fix] Avoid reusing g_error (Fixes #1262) * [Fix] Do not reset loaded ANN when learning is requested * [Fix] Fix another issue with external deps in SA * [Fix] Fix body trie matching * [Fix] Fix checking of DKIM bodies that needs just `\n` to be added * [Fix] Fix fuzzy hashes count * [Fix] Fix keys names * [Fix] Fix length calculations for url encoded urls * [Fix] Fix matching of the same patterns from different tries * [Fix] Fix name of var * [Fix] Fix parsing of URLs with spaces and other bad chars * [Fix] Fix probability calculations for fuzzy redis backend * [Fix] Fix signing headers creation in DKIM * [Fix] Plug memory leak * [Fix] Really fix chained SA dependencies * [Fix] Really increase lock lifetime * [Fix] Use checksum to avoid intersection between different ANNs * [Fix] Use rspamd hashes in embedded ucl * [Fix] Yet another change for testing external deps Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
cbdf99ae2e
|
Release 1.4.1
* [Feature] ASN support in Clickhouse module * [Feature] Add clickhouse plugin * [Feature] Add generic tool to add universal maps for lua modules * [Feature] Add logger.debugm to debug lua modules * [Feature] Allow to register metrics symbols using register_symbol * [Feature] Allow to specify prefix for fann_redis * [Feature] Clickhouse: support different masks for IPv4/IPv6 * [Feature] Support forcing action in antivirus plugin * [Fix] Add handling of regexp maps * [Fix] Allow backslashes in http urls * [Fix] Avoid mapping of empty files * [Fix] Do not load tld file to speed up rspamadm * [Fix] Do not resolve numeric IP addresses due to ipv6 insanity * [Fix] Filter incorrect training data * [Fix] Fix Fuzzyconvert tool when password or DB is given * [Fix] Fix build with custom glib/gmime * [Fix] Fix converting of learn count from sqlite to redis * [Fix] Fix crashes with invalid received and task:set_from_ip * [Fix] Fix external dependencies for SA module * [Fix] Fix fann_redis when number of scores has been changed * [Fix] Fix hyperscan usage for non compatible platforms * [Fix] Fix loading of maps from UCL objects * [Fix] Fix memory leak for task-less redis requests * [Fix] Fix mid module with new maps syntax * [Fix] Fix parsing of URLs with username * [Fix] Fix re cache initialisation * [Fix] Fix replacements to sanitize '%' character * [Fix] Fix set and regexp like static maps * [Fix] Fix some issues in redis settings * [Fix] Fix static IP maps * [Fix] Fix total learns counter for redis stats * [Fix] Fix usage of config during reload * [Fix] Fix various warnings and issues * [Fix] Invalidate ANN if training data is incorrect * [Fix] Miltiple fixes to fann_redis module * [Fix] More fixes for URLs with backslashes * [Fix] Properly get options for ip_score module * [Fix] Relax requirements for Received as gmail cannot RFC * [Fix] Remove or fix hyperscan incompatible regexps * [Fix] Settings: correctly read redis config * [Rework] Rework lua logger interface slightly * [Rework] Use new maps add function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
919cbd477d
|
Release 1.4.0
* [CritFix] Add guards for inactive redis connections
* [CritFix] Another fix for proxying files using rspamd_proxy
* [CritFix] Cleanup inactive redis connections
* [CritFix] Do not sometimes try to exec posfilters before classification
* [CritFix] Fix application of IPv6 mask
* [CritFix] Fix chunked encoding when reading messages
* [CritFix] Fix file mode for rspamd_proxy
* [CritFix] Fix hyperscan compilation on regexp change
* [CritFix] Fix issue with finding of end of lines pointers
* [CritFix] Fix iteration over headers array (introduced in 1.4)
* [CritFix] Fix processing of learned tokens count for redis backend
* [CritFix] Fix race condition in checking of cached maps
* [CritFix] Fix workers scripts by sharing workers configs
* [CritFix] Introduce raw content to text parts
* [CritFix] Plug memory leak and potential memory corruption
* [Feature] Adaptive ratelimits
* [Feature] Add ASN -> rbldnsd script for asn.rspamd.com
* [Feature] Add DMARC_NA symbol
* [Feature] Add F-Prot support to antivirus module
* [Feature] Add HTTP backend to metadata exporter
* [Feature] Add Lua API module for monitored objects
* [Feature] Add R_DKIM_NA / R_SPF_NA / AUTH_NA symbols
* [Feature] Add R_DKIM_PERMFAIL symbol
* [Feature] Add R_SPF_PERMFAIL symbol
* [Feature] Add Sophos antivirus support
* [Feature] Add ZSTD compression to Lua API
* [Feature] Add `mid` Lua module
* [Feature] Add `one_param` flag for metric symbols
* [Feature] Add a generic lua classifier
* [Feature] Add a very basic interface to access workers data from on_load
* [Feature] Add ability to delete a hash by its data to fuzzy_check plugin
* [Feature] Add ability to enable/disable symbols via dynamic_conf
* [Feature] Add ability to lookup settings by key
* [Feature] Add common way to disable Lua modules
* [Feature] Add compression support to rspamd client
* [Feature] Add condition to do antiviral check
* [Feature] Add configuration for lua classifiers
* [Feature] Add configuration knobs for the errors circular buffer
* [Feature] Add decompression support in rspamd client
* [Feature] Add errors exporter to the controller
* [Feature] Add expected value for monitored DNS resources
* [Feature] Add exporter from error ringbuf to ucl
* [Feature] Add extended version for fann creation function
* [Feature] Add ffi friendly version of process_regexp function
* [Feature] Add frequency and time display to webui
* [Feature] Add fuzzy_delhash command to rspamc client
* [Feature] Add implementation of redis connections pool
* [Feature] Add latency and offline time monitoring
* [Feature] Add learning support for lua classifiers
* [Feature] Add max-size and timeout options to CGP helper
* [Feature] Add method to enable/disable symbols in config
* [Feature] Add methods to get metric's actions and symbols from Lua
* [Feature] Add mmap support to lua_text
* [Feature] Add monitored object for surbl plugin
* [Feature] Add more exceptions to surbl whitelist
* [Feature] Add more meta-tokens to bayes
* [Feature] Add neural net classifier to fann_scores module
* [Feature] Add neural net serialization/deserialization
* [Feature] Add new dynamic conf module
* [Feature] Add periodic events support for lua_config
* [Feature] Add plugin to check MX'es for the sender's domain
* [Feature] Add preliminary monitored module
* [Feature] Add preliminary support of dynamic conf updates in Redis
* [Feature] Add preliminary version of clamav plugin
* [Feature] Add redis cache to asn module
* [Feature] Add replies compression
* [Feature] Add spamhaus DROP dnsbl
* [Feature] Add support for dictionary in client compression
* [Feature] Add support for fuzzy learn and unlearn from lua
* [Feature] Add support for input encryption
* [Feature] Add support of min_learns to neural net classifier
* [Feature] Add termination callbacks for workers
* [Feature] Add user-agent for rspamc
* [Feature] Add utility to perform classifier tests
* [Feature] Add zstd compression library
* [Feature] Allow HTTPS requests in lua_http
* [Feature] Allow conditions for pre and postfilters
* [Feature] Allow custom functions for ratelimits
* [Feature] Allow for excluding messages from AV scanning based on size
* [Feature] Allow for getting worker stats from Lua
* [Feature] Allow getting task UID from Lua
* [Feature] Allow parsing of mailbox messages from the commandline
* [Feature] Allow plugins to publish their lua API via rspamd_plugins
* [Feature] Allow to compare other systems with Rspamd
* [Feature] Allow to execute Lua scripts by controller
* [Feature] Allow to have a function to set custom greylist message
* [Feature] Allow to iterate over multiple tags
* [Feature] Allow to pass extra data from plugins to log helper
* [Feature] Allow to plan new periodics at different time
* [Feature] Allow to reset hashes
* [Feature] Allow to run rspamadm lua just as a lua interpreter
* [Feature] Allow to store settings in redis
* [Feature] Allow to update dynamic conf in Redis
* [Feature] Allow to use dictionaries for compression
* [Feature] Allow to use md5, sha1, sha256, sha384 and sha512 hashes in Lua
* [Feature] Allow whitelisting by IP for greylisting plugin
* [Feature] Antivirus: Support whitelists & pattern-matching sig names
* [Feature] Backport pack/unpack routines from Lua 5.3
* [Feature] Check settings with equal priopities in alphabetical order
* [Feature] Compress neural net in redis
* [Feature] Consider more tags when doing WHITE_ON_WHITE rule
* [Feature] Descriptive options for DMARC failure symbols
* [Feature] Descriptive options for RBL symbols
* [Feature] Enable configuration for monitored objects
* [Feature] Execute on_load scripts with ev_base ready
* [Feature] Fann scores now uses metadata from a message
* [Feature] Implement FANN threaded learning
* [Feature] Implement classifying for lua classifiers
* [Feature] Implement finish scripts for worker processes
* [Feature] Implement monitoring for DNS resources
* [Feature] Implement real priorities for pre and post filters
* [Feature] Insert two symbols: FANN_HAM and FANN_SPAM instead of one
* [Feature] Module to push metadata/messages to redis pubsub
* [Feature] Monitor RBL records
* [Feature] Move fann_classifier to a separate plugin
* [Feature] Normalize all ANN inputs
* [Feature] Preliminary version of metric exporter module
* [Feature] Preserve decompression context between tasks
* [Feature] Ratelimit: Support dynamic bucket size/leak rate
* [Feature] Relax FORGED_RECIPIENTS: allow senders to BCC themselves
* [Feature] Remove symbols weights on composites processing
* [Feature] Return symbol scores when getting resulting symbols
* [Feature] Rework lua tcp module
* [Feature] Rule to detect some obvious X-PHP-Originating-Script forgeries
* [Feature] Rule to identify some X-PHP-Script forgeries
* [Feature] Rules for scoring Google Message-ID fixes
* [Feature] Send hashes values to reply
* [Feature] Set expire for dmarc reports
* [Feature] Stop using cymru zone as it is unstable
* [Feature] Stop using of GLists for headers, improve performance
* [Feature] Store `for` in task:get_received_headers
* [Feature] Store `for` part in received headers
* [Feature] Store enabled flag for webui session
* [Feature] Store error messages in ring buffer
* [Feature] Support compressed maps
* [Feature] Support excluding selected users from ratelimits
* [Feature] Support looking up NS records in lua_dns
* [Feature] Support modern style SURBL configuration
* [Feature] Support multiple hashes in delhash path
* [Feature] Support new messages in rspamc
* [Feature] Support requests without reads in lua_tcp
* [Feature] Support setting task message from Lua
* [Feature] Track visibility of HTML elements
* [Feature] Try to add CRLF when checking DKIM
* [Feature] Try to guess line endings when folding headers
* [Feature] Try to improve normalization function for bayes
* [Feature] Use FFI to optimize SA module
* [Feature] Use length based arguments for redis, allow lua_text as arg
* [Feature] Use more layers for fann and another normalization
* [Feature] User-defined ratelimits
* [Feature] Utility to convert fuzzy storage from sqlite to redis
* [Feature] Yield DMARC_DNSFAIL on lookup failure
* [Fix] Adopt fuzzy storage for flexible backends
* [Fix] Allow plain IP addresses in Rspamd maps
* [Fix] Another fix for brain-damaged hiredis
* [Fix] Another fix for rdns write errors
* [Fix] Another fix for rdns_make_request_full invocation
* [Fix] Another fix in DKIM canonicalization
* [Fix] Another memory leak plugged
* [Fix] Another try to deal with posix idiotizm
* [Fix] Another try to fix RDNS events processing logic
* [Fix] Avoid double frees in HEAD requests
* [Fix] Avoid extra symbols for RBLs
* [Fix] Banish table.maxn from Lua parts
* [Fix] Check for socket error before connection in lua_tcp
* [Fix] Correctly propagate redis timeouts to Lua
* [Fix] Do not add extra newline in MIME mode
* [Fix] Do not be cheated by system hiredis
* [Fix] Do not classify when a message has not enough tokens
* [Fix] Do not crash on redis errors
* [Fix] Do not distinguish NXDOMAIN and NOREC for monitored
* [Fix] Do not replan retransmits if merely one server is defined
* [Fix] Do not use headers to calculate messages digests
* [Fix] Don't force action in replies module for authenticated users/local networks
* [Fix] Explicitly ban default passwords in webui
* [Fix] Finally fix ambiguity between parsed and resolved spf elts
* [Fix] Fix 'decoded' value in task:get_header_full()
* [Fix] Fix DKIM calculations
* [Fix] Fix DKIM signing for messages with no newline at the end
* [Fix] Fix DNS request in monitored
* [Fix] Fix DNS write errors processing
* [Fix] Fix HTTP methods other than GET and POST
* [Fix] Fix PERMFAIL for v6/v4 ambiguities
* [Fix] Fix absurdic scores for HFILTER_URL_ONLY
* [Fix] Fix actions in rolling history
* [Fix] Fix actrie patterns
* [Fix] Fix applying of lua dynamic confg
* [Fix] Fix autolearning errors and redis cache
* [Fix] Fix bayes learn_condition
* [Fix] Fix build with the recent OpenSSL
* [Fix] Fix caching and compressed maps
* [Fix] Fix check plain text part
* [Fix] Fix crash on OpenBSD in `url_email_start`
* [Fix] Fix double free in SPF
* [Fix] Fix extraction of shingles from redis fuzzy storage
* [Fix] Fix false sharing for symbols in the cache
* [Fix] Fix float usage in util:get_time
* [Fix] Fix folding algorithm to deal with empty tokens
* [Fix] Fix format string
* [Fix] Fix format string usage in controller errors handling
* [Fix] Fix handling of '\0' in lua_tcp
* [Fix] Fix handling of HTTP HEAD methods
* [Fix] Fix hash creation
* [Fix] Fix hiredis stupidity
* [Fix] Fix implicit settings module settingsup
* [Fix] Fix interaction with lua GC to avoid craches
* [Fix] Fix ip_score module registration
* [Fix] Fix issue with empty messages and dkim
* [Fix] Fix issues with CGP helper
* [Fix] Fix issues with the recent SPF changes
* [Fix] Fix key name to load ANN correctly
* [Fix] Fix lua tcp module by saving `do_read` in callback data
* [Fix] Fix memory leak in client when using compression
* [Fix] Fix min_learns option
* [Fix] Fix on_finish scripts and async handlers
* [Fix] Fix options for SPF dnsfail symbol
* [Fix] Fix parsing includes and redirects in SPF
* [Fix] Fix parsing of lua comments with empty lines
* [Fix] Fix parsing of unquoted HTML attributes
* [Fix] Fix periodic events and redis
* [Fix] Fix processing of fuzzy learns from Lua
* [Fix] Fix processing of redirect in SPF includes
* [Fix] Fix processing of symbols when reject limit is reached
* [Fix] Fix refcounts when map is specified by IP
* [Fix] Fix rspamd{session} class in Lua API
* [Fix] Fix setting ratelimit key for 'ip' bucket
* [Fix] Fix some cases of TLD urls detector
* [Fix] Fix statconvert tool
* [Fix] Fix stats for backend-less classifiers
* [Fix] Fix training script for fann_redis
* [Fix] Fix variable in ann module
* [Fix] Fix various errors in lua dynamic conf plugin
* [Fix] Forget old ANN when max_usages is reached to avoid overtrain
* [Fix] Further canonicalization fixes
* [Fix] Further fixes for fann_redis prefixes
* [Fix] Handle failures for inactive pooled connections
* [Fix] Improve multimap info message
* [Fix] More fixes in ANN loading
* [Fix] More fixes to fann_redis
* [Fix] More issues in fann_redis
* [Fix] More spaces fix in DKIM signature
* [Fix] Multiple fixes to asn script, add IPv6 support
* [Fix] Multiple issues in fann_redis
* [Fix] No greylist rejected messages
* [Fix] One more attempt to fix lua_redis
* [Fix] One more check for readdir...
* [Fix] Params should be treated as a hash
* [Fix] Plug memory leak in regexp desctructor
* [Fix] Process headers only once
* [Fix] Properly handle nil values in ratelimit plugin
* [Fix] Really fix redis shingles check
* [Fix] Remove fann with incorrect layers count
* [Fix] Remove mentions of deleted include
* [Fix] Remove some incompatible functions
* [Fix] Settings: fix `authenticated` parameter (#886)
* [Fix] Skip MX check for authenticated users and local networks
* [Fix] Slightly fix ANN routines
* [Fix] Stop caching records with DNS failures
* [Fix] Treat all errors in redis_pool as fatal errors for a connection
* [Fix] Try avoid false-positives in HEADER_FORGED_MDN rule
* [Fix] Try to avoid race condition when using rrd
* [Fix] Try to reload redis scripts if they are missing
* [Fix] Unbreak once_received skipping for local networks
* [Fix] Unlock ANN on error
* [Fix] Use memmove for overlapping regions
* [Fix] Use real size instead of displayed for core limits
* [Fix] Use the correct macro to get the size of control
* [Fix] Various fixes for errors ringbuffer
* [Fix] Yield R_SPF_DNSFAIL if lookup of included record fails
* [Fix] mid: fix map initialization
* [Fix] mid: handle incorrect rgexps in the map
* [Rework] Add extract training data function to fann_redis
* [Rework] Add preliminary train tests
* [Rework] Add redis storage feature to fann_redis
* [Rework] Adopt fuzzy storage for abstract backend
* [Rework] Adopt plugins
* [Rework] First reiteration on fann scores
* [Rework] Implement loading/invalidating
* [Rework] Make lua_redis task agnostic
* [Rework] Make rspamd protocol messages useful
* [Rework] Massive removal of legacy code
* [Rework] More cleanup actions
* [Rework] Remove legacy code never used for classifiers
* [Rework] Remove outdated and unused lua_session module
* [Rework] Reorganize fuzzy backend structure
* [Rework] Reorganize the internal backend structure
* [Rework] Restore old fann_scores, move common parts
* [Rework] Rework and simplify rbl plugin
* [Rework] Rework parsing of DMARC records
Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>
|
7 years ago |
|
Vsevolod Stakhov
|
c30af6a8c9
|
Release 1.3.4
* [Feature] ASN module; support matching ASN/country in multimap * [Feature] Add SPF method in spf return result * [Feature] Add Yandex and Mail.ru forwarding rules * [Feature] Add mempool maps in multimap * [Feature] Add rule for identifying mail sent by eval()'d PHP code * [Feature] Add support of stub DNSSEC resolver to rdns * [Feature] Add task:get_digest method * [Feature] Allow for more fine-grained scoring for ip_score * [Feature] Allow to get digest of a mime part from lua * [Feature] Allow to print message digest in logs * [Feature] Fold DKIM-Signature header * [Feature] Implement encrypted logs * [Feature] Log URLs encrypted if we have log encryption pubkey * [Feature] Pass authenticated bit to lua * [Feature] Read redis backend statistics configuration from global section * [Feature] Show the exact value matched for multima symbols * [Feature] Store task checksum * [Fix] Avoid setting limits when required elements are missing * [Fix] DMARC: Fix alignment checking for subdomains * [Fix] DMARC: deal with missing and spurious spaces * [Fix] Defer insertion of results in ip_score to avoid skewing stats * [Fix] Disable DMARC for local/authorized mail * [Fix] Fix handling of proxied headers in controller * [Fix] Fix hex printing of strings * [Fix] Fix issue with spaces in maps * [Fix] Fix parsing of forwarded IP * [Fix] Fix reload in some plugins and workers * [Fix] Fix reloading on SIGHUP * [Fix] Fix some border cases for DKIM canonicalization * [Fix] Fix url maps * [Fix] Make dnssec configurable option disabled by default for now * [Fix] rspamadm statconvert: force db to be a string * [Fix] rspamadm statconvert: use db/password for learn cache * [Rework] Rework flags in rspamd logger Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |
|
Vsevolod Stakhov
|
8466d1816d |
Release 1.3.3
* [CritFix] Check hyperscan cache sanity before loading * [CritFix] Fix setting of fuzzy keys (completely breaks fuzzy storage) * [Feature] Add SARBL (sarbl.org) uribl * [Feature] Add `--search-pattern` option to rspamd_stats * [Feature] Add some sanity check for very long from/to log elements * [Feature] Allow to create hashes from string in a single step * [Feature] Fix order of pre and postfilters * [Feature] Improve lua URLs API * [Feature] Improve message about fuzzy rules * [Feature] Pre-calculate blake2 digest for all parts * [Feature] Print radix duplicate keys as IP addresses * [Feature] Simple mechanism for disabling RBLs in local.d/rbl.conf * [Feature] Use faster hash function for fuzzy storage * [Feature] rspamd_stats: support log directory reading * [Fix] Add sanity check for url filters * [Fix] Do not show rmilter section as a fake metric in rspamc * [Fix] Fix URL filters * [Fix] Fix a stupid mistake in util.strequal_caseless * [Fix] Fix blake2b hash of the string "rspamd" * [Fix] Fix filename maps filter * [Fix] Fix finding tld in util.get_tld * [Fix] Fix multimap content filters * [Fix] Fix returning boolean from Lua * [Fix] Fix returning of REDIS_NIL * [Fix] Try to deal with multiple workers terminated * [Fix] Use forced DNS request when calling for lua_http * [Rework] Rework multimap filters, add redis maps Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru> |
7 years ago |