Release 1.1.01.1.0

* Incompatible change: sqlite3 and per_user behaviour:
	Now both redis and sqlite3 follows the common principles for per-user
	statistics:
	1) If per-user statistics is enabled check per-user tokens ONLY
	2) If per-user statistics is not enabled then check common tokens ONLY
	If you need old behaviour, then you'd need to use separate classifier
	for per-user statistics.
* Implement redis statistics backend and cache
* Implement autolearning for statistics
* Reworked statistics architecture from scratch
* Add hyperscan (https://github.com/01org/hyperscan) engine for regular
	expressions:
	- add lazy loader for hyperscan databases
	- rework regexp cache to have joint pcre/hyperscan scanning
	- implement hyperscan pre-filter support
	- add compilation guards for bad expressions
	- implement `rspamadm control recompile` command
	- implement hyperscan cache monitoring
	- slides: <https://highsecure.ru/rspamd-hyperscan.pdf>
* Implement flexible task logging
* Rework fuzzy worker:
	- it is now possible to run multiple fuzzy workers;
	- implement lazy writing as sqlite3 is bad at concurrent writing;
	- add retries for simple sql commands in fuzzy backend;
	- use fine-grained transactions for fuzzy;
	- implement new multi-pubkeys mode;
	- allow encrypted only storages;
	- rework statistics for fuzzy;
	- add `rspamadm control fuzzystat` command for extended statistics;
	- implement human readable output for the previous command;
	- add condition script for learning fuzzy storage;
* Various fixes to SPF:
	- fix `redirect` records;
	- fix domains when parsing mx/ptr/a records in includes/redirects;
	- fix issues with multiple addresses in SPF records;
	- ignore SPF results in case of DNS failure;
	- adjust TTL of records when resolving subelements of SPF records;
	- always select `v=spf1` line if it is available
	- do not cache records with DNS failure in subrequests;
	- ignore records with temporary fails during subrequests resolving;
	- fix `RDNS_RC_NOREC` support;
* Add clang plugin for static analysis:
	- implement static checks for `rspamd_printf` format strings;
* Add 'allow_raw_input' option for non-mime messages
* Recognize types using libmagic
* Fix parsing of IPv6 received headers
* Add new interface of communication between workers in rspamd
* Add support for named socketpairs
* Don't write URLs by default as it is too verbose
* Set status for HTTP replies
* Try load `rspamd.conf.override`
* Implement words decaying for text parts to limit many checks
* Improve support of SA rules and plugins:
	- add check_for_shifted_date and check_for_missing_to_header eval rules;
	- add 'check_relays_unparseable' support;
	- add `check_for_mime('mime_attachement')` function;
	- use new re_cache interface for all SA rules;
	- add support for `Mail::SpamAssassin::Plugin::MIMEHeader`;
	- add support of 'special' SA headers to `exists` function;
	- fix issue when SA metas contain other metas;
	- fix freemail rules;
* Many fixes to the URL parser
* Match any newline character in regexps
* Fix resolving of upstreams and detection of poor IPv6 configurations
* Parse upstreams selection algorithm from the configuration line
* Add `reresolve` command to the control interface
* Generate fuzzy hashes from task metadata (URLs and headers)
* Add method to check if IP is local and `local_addrs` option
* Implement forced timeout for delayed filters
* Disable fast path of pcre-jit as it seems to be broken
* Bayes fixes:
	- new normalizer function;
	- really use weights of tokens from the OSB algorithm;
	- restore multiple classifiers support;
* Rules changes:
	- add `R_SUSPICIOUS_URL` rule that detects obfuscated URL's;
	- improve empty image rule;
	- rework `FORGED_RECIPIENTS` rule;
	- reduce weight of `SUSPICIOUS_RECIPS`;
	- fix `*_NORESOLVE_MX` symbols in hfilter;
	- add `SUBJ_ALL_CAPS` rule with support of UTF8
	- add spamhaus SBL to uribl
	- fix `SUSPICIOUS_RECIPS` and `SORTED_RECIPS` rules
	- remove `R_TO_SEEMS_AUTO` as it generates a lot of FP;
	- add new Message-ID regexp for Thunderbird (by @moisseev);
* Plugins changes:
	- allow ratelimit plugin to set symbol instead of pre-result
	- support IP DNS black lists for URIBL (e.g spamhaus SBL);
	- drop deprecated SURBL bits (by @fatalbanana)
	- rename `JP_SURBL_MULTI` to `ABUSE_SURBL` (by @fatalbanana)
	- add `SURBL_BLOCKED` (by @fatalbanana)
	- add `CR_SURBL`
	- SURBL: allow fallthrough to default symbol (by @fatalbanana)
	- Settings: fix IP match (by @fatalbanana)
	- SURBL: add missing symbols to metric (by @fatalbanana)
	- allow processing images urls for SURBL
	- unconditionally disable SPF for authenticated users and local networks
* Rework ratelimit plugin
	- switch to `rates` instead of old and stupid strings to setup;
	- check if a bucket is zero and disable the corresponding limits'
	- turn off all buckets by default;
	- check either `rcpt` or `user` buckets, not all together'
	- document new `rates` and `symbol` options;
	- inform user about what buckets are used in the configuration;
* Add neural network **experimental** plugin
* Add a sample script to learn neural network from rspamd logs
* Add documentation strings support to rspamd:
	- add strings for the main configuration options;
	- document workers options;
	- add internal plugin options;
	- create `rspamadm confighelp` routine;
	- implement human readable output for the previous command;
	- add subtree search support;
	- add keyword search support;
* Documentation improvements, tutorials section, statistics description
* Many other minor and major bugfixes not noted here

Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>

1 files changed, 118 insertions, 0 deletions

diff --git a/ChangeLog b/ChangeLog
index f1ad2220a..ace5a2050 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,121 @@
+1.1.0:
+	* Incompatible change: sqlite3 and per_user behaviour:
+		Now both redis and sqlite3 follows the common principles for per-user
+		statistics:
+		1) If per-user statistics is enabled check per-user tokens ONLY
+		2) If per-user statistics is not enabled then check common tokens ONLY
+		If you need old behaviour, then you'd need to use separate classifier
+		for per-user statistics.
+	* Implement redis statistics backend and cache
+	* Implement autolearning for statistics
+	* Reworked statistics architecture from scratch
+	* Add hyperscan (https://github.com/01org/hyperscan) engine for regular
+	  expressions:
+		- add lazy loader for hyperscan databases
+		- rework regexp cache to have joint pcre/hyperscan scanning
+		- implement hyperscan pre-filter support
+		- add compilation guards for bad expressions
+		- implement `rspamadm control recompile` command
+		- implement hyperscan cache monitoring
+		- slides: <https://highsecure.ru/rspamd-hyperscan.pdf>
+	* Implement flexible task logging
+	* Rework fuzzy worker:
+		- it is now possible to run multiple fuzzy workers;
+		- implement lazy writing as sqlite3 is bad at concurrent writing;
+		- add retries for simple sql commands in fuzzy backend;
+		- use fine-grained transactions for fuzzy;
+		- implement new multi-pubkeys mode;
+		- allow encrypted only storages;
+		- rework statistics for fuzzy;
+		- add `rspamadm control fuzzystat` command for extended statistics;
+		- implement human readable output for the previous command;
+		- add condition script for learning fuzzy storage;
+	* Various fixes to SPF:
+		- fix `redirect` records;
+		- fix domains when parsing mx/ptr/a records in includes/redirects;
+		- fix issues with multiple addresses in SPF records;
+		- ignore SPF results in case of DNS failure;
+		- adjust TTL of records when resolving subelements of SPF records;
+		- always select `v=spf1` line if it is available
+		- do not cache records with DNS failure in subrequests;
+		- ignore records with temporary fails during subrequests resolving;
+		- fix `RDNS_RC_NOREC` support;
+	* Add clang plugin for static analysis:
+		- implement static checks for `rspamd_printf` format strings;
+	* Add 'allow_raw_input' option for non-mime messages
+	* Recognize types using libmagic
+	* Fix parsing of IPv6 received headers
+	* Add new interface of communication between workers in rspamd
+	* Add support for named socketpairs
+	* Don't write URLs by default as it is too verbose
+	* Set status for HTTP replies
+	* Try load `rspamd.conf.override`
+	* Implement words decaying for text parts to limit many checks
+	* Improve support of SA rules and plugins:
+		- add check_for_shifted_date and check_for_missing_to_header eval rules;
+		- add 'check_relays_unparseable' support;
+		- add `check_for_mime('mime_attachement')` function;
+		- use new re_cache interface for all SA rules;
+		- add support for `Mail::SpamAssassin::Plugin::MIMEHeader`;
+		- add support of 'special' SA headers to `exists` function;
+		- fix issue when SA metas contain other metas;
+		- fix freemail rules;
+	* Many fixes to the URL parser
+	* Match any newline character in regexps
+	* Fix resolving of upstreams and detection of poor IPv6 configurations
+	* Parse upstreams selection algorithm from the configuration line
+	* Add `reresolve` command to the control interface
+	* Generate fuzzy hashes from task metadata (URLs and headers)
+	* Add method to check if IP is local and `local_addrs` option
+	* Implement forced timeout for delayed filters
+	* Disable fast path of pcre-jit as it seems to be broken
+	* Bayes fixes:
+		- new normalizer function;
+		- really use weights of tokens from the OSB algorithm;
+		- restore multiple classifiers support;
+	* Rules changes:
+		- add `R_SUSPICIOUS_URL` rule that detects obfuscated URL's;
+		- improve empty image rule;
+		- rework `FORGED_RECIPIENTS` rule;
+		- reduce weight of `SUSPICIOUS_RECIPS`;
+		- fix `*_NORESOLVE_MX` symbols in hfilter;
+		- add `SUBJ_ALL_CAPS` rule with support of UTF8
+		- add spamhaus SBL to uribl
+		- fix `SUSPICIOUS_RECIPS` and `SORTED_RECIPS` rules
+		- remove `R_TO_SEEMS_AUTO` as it generates a lot of FP;
+		- add new Message-ID regexp for Thunderbird (by @moisseev);
+	* Plugins changes:
+		- allow ratelimit plugin to set symbol instead of pre-result
+		- support IP DNS black lists for URIBL (e.g spamhaus SBL);
+		- drop deprecated SURBL bits (by @fatalbanana)
+		- rename `JP_SURBL_MULTI` to `ABUSE_SURBL` (by @fatalbanana)
+		- add `SURBL_BLOCKED` (by @fatalbanana)
+		- add `CR_SURBL`
+		- SURBL: allow fallthrough to default symbol (by @fatalbanana)
+		- Settings: fix IP match (by @fatalbanana)
+		- SURBL: add missing symbols to metric (by @fatalbanana)
+		- allow processing images urls for SURBL
+		- unconditionally disable SPF for authenticated users and local networks
+	* Rework ratelimit plugin
+		- switch to `rates` instead of old and stupid strings to setup;
+		- check if a bucket is zero and disable the corresponding limits'
+		- turn off all buckets by default;
+		- check either `rcpt` or `user` buckets, not all together'
+		- document new `rates` and `symbol` options;
+		- inform user about what buckets are used in the configuration;
+	* Add neural network **experimental** plugin
+	* Add a sample script to learn neural network from rspamd logs
+	* Add documentation strings support to rspamd:
+		- add strings for the main configuration options;
+		- document workers options;
+		- add internal plugin options;
+		- create `rspamadm confighelp` routine;
+		- implement human readable output for the previous command;
+		- add subtree search support;
+		- add keyword search support;
+	* Documentation improvements, tutorials section, statistics description
+	* Many other minor and major bugfixes not noted here
+
 1.0.11:
 	* Fix spf redirects
 	* Fix domains when parsing mx/ptr/a records in includes/redirects