Merge remote-tracking branch 'upstream/master' into documentation

* upstream/master: (90 commits)
  [Fix] Plug memory leak in proxy
  [Feature] Do not print garbadge in --compact output
  [Fix] Fix encrypted proxy requests
  [Fix] Do not delete uninitialized events
  [Feature] Add protection against open files limit and accepting sockets
  [Fix] Another fix for redis timeouts
  [Fix] Fix order of initialization
  [Feature] Use file lock in logger to avoid deadlocks
  [Fix] Fix errors handling in the proxy
  [Fix] More fixes for redis refcounts
  [Fix] Initialize parser scripts properly
  [Fix] Try to fix issue in redis stats backend when task is closed
  [Fix] Fix usage of rdns reply structure
  [Fix] Fix symbol name for spf soft fail
  [Fix] Fix setting path for lua
  [Doc] Update regexp module documentation
  [Minor] Fix names
  [Fix] Add missing types
  [Feature] Implement braced regexp quantifiers
  [Fix] Implement new automata to skip empty lines for dkim signing
  ...

3 files changed, 83 insertions, 20 deletions

diff --git a/doc/markdown/modules/multimap.md b/doc/markdown/modules/multimap.md
index 290df9e49..cede3bc94 100644
--- a/doc/markdown/modules/multimap.md
+++ b/doc/markdown/modules/multimap.md
@@ -22,6 +22,7 @@ Multimap module allows to build rules based on the dynamic maps content. Rspamd
 map types in this module:
 
 * `hash map` - a list of domains or `user@domain`
+* `regexp map` - a list of regular expressions
 * `ip map` - an effective radix trie of `ip/mask` values (supports both IPv4 and IPv6 addresses)
 * `cdb` - constant database format (files only)
 
@@ -36,6 +37,8 @@ The module itself contains a set of rules in form:
 
 	symbol { type = type; map = uri; [optional params] }
 
+### Map types
+
 Type attribute means what is matched with this map. The following types are supported:
 
 * `ip` - matches source IP of message (radix map)
@@ -52,7 +55,7 @@ Maps can also be specified as [CDB](http://www.corpit.ru/mjt/tinycdb.html) datab
 
 	map = "cdb:///path/to/file.cdb";
 
-Here is an example configuration of multimap module:
+### Pre-filter maps
 
 To enable pre-filter support, you should specify `action` parameter which can take the
 following values:
@@ -73,13 +76,28 @@ multimap {
 }
 ~~~
 
+### Regexp maps
+
+
 All maps but `ip` and `dnsbl` support `regexp` mode. In this mode, all keys in maps are treated as regular expressions, for example:
 
 	/example\d+\.com/i
 	/other\d+\.com/i test
+	# Comments are still enabled
 
 For performance considerations, use only expressions supported by [hyperscan](http://01org.github.io/hyperscan/dev-reference/compilation.html#pattern-support) as this engine provides blazing performance at no additional cost. Currently, there is no way to distinguish what particular regexp was matched in case if multiple regexp were matched.
 
+To enable regexp mode, you should set `regexp` option to `true`:
+
+~~~ucl
+sender_from_whitelist_user {
+  type = "from";
+  map = "file:///tmp/from.map";
+  symbol = "SENDER_FROM_WHITELIST";
+  regexp = true;
+}
+~~~
+
 ### Map filters
 
 It is also possible to apply a filtering expression before checking value against some map. This is mainly useful
diff --git a/doc/markdown/modules/regexp.md b/doc/markdown/modules/regexp.md
index f08079bff..01d7a0635 100644
--- a/doc/markdown/modules/regexp.md
+++ b/doc/markdown/modules/regexp.md
@@ -60,12 +60,24 @@ The match type is defined by special flags after the last `/` symbol:
 * `B` - MIME header regexp (applied for headers in MIME parts only)
 * `R` - full headers content (applied for all headers undecoded and for the message only - **not** including MIME headers)
 * `M` - raw message regexp
-* `P` - part regexp
+* `P` - part regexp without HTML tags
+* `Q` - part regexp with HTML tags
+* `C` - spamassassin `BODY` regexp analogue(see http://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Conf.txt)
+* `D` - spamassassin `RAWBODY` regexp analogue
 * `U` - URL regexp
 
+From 1.3, it is also possible to specify long regexp types for convenience in curly braces:
 
-We strongly discourage from using of raw message regexps as they are expensive and
-should be replaced by [trie](trie.md) rules if possible.
+* `{header}` - header regexp
+* `{raw_header}` - undecoded header regexp (e.g. without quoted-printable decoding)
+* `{mime_header}` - MIME header regexp (applied for headers in MIME parts only)
+* `{all_header}` - full headers content (applied for all headers undecoded and for the message only - **not** including MIME headers)
+* `{body}` - raw message regexp
+* `{mime}` - part regexp without HTML tags
+* `{raw_mime}` - part regexp with HTML tags
+* `{sa_body}` - spamassassin `BODY` regexp analogue(see http://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Conf.txt)
+* `{sa_raw_body}` - spamassassin `RAWBODY` regexp analogue
+* `{url}` - URL regexp
 
 Each regexp also supports the following flags:
 
@@ -122,7 +134,7 @@ Here is an example of table form definition of regexp rule:
 
 ~~~lua
 config['regexp']['RE_TEST'] = {
-    re = '/test/P',
+    re = '/test/i{mime}',
     score = 10.0,
     condition = function(task)
         if task:get_header('Subject') then
diff --git a/doc/markdown/modules/whitelist.md b/doc/markdown/modules/whitelist.md
index ec4671e58..5b2417194 100644
--- a/doc/markdown/modules/whitelist.md
+++ b/doc/markdown/modules/whitelist.md
@@ -1,6 +1,6 @@
 # Whitelist module
 
-Whitelist module is intended to negate scores for some messages that are known to
+Whitelist module is intended to negate or increase scores for some messages that are known to
 be from the trusted sources. Due to `SMTP` protocol design flaws, it is quite easy to
 forge sender. Therefore, rspamd tries to validate sender based on the following additional
 properties:
@@ -14,15 +14,30 @@ properties:
 Whitelist configuration is quite straightforward. You can define a set of rules within
 `rules` section. Each rule **must** have `domains` attribute that specifies either
 map of domains (if specified as a string) or a direct list of domains (if specified as an array).
-The following optional parameters are allowed:
+
+### Whitelist constraints
+
+The following constraints are allowed:
 
 - `valid_spf`: require a valid SPF policy
 - `valid_dkim`: require DKIM validation
 - `valid_dmarc`: require a valid DMARC policy
 
-These options are combined using `AND` operator, therefore `valid_dkim = true` and
+### Whitelist rules modes
+
+Each whitelist rule can work in 3 modes:
+
+- `whitelist` (default): add symbol when a domain has been found and one of constraints defined is satisfied (e.g. `valid_dmarc`)
+- `blacklist`: add symbol when a domain has been found and one of constraints defined is *NOT* satisfied (e.g. `valid_dmarc`)
+- `strict`: add symbol with negative (ham) score when a domain has been found and one of constraints defined is satisfied (e.g. `valid_dmarc`) and add symbol with **POSITIVE** (spam) score when some of constraints defined has failed
+
+If you do not define any constraints, then all both `strict` and `whitelist` rules just insert result for all mail from the specified domains. For `blacklist` rules the result has normally positive score.
+
+These options are combined using `AND` operator for `whitelist` and using `OR` for `blacklist` and `strict` rules. Therefore, if `valid_dkim = true` and
 `valid_spf = true` would require both DKIM and SPF validation to whitelist domains from
-the list.
+the list. On the contrary, for blacklist and strict rules any violation would cause positive score symbol being inserted.
+
+### Optional settings
 
 You can also set the default metric settings using the ordinary attributes, such as:
 
@@ -50,34 +65,52 @@ whitelist {
             valid_spf = true;
             domains = [
                 "github.com",
-            ]
-            score = -1.0
+            ];
+            score = -1.0;
         }
-    
+
         WHITELIST_DKIM = {
             valid_dkim = true;
             domains = [
                 "github.com",
-            ]
-            score = -2.0
+            ];
+            score = -2.0;
         }
-        
+
         WHITELIST_SPF_DKIM = {
             valid_spf = true;
             valid_dkim = true;
             domains = [
                 ["github.com", 2.0],
-            ]
-            score = -3.0
+            ];
+            score = -3.0;
         }
-        
+
+        STRICT_SPF_DKIM = {
+            valid_spf = true;
+            valid_dkim = true;
+            strict = true;
+            domains = [
+                ["paypal.com", 2.0],
+            ];
+            score = -3.0; # For strict rules negative score should be defined
+        }
+
+        BLACKLIST_DKIM = {
+            valid_spf = true;
+            valid_dkim = true;
+            blacklist = true;
+            domains = "/some/file/blacklist_dkim.map";
+            score = 3.0; # Mention positive score here
+        }
+
         WHITELIST_DMARC_DKIM = {
             valid_dkim = true;
             valid_dmarc = true;
             domains = [
                 "github.com",
-            ]
-            score = -7.0
+            ];
+            score = -7.0;
         }
     }
 }