Rspamd purposes and features.
*****************************

Introduction.
=============

Rspamd filtering system is created as a replacement of popular spamassassin spamd and is designed to be fast, modular and easily extendable system. Rspamd core is written in C language using event driven paradigma. Plugins for rspamd can be written in lua. Rspamd is designed to process connections completely asynchronous and do not block anywhere in code. Spam filtering system contains of several processes among them are:

  * Main process

  * Workers processes

  * Controller process

  * Other processes

Main process manages all other processes, accepting signals from OS (for example SIGHUP) and spawn all types of processes if any of them die. Workers processes do all tasks for filtering e-mail (or HTML messages in case of using rspamd as non-MIME filter). Controller process is designed to manage rspamd itself (for example get statistics or learning rspamd). Other processes can do different jobs among them now are implemented LMTP worker that implements LMTP protocol for filtering mail and fuzzy hashes storage server.

Features.
=========

The main features of rspamd are:

  * Completely asynchronous filtering that allows a big number of simultenious connections.

  * Easily extendable architecture that can be extended by plugins written in lua and by dynamicaly loaded plugins written in c.

  * Ability to work in cluster: rspamd is able to perform statfiles synchronization, dynamic load of lists via HTTP, to use distributed fuzzy hashes storage.

  * Advanced statistics: rspamd now is shipped with winnow-osb classifier that provides more accurate statistics than traditional bayesian algorithms based on single words.

  * Internal optimizer: rspamd first of all try to check rules that were met more often, so for huge spam storms it works very fast as it just checks only that rules that can happen and skip all others.

  * Ability to manage the whole cluster by using controller process.

  * Compatibility with existing spamassassin SPAMC protocol.

  * Extended RSPAMC protocol that allows to pass many additional data from SMTP dialog to rspamd filter.

  * Internal support of IMAP in rspamc client for automated learning.

  * Internal support of many anti-spam technologies, among them are SPF and SURBL.

  * Active support and development of new features.

Installation of rspamd.
***********************

Obtaining of rspamd.
====================

The main rspamd site is sourceforge (http://rspamd.sourceforge.net/). Here you can obtain source code package as well as pre-packed packages for different operating systems and architectures. Also, you can use SCM mercurial (http://mercurial.selenic.com) for accessing rspamd development repository that can be found here: http://rspamd.hg.sourceforge.net:8000/hgroot/rspamd/rspamd. Rspamd is shipped with all modules and sample config by default. But there are some requirements for building and running rspamd.

Requirements.
=============

For building rspamd from sources you need CMake system. CMake is very nice source building system and I decided to use it instead of GNU autotools. CMake can be obtained here: http://cmake.org. Also rspamd uses gmime and glib for MIME parsing and many other purposes (note that you are NOT required to install any GUI libraries -  nor glib, nor gmime are GUI libraries). Gmime and glib can be obtained from gnome site: http://ftp.gnome.org/. For plugins and configuration system you also need lua language interpreter and libraries. They can be easily obtained from official lua site (http://lua.org). Also for rspamc client you need perl interpreter that could be installed from http://www.perl.org.

Building and Installation.
==========================

Build process of rspamd is rather simple:

  * Configure rspamd build environment, using cmake:

          $ cmake .
          ...
          -- Configuring done
          -- Generating done
          -- Build files have been written to: /home/cebka/rspamd

     For special configuring options you can use

          $ ccmake .
           CMAKE_BUILD_TYPE
           CMAKE_INSTALL_PREFIX             /usr/local
           DEBUG_MODE                       ON
           ENABLE_GPERF_TOOLS               OFF
           ENABLE_OPTIMIZATION              OFF
           ENABLE_PERL                      OFF
           ENABLE_PROFILING                 OFF
           ENABLE_REDIRECTOR                OFF
           ENABLE_STATIC                    OFF

     Options allows building rspamd as static module (note that in this case dynamicaly loaded plugins are NOT supported), linking rspamd with google performance tools for benchmarking and include some other flags while building.

  * Build rspamd sources:

          $ make
          [  6%] Built target rspamd_lua
          [ 11%] Built target rspamd_json
          [ 12%] Built target rspamd_evdns
          [ 12%] Built target perlmodule
          [ 58%] Built target rspamd
          [ 76%] Built target test/rspamd-test
          [ 85%] Built target utils/expression-parser
          [ 94%] Built target utils/url-extracter
          [ 97%] Built target rspamd_ipmark
          [100%] Built target rspamd_regmark

  * Install rspamd (as superuser):

          # make install
          Install the project...
          ...

After installation you would have several new files installed:

  * Binaries:

       * PREFIX/bin/rspamd - main rspamd executable

       * PREFIX/bin/rspamc - rspamd client program

  * Sample configuration files and rules:

       * PREFIX/etc/rspamd.xml.sample - sample main config file

       * PREFIX/etc/rspamd/lua/*.lua - rspamd rules

  * Lua plugins:

       * PREFIX/etc/rspamd/plugins/lua/*.lua - lua plugins

For FreeBSD system there also would be start script for running rspamd in PREFIX/etc/rc.d/rspamd.sh.

Running rspamd.
===============

Rspamd can be started by running main rspamd executable - PREFIX/bin/rspamd. There are several command-line options that can be passed to rspamd. All of them can be displayed by passing --help argument:

     $ rspamd --help
     Usage:
       rspamd [OPTION...] - run rspamd daemon

     Summary:
       Rspamd daemon version 0.3.0

     Help Options:
       -?, --help               Show help options

     Application Options:
       -t, --config-test        Do config test and exit
       -f, --no-fork            Do not daemonize main process
       -c, --config             Specify config file
       -u, --user               User to run rspamd as
       -g, --group              Group to run rspamd as
       -p, --pid                Path to pidfile
       -V, --dump-vars          Print all rspamd variables and exit
       -C, --dump-cache         Dump symbols cache stats and exit
       -X, --convert-config     Convert old style of config to xml one

All options are optional: by default rspamd would try to read PREFIX/etc/rspamd.xml config file and run as daemon. Also there is test mode that can be turned on by passing -t argument. In test mode rspamd would read config file and checks its syntax, if config file is OK, then exit code is zero and non zero otherwise. Test mode is useful for testing new config file without restarting of rspamd. With -C and -V arguments it is possible to dump variables or symbols cache data. The last ability can be used for determining which symbols are most often, which are most slow and to watch to real order of rules inside rspamd. -X option can be used to convert old style (pre 0.3.0) config to xml one:

     $ rspamd -c ./rspamd.conf -X ./rspamd.xml

After this command new xml config would be dumped to rspamd.xml file.

Managing rspamd with signals.
=============================

First of all it is important to note that all user's signals should be sent to rspamd main process and not to its children (as for child processes these signals may have other meanings). To determine which process is main you can use two ways:

  * by reading pidfile:

          $ cat pidfile

  * by getting process info:

          $ ps auxwww | grep rspamd
          nobody 28378  0.0  0.2 49744  9424   rspamd: main process (rspamd)
          nobody 64082  0.0  0.2 50784  9520   rspamd: worker process (rspamd)
          nobody 64083  0.0  0.3 51792 11036   rspamd: worker process (rspamd)
          nobody 64084  0.0  2.7 158288 114200 rspamd: controller process (rspamd)
          nobody 64085  0.0  1.8 116304 75228  rspamd: fuzzy storage (rspamd)

          $ ps auxwww | grep rspamd | grep main
          nobody 28378  0.0  0.2 49744  9424   rspamd: main process (rspamd)

After getting pid of main process it is possible to manage rspamd with signals:

  * SIGHUP - restart rspamd: reread config file, start new workers (as well as controller and other processes), stop accepting connections by old workers, reopen all log files. Note that old workers would be terminated after one minute that should allow to process all pending requests. All new requests to rspamd would be processed by newly started workers.

  * SIGTERM - terminate rspamd system.

These signals may be used in start scripts as it is done in FreeBSD start script. Restarting of rspamd is doing rather softly: no connections would be dropped and if new config is syntaxically incorrect old config would be used.

Configuring of rspamd.
**********************

Principles of work.
===================

We need to define several terms to explain configuration of rspamd. Rspamd operates with rules, each rule defines some actions that should be done with message to obtain result. Result is called symbol - a symbolic representation of rule. For example, if we have a rule to check DNS record for a url that contains in message we may insert resulting symbol if this DNS record is found. Each symbol has several attributes:

  * name - symbolic name of symbol (usually uppercase, e.g. MIME_HTML_ONLY)

  * weight - numeric weight of this symbol (this means how important this rule is), may be negative

  * options - list of symbolic options that defines additional information about processing this rule

Weights of symbols are called factors. Also when symbol is inserted it is possible to define additional multiplier to factor. This can be used for rules that have dynamic weights, for example statistical rules (when probability is higher weight must be higher as well).

All symbols and corresponding rules are combined in metrics. Metric defines a group of symbols that are designed for common purposes. Each metric has maximum weight: if sum of all rules' results (symbols) is bigger than this limit then this message is considered as spam in this metric. The default metric is called default and rules that have not explicitly specified metric would insert their results to this default metric.

Let's impress how this technics works:

 1. First of all when rspamd is running each module (lua, internal or external dynamic module) can register symbols in any defined metric. After this process rspamd has a cache of symbols for each metric. This cache can be saved to file for speeding up process of optimizing order of calling of symbols.

 2. Rspamd gets a message from client and parse it with mime parsing and do other parsing jobs like extracting text parts, urls, and stripping html tags.

 3. For each metric rspamd is looking to metric's cache and select rules to check according to their order (this order depends on frequence of symbol, its weight and execution time).

 4. Rspamd calls rules of metric till the sum weight of symbols in metric is less than its limit.

 5. If sum weight of symbols is more than limit the processing of rules is stopped and message is counted as spam in this metric.

After processing rules rspamd is also does statistic check of message. Rspamd statistic module is presented as a set of classifiers. Each classifier defines algorithm of statistic checks of messages. Also classifier definition contains definition of statistic files (or statfiles shortly). Each statfile contains of number of patterns that are extracted from messages. These patterns are put into statfiles during learning process. A short example: you define classifier that contains two statfiles: ham and spam. Than you find 10000 messages that are spam and 10000 messages that contains ham. Then you learn rspamd with these messages. After this process ham statfile contains patterns from ham messages and spam statfile contains patterns from spam messages. Then when you are checking message via this statfiles messages that are like spam would have more probability/weight in spam statfile than in ham statfile and classifier would insert symbol of spam statfile and would calculate how this message is like patterns that are contained in spam statfile. But rspamd is not limiting you to define one classifier or two statfiles. It is possible to define a number of classifiers and a number of statfiles inside a classifier. It can be useful for personal statistic or for specific spam patterns. Note that each classifier can insert only one symbol - a symbol of statfile with max weight/probability. Also note that statfiles check is allways done after all rules. So statistic can correct result of rules.

Now some words about modules. All rspamd rules are contained in modules. Modules can be internal (like SURBL, SPF, fuzzy check, email and others) and external written in lua language. In fact there is no differ in the way, how rules of these modules are called:

 1. Rspamd loads config and loads specified modules.

 2. Rspamd calls init function for each module passing configurations arguments.

 3. Each module examines configuration arguments and register its rules (or not register depending on configuration) in rspamd metrics (or in a single metric).

 4. During metrics process rspamd calls registered callbacks for module's rules.

 5. These rules may insert results to metric.

So there is no actual difference between lua and internal modules, each are just providing callbacks for processing messages. Also inside callback it is possible to change state of message's processing. For example this can be done when it is required to make DNS or other network request and to wait result. So modules can pause message's processing while waiting for some event. This is true for lua modules as well.

Rspamd config file structure.
=============================

Rspamd config file is placed in PREFIX/etc/rspamd.xml by default. You can specify other location by passing -c option to rspamd. Rspamd config file contains configuration parameters in XML format. XML was selected for rather simple manual editing config file and for simple automatic generation as well as for dynamic configuration. I've decided to move rules logic from XML file to keep it small and simple. So rules are defined in lua language and rspamd parameters are defined in xml file (rspamd.xml). Configuration rules are included by <lua> tag that have src attribute that defines relative path to lua file (relative to placement of rspamd.xml):

     <lua src="rspamd/lua/rspamd.lua">fake</lua>

Note that it is not currently possible to have empty tags. I hope this restriction would be fixed in future. Rspamd xml config consists of several sections:

  * Main section - section where main config parameters are placed.

  * Workers section - section where workers are described.

  * Classifiers section - section where you define your classify logic

  * Modules section - a set of sections that describes module's rules (in fact these rules should be in lua code)

  * Metrics section - a section where you can set weights of symbols in metrics and metrics settings

  * Logging section - a section that describes rspamd logging

  * Views section - a section that defines rspamd views

So common structure of rspamd.xml can be described this way:

     <? xml version="1.0" encoding="utf-8" ?>
     <rspamd>
      <!-- Main section directives -->
      ...
      <!-- Workers directives -->
      <worker>
       ...
      </worker>
      ...
      <!-- Classifiers directives -->
      <classifier>
       ...
      </classifier>
      ...
      <!-- Logging section -->
      <logging>
       <type>console</type>
       <level>info</level>
       ...
      </logging>
      <!-- Views section -->
      <view>
       ...
      </view>
      ...
      <!-- Modules settings -->
      <module name="regexp">
       <option name="test">test</option>
       ...
      </module>
      ...
     </rspamd>

Each of these sections would be described further in details.

Rspamd configuration atoms.
===========================

There are several primitive types of rspamd configuration parameters:

  * String - common string that defines option.

  * Number - integer or fractional number (e.g.: 10 or -1.5).

  * Time - ammount of time in milliseconds, may has suffixes:

       * s - for seconds (e.g. 10s);

       * m - for minutes (e.g. 10m);

       * h - for hours (e.g. 10h);

       * d - for days (e.g. 10d);

  * Size - like number numerci reprezentation of size, but may have a suffix:

       * k - 'kilo' - number * 1024 (e.g. 10k);

       * m - 'mega' - number * 1024 * 1024 (e.g. 10m);

       * g - 'giga' - number * 1024 * 1024 * 1024 (e.g. 1g);

     Size atoms are used for memory limits for example.

  * Lists - path to dynamic rspamd list (e.g. http://some.host/some/path).

While practically all atoms are rather trivial to understand rspamd lists may cause some confusion. Lists are widely used in rspamd for getting data that can be often changed for example white or black lists, lists of ip addresses, lists of domains. So for such purposes it is possible to use files that can be get either from local filesystem (e.g. file:///var/run/rspamd/whitelsist) or by HTTP (e.g. http://some.host/some/path/list.txt). Rspamd constantly looks for changes in this files, if using HTTP it also set If-Modified-Since header and check for Not modified reply. So it causes no overhead when lists are not modified and may allow to store huge lists and to distribute them over HTTP. Monitoring of lists is done with some random delay (jitter), so if you have many rspamd servers in cluster that are monitoring a single list they would come to check or download it in slightly different time. The two most common list formats are IP list and domains list. IP list contains of ip addresses in dot notation (e.g. 192.168.1.1) or ip/network pairs in CIDR notation (e.g. 172.16.0.0/16). Items in lists are separated by newline symbol. Lines that begin with # symbol are considered as comments and are ignored while parsing. Domains list is very like ip list with difference that it contains domain names.

Main rspamd configuration section.
==================================

Main rspamd configurtion section contains several definitions that determine main parameters of rspamd for example path to pidfile, temporary directory, lua includes, several limits e.t.c. Here is list of this directives explained:

Tag   Mean

<TEMPDIR>
Defines temporary directory for rspamd. Default is to use TEMP
environment variable or /tmp.

<PIDFILE>
Path to rspamd pidfile. Here would be stored a pid of main process.
Pidfile is used to manage rspamd from start scripts.

<STATFILE_POOL_SIZE>
Limit of statfile pool size: a total number of bytes that can be used for
mapping statistic files. Rspamd is using LRU system and would unmap the most
unused statfile when this limit would be reached. The common sense is to set
this variable equal to total size of all statfiles, but it can be less than this
in case of dynamic statfiles (for per-user statistic).

<FILTERS>
List of enabled internal filters. Items in this list can be separated by
spaces, semicolons or commas. If internal filter is not specified in this line
it would not be loaded or enabled.

<RAW_MODE>
Boolean flag that specify whether rspamd should try to convert all
messages to UTF8 or not. If RAW_MODE is enabled all messages are
processed as is and are not converted. Raw mode is faster than utf mode
but it may confuse statistics and regular expressions.

<LUA>
Defines path to lua file that should be loaded fro configuration. Path to
this file is defined in src attribute. Text inside tag is required but
is not parsed (this is stupid limitation of parser's design).

Rspamd logging configuration.
=============================

Rspamd has a number of logging variants. First of all there are three types of logs that are supported by rspamd: console loggging (just output log messages to console), file logging (output log messages to file) and logging via syslog. Also it is possible to filter logging to specific level:

  * error - log only critical errors

  * warning - log errors and warnings

  * info - log all non-debug messages

  * debug - log all including debug messages (huge amount of logging)

Also it is possible to turn on debug messages for specific ip addresses. This ability is usefull for testing.

For each logging type there are special mandatory parameters: log facility for syslog (read syslog (3) manual page for details about facilities), log file for file logging. Also file logging may be buffered for speeding up. For reducing logging noise rspamd detects for sequential identic log messages and replace them with total number of repeats:

     #81123(fuzzy): May 11 19:41:54 rspamd file_log_function: Last message repeated 155 times
     #81123(fuzzy): May 11 19:41:54 rspamd process_write_command: fuzzy hash was successfully added

Here is summary of logging parameters:

Tag   Mean
<TYPE>
Defines logging type (file, console or syslog). For each type mandatory
attriute must be present:

  * filename - path to log file for file logging type;

  * facility - syslog logging facility.

<LEVEL>
Defines loggging level (error, warning, info or debug).

<LOG_BUFFER>
For file and console logging defines buffer in bytes (kilo, mega or giga
bytes) that would be used for logging output.

<LOG_URLS>
Flag that defines whether all urls in message would be logged. Useful for
testing.

<DEBUG_IP>
List that contains ip addresses for which debugging would be turned on. For
more information about ip lists look at config atoms section.

Metrics configuration.
======================

Setting of rspamd metrics is the main way to change rules' weights. You can set up weights for all rules: for those that have static weights (for example simple regexp rules) and for those that have dynamic weights (for example statistic rules). In all cases the base weight of rule is multiplied by metric's weight value. For static rules base weight is usually 1.0. So we have:

  * w_symbol = w_static * factor - for static rules

  * w_symbol = w_dynamic * factor - for dynamic rules

Also there is an ability to add so called "grow factor" - additional multiplier that would be used when we have more than one symbol in metric. So for each added symbol this factor would increment its power. This can be written as: w_total = w_1 * gf ^ 0 + w_2 * gf ^ 1 + ... + w_n * gf ^ n - 1 Grow multiplier is used to increment weight of rules when message got many symbols (likely spammy). Note that only rules with positive weights would increase grow factor, those with negative weights would just be added. Also note that grow factor can be less than 1 but it is uncommon use (in this case we would have weight lowering when we have many symbols for this message). Metrics can be set up with config section(s) metric:

     <metric>
      <name>test_metric</name>
      <action>reject</action>
      <symbol weight="0.1">MIME_HTML_ONLY</symbol>
      <grow_factor>1.1</grow_factor>
     </metric>

Note that you basically need to add symbols to metric when you add additional rules. The decision of weight of newly added rule basically depends on its importance. For example you are absolutely sure that some rule would add a symbol on only spam messages, so you can increase weight of such rule so it would filter such spam. But if you increase weight of rules you should be more or less sure that it would not increase false positive errors rate to unacceptable level (false positive errors are errors when good mail is treated as spam). Rspamd comes with a set of default rules and default weights of that rules are placed in rspamd.xml.sample. In most cases it is reasonable to change them for your mail system, for example increase weights of some rules or decrease for others. Also note that default grow factor is 1.0 that means that weights of rules do not depend on count of added symbols. For some situations it useful to set grow factor to value more than 1.0. Also by modifying weights it is possible to manage static multiplier for dynamic rules.

Workers configuration.
======================

Workers are rspamd processes that are doing specific jobs. Now are supported 4 types of workers:

 1. Normal worker - a typical worker that process messages.

 2. Controller worker - a worker that manages rspamd, get statistics and do learning tasks.

 3. Fuzzy storage worker - a worker that contains a collection of fuzzy hashes.

 4. LMTP worker - experimental worker that acts as LMTP server.

These types of workers has some common parameters:

Parameter   Mean
<type>
Type of worker (normal, controller, lmtp or fuzzy)
<bind_socket>
Socket credits to bind this worker to. Inet and unix sockets are supported:

     <bind_socket>localhost:11333</bind_socket>
     <bind_socket>/var/run/rspamd.sock</bind_socket>

Also for inet sockets you may specify * as address to bind to all
available inet interfaces:

     <bind_socket>*:11333</bind_socket>

<count>
Number of worker processes of this type. By default this number is
equialent to number of logical processors in system.
<maxfiles>
Maximum number of file descriptors available to this worker process.
<maxcore>
Maximum size of core file that would be dumped in cause of critical errors
(in mega/kilo/giga bytes).

Also each of workers types can have specific parameters:

  * Normal worker:

       * <CUSTOM_FILTERS> - path to dynamically loaded plugins that would do real check of incoming messages. These modules are described further.

       * <MIME> - if this parameter is "no" than this worker assumes that incoming messages are in non-mime format (e.g. forum's messages) and standart mime headers are added to them.

  * Controller worker:

       * <PASSWORD> - a password that would be used to access to contorller's privilleged commands.

  * Fuzzy worker:

       * <HASHFILE> - a path to file where fuzzy hashes would be permamently stored.

       * <USE_JUDY> - if libJudy is present in system use it for faster storage.

       * <FREQUENT_SCORE> - if judy is not turned on use this score to place hashes with score that is more than this value to special faster list (this is designed to increase lookup speed for frequent hashes).

       * <EXPIRE> - time to expire of fuzzy hashes after their placement in storage.

These parameters can be set inside worker's definition:

     <worker>
       <type>fuzzy</type>
       <bind_socket>*:11335</bind_socket>
       <count>1</count>
       <maxfiles>2048</maxfiles>
       <maxcore>0</maxcore>
     <!-- Other params -->
         <param name="use_judy">yes</param>
         <param name="hashfile">/spool/rspamd/fuzzy.db</param>
         <param name="expire">10d</param>
     </worker>

The purpose of each worker's type would be described later. The main parameters that could be defined are bind sockets for workers, their count, password for controller's commands and parameters for fuzzy storage. Default config provides reasonable values of this parameters (except password of course), so for basic configuration you may just replace controller's password to more secure one.

Classifiers configuration.
==========================

Common classifiers options.
---------------------------

Each classifier has mandatory option TYPE that defines internal algorithm that is used for classifying. Currently only winnow is supported. You can read theoretical description of algorithm used here: http://www.siefkes.net/papers/winnow-spam.pdf

The common classifier configuration consists of base classifier parameters and definitions of two (or more than two) statfiles. During classify process rspamd check each statfile in classifier and select those that has more probability/weight than others. If all statfiles has zero weight this classifier do not add any symbols. Among common classifiers options are:

Tag   Mean
<TOKENIZER>
Tokenizer to extract tokens from messages. Currently only osb
tokenizer is supported
<METRIC>
Metric to which this classifier would insert symbol.

Also option MIN_TOKENS is supported to specify minimum number of tokens to work with (this is usefull to avoid classifying of short messages as statistic is practically useless for small amount of tokens). Here is example of base classifier config:

     <classifier type="winnow">
      <tokenizer>osb-text</tokenizer>
      <metric>default</metric>
      <option name="min_tokens">20</option>
      <statfile>
       ...
      </statfile>
     </classifier>

Statfiles options.
------------------

The most common statfile options are SYMBOL and SIZE. The first one defines which symbol would be inserted if this statfile would have maximal weight inside classifier and size defines statfile size on disk and in memory. Note that statfiles are mapped directly to memory and you should practically note parameter STATFILE_POOL_SIZE of main section which defines maximum ammount of memory for mapping statistic files. Also note that statistic files are of constant size: if you defines 100 megabytes statfile it would occupy 100 megabytes of disc space and 100 megabytes of memory when it is used (mapped). Each statfile is indexed by tokens and contains so called "token chains". This mechanizm would be described further but note that each statfile has parameter "free tokens" that defines how much space is available for new tokens. If statfile has no free space the most unused tokens would be removed from statfile.

Here is list of common options of statfiles:

Tag   Mean
<SYMBOL>
Defines symbol to insert for this statfile.
<SIZE>
Size of this statfile in bytes (kilo/mega/giga bytes).
<PATH>
Filesystem path to statistic file.
<NORMALIZER>
Defines weight normalization structure. Can be lua function name or
internal normalizer. Internal normalizer is defined in format:
"internal:<max_weight>" where max_weight is fractional number that limits the
maximum weight of this statfile's symbol (this is so called dynamic weight).
<BINLOG>
Defines binlog affinity: master or slave. This option is used for statfiles
binary sync that would be described further.
<BINLOG_MASTER>
Defines credits of binlog master for this statfile.
<BINLOG_ROTATE>
Defines rotate time for binlog.

Internal normalization of statfile weight works in this way:

  * R_score = 1 when W_statfile < 1

  * R_score = W_statfile ^ 2 when 1 < W_statfile < max / 2

  * R_score = W_statfile when max / 2 < W_statfile < max

  * R_score = max when W_statfile > max

The final result weight would be: weight = R_score * W_weight. Here is sample classifier configuration with two statfiles that can be used for spam/ham classifying:

        <symbol weight="-1.00">WINNOW_HAM</symbol>
        <symbol weight="1.00">WINNOW_SPAM</symbol>
     ...

     <!-- Classifiers section -->
     <classifier type="winnow">
      <tokenizer>osb-text</tokenizer>
      <metric>default</metric>
      <option name="min_tokens">20</option>
      <statfile>
       <symbol>WINNOW_HAM</symbol>
       <size>100M</size>
       <path>/var/run/rspamd/data.ham</path>
       <normalizer>internal:3</normalizer>
      </statfile>
      <statfile>
       <symbol>WINNOW_SPAM</symbol>
       <size>100M</size>
       <path>/var/run/rspamd/data.spam</path>
       <normalizer>internal:3</normalizer>
      </statfile>
     </classifier>
     <!-- End of classifiers section -->

In this sample we define classifier that contains two statfiles: WINNOW_SPAM and WINNOW_HAM. Each statfile has 100 megabytes size (so they would occupy 200Mb while classifying). Also each statfile has maximum weight of 3 so with such weights (-1 for WINNOW_HAM and 1 for WINNOW_SPAM) the result weight of symbols would be 0..3 for WINNOW_SPAM and 0..-3 for WINNOW_HAM.

Modules config.
===============

Lua modules loading.
--------------------

For loading custom lua modules you should use <modules> section:

     <modules>
      <module>/usr/local/etc/rspamd/plugins/lua</module>
     </modules>

Each <module> directive defines path to lua modules. If this is a directory so all *.lua files inside that directory would be loaded. If this is a file it would be loaded directly.

Modules configuration.
----------------------

Each module can have its own config section (this is true not only for internal module but also for lua modules). Such section is called <module> with mandatory attribute "name". Each module can be configured by <option> directives. These directives must also have "name" attribute. So module configuration is done in param = value style:

     <module name="fuzzy_check">
       <option name="servers">localhost:11335</option>
       <option name="symbol">R_FUZZY</option>
       <option name="min_length">300</option>
       <option name="max_score">10</option>
     </module>

The common parameters are:

  * symbol - symbol that this module should insert.

But each module can have its own unique parameters. So it would be discussed furhter in detailed modules description. Also note that for internal modules you should edit <filters> parameter in main section: this parameter defines which internal modules would be turned on in this configuration.

Views config.
=============

It is possible to make different rules for different networks/senders/recipients. For this purposes you can use rspamd views: maps of conditions (ip, sender, recipients) and actions, associated with them. For example you can turn rspamd off for specific conditions by using skip_check action or check only specific rules. Views are defined inside <view> xml section. Here is list of available tags inside section:

Tag   Mean
<SKIP_CHECK>
Boolean flag (yes or no) that specifies whether rspamd checks should be
turned off for this ip
<SYMBOLS>
Defines comma-separated list of symbols that should be checked for this
view
<IP>
Map argument that defines path to list of ip addresses (may be with CIDR
masks) to which this view should be applied.
<CLIENT_IP>
Map argument that defines path to list of ip addresses of rspamd clients
to which this view should be applied. Note that this is ip of rspamd client not
ip of message's sender.
<FROM>
Map argument that defines path to list of senders to which this view should
be applied.

Here is an example view definition

     <view>
      <skip_check>yes</skip_check>
      <ip>file:///usr/local/etc/rspamd/whitelist</ip>
     </view>

Rspamd clients interaction.
***************************

Introduction.
=============

After you have basic config file you may test rspamd functionality by using whether telnet like utility or rspamc client. For testing newly installed config it is possible to run config file test:

     $ rspamd -t
     syntax OK

Rspamc utility is written in perl language and uses perl modules that are shipped with rspamd: Mail::Rspamd::Client for client's protocol and Mail::Rspamd::Config for reading and writing configuration. The documentation for these modules can be found by commands:

     $ perldoc Mail::Rspamd::Client
     $ perldoc Mail::Rspamd::Config

So other way to access rspamd is to use perl client API:

     use Mail::Rspamd::Client;
     my $config = {
     	hosts => ['localhost:11333'],
     };

     my $client = new Mail::Rspamd::Client(%config);

     if (! $client->ping()) {
     	die "Cannot ping rspamd: $client->{error}";
     }

     my $result = $client->check($testmsg);

     if ($result->{'default'}->{isspam} eq 'True') {
     	# do something with spam message here
     }

Rspamc protocol.
================

Rspamc protocol is an extension over traditional spamc protocol that is used by spamassassin. This protocol looks like traditional HTTP session: first line is method with version, headers can be passed by next lines and the message itself is waited after empty line:

     <REQUEST>
     SYMBOLS RSPAMC/1.1
     Content-Length: 2200

     <message octets>

     <REPLY>
     RSPAMD/1.1 0 OK
     Metric: default; True; 10.40 / 10.00 / 0.00
     Symbol: R_UNDISC_RCPT
     Symbol: ONCE_RECEIVED
     Symbol: R_MISSING_CHARSET
     Urls:

The format of method line can be presented as:

     <COMMAND> RSPAMC/<version>

Version can be 1.0 and 1.1. The main difference that in 1.1 metrics output also has reject score - hard limit of score for metric. This would be discussed while describing user's options. Commands are:

Command   Mean
CHECK
Check a message and output results for each metric. But do not output
symbols.
SYMBOLS
Same as CHECK but output symbols.
PROCESS
Same as SYMBOLS but output also original message with inserted
X-Spam headers.
PING
Do not do any processing, just check rspamd state:

     $ telnet localhost 11333
     Trying 127.0.0.1...
     Connected to localhost.
     Escape character is '^]'.
     PING RSPAMC/1.1

     RSPAMD/1.1 0 PONG
     Connection closed by foreign host.

After command there should be one mandatory header: Content-Length that defines message's length in bytes and optional headers:

Header   Mean
DELIVER-TO:
Defines actual delivery recipient of message. Can be used for personalized
statistic and for user specific options.
IP:
Defines IP from which this message is received.
HELO:
Defines SMTP helo.
FROM:
Defines SMTP mail from command data.
QUEUE-ID:
Defines SMTP queue id for message (can be used instead of message id in
logging).
RCPT:
Defines SMTP recipient (it may be several Rcpt: headers).
PASS:
If this header has "all" value, all filters would be checked for
this message.
SUBJECT:
Defines subject of message (is used for non-mime messages).
USER:
Defines SMTP user (this is currently unused in rspamd however).

So rspamc protocol allows to pass many data from MTA to rspamd. This is used to increase speed of processing and for building filters (like SPF filter). Also note that rspamd support spamassassin spamc protocol and you can even pass rspamc headers in spamc mode, but reply of rspamd in spamc mode would be much shorter: it would only use "default" metric and won't show additional options for symbols. Rspamc reply looks like this:

     RSPAMD/1.1 0 OK
     Metric: default; True; 10.40 / 10.00 / 0.00
     Symbol: R_UNDISC_RCPT
     Symbol: ONCE_RECEIVED
     Symbol: R_MISSING_CHARSET
     Urls:

First line is method reply: <PROTOCOL>/<VERSION> <ERROR_CODE> <ERROR_REPLY>. Error code is 0 when no error occured. After first reply line there are metrics output. For SYMBOLS and PROCESS commands there are symbols lines after each metric. And for PROCESS command there would be original message after all metrics results. Metric result line looks like this:

     Metric: <name>; <result>; <score> / <required_score> / <reject_score>

For 1.0 version of rspamc protocol reject_score parameter is not printed. Symbol line looks like this:

     Symbol: <Name>[; param1[, param2...]]

Some symbols can have parameters attached. It is useful for example for RBL checks (you can insert additional data after symbol name), for statistic and fuzzy checks. Also rspamd inserts Urls line in which all urls that are contained in message are printed in comma-separated list. Note that this protocol is used for normal workers. Controller, fuzzy storage and lmtp/smtp workers are using other protocols. For example controller's protocol is oriented on interactive sessions: you can pass many commands to controller before disconnecting. Fuzzy storage is using UDP for making interaction with storage faster. LMTP/SMTP workers are using lmtp and smtp protocols. All of these protocols would be described in further chapters about rspamd workers.

Controller protocol.
====================

Rspamd controller can also be accessed by telnet, by rspamc client or by using perl module Mail::Rspamd::Client. Controller protocol accepts commands and it is possible to send several commands during a single session. Here is an example telnet session:

     >telnet localhost 11334
     Trying 127.0.0.1...
     Connected to localhost.
     Escape character is '^]'.
     Rspamd version 0.3.0 is running on spam1.rambler.ru
     stat
     Messages scanned: 1526901
     Messages treated as spam: 238171, 15.60%
     Messages treated as ham: 1288730, 84.40%
     Messages learned: 0
     Connections count: 1529758
     Control connections count: 15
     Pools allocated: 3059589
     Pools freed: 3056134
     Bytes allocated: 98545852799
     Memory chunks allocated: 8745374
     Shared chunks allocated: 7
     Chunks freed: 8737507
     Oversized chunks: 768784
     Fuzzy hashes stored: 0
     Fuzzy hashes expired: 0
     Statfile: WINNOW_SPAM (version 186); length: 100.0 MB; free blocks: 748504; total blocks: 6553581; free: 11.42%
     Statfile: WINNOW_HAM (version 186); length: 100.0 MB; free blocks: 748504; total blocks: 6553581; free: 11.42%
     END

So you can see that reply from controller is ended with line that contains word END. It is also possible to get summary help for controller's commands:

     help
     Rspamd CLI commands (* - privilleged command):
         help - this help message
     (*) learn <statfile> <size> [-r recipient] [-m multiplier] [-f from] [-n] - learn message to specified statfile
         quit - quit CLI session
     (*) reload - reload rspamd
     (*) shutdown - shutdown rspamd
         stat - show different rspamd stat
         counters - show rspamd counters
         uptime - rspamd uptime
     END

Note that some commands are privilleged ones - you are required to enter a password for them:

     >telnet localhost 11334
     Trying 127.0.0.1...
     Connected to localhost.
     Escape character is '^]'.
     Rspamd version 0.3.0 is running on spam1.rambler.ru
     reload
     not authorized
     END

     password q1
     password accepted
     END

     reload
     reload request sent
     END
     Connection closed by foreign host.

This password is configured in rspamd.xml in worker section where you are describing controller:

     <worker>
       <type>controller</type>
       ...
     <!-- Other params -->
         <param name="password">q1</param>
     </worker>

In many cases it is more easy to use rspamc to access controller. Here is example of learning statfiles using rspamc CLI:

     % rspamc -h localhost:11334 -P q1 -s WINNOW_HAM learn < /tmp/exim.eml
     Results for host localhost:11334:

     Learn succeed. Sum weight: 1.51

     % rspamc -h localhost:11334 -P q1 -s WINNOW_SPAM learn < /tmp/bad.eml
     Results for host localhost:11334:

     Learn succeed. Sum weight: 1.51

Note that rspamc handles password issues and other things like timeouts and error handling inside and makes this tasks rather easy.

More about rspamc client.
=========================

Rspamc is small and simple client that allows to simplify common tasks for rspamd manage. Rspamc is written in perl language and requires some modules for its work:

  * Mail::Rspamd::Client - a module that contains common function for accessing rspamd, shipped with rspamd and installed automatically

  * Term::Cap - a module that allows basic interaction with terminal, can be obtained via cpan (http://www.cpan.org).

Rspamc accepts several command line options:

     % rspamc --help
     Usage: rspamc.pl [-h host] [-H hosts_list] [-P password] [-c conf_file] [-s statfile] [-d user@domain] [command] [path]
     -h         host to connect (in format host:port) or unix socket path
     -H         path to file that contains list of hosts
     -P         define control password
     -c         config file to parse
     -s         statfile to use for learn commands

     Additional options:
     -d         define deliver-to header
     -w         define weight for fuzzy operations
     -S         define search string for IMAP operations
     -i         emulate that message was send from specified IP
     -p         pass message throught all filters

     Notes:
     imap format: imap:user:<username>:password:[<password>]:host:<hostname>:mbox:<mboxname>
     Password may be omitted and then it would be asked in terminal
     imaps requires IO::Socket::SSL

     IMAP search strings samples:
     ALL - All messages in the mailbox;
     FROM <string> - Messages that contain the specified string in the envelope structure's FROM field;
     HEADER <field-name> <string> - Messages that have a header with the specified field-name and that
                  contains the specified string in the text of the header (what comes after the colon);
     NEW - Messages that have the Recent flag set but not the Seen flag.
                  This is functionally equivalent to "(RECENT UNSEEN)".
     OLD - Messages that do not have the Recent flag set.
     SEEN - Messages that have the Seen flag set.
     SENTBEFORE <date> - Messages whose [RFC-2822] Date: header (disregarding time and timezone)
                  is earlier than the specified date.
     TO <string> - Messages that contain the specified string in the envelope structure's TO field.
     TEXT <string> - Messages that contain the specified string in the header or body of the message.
     OR <search-key1> <search-key2> - Messages that match either search key (same for AND and NOT operations).

     Version:   0.3.0

After options you should specify command to execute, for example:

     % rspamc symbols < /tmp/exim.eml

After command name you may specify objects to apply to: files, directories or even imap folders:

  * A single file:

          % rspamc symbols /tmp/exim.eml

  * A list of files:

          % rspamc symbols /tmp/*.eml

  * Directories:

          % rspamc symbols /tmp/*.eml /tmp/to_scan/

  * IMAP folder:

          % rspamc symbols imap:user:username:password::host:localhost:mbox:INBOX
          Enter IMAP password:

     Note that it is possible to specify empty password and be prompted for a password during execution (you also need perl module Term::ReadKey for turning on noecho input of password).

For fetching imap messages you may also use search string by specifying -S option. Some examples of IMAP search strings can be found in a help message. For more complex things you may read rfc3501 about imap4 search strings. This may be found for example here: http://www.faqs.org/rfcs/rfc3501.html. IMAP access may be usefull for setting up automatic learning scripts. Also it is possible to use SSL version of imap by specifying imaps instead imap as first component. Note that for SSL access you need IO::Socket::SSL perl module.

Statistics and hashes storage.
******************************

Introduction.
=============

First of all we need to strictly define purposes of hashes and statistic. Hashes are used to find very close messages (for example messages where there are only several words changed), while statistic can find probability of belonging message to specified class of messages. So when you learn rspamd with message's hash you just add this hash to storage and when you learn rspamd statistic you add tokens from message to specified class. So statistic is probabilistic method to filter message, while fuzzy hashes can detect specific patterns in messages and filter them.

Classifiers and statistic.
==========================

Tokenization.
-------------

Now rspamd supports OSB-Winnow statistic algorithm. Let's describe it in details. First of all message is separeted into a set of tokens. The algorithm of extracting tokens is rather simple now:

 1. Extract graph symbols till first non-graph symbol (whitespace, punctuation etc), the group of graph symbols forms a token, non-graphs are separators.

 2. Fill an array with token till window size is reached (currently this size is 5 tokens).

 3. Get pairs of tokens from array and extract their hashes:

       * * . . . * -> token1 (h1, h5);

       * . * . . * -> token2 (h2, h5);

       * . . * . * -> token3 (h3, h5);

       * . . . * * -> token4 (h4, h5);

 4. Insert these tokens to statfile (indexed by first hash).

 5. Shift window on next word.

So after tokenizing process we would have tokens each of that contains 2 hashes of 2 words from message. This mechanics allows to count not only words itself but also its combinations into a message, so providing more accurate statistic.

Classifying.
------------

For classifying process winnow algorithm is used. In this statistic algtorithm we operate not with probabilities but with weights. Each token has its own weight and when we learn some statfile with tokens rspamd does several things:

 1. Try to find token inside statfile.

 2. If a token found multiply its weight by so called promotion factor (that is now 1.23).

 3. If token not found insert it into statfile with weight 1.

If it is needed to lower token weight, so its weight is multiplied with demotion factor (currently 0.83). Classify process is even more simple:

 1. Extract tokens from a message.

 2. For each statfile check weight of obtained tokens and store summary weight.

 3. Compare sums for each statfile and select statfile with the most big sum.

 4. Do weight normalization and insert symbol of selected statfile.

Statfiles synchronization.
--------------------------

Rspamd allows to make master/slave statfiles synchronization. This is done by writing changes to statfiles to special binary log. Binary log is a file on filesystem named like statfile but with .binlog suffix. Binary log consist of two level indexes and binary changes to each statfile. So after each learning process the version of affected statfiles is increased by 1 and a record is written to binary log. Binary logs have fixed size limit and may have time limit (rotate time). The process of synchronization may be described as:

 1. Slave rspamd periodically asks master for version of statfiles monitored.

 2. If master has version that is larger than slave's one the synchronization process starts.

 3. During synchronization process master looks at version reported by client in binary log.

 4. If version is found all records that are after client's version are sent to client.

 5. Client accepts changes and apply binary patches one-by-one incrementing statfile's version.

 6. If version that client reports is not found in binary log the completely statfile is sent to client (slow way, but practically that would take place only once for fresh slaves).

Here is example configuration for master statfile:

      <statfile>
       <symbol>WINNOW_HAM</symbol>
       <size>100M</size>
       <path>/spool/rspamd/data.ham</path>
       <normalizer>internal:3</normalizer>
       <binlog>master</binlog>
       <binlog_rotate>1d</binlog_rotate>
      </statfile>

Here we define binlog affinity (master) that automatically create binlog file /spool/rspamd/data.ham.binlog and set up time limit for it (1 day). For slaves you should first of all set up controller worker to accept network connections (statfile synchronization is done via controller workers). The second task is to define affinity for slave and master's address:

      <statfile>
       <symbol>WINNOW_HAM</symbol>
       <size>100M</size>
       <path>/spool/rspamd/data.ham</path>
       <normalizer>internal:3</normalizer>
       <binlog>slave</binlog>
       <binlog_master>spam10:11334</binlog_master>
      </statfile>

Conclusion.
-----------

Statfiles synchronization allows to set up rspamd cluster that uses the common statfiles and easily learn the whole cluster without unnecessary overhead.

Hashes and hash storage.
========================

Fuzzy hashes.
-------------

Hashes that are used in rspamd for messages are not cryptoghraphic. Instead of them fuzzy hashes are used. Fuzzy hashes is technics that allows to obtain common hashes for common messages (for cryptographic hashes you usually get very different hashes even if input messages are very common but not identical). The main principle of fuzzy hashing is to break up text parts of message into small pieces (blocks) and calculate hash for each block using so called rolling hash. After this process the final hash is forming by setting bytes in it from blocks. So if we have 2 messages each of that contains 100 blocks and 99 of them are identical we would have 2 hashes that differs only in one byte. So we can consider that one message is 99% like other message.

Fuzzy storage.
--------------

In rspamd hashes can be stored in fuzzy storage. Fuzzy storage is a special worker that can store hashes and reply about score of hashes. Inside fuzzy storage each hash has its own weight and list number. List number is integer that specify to which list this hash is related. This number can be used in fuzzy_check plugin inside rspamd to add custom symbol. There are two ways of storing fuzzy hashes: store them in a set of linear linked lists and storing hashes in very fast judy tree. First way is good for a relatively small number of fuzzy hashes. Also in this case fuzzy match is used, so you can find not only identical hashes but also common hashes. But for large number of hashes this method is very slow. The second way requires libJudy in system (can be found at http://judy.sourceforge.net) and turns off fuzzy matching - only identical hashes would be found. On the other hand you may store millions of hashes in judy tree not loosing nor memory, nor CPU.

Conclusion.
-----------

Fuzzy hashes is efficient way to make up different black or white lists. Fuzzy storage can be distributed over several machines (if you specify several storage servers rspamd would select upstream by hash of fuzzy hash). Also storage can contain several lists identified by number. Each hash has its own weight that allows to set up dynamic rules that add different score from different hashes.

Rspamd modules.
***************

Introduction.
=============

This chapter describes modules that are shipped with rspamd. Here you can find details about modules configuration, principles of working, tricks to make spam filtering effective. First sections describe internal modules written in C: regexp (regular expressions), surbl (black list for URLs), fuzzy_check (checks for fuzzy hashes), chartable (check for character sets in messages) and emails (check for blacklisted email addresses in messages). Modules configuration can be done in lua or in config file itself.

Lua configuration.
------------------

You may use lua for setting configuration options for modules. With lua you can write rather complex rules that can contain not only text lines, but also some lua functions that would be called while processing messages. For loading lua configuration you should add line to rspamd.xml:

     <lua src="/usr/local/etc/rspamd/lua/my.lua">fake</lua>

It is possible to load several scripts this way. Inside lua file there would be defined global table with name CONFIG. This table should contain configuration options for modules indexed by module. This can be written this way:

     config['module_name'] = {}
     local mconfig = config['module_name']

     mconfig['option_name'] = 'option value'

     local a = 'aa'
     local b = 'bb'

     mconfig['other_option'] = string.format('%s, %s', a, b)

In this simple example we defines new element of table that is associated with module named 'module_name'. Then we assign to it an empty table ({}) and associate local variable mconfig. Then we set some elements of this table, that is equialent to setting module options like that:

     option_name = option_value
     other_option = aa, bb

Also you may assign to elements of modules tables some functions. That functions should accept one argument - worker task object and return result specific for that option: number, string, boolean. This can be shown on this simple example:

     local function test (task)
     	if task:get_ip() == '127.0.0.1' then
     		return 1
     	else
     		return 0
     	end
     end

     mconfig['some_option'] = test

In this example we assign to module option 'some_option' a function that check for message's ip and return 1 if that ip is '127.0.0.1'.

So using lua for configuration can help for making complex rules and for structuring rules - you can place options for specific modules to specific files and use lua function dofile for loading them (or add other <lua> tag to rspamd.xml).

XML configuration.
------------------

Options for rspamd modules can be set up from xml file too. This can be used for simple and/or temporary rules and should not be used for complex rules as this would make xml file too hard to read and edit. Thought it is surely possible but not recommended from points of config file understanding. Here is a simple example of module config options:

     <module name="module_name">
      <option name="option_name">option_value</option>
      <option name="other_option">aa, bb</option>
     </module>

Note that you need to encode xml entitles like & - &amp; and so on. Also only utf8 encoding is allowed. In sample rspamd configuration all modules except regexp module are configured via xml as they have only settings and regexp module has rules that are sometimes rather complex.

Regexp module.
==============

Introduction.
-------------

Regexp module is one of the most important rspamd modules. Regexp module can load regular expressions and filter messages according to them. Also it is possible to use logical expressions of regexps to create complex rules of filtering. It is allowed to use logical operators:

  * & - logical AND function

  * | - logical OR function

  * ! - logical NOT function

Also it is possible to use brackets for making priorities in expressions. Regexp module operates with regexp items that can be combined with logical operators into logical regexp expresions. Each expression is associated with its symbol and if it evaluates to true with this message the symbol would be inserted. Note that rspamd uses internal optimization of logical expressions (for example if we have expression 'rule1 & rule2' rule2 would not be evaluated if rule1 is false) and internal regexp cache (so if rule1 and rule2 have common items they would be evaluated only once). So if you need speed optimization of your rules you should take this fact into consideration.

Regular expressions.
--------------------

Rspamd uses perl compatible regular expressions. You may read about perl regular expression syntax here: http://perldoc.perl.org/perlre.html. In rspamd regular expressions must be enclosed in slashes:

     /^\\d+$/

If '/' symbol must be placed into regular expression it should be escaped:

     /^\\/\\w+$/

After last slash it is possible to place regular expression modificators:

Modificator   Mean
i   Ignore case for this expression.
m   Assume this expression as multiline.
s   Assume . as all characters including newline
characters (should be used with m flag).
x   Assume this expression as extended regexp.
u   Performs ungreedy matches.
o   Optimize regular expression.
r   Assume this expression as raw (this is actual for
utf8 mode of rspamd).
H   Search expression in message's headers.
X   Search expression in raw message's headers (without mime
decoding).
M   Search expression in the whole message (must be used
carefully as the whole message would be checked with this expression).
P   Search expression in all text parts.
U   Search expression in all urls.

You can combine flags with each other:

     /^some text$/iP

All regexp must be with type: H, X, M, P or U as rspamd should know where to search for specified pattern. Header regexps (H and X) have special syntax if you need to check specific header, for example From header:

     From=/^evil.*$/Hi

If header name is not specified all headers would be matched. Raw headers is matching is usefull for searching for mime specific headers like MIME-Version. The problem is that gmime that is used for mime parsing adds some headers implicitly, for example MIME-Version and you should match them using raw headers. Also if header's value is encoded (base64 or quoted-printable encoding) you can search for decoded version using H modificator and for raw using X modificator. This is usefull for finding bad encodings types or for unnecessary encoding.

Internal function.
------------------

Rspamd provides several internal functions for simplifying message processing. You can use internal function as items in logical expressions as they like regular expressions return logical value (true or false). Here is list of internal functions with their arguments:

Function   Arguments   Description
header_exists
header name
Returns true if specified header exists.

compare_parts_distance
number
If message has two parts (text/plain and text/html) compare how much they
differs (html messages are compared with stripped tags). The difference is
number in percents (0 is identically parts and 100 is totally different parts).
So if difference is more than number this function returns true.

compare_transfer_encoding
string
Compares header Content-Transfer-Encoding with specified string.

content_type_compare_param
param_name, param_value
Compares specified parameter of Content-Type header with regexp or certain
string:

     content_type_compare_param(Charset, /windows-\d+/)
     content_type_compare_param(Charset, ascii)

content_type_has_param
param_name
Returns true if content-type has specified parameter.

content_type_is_subtype
subtype_name
Return true if content-type is of specified subtype (for example for
text/plain subtype is 'plain').

content_type_is_type
type_name
Return true if content-type is of specified type (for example for
text/plain subtype is 'text'):

     content_type_is_type(text)
     content_type_is_subtype(/?.html/)

regexp_match_number
number,[regexps list]
Returns true if specified number of regexps matches for this message. This
can be used for making rules when you do not know which regexps should match but
if 2 of them matches the symbol shoul be inserted. For example:

     regexp_match_number(2, /^some evil text.*$/Pi, From=/^hacker.*$/H, header_exists(Subject))

has_only_html_part
nothing
Returns true when message has only HTML part

compare_recipients_distance
number
Like compare_parts_distance calculate difference between recipients. Number
is used as minimum percent of difference. Note that this function would check
distance only when there are more than 5 recipients in message.

is_recipients_sorted
nothing
Returns true if recipients list is sorted. This function would also works
for more than 5 recipients.

is_html_balanced
nothing
Returns true when all HTML tags in message are balanced.

has_html_tag
tag_name
Returns true if tag 'tag_name' exists in message.

check_smtp_data
item, regexp
Returns true if specified part of smtp dialog matches specified regexp. Can
check HELO, FROM and RCPT items.

These internal functions can be easily implemented in lua but I've decided to make them built-in as they are widely used in our rules. In fact this list may be extended in future.

Dynamic rules.
--------------

Rspamd regexp module can use dynamic rules that can be written in json syntax. Dynamic rules are loaded at runtime and can be modified while rspamd is working. Also it is possible to turn dynamic rules for specific networks only and add rules that does not contain any regexp (this can be usefull for dynamic lists for example). Dynamic rules can be obtained like any other dynamic map via file monitoring or via http. Here are examples of dynamic rules definitions:

     <module name="regexp">
       <option name="dynamic_rules">file:///tmp/rules.json</option>
     </module>

or for http map:

     <module name="regexp">
       <option name="dynamic_rules">http://somehost/rules.json</option>
     </module>

Rules are presented as json array (in brackets '[]'). Each rule is json object. This object can have several properties (properties with * are required):

Property   Mean
symbol(*)
Symbol for rule.
factor(*)
Factor for rule.
rule
Rule itself (regexp expression).
enabled
Boolean flag that define whether this rule is enabled (rule is enabled if
this flag is not present by default).
networks
Json array of networks (in CIDR format, also it is possible to add negation
by prepending ! symbol before item.

Here is an example of dynamic rule:

     [
     	{
     		"rule": "/test/rP",
     		"symbol": "R_TMP_1",
     		"factor": 1.1,
     		"networks": ["!192.168.1.0/24", "172.16.0.0/16"],
     		"enabled": false
     	}
     ]

Note that dynamic rules are constantly monitored for changes and are reloaded completely when modification is detected. If you change dynamic rules they would be reloaded in a minute and would be applied for new messages.

Conclusion.
-----------

Rspamd regexp module is powerfull tool for matching different patterns in messages. You may use logical expressions of regexps and internal rspamd functions to make rules. Rspamd is shipped with many rules for regexp module (most of them are taken from spamassassin rules as rspamd originally was a replacement of spamassassin) so you can look at them in ETCDIR/rspamd/lua/regexp directory. There are many built-in rules with detailed comments. Also note that if you add logical rule into XML file you need to escape all XML entitles (like & operators). When you make complex rules from many parts do not forget to add brackets for parts inside expression as you would not predict order of checks otherwise. Rspamd regexp module has internal logical optimization and regexp cache, so you may use identical regexp many times - they would be matched only once. And in logical expression you may optimize performance by putting likely TRUE regexp first in OR expression and likely FALSE expression first in AND expression. A number of internal functions can simplify complex expressions and for making common filters. Lua functions can be added in rules as well (they should return boolean value).

SURBL module.
=============

Surbl module is designed for checking urls via blacklists. You may read about surbls at http://www.surbl.org. Here is the sequence of operations that is done by surbl module:

 1. Extract all urls in message and get domains for each url.

 2. Check to special list called '2tld' and extract 3 components for domains from that list and 2 components for domains that are not listed:

          http://virtual.somehost.domain.com/some_path
          -> somehost.domain.com if domain.com is in 2tld list
          -> domain.com if not in 2tld

 3. Remove duplicates from domain lists

 4. For each registered surbl do dns request in form domain.surbl_name

 5. Get result and insert symbol if that name resolves

 6. It is possible to examine bits in returned IP address and insert different symbol for each bit that is turned on in result.

All DNS requests are done asynchronously so you may not bother about blocking. SURBL module has several configuration options:

  * metric - metric to insert symbol to.

  * 2tld - list argument of domains for those 3 components of domain name would be extracted.

  * max_urls - maximum number of urls to check.

  * whitelist - map of domains for which surbl checks would not be performed.

  * suffix - a name of surbl. It is possible to add several suffixes:

          suffix_RAMBLER_URIBL = insecure-bl.rambler.ru
          or in xml:
           <param name="suffix_RAMBLER_URIBL">insecure-bl.rambler.ru</param>

     It is possible to add %b to symbol name for checking specific bits:

          suffix_%b_SURBL_MULTI = multi.surbl.org
          then you may define replaces for %b in symbol name for each bit in result:
          bit_2 = SC -> sc.surbl.org
          bit_4 = WS -> ws.surbl.org
          bit_8 = PH -> ph.surbl.org
          bit_16 = OB -> ob.surbl.org
          bit_32 = AB -> ab.surbl.org
          bit_64 = JP -> jp.surbl.org

     So we make one DNS request and check for specific list by checking bits in result ip. This is described in surbl page: http://www.surbl.org/lists.html#multi. Note that result symbol would NOT contain %b as it would be replaced by bit name. Also if several bits are set several corresponding symbols would be added.

Also surbl module can use redirector - a special daemon that can check for redirects. It uses HTTP/1.0 for requests and accepts a url and returns resolved result. Redirector is shipped with rspamd but not enabled by default. You may enable it on stage of configuring but note that it requires many perl modules for its work. Rspamd redirector is described in details further. Here are surbl options for working with redirector:

  * redirector: adress of redirector (in format host:port)

  * redirector_connect_timeout (seconds): redirector connect timeout (default: 1s)

  * redirector_read_timeout (seconds): timeout for reading data (default: 5s)

  * redirector_hosts_map (map string): map that contains domains to check with redirector

So surbl module is an easy to use way to check message's urls and it may be used in every configuration as it filters rather big ammount of email spam and scam.

SPF module.
===========

SPF module is designed to make checks of spf records of sender's domains. SPF records are placed in TXT DNS items for domains that have enabled spf. You may read about SPF at http://en.wikipedia.org/wiki/Sender_Policy_Framework. There are 3 results of spf check for domain:

  * ALLOW - this ip is allowed to send messages for this domain

  * FAIL - this ip is not allowed to send messages for this domain

  * SOFTFAIL - it is unknown whether this ip is allowed to send mail for this domain

SPF supports different mechanizms for checking: dns subrequests, macroses, includes, blacklists. Rspamd supports the most of them. Also for security reasons there is internal limits for DNS subrequests and inclusions recursion. SPF module support very small ammount of options:

  * metric (string): metric to insert symbol (default: 'default')

  * symbol_allow (string): symbol to insert (default: 'R_SPF_ALLOW')

  * symbol_fail (string): symbol to insert (default: 'R_SPF_FAIL')

  * symbol_softfail (string): symbol to insert (default: 'R_SPF_SOFTFAIL')

Chartable module.
=================

Chartable is a simple module that detects different charsets in a message. This module is aimed to protect from emails that contains symbols from different character sets that looks like each other. Chartable module works differently for raw and utf modes: in utf modes it detects different characters from unicode tables and in raw modes only ASCII and non-ASCII symbols. Configuration of whis module is very simple:

  * metric (string): metric to insert symbol (default: 'default')

  * symbol (string): symbol to insert (default: 'R_BAD_CHARSET')

  * threshold (double): value that would be used as threshold in expression N_charset-changes / N_chars (e.g. if threshold is 0.1 than charset change should occure more often than in 10 symbols), default: 0.1

Fuzzy check module.
===================

Fuzzy check module provides a client for rspamd fuzzy storage. Fuzzy check can work with a cluster of rspamd fuzzy storages and the specific storage is selected by value of hash of message's hash. The available configuration options are:

  * metric (string): metric to insert symbol (default: 'default')

  * symbol (string): symbol to insert (default: 'R_FUZZY')

  * max_score (double): maximum score to that weights of hashes would be normalized (default: 0 - no normalization)

  * fuzzy_map (string): a string that contains map in format { fuzzy_key => [ symbol, weight ] } where fuzzy_key is number of fuzzy list. This string itself should be in format 1:R_FUZZY_SAMPLE1:10,2:R_FUZZY_SAMPLE2:1 etc, where first number is fuzzy key, second is symbol to insert and third - weight for normalization

  * min_length (integer): minimum length (in characters) for text part to be checked for fuzzy hash (default: 0 - no limit)

  * whitelist (map string): map of ip addresses that should not be checked with this module

  * servers (string): list of fuzzy servers in format "server1:port,server2:port" - these servers would be used for checking and storing fuzzy hashes

Forged recipients.
==================

Forged recipients is a lua module that compares recipients provided by smtp dialog and recipients from To: header. Also it is possible to compare From: header with SMTP from. So you may set symbol_rcpt option to set up symbol that would be inserted when recipients differs and symbol_sender when senders differs.

Maillist.
=========

Maillist is a module that detects whether this message is send by using one of popular mailing list systems (among supported are ezmlm, mailman and subscribe.ru systems). The module has only option symbol that defines a symbol that would be inserted if this message is sent via mailing list.

Once received.
==============

This lua module checks received headers of message and insert symbol if only one received header is presented in message (that usually signals that this mail is sent directly to our MTA). Also it is possible to insert strict symbol that indicates that host from which we receive this message is either unresolveable or has bad patterns (like 'dynamic', 'broadband' etc) that indicates widely used botnets. Configuration options are:

  * symbol: symbol to insert for messages with one received header.

  * symbol_strict: symbol to insert for messages with one received header and containing bad patterns or unresolveable sender.

  * bad_host: defines pattern that would be count as "bad".

  * good_host: defines pattern that would be count as "good" (no strict symbol would be inserted), note that "good" has a priority over "bad" pattern.

You can define several "good" and "bad" patterns for this module.

Received rbl.
=============

Received rbl module checks for all received headers and make dns requests to IP black lists. This can be used for checking whether this email was transfered by some blacklisted gateway. Here are options available:

  * symbol: symbol to insert if message contains blacklisted received headers

  * rbl: a name of rbl to check, it is possible to define specific symbol for this rbl by adding symbol name after semicolon:

          rbl = pbl.spamhaus.org:RECEIVED_PBL

Conclusion.
===========

Rspamd is shipped with some ammount of modules that provides basic functionality fro checking emails. You are allowed to add custom rules for regexp module and to set up available parameters for other modules. Also you may write your own modules (in C or Lua) but this would be described further in this documentation. You may set configuration options for modules from lua or from xml depends on its complexity. Internal modules are enabled and disabled by filters configuration option. Lua modules are loaded and usually can be disabled by removing their configuration section from xml file or by removing corresponding line from modules section.