aboutsummaryrefslogtreecommitdiffstats
path: root/doc/rspamd.texi
blob: 73c1cbba0e5c7bc4b3a4737718ca733d00b7fd81 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
\input texinfo
@settitle "Rspamd Spam Filtering System"
@titlepage

@title Rspamd Spam Filtering System
@subtitle A User's Guide for Rspamd

@author Vsevolod Stakhov


@end titlepage
@contents

@chapter Rspamd purposes and features.

@section Introduction.
Rspamd filtering system is created as a replacement of popular
@code{spamassassin}
spamd and is designed to be fast, modular and easily extendable system. Rspamd
core is written in @code{C} language using event driven paradigma. Plugins for rspamd
can be written in @code{lua}. Rspamd is designed to process connections
completely asynchronous and do not block anywhere in code. Spam filtering system
contains of several processes among them are:
@itemize @bullet
@item Main process
@item Workers processes
@item Controller process
@item Other processes
@end itemize
Main process manages all other processes, accepting signals from OS (for example
SIGHUP) and spawn all types of processes if any of them die. Workers processes
do all tasks for filtering e-mail (or HTML messages in case of using rspamd as
non-MIME filter). Controller process is designed to manage rspamd itself (for
example get statistics or learning rspamd). Other processes can do different
jobs among them now are implemented @code{LMTP} worker that implements
@code{LMTP} protocol for filtering mail and fuzzy hashes storage server. 

@section Features.
The main features of rspamd are:
@itemize @bullet
@item Completely asynchronous filtering that allows a big number of simultenious
connections.
@item Easily extendable architecture that can be extended by plugins written in
@code{lua} and by dynamicaly loaded plugins written in @code{c}.
@item Ability to work in cluster: rspamd is able to perform statfiles
synchronization, dynamic load of lists via HTTP, to use distributed fuzzy hashes
storage.
@item Advanced statistics: rspamd now is shipped with winnow-osb classifier that
provides more accurate statistics than traditional bayesian algorithms based on
single words.
@item Internal optimizer: rspamd first of all try to check rules that were met
more often, so for huge spam storms it works very fast as it just checks only
that rules that @emph{can} happen and skip all others.
@item Ability to manage the whole cluster by using controller process.
@item Compatibility with existing @code{spamassassin} SPAMC protocol.
@item Extended @code{RSPAMC} protocol that allows to pass many additional data
from SMTP dialog to rspamd filter.
@item Internal support of IMAP in rspamc client for automated learning.
@item Internal support of many anti-spam technologies, among them are
@code{SPF} and @code{SURBL}.
@item Active support and development of new features.
@end itemize

@chapter Installation of rspamd.

@section Obtaining of rspamd.

The main rspamd site is @url{http://rspamd.sourceforge.net/, sourceforge}. Here
you can obtain source code package as well as pre-packed packages for different
operating systems and architectures. Also, you can use SCM
@url{http://mercurial.selenic.com, mercurial} for accessing rspamd development
repository that can be found here:
@url{http://rspamd.hg.sourceforge.net:8000/hgroot/rspamd/rspamd}. Rspamd is
shipped with all modules and sample config by default. But there are some
requirements for building and running rspamd.

@section Requirements.

For building rspamd from sources you need @code{CMake} system. CMake is very
nice source building system and I decided to use it instead of GNU autotools.
CMake can be obtained here: @url{http://cmake.org}. Also rspamd uses gmime and
glib for MIME parsing and many other purposes (note that you are NOT required
to install any GUI libraries -  nor glib, nor gmime are GUI libraries). Gmime
and glib can be obtained from gnome site: @url{http://ftp.gnome.org/}. For
plugins and configuration system you also need lua language interpreter and
libraries. They can be easily obtained from @url{http://lua.org, official lua
site}. Also for rspamc client you need @code{perl} interpreter that could be
installed from @url{http://www.perl.org}.

@section Building and Installation.

Build process of rspamd is rather simple:
@itemize @bullet
@item Configure rspamd build environment, using cmake:
@example
$ cmake .
...
-- Configuring done
-- Generating done
-- Build files have been written to: /home/cebka/rspamd
@end example
@noindent
For special configuring options you can use
@example
$ ccmake .
 CMAKE_BUILD_TYPE                                                                                                                                           
 CMAKE_INSTALL_PREFIX             /usr/local                                                                                                                
 DEBUG_MODE                       ON                                                                                                                        
 ENABLE_GPERF_TOOLS               OFF                                                                                                                       
 ENABLE_OPTIMIZATION              OFF                                                                                                                       
 ENABLE_PERL                      OFF                                                                                                                       
 ENABLE_PROFILING                 OFF                                                                                                                       
 ENABLE_REDIRECTOR                OFF                                                                                                                       
 ENABLE_STATIC                    OFF                                                                                                                       
@end example
@noindent
Options allows building rspamd as static module (note that in this case
dynamicaly loaded plugins are @strong{NOT} supported), linking rspamd with
google performance tools for benchmarking and include some other flags while
building.
@item Build rspamd sources:
@example
$ make
[  6%] Built target rspamd_lua
[ 11%] Built target rspamd_json
[ 12%] Built target rspamd_evdns
[ 12%] Built target perlmodule
[ 58%] Built target rspamd
[ 76%] Built target test/rspamd-test
[ 85%] Built target utils/expression-parser
[ 94%] Built target utils/url-extracter
[ 97%] Built target rspamd_ipmark
[100%] Built target rspamd_regmark
@end example
@noindent
@item Install rspamd (as superuser):
@example
# make install
Install the project...
...
@end example
@noindent
@end itemize

After installation you would have several new files installed:
@itemize @bullet

@item Binaries:
@itemize @bullet
@item PREFIX/bin/rspamd - main rspamd executable
@item PREFIX/bin/rspamc - rspamd client program
@end itemize
@item Sample configuration files and rules:
@itemize @bullet
@item PREFIX/etc/rspamd.xml.sample - sample main config file
@item PREFIX/etc/rspamd/lua/*.lua - rspamd rules
@end itemize
@item Lua plugins:
@itemize @bullet
@item PREFIX/etc/rspamd/plugins/lua/*.lua - lua plugins
@end itemize

@end itemize
For @code{FreeBSD} system there also would be start script for running rspamd in
@emph{PREFIX/etc/rc.d/rspamd.sh}.

@section Running rspamd.

Rspamd can be started by running main rspamd executable -
@code{PREFIX/bin/rspamd}. There are several command-line options that can be
passed to rspamd. All of them can be displayed by passing --help argument:
@example
$ rspamd --help
Usage:
  rspamd [OPTION...] - run rspamd daemon

Summary:
  Rspamd daemon version 0.3.0

Help Options:
  -?, --help               Show help options

Application Options:
  -t, --config-test        Do config test and exit
  -f, --no-fork            Do not daemonize main process
  -c, --config             Specify config file
  -u, --user               User to run rspamd as
  -g, --group              Group to run rspamd as
  -p, --pid                Path to pidfile
  -V, --dump-vars          Print all rspamd variables and exit
  -C, --dump-cache         Dump symbols cache stats and exit
  -X, --convert-config     Convert old style of config to xml one
@end example
@noindent

All options are optional: by default rspamd would try to read
@code{PREFIX/etc/rspamd.xml} config file and run as daemon. Also there is test
mode that can be turned on by passing @emph{-t} argument. In test mode rspamd
would read config file and checks its syntax, if config file is OK, then exit
code is zero and non zero otherwise. Test mode is useful for testing new config
file without restarting of rspamd. With @emph{-C} and @emph{-V} arguments it is
possible to dump variables or symbols cache data. The last ability can be used
for determining which symbols are most often, which are most slow and to watch
to real order of rules inside rspamd. @emph{-X} option can be used to convert
old style (pre 0.3.0) config to xml one:
@example
$ rspamd -c ./rspamd.conf -X ./rspamd.xml
@end example
@noindent
After this command new xml config would be dumped to rspamd.xml file.

@section Managing rspamd with signals.
First of all it is important to note that all user's signals should be sent to
rspamd main process and not to its children (as for child processes these
signals may have other meanings). To determine which process is main you can use
two ways:
@itemize @bullet
@item by reading pidfile:
@example
$ cat pidfile
@end example
@noindent
@item by getting process info:
@example
$ ps auxwww | grep rspamd
nobody 28378  0.0  0.2 49744  9424   rspamd: main process (rspamd)
nobody 64082  0.0  0.2 50784  9520   rspamd: worker process (rspamd)
nobody 64083  0.0  0.3 51792 11036   rspamd: worker process (rspamd)
nobody 64084  0.0  2.7 158288 114200 rspamd: controller process (rspamd)
nobody 64085  0.0  1.8 116304 75228  rspamd: fuzzy storage (rspamd)

$ ps auxwww | grep rspamd | grep main
nobody 28378  0.0  0.2 49744  9424   rspamd: main process (rspamd)
@end example
@noindent
@end itemize

After getting pid of main process it is possible to manage rspamd with signals:
@itemize @bullet
@item SIGHUP - restart rspamd: reread config file, start new workers (as well as
controller and other processes), stop accepting connections by old workers,
reopen all log files. Note that old workers would be terminated after one minute
that should allow to process all pending requests. All new requests to rspamd
would be processed by newly started workers.
@item SIGTERM - terminate rspamd system.
@end itemize

These signals may be used in start scripts as it is done in @code{FreeBSD} start
script. Restarting of rspamd is doing rather softly: no connections would be
dropped and if new config is syntaxically incorrect old config would be used.

@chapter Configuring of rspamd.

@section Principles of work.

We need to define several terms to explain configuration of rspamd. Rspamd
operates with @strong{rules}, each rule defines some actions that should be done with
message to obtain result. Result is called @strong{symbol} - a symbolic
representation of rule. For example, if we have a rule to check DNS record for
a url that contains in message we may insert resulting symbol if this DNS record
is found. Each symbol has several attributes:
@itemize @bullet
@item name - symbolic name of symbol (usually uppercase, e.g. MIME_HTML_ONLY)
@item weight - numeric weight of this symbol (this means how important this rule is), may
be negative
@item options - list of symbolic options that defines additional information about
processing this rule
@end itemize

Weights of symbols are called @strong{factors}. Also when symbol is inserted it
is possible to define additional multiplier to factor. This can be used for
rules that have dynamic weights, for example statistical rules (when probability
is higher weight must be higher as well).

All symbols and corresponding rules are combined in @strong{metrics}. Metric
defines a group of symbols that are designed for common purposes. Each metric
has maximum weight: if sum of all rules' results (symbols) is bigger than this
limit then this message is considered as spam in this metric. The default metric
is called @emph{default} and rules that have not explicitly specified metric
would insert their results to this default metric.

Let's impress how this technics works:
@enumerate 1
@item First of all when rspamd is running each module (lua, internal or external
dynamic module) can register symbols in any defined metric. After this process
rspamd has a cache of symbols for each metric. This cache can be saved to file
for speeding up process of optimizing order of calling of symbols.
@item Rspamd gets a message from client and parse it with mime parsing and do
other parsing jobs like extracting text parts, urls, and stripping html tags.
@item For each metric rspamd is looking to metric's cache and select rules to
check according to their order (this order depends on frequence of symbol, its
weight and execution time).
@item Rspamd calls rules of metric till the sum weight of symbols in metric is
less than its limit.
@item If sum weight of symbols is more than limit the processing of rules is
stopped and message is counted as spam in this metric.
@end enumerate

After processing rules rspamd is also does statistic check of message. Rspamd
statistic module is presented as a set of @strong{classifiers}. Each classifier
defines algorithm of statistic checks of messages. Also classifier definition
contains definition of @strong{statistic files} (or @strong{statfiles} shortly).
Each statfile contains of number of patterns that are extracted from messages.
These patterns are put into statfiles during learning process. A short example:
you define classifier that contains two statfiles: @emph{ham} and @emph{spam}.
Than you find 10000 messages that are spam and 10000 messages that contains ham.
Then you learn rspamd with these messages. After this process @emph{ham}
statfile contains patterns from ham messages and @emph{spam} statfile contains
patterns from spam messages. Then when you are checking message via this
statfiles messages that are like spam would have more probability/weight in
@emph{spam} statfile than in @emph{ham} statfile and classifier would insert
symbol of @emph{spam} statfile and would calculate how this message is like
patterns that are contained in @emph{spam} statfile. But rspamd is not limiting
you to define one classifier or two statfiles. It is possible to define a number
of classifiers and a number of statfiles inside a classifier. It can be useful
for personal statistic or for specific spam patterns. Note that each classifier
can insert only one symbol - a symbol of statfile with max weight/probability.
Also note that statfiles check is allways done after all rules. So statistic can
@strong{correct} result of rules.

Now some words about @strong{modules}. All rspamd rules are contained in
modules. Modules can be internal (like SURBL, SPF, fuzzy check, email and
others) and external written in @code{lua} language. In fact there is no differ
in the way, how rules of these modules are called:
@enumerate 1
@item Rspamd loads config and loads specified modules.
@item Rspamd calls init function for each module passing configurations
arguments.
@item Each module examines configuration arguments and register its rules (or
not register depending on configuration) in rspamd metrics (or in a single
metric).
@item During metrics process rspamd calls registered callbacks for module's
rules.
@item These rules may insert results to metric.
@end enumerate

So there is no actual difference between lua and internal modules, each are just
providing callbacks for processing messages. Also inside callback it is possible
to change state of message's processing. For example this can be done when it is
required to make DNS or other network request and to wait result. So modules can
pause message's processing while waiting for some event. This is true for lua
modules as well.

@bye