You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

ChangeLog 58KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376
  1. 1.1.2:
  2. * Fix stat_cache closing
  3. * Add checkpoints to sqlite3 learn cache
  4. * Do not recompile lua generated headers all the time
  5. * Increase number of messages learned
  6. * Fix issues with dual stack and hfilter
  7. * Disable MID checks for hfilter by default
  8. * Fix cache definitions in multiple classifier and no type
  9. * Don't crash if learn cache failed to initialize
  10. * Fix googlegroups support in maillist plugin
  11. * Rework flags LUA API:
  12. - Allow to check for a specific flag
  13. - Add `learn_spam`, `learn_ham` and `broken_headers` flags
  14. - Unify internal functions
  15. * Add `BROKEN_HEADERS` rule
  16. * Add support for forged confirmation headers (by @AdUser)
  17. * Allow `any`, `mime` and `smtp` for get_from/get_recipients
  18. * Add mime types checking plugin
  19. * Add rule to detect spammers attempts to cheat mime parsing
  20. * Rework parsing of IP addresses in configuration (better IPv6 support)
  21. * Add `util.parse_mail_address` function to LUA API
  22. * Add lua sqlite3 module
  23. * Implement synchronous redis call
  24. * Ratelimit: avoid possible indexing of nil value (Fixes #498) (by @fatalbanana)
  25. * Add stat_convert command to convert stats tokens from sqlite3 to redis
  26. * Implement redis advanced lua api with pipelining
  27. * Fix memory leak on redis stat (#500)
  28. * Fix user/language learn count in sqlite statistics (#496) (by @fatalbanana)
  29. * Fix build with custom pcre
  30. * Fix fuzzy relearning (#498)
  31. * Improve planning of asynchronous tasks
  32. * Show slow rules in log
  33. * Add warning for slow regexps
  34. * Add base32 decode/encode routines to lua util
  35. * Allow converting of learn cache from sqlite to redis
  36. * Add methods to check if a messages has from/rcpts
  37. * Improve and fix multimap plugin:
  38. - Restore 'header' maps
  39. - Add filters for headers
  40. - Add 'email:addr', 'email:user', 'email:domain' and 'email:name' filters
  41. - Add generic regexp filters
  42. * Disable reload command in rc scripts
  43. * Improve runtime CPU dispatcher for libcryptobox
  44. * Add preliminary support of digital signatures via ed25519
  45. * Add detection for RDRAND support
  46. * Print configuration of crypto on start
  47. * A in SPF presumes AAAA lookup as well
  48. 1.1.1:
  49. * Fix duplicated XBL symbol
  50. * Reduce log severity for ratelimit missing servers
  51. * Fix XBL composite to avoid duplicate symbols
  52. * Reduce weight of URL_ONLY rule due to FP rate
  53. * Disable fuzzy hashes from the metadata for now
  54. * Fix processing of empty messages (#486)
  55. * Always treat DNS timeouts as temporary fail for SPF
  56. * Fix issue with SPF double IP stack (#483)
  57. * Use X-Forwarded-For when checking secure_ip (#488)
  58. * Fix hash calculation for sqlite stats
  59. * Fix memory corruption on punycode
  60. * Fix strings allocation in punycode
  61. * Fix error message (#491)
  62. 1.1.0:
  63. * Incompatible change: sqlite3 and per_user behaviour:
  64. Now both redis and sqlite3 follows the common principles for per-user
  65. statistics:
  66. 1) If per-user statistics is enabled check per-user tokens ONLY
  67. 2) If per-user statistics is not enabled then check common tokens ONLY
  68. If you need old behaviour, then you'd need to use separate classifier
  69. for per-user statistics.
  70. * Implement redis statistics backend and cache
  71. * Implement autolearning for statistics
  72. * Reworked statistics architecture from scratch
  73. * Add hyperscan (https://github.com/01org/hyperscan) engine for regular
  74. expressions:
  75. - add lazy loader for hyperscan databases
  76. - rework regexp cache to have joint pcre/hyperscan scanning
  77. - implement hyperscan pre-filter support
  78. - add compilation guards for bad expressions
  79. - implement `rspamadm control recompile` command
  80. - implement hyperscan cache monitoring
  81. - slides: <https://highsecure.ru/rspamd-hyperscan.pdf>
  82. * Implement flexible task logging
  83. * Rework fuzzy worker:
  84. - it is now possible to run multiple fuzzy workers;
  85. - implement lazy writing as sqlite3 is bad at concurrent writing;
  86. - add retries for simple sql commands in fuzzy backend;
  87. - use fine-grained transactions for fuzzy;
  88. - implement new multi-pubkeys mode;
  89. - allow encrypted only storages;
  90. - rework statistics for fuzzy;
  91. - add `rspamadm control fuzzystat` command for extended statistics;
  92. - implement human readable output for the previous command;
  93. - add condition script for learning fuzzy storage;
  94. * Various fixes to SPF:
  95. - fix `redirect` records;
  96. - fix domains when parsing mx/ptr/a records in includes/redirects;
  97. - fix issues with multiple addresses in SPF records;
  98. - ignore SPF results in case of DNS failure;
  99. - adjust TTL of records when resolving subelements of SPF records;
  100. - always select `v=spf1` line if it is available
  101. - do not cache records with DNS failure in subrequests;
  102. - ignore records with temporary fails during subrequests resolving;
  103. - fix `RDNS_RC_NOREC` support;
  104. * Add clang plugin for static analysis:
  105. - implement static checks for `rspamd_printf` format strings;
  106. * Add 'allow_raw_input' option for non-mime messages
  107. * Recognize types using libmagic
  108. * Fix parsing of IPv6 received headers
  109. * Add new interface of communication between workers in rspamd
  110. * Add support for named socketpairs
  111. * Don't write URLs by default as it is too verbose
  112. * Set status for HTTP replies
  113. * Try load `rspamd.conf.override`
  114. * Implement words decaying for text parts to limit many checks
  115. * Improve support of SA rules and plugins:
  116. - add check_for_shifted_date and check_for_missing_to_header eval rules;
  117. - add 'check_relays_unparseable' support;
  118. - add `check_for_mime('mime_attachement')` function;
  119. - use new re_cache interface for all SA rules;
  120. - add support for `Mail::SpamAssassin::Plugin::MIMEHeader`;
  121. - add support of 'special' SA headers to `exists` function;
  122. - fix issue when SA metas contain other metas;
  123. - fix freemail rules;
  124. * Many fixes to the URL parser
  125. * Match any newline character in regexps
  126. * Fix resolving of upstreams and detection of poor IPv6 configurations
  127. * Parse upstreams selection algorithm from the configuration line
  128. * Add `reresolve` command to the control interface
  129. * Generate fuzzy hashes from task metadata (URLs and headers)
  130. * Add method to check if IP is local and `local_addrs` option
  131. * Implement forced timeout for delayed filters
  132. * Disable fast path of pcre-jit as it seems to be broken
  133. * Bayes fixes:
  134. - new normalizer function;
  135. - really use weights of tokens from the OSB algorithm;
  136. - restore multiple classifiers support;
  137. * Rules changes:
  138. - add `R_SUSPICIOUS_URL` rule that detects obfuscated URL's;
  139. - improve empty image rule;
  140. - rework `FORGED_RECIPIENTS` rule;
  141. - reduce weight of `SUSPICIOUS_RECIPS`;
  142. - fix `*_NORESOLVE_MX` symbols in hfilter;
  143. - add `SUBJ_ALL_CAPS` rule with support of UTF8
  144. - add spamhaus SBL to uribl
  145. - fix `SUSPICIOUS_RECIPS` and `SORTED_RECIPS` rules
  146. - remove `R_TO_SEEMS_AUTO` as it generates a lot of FP;
  147. - add new Message-ID regexp for Thunderbird (by @moisseev);
  148. * Plugins changes:
  149. - allow ratelimit plugin to set symbol instead of pre-result
  150. - support IP DNS black lists for URIBL (e.g spamhaus SBL);
  151. - drop deprecated SURBL bits (by @fatalbanana)
  152. - rename `JP_SURBL_MULTI` to `ABUSE_SURBL` (by @fatalbanana)
  153. - add `SURBL_BLOCKED` (by @fatalbanana)
  154. - add `CR_SURBL`
  155. - SURBL: allow fallthrough to default symbol (by @fatalbanana)
  156. - Settings: fix IP match (by @fatalbanana)
  157. - SURBL: add missing symbols to metric (by @fatalbanana)
  158. - allow processing images urls for SURBL
  159. - unconditionally disable SPF for authenticated users and local networks
  160. * Rework ratelimit plugin
  161. - switch to `rates` instead of old and stupid strings to setup;
  162. - check if a bucket is zero and disable the corresponding limits'
  163. - turn off all buckets by default;
  164. - check either `rcpt` or `user` buckets, not all together'
  165. - document new `rates` and `symbol` options;
  166. - inform user about what buckets are used in the configuration;
  167. * Add neural network **experimental** plugin
  168. * Add a sample script to learn neural network from rspamd logs
  169. * Add documentation strings support to rspamd:
  170. - add strings for the main configuration options;
  171. - document workers options;
  172. - add internal plugin options;
  173. - create `rspamadm confighelp` routine;
  174. - implement human readable output for the previous command;
  175. - add subtree search support;
  176. - add keyword search support;
  177. * Documentation improvements, tutorials section, statistics description
  178. * Many other minor and major bugfixes not noted here
  179. 1.0.11:
  180. * Fix spf redirects
  181. * Fix domains when parsing mx/ptr/a records in includes/redirects
  182. * Fix unfolded base64 encoding
  183. * Fix GError use-after-free
  184. * Do not rewrite the original url when using redirector
  185. * Fix parsing of fragment in urls
  186. * Fix processing of HTML tags
  187. * Improve empty image rule
  188. * Avoid long double type
  189. * Fix tokens weights in OSB algorithm
  190. * Improve debugging for bayes
  191. 1.0.10:
  192. * Fix settings application (#416)
  193. * Fix another issue with fixed strings
  194. * Fix hash function invocation
  195. * Use the proper string for make_dns_request in lua_http
  196. * Fix scan time output
  197. * Update webui:
  198. - fix labels for greylisting
  199. - fix dimension of scan time
  200. 1.0.9:
  201. * Emergency fix in keyed blake2 to fix fuzzy hashes and encrypted password
  202. * Support passwords longer than 64 symbols
  203. 1.0.8:
  204. * Add function to traverse AST atoms
  205. * Allow dependencies on rspamd symbols for SA metas
  206. * Fix memory corruption when timeout is removed in fuzzy check
  207. * Fix encrypted fuzzy add processing
  208. * Avoid use-after-free in controller session destructor
  209. * Use session pool instead of task pool in fuzzy check
  210. * Fix assembly in i386 mode (#413, #412)
  211. 1.0.7:
  212. * Plugged memory leaks in internet address object & html parser
  213. * Fixed static build
  214. * Fixed multiple sigchld processing
  215. * Fixed deletion of signal events after event processing loop
  216. * Fixed build on ARM (#404 - reported by @Gottox)
  217. * Fixed setting the default mask for SPF.
  218. * Fixed sanitisation of HTTP query values
  219. * Fixed parsing of the last header in encrypted HTTP messages
  220. * Additions and fixes for test suite & benchmarks
  221. * Added openssl aes-256-gcm support to libcryptobox & HTTP server
  222. * Implemented support for starting multiple HTTP servers
  223. * Implemented batch accept in HTTP server
  224. * Added module to get data from HTTP headers (#285 - reported by @msimerson)
  225. * Added `rspamadm control` command
  226. * Added ability to sort counters output.
  227. * Added ability to specify custom headers for rspamc client
  228. * Fix architecture detection
  229. * Converted history storage to the UCL format
  230. * Allow flexible number of rows in history
  231. * Fix action badges in WebUI
  232. * Add universal cryptobox hash API
  233. * Migrated to the optimized blake2b implementation adopted from Andrew Moon
  234. * Allow explicit loading of specific modules
  235. * Always load settings module
  236. * Allow to add symbols from settings
  237. * Fix double free in the controller fuzzy learn command
  238. * Avoid endless loop when cannot open sqlite db
  239. * Updated libucl
  240. 1.0.6:
  241. * Fix build on i386
  242. * Update CentOS7 service file patch (by @fatalbanana)
  243. * Fix path to rspamadm in Debian init script (by @fatalbanana)
  244. * Fix broken '_SC_GETPW_R_SIZE_MAX' on FreeBSD
  245. * Fix portability issues
  246. * Use cryptobox chacha for libottery
  247. * Better support of 32 bit builds
  248. * Fix header name tokens setup
  249. * Fix levenstein distance method for words
  250. * Add workaround for old libevent (#400)
  251. * Fix microseconds in termination timer
  252. * Fix some more issues with fixed strings
  253. * Explicitly test CPU instructions even after CPUID call
  254. * Do not check out of boundary memory
  255. * Do not output broken emails
  256. * Fix unknown symbols registration
  257. * Handle SIGILL using longjmp
  258. * Block signals when exiting event loop
  259. * Fix incorrect allocation size
  260. * Slightly optimize alignment
  261. * Restore rspamd -t for compatibility
  262. * Add more sanity checks for emails
  263. 1.0.5:
  264. * Add rspamd control interface:
  265. - support `stat` command to get runtime stats of rspamd workers
  266. - support `reload` command to reload runtime elements (e.g. sqlite3 databases)
  267. * Rework curve25519 library for modular design:
  268. - add Sandy2x implementation by Tung Chou
  269. - fix CPU detection for variables loading assembly
  270. - add testing for curve25519 ECDH
  271. * New fixed strings library
  272. * Add `R_SUSPICIOUS_IMAGES` rule
  273. * Enable mmap in sqlite3
  274. * Use new strings in the HTTP code
  275. * Improve google perftools invocation
  276. * Improve performance profiling in http test
  277. * Reorganize includes to reduce namespace pollution
  278. * Allow specific sections printing in configdump command
  279. * Rework workers signals handlers to be chained if needed
  280. * Update socketpair utility function
  281. * Add control_path option for rspamd control protocol
  282. * Fix ownership when listening on UNIX sockets
  283. * Rework signals processing in main
  284. * Remove extra tools from rspamd (they live in rspamadm now)
  285. * Remove global rspamd_main
  286. * Add global timeout for the overall task processing (8 seconds by default)
  287. * Sanitize NULL values for fuzzy backend
  288. * Store NM between encrypt/decrypt
  289. * Add textpart:get_words_count method
  290. * Fix generic DNS request in lua
  291. * Tune hfilter weights
  292. * Add support of IPv6 in hfilter
  293. * Fix parsing of HTTP headers with IP addresses
  294. * Sync with the recent libucl
  295. * Various minor bugfixes
  296. 1.0.4:
  297. * Add configdump routine to rspamadm
  298. * Implement retransmits for fuzzy_check plugin
  299. * Fix events processing for learning anf checking fuzzy hashes
  300. * Avoid dependency on unneeded and uncompatible glib include
  301. * Add `historyreset` command to the controller
  302. * Fix loading of tokenizer config from dump (#389)
  303. * Add sorting hints for the history
  304. * Allow custom lua scripts for users/languages extraction (#388)
  305. * Do not add FORGED_RECIPIENTS when 'To' is missing (#387)
  306. * Do not add R_UNDISC_RCPT when 'To' is missing (#387)
  307. * Add encryption to fuzzy check plugin
  308. * Add encryption for fuzzy storage
  309. * Add new epoch for encrypted fuzzy request
  310. * Add encryption for `rspamd.com` storage
  311. * Remove gmime processing for LDA mode as it is deadly broken
  312. * Add routine to find end of headers position in mime messages
  313. * Fix LDA headers folding
  314. * Init libraries in rspamc client as well to avoid locale issues
  315. * Avoid collision with locally installed includes
  316. * Allocate and free memory with the same allocator in rspamadm (#385)
  317. * Preserve expired fuzzy hashes counter
  318. * Improvements in webui:
  319. - Add favicon.ico
  320. - Rework history table
  321. - Fix sorting for the history
  322. - Migrate to bootstrap 3 and jquery 2
  323. - Fix css bugs
  324. - Add glyphicons
  325. - Add reset history
  326. - Improve history buttons
  327. - Redraw graph to avoid display issues
  328. - Webui is now MIT licensed to match licensing policy of rspamd
  329. 1.0.3:
  330. * Fix piechart clean slice (#380)
  331. * Fix controller crashes when GString is reallocated (#381)
  332. * Correctly set locale before start
  333. * Set C locale for numeric values
  334. * Add rspamadm routine:
  335. - add `pw` command to manage passwords
  336. - add `help` command for displaying help
  337. - add `configtest` command to check configuration files
  338. - add `keypair` command for generating encryption keys
  339. - add `fuzzy_merge` routine to merge fuzzy sqlite databases
  340. - add a simple manual page for rspamadm
  341. * Allow metric registration for composite expressions
  342. * Add strict mode for configtest
  343. * Add logger counters
  344. * Save and show learned messages count (#383)
  345. * Add `no_stat` flag
  346. * Add `task:set_flag` and `task:get_flags` (#382)
  347. * Enable foreign keys in sqlite3
  348. * Remove orphaned shingles from fuzzy storage
  349. * Optimize synchronization steps for fuzzy storage
  350. * Allow delayed conditions registration
  351. * Add lua API for conditions registering
  352. 1.0.2:
  353. * Fix critical bug in webui that prevents password from being sent
  354. * Rework webui view:
  355. - Switch to d3.js for graphs
  356. - Improve piechart look
  357. - Rework colors for piechart
  358. - Fix layout for symbols
  359. - Fix refresh button
  360. * Add descriptions for whitelist maps
  361. * Fix build on arm (#379)
  362. * Fix issue with the last element in the radix trie
  363. * Add more tests for radix trie algorithm
  364. * Allow to extract URLs from query strings of other URLs (#361)
  365. * Initialize rrd fields before writing to file
  366. * Fix double free if no password has been specified
  367. 1.0.1:
  368. * Add writing to rrd from the controller
  369. * Fixed lots of bugs in rrd code
  370. * Adopt new DNS API in hfilter plugin (by @AlexeySa)
  371. * Allow only one controller process to manage rrd file
  372. * Set event base for fuzzy calls
  373. * Improve fuzzy IO errors logging
  374. * Add rra extraction function to rrd library
  375. * Add graph handler to the controller
  376. * Cache correct passwords to avoid too high CPU usage when working with webui
  377. * Controller sockets are owned by router do not export them to task
  378. * Optimize logging by skipping hash table search if it's empty
  379. * Fix loading issue whith broken statfiles
  380. * Print assertions from glib to rspamd logger
  381. * Load legacy `lua/rspamd.local.lua`
  382. * Update webui with some fixes to learning and scanning
  383. 1.0.0:
  384. * Rework symbols processing:
  385. - Improve sorting logic for symbols
  386. - Organize processing into multiple stages
  387. - Added asynchronous watchers for symbols
  388. - Added ability to organize dependencies between symbols
  389. * Fixed URL redirector:
  390. - Use optimized POE loop
  391. - Organize dependencies
  392. - Fix startup
  393. * New sqlite3 backend:
  394. - Allow to have per-languages and per-user statistics
  395. - Allow sqlite3 to be used as statistics backend
  396. * Store tokenizer configuration within statfiles
  397. * Improve bayes statistics:
  398. - Use headers and images metainformation in bayes
  399. - Suggest using of pre-processed tokens for statistics
  400. - Fix tokens normalization for OSB algorithm
  401. * Rewrite url parsing:
  402. - Fix numerous issues with url extraction and normalization
  403. - Fix mailto urls
  404. * Fix settings plugin to allow custom actions scores
  405. * Improve rbl plugin
  406. * Allow capturing patterns in rspamd lua regexp library
  407. * Add GTUBE support
  408. * Fix spamc legacy support
  409. * Add DKIM support to RBL module
  410. * Fix issues with multiple DKIM signatures
  411. * Fix issue if rspamd cannot create statfiles (#331)
  412. * Rework parts and task structure:
  413. - Now text_parts, parts and received are arrays
  414. - Pre-allocate arrays with some reasonable defaults
  415. - Use arrays instead of lists in plugins and checks
  416. - Remove unused fields from task structure
  417. - Rework mime_foreach callback function
  418. - Remove deprecated scan_milliseconds field
  419. * Add ip_score plugin support (not enabled by default):
  420. - Can check for asn/country and network using DNS lookups
  421. - Can store and load reputation from redis server
  422. * Improve PARTS_DIFFER rule to count merely different words
  423. * New HTML parser:
  424. - Parses HTML parts using a set of state machines
  425. - Extracts useful data and exports it to lua functions:
  426. + Styles
  427. + Images
  428. + URLs
  429. + Colors
  430. + Structure elements
  431. - Added HTML rules for some checks
  432. * New version of LUA DNS API
  433. * Table versions of many functions in LUA API
  434. * Improve rspamc client:
  435. - Print execution time
  436. - Allow executing of external commands and passing output to them
  437. - Allow mime output mode when rspamc alters message according to rspamd
  438. checks and send it to an external command or stdout
  439. * Allow scanning of local files using HTTP requests
  440. * Rework configuration system:
  441. - Rules are now moved from the $CONFDIR to $RULESDIR to avoid ambiguity
  442. - All modules configurations are now split in $CONFDIR/modules.d/* to
  443. simplify upgrades
  444. - Move hfilter to plugins
  445. - Allow plugins and rules to define default scores to simplify metrics
  446. setup
  447. - Include overrides for all modules to honor local/automatic parameters
  448. - Tune scores for many modules
  449. * Rework and enable DMARC plugin
  450. * Add whitelist plugin for SPF/DKIM/DMARC based whitelisting
  451. * Add some common domains to whitelists shipped with rspamd
  452. * Rework logging:
  453. - Now each log entry supports module name and a `tag`. Tag is used to
  454. identify unique objects (such as tasks) when checking log files
  455. - It is possible to turn on debugging for the specific modules
  456. - Systemd logging is fixed
  457. * Improve spamassassin plugin.
  458. - Now headers are matched more like SA
  459. - Improve support of Message-ID
  460. - Add support of ToCc header type
  461. - Fix :addr and :name in headers regexps
  462. * Resurrect rrd support code
  463. * Save controller stats between restarts
  464. * Fixed tonns of bugs
  465. * Added tonns of minor improvements and features
  466. * Added more unit tests
  467. * Create functional tests framework
  468. * Added documentation for missing modules
  469. * Added rpm/deb repositories and scripts
  470. * Updated WebUI and libucl externals
  471. 0.9.10:
  472. * Do not dereference null pointer on learning.
  473. * Fix some extreme cases in BAYES.
  474. * Add a workaround to avoid bad HTML messages breaking.
  475. * Build with -O2 flags by default.
  476. * Add constraints to limit DNS requests count per task.
  477. * Add workaround for SURBL DNS flood.
  478. * Set error if rspamd cannot learn anything.
  479. 0.9.9:
  480. * Don't use RWL_SPAMHAUS_WL (unknown result) for whitelisting (by @fatalbanana)
  481. * Import updated public suffix list (by @fatalbanana)
  482. * Remove debug message
  483. * Fix settings (by @fatalbanana)
  484. * Remove duplicated symbol registration
  485. * Use WAL for fuzzy storage
  486. * RBL fixes (by @fatalbanana):
  487. - silence errors;
  488. - yield unknown results from RBLs;
  489. - fix scoring for DNSWL;
  490. - fix use of RBL name as symbol;
  491. - ignore RBL names that would not be yielded;
  492. * Support captures in regular expressions
  493. * Add captures support to lua_regexp
  494. * Support dist on FreeBSD and Darwin
  495. * Add RCVD_IN_DNSWL_NONE as whitelisting exclusion (by @fatalbanana)
  496. * Multiple fixes to URL detection:
  497. - support port definition;
  498. - fix query and path recognition;
  499. - fix parsing of multiple slashes in URL;
  500. - fix parsing query just after port;
  501. - fix path field in `url:to_table` method;
  502. - improve support of IP based URLs.
  503. * Set ignore_whitelists = true for RECEIVED_SPAMHAUS_XBL (by @fatalbanana)
  504. * Add GTUBE support
  505. * Ignore User header in SA mode
  506. 0.9.8:
  507. * Fix critical bug in bayes classifier (#305)
  508. * Fix critical bug in RBL module (by @fatalbanana)
  509. * Fix and rework settings plugin.
  510. * Fix get_all_opts for a case of non-iterable options.
  511. * Use tld for redirector's matching.
  512. 0.9.7:
  513. * Add whitelist_exception setting to RBL module (by @fatalbanana)
  514. * Don't use RWL_MAILSPIKE_POSSIBLE or DNSWL_BLOCKED for whitelisting (by
  515. @fatalbanana)
  516. * Fix extreme cases in bayes classifier.
  517. * Fix parsing of urls with '?' at the end of hostname.
  518. * Update interface.
  519. * Fix number of issues with webui interaction.
  520. * Fix saving maps.
  521. * Allow user@ and @domain matches in multimap.
  522. * Fix issues with bounces From processing.
  523. * Fix abs/fabs misuse.
  524. * Fix builds on suse and arch linux distributions.
  525. 0.9.6:
  526. * Fix memory leak if mime cannot be parsed.
  527. * Fix dkim cache expiration.
  528. * Fix issues with redirector HTTP response.
  529. * Fix abnormal connection closing with certains messages with a high score
  530. (issue #296)
  531. * Fix redirector installation.
  532. * Use specific POE loop for some systems.
  533. * Fix number of issues in URL redirector.
  534. * Fix selecting URLs for sending to redirector.
  535. 0.9.5:
  536. * Avoid double free when extending HTTP message.
  537. * Fix double free if multiple classifiers are defined.
  538. * Fix misprint in spamassassin plugin.
  539. * Fix cpuid invocation on i386.
  540. * Fix ownership issues for zero-copy decode.
  541. * Allow __len metamethod on rspamd{text}.
  542. * Add base64 decoding lua utility.
  543. * Fix build on FreeBSD
  544. * Skip spaces at the beginning of mime messages.
  545. * DBL_ABUSE_REDIR should not have significant weight.
  546. * Allow to split by lua_regexp rspamd{text} objects.
  547. * Allow to specify custom stop pattern for lua_tcp.
  548. 0.9.4:
  549. * Fix critical bugs in tokenization algorithm
  550. * Write unit tests for tokenization
  551. * Add documentation for lua_tcp
  552. * Switch off legacy tokenization by default.
  553. * Fix critical bugs in words normalization
  554. * Add lua bindings to tokenizer.
  555. * Implement storing of HTTP headers inside task
  556. * Add lua API to accerss HTTP headers data
  557. * Implemented base64 encoding suitable for MIME
  558. * Use caseless hash and equal functions for HTTP request headers.
  559. * Improve debian architectures support (by @dottedmag)
  560. 0.9.3:
  561. * Revert incorrect regexp change that broke the default rules
  562. * Fix lua_tcp module
  563. 0.9.2:
  564. * Fix error on spawning unique workers.
  565. * Add preliminary version of generic LUA TCP requests API.
  566. * Use lua 5.1 if luajit is not available (Arm64, PowerPC, s390x etc)
  567. * Fix fuzzy mime strings with only type.
  568. * Improve thunderbird sanity checks.
  569. * Fix critical bug on matching regular expressions.
  570. * Make hiredis optional dependency.
  571. * Fix multiple bugs in daemon reloading
  572. 0.9.1:
  573. * Restore utf8 validation for regular expressions to avoid crashes
  574. * Fix symbols displaying in the interface
  575. * Add symbol groups to the interface
  576. * Fix maps ID parsing in the controller
  577. * Add multimap and regexp modules documentation
  578. * Backport fixes from libucl
  579. * Fix debian package (by @dottedmag)
  580. * Rework XXH32 invocations
  581. 0.9.0:
  582. * Add support of the fast and secure protocol level encryption:
  583. - curve25519 is used for key exchange;
  584. - chacha20/poly1305 cryptobox construction for bulk encryption;
  585. - zero latency overhead;
  586. - encrypting and balancing HTTP proxy worker
  587. * Rework expressions and create new expressions library:
  588. - aggressive optimizations based on the abstract syntax tree;
  589. - abstract expressions support (regular expressions, functions, lua modules
  590. composites and so on)
  591. - New comparision and '+' operators support
  592. - New greedy algorithm to minimize execution time of expressions and
  593. all symbols
  594. - Dynamic expressions benchmark and reoptimizations
  595. * Many improvements to the LUA API:
  596. - reworked logger module allowing to do pretty print of the most of lua
  597. types (including tables and userdata classes)
  598. - reworked lua redis and lua HTTP to support more features
  599. - added opaque type for passing large text chunks without copying
  600. - new regexp module with many auxiliary functions (e.g. `re:split`)
  601. * LuaJIT is now the default requirement for rspamd allowing to speed up lua
  602. execution by a large margin (however, plain lua is still supported)
  603. * New plugins:
  604. - spamassassin rules plugin that allows to load and re-use the most of
  605. SA rules natively
  606. - DMARC plugin that evaluates SPF and DKIM policies to the domain policies
  607. - many old plugins has been reworked to implement new features and improve
  608. stability
  609. * New aho-corasic trie implementation from @mischasan that allows to load and
  610. use hundreds of thousands of patterns with no influence on load
  611. * Support of PCRE JIT and PCRE JIT fast path modes that significantly improves
  612. the performance of regular expressions if supported by PCRE
  613. * New URLs parser and extractor:
  614. - removed legacy code that was useless for url finding
  615. - reworked algorithms of URL parsing for more precise and accurate results
  616. - added top-level-domains tree from http://publicsuffix.org
  617. - improved emails parsing
  618. - removed many phishing false positives due to TLD tree check
  619. * New statistics infrastructure:
  620. - created a separate layer of statistic library
  621. - improved OSB-Bayes by re-weighting tokens according to the original
  622. academic paper and `crm114` implementation, which reduced false positives
  623. rate significantly
  624. - created learn cache to avoid double learning of statistics and providing
  625. an efficient way to re-learn class for a message
  626. - created abstract layers for different statistics backends
  627. - implemented new tokenization algorithms with fast or secure (siphash)
  628. hashes to generate statistics features
  629. * Reworked utf8 tokenization that previously corrupted all UTF8 words (minor
  630. incompatibility with old fuzzy hashes with utf-8 symbols)
  631. * SPF module has been completely rewritten to support complex cases of
  632. `include` and `redirect` within SPF records
  633. * DKIM module now supports multiple signatures
  634. * Controller passwords can now be stored encrypted by `PBKDF2-HMAC` in the
  635. configuration file
  636. * Many hand-written HTTP clients has been replaced with the common rspamd
  637. http module
  638. * New test framework:
  639. - import lua `telescope` test framework
  640. - add unit tests for many rspamd modules and routines
  641. - create a unit test for each possible bug found
  642. - use luajit ffi for testing C code
  643. - added preliminary support of functional testing by creating tasks from lua
  644. * Randomize hash seed to avoid certain hash tables vulnerabilities
  645. * Documentation improvements:
  646. - added documentation for the vast majority of rspamd modules
  647. - added documentation for rspamd protocol
  648. - added documentation for the most of rspamd LUA extensions
  649. * Fixed tonns of bugs and memory leaks
  650. * Added tonns of minor features
  651. 0.8.3:
  652. * Various critical fixes in distribution (by @dottedmag and @fatalbanana)
  653. * Fixed bugs in url detector to parse certain patterns
  654. * Add default host and helo for a client
  655. * Some sanity checks for tokenizer and classifier
  656. * Reiterate on systemd support
  657. * Fix missing symbol registration
  658. * Add support of spamc compatible output
  659. * Filter double-dots in rbl.lua validate_dns (by @fatalbanana)
  660. * Update ucl submodule due to critical bugfix
  661. 0.8.2:
  662. * Create fuzzy db if it does not exist
  663. * Fix: Centos init script: configtest() (by @AlexeySa)
  664. * Enable one_shot for RECEIVED_SPAMHAUS_XBL - Fixes #102 (by @fatalbanana)
  665. * Update Exim patch (by @fatalbanana)
  666. * Fix processing of unix sockets.
  667. * Allow applying settings to authenticated users (by @fatalbanana)
  668. * Make settings priorities work as documented (by @fatalbanana)
  669. * Fix race condition in symbols planner
  670. * Add DNSWL_BLOCKED symbol (by @fatalbanana)
  671. * Make Exim pass usernames to rspamd (by @fatalbanana)
  672. * Update RBL module (by @fatalbanana):
  673. - fix indentation;
  674. - collapse loops;
  675. - avoid calling for un-needed information;
  676. - allow disabling RBLs for authenticated users
  677. * once_received.lua: Fix indentation & add exclusion for authenticated users (by @fatalbanana)
  678. * hfilter.lua: Add exclusion for authenticated users (by @AlexeySa)
  679. * Updates to hfilter rules (by @AlexeySa)
  680. * Set empty <> user or addr for msgs without FROM (by @eneq123)
  681. * Fix: attempt to index field '?' (a nil value) (by @eneq123)
  682. * Fix: if not exist Date-header (by @AlexeySa)
  683. * Add task:get_content() method.
  684. * rbl.lua: Ignore private IP space (by @fatalbanana)
  685. * Allow to check radix maps from lua by rspamd{ip}
  686. * Make local exclusions configurable per-RBL (by @fatalbanana)
  687. * Add rspamd_config:radix_from_config() (by @fatalbanana)
  688. * Support emails dnsbl in rbl (by @fatalbanana)
  689. * Complete rework of url extraction logic
  690. * Allow customizations for unix sockets. (fixes #182)
  691. * Set lua path according to rspamd settings.
  692. * Import lua-functional for plugins stuff.
  693. * Completely rewrite multimap plugin in functional style.
  694. * Fix FORGED_MUA_THUNDERBIRD_MSGID (fixes #186)
  695. * Check IPv6 addresses at dnswl.org and Spamhaus whitelist (by @fatalbanana)
  696. * Add lowercase utility for utf8 strings.
  697. * Various fixes to build system
  698. * Updated debian configuration infrastructure (by @dottedmag)
  699. 0.8.1:
  700. * Add sqlite and perl as dependencies for RPM/Debian packages (by @fatalbanana)
  701. * Remove whitelist.lua from RPM file list (by @fatalbanana)
  702. * Make Exim pass hostnames to rspamd (by @fatalbanana)
  703. * Fix building on Fedora (by @fatalbanana)
  704. * Add toggle for disabling installation of systemd units on Linux (by @fatalbanana)
  705. * Fix double format rounding that caused output corruption (reported by @fatalbanana)
  706. * Revert broken change for destructors ordering that led to memory corruption
  707. * Do not reset symbols case of settings if parsed from lua (reported by @andrejzverev)
  708. * Fix build on SunOS (by @wiedi)
  709. * Fix multiple crashes on broken DKIM DNS records
  710. * Fix critical issue with composites weights removing
  711. * Fix memory corruption in composites processing code
  712. * Ignore non-SPF TXT records when parsing SPF includes
  713. 0.8.0:
  714. * New fuzzy check logic:
  715. - use shingles algorithm for fuzzy matching
  716. - use blake2 instead of md5 for larger output space
  717. - combine fuzzy and strict matching
  718. - allow to organize private storages by means of keys
  719. - preserve compatibility with previous versions
  720. * New fuzzy storage:
  721. - use sqlite instead of own memory based hash tables
  722. - rework commands interface
  723. - add conversion from the old format
  724. - add fuzzy match by shignles
  725. - support old rspamd versions
  726. * Add lemmatizing for words used in fuzzy hashes that allows to improve match
  727. quality by using of the first forms of all words
  728. * Rework language detection
  729. * Fix several critical bugs, memory leaks and deadlocks:
  730. - memory leak in HTML nodes parsing
  731. - deadlock in logger code
  732. - deadlock in signals processing
  733. - crashes in fuzzy_storage
  734. - crashes in tokenizers if the input was empty
  735. * Import new libucl with several bugfixes and improvements
  736. * Support listening on ipv6 addresses only
  737. * Fix macro expansion in SPF module
  738. * Several bugfixes in DKIM module
  739. * Add load headers support for mime parts to the lua API
  740. * Add documentation for:
  741. - workers in general
  742. - fuzzy_storage worker
  743. - fuzzy_check plugin
  744. - mimepart and textpart lua API modules
  745. 0.7.6:
  746. * Apply boundary fix for dkim simple canonization
  747. * Fix ping command
  748. * Return nil if header was not found in lua_task
  749. * Fix hang in upstreams revive logic
  750. * Decode entitles when normalizing HTML parts
  751. * Fix logic of finding URLs in HTML parts
  752. * Do not include \0 into length of text when performing conversion to utf8
  753. * Fix raw vs parsed reperesentations
  754. Raw parts are now:
  755. - decoded b64/qp, but *NOT* converted to utf-8
  756. Processed parts are now:
  757. - converted to UTF-8
  758. - normalized if needed (e.g. HTML tags are stripped)
  759. * Rework DKIM canonization to line based
  760. * Fix fuzzy hashes addding
  761. * Use more specific hash function for fuzzy
  762. * Fix leaking of iconv descriptors
  763. * Fix PTR resolving in lua resolver
  764. * Rework spf module.
  765. - Copy data to memory pool as cached record might be destroyed causing
  766. freed memory being passed to the protocol output (use after free)
  767. - Allow SPF_NEUTRAL policy to be handled separately
  768. - Add R_SPF_NEUTRAL to the default config
  769. * Rework `register_symbols` function
  770. * Allow to disable components of hfilter
  771. 0.7.5:
  772. * Fix owner when creating folder /run/rspamd (by @sfirmery)
  773. * Fix IP validity checks
  774. * Decode URLs obtained from HTML tags
  775. * Fix crash with unweighted upstreams
  776. * Stop processing headers in parts
  777. * Set sockaddr.sa_family properly when connectig to upstreams
  778. * Fix reload issues in surbl and fuzzy_check (reported by @citrin)
  779. * Fix timeouts in redirector
  780. * Improve lua errors reporting
  781. * Fix lua closures processing in libucl
  782. * Rework calling of lua functions from regexp module
  783. * Choose raw regexp for raw headers
  784. * Rework conversion to utf since glib one is broken
  785. * Ignore SGML style tags in html
  786. * Fix old bug with non-capturing https urls
  787. * Fix memory corruption on fuzzy reload (reported by @citrin)
  788. * Fix percents display in rspamc
  789. * Fix buffer update for DKIM
  790. * Do not validate utf for raw headers
  791. 0.7.4:
  792. * Fix build under *BSD
  793. * Detect HAN unicode script
  794. * Implement language detection heuristic for text parts
  795. * Fix time output in history
  796. * Improve piechart coloring
  797. * Fix \r\n conversion in DKIM module (reported by @citrin)
  798. * Try to detect systems with no IPv6 support
  799. * Fix multiple/single values in use settings (reported by @citrin)
  800. * Rework IP addresses in upstreams:
  801. - Select ipv4/unix addresses if they exist and use ipv6 for ipv6 only
  802. upstreams (since the support of ipv6 is poor in many OSes and
  803. environments)
  804. - Free IP list on upstream destruction
  805. - Add test cases for addresses selection
  806. - Allow adding of free form IP addresses to upstreams
  807. * Fix endiannes in lua_radix search (reported by @citrin)
  808. * Soft shutdown should also set wanna_die flag (reported by @citrin)
  809. * Stop use-after-free in event loop termination
  810. * Fix processing of very short messages in DKIM (reported by @citrin)
  811. * Detect systems without shared mutexes
  812. * Fix issues with PTR and MX elements in SPF parser (reported by @citrin)
  813. 0.7.3:
  814. * New upstreams code:
  815. - simplify upstreams API;
  816. - unify strings parsing in upstreams definition;
  817. - add configuration options for the upstreams;
  818. - for failed upstreams re-resolve their addresses;
  819. - use all resolved addresses for an upstream (round-robin);
  820. - implement stable hashing and use it by default for upstreams;
  821. - add unit test for upstreams module.
  822. * Rework signals processing in all rspamd workers:
  823. - signals are now processed in the event loop;
  824. - implement the most common signal handlers for all workers;
  825. - add callbacks for workers specific signal handlers
  826. * Fix critical issue with fuzzy storage:
  827. Fuzzy stroage could not save any hashes on termination due to bugged
  828. signals handling
  829. * Fix roll history IP storage
  830. * Rework ipv4/ipv6 handling in parsing addresses:
  831. - turn off support of IPV6_V6ONLY socket option;
  832. - create ipv6 socket prior to ipv4 one to handle systems with v6/v4
  833. sockets enabled (Linux)
  834. * Remove CBL as it's wholly included in Spamhaus XBL (by @fatalbanana)
  835. * Remove nszones.com fake RBL (by @citrin)
  836. * Fix upstreams interaction for fuzzy_check
  837. * Verify spf PTR records (reported by @citrin)
  838. * Fix spf MX records parsing
  839. * Add compatibility for old libevent (by @yellowman)
  840. * Sync bugfixes from libucl
  841. 0.7.2:
  842. * Convert all maps to the compressed radix trie
  843. * Allow IPv6 addresses in IP maps
  844. * Remove dynamic items support from symbols cache
  845. * Allow hex encoded output of strings
  846. * Fix bug with control connections count
  847. * Process fuzzy weight correctly (reported by @fatalbanana)
  848. * Remove extra reference retain of http connection on error
  849. * Remove deprecated options from the default config
  850. * Add `one_shot` attr to metric's symbols
  851. * Doc: add documentation for metrics
  852. * Add Upstart job to debian packaging (by @CameronNemo)
  853. * Config: improve SURBL symbols descriptions (by @citrin)
  854. * Config: reflect SURBL changes (by @citrin):
  855. - Outblaze removed, malware moved to separate list:
  856. http://www.surbl.org/news/internal/MW-malware-sublist-added-to-multi
  857. * Fix C modules initialization on restart
  858. * Treat single IP as a single IP in radix lists (reported by @citrin)
  859. * Do not touch file and core limits if not asked explicitly (reported by @citrin)
  860. * Improve logging for fuzzy errors
  861. * Block SIGPIPE for HTTP writing
  862. * Doc: update manual pages
  863. * Fix HTTP connection termination
  864. * Reduce default number of parallel requests to 8
  865. * Sync with libucl include features
  866. 0.7.1:
  867. * Fix typo in stat output.
  868. * Fix issues with includes crossing with the system includes
  869. * Restore testing framework
  870. * Add radix trie test suite
  871. * Implement new path-compressed radix trie.
  872. - The performance benefit over the old algorithm is about 1.5 times.
  873. - Memory usage is significantly lower as well.
  874. - Now radix trie can accept any IPv4/IPv6 values
  875. * Various improvements to the memory pools code
  876. * Fix writing reply to a client when no filters are defined
  877. * Write base32 encoded fuzzy
  878. * Fix 'soft reject' action
  879. * Fix rspamd reload and modules reconfiguration
  880. * Fix subject rewriting for the default subject
  881. * Fix states for processing task and pre-filters
  882. * Fix issues with connection closing
  883. * Fix crashes in rdns
  884. * Fix ratelimit pre-filter
  885. * Update exim patch.
  886. - Update to the recent exim version
  887. - Strip extra leading src/ from the patch
  888. - Remove sendfile since it was broken
  889. - Fix rspamd spam report for exim
  890. * Improve documentation
  891. 0.7.0:
  892. * Use HTTP protocol for all operatiosn
  893. * Webui worker is now removed and controller works as webui
  894. * Allow to serve static files via controller by option `static_dir`
  895. * Rspamd interface is now a part of rspamd
  896. * Rspamc client has been rewritten to use HTTP and non-blocking mode
  897. allowing to start multiple operations simultaneously (see `-n` option)
  898. * Lua API was completely reworked to satisfy modern standards of LUA:
  899. * Module `lua-message` was removed
  900. * Reduced number of superglobals registered by rspamd
  901. * Many functions has been redesigned
  902. * Symbols registration is now more convenient
  903. * Users settings has been rewritten as lua plugin
  904. * Reworked headers system as gmime's based one misses many headers and is
  905. very slow to get headers values
  906. * Reorganized code and removed embedded jannsson by using UCL for all json
  907. parsing
  908. * Migrated to `librdns` for DNS resolving that improves concurrency for
  909. DNS requests significantly
  910. * Fixed tonns of bugs in MIME processing
  911. * Improved metrcis and default symbol's weights
  912. * Added new RBL's
  913. * Fixed a number of issues in the modules
  914. * Removed several memory leaks found
  915. * Fix unicode processing
  916. * Fix fuzzy checking for unicode parts
  917. * Significantly improve documentation and especially LUA API docs
  918. * See migration notes at https://rspamd.com/doc/migration.html
  919. 0.6.8:
  920. * Controller now listen for localhost and not for 127.0.0.1 by default
  921. * Allow FCrDNS-style RBL lookups (by @fatalbanana)
  922. * Reduce threshold for parts_differ function.
  923. * Fix hostname lookup for rdns rbl (by @AlexeySa)
  924. * Fix HFILTER_URL_ONELINE to reduce false positive rate.
  925. * Fix whitelist module.
  926. * Allow override system predefined settings without touching system ones
  927. by .try_include macro (by @andrejzverev)
  928. * Check for [ip.address]-style HELO and suppress lookups. (by
  929. @fatalbanana)
  930. * Optimize hfilter (by @AlexeySa)
  931. * Fix issue with random numbers generator in dns.
  932. * Use more clever time values to setup entropy.
  933. * Synced with the recent libucl.
  934. 0.6.7:
  935. * Use ChaCha20 for DNS generator (more secure DNS id)
  936. * Unknown symbols now has zero weight and not 1.0
  937. * Fix fuzzy hashes expire time
  938. * Fix critical issue in statfiles on FreeBSD 9 (and some other platforms)
  939. * Add .include_map macro to ucl parser
  940. * Update libucl
  941. * Fix headers end detection for DKIM module
  942. * Fix a bug in received headers parser
  943. * Validate IP addresses before pushing them to lua
  944. * Start new documentation project
  945. * Fixed tonns of other minor bugs
  946. * Start to prepare for 0.7 with HTTP protocol and new settings
  947. 0.6.6:
  948. * Removed issue with BUFSIZ limitation in the controller output
  949. * Simplify logging symbols escaping
  950. * Adjusted weights for several rules
  951. * Improve spamhaus rbl support
  952. * Removed PBL for received headers checks
  953. * Added hfilter module that performs various HELO and IP checks.
  954. * Rspamd can now be reloaded by HUP signal
  955. * Fuzzy storage should expire hashes properly
  956. * Build system has been reworked for better supportof pkg-config
  957. * Various minor bugfixes
  958. 0.6.5:
  959. * Fixed critical bug in DNS resolver, introduced in 0.6.4
  960. * Improved multimap and rbl plugins to skip
  961. * Add dns_sockets option for tuning sockets per server in DNS resolver
  962. * Improved packages for rspamd
  963. 0.6.4:
  964. * Added io channels for DNS request to balance load and reduce id
  965. collisions chance
  966. * Fixed a bug in SPF filter that may cause core dump in specific
  967. circumstances
  968. * FIxed default config for rbl module
  969. * It is possible to get a list of rspamc commands with their descriptions
  970. * Added SORBS bl to the default config
  971. * 2tld file for surbl module has been significantly extended
  972. * Perl modules has been removed from the code.
  973. * Fixed an issue in libucl when parsing macros
  974. 0.6.3:
  975. * Fixed issues with DNS:
  976. - labels decompression algorithm was fixed;
  977. - added resolve_mx to lua API;
  978. - fixed modules that use DNS.
  979. * Lua modules once_received and emails reworked for new resolver API and UCL.
  980. * Debian package was polished.
  981. * Fixed a bug in fuzzy_check module that prevents correct processing messages
  982. without valid parts.
  983. 0.6.2:
  984. * Fuzzy check module has been reworked:
  985. - now fuzzy_check operates with a group of rules, that define which
  986. servers sre to be checked;
  987. - it is possible to specify read_only groups to avoid learning errors;
  988. - turn fuzzy_check to one_shot mode permanently;
  989. - fuzzy_check module configuration is now incompatible with the previous
  990. versions.
  991. * Imported bugfixes from libucl.
  992. * Fixed whitelist plugin.
  993. * Fixed statfiles resizing.
  994. * Improved logging initialization order.
  995. * Fixed race condition in the controller worker.
  996. 0.6.1:
  997. * Critical bugfixes:
  998. - fixed build system;
  999. - fixed in_class setting in bayes learning;
  1000. 0.6.0:
  1001. * Use UCL instead xml for configuration (https://github.com/vstakhov/libucl)
  1002. * Fix statistics module normalization
  1003. * Rework the many modules for the new configuration:
  1004. - surbl has incompatible configuration;
  1005. - fuzzy_check has incompatible configuration;
  1006. - multimap has now new configuration;
  1007. - received_rbl is removed and replaced by rbl module.
  1008. * Removed deprecated options:
  1009. - statfile_pool_size;
  1010. - action and required/reject score for a metric.
  1011. * Simplify build system and unify configuration for all platforms.
  1012. * Improved URL detector (reduced false positive rate).
  1013. * Lua 5.2 is now the default and fully supported version.
  1014. * Tons of bugfixes and minor improvements.
  1015. 0.5.6:
  1016. * Fix bug with counters incrementing in rolling history
  1017. * Detect expl and exp2l as some systems do not have it
  1018. * Support input streams without Content-Length
  1019. * Implement counters output via rspamc and controller interface
  1020. * Fix bug with udp sockets in fuzzy storage
  1021. 0.5.5:
  1022. * New bayes normalizator based on inverse chi-square function
  1023. * Various fixes to fuzzy storage
  1024. * Allow update fuzzy storage only from specific IP addresses
  1025. * Better support of IPv6 and address selection algorithms
  1026. * Add CentOS spec file
  1027. 0.5.4:
  1028. * Fixed issues with diff algorithm
  1029. * Added support of RRD statistics
  1030. * Add webui worker for interface interaction
  1031. * Fix a lot of issues with dynamic conf
  1032. * Fix critical memory leak in settings code
  1033. * Improve stability of the system
  1034. 0.5.3:
  1035. * Added dynamic options
  1036. * Added advanced metaclassfication
  1037. * Added RESTfull API for controller
  1038. * Improved hashing algorithms
  1039. * Various fixes for rspamc client:
  1040. - allow interacting with unix sockets
  1041. - librspamdclient major cleanup
  1042. - bayes is now default classifier
  1043. 0.5.2:
  1044. * Added lua bindings for:
  1045. - basic mime parts, that allows checkign attachements for example;
  1046. - DNS resolver;
  1047. * Existing lua bindings now works without task object allowing to use them
  1048. in custom code.
  1049. * Threads system was reworked to avoid global lua interpreter lock.
  1050. * DKIM module now converts all line endings to CRLF how opendkim does.
  1051. * URL detector is now more accurate for text parts.
  1052. * Several critical bugs and memory leaks were fixed.
  1053. 0.5.1:
  1054. * Added lua worker type to handle network connections in lua.
  1055. * Added lua bindings for async_session, IO dispatcher, memory_pool, and
  1056. worker.
  1057. * Composites can now uses other composites in expressions.
  1058. * Fixes for debian package and for FreeBSD >= 9.1
  1059. * Add support of gmime-2.6 if gmime-2.4 is not found.
  1060. * Improve url detection and phishing detection.
  1061. * Add lua mime_part library to get an access to all message part
  1062. attributes (like filename, length, type).
  1063. 0.5.0:
  1064. * Added SMTP lightweight balancing proxy with XCLIENT support.
  1065. * Added lua bindings for upstreams objects and API.
  1066. * New pre-filters are implemented to support initial checking for messages.
  1067. * Added ratelimit plugin that uses redis protocol to store data.
  1068. * Added ipv6 support to spf and some other modules.
  1069. * Unbreak spf plugin.
  1070. * Allow options with the same name be threated as list.
  1071. * DKIM plugin an parsing code was added.
  1072. * Separate build system to put logic in several shared libraries.
  1073. * Many bugfixes.
  1074. 0.3.2:
  1075. * Add error handling for regexps
  1076. * Fix quit command in controller interface
  1077. * Write symbols weights to rspamc output
  1078. * Improve logic of selecting rspamc version
  1079. * Do not try to parse broken DNS replies
  1080. * Add 'raw' flag to FROM_EXCESS_BASE64 rule (requested by citrin)
  1081. * Output message id in rspamc reply
  1082. * Fix inserting composite symbol
  1083. * Fix output of log line
  1084. * Document composites
  1085. * Add logging for fuzzy checks
  1086. * Add logging for learning
  1087. * Improve logic of learning messages - do not learn more than specific threshold
  1088. * Fix inserting results for symbols that were incorrectly (for example more than 1 time) defined in config file
  1089. * Do not output control characters if output is not terminal
  1090. * Fix some logic errors in learning
  1091. * Consider lua plugins errors as fatal configuration errors
  1092. * Fix wirting message id during fuzzy_add command
  1093. * Display weight of symbols correctly
  1094. * Fixes to winnow learning
  1095. * One more try to improve accuracy of winnow algorithm
  1096. * Add bayesian classifier (initial version)
  1097. * Remove normalizer as it is winnow specific thing, so all statistic algorithms now returns value from 0 to 1
  1098. * Some fixes to fuzzy hashes expiration:
  1099. * Fix assertion while look up value in NULL hash (found by cirtin)
  1100. * Fix normalization for systems that have not tanhl function
  1101. * Ignore rfc822 group addresses
  1102. * Move images library to core rspamd
  1103. * Add lua api to access images properties
  1104. * Add post filters to lua API - filters that would be called after all message's processing
  1105. * Add ability to check for specified symbol in task results from lua
  1106. * Add ability to check for metric's results from lua
  1107. * Add ability to learn specified statfile form lua
  1108. * Add ability to extract filename and size of images from lua
  1109. * Fix assertion while extracting internet address
  1110. * Fix races in fuzzy storage
  1111. * Make spf parser case insensitive
  1112. * Add ability to check hashes of selected mime types
  1113. * Add ability to set minimum size in bytes for mime types
  1114. * Add ability to set minimum dimensions for images
  1115. * Assume all text/* content types as text/plain
  1116. * Fix getting data wrapper for gmime24
  1117. * Many fixes to statfile syncronization system
  1118. * Fixed statfile pool initialization and synchronization with disk
  1119. * Prepare 0.3.2
  1120. * Fix Mail::Rspamd::Config for new rspamd features
  1121. * Use Mail::Rspamd::Config in rspamc client
  1122. * Write user's name to rspamd log
  1123. * Prepare rspamd build infrastructure for creating rpm and deb packages
  1124. * Fix depends
  1125. * Add start script for linux systems
  1126. * Fix shared usage of statfiles
  1127. * Add invalidation of statfiles in case of learning, so now statfiles
  1128. * This should fix shared usage of statfile pool by several processes
  1129. * Fix misprint (reported by az)
  1130. * Fix stupid error when all checks can be done in a single pass
  1131. * New trie based url scanner (based on libcamel)
  1132. * Small fixes to rspamd perl client
  1133. * Write fuzzy hashes info to log
  1134. * Add trie interface to lua api
  1135. * Explain sample config and cleanup it
  1136. 0.3.1:
  1137. * Add modules documentation
  1138. * Continue implementing smtp proxy
  1139. * Implement new learning system, now rspamd should be much more intelligent while learning messages
  1140. * Convert statistic sums to use long double for counters
  1141. * Use hyperbolic tangent for internal normalizer
  1142. * In classify normalize result after comparing, not before
  1143. * New symbols sorter
  1144. * Fix strict aliasing while compiling with optimization
  1145. * Fix tanhl detection for platforms that have not implementation of it
  1146. * Remove several compile warnings
  1147. * Add experimental support of dynamic rules to regexp module
  1148. * Document views configuration
  1149. * Several fixes to documentation
  1150. * Add more logic for dynamic rules
  1151. * Add documentation for dynamic rules
  1152. * Add ability to make negations in networks in dynamic rules
  1153. * Clean up cache items correctly
  1154. * Implement basic SMTP dialog:
  1155. * Implement interaction with smtp upstream (with support of XCLIENT)
  1156. * Check messages received via smtp proxy
  1157. * Add support for sendfile in io dispatcher
  1158. * Fix issues with compatibility of worker_task and smtp proxy
  1159. * Proxy DATA command
  1160. * Fix SMTP
  1161. * Change metric logic
  1162. * Completely remove lex/yacc readers for config
  1163. * Make common sense of metric/action and symbols
  1164. * Sync changes with all plugins
  1165. * Incorrectly removed in previous commit
  1166. * Fix misprint (by Andrej Zverev)
  1167. * announce the "password" keyword in usage list
  1168. * Implement initial version of greylisting triplets storage
  1169. * Fix issues with smtp worker
  1170. * Fix QUIT command in SMTP worker
  1171. * Some fixes about new metrics system (may be incomplete)
  1172. * Get weights of symbol from default metric for symbols cache
  1173. * Fix setting task->from/task->rctp in smtp client
  1174. * Copy from and rcpt correctly
  1175. * Some performance improvements to IO dispatcher (do not drain the whole buffer after a single line readed)
  1176. * Fix smtp data input
  1177. * Fix misprint
  1178. * Add limit of maximum allowed smtp session errors
  1179. * New logic of SURBL module:
  1180. * Use system mkstemp(3) on systems where it is available as glib implementation
  1181. * Try to fix memmove issues in io dispatcher
  1182. * Remove debug from SURBL module
  1183. * Rewrite buffered input for line policy (again)
  1184. * Fix issue with links that are ip addresses in numeric form in surbl
  1185. * On Darwin use BSD style sendfile definition
  1186. * Reorganize platform specific knobs in CMakeLists
  1187. * Use gettimeofday on systems that have not clock_getres
  1188. * Use ftime for dns trans id generation on systems without clock_getres
  1189. * Darwin sendfile(2) support
  1190. * TIMEDB->TIMEB
  1191. * More to previous commit
  1192. * Pass env from main() arguments instead of platform specific global environ
  1193. * Fix compatibility issues
  1194. * Fix -lintl detection
  1195. * Init some variables to avoid problems
  1196. * Remove garbadge (gnome terminal sucks)
  1197. * Add more information about why we drop smtp connection
  1198. * Fix mkstemp call
  1199. * Send to upstream QUIT command at the end of session
  1200. * Check return value of each rspamd_dispatcher_write as in case of write errors sessions can be destroyed early
  1201. * Fix states in smtp dialog
  1202. * Use rspamd_snprintf instead of libc one
  1203. * Fix URLS command
  1204. * Fix reconfigure process of surbl module
  1205. * Fix destroying smtp session (unmap memory and do not delete pool early)
  1206. * Delete pool after using its variables
  1207. * Delay timer must be registered in async session to correctly handle connection termination
  1208. * Register dns requests in session too
  1209. * Make session before registering events
  1210. * Remove events in handlers
  1211. * Add ability to set filters for smtp worker for each smtp stage
  1212. * Add very initial version of DNS resolver (many things to be done)
  1213. * Announce weights and sync
  1214. * Fix few typo
  1215. * Understand short names of facility in logging config
  1216. * Add ability to make whitelist for spf checks
  1217. * Misprint != -> ==
  1218. * Handle lua tag in way that it is not required to write additional text:
  1219. * Strip all starting whitespace symbols from xml texts
  1220. * Fix stupid bug in calculating buffer length while reading file maps
  1221. * Add resolv.conf parsing into dns.c
  1222. * Fix microseconds<->milliseconds conversions
  1223. * Take callback argument in Mail::Rspamd::Client for processing files and directories
  1224. * Print results if rspamc is called for a directory
  1225. * Fix stupid error with surbl module reconfig (another one, blame me)
  1226. * Do not show duplicate urls in url header
  1227. * Fix detection of numeric urls (reported by citrin)
  1228. * Write real time of message's scan to log (not only virtual)
  1229. * Fix chartable module in utf mode
  1230. * Fix parsing of some broken urls
  1231. * Add ability to test regexp with 'T' flag
  1232. * Write more code for DNS resolver:
  1233. * Make DNS resolver working
  1234. * Many improvements to rspamd test suite: now it CAN be used for testing rspamd functionality
  1235. * Write DNS resolver tests
  1236. * Fix issues with memory_pool mutexes and with creating of statfiles
  1237. * Forgotten in previous commit
  1238. * Add support for parsing SPF and SRV records
  1239. * Fix PTR parsing
  1240. * Add tests
  1241. * Make SURBL module to use rspamd dns module
  1242. * Several fixes to DNS logic
  1243. * Remove evdns and use only rspamd resolver
  1244. * Very hard to detect problem with race among error in socket and destroying task while we are writing to socket and go through a hash table
  1245. * Fix resolving in smtp module
  1246. * Init events before configuring resolver in smtp worker
  1247. * Set resolver inside task
  1248. * Fix reload signal (reported by citrin)
  1249. * Some improvements to redirector
  1250. * Call has_forked method to inform POE about fork
  1251. * Fix lua DNS code
  1252. * Decompress labels in DNS packets more strictly
  1253. * Fix some problems with TXT records
  1254. * Try to fix removing of DNS events
  1255. * Do not insert unparsed RR's into reply
  1256. * Calling callbacks may cause destroying session from which we are calling callback so we MUST call callback as the latest action
  1257. * Fix check_smtp_data function
  1258. * Add ability to make views by recipient
  1259. * Add ability to set metric's action from config file
  1260. * Fix bug with writing garbadge if message has no urls or no messages
  1261. * Fix bug with incorrect behaviour of compare_parts_distance function
  1262. * Add ability to assign several actions to one metric
  1263. * Report action in rspamc protocol
  1264. * Mail::Rspamd::Client and rspamc can now understand Action header too
  1265. * Write action to log as well
  1266. * Make valgrind happy about comparing symbols
  1267. * Add more debug to comparing parts distance function
  1268. * Write action even if message has no symbols
  1269. * Make improvements to HTML entites decoder: now it replaces entities with common characters and
  1270. * Add -d option to force debug output
  1271. * Assume 7bit as default transfer encoding
  1272. * Do not overwrite lua plugins and configs if they already exists in target directory
  1273. * Improve logging
  1274. * Write queue id to log
  1275. * Remove test messages from cmake
  1276. * Reopen log file by USR1 signal
  1277. * Add reopenlog method to FreeBSD rc script
  1278. * Adopt foreach for cmake 2.6
  1279. * Fix to rc script
  1280. * Do not try to resolve names with several dots in a row
  1281. * Fix surbl request formatting for ip addresses
  1282. * Handle cases of broken requests
  1283. * Fix problems with parsing compressed names
  1284. * Fix TXT records parsing
  1285. * Fix expanding spf macros that may fail in rare cases
  1286. * Fix another error with early task destroying
  1287. * Handle empty from header
  1288. * Improve reopenlog command in rc script
  1289. * Strip trailing whitespace characters in Mail::Rspamd::Client
  1290. * Use ungreedy match to strip trailing whitespaces
  1291. * Stupid error in calculation compressed label length
  1292. * Some optimizations to client library
  1293. * Do not compare empty parts
  1294. * Empty and non-empty parts are allways different
  1295. * Save in regexp cache the whole regexp with header name (if exists) and with flags
  1296. * Add rspamd_log variable to lua plugins to access logging functions
  1297. * Each part in rspamd task now can have parent part
  1298. * Check for parts distance only for multipart/alternative subparts
  1299. * Do not check attachements even if they are text (but attached as file)
  1300. * Do not die if write (2) returned ENOSPACE while doing logging, turn on throttling mode instead (1 write try in a second)
  1301. * Add ability to turn on debug for specific symbols
  1302. * Add ability to configure dns timeouts and dns retransmits in config file
  1303. * More debug
  1304. * Fix extracting arguments in lua logger interface
  1305. * Turn off debug_ip during reload if it was disabled by new config
  1306. * Improve lua logging
  1307. * Pre-init symbols cache when rereading config
  1308. * Fix lua representing of invalid ip (nil, not 255.255.255.255)
  1309. * Fix R_TO_SEEMS_AUTO rule (by citrin)
  1310. * Add multimap lua plugin
  1311. * Fix some multimap issues
  1312. * Try to save images hashes to fuzzy storage to stop some annoying spammers
  1313. * Allocate some more bytes for read buffer to avoid incorrect behavoiur
  1314. * Add ability to check dns black lists by multimap module
  1315. * Add multimap documentation
  1316. * Fix labels parsing
  1317. * Another try to save regexps in cache correctly
  1318. * Improve test logs for regexps
  1319. * Fix parsing txt records to avoid reading of uninitialized data
  1320. * Fix error with writing symbols cache file
  1321. * Fix error while working in utf mode when raw regexps was not created properly
  1322. * Do not add extra byte while converting text to utf
  1323. * Add error handling for regexps