From 2a1fa7d08ec1703fd47b37ab014699f001fb3344 Mon Sep 17 00:00:00 2001 From: Ivan Stakhov <50211739+LeftTry@users.noreply.github.com> Date: Mon, 3 Jun 2024 15:29:11 +0500 Subject: Upgraded replies and known senders modules (#4895) * Made the individual replies_set for senders and their recipients. Made the global replies_set for verified recipients. * Made the individual replies_set for senders and their recipients. Made the global replies_set for verified recipients. * FIXED. Made the individual replies_set for senders and their recipients. Made the global replies_set for verified recipients. * Made the individual replies_set for senders and their recipients. Made the global replies_set for verified recipients. * Added pre-test for replies set * Update functional of replies_set * Few changes to replies and added check for incoming mail * Few changes in known_senders in check_known_incoming_mail_callback * Few changes in known_senders and replies * An attempt to write test(not tested) * Clean up * Clean up * Clean up * Added tests for replies and known_senders (all tests failed, debug required) * Moved replies test to the 001_merged * Cleared up code * Few changes to replies * Small changes in score of CHECK_INC_MAIL symbol * Small debug in known_senders * Plugin known_senders is fully working * Troubleshooting replies module * Changed symbol for check_known_incoming_mail_callback * Added test for failed incoming mail check * Little rework * Rewritten test for more appropriate * Rewritten tests for replies module. All test passed(debugging not adding to global set) * Debugged replies module * Replies module works and tested(needs performance improvements) * Cleaned up code * Improved readability and cleaned up code * Connected auth back(Tests not working, needs user) * Added test for incoming mail check in known senders module * Debugged. Works normally(tested, needs to add user) * Debug + clean up. Tested. Works. User auth required for tests * Improved performance * Small changes * Changed adding to global replies set logic + improved logs messaging * Added authenticated user to tests * Cleaned up * Made a few changes according to the comments on pull request * [Rework] Added removal of extra senders and recipients in global and local replies sets * [Minor] Small cleanup * [Minor] Cleaned up code * [Fix] Fixed call of incorrect function when making key * [Rework] Reworked scripts. Added ZADD redis call for local and global replies set * [Minor] Cleaned up code * [Fix] Improved performance and eliminated unnecessary invocations * [Minor] Reassigned script ids * [Feature] Made a check for local set * [Fix] Upgraded tests for known senders * [Fix] Upgraded tests for known senders * [Fix] Fixed performance of verification of local replies set * [Minor] Cleaned up code * [Feature] Added new test to the known_senders tests * [Test] Ubuntu test * [Fix] Fixing local replies test check * [Fix] Fixed code for local replies set check(was not working in previous versions of redis) * [Fix] Reorganized code to more convenient style and made better loading for scripts * [Minor] Code has been rewritten in a more appropriate format * [Minor] Fixed debug messaging * [Fix] Reworked expiration of replies sets * [Minor] Upgrade code style * [Fix] Small fix * [Feature] Change LFU logic of global replies set to LRU logic * Made the individual replies_set for senders and their recipients. Made the global replies_set for verified recipients. * Made the individual replies_set for senders and their recipients. Made the global replies_set for verified recipients. * FIXED. Made the individual replies_set for senders and their recipients. Made the global replies_set for verified recipients. * Made the individual replies_set for senders and their recipients. Made the global replies_set for verified recipients. * Added pre-test for replies set * Update functional of replies_set * Few changes to replies and added check for incoming mail * Few changes in known_senders in check_known_incoming_mail_callback * Few changes in known_senders and replies * An attempt to write test(not tested) * Clean up * Clean up * Clean up * Added tests for replies and known_senders (all tests failed, debug required) * Moved replies test to the 001_merged * Cleared up code * Few changes to replies * Small changes in score of CHECK_INC_MAIL symbol * Small debug in known_senders * Plugin known_senders is fully working * Troubleshooting replies module * Changed symbol for check_known_incoming_mail_callback * Added test for failed incoming mail check * Little rework * Rewritten test for more appropriate * Rewritten tests for replies module. All test passed(debugging not adding to global set) * Debugged replies module * Replies module works and tested(needs performance improvements) * Cleaned up code * Improved readability and cleaned up code * Connected auth back(Tests not working, needs user) * Added test for incoming mail check in known senders module * Debugged. Works normally(tested, needs to add user) * Debug + clean up. Tested. Works. User auth required for tests * Improved performance * Small changes * Changed adding to global replies set logic + improved logs messaging * Added authenticated user to tests * Cleaned up * Made a few changes according to the comments on pull request * [Rework] Added removal of extra senders and recipients in global and local replies sets * [Minor] Small cleanup * [Minor] Cleaned up code * [Fix] Fixed call of incorrect function when making key * [Rework] Reworked scripts. Added ZADD redis call for local and global replies set * [Minor] Cleaned up code * [Fix] Improved performance and eliminated unnecessary invocations * [Minor] Reassigned script ids * [Feature] Made a check for local set * [Fix] Upgraded tests for known senders * [Fix] Upgraded tests for known senders * [Fix] Fixed performance of verification of local replies set * [Minor] Cleaned up code * [Feature] Added new test to the known_senders tests * [Test] Ubuntu test * [Fix] Fixing local replies test check * [Fix] Fixed code for local replies set check(was not working in previous versions of redis) * [Fix] Reorganized code to more convenient style and made better loading for scripts * [Minor] Code has been rewritten in a more appropriate format * [Minor] Fixed debug messaging * [Fix] Reworked expiration of replies sets * [Minor] Upgrade code style * [Fix] Small fix * [Feature] Change LFU logic of global replies set to LRU logic * [Fix] Fix test conflict * [Minor] Revert rename * [Minor] Clean up code * [Fix] Fix commit history --- test/functional/cases/400_known_senders.robot | 39 ++++++++++++++++++ test/functional/cases/410_replies.robot | 47 ++++++++++++++++++++++ test/functional/configs/merged.conf | 1 + test/functional/configs/replies.conf | 6 +++ test/functional/messages/inc_mail_known_sender.eml | 11 +++++ .../messages/inc_mail_unknown_sender.eml | 11 +++++ test/functional/messages/replyto_1_1.eml | 11 +++++ test/functional/messages/replyto_1_2.eml | 11 +++++ test/functional/messages/replyto_1_2_s.eml | 11 +++++ test/functional/messages/replyto_2_2.eml | 11 +++++ test/functional/messages/set_replyto_1_1.eml | 12 ++++++ test/functional/messages/set_replyto_1_2_first.eml | 11 +++++ test/functional/messages/set_replyto_2_2.eml | 11 +++++ 13 files changed, 193 insertions(+) create mode 100644 test/functional/cases/410_replies.robot create mode 100644 test/functional/configs/replies.conf create mode 100644 test/functional/messages/inc_mail_known_sender.eml create mode 100644 test/functional/messages/inc_mail_unknown_sender.eml create mode 100644 test/functional/messages/replyto_1_1.eml create mode 100644 test/functional/messages/replyto_1_2.eml create mode 100644 test/functional/messages/replyto_1_2_s.eml create mode 100644 test/functional/messages/replyto_2_2.eml create mode 100644 test/functional/messages/set_replyto_1_1.eml create mode 100644 test/functional/messages/set_replyto_1_2_first.eml create mode 100644 test/functional/messages/set_replyto_2_2.eml (limited to 'test') diff --git a/test/functional/cases/400_known_senders.robot b/test/functional/cases/400_known_senders.robot index f258113da..d827acc0e 100644 --- a/test/functional/cases/400_known_senders.robot +++ b/test/functional/cases/400_known_senders.robot @@ -7,6 +7,9 @@ Variables ${RSPAMD_TESTDIR}/lib/vars.py *** Variables *** ${CONFIG} ${RSPAMD_TESTDIR}/configs/known_senders.conf +${SETTINGS_REPLIES} {symbols_enabled = [REPLIES_CHECK, REPLIES_SET, REPLY]} +${SYMBOL_GLOBAL} INC_MAIL_KNOWN_GLOBALLY +${SYMBOL_LOCAL} INC_MAIL_KNOWN_LOCALLY ${REDIS_SCOPE} Suite ${RSPAMD_SCOPE} Suite @@ -34,3 +37,39 @@ UNKNOWN SENDER WRONG DOMAIN RESCAN ... Settings={symbols_enabled [KNOWN_SENDER]} Do Not Expect Symbol KNOWN_SENDER Do Not Expect Symbol UNKNOWN_SENDER + +INCOMING MAIL SENDER IS UNKNOWN + Scan File ${RSPAMD_TESTDIR}/messages/inc_mail_unknown_sender.eml + ... Settings={symbols_enabled [${SYMBOL_GLOBAL}, ${SYMBOL_LOCAL}]} + Do Not Expect Symbol ${SYMBOL_GLOBAL} + Do Not Expect Symbol ${SYMBOL_LOCAL} + +INCOMING MAIL SENDER IS KNOWN RECIPIENTS ARE UNKNOWN + Scan File ${RSPAMD_TESTDIR}/messages/set_replyto_1_1.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Scan File ${RSPAMD_TESTDIR}/messages/replyto_1_1.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Scan File ${RSPAMD_TESTDIR}/messages/inc_mail_known_sender.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings={symbols_enabled [${SYMBOL_GLOBAL}, ${SYMBOL_LOCAL}]} + Expect Symbol ${SYMBOL_GLOBAL} + Do Not Expect Symbol ${SYMBOL_LOCAL} + +INCOMING MAIL SENDER IS KNOWN RECIPIENTS ARE KNOWN + Scan File ${RSPAMD_TESTDIR}/messages/set_replyto_1_1.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Scan File ${RSPAMD_TESTDIR}/messages/replyto_1_1.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Scan File ${RSPAMD_TESTDIR}/messages/inc_mail_known_sender.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Scan File ${RSPAMD_TESTDIR}/messages/inc_mail_known_sender.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings={symbols_enabled [${SYMBOL_GLOBAL}, ${SYMBOL_LOCAL}]} + Expect Symbol ${SYMBOL_GLOBAL} + Expect Symbol ${SYMBOL_LOCAL} + diff --git a/test/functional/cases/410_replies.robot b/test/functional/cases/410_replies.robot new file mode 100644 index 000000000..23ad9df35 --- /dev/null +++ b/test/functional/cases/410_replies.robot @@ -0,0 +1,47 @@ +*** Settings *** +Suite Setup Rspamd Redis Setup +Suite Teardown Rspamd Redis Teardown +Library ${RSPAMD_TESTDIR}/lib/rspamd.py +Resource ${RSPAMD_TESTDIR}/lib/rspamd.robot +Variables ${RSPAMD_TESTDIR}/lib/vars.py + +*** Variables *** +${CONFIG} ${RSPAMD_TESTDIR}/configs/replies.conf +${SETTINGS_REPLIES} {symbols_enabled = [REPLIES_CHECK, REPLIES_SET, REPLY]} +${SYMBOL} REPLY +${REDIS_SCOPE} Suite +${RSPAMD_SCOPE} Suite + +*** Test Cases *** +Reply to 1 sender 1 recipients + Scan File ${RSPAMD_TESTDIR}/messages/set_replyto_1_1.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Scan File ${RSPAMD_TESTDIR}/messages/replyto_1_1.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Expect Symbol ${SYMBOL} + +Reply to 1 sender 2 recipients first is set second is not + Scan File ${RSPAMD_TESTDIR}/messages/set_replyto_1_2_first.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Scan File ${RSPAMD_TESTDIR}/messages/replyto_1_2.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Expect Symbol ${SYMBOL} + +Reply to 1 sender 2 recipients 1 rcpt is same + Scan File ${RSPAMD_TESTDIR}/messages/replyto_1_2_s.eml + ... IP=8.8.8.8 User=user@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Expect Symbol ${SYMBOL} + +Reply to another sender 2 recipients + Scan File ${RSPAMD_TESTDIR}/messages/set_replyto_2_2.eml + ... IP=8.8.8.8 User=another@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Scan File ${RSPAMD_TESTDIR}/messages/replyto_2_2.eml + ... IP=8.8.8.8 User=another@emailbl.com + ... Settings=${SETTINGS_REPLIES} + Expect Symbol ${SYMBOL} diff --git a/test/functional/configs/merged.conf b/test/functional/configs/merged.conf index 0718d0210..2b3640048 100644 --- a/test/functional/configs/merged.conf +++ b/test/functional/configs/merged.conf @@ -35,5 +35,6 @@ lua = "{= env.TESTDIR =}/lua/magic.lua" # 380_external_relay lua = "{= env.TESTDIR =}/lua/external_relay.lua" + .include(priority=1,duplicate=merge) "{= env.TESTDIR =}/configs/merged-local.conf" .include(priority=2,duplicate=replace) "{= env.TESTDIR =}/configs/merged-override.conf" diff --git a/test/functional/configs/replies.conf b/test/functional/configs/replies.conf new file mode 100644 index 000000000..e4d9b34f5 --- /dev/null +++ b/test/functional/configs/replies.conf @@ -0,0 +1,6 @@ +.include "{= env.TESTDIR =}/../../conf/rspamd.conf" + +lua = "{= env.TESTDIR =}/lua/test_coverage.lua" + +.include(priority=1,duplicate=merge) "{= env.TESTDIR =}/configs/merged-local.conf" +.include(priority=2,duplicate=replace) "{= env.TESTDIR =}/configs/merged-override.conf" \ No newline at end of file diff --git a/test/functional/messages/inc_mail_known_sender.eml b/test/functional/messages/inc_mail_known_sender.eml new file mode 100644 index 000000000..29f4cbeb7 --- /dev/null +++ b/test/functional/messages/inc_mail_known_sender.eml @@ -0,0 +1,11 @@ +in-reply-to: 00020 +message-id: 000200 +From: +To: , +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test diff --git a/test/functional/messages/inc_mail_unknown_sender.eml b/test/functional/messages/inc_mail_unknown_sender.eml new file mode 100644 index 000000000..1643e8f66 --- /dev/null +++ b/test/functional/messages/inc_mail_unknown_sender.eml @@ -0,0 +1,11 @@ +in-reply-to: 00020 +message-id: 000200 +From: +To: +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test diff --git a/test/functional/messages/replyto_1_1.eml b/test/functional/messages/replyto_1_1.eml new file mode 100644 index 000000000..ff3c75417 --- /dev/null +++ b/test/functional/messages/replyto_1_1.eml @@ -0,0 +1,11 @@ +in-reply-to: 0002 +message-id: 00020 +From: +To: +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test diff --git a/test/functional/messages/replyto_1_2.eml b/test/functional/messages/replyto_1_2.eml new file mode 100644 index 000000000..5acbd584e --- /dev/null +++ b/test/functional/messages/replyto_1_2.eml @@ -0,0 +1,11 @@ +in-reply-to: 0012 +message-id: 00120 +From: +To: , +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test \ No newline at end of file diff --git a/test/functional/messages/replyto_1_2_s.eml b/test/functional/messages/replyto_1_2_s.eml new file mode 100644 index 000000000..11546939b --- /dev/null +++ b/test/functional/messages/replyto_1_2_s.eml @@ -0,0 +1,11 @@ +in-reply-to: 0012 +message-id: 00121 +From: +To: , +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test \ No newline at end of file diff --git a/test/functional/messages/replyto_2_2.eml b/test/functional/messages/replyto_2_2.eml new file mode 100644 index 000000000..de56ecbf1 --- /dev/null +++ b/test/functional/messages/replyto_2_2.eml @@ -0,0 +1,11 @@ +in-reply-to: 0022 +message-id: 00220 +From: +To: , +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test \ No newline at end of file diff --git a/test/functional/messages/set_replyto_1_1.eml b/test/functional/messages/set_replyto_1_1.eml new file mode 100644 index 000000000..267d809d7 --- /dev/null +++ b/test/functional/messages/set_replyto_1_1.eml @@ -0,0 +1,12 @@ +in-reply-to: 0002 +message-id: 0002 +From: +To: +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test + diff --git a/test/functional/messages/set_replyto_1_2_first.eml b/test/functional/messages/set_replyto_1_2_first.eml new file mode 100644 index 000000000..0985a2d98 --- /dev/null +++ b/test/functional/messages/set_replyto_1_2_first.eml @@ -0,0 +1,11 @@ +in-reply-to: 0012 +message-id: 0012 +From: +To: +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test diff --git a/test/functional/messages/set_replyto_2_2.eml b/test/functional/messages/set_replyto_2_2.eml new file mode 100644 index 000000000..63e7d7a6c --- /dev/null +++ b/test/functional/messages/set_replyto_2_2.eml @@ -0,0 +1,11 @@ +in-reply-to: 0022 +message-id: 0022 +From: +To: +Subject: 06.07.2015 +Date: Mon, 6 Jul 2015 10:35:56 +0200 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="Windows-1251" + +test \ No newline at end of file -- cgit v1.2.3