From 478a75e719c8c8bf2333465a34ccf3991746d141 Mon Sep 17 00:00:00 2001 From: Andrew Lewis Date: Sun, 14 Aug 2016 17:28:46 +0200 Subject: [PATCH] [Test] Rework fuzzy tests - Add fuzzy & negative match tests - Check attachment & text samples - Add suites for different algorithms; keyed/unkeyed - Move overwrite/delete tests into general suite --- .../cases/120_fuzzy/encrypted.robot | 18 +-- .../cases/120_fuzzy/fasthash-keyed.robot | 14 +++ .../functional/cases/120_fuzzy/fasthash.robot | 14 +++ test/functional/cases/120_fuzzy/general.robot | 14 +++ test/functional/cases/120_fuzzy/lib.robot | 117 ++++++++++++++++-- .../cases/120_fuzzy/mumhash-keyed.robot | 14 +++ test/functional/cases/120_fuzzy/mumhash.robot | 14 +++ test/functional/cases/120_fuzzy/plain.robot | 18 --- .../cases/120_fuzzy/siphash-keyed.robot | 14 +++ test/functional/cases/120_fuzzy/siphash.robot | 14 +++ .../cases/120_fuzzy/xxhash-keyed.robot | 14 +++ test/functional/cases/120_fuzzy/xxhash.robot | 14 +++ test/functional/configs/fuzzy.conf | 1 + test/functional/lib/rspamd.py | 5 + test/functional/lib/rspamd.robot | 2 +- .../messages/spam_message.eml.fuzzy1 | 44 +++++++ 16 files changed, 290 insertions(+), 41 deletions(-) create mode 100644 test/functional/cases/120_fuzzy/fasthash-keyed.robot create mode 100644 test/functional/cases/120_fuzzy/fasthash.robot create mode 100644 test/functional/cases/120_fuzzy/general.robot create mode 100644 test/functional/cases/120_fuzzy/mumhash-keyed.robot create mode 100644 test/functional/cases/120_fuzzy/mumhash.robot delete mode 100644 test/functional/cases/120_fuzzy/plain.robot create mode 100644 test/functional/cases/120_fuzzy/siphash-keyed.robot create mode 100644 test/functional/cases/120_fuzzy/siphash.robot create mode 100644 test/functional/cases/120_fuzzy/xxhash-keyed.robot create mode 100644 test/functional/cases/120_fuzzy/xxhash.robot create mode 100644 test/functional/messages/spam_message.eml.fuzzy1 diff --git a/test/functional/cases/120_fuzzy/encrypted.robot b/test/functional/cases/120_fuzzy/encrypted.robot index 45408d7f8..21fd87908 100644 --- a/test/functional/cases/120_fuzzy/encrypted.robot +++ b/test/functional/cases/120_fuzzy/encrypted.robot @@ -1,20 +1,14 @@ *** Settings *** -Suite Setup Encrypted Fuzzy Setup +Suite Setup Fuzzy Setup Encrypted Siphash Suite Teardown Generic Teardown Resource lib.robot *** Test Cases *** Fuzzy Add - Fuzzy Add Test + Fuzzy Multimessage Add Test -Fuzzy Delete - Fuzzy Delete Test +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test -Fuzzy Overwrite - Fuzzy Overwrite Test - -*** Keywords *** -Encrypted Fuzzy Setup - Set Suite Variable ${SETTINGS_FUZZY_WORKER} "keypair": {"pubkey": "${KEY_PUB1}", "privkey": "${KEY_PVT1}"}; "encrypted_only": true; - Set Suite Variable ${SETTINGS_FUZZY_CHECK} encryption_key = "${KEY_PUB1}"; - Generic Setup +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/cases/120_fuzzy/fasthash-keyed.robot b/test/functional/cases/120_fuzzy/fasthash-keyed.robot new file mode 100644 index 000000000..d135eaca0 --- /dev/null +++ b/test/functional/cases/120_fuzzy/fasthash-keyed.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Keyed Fasthash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test + +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/cases/120_fuzzy/fasthash.robot b/test/functional/cases/120_fuzzy/fasthash.robot new file mode 100644 index 000000000..75d596c73 --- /dev/null +++ b/test/functional/cases/120_fuzzy/fasthash.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Plain Fasthash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test + +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/cases/120_fuzzy/general.robot b/test/functional/cases/120_fuzzy/general.robot new file mode 100644 index 000000000..7d1b0171a --- /dev/null +++ b/test/functional/cases/120_fuzzy/general.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Plain Siphash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Delete + Fuzzy Multimessage Delete Test + +Fuzzy Overwrite + Fuzzy Multimessage Overwrite Test diff --git a/test/functional/cases/120_fuzzy/lib.robot b/test/functional/cases/120_fuzzy/lib.robot index e5e9cbcb7..bde6e2dc5 100644 --- a/test/functional/cases/120_fuzzy/lib.robot +++ b/test/functional/cases/120_fuzzy/lib.robot @@ -1,48 +1,145 @@ *** Settings *** +Library OperatingSystem Library ${TESTDIR}/lib/rspamd.py Resource ${TESTDIR}/lib/rspamd.robot Variables ${TESTDIR}/lib/vars.py *** Variables *** +${ALGORITHM} ${EMPTY} ${CONFIG} ${TESTDIR}/configs/fuzzy.conf ${FLAG1_NUMBER} 50 ${FLAG1_SYMBOL} R_TEST_FUZZY_DENIED ${FLAG2_NUMBER} 51 ${FLAG2_SYMBOL} R_TEST_FUZZY_WHITE -${MESSAGE} ${TESTDIR}/messages/bad_message.eml +@{MESSAGES} ${TESTDIR}/messages/spam_message.eml ${TESTDIR}/messages/zip.eml +@{RANDOM_MESSAGES} ${TESTDIR}/messages/bad_message.eml ${TESTDIR}/messages/zip-doublebad.eml ${RSPAMD_SCOPE} Suite +${SETTINGS_FUZZY_WORKER} ${EMPTY} +${SETTINGS_FUZZY_CHECK} ${EMPTY} *** Keywords *** Fuzzy Add Test - Set Suite Variable ${RSPAMD_FUZZY_ADD} 0 + [Arguments] ${message} + Set Suite Variable ${RSPAMD_FUZZY_ADD_${message}} 0 ${result} = Run Rspamc -h ${LOCAL_ADDR}:${PORT_CONTROLLER} -w 10 -f - ... ${FLAG1_NUMBER} fuzzy_add ${MESSAGE} + ... ${FLAG1_NUMBER} fuzzy_add ${message} Check Rspamc ${result} Sync Fuzzy Storage - ${result} = Scan Message With Rspamc ${MESSAGE} + ${result} = Scan Message With Rspamc ${message} Check Rspamc ${result} ${FLAG1_SYMBOL} - Set Suite Variable ${RSPAMD_FUZZY_ADD} 1 + Set Suite Variable ${RSPAMD_FUZZY_ADD_${message}} 1 Fuzzy Delete Test - Run Keyword If ${RSPAMD_FUZZY_ADD} == 0 Fail "Fuzzy Add was not run" + [Arguments] ${message} + Run Keyword If ${RSPAMD_FUZZY_ADD_${message}} == 0 Fail "Fuzzy Add was not run" ${result} = Run Rspamc -h ${LOCAL_ADDR}:${PORT_CONTROLLER} -f ${FLAG1_NUMBER} fuzzy_del - ... ${MESSAGE} + ... ${message} Check Rspamc ${result} Sync Fuzzy Storage - ${result} = Scan Message With Rspamc ${MESSAGE} + ${result} = Scan Message With Rspamc ${message} Follow Rspamd Log Should Not Contain ${result.stdout} ${FLAG1_SYMBOL} Should Be Equal As Integers ${result.rc} 0 +Fuzzy Fuzzy Test + [Arguments] ${message} + Run Keyword If ${RSPAMD_FUZZY_ADD_${message}} != 1 Fail "Fuzzy Add was not run" + @{path_info} = Path Splitter ${message} + @{fuzzy_files} = List Files In Directory @{pathinfo}[0] @{pathinfo][1].fuzzy* absolute=1 + : FOR ${i} IN @{fuzzy_files} + \ ${result} = Scan Message With Rspamc ${i} + \ Check Rspamc ${result} ${FLAG1_SYMBOL} + +Fuzzy Miss Test + [Arguments] ${message} + ${result} = Scan Message With Rspamc ${message} + Check Rspamc ${result} ${FLAG1_SYMBOL} inverse=1 + Fuzzy Overwrite Test + [Arguments] ${message} ${flag_numbers} = Create List ${FLAG1_NUMBER} ${FLAG2_NUMBER} : FOR ${i} IN @{flag_numbers} \ ${result} = Run Rspamc -h ${LOCAL_ADDR}:${PORT_CONTROLLER} -w 10 - \ ... -f ${i} fuzzy_add ${MESSAGE} + \ ... -f ${i} fuzzy_add ${message} \ Check Rspamc ${result} Sync Fuzzy Storage - ${result} = Scan Message With Rspamc ${MESSAGE} + ${result} = Scan Message With Rspamc ${message} Follow Rspamd Log Should Not Contain ${result.stdout} ${FLAG1_SYMBOL} Should Contain ${result.stdout} ${FLAG2_SYMBOL} Should Be Equal As Integers ${result.rc} 0 + +Fuzzy Setup Encrypted + [Arguments] ${algorithm} + ${worker_settings} = Set Variable "keypair": {"pubkey": "${KEY_PUB1}", "privkey": "${KEY_PVT1}"}; "encrypted_only": true; + ${check_settings} = Set Variable encryption_key = "${KEY_PUB1}"; + Fuzzy Setup Generic ${algorithm} ${worker_settings} ${check_settings} + +Fuzzy Setup Encrypted Keyed + [Arguments] ${algorithm} + ${worker_settings} = Set Variable "keypair": {"pubkey": "${KEY_PUB1}", "privkey": "${KEY_PVT1}"}; "encrypted_only": true; + ${check_settings} = Set Variable fuzzy_key = "mYN888sydwLTfE32g2hN"; fuzzy_shingles_key = "hXUCgul9yYY3Zlk1QIT2"; encryption_key = "${KEY_PUB1}"; + Fuzzy Setup Generic ${algorithm} ${worker_settings} ${check_settings} + +Fuzzy Setup Plain + [Arguments] ${algorithm} + Fuzzy Setup Generic ${algorithm} ${EMPTY} ${EMPTY} + +Fuzzy Setup Keyed + [Arguments] ${algorithm} + ${check_settings} = Set Variable fuzzy_key = "mYN888sydwLTfE32g2hN"; fuzzy_shingles_key = "hXUCgul9yYY3Zlk1QIT2"; + Fuzzy Setup Generic ${algorithm} ${EMPTY} ${check_settings} + +Fuzzy Setup Generic + [Arguments] ${algorithm} ${worker_settings} ${check_settings} + Set Suite Variable ${SETTINGS_FUZZY_WORKER} ${worker_settings} + Set Suite Variable ${SETTINGS_FUZZY_CHECK} ${check_settings} + Set Suite Variable ${ALGORITHM} ${algorithm} + Generic Setup + +Fuzzy Setup Plain Fasthash + Fuzzy Setup Plain fasthash + +Fuzzy Setup Plain Mumhash + Fuzzy Setup Plain mumhash + +Fuzzy Setup Plain Siphash + Fuzzy Setup Plain siphash + +Fuzzy Setup Plain Xxhash + Fuzzy Setup Plain xxhash + +Fuzzy Setup Keyed Fasthash + Fuzzy Setup Keyed fasthash + +Fuzzy Setup Keyed Mumhash + Fuzzy Setup Keyed mumhash + +Fuzzy Setup Keyed Siphash + Fuzzy Setup Keyed siphash + +Fuzzy Setup Keyed Xxhash + Fuzzy Setup Keyed xxhash + +Fuzzy Setup Encrypted Siphash + Fuzzy Setup Encrypted siphash + +Fuzzy Multimessage Add Test + : FOR ${i} IN @{MESSAGES} + \ Fuzzy Add Test ${i} + +Fuzzy Multimessage Fuzzy Test + : FOR ${i} IN @{MESSAGES} + \ Fuzzy Fuzzy Test ${i} + +Fuzzy Multimessage Miss Test + : FOR ${i} IN @{RANDOM_MESSAGES} + \ Fuzzy Miss Test ${i} + +Fuzzy Multimessage Delete Test + : FOR ${i} IN @{MESSAGES} + \ Fuzzy Delete Test ${i} + +Fuzzy Multimessage Overwrite Test + : FOR ${i} IN @{MESSAGES} + \ Fuzzy Overwrite Test ${i} diff --git a/test/functional/cases/120_fuzzy/mumhash-keyed.robot b/test/functional/cases/120_fuzzy/mumhash-keyed.robot new file mode 100644 index 000000000..cfef38e57 --- /dev/null +++ b/test/functional/cases/120_fuzzy/mumhash-keyed.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Keyed Mumhash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test + +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/cases/120_fuzzy/mumhash.robot b/test/functional/cases/120_fuzzy/mumhash.robot new file mode 100644 index 000000000..6e2714b64 --- /dev/null +++ b/test/functional/cases/120_fuzzy/mumhash.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Plain Mumhash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test + +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/cases/120_fuzzy/plain.robot b/test/functional/cases/120_fuzzy/plain.robot deleted file mode 100644 index 2fc2fd5ef..000000000 --- a/test/functional/cases/120_fuzzy/plain.robot +++ /dev/null @@ -1,18 +0,0 @@ -*** Settings *** -Suite Setup Generic Setup -Suite Teardown Generic Teardown -Resource lib.robot - -*** Variables *** -${SETTINGS_FUZZY_WORKER} ${EMPTY} -${SETTINGS_FUZZY_CHECK} ${EMPTY} - -*** Test Cases *** -Fuzzy Add - Fuzzy Add Test - -Fuzzy Delete - Fuzzy Delete Test - -Fuzzy Overwrite - Fuzzy Overwrite Test diff --git a/test/functional/cases/120_fuzzy/siphash-keyed.robot b/test/functional/cases/120_fuzzy/siphash-keyed.robot new file mode 100644 index 000000000..6ed321ca9 --- /dev/null +++ b/test/functional/cases/120_fuzzy/siphash-keyed.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Keyed Siphash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test + +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/cases/120_fuzzy/siphash.robot b/test/functional/cases/120_fuzzy/siphash.robot new file mode 100644 index 000000000..5ede462de --- /dev/null +++ b/test/functional/cases/120_fuzzy/siphash.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Plain Siphash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test + +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/cases/120_fuzzy/xxhash-keyed.robot b/test/functional/cases/120_fuzzy/xxhash-keyed.robot new file mode 100644 index 000000000..b9840dc0d --- /dev/null +++ b/test/functional/cases/120_fuzzy/xxhash-keyed.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Keyed Xxhash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test + +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/cases/120_fuzzy/xxhash.robot b/test/functional/cases/120_fuzzy/xxhash.robot new file mode 100644 index 000000000..4cb7fef75 --- /dev/null +++ b/test/functional/cases/120_fuzzy/xxhash.robot @@ -0,0 +1,14 @@ +*** Settings *** +Suite Setup Fuzzy Setup Plain Xxhash +Suite Teardown Generic Teardown +Resource lib.robot + +*** Test Cases *** +Fuzzy Add + Fuzzy Multimessage Add Test + +Fuzzy Fuzzy + Fuzzy Multimessage Fuzzy Test + +Fuzzy Miss + Fuzzy Multimessage Miss Test diff --git a/test/functional/configs/fuzzy.conf b/test/functional/configs/fuzzy.conf index e8d5db628..fcd744f92 100644 --- a/test/functional/configs/fuzzy.conf +++ b/test/functional/configs/fuzzy.conf @@ -50,6 +50,7 @@ fuzzy_check { min_bytes = 100; rule { + algorithm = "${ALGORITHM}"; servers = "${LOCAL_ADDR}:${PORT_FUZZY}"; symbol = "R_TEST_FUZZY"; max_score = 10.0; diff --git a/test/functional/lib/rspamd.py b/test/functional/lib/rspamd.py index 4da8712a3..d45faa794 100644 --- a/test/functional/lib/rspamd.py +++ b/test/functional/lib/rspamd.py @@ -72,6 +72,11 @@ def HTTP(method, host, port, path, data=None, headers={}): def make_temporary_directory(): return tempfile.mkdtemp() +def path_splitter(path): + dirname = os.path.dirname(path) + basename = os.path.basename(path) + return [dirname, basename] + def read_log_from_position(filename, offset): offset = long(offset) f = open(filename, 'rb') diff --git a/test/functional/lib/rspamd.robot b/test/functional/lib/rspamd.robot index 146eabd71..7ec79ce6e 100644 --- a/test/functional/lib/rspamd.robot +++ b/test/functional/lib/rspamd.robot @@ -127,4 +127,4 @@ Sync Fuzzy Storage ${result} = Run Process ${RSPAMADM} control -s ${TMPDIR}/rspamd.sock fuzzy_sync Log ${result.stdout} Follow Rspamd Log - Sleep 1s Try give fuzzy storage time to sync + Sleep 0.005s Try give fuzzy storage time to sync diff --git a/test/functional/messages/spam_message.eml.fuzzy1 b/test/functional/messages/spam_message.eml.fuzzy1 new file mode 100644 index 000000000..e4b2a0853 --- /dev/null +++ b/test/functional/messages/spam_message.eml.fuzzy1 @@ -0,0 +1,44 @@ +Reply-To: +From: "Bob" +Subject: Hello +Date: Sun, 14 Aug 2016 13:35:02 +0200 +MIME-Version: 1.0 +Content-Type: text/plain +Content-Transfer-Encoding: 7bit + +Dear Western Union Beneficiary, + + +UNITED NATIONS PAYMENT NOTIFICATION. + +We wish to inform you the United Nations (UN) authorized us to remit to you a total amount of $920,000.00, (Nine Hundred and Twenty Thousand United States Dollars). + +Your Cash prize was paid out to us by the United Nations, and they have successfully succeeded in depositing your whole funds with us here at Western Union London United Kingdom. + +They have now ordered us to take full responsibility in the transfer process of your funds and thus commence the immediate remittance of your funds to you. + +Be duly informed that because of our Western Union transfer policy, your funds will be paid to you via our Western Union Daily Transfer limit of $4,600.00 USD. This means that you will receive a amount of $4,600.00 USD daily, and this amount Can be collected from any of our numerous Western Union outlets in your current location. + +To begin the claim process of your daily payment as stated above, kindly furnish us with the following; + +Full Name: +Address: +Phone Number: + +Upon receipt of the above mentioned details, your first transaction will be activated and we shall then Proceed to provide you with the Money Transfer Control Number (MTCN) for the First installment and we will continue to email you others after 12 hours of Receiving each payment. + +For information on your payment status; + +Contact Person +Henry Adams +Email: h_ada15@aol.co.uk + +OR call our 24 hours Helpline @ +440000000000, for any inquiries on the above message. + + +Yours truly, + +Diana Mckay +For: Western Union London, United Kingdom. +WESTERN UNION... Over 380,000 Outlets Worldwide + -- 2.39.5