Speedup lua coverage collecting for functional test
luacov-coveralls merge mode (-j flag) was created to join reports
containing coverage for different source files (e.g. C and Lua code).
Coverage for the same file in two report is not merged, instead one
source file is added several times to source_files array in JSON. As
a result if we use luacov-coveralls -j on report for same source files
it ends up spending a lot of time on parsing and dumping big JSON files.
This change reduces functional test time from 7+ minutes to 4+ minutes.
Clickhouse test sometimes fails with an error:
ConnectionError: HTTPConnectionPool(host='localhost', port=18123): Max retries exceeded with url: /?default_format=JSONEachRow (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f40ad96fdd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Fix this by waiting for TCP port instead pid file.
Share robot log and report on https://ci.rspamd.com/testlogs/
Sending functional test log (about 5Mb) in a email attachmend was temporary
solution.
Upload them to https://ci.rspamd.com/testlogs/ instead.
Link to log/report will be printed in build log.
* Add utility to prepare C coverage for upload to coveralls.io
It turned out that it is more easy to write own script then debug
and fix coveralls utility (https://github.com/eddyxu/cpp-coveralls).
gcov-coveralls.py can be used as a replacement for coveralls.
* Save coverage data from .gcda files only once
Coverage data in .gcda files is merged after each binary invocation,
so we can run all test and then gather coverage data.
If we dump them two times execution counts will be more then they be.
* Switch from coveralls (cpp-coveralls) to own script
Problem with coveralls was, that coverage for source files outside build
directory was not added to the report.
* Add tool to dump info from json for coveralls.io
* Add debug
* Fix: don't die if there is no service_job_id in json
* Debug
* Fix dump_coveralls.py
* Rename to gcov_coveralls.py (s/-/_/)
For most files in this repo '_' is used as separator.
* Don't add source code to coveralls JSON
According to https://docs.coveralls.io/api-introduction
Coverals don't need source code, only MD5 digest to tracks changes.
Anyway source code is already added by luacov-coveralls and source_digest
is added by cpp-coveralls and gcov_coveralls.py
Both options seems to work for now.
* Provide path to source directory to merge_coveralls.py
merge_coveralls.py has code to filter files and remove prefixes.
When --root points to source directory merge_coveralls.py can strip
prefix from absolute path in JSONs generated by luacov-coveralls.
* Style
Don't add parameters with default values.
* Make --output optional
It useful mainly for debugging. We can send report without saving it.
* Log CI_COMMIT_AUTHOR env var
It is not clear from drone.io source how CI_COMMIT_AUTHOR variable is
set. Log it to see what it means.
* Move merge_coveralls.py to test/tools
This script is used not only for funcional test coverage, but for
rspamd-test coverage too.
* Remove debug
* Style
Use more compact formatting.
* Write comment about parallel tests running [SKIP CI]
Document why running tests in parallel may be bad idea (but still do so).
* Fix typo [SKIP CI]
Currently functional test sometimes fails with and error:
Teardown failed: NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=2259)
Rewrite process termination - there is no need to send signals in a loop
many times.
* Build as nobody
This showld fix coverage files writing. At least some coverage files
(*.gcda, *.gcno) are written by rspamd running as nobody.
* Comment luacov-coveralls
* Run rspamd-test as nobody too
* Run coveralls from build root (/rspamd/build)
It need to find all `.gcda` files.
* Hack for coveralls was moved to Docker image
* Add test debug
* More debug
* Workaround to writing *.gcda from different users
*.gcda files written in parallel by root and by nobody
* Remove debug
* Use current build number instead of previous
* Remove unused variable
* Add more parameters
- If CI detected we don't need to check if service_job_id and other
fields in json already filled - this info anyway comes from environment
- coveralls.io API is not documented. Code from:
https://github.com/lemurheavy/coveralls-ruby/
and
https://github.com/coveralls-clients/coveralls-python/
was used as an example.
* In case of error there is no url in json respnse
Print message field anyway - it can contain useful error message.
* Add debug
* Simplify
* Fix: for pull request CI_BUILD_EVENT set to 'pull_request'
* Remove debug
* Start switchind test to custom Docker images
Using Docker images with pre-installed delendencies will speedup
testing.
* Debug build in drone.io
* Use variable instead hardcoded path to the source
* Debug variable substitution by drone
Also according to comments in code DRONE_WORKSPACE is deprecated and
CI_WORKSPACE is a new form.
* One more try to use variables.
It looks like drone uses same rules as docker-compose:
https://docs.docker.com/compose/compose-file/#variable-substitution
* Cleanup debug
* Less verbose make output
* Be consistent in variable escaping
It works without escaping only becuse ${VAR} currently expanded by
drone, but $VAR is not.
* Use alpine base image for nodejs
It is smaller.
* Return rspamd-test
* Revert variable escaping
Wihtout escaping it is more easy to copy-past command for testing.
Just don't use ${var} when var should be expanded by shell and not
drone.
* Add debug
* Fix yaml parsing error
* Fix exit code capture
It should be done on the same line, because drone insert 'echo + cmd'
between commands.
* Remove debug
* Readability
No functional changes.
* Return functional tests
* Be consistent
* Add git stub for coveralls
* Send coverage
* Fix yaml parsing error
* Add links to Dockerfiles
* Send test log in case of test failure
XXX if rspamd-test failed, but functional was successful robot log will be
attached (but not useful).
* Don't run rspamd-test and functional tests in parallel
It looks like there is a race condition in writing coverage files.
* Try to run tests in parallel
* Add debug
* Fix parallel test run
test/rspamd-test and functional test both use luacov.stats.out file
in current directory. Run ./rspamd-test from anther directory.
* Use drone condition for sending coveralls report
Also don't send report if empty tocken was provided.
* Add mininal python3 compatibility
Anton Yuzhaninov
Vsevolod Stakhov
Mikhail Galanin
Mikhail Galanin