Fix coverage (#2603)

* Add utility to prepare C coverage for upload to coveralls.io

It turned out that it is more easy to write own script then debug
and fix coveralls utility (https://github.com/eddyxu/cpp-coveralls).

gcov-coveralls.py can be used as a replacement for coveralls.

* Save coverage data from .gcda files only once

Coverage data in .gcda files is merged after each binary invocation,
so we can run all test and then gather coverage data.

If we dump them two times execution counts will be more then they be.

* Switch from coveralls (cpp-coveralls) to own script

Problem with coveralls was, that coverage for source files outside build
directory was not added to the report.

* Add tool to dump info from json for coveralls.io

* Add debug

* Fix: don't die if there is no service_job_id in json

* Debug

* Fix dump_coveralls.py

* Rename to gcov_coveralls.py (s/-/_/)

For most files in this repo '_' is used as separator.

* Don't add source code to coveralls JSON

According to https://docs.coveralls.io/api-introduction
Coverals don't need source code, only MD5 digest to tracks changes.

Anyway source code is already added by luacov-coveralls and source_digest
is added by cpp-coveralls and gcov_coveralls.py
Both options seems to work for now.

* Provide path to source directory to merge_coveralls.py

merge_coveralls.py has code to filter files and remove prefixes.
When --root points to source directory merge_coveralls.py can strip
prefix from absolute path in JSONs generated by luacov-coveralls.

* Style

Don't add parameters with default values.

* Make --output optional

It useful mainly for debugging. We can send report without saving it.

* Log CI_COMMIT_AUTHOR env var

It is not clear from drone.io source how CI_COMMIT_AUTHOR variable is
set. Log it to see what it means.

* Move merge_coveralls.py to test/tools

This script is used not only for funcional test coverage, but for
rspamd-test coverage too.

* Remove debug

* Style

Use more compact formatting.

* Write comment about parallel tests running [SKIP CI]

Document why running tests in parallel may be bad idea (but still do so).

* Fix typo [SKIP CI]

1 files changed, 19 insertions, 8 deletions

diff --git a/.drone.yml b/.drone.yml
index 9c1b5d09a..24d4d8ec8 100644
--- a/.drone.yml
+++ b/.drone.yml
@@ -12,6 +12,8 @@ pipeline:
       - install -d -o nobody -g nogroup /rspamd/build /rspamd/install
       # lua-torch CMakeLists writes to src dir
       - chown nobody $CI_WORKSPACE/contrib/lua-torch/nn
+      # for debug
+      - echo $CI_COMMIT_AUTHOR
 
   build:
     # https://github.com/rspamd/rspamd-build-docker/blob/master/ubuntu-build/Dockerfile
@@ -44,6 +46,16 @@ pipeline:
       # checks are configured in .tidyallrc at the top of rspamd repo
       - tidyall --all --root-dir $CI_WORKSPACE --check-only --no-cache --data-dir /tmp/tidyall
 
+  # We run rspamd-test (unit test) and functional test (runned by robot) in
+  # parallel to save time. To avoid conflict in saving lua coverage we run them
+  # from different directories. For C code coverage counters is saved to .gcda
+  # files and binary contain absolute path to them, so rspamd-test and
+  # processes started by functional test are writing to the same files. On
+  # process exit new coverage data merged with existing content of .gcda file.
+  # Race is possible if rspamd-test and some rspamd process in functional test
+  # will try to write .gcda file simultaneous.  But it is very unlikely and
+  # performance is more important then correct coverage data.
+
   rspamd-test:
     # https://github.com/rspamd/rspamd-build-docker/blob/master/ubuntu-test/Dockerfile
     image: rspamd/ci-ubuntu-test
@@ -68,8 +80,6 @@ pipeline:
       # luacov-coveralls reads luacov.stats.out written by rspamd-test using luacov module
       # and writes json report for coveralls.io service
       - luacov-coveralls -o /rspamd/build/unit_test_lua.json --dryrun
-      - cd /rspamd/build
-      - coveralls --dump coverage.rspamd-test.dump
       - exit $EXIT_CODE
 
   functional:
@@ -82,18 +92,19 @@ pipeline:
       # some rspamd processes during this test work as root and some as nobody
       # use umask to create world-writable files so nobody can write to *.gcda files created by root
       - umask 0000
-      - set +e
-      - RSPAMD_INSTALLROOT=/rspamd/install robot --xunit xunit.xml --exclude isbroken $CI_WORKSPACE/test/functional/cases; EXIT_CODE=$?
-      - set -e
-      - coveralls --dump coverage.functional.dump
-      - exit $EXIT_CODE
+      - RSPAMD_INSTALLROOT=/rspamd/install robot --xunit xunit.xml --exclude isbroken $CI_WORKSPACE/test/functional/cases
 
   send-coverage:
     image: rspamd/ci-ubuntu-test
     secrets: [ coveralls_repo_token ]
     commands:
       - cd /rspamd/build
-      - $CI_WORKSPACE/test/functional/util/merge_coveralls.py --input coverage.functional.dump coverage.rspamd-test.dump unit_test_lua.json lua_coverage_report.json --output out.josn --token=$COVERALLS_REPO_TOKEN
+      # extract coverage data for C code from .gcda files and save it in a format suitable for coveralls.io
+      - $CI_WORKSPACE/test/tools/gcov_coveralls.py --exclude test --prefix /rspamd/build --prefix $CI_WORKSPACE --out coverage.c.json
+      # * merge coverage for C and Lua code
+      # * remove prefixes from absolute paths (in luacov-coveralls files), filter test, contrib, e. t.c
+      # * upload report to coveralls.io
+      - $CI_WORKSPACE/test/tools/merge_coveralls.py --root $CI_WORKSPACE --input coverage.c.json unit_test_lua.json lua_coverage_report.json --token=$COVERALLS_REPO_TOKEN
     when:
       branch: master
       # don't send coverage report for pull request