rspamd/test/coverage.md

Coverage collection explained

Hi mate. In short, you don’t wanna know this. Believe me, you don’t. Please, close this file and forget about it.

Surely? You still here?

Please, stop it until it’s too late.

You were warned.

Preamble

RSPAMD is written mainly in two languages: C and Lua. Coverage for each of them is being collected using different tools and approaches and is sent into coveralls.io. Each approach is not quite compatible to other tools. This document describes how we crutch them to work together.

C coverage

In general, pretty boring. When you run cmake with “-DENABLE_COVERAGE=ON” flag, it adds “--coverage” flag to both CFLAGS and LDFLAGS. So that each run of generated binary will create *.gcda file containing coverage data.

However, there are some moment to highlight:

• RSPAMD is run under “nobody” user. Hence, directories and files should be writable for this user.
• To make it possible, we explicitly run umask 0000 in “build” and “functional” stages in .circleci/config.yml
• After run, we persist coverage data in “coverage.${CIRCLE_JOB}.dump” during this build flow, see capture_coverage_data, to use it on the final stage.
• we use cpp-coveralls because it is able to save data for coveralls without actually sending it. We send on our own along with Lua-coverage.

Lua coverage

Lua coverage is collected for unit-tests and functional test suite. First part contains nothing interesting, just see test/lua/tests.lua.

“Functional” part is completely unobvious.

1. Coverage collecting is initiated and dumped in test/functional/lua/test_coverage.lua (there are a lot of comments inside). This file should be included on the very early stage of test run. Usually it’s included via config.
2. Coverage is dumped into ${TMPDIR}/%{worker_name}.luacov.stats.out
3. All worker coverage reports are merged into lua_coverage_report.json (see collect_lua_coverage())
4. finally, lua_coverage_report.json is persisted in build flow (see functional stage)

Altogether

Finally, we get all the reports:

• coverage.functional.dump
• coverage.rspamd-test.dump
• lua_coverage_report.json
• unit_test_lua.json

and merge them and send the resulting report using test/functional/util/merge_coveralls.py. Also, this scripts maps installed paths into corresponding repository paths and removes unneeded files (i.e. test sources).