ctest writes a test whose executable is missing, for example because it failed to link, as status="notrun" with <skipped message="Unable to find executable">, the same shape as a test that skipped itself. sanitizer_report counted both as skipped, which hid a test that did not build. A skipped element now means a skip only when its message is a ctest skip status (SKIP_RETURN_CODE=<n> or SKIP_REGULAR_EXPRESSION_MATCHED, as tools/check_skips.py reads it) or when the status is not a failing one; any other not-run test counts as failed. On a local ASan and UBSan run of main this moves atrac9_gradient, whose executable does not link, from the 721 skipped tests to the failed ones.
Check that the SUMMARY line wins over the ERROR line, that LSan and TSan lines and a UBSan runtime error with its source location are found, that output without a sanitizer line gives nothing, that long lines are shortened, and that the failed test table gains the report column only when a failure has a report.
Read the system-out and system-err of each testcase and keep the first sanitizer line: the SUMMARY line of ASan, LSan or TSan, else their ERROR or WARNING line, else a UBSan runtime error line with its source location. When any unexpected failure has one, the failed test table gains a First sanitizer report column, shortened to 200 characters and escaped for Markdown.
Check that a listed test that passed appears in the passed table with its reason, that a listed test missing from the run appears in the not-run table, that a skipped listed test appears in neither, and that a run whose known failures all still fail has neither table.
List the known failures of the selected sanitizers whose test passed, so a fixed test can leave the list, and those that are not in the run at all, such as a renamed test. Neither affects the exit code.
Check that known failures apply only to the sanitizers they name and that blank lines are skipped, that a listed failure moves to the Known failure table with its reason escaped for Markdown and stops failing the run while an unlisted one still fails it, that --known without --sanitizer is a usage error, and that malformed lines are rejected with exit code 2 and their line number.
sanitizer_report.py --known reads a text file with one known failure per line: the test name, the comma-separated sanitizers it fails under and the reason. With --sanitizer naming the sanitizers of the build, a failure of a listed test moves to a Known failure table with its reason and no longer fails the run. Blank lines are ignored; a line without a reason, an unknown sanitizer and a duplicate test and sanitizer pair are errors that name the line.
Check how sanitizer_report classifies passed, failed, skipped, not run and disabled tests, the counts and sorted table of the summary, the exit code for runs with and without failures, the exit code 2 for unreadable or malformed reports, and that --summary appends to an existing file.
Add tools/sanitizer_report.py, which reads the JUnit file that ctest --output-junit writes, counts passed, skipped and failed tests, lists the failed ones in a Markdown summary and exits with 1 when any test failed. A test that ctest did not run counts as failed, and a disabled test as skipped.
check_stub runs check_conventions.py in a temporary repository with a
10 s limit. The slowest tests of this file take 8 to 11 s each on a
Windows 10 machine here, and in a full ctest run the call exceeded the
limit (subprocess.TimeoutExpired after 10 seconds), so
conventions_function_bodies_shard_5 failed although it passes on its
own. The limit is now 120 s, the same as the ctest timeout of the shard.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
MarcZX10