A failed inspect or conversion hid the progress panel that held the
conversion log, cleared the selected archives and showed a single line,
so the only evidence of the failure was gone and a multi-GB input had to
be picked again. Compose the message with the phase, the current input,
the target, the libraries tag and the last 150 log lines, and show it in
the error box under "Failure details" so it can be copied into a report.
The report is a pure function in failure-report.mjs and is covered by
node tests.
Check that an empty testsuite and an empty testsuites file both exit with 2, print nothing and leave the --summary file untouched. The test fails with the previous behavior, which reported 0 tests and exit code 0.
A JUnit file without testcases, as ctest writes when it finds no tests in a build directory, gave '0 tests' and exit code 0, so a run that built nothing passed. Treat it as an unreadable report, with exit code 2.
Parse .github/sanitizer-known-failures.txt for every sanitizer and check that its entries stay sorted by test name, so a malformed or misplaced entry fails the sanitizer_report test in the regular build of the pull request that adds it, instead of the next nightly run. Names that no longer match a test are reported by the nightly summary, since many test names are generated in CMake loops.
Use the shapes ctest writes: a not-run test with SKIP_RETURN_CODE=77 or SKIP_REGULAR_EXPRESSION_MATCHED is a skip, while a not-run test whose executable is missing ('Unable to find executable') or whose skipped element has no message is a failure that fails the run. The test fails with the previous classification.
ctest writes a test whose executable is missing, for example because it failed to link, as status="notrun" with <skipped message="Unable to find executable">, the same shape as a test that skipped itself. sanitizer_report counted both as skipped, which hid a test that did not build. A skipped element now means a skip only when its message is a ctest skip status (SKIP_RETURN_CODE=<n> or SKIP_REGULAR_EXPRESSION_MATCHED, as tools/check_skips.py reads it) or when the status is not a failing one; any other not-run test counts as failed. On a local ASan and UBSan run of main this moves atrac9_gradient, whose executable does not link, from the 721 skipped tests to the failed ones.
Check that the SUMMARY line wins over the ERROR line, that LSan and TSan lines and a UBSan runtime error with its source location are found, that output without a sanitizer line gives nothing, that long lines are shortened, and that the failed test table gains the report column only when a failure has a report.
Read the system-out and system-err of each testcase and keep the first sanitizer line: the SUMMARY line of ASan, LSan or TSan, else their ERROR or WARNING line, else a UBSan runtime error line with its source location. When any unexpected failure has one, the failed test table gains a First sanitizer report column, shortened to 200 characters and escaped for Markdown.
Check that a listed test that passed appears in the passed table with its reason, that a listed test missing from the run appears in the not-run table, that a skipped listed test appears in neither, and that a run whose known failures all still fail has neither table.
List the known failures of the selected sanitizers whose test passed, so a fixed test can leave the list, and those that are not in the run at all, such as a renamed test. Neither affects the exit code.
Check that known failures apply only to the sanitizers they name and that blank lines are skipped, that a listed failure moves to the Known failure table with its reason escaped for Markdown and stops failing the run while an unlisted one still fails it, that --known without --sanitizer is a usage error, and that malformed lines are rejected with exit code 2 and their line number.
sanitizer_report.py --known reads a text file with one known failure per line: the test name, the comma-separated sanitizers it fails under and the reason. With --sanitizer naming the sanitizers of the build, a failure of a listed test moves to a Known failure table with its reason and no longer fails the run. Blank lines are ignored; a line without a reason, an unknown sanitizer and a duplicate test and sanitizer pair are errors that name the line.
Check how sanitizer_report classifies passed, failed, skipped, not run and disabled tests, the counts and sorted table of the summary, the exit code for runs with and without failures, the exit code 2 for unreadable or malformed reports, and that --summary appends to an existing file.
Add tools/sanitizer_report.py, which reads the JUnit file that ctest --output-junit writes, counts passed, skipped and failed tests, lists the failed ones in a Markdown summary and exits with 1 when any test failed. A test that ctest did not run counts as failed, and a disabled test as skipped.
Adrià