A YAML comment line such as '# python3 tools/tests/test_x.py' counted as a run, because only a comment after a space was stripped. Skip lines whose first non-blank character is '#', also inside run blocks, and strip a trailing comment after any whitespace.
check_test_registration.py only audits built test executables, so a Python test that no add_test or workflow names is silently skipped. Add a --scripts mode that lists the tracked test_*.py scripts and fails for any not run by an add_test of a CMakeLists or a python step of a workflow, and register it as a ctest so the relinker builds run it.
Ninja Multi-Config writes one add_test per configuration and a final NOT_AVAILABLE one for the same name. The audit kept only the last, so every test executable was reported as never run.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
tools/check_test_registration.py invoked bare ninja from PATH, so the audit failed with FileNotFoundError when the configured Ninja lived outside PATH. Read CMAKE_MAKE_PROGRAM from the build directory CMakeCache.txt and fail honestly when it is missing. Adapt the mocked built tests to a configured build and add a regression that runs a real Ninja from a path with spaces outside PATH.
#1331 registered the agc_pipeline_cache ctest, which runs
agc_driver_recorder_tests, without removing that executable from
KNOWN_UNRUN, so check_test_registration fails on main:
'agc_driver_recorder_tests is run by ctest now; remove it from KNOWN_UNRUN'.
The list is empty now; keep it a set for the set operations below.
The mesh test was built only on request and never registered, so the
translation of NGG geometry programs to mesh shaders ran in no test
run. On an RTX 4060 it passes. Register it with the other AGC driver
tests.
Three cases would fail or mislead under ctest. Without a Vulkan device
the test failed, and the Windows job has none. A device without
VK_EXT_mesh_shader exited 0, so ctest reported a pass for a test that
drew nothing. On llvmpipe, with the Linux job's Mesa 25.2.8 and Ubuntu
24.04's LLVM 20.1.2, every draw that builds a mesh pipeline stops the
process with "LLVM ERROR: Cannot emit physreg copy instruction"; Mesa
26.2.4 with LLVM 23.1 segfaults in LLVM's code generation instead. The
SPIR-V of all 18 modules the test builds passes spirv-val. Mesa #14376
reports the same error for lavapipe mesh shaders that index shared
memory per invocation; that the cause is the same is not established.
The test now opens its device with OpenVulkanTestDevice and exits 77
without a device, without VK_EXT_mesh_shader and on llvmpipe, saying
why. ANYPS5_REQUIRE_VULKAN still turns a missing device into a
failure. The Linux job therefore skips the test; it runs on hosts with
a GPU.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
guest_formatting_tests runs only on Windows and windows_exception_tests only
exists there, so the stale-entry check failed the Windows job.
Co-Authored-By: Claude <[email protected]>
The Windows job failed the check with two names that are not test
executables. `libhost_thread_local_test_library` is a support library
whose RUNTIME_OUTPUT_DIRECTORY puts it in the test directory; on Linux
its output is a library artifact and never lands there, which is why the
Linux job stayed green. `windows_exception_tests` is a MINGW-only test
executable that no `add_test` registers.
Ninja prints the rule beside each target, and CMake links an executable
with `CXX_EXECUTABLE_LINKER__<target>_<config>` and a shared library with
`CXX_SHARED_LIBRARY_LINKER__...`, on every platform. Requiring EXECUTABLE
in the rule leaves the library out and keeps the Linux count at 362.
`windows_exception_tests` is a real instance of the class this check
looks for, so record it beside the other five rather than fail the job
that found it.
The Windows job failed on the new step with
"no test executables found under build". Ninja prints the target names
with the platform separator, so the Windows build graph lists
tests\name_tests.exe where Linux lists tests/name_tests, and the reader
matched neither the backslash nor the .exe suffix.
Accept both separators and a .exe suffix. Verified by driving the
function with recorded Ninja output for both platforms, which now
return the same names, and by re-running the check against the Linux
tree, which still reports 362 test executables and the same five.
A test executable can be declared with add_test_executable and never
registered with add_test. It is then built and never run, and ctest
reports a full pass over what it knows, so nothing in CI says so.
Measured on main, five test executables are in this state: the driver's
mesh and recorder tests, the guest formatting tests, the video out flip
tests and the BDA device tests.
tools/check_test_registration.py compares the test executables CMake
builds, read from the build graph, against the add_test calls in the
generated CTestTestfile.cmake files, which are written at configure time
and so do not depend on the build having finished. It fails on a test
executable that nothing runs, and records those five by name so the job
stays green until they are dealt with.
Both jobs run it right after configure, before anything is built.
Dean