A run of adjacent heap mirrors is collected as one range. Once a refused
heap keeps its mirror after its range has left the write watch, such a run
can no longer be collected whole, and each of its mirrors is collected on
its own: the unwatched one is refreshed in full and its watched neighbour
by the blocks the CPU wrote.
The test registers two adjacent heaps, takes a block of the first off the
watch and writes to both, and checks that the next build keeps both
mirrors and serves both writes, with the address-space cache and without
it. Refreshing the whole run as unwatched instead leaves the neighbour's
mirror without its write. The heaps share one allocation and the mock
device imports nothing, so the test runs on every platform.
The sparse heap mirror unwatch tests were registered only outside Windows.
Their Windows path takes the heap from the guest arena and commits its
first half, as the sparse heap mirror tests already do there, and the heap
is the only range they register, so no neighbouring range can join its
import run. GuestMemory::Unwatch excludes the block from the arena's watch,
which the heap mirror unwatch tests already exercise on Windows. Register
both on every platform, with the libc directory on PATH as the heap mirror
unwatch tests have it.
ShaderRegistry.cpp prepares the registered shader types 0, 1, 2, 4, 6 and 7, skips 5, and logs and leaves every other type unprepared at registration; Hello Neighbor 2, Ad Infinitum and Ghostrunner register type 8.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A heap mirror whose range had left the write watch was dropped and its
range planned again. When the host import budget refused the range, that
plan was a per-build copy, and past APS5_ADDRESS_COPY_MAX_MIB (64 MiB) the
copy stopped the process ("an address-based build copies ... past ..."),
where refreshing the mirror in full would have carried on.
Plan such a range again only when the import path takes it now. A cached
address space whose heap or sparse mirror has left the watch is rebuilt
only if importing the mirror's range succeeds, and the rebuild then finds
that import; otherwise the space is kept and the mirror refreshed in full,
every block counted as written. A build that plans its ranges tries the
import first, so acquireMirror keeps the existing mirror of a range that
reaches it, and a sparse mirror whose pieces are unwatched is refreshed in
full instead of planned again.
SetAddressCopyLimit sets the copy cap in-process, as SetHostImportBudget
sets the budget. The heap and sparse unwatch tests set it below the heap
the old plan would copy: the refused heap's mirror is kept and refreshed in
full (its unwatched block and a watched block the CPU wrote both reach it),
and once the budget holds the heap the next build imports it and releases
the mirror.
Trails in the Sky 2nd Chapter creates a custom rack with module id 0x20, which Custom.cpp rejects; the module and its structure are unknown (#4246).
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Broken Sword - Shadow of the Templars: Reforged stops at AJM decoding codec 23, which is not one of the codec ids known in Ajm.cpp (MP3 0, ATRAC9 1, M4AAC 2, Opus 24).
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A build checks that its heap mirrors' ranges are write-watched before it
reuses them and refreshes them later in the same build. A map-time import
on another thread can take a range off the watch in between, and
refreshHeapMirrors and refreshHeapRuns then threw "a heap mirror's range is
no longer write-watched".
When a run of heap mirrors cannot be collected whole, collect each mirror
on its own. A mirror whose range is no longer watched is refreshed in full,
every block counted as written, and counted as an unwatched heap; the run's
other mirrors keep their per-block refresh.
The sample order planning test builds pipelines from graphics pipeline
libraries on a mock device whose object count, handle counter and
recorded sample masks are plain globals. With link-time optimization on,
the Context default, each linked pipeline also queues an optimized link
on the pipeline library worker thread. That link calls the mock's
vkCreateGraphicsPipelines, and its vkDestroyPipeline when the pipeline
was released first, while the test thread builds the next pipeline and
destroys both. An increment or decrement of the object count can be
lost, so the leak check fails now and then: on Windows, and on Linux in
8 of 20000 runs with eight runs at a time on four cores.
No object is left alive: ClearPipelineLibraries waits for the worker
before the check. Waiting with WaitForOptimizedPipelines after each
pipeline does not close the overlap either, as the pipeline's
constructor still destroys the shader modules a cached library part did
not keep after it queues the link.
The test now builds its library pipelines without the optimized link, as
on a device without link-time optimization, so every mock call stays on
the test thread.
The check that statvfs leaves no descriptor open listed /proc/self/fd,
which macOS does not have. macOS lists the process's descriptors in
/dev/fd; Linux keeps /proc/self/fd.
statvfs opened the path with O_PATH, which macOS does not have, so
libSceLibcInternal did not build there. No open mode stands in for it:
on macOS O_RDONLY and O_EVTONLY fail with EACCES on a file or directory
without read permission, and O_SEARCH on such a directory, where
FreeBSD's statvfs only needs search permission on the path.
On macOS statvfs now asks the host statvfs by path, which needs exactly
that, fills the kernel statfs the way _fstatfs fills it from the host
fstatvfs, and converts it with the code fstatvfs uses. A path that must
name a directory and does not fails with ENOTDIR, as the O_DIRECTORY open
does elsewhere. Other hosts are unchanged.
fstatvfs kept the guest struct statvfs and the sfs2svfs conversion from
the kernel's statfs to itself. Both move to include/Statvfs.hpp so that
statvfs can convert a volume the same way without opening the path.
Nothing changes for fstatvfs.
The fchdir entry still said FreeBSD's search-permission check was not
modelled, although it has run on Linux since 0e4d94ce3 and now runs on
every host but Windows.
The case that removes search permission from a directory and expects
fchdir to fail with EACCES, leave the working directory unchanged and
succeed again once search is allowed ran only on Linux. It now runs on
every host but Windows, with the POSIX headers it needs.
fchdir checked search permission on the directory with faccessat only on
Linux, so on macOS a guest could change into a directory it may not
search, where FreeBSD fails with EACCES and keeps the working directory.
faccessat with AT_EACCESS is POSIX and behaves the same on macOS, so the
check now runs on every host but Windows.
unlinkat's rmdir and unlink paths call ::rmdir and ::unlink outside the
Windows branch, but <unistd.h> and the headers beside it were included
only under __linux__, so libkernel_sys did not build on macOS. They are
now included on every host that is not Windows, the same split the code
that uses them already makes.
The shader disk cache's writer, started with the cache at the first
shader lookup, was detached and never stopped. It stayed parked in a
timed wait on its queue until the process exit killed it, and entries
still queued at exit were never written. A process that exited right
after its first shaders could kill the writer inside its first wait,
which takes winpthreads' global condition-variable lock, as with the
driver's own threads.
The writer is now joinable. The driver's shutdown stops it after the
other driver threads: the writer writes the entries still queued and
leaves, and the shutdown joins it. A store after that is written on the
caller, and a cache first used after the shutdown starts no writer.
ShaderDiskCache::Shutdown joins the names the cache key check has
reviewed, since it changes nothing a recompile produces.
The committer of pipelined draws, started at the first pipelined
submission, was never stopped. The driver's shutdown joined the queue
workers and left the committer parked on its queue, and a commit the
last submission left queued, such as its deferred completion, could
still run after the shutdown had reset the device. A process that
exited right after its first pipelined submissions could kill the
committer inside its first wait on the queue, which takes winpthreads'
global condition-variable lock, as with the preparation workers.
The driver's shutdown now stops the committer after joining the queue
workers and before taking the GPU lock. The committer runs the commits
still queued and leaves, and the shutdown joins it. A commit enqueued
after that runs on the caller.
ARCHITECTURE.md and the debt entry describe how imports under 1 MiB are
settled without waiting and return to the write watch, which the debt
entry listed as not implemented.
Since #6060 a Windows host import of private write-watched memory stays
in the write watch only after its settle, a 4-byte GPU read that the
importing thread waits for, and only imports of 1 MiB or more are
settled: SuperTuxKart imports a 64 KiB buffer around every submission,
and the wait cost it about 19% of its frame rate. A smaller import is
compared instead, and its range leaves the write watch for good.
Resident Evil Village maps its GPU heap in 64 KiB pieces, so every
texture over that heap is compared with guest memory.
An import under 1 MiB is still compared when it is made, but its read is
now queued. The next import lookup that holds the device lock records
the reads queued since into one batch on the settle queue and submits
it, so the thread that maps and imports neither submits nor waits.
Once the batch's fence has signalled, a later lookup returns the
imports' blocks to the write watch, except where storage results are
pending over them, which waits for a later lookup. An import retired
while its read is in flight waits for that batch first.
A settled import whose range had already left the write watch, such as
a whole range imported over a piece that was compared, returns to it
the same way.
A sparse heap mirror, made for a heap with uncommitted pages, is
refreshed from the write watch through its committed pieces. The check
that a reused heap mirror's range is still watched covered the heap
mirrors acquireMirror reuses and the cached space's heap runs, not
sparse ones: once a committed piece left the watch, refreshSparseMirror
threw "a sparse heap mirror's range is no longer write-watched" from
acquireMirror's reuse and from a cache hit's sparse refresh.
refreshSparseMirror now returns false when a piece is no longer
watched. Both callers already take false as not reusable: acquireMirror
makes the mirror again, finds the range unwatched, drops the stale
mirror and leaves the range to a copy, and a cache hit gives up the
cached space.
agc_sparse_heap_mirror_unwatch mirrors a 1 MiB reservation with 256 KiB
committed, takes one committed block off the write watch and stores new
bytes there from the CPU; the next build must not reuse the mirror and
must serve the new bytes. agc_sparse_heap_mirror_unwatch_no_space_cache
runs the same without the address-space cache. Without the fix both
throw. They are registered on Linux; the Windows path uses the guest
arena as the existing sparse heap test does and has not run on Windows.
A sleep of 1 ms or less returns once the AGC driver sees a GPU batch
complete (a thread waits on the recorder's timeline semaphore), runs a
batch's completion actions, or VideoOut completes a flip. A thread
returns early at most 8 times in a run of back-to-back sleeps; the run
restarts when the thread submits AGC work or did not sleep for the
requested time. APS5_NO_SLEEP_WAKE=1 keeps every sleep in full.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The Linux output passed a PT_TLS whose initialization image is outside every PT_LOAD, whose file size exceeds its memory size, whose alignment is not a power of two, or that follows another PT_TLS, straight to the dynamic loader, which reads the image from p_vaddr when it sets up TLS. The Windows and macOS writers already refuse these layouts. Reject them in ElfReader so every target, and guest modules, see the same rule. An empty PT_TLS placeholder is still accepted.
GuestMemory::Rewatch takes ranges that left the write watch back into
it. Only whole tracker blocks whose pages can be collected return:
their write-watch marks are collected and dropped, and every block is
stamped as a CPU store with a new generation, so a snapshot or a proof
made before the block left the watch reads as changed, and the blocks'
untracked hints stop applying. It does nothing outside Windows.
Driver::stop reset the device and then stopped the worker. No job could
outlive the device, because ClearPipelineLibraries waits for the
device's jobs to finish, but that was only visible inside the wait.
The worker now stops first: a running job finishes, queued jobs are
dropped, and no job runs once the device's teardown starts.
agc_driver_draw_cache_reuse checks that the driver's own shutdown
leaves no library worker running.
sceAgcLinkShaders resolves the rect-list pairing of a pipeline when the
title creates it, and three cases ended the title there although the
rect-list draw that uses the pipeline builds its pair itself:
- a pipeline without a pixel shader reached ResolveGraphicsAbi with a
null pixel, which required both stages, although the draw pairs the
front stage with the null pixel program;
- a stage with no artifact yet, such as a pixel shader that exports
stencil on a device without VK_EXT_shader_stencil_export, failed the
missing stage artifacts check;
- a pair that BuildRectListShaders refuses, such as a pixel input with
passthrough interpolation, threw from the builder.
The ABI now skips the first two, and logs the third once as a deferral
of the pixel shader without recording the pair, so the pair's draw
builds it again and fails with the builder's reason. When a draw builds
the rectangles of its front stage, only the pair of its own fragment can
fail it; the pair of another fragment that has no artifact or that the
builder refuses is left to that fragment's draw.
A new mode resolves a front stage with a null pixel, with a pixel before
it has an artifact, with a pixel that never gets one and with the first
pixel's passthrough variant, and expects one deferral with the
builder's reason. It then draws the front stage as a rect list with the
null pixel program, which succeeds although the other two pairs cannot
be built, and with the passthrough pixel, which fails with the
builder's reason.
PrepareGraphicsStages decodes each front-end stage's vertex inputs from
its AGC header before it prepares the stage, outside the catch that
defers a stage the recompiler cannot prepare. A header the decode
refuses, such as a semantic with a statically bound vertex buffer,
threw out of sceAgcDriverResolveGraphicsStagesAbi and ended a title that
creates its pipelines ahead of drawing. Registration leaves such a stage
alone when it is an NGG vertex shader that uses its workgroup, or a
mesh, hull or domain stage, so the pipeline ABI is the first to decode
it.
A decode failure now defers the stage the same way a preparation
failure does. The draw that uses it decodes the same header with the
draw's own user data and fails there with the same reason.
A new mode registers an NGG vertex shader with vertex buffer and
attribute tables and one statically bound semantic, resolves its
pipeline ABI twice inside a scope, and expects one deferral with the
decode's reason and no front-end build, then the same reason from its
first draw.
A failure stored by a shader's background registration preparation was
rethrown by every pipeline ABI call that touched the stage. The
ShaderPreparationScope that sceAgcLinkShaders, sceAgcCreatePrimState and
sceAgcCreateInterpolantMapping open settled the stages and threw first,
and ResolveShaderAbi, ResolveGraphicsStagesAbi and ResolveGraphicsAbi
threw it again from their reads of the stage. A title that creates its
pipelines ahead of drawing ended there, although the draw or dispatch
that uses the stage rethrows the same stored failure.
Settling a scope no longer rethrows. Each ABI call logs such a stage
once with its stored reason, marks it deferred and prepares the other
stages of the call as before. ResolveGraphicsStagesAbi keeps its ABI
record on the first stage that did not fail, PrepareGraphicsStages skips
a failed program, and ResolvePreparedGraphics records no rectangle for
a failed stage and never adds a failed fragment to a front stage, where
the rectangle draw of another fragment would meet it.
The compute mode no longer expects the stored failure from the ABI call.
A new mode registers a compute, a pixel and a vertex shader whose
registration preparation fails, resolves them through every ABI entry
point inside a scope twice, and expects three deferrals with the stored
reason the first time and none the second, then the stored reason from
the compute shader's dispatch.
agc_heap_mirror_unwatch registers a 2 MiB heap, a 1 MiB idle range and a
64 KiB busy range, imports the idle and busy ranges and sets the import
budget so the heap's import is refused: the first address-based build
mirrors the heap. One 64 KiB block of the heap then leaves the write
watch and the CPU stores new bytes there, so the second build must drop
the mirror and serve the new bytes, here from a copy.
agc_heap_mirror_unwatch_no_space_cache runs the same without the
address-space cache.
agc_heap_mirror_unwatched_import takes the heap off the watch through a
host import instead: once the budget holds the heap, an import that does
not keep the watch (ImportWatch::Unwatch) covers it, and the next build
must take that import instead of the cached space's mirror. It is not
registered on Windows, where an import of private memory may keep the
watch through a shared alias or a settle, which the mock does not model.
Without the fix each one throws "a heap mirror's range is no longer
write-watched": the first and third from the cached space's heap runs,
the second from the build's own heap refresh.
load_summary returned None for a missing file whether the path was the
default aps5-summary.jsonl next to the trace or one named with
--summary. A mistyped --summary path was therefore silently ignored:
the digest printed "no summary line found" and, without the window end,
dropped the last frame of the window. Report an explicitly named file
that does not exist as a usage error; the default stays optional.
The ci-gate check read a job's needs and if only two spaces deeper
than the job name. In a workflow indented by four spaces those keys
were never found, so a job gated with needs: eligibility and an if on
its outputs was reported as ungated. Read the keys at the indentation
of the job's first child instead.
The comment check looked at each added .py, .cmake and CMakeLists.txt
line on its own and treated any # outside a string closed on that line
as a comment. A # inside a Python triple-quoted string or a CMake
quoted or bracket argument that spans lines was reported, so adding an
#include to the C++ held in PatchSpirvTools.cmake's [=[ ]=] arguments
failed the check. An escaped \# in CMake was reported too, while the
inner lines of a multi-line #[[ ]] bracket comment were not.
Find comments over the whole file instead: Python COMMENT tokens from
tokenize, and a CMake scan that skips quoted and bracket arguments and
escapes and covers every line of a bracket comment. A Python file that
does not tokenize keeps the previous per-line check.
inflate returns Z_BUF_ERROR both when the output is full and when the
input ends early. sceZlibInflate treated every Z_BUF_ERROR with no
output space left as insufficient output, so a stream cut off after
its last data byte, such as one missing its Adler-32 trailer, reported
0x8112001C when the output capacity equalled its decoded length, and
0x811200FF with any larger buffer. Report insufficient output only
while input remains, as zlib's uncompress2 and miniz's mz_uncompress2
do and as the technical debt entry describes.
Yunus Shaikh