SPIRV-Cross emits the lock's acquire compare-exchange and release
exchange as relaxed Metal atomics, so the plain LDS loads and stores of
a locked 64-bit update were not ordered against the lock and contended
ds_add_u64 lost updates on an Apple M5 Pro. On MoltenVK the driver now sets
SpirvTarget::ldsLockFences, and the lock brackets its critical section
with workgroup-scope acquire and release OpMemoryBarriers, which become
threadgroup atomic_thread_fence with no execution barrier.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Order memory deltas and video output registration before the next
recorded call, and serialize the deadlock breaker with delta application
- Capture the blocks holding shader code with each shader delta
- Reserve explicit and automatic EOP flip tokens under one lock
- Record video output unregistration so replay can close and reopen a handle
- Clear already mapped ranges before restoring the snapshot so elided zero
blocks replay as zero
Co-Authored-By: Claude Opus 5.5 <[email protected]>
APS5_FRAME_CAPTURE=<dir> makes the AGC driver record a few frames of a
title: the shader registrations and ABI resolutions, video output and
EOP flip reservations it made since it started, a snapshot of every
readable guest allocation (the title image included) taken at a frame
boundary once the driver is idle, and for the next frames each
submission, flip and suspend point with the thread that made it, its
serial progress and the guest pages the write tracker saw the CPU write
before it. At the end it stores the presented display buffers as
reference images. APS5_FRAME_CAPTURE_AT_FLIP, _TRIGGER and _FRAMES pick
the frames.
agc_frame_replay maps the captured memory at its addresses and replays
the calls of each captured thread on a thread of its own, in captured
order, through the current build of the driver with no title running.
It paces GPU work to the in-flight depth the title had, releases GPU
waits that the title's CPU threads released without calling the driver
(AgcDriverAwaitedWaits), and reports frame times, released waits and
how closely the replayed display buffers match the references.
Tests: agc_frame_capture (file format) and agc_driver_frame_replay
(two captured frames of CPU writes and DMA copies replayed and
compared). Docs: BUILD.md, ENVIRONMENT.md and the libSceAgcDriver debt
file.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The recompiler orders an LDS atomic, and the compare-exchange loop that
writes a byte or a halfword of LDS (ds_write_b8/b16), with a
workgroup-scope OpMemoryBarrier after the update. AMD's proprietary
Windows driver 32.0.21042.62 (Radeon 0x13c0, RDNA 2) compiles that
memory barrier to s_barrier, as the code objects in its pipeline cache
show. Such updates usually sit in branches that only some waves of a
workgroup take, so those waves execute extra s_barriers, which pair
with the other waves' barriers, and the other waves read LDS before it
is written. An OpMemoryBarrier is not an execution barrier and SPIR-V
allows it in non-uniform control flow, so the SPIR-V is valid; a
device-scope OpMemoryBarrier compiles to GL0 and GL1 cache
invalidations without s_barrier there.
PS5LM Plus (PPSA99581, llama.cpp's Vulkan backend) drifted into
garbage on that driver with flash attention on. Its flash-attention
shader stores two halfwords to LDS in a branch that two of its four
waves take, inside the KV loop. The driver's code had 23 s_barrier for
the guest's 21; a replay of the dispatch differed from NVIDIA in 159
words, up to 1.7e38, and now has 21 and differs only in the last bit
of 9 words.
On that driver the fence is now device scope, which orders everything
the workgroup-scope fence did. Other drivers keep the workgroup scope.
The flag is part of the shader cache key, and agc_shader_replay takes
--device-scope-lds-memory-barriers to replay a request the way that
driver compiles it. The LDS update helper already fences, so the
second, identical fence after it in ds_min/max_f32 and the other
read-modify-write LDS operations is dropped.
agc_driver_lds_subword_store_branch stores a halfword to LDS in a
branch that waves 0-1 of four take, delays those waves and then
exchanges LDS slots across one s_barrier. Before, thread 64 read a
stale slot on the AMD driver; it passes on AMD and NVIDIA now.
agc_subgroup_barrier checks the driver quirk and that ds_write_b16,
ds_add_u32 and ds_max_f32 each emit one fence of the expected scope.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01BM2kfDbr6MydxsfTckZJDR
A guest wave's lanes see each other's LDS writes without a barrier, so
the recompiler adds a subgroup-scope OpControlBarrier between an LDS
write and the next LDS access when the guest wave is one host subgroup
(WaveLdsScope). AMD's proprietary Windows driver 32.0.21042.62 (Vulkan
driver 2.0.353, "AMD proprietary shader compiler", Radeon 0x13c0)
compiles that barrier to s_barrier, as the pipeline's internal ISA
shows (26 s_barrier for the guest's 9 in a small LDS reduction). In a
workgroup of several waves, the waves that enter a branch holding LDS
accesses then execute extra s_barriers, which pair with the other
waves' barriers, and those waves read LDS before it is written. LLPC
compiles a subgroup-scope control barrier to a wave barrier instead.
PS5LM Plus (PPSA99581) answered with garbage on that driver: the first
compute dispatch of its first graph, an RMS norm with a 512-thread
wave32 LDS tree reduction, scaled every row except its first 32
elements by a partial sum, while NVIDIA matched a NumPy reference and
the inputs were identical.
On that driver the barrier is now a subgroup-scope OpMemoryBarrier,
which it compiles to no instruction. A 32-lane subgroup is one hardware
wave there, which runs its lanes in lockstep and issues its LDS
instructions in order; Vulkan does not guarantee that, so it is
recorded as debt. Other drivers keep the control barrier. The flag is
part of the shader cache key, and agc_shader_replay takes
--wave-lds-memory-barriers to replay a request the way that driver
compiles it.
agc_driver_lds_tree_reduction sums over eight wave32 waves the way the
guest does, with its last four steps inside one wave and no s_barrier
between them: before, thread 32 read 1936 instead of 32896 on the AMD
driver; it passes on AMD and NVIDIA now. agc_subgroup_barrier checks
the driver quirk and the emitted barriers on every runner. PS5LM Plus
now answers " Paris. It is the largest city in Europe ..." on the AMD
iGPU without flash attention, as on NVIDIA and the CPU.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
With APS5_DUMP_SHADERS set, PrepareShaderWithDiagnostics wrote a request
whose preparation failed under the same shader_<address>.req name that
dispatch and draw captures use. A preparation request has no captured
resources: the registration-time one carries placeholder (zero) user
data and only the code and header memory. Replaying such a file as a
dispatch capture follows the null user-data pointer into the SRT and
fails in SrtWalker, which looks like a walker bug but is not one.
dumpRequest also de-duplicated by address, so once a preparation
failure had been dumped, a later dispatch dump of the same shader was
silently dropped and the real capture was lost.
Preparation-failure dumps are now written as shader_<address>.prepare.req,
and dumpRequest de-duplicates by file name, so the later dispatch dump
of the same address is still written. agc_shader_replay's usage text
says to replay .prepare.req files with --prepare.
The FailureCapture test expects the .prepare.req name, no plain .req
from a preparation failure, and a subsequent dispatch dump of the same
address to be written with the serialized request.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A load selected instead of branched still accesses memory, so its fallback
element must be one no write can race with: scratch keeps element 0, LDS in
workgroup or function memory reads a sentinel dword appended after the
program's LDS and lock, and storage buffers read element 0xfffffffe, which
robustBufferAccess2 turns into a zero load without a memory access. The
device enables robustBufferAccess2 when it has it and the target carries the
flag; without it, and for LDS in device memory, buffer loads keep their
selections. The aligned high word of a buffer read re-reads the low word's
element instead of word 0. agc_guarded_memory_loads traces the emitted loads
in both device modes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Add a CPU replay mode that reaches the runtime descriptor checks and specialization used by prepared shader invocations. Test active and inactive guarded descriptors and continuation after a failed request.
Co-Authored-By: Codex <[email protected]>
A hang with threads in Driver::waitMemory takes a ten-minute boot to
reproduce, and the existing switches show only the first submissions of
one queue (APS5_DUMP_QUEUE) or a rejected one (APS5_DUMP_REJECTED).
Capture. APS5_DUMP_SUBMITS=<dir> writes every submission the driver
accepts. <dir>/index.txt starts with "aps5-submissions 1" and "ordinal
queue type serial time_us dwords kind file held conditional", then one
line per submission: acceptance ordinal, queue id, graphics or compute,
serial, microseconds on the APS5_TRACE_GPU clock, dword count, kind,
file name, the dword offsets of the waits held at submit
(Submission::heldAtSubmit) and the COND_EXEC and predicated indirect
buffer ranges (Submission::conditionalEnds) as start:end, "-" for none.
<dir>/submission_<ordinal>.bin holds the copied dwords, indirect
buffers inlined. REWIND tails are listed as rewind with the serial of
their submission, suspend points as suspend without a file.
SubmissionCapture::Prepare copies the commands before the driver lock;
Take, under it after reserveOutputs, sets only the ordinal, time and
held waits and does not allocate; the file and then its index line are
written after the lock. A failed write reports the failure to the driver
(Driver::ReportFailure) and throws, so every later driver call throws
too. Unset, Submit tests a static pointer. An index written before the
held and conditional columns ("ordinal queue type serial time_us dwords
kind file") is read with both unknown.
Analysis. agc_pm4_replay <dir> [--full] decodes each submission and
throws on a packet the driver rejects (a corrupt capture). It does not
model cross-queue timing: a wait is released only when the capture
alone proves that the driver releases it (WaitMemory.cpp,
LabelHistory.cpp), and unknown, with the reason, when its outcome
depends on when another queue's stores run. A memory wait is released
when its compare always passes, when it was held at submit, when the
last store to the awaited bytes before it on its queue covers them with
an accepted value, is not conditional and no other queue stores to
those bytes anywhere in the capture, or when a RELEASE_MEM or
WRITE_DATA label earlier in the waiting submission that the label
history keeps (KeepsLabelStore, not conditional) stores an accepted
value and fewer than LabelStoreHistory other stores to the awaited
bytes can run between it and the wait: those between them on the
waiting queue and every store of another queue to those bytes, since
the history keeps the last stores per dword over all queues and finds
only a store that ran after the waiting submission entered the driver.
A REWIND is released when its tail is in the capture; one valid at
submit without a captured tail is unknown, because the driver reads the
valid bit again when the queue reaches it. A wait is unreleased when it was not held at submit and no
store that can run before it may write an accepted value (a store of
part of the awaited bytes or of a run-time value may; a store later on
its queue runs after it): the GPU does not release it, a CPU or shader
store could. A REWIND without a tail that was not valid at submit is unreleased. Every other wait is
unknown: "depends on cross-queue timing (stores on queue X at #...)"
when another queue stores to the awaited bytes, otherwise because the
last store before it on its queue is conditional, stores a run-time
value or part of the bytes, or overwrites an accepted store the capture
does not prove still in the label history. Without the held column a
wait that would be unreleased is unknown; without the conditional
column every packet after a COND_EXEC or predicated INDIRECT_BUFFER is
conditional to the end of its submission. The outcomes depend on the
order of each queue's stores and on which stores other queues make, not
on the submit order across queues. The report counts the outcomes,
routes and cross-queue unknowns, says when the index lacks the columns,
lists up to 32 unreleased and then unknown waits newest first with the
packets around them and their stores, then the last submissions of
each queue; exit code 1 when a wait is unreleased, 2 on errors.
Pm4 gains DecodeStores, DecodeWait, DecodeConditionalWrite,
DecodeConstantRamDump, the COPY_DATA and DMA_DATA field helpers, the
sample dump layout and RewindValid; Execute, ResolveStore, Validate,
DecodeMemoryCopy, the driver's sample dump, noteHeldAtSubmit and
executeRewindTail use them. LabelStoreHistory and KeepsLabelStore move
to LabelStores.hpp, used by noteLabelStore and the analysis.
Test. agc_driver_pm4_replay submits command buffers built with the
libSceAgc builders through the driver with the switch set and checks the
index, the held waits, the COND_EXEC range, the dwords, the decode and
the outcome of eight waits, including WRITE_DATA 1, WRITE_DATA 0,
WAIT == 1 in one submission, which the driver releases through its
label history. Hand-made captures check each release route, a store of
another queue to the awaited bytes that entered before or after
(unknown), the label history limit with later stores on the waiting
queue or on another queue, a fill the history does not keep, a store
in an earlier submission overwritten later, later stores, rejected
values and a missing rewind tail (unreleased), the six captures of the
reviews (none released but the wait held at submit), five groups of
waits in eight submission orders with identical outcomes, an index
without the held and conditional columns and corrupt captures. A
directory in place of the next submission file makes the write fail,
and the submit, the next submit and shutdown throw.
agc_driver_pm4_replay_tool runs the tool through
tests/Pm4ReplayTool.cmake and checks exit codes 1 and 2.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
agc_shader_replay compiles a request through Recompile, the per-draw path.
Shaders the driver prepares at registration go through PrepareShader
instead and produce a different, larger artifact, so a failure or the size
of that artifact could not be reproduced from the dumped request.
--prepare compiles the request with PrepareShader and prints the artifact's
SPIR-V size, or the first line of the failure.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--code writes the shader's code as <request name>.code in the working directory, as raw
little-endian words for llvm-mc -triple amdgcn -mcpu=gfx1030 --disassemble. --asm prints only
the first word of each instruction.
image_sample_*_o takes its texel offset from an address VGPR. The emitter
accepted only an immediate there and failed with "requires a constant texel
offset for image sampling" for any other, including an offset the program
moves into the VGPR under EXEC.
An offset that resolves to a constant at its packed address position stays a
ConstOffset. Any other becomes the Offset image operand, which Vulkan allows
on sampling instructions only with VK_KHR_maintenance8's maintenance8 feature
and the ImageGatherExtended capability. The device enables maintenance8 when
available and reports it through SpirvTarget::nonConstantImageOffsets, which
the cache key, the request serialization (version 6) and the SPIRV-Tools
validator options follow. Without it the sample still fails.
agc_shader_replay --maintenance8 recompiles a request for such a device.
A selection whose region (the blocks reachable from its arms before its
merge) is also entered from outside the header cannot be expressed as a
SPIR-V selection construct, and the structurizer gave up with "semantic
block cloning is disabled". Pixel shaders hit this through a shared
discard tail (s_mov_b64 exec, 0 / exp null / s_endpgm) that several
selections branch to, compute shaders through short-circuit conditions
that jump into a shared body, and ray-traced compute passes through an
s_cbranch_execz that skips past a join into the middle of a selection.
Such a region is made single-entry in one of two ways, chosen per region
by its estimated translated size:
- Cloning (node splitting): the blocks the outside edges reach inside the
region are copied and the outside edges redirected to the copies. When
the header's merge is a shared linear return tail, giving the header a
private copy of that tail is tried first. Each block now carries an
estimate of the SPIR-V words it translates to. The per-instruction
weights come from a fit of the recompiler's per-block output (before
spirv-opt, phis excluded) over 8,300 blocks of captured shaders: VALU
60, buffer and flat 120, SMEM 180, image 220, LDS 400. SALU (8) and
exports (20) fit near zero and keep a small weight. Lane ops (120) and
image_bvh_intersect_ray (2000) were too rare there to fit and are
weighted for the wave and traversal expansions they need. A region is cloned only when it is at most 1024 estimated
words and at most an eighth of the program (at least 256), and while
all clones stay within a quarter of the program.
- Routing through a junction, for larger regions: every edge into the
entered part (from outside, and from the header's own arms) and every
edge from the arms to the merge is sent through a new empty junction
block. Each rerouted edge first passes empty blocks that assign
Boolean route variables (the IR's goto variables); the junction tests
them in order to reach the edge's original target. The entered part
then follows the junction and is emitted once. Branch conditions are
still evaluated only at the end of their original blocks: a route
variable records the edge a block already took, so a later SCC, VCC or
EXEC write cannot change the route, and in the two-lane wave64 layout
the source branch is combined across both halves as before while the
route variable holds the same constant in both. Routing is refused
when a rerouted edge is a back edge or leaves an indirect branch, when
the region wraps around to a dominating block, or when it would change
the natural loops or make the graph irreducible; cloning bounded by the
program's estimated size is the fallback, and otherwise it throws with
the header, the entered block and both reasons.
Selections whose arms only meet again after private return tails had no
post-dominator merge. When the existing rules find none, the merge is
now looked up with edges into private return tails ignored, or taken as
the other arm when one arm only returns; the candidate must still exit
the construct only through the merge. applyBlockOrder also remaps
indirect branch targets.
agc_shader_control_flow_tests (under ctest) covers hand-assembled
gfx1030 programs: the cloned shapes above, and routed ones (an
s_cbranch_execz into an expensive region with a loop, a region inside a
loop, nested entered regions, a shared body reached after SCC is
overwritten, early returns beside an entered region, an arm that only
returns). It checks that every selection construct exits only through
its merge, follows route variables to check that the structured graph
runs the same instruction ranges under the same branch conditions as
the original, bounds the cloned instructions, recompiles each program
for 64- and 32-lane subgroups, and keeps the routed execz program within
10% of the SPIR-V of the same program without the entering branch.
Given .req files it checks their graphs instead. agc_shader_replay --cfg
prints a request's graph, now with each block's estimate, before and
after structurization.
The geometry path's merged ES/GS programs (VGT_SHADER_STAGES_EN.GS_EN)
were compiled as mesh-stage programs whose input metadata was never
filled in, so every such draw failed with "mesh shader translation
configuration is not supported".
Recompiler:
- BuildShaderStageInputInfo takes the draw's MeshConfiguration; the mesh
input helpers (primitive size, step and counts) are implemented.
- The prologue emulates one NGG subgroup per workgroup: s2 GS_TG_INFO,
s3 the merged wave info per wave, the GS vertex offsets (ES thread
index times VGT_ESGS_RING_ITEMSIZE, odd strip triangles swapped), the
primitive id, and the ES vertex and instance ids. An indexed draw's
vertex id is read from the index buffer through a raw V# the driver
puts in hidden user words 4..7.
- s_sendmsg GS_ALLOC_REQ becomes MeshAllocate. The epilogue zeroes the
allocation first, clamps it to the output limits, culls primitives no
thread exported and decodes the 9-bit vertex indices.
- The draw parameters live in the last six push dwords. A BDA fault in
a mesh program no longer stops the invocation (as for other programs
with barriers, the fault is recorded and the read returns zero), so
every invocation reaches the subgroup barriers and the epilogue.
- The cache keys cover the mesh configuration; requests serialize its
new esgsItemSize field (version 4).
Driver:
- DecodeShaderStages reads VGT_ESGS_RING_ITEMSIZE (now in the draw key)
and sizes the subgroup for its ES vertices, GS primitives and outputs.
- Stages compile below the mesh draw parameters, which every mesh draw
pushes; the mesh stage always takes part in the push range.
- The index-buffer V# is set per draw (per record on the CPU-resolved
indirect path); the index-buffer alias check spares read-only
elements.
- The workgroup and LocalSize checks count two guest lanes per
invocation on 32-wide subgroups; mesh draws may carry a first vertex
or instance.
agc_driver_mesh_tests (EXCLUDE_FROM_ALL, needs VK_EXT_mesh_shader)
recompiles a hand-assembled merged ES/GS program in the PS5 compiler's
layout and draws indexed and non-indexed triangle lists in one- and
two-wave subgroups and a triangle strip across odd-starting subgroups,
reading the pixels back. agc_shader_replay prints a request's draw and
mesh configuration.
Only the lines this branch added lose their comments; the commits before
keep them, with the reasoning behind each change, for whoever digs into
that code later.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--spv saves <request>.spv in the working directory for spirv-val; the usage
line lists --asm and --mem too.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- The loop guard's report names the shader (its code address when its plan
was built, now the translator's shader hash), and LoopGuardTripped() tells
whether any dispatch reported one.
- APS5_WATCH_MEMORY=<hex address>:<hex bytes> (with APS5_SYNC_DISPATCH=1)
keeps the last 64 contents of a range with the dispatch that left each, and
writes them to watch_<n>_<program>.bin when the loop guard first trips; the
address matches any heap base above bit 36.
- agc_shader_replay --mem prints the guest buffer V#s and which the shader
writes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A dispatch sized in threads (USE_THREAD_DIMENSIONS) launches a partial last
workgroup on RDNA2, but the host rounds it up to whole groups, so the extra
threads ran: past the end of a node list they read past it and wrote over
live records.
Such dispatches now compile a separate variant (ShaderComputeStageInfo::
partialThreads, keyed only on whether it is set) whose entry EXEC excludes
threads at or past the requested size. The size reaches the shader as three
shader data dwords after the memory offsets (StageInputKind::
DispatchThreadLimit), is mixed into the dispatch cache key and the result
memo hash, and serialized in recompile requests (version 3). Whole-group
dispatches keep their variant.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Replaces the synchronous per-command execution model, and with it the
upstream render-target, pipeline-cache, tile-buffer, batched-draw and
memory-ownership commits (5a280ce, 7ed5a0b, 46c799d, 598caac, 40df528),
whose files are removed.
- Recorder: one open command buffer per device; dispatches, resident
draws, uploads, write-backs, labels, fills and copies record into it;
timeline-semaphore serials let drains wait without the GPU mutex;
completions run in order at reap.
- one worker thread per PM4 queue; labels and stores written by the GPU
into the host import; GPU-side DISPATCH_INDIRECT; the buffer-fill
kernel replaced by vkCmdFillBuffer; a dispatch cache validated by
write watch; filler packets tolerated; rejected submissions dumped.
- guest memory: page-state cache, 64 KiB write generations, host imports
with VK_EXT_external_memory_host, persistent mirrors of the executable
image, GPU copies of misaligned sub-ranges.
- textures: shader-written images stay resident with deferred
write-back (adjacent-generation rule for packed video planes), sampled
views of storage images, DCC key handling, addrlib swizzle equations,
bidirectional detiler.
- draws: MRT, more formats, dynamic viewport/scissor, cached pipelines,
resident 64 KiB targets, draws recorded into the recorder.
- ShaderResources: two-stage build, descriptor and sampler caches, a
content-keyed resource cache; presentation from the resident image.
- a label deferred behind a completion is no longer re-stored by the
CPU at reap time unless a write-back overwrote it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_0177gthrWEBBfBu5xJDXgBfo
Dean