Every shader source keeps up to 256 materialized results of
Recompile(request, capture) in its result memo, and as many patch
recipes in its masked memo, with no limit on their bytes. In Ghost of
Tsushima's opening the memos held about 500 MiB after 5 minutes and
kept growing, at a 13 % hit rate.
Each memo entry now counts the bytes its result's vectors keep (the
bindings' vectors included; the module is shared with the prepared
module cache), plus its patch recipe in the masked memo. A result held
by both tables counts once in each. Past a budget, 256 MiB unless
APS5_RESULT_MEMO_MB sets another (0 sets none), the least recently used
entries of both tables of all sources are dropped until three quarters
of the budget is left. The recency stamps come from one clock for all
sources and both tables so they compare across them. The profile line
also reports the trimmed and held entries and their MiB.
The trim takes its lock blocking and repeats, re-reading the bytes held,
until they are within the budget, so the insertion that pushed them
over has its trim run after any other; it stops when nothing is left to
drop.
SetResultMemoBudget and ResultMemoBytes let agc_shader_result_memo_budget
check that a memo over its budget keeps the most recently used results
and drops the oldest one, with the masked memo off. A one-shot hook
inside the trim (SetResultMemoTrimHook) lets
agc_shader_result_memo_budget_threads insert two results from other
threads after the trim has read the bytes held: the memo must still end
within its budget. agc_shader_result_memo_budget_masked, with the masked
memo on, checks that a first result counts its masked entry too and that
a trim empties the masked memo.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The specialized module caches (by variant and constants, by binding plan
and by registration key) and the SPIR-V interning table held every
module's words for the rest of the run, after the results that used them
were gone. They now keep a module's interface (bindings, push data,
specialization ID and content hash) and only a weak reference to its
words; a lookup after the words were released builds them again from the
same key, checks the content hash, and keeps the specialization ID that
names the module's pipelines. The interning table keeps weak references
and drops expired ones from the bucket it searches.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A shader source loading its resource plan from the disk cache decoded
its own copy, also when another source had already loaded the same
plan: the same code registered under another plan key (another host
subgroup size or interface) stores byte-identical plan entries. In Ghost
of Tsushima's cache, 14,043 plan entries hold 8,170 distinct payloads.
Loaded plans now go through a table of weak references keyed by two
hashes of the payload, its size and the shader address the plan carries,
so equal payloads for one shader decode once and share the plan, its
compact form and its compiled SRT tape while any source uses it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The recompiler orders an LDS atomic, and the compare-exchange loop that
writes a byte or a halfword of LDS (ds_write_b8/b16), with a
workgroup-scope OpMemoryBarrier after the update. AMD's proprietary
Windows driver 32.0.21042.62 (Radeon 0x13c0, RDNA 2) compiles that
memory barrier to s_barrier, as the code objects in its pipeline cache
show. Such updates usually sit in branches that only some waves of a
workgroup take, so those waves execute extra s_barriers, which pair
with the other waves' barriers, and the other waves read LDS before it
is written. An OpMemoryBarrier is not an execution barrier and SPIR-V
allows it in non-uniform control flow, so the SPIR-V is valid; a
device-scope OpMemoryBarrier compiles to GL0 and GL1 cache
invalidations without s_barrier there.
PS5LM Plus (PPSA99581, llama.cpp's Vulkan backend) drifted into
garbage on that driver with flash attention on. Its flash-attention
shader stores two halfwords to LDS in a branch that two of its four
waves take, inside the KV loop. The driver's code had 23 s_barrier for
the guest's 21; a replay of the dispatch differed from NVIDIA in 159
words, up to 1.7e38, and now has 21 and differs only in the last bit
of 9 words.
On that driver the fence is now device scope, which orders everything
the workgroup-scope fence did. Other drivers keep the workgroup scope.
The flag is part of the shader cache key, and agc_shader_replay takes
--device-scope-lds-memory-barriers to replay a request the way that
driver compiles it. The LDS update helper already fences, so the
second, identical fence after it in ds_min/max_f32 and the other
read-modify-write LDS operations is dropped.
agc_driver_lds_subword_store_branch stores a halfword to LDS in a
branch that waves 0-1 of four take, delays those waves and then
exchanges LDS slots across one s_barrier. Before, thread 64 read a
stale slot on the AMD driver; it passes on AMD and NVIDIA now.
agc_subgroup_barrier checks the driver quirk and that ds_write_b16,
ds_add_u32 and ds_max_f32 each emit one fence of the expected scope.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01BM2kfDbr6MydxsfTckZJDR
The slot counter is reset only after earlier fragment writes, and the overflow flag stays until WaitIdle. The storage buffer is limited to NVIDIA's proprietary driver, which is where the large function array does not finish.
The recompiler marks every guest f32 add, subtract and multiply
NoContraction, because the guest rounds each one separately.
SPIRV-Cross turns each of them into a call to a helper that Metal
compiles without optimization, which dominates fragment time on
MoltenVK, and its multiply helper, fma(a, b, +0.0), turns a -0.0
product into +0.0.
On MoltenVK they are now GLSL Fma whose second step is exact, which
gives the separately rounded result: a * b is fma(a, b, -0.0), a + b
is fma(1.0, a, b) and a - b is fma(-1.0, b, a). The sin, cos and log2
series and the offset image coordinates and gather texel scale use the
same forms. The choice is a SpirvTarget flag set for
VK_DRIVER_ID_MOLTENVK, part of the shader cache key and of the request
serialization (version 44); other drivers are unchanged.
agc_driver_f32_separate_rounding checks a multiply feeding an add or a
subtract on rows where fusing them changes the result. It,
agc_driver_f32_arithmetic_nan and agc_driver_f32_denormal_flush also
run the fused forms on devices that do not use them, so lavapipe checks
them against the same rows.
Co-authored-by: Ycaro Gabriel <[email protected]>
v_add_f32, v_sub_f32, v_subrev_f32, v_mul_f32 and v_mul_legacy_f32
follow the FLOAT_MODE f32 denormal field with f32 round to nearest
even, as measured on RDNA2 (gfx1030): mode 1 keeps denormal inputs and
flushes a denormal result, and mode 2 flushes denormal inputs and keeps
a denormal result. Both need the device to keep the f32 denormals the
shader keeps, so they translate only when the module declares
DenormPreserve 32 and throw otherwise. The result flush now applies
when only the result flush bit is set; it was skipped unless the input
flush bit was set too. A constant factor no longer skips the tiny
product check while denormal inputs are kept, as a kept denormal times
such a constant can still be tiny. Directed rounding in these modes
still throws.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Each GCN instruction is re-encoded to its RDNA form and decoded by the
existing RDNA decoder, so the PS4 path shares the recompiler. Covers the
legacy VS stage (program at the VS slot, user SGPRs from s0, vertex and
instance index in v0-v3), fetch shaders linked as user-data calls, MIMG
(image_sample and friends), PS4 buffer descriptors (GCN DATA_FORMAT /
NUM_FORMAT, 44-bit base) and T# image descriptors (GCN tiling in word 7).
The GCN switch is keyed into the shader disk cache. PS5 behaviour is
unchanged: every PS4-only rule is gated on GcnGuestShaders().
When the startup probe chose the hardware f32 flush, every module declared
DenormFlushToZero 32, also shaders whose FLOAT_MODE keeps f32 denormals
(f32 denormal field 1, 2 or 3), so the device flushed what they keep and
DenormPreserve 32 was never declared. The hardware flush now applies to
shaders that flush f32 denormals (field 0, or no float mode); the others
declare DenormPreserve 32 where the device supports it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
fdfawef324 notes that current LLVM spills SGPRs into VGPRs it allocates like any other
register, so one window register can be an ordinary register early in a
program and hold spill lanes later. ReadLaneEliminator only followed a
spilled lane across M0-indexed window stores for slots never used as
ordinary registers, so RESIDENT EVIL 4 (PPSA07411) and LEGO Horizon
Adventures (PPSA14632) kept their reloaded descriptors as run-time
values and stopped. A known lane can only come from a v_writelane chain,
so keep it across a computed-index store instead; a constant-index store
of the slot still ends the spill. This is fdfawef324's suggestion on
#6277.
That such a store misses the spill cannot be proven for these programs,
so it is checked at run time where a replaced read depends on it: a
backward liveness over each slot and lane whose known value a replaced
read used puts a WindowLaneGuard before every computed-index store at
which that lane is live, and the guard records a WindowLaneSpill fault
when the store changes that lane of the slot. The driver reads the fault
buffer back for every program with such a guard and reports the dispatch
or draw as failed, so a broken assumption is no longer silent. This also
covers the pure spill slots main already folds.
Co-authored-by: fdfawef324 <[email protected]>
Co-Authored-By: Claude <[email protected]>
AcquireFrontend built the frontend's SwappcInfo with a wave size of 0, while the
driver's LinkUserDataCallees passes the dispatch's wave size. A wave64 v_cmp_e64
that reuses a call's link pair after the return, followed by a v_cndmask_b32_e64
that reads it, was refused as escaping the call/return model. The frontend now
receives the dispatch's wave size, and a frontend built for one wave size is not
reused for another.
The recompiler asked for DenormPreserve only for 16-bit and 64-bit floats,
so a device that flushes 32-bit denormals by default (RADV, MoltenVK) flushed
them whatever the guest's float mode said. target.float32DenormPreserve is set
from shaderDenormPreserveFloat32 when denormBehaviorIndependence allows a 32-bit
mode, is part of the cache key and of the request serialization (version 41),
and makes the module declare DenormPreserve 32 unless the 32-bit flush is
already taken in hardware. APS5_NO_HW_F32_PRESERVE=1 turns it off.
Preserve phi graphs with transitive non-phi users and remove unused cycles before building the SRT plan. Extend the IR regression with dead and live cycles, an empty program, and repeated elimination.
AI-assisted: OpenAI Codex
A CB_TARGET_MASK that writes fewer than four components of a 4_4_4_4
target leaves the other nibbles of each packed word as they were, which
an R16_UINT attachment cannot do. Such draws now close the attachment's
color write mask, and the pixel shader merges the written nibbles into
the target through a storage image: inside a pixel interlock it reads
the word, keeps the bits the mask leaves alone and stores it back.
The merge (export index, word mask, sample count) is part of the static
shader key, request version 40 and shader disk cache format 42; the
storage image is appended to the shader's images with a single runtime
mode, and its T# is encoded from the target's surface at draw time, so
the draw resolves it to the target's resident image, which Draw
requires. Merging shaders declare early fragment tests; shaders that
kill under late tests, export depth, stencil or coverage, use
alpha-to-coverage or a quad sample mask are refused, as are
multisampled, mipmapped, volume and array targets and more than one
merged target per draw.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A guest wave's lanes see each other's LDS writes without a barrier, so
the recompiler adds a subgroup-scope OpControlBarrier between an LDS
write and the next LDS access when the guest wave is one host subgroup
(WaveLdsScope). AMD's proprietary Windows driver 32.0.21042.62 (Vulkan
driver 2.0.353, "AMD proprietary shader compiler", Radeon 0x13c0)
compiles that barrier to s_barrier, as the pipeline's internal ISA
shows (26 s_barrier for the guest's 9 in a small LDS reduction). In a
workgroup of several waves, the waves that enter a branch holding LDS
accesses then execute extra s_barriers, which pair with the other
waves' barriers, and those waves read LDS before it is written. LLPC
compiles a subgroup-scope control barrier to a wave barrier instead.
PS5LM Plus (PPSA99581) answered with garbage on that driver: the first
compute dispatch of its first graph, an RMS norm with a 512-thread
wave32 LDS tree reduction, scaled every row except its first 32
elements by a partial sum, while NVIDIA matched a NumPy reference and
the inputs were identical.
On that driver the barrier is now a subgroup-scope OpMemoryBarrier,
which it compiles to no instruction. A 32-lane subgroup is one hardware
wave there, which runs its lanes in lockstep and issues its LDS
instructions in order; Vulkan does not guarantee that, so it is
recorded as debt. Other drivers keep the control barrier. The flag is
part of the shader cache key, and agc_shader_replay takes
--wave-lds-memory-barriers to replay a request the way that driver
compiles it.
agc_driver_lds_tree_reduction sums over eight wave32 waves the way the
guest does, with its last four steps inside one wave and no s_barrier
between them: before, thread 32 read 1936 instead of 32896 on the AMD
driver; it passes on AMD and NVIDIA now. agc_subgroup_barrier checks
the driver quirk and the emitted barriers on every runner. PS5LM Plus
now answers " Paris. It is the largest city in Europe ..." on the AMD
iGPU without flash attention, as on NVIDIA and the CPU.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A registered type-3 hull program holds the local and the hull part of a
tessellation draw in one binary, addressed through SPI_SHADER_PGM_LO_LS
(SH 0x148) with no SPI_SHADER_PGM_LO_HS: the local part runs on the lanes
below s3[7:0] and writes LDS, an s_barrier follows, and the hull part runs
on the lanes below s3[15:8] and may store the registers the local part
filled. The draw decode asked 0x148 for a type-5 local half, and
PPSA11386 (007 First Light) stopped in its level load with "registered
program refers to an incompatible shader binary type".
The draw decode now gives a registered type-3 program both the local and
the hull role and marks the tessellation configuration mergedHull. The
local stage only passes gl_VertexIndex and gl_InstanceIndex in a 16-byte
record. The hull stage runs the whole program, one invocation per output
control point: v2 and v5 come from the record of control point
min(gl_InvocationID, inputs - 1), v3 is the relative vertex id, and the
LDS becomes a per-control-point output array that a control point writes
in its own record and reads in any record of its patch after the barrier.
Registers live across the barrier stay in the invocation.
The hull stage of a merged program runs at one lane per invocation on
every device (HostSubgroupSize 1). Its lanes are gl_InvocationID, not host
subgroup invocations, so the s_cbranch_execz around the local part tests
the invocation's own lane. With the device's subgroup it took a subgroup
ballot, and lavapipe, whose subgroup stages lack the tessellation control
stage, rejected the draw ("subgroup capability 61 is unsupported for
shader stage 2").
The lowering refuses more input than output control points, LDS writes
outside the own record, reads outside the patch's records, barriers that
some invocations do not reach or that sit in a loop, reads or writes of
another lane's register, outputs beyond the device limits, and fetch
subroutines. Type 3 is prepared with its stage group like type 5 and
takes its float mode from RSRC1_HS. The hull user-data address (SH
0x102/0x103) is optional for a merged program: s[0:1] take it, unchecked,
when the draw sets it and are zero otherwise.
Both stages refer to one binary, so a draw lists each code and header
region once (AppendProgramMemory); a request's memory regions must not
overlap. An indirect draw's record words go into every program whose user
data holds the register, so the hull stage of a merged program reads
them too.
mergedHull is part of the recompile cache key, the draw decode comparison
and the request serialization (version 39).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
specializeModule caches a module per artifact variant and constant set,
so artifacts whose SPIR-V is the same and that get the same constants
each build and keep their own copy of the same specialized words. The
built words are now interned by content (both content hashes, then the
words themselves), and later equal modules share one copy. Each module
keeps its own specialization ID.
In Crimson Desert, 4500 specialized modules built in the first 235 s
held 916.6 MiB of words, of which 133.5 MiB were distinct.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
StencilReplaySafe refused every program with usesDma, including pixel
shaders whose only BDA accesses are reads (Onimusha: Way of the Sword,
PPSA27836: one global load, then image and buffer reads). Replaying such
a shader per reference value is idempotent: its only side effect is the
first-fault record, a compare-exchange that keeps the first fault; a
fault is never retried but thrown by CheckFault, and replayed draws with
a BDA table already wait and check faults before publishing. Written-page
tracking exists only in programs with bdaWrites, which stay refused.
Compiled shaders now carry bdaWrites, and ShadersWriteMemory counts a BDA
page table as a memory write only for a program that writes through it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A data-only hit (#494) reused a cached stage only when every user data word
of the stage was unchanged. A stage records the user words it depends on when
it is captured: the words its SRT walk reads as values or bases
(SrtUserDataUse), the words its binding layout forwards into shader data
(userDataRegisterReads), and the vertex fetch table words of its vertex stage
info. A cached stage is now a candidate when only words outside that set
changed; its captured memory is validated as before.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A pixel shader that reads per-vertex parameters (v_interp_mov of P10 and
P20) on a device without VK_KHR_fragment_shader_barycentric now works on
the mesh path too. The mesh program stages each such parameter in
workgroup memory and writes the values of the three vertices of every
primitive as a per-primitive output, in the order of the triangle
indices; the pixel shader reads them from there.
The emulation throws when the staging, the output components or the
output memory exceed the device's mesh limits, or when the mesh
configuration uses a provoking vertex other than the first.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
