Every shader source keeps up to 256 materialized results of
Recompile(request, capture) in its result memo, and as many patch
recipes in its masked memo, with no limit on their bytes. In Ghost of
Tsushima's opening the memos held about 500 MiB after 5 minutes and
kept growing, at a 13 % hit rate.
Each memo entry now counts the bytes its result's vectors keep (the
bindings' vectors included; the module is shared with the prepared
module cache), plus its patch recipe in the masked memo. A result held
by both tables counts once in each. Past a budget, 256 MiB unless
APS5_RESULT_MEMO_MB sets another (0 sets none), the least recently used
entries of both tables of all sources are dropped until three quarters
of the budget is left. The recency stamps come from one clock for all
sources and both tables so they compare across them. The profile line
also reports the trimmed and held entries and their MiB.
The trim takes its lock blocking and repeats, re-reading the bytes held,
until they are within the budget, so the insertion that pushed them
over has its trim run after any other; it stops when nothing is left to
drop.
SetResultMemoBudget and ResultMemoBytes let agc_shader_result_memo_budget
check that a memo over its budget keeps the most recently used results
and drops the oldest one, with the masked memo off. A one-shot hook
inside the trim (SetResultMemoTrimHook) lets
agc_shader_result_memo_budget_threads insert two results from other
threads after the trim has read the bytes held: the memo must still end
within its budget. agc_shader_result_memo_budget_masked, with the masked
memo on, checks that a first result counts its masked entry too and that
a trim empties the masked memo.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The specialized module caches (by variant and constants, by binding plan
and by registration key) and the SPIR-V interning table held every
module's words for the rest of the run, after the results that used them
were gone. They now keep a module's interface (bindings, push data,
specialization ID and content hash) and only a weak reference to its
words; a lookup after the words were released builds them again from the
same key, checks the content hash, and keeps the specialization ID that
names the module's pipelines. The interning table keeps weak references
and drops expired ones from the bucket it searches.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The recompiler orders an LDS atomic, and the compare-exchange loop that
writes a byte or a halfword of LDS (ds_write_b8/b16), with a
workgroup-scope OpMemoryBarrier after the update. AMD's proprietary
Windows driver 32.0.21042.62 (Radeon 0x13c0, RDNA 2) compiles that
memory barrier to s_barrier, as the code objects in its pipeline cache
show. Such updates usually sit in branches that only some waves of a
workgroup take, so those waves execute extra s_barriers, which pair
with the other waves' barriers, and the other waves read LDS before it
is written. An OpMemoryBarrier is not an execution barrier and SPIR-V
allows it in non-uniform control flow, so the SPIR-V is valid; a
device-scope OpMemoryBarrier compiles to GL0 and GL1 cache
invalidations without s_barrier there.
PS5LM Plus (PPSA99581, llama.cpp's Vulkan backend) drifted into
garbage on that driver with flash attention on. Its flash-attention
shader stores two halfwords to LDS in a branch that two of its four
waves take, inside the KV loop. The driver's code had 23 s_barrier for
the guest's 21; a replay of the dispatch differed from NVIDIA in 159
words, up to 1.7e38, and now has 21 and differs only in the last bit
of 9 words.
On that driver the fence is now device scope, which orders everything
the workgroup-scope fence did. Other drivers keep the workgroup scope.
The flag is part of the shader cache key, and agc_shader_replay takes
--device-scope-lds-memory-barriers to replay a request the way that
driver compiles it. The LDS update helper already fences, so the
second, identical fence after it in ds_min/max_f32 and the other
read-modify-write LDS operations is dropped.
agc_driver_lds_subword_store_branch stores a halfword to LDS in a
branch that waves 0-1 of four take, delays those waves and then
exchanges LDS slots across one s_barrier. Before, thread 64 read a
stale slot on the AMD driver; it passes on AMD and NVIDIA now.
agc_subgroup_barrier checks the driver quirk and that ds_write_b16,
ds_add_u32 and ds_max_f32 each emit one fence of the expected scope.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01BM2kfDbr6MydxsfTckZJDR
The slot counter is reset only after earlier fragment writes, and the overflow flag stays until WaitIdle. The storage buffer is limited to NVIDIA's proprietary driver, which is where the large function array does not finish.
The recompiler marks every guest f32 add, subtract and multiply
NoContraction, because the guest rounds each one separately.
SPIRV-Cross turns each of them into a call to a helper that Metal
compiles without optimization, which dominates fragment time on
MoltenVK, and its multiply helper, fma(a, b, +0.0), turns a -0.0
product into +0.0.
On MoltenVK they are now GLSL Fma whose second step is exact, which
gives the separately rounded result: a * b is fma(a, b, -0.0), a + b
is fma(1.0, a, b) and a - b is fma(-1.0, b, a). The sin, cos and log2
series and the offset image coordinates and gather texel scale use the
same forms. The choice is a SpirvTarget flag set for
VK_DRIVER_ID_MOLTENVK, part of the shader cache key and of the request
serialization (version 44); other drivers are unchanged.
agc_driver_f32_separate_rounding checks a multiply feeding an add or a
subtract on rows where fusing them changes the result. It,
agc_driver_f32_arithmetic_nan and agc_driver_f32_denormal_flush also
run the fused forms on devices that do not use them, so lavapipe checks
them against the same rows.
Co-authored-by: Ycaro Gabriel <[email protected]>
ResultKeyMemo declared std::atomic<std::shared_ptr<...>>. libc++ has no
atomic<shared_ptr> specialization, so with Apple clang the recompiler
headers stop at std::atomic's trivially-copyable static assertion
(request_serializer_tests in the macOS relinker build). The memo now uses
AtomicSharedPtr from libc, which is that specialization where the library
provides it and a locked shared_ptr with the same load and store otherwise.
The header joins the shader disk cache key's sources, and
request_serializer_tests gets core/libs on its include path.
A FLAT_SHADE input without pass-through that a pixel shader reads
through v_interp_mov_f32 P20 and not P10 stayed an explicit per-vertex
input and read vertex 2 minus vertex 0. For a flat value that is the same
on every vertex the difference is 0, and with the first provoking vertex
nothing refused it.
ps5-opengl lowers every flat input to a load of one vertex: vertex 1
(P10) by default, which #4030 reads as the provoking vertex, and vertex 2
(P20) for triangles when GL's flatshade_first is set. SuperTuxKart's
deferred point lights are drawn that way, so all twelve flat inputs of
pointlight.frag read 0, the attenuation 0 * -inf was NaN, nothing was
discarded, and the starting-line referee's light quad blacked out the
scene behind it (#8364).
A P20-only read now returns the provoking vertex's value, as P10 and P0
do. The lane was not measured on the console either, so FragmentParameter
records it and a pipeline using the last provoking vertex throws. An input
read through both P10 and P20 keeps its per-vertex differences.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
fdfawef324 notes that current LLVM spills SGPRs into VGPRs it allocates like any other
register, so one window register can be an ordinary register early in a
program and hold spill lanes later. ReadLaneEliminator only followed a
spilled lane across M0-indexed window stores for slots never used as
ordinary registers, so RESIDENT EVIL 4 (PPSA07411) and LEGO Horizon
Adventures (PPSA14632) kept their reloaded descriptors as run-time
values and stopped. A known lane can only come from a v_writelane chain,
so keep it across a computed-index store instead; a constant-index store
of the slot still ends the spill. This is fdfawef324's suggestion on
#6277.
That such a store misses the spill cannot be proven for these programs,
so it is checked at run time where a replaced read depends on it: a
backward liveness over each slot and lane whose known value a replaced
read used puts a WindowLaneGuard before every computed-index store at
which that lane is live, and the guard records a WindowLaneSpill fault
when the store changes that lane of the slot. The driver reads the fault
buffer back for every program with such a guard and reports the dispatch
or draw as failed, so a broken assumption is no longer silent. This also
covers the pure spill slots main already folds.
Co-authored-by: fdfawef324 <[email protected]>
Co-Authored-By: Claude <[email protected]>
A point list routed through the vertex path was refused, and so was
PA_CL_VS_OUT_CNTL.USE_VTX_POINT_SIZE on it. OliSe Player (PPSA01153)
stops on the latter with two of its MilkDrop presets.
The vertex path now draws point lists as Vulkan points. The vertex stage
writes PointSize: the misc export's X clamped to PA_SU_POINT_MINMAX when
USE_VTX_POINT_SIZE is set, and the PA_SU_POINT_SIZE height otherwise.
The point state travels in the recompile request (version 31) and the
cache and draw keys. Non-square register sizes, inverted clamps, point
sprite texture coordinates and, without largePoints, sizes other than
one pixel throw.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
main 7ce3b1df3e took disk-cache format 43 (#8295) for its own layout, so this
branch's format moves to 44. The request serializer stays at 42: main
still writes 41.
The recompiler asked for DenormPreserve only for 16-bit and 64-bit floats,
so a device that flushes 32-bit denormals by default (RADV, MoltenVK) flushed
them whatever the guest's float mode said. target.float32DenormPreserve is set
from shaderDenormPreserveFloat32 when denormBehaviorIndependence allows a 32-bit
mode, is part of the cache key and of the request serialization (version 41),
and makes the module declare DenormPreserve 32 unless the 32-bit flush is
already taken in hardware. APS5_NO_HW_F32_PRESERVE=1 turns it off.
The refusal of a merging shader that writes guest memory under late
tests fired in two cases where the early tests change nothing.
The first was a draw with no depth, stencil or depth-bounds test. Early
tests reject no pixel there, so every write the console makes still
happens. The target merge now carries whether the draw has such a
test, and the refusal applies only then. This adds one byte to the
version 40 merge fields of the recompile request.
The second was any GDS binding, including a shader that only reads GDS.
The refusal now looks for GDS writes, atomics, appends and consumes.
Tests: the depth and stencil decode checks the new flag, and the
static shader key depends on it. A counter-adding shader under late
tests compiles without a test and throws with one. A late-test merge
that reads GDS compiles, and one that adds to GDS throws.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A packed 4_4_4_4 partial write merges inside the pixel shader after early
depth and stencil tests. Moving the tests ahead of a pixel kill only changes
the result when the draw writes depth or stencil, so the target merge now
carries whether the draw does (depth write, or stencil with a write mask and
a non-keep operation), and the late-test kill refusal applies only then.
The flag is part of the static shader key, the serialized request (now
version 40) and the pixel input info.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A CB_TARGET_MASK that writes fewer than four components of a 4_4_4_4
target leaves the other nibbles of each packed word as they were, which
an R16_UINT attachment cannot do. Such draws now close the attachment's
color write mask, and the pixel shader merges the written nibbles into
the target through a storage image: inside a pixel interlock it reads
the word, keeps the bits the mask leaves alone and stores it back.
The merge (export index, word mask, sample count) is part of the static
shader key, request version 40 and shader disk cache format 42; the
storage image is appended to the shader's images with a single runtime
mode, and its T# is encoded from the target's surface at draw time, so
the draw resolves it to the target's resident image, which Draw
requires. Merging shaders declare early fragment tests; shaders that
kill under late tests, export depth, stencil or coverage, use
alpha-to-coverage or a quad sample mask are refused, as are
multisampled, mipmapped, volume and array targets and more than one
merged target per draw.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
CB_COLOR_INFO format 19 with UNORM and COMP_SWAP 2 lists A, B, G, R from the
least significant bits. No Vulkan device is required to render into the 4-bit
formats and NVIDIA exposes none of them as color attachments, so the target
is an R16_UINT image and the pixel shader packs the exports into the guest
layout, as the 10_11_11 target does. Writes of fewer than four components,
blending, DCC and truncating rounding are refused.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A guest wave's lanes see each other's LDS writes without a barrier, so
the recompiler adds a subgroup-scope OpControlBarrier between an LDS
write and the next LDS access when the guest wave is one host subgroup
(WaveLdsScope). AMD's proprietary Windows driver 32.0.21042.62 (Vulkan
driver 2.0.353, "AMD proprietary shader compiler", Radeon 0x13c0)
compiles that barrier to s_barrier, as the pipeline's internal ISA
shows (26 s_barrier for the guest's 9 in a small LDS reduction). In a
workgroup of several waves, the waves that enter a branch holding LDS
accesses then execute extra s_barriers, which pair with the other
waves' barriers, and those waves read LDS before it is written. LLPC
compiles a subgroup-scope control barrier to a wave barrier instead.
PS5LM Plus (PPSA99581) answered with garbage on that driver: the first
compute dispatch of its first graph, an RMS norm with a 512-thread
wave32 LDS tree reduction, scaled every row except its first 32
elements by a partial sum, while NVIDIA matched a NumPy reference and
the inputs were identical.
On that driver the barrier is now a subgroup-scope OpMemoryBarrier,
which it compiles to no instruction. A 32-lane subgroup is one hardware
wave there, which runs its lanes in lockstep and issues its LDS
instructions in order; Vulkan does not guarantee that, so it is
recorded as debt. Other drivers keep the control barrier. The flag is
part of the shader cache key, and agc_shader_replay takes
--wave-lds-memory-barriers to replay a request the way that driver
compiles it.
agc_driver_lds_tree_reduction sums over eight wave32 waves the way the
guest does, with its last four steps inside one wave and no s_barrier
between them: before, thread 32 read 1936 instead of 32896 on the AMD
driver; it passes on AMD and NVIDIA now. agc_subgroup_barrier checks
the driver quirk and the emitted barriers on every runner. PS5LM Plus
now answers " Paris. It is the largest city in Europe ..." on the AMD
iGPU without flash attention, as on NVIDIA and the CPU.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A registered type-3 hull program holds the local and the hull part of a
tessellation draw in one binary, addressed through SPI_SHADER_PGM_LO_LS
(SH 0x148) with no SPI_SHADER_PGM_LO_HS: the local part runs on the lanes
below s3[7:0] and writes LDS, an s_barrier follows, and the hull part runs
on the lanes below s3[15:8] and may store the registers the local part
filled. The draw decode asked 0x148 for a type-5 local half, and
PPSA11386 (007 First Light) stopped in its level load with "registered
program refers to an incompatible shader binary type".
The draw decode now gives a registered type-3 program both the local and
the hull role and marks the tessellation configuration mergedHull. The
local stage only passes gl_VertexIndex and gl_InstanceIndex in a 16-byte
record. The hull stage runs the whole program, one invocation per output
control point: v2 and v5 come from the record of control point
min(gl_InvocationID, inputs - 1), v3 is the relative vertex id, and the
LDS becomes a per-control-point output array that a control point writes
in its own record and reads in any record of its patch after the barrier.
Registers live across the barrier stay in the invocation.
The hull stage of a merged program runs at one lane per invocation on
every device (HostSubgroupSize 1). Its lanes are gl_InvocationID, not host
subgroup invocations, so the s_cbranch_execz around the local part tests
the invocation's own lane. With the device's subgroup it took a subgroup
ballot, and lavapipe, whose subgroup stages lack the tessellation control
stage, rejected the draw ("subgroup capability 61 is unsupported for
shader stage 2").
The lowering refuses more input than output control points, LDS writes
outside the own record, reads outside the patch's records, barriers that
some invocations do not reach or that sit in a loop, reads or writes of
another lane's register, outputs beyond the device limits, and fetch
subroutines. Type 3 is prepared with its stage group like type 5 and
takes its float mode from RSRC1_HS. The hull user-data address (SH
0x102/0x103) is optional for a merged program: s[0:1] take it, unchecked,
when the draw sets it and are zero otherwise.
Both stages refer to one binary, so a draw lists each code and header
region once (AppendProgramMemory); a request's memory regions must not
overlap. An indirect draw's record words go into every program whose user
data holds the register, so the hull stage of a merged program reads
them too.
mergedHull is part of the recompile cache key, the draw decode comparison
and the request serialization (version 39).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
specializeModule caches a module per artifact variant and constant set,
so artifacts whose SPIR-V is the same and that get the same constants
each build and keep their own copy of the same specialized words. The
built words are now interned by content (both content hashes, then the
words themselves), and later equal modules share one copy. Each module
keeps its own specialization ID.
In Crimson Desert, 4500 specialized modules built in the first 235 s
held 916.6 MiB of words, of which 133.5 MiB were distinct.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
StencilReplaySafe refused every program with usesDma, including pixel
shaders whose only BDA accesses are reads (Onimusha: Way of the Sword,
PPSA27836: one global load, then image and buffer reads). Replaying such
a shader per reference value is idempotent: its only side effect is the
first-fault record, a compare-exchange that keeps the first fault; a
fault is never retried but thrown by CheckFault, and replayed draws with
a BDA table already wait and check faults before publishing. Written-page
tracking exists only in programs with bdaWrites, which stay refused.
Compiled shaders now carry bdaWrites, and ShadersWriteMemory counts a BDA
page table as a memory write only for a program that writes through it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A data-only hit (#494) reused a cached stage only when every user data word
of the stage was unchanged. A stage records the user words it depends on when
it is captured: the words its SRT walk reads as values or bases
(SrtUserDataUse), the words its binding layout forwards into shader data
(userDataRegisterReads), and the vertex fetch table words of its vertex stage
info. A cached stage is now a candidate when only words outside that set
changed; its captured memory is validated as before.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Adrià