ARCHITECTURE.md and the libSceAgcDriver debt entry now say that a
settled Windows import under 1 MiB returns to the write watch only when
no storage image overlaps it, and why.
Main's #7269 keeps the synchronous settle's buffer, memory and fence
per device in ImportSettle, so clearing one device's imports releases
only that device's objects. This branch's settle batches move there
too: each device's ImportSettle holds its batch target, memory and
batches with their fences, so switching devices no longer drops a
device's batch handles, and clearing a device waits for and destroys
only its own batches. The per-import settle state is renamed
pendingSettles to stay clear of main's per-device map, and a submitted
settle records the device whose batch it is in.
Main's #6402 stores to image tables: a table with a written entry takes
the fresh path, as it does on main, and so does one without a sampled
table binding, whose entries the fresh path rejects. The shared place
step uses the binding as a pointer. Main's #6614 frees descriptor sets
from shared pools one by one, while this branch keeps only dedicated
pools, so both stay. #7101's bulk key read counts its unmapped-page
skips as unmapped, and its key words sit next to this branch's write
stamp and compared runs.
Main's #7101 reads a key buffer's words in one call where this branch
names the reason a narrowing is skipped. Keep main's bulk read and
count both of its unmapped-page skips as unmapped.
MayEvaluate and the evaluator track the first-lane reads they are inside, so a read reached again through a loop-carried phi or a constant slot is not evaluable instead of recursing.
The deferred settle of small Windows imports could only be turned off
through the test hook. The variable turns it off for a run, so a title can
be compared with and without it on one build; unset, nothing changes.
Main's #5905 settles Linux dma-buf imports with settleImport, which is
now built on both platforms. The batched settle of Windows imports under
1 MiB, its counters and adoptSettleDevice's batch reset stay Windows-only,
and ClearHostImports drains the batches only there.
The automatic window needed 8 logical processors, which a 4-core CPU with SMT passes while the waited-for thread may run on a sibling thread, and ignored the process affinity. It now needs more than 8 processors in the affinity mask (CpuTopology), falling back to hardware_concurrency.
Brings #5872 (shader preparation on first use), main's async driver log and its geometry subgroup checks. The prepared-shader modes are the union of both lists. With this branch, the pipeline ABI of a lazily registered shader it cannot prepare leaves the stage deferred instead of throwing, so LazyRegistration now requires the stage to stay deferred with no prepared entry after that call; its dispatch and shutdown failure checks are unchanged. StderrCapture flushes the async log when it starts and before it reads, since a deferral's log line can otherwise still be queued. The undecodable-state check now matches the decode's VGT_SHADER_STAGES_EN reason rather than one message, which main's subgroup output check now gives first.
On Windows the atexit handlers of a DLL run after the process has ended
its other threads, so the logger thread can be gone with lines still
queued and the unbounded Flush waits forever (guest_shader_link timed
out on 4 of the last 7 completed main builds since #6371). Run the bounded
FlushAtEnd there instead.
Co-Authored-By: Claude <[email protected]>
At the Minecraft Dungeons II title the queue-0 worker, the draw committer and the compute worker
hand work to each other 49,000 times per 10 s through the GPU mutex, the committer's condition
variable and DrawPipeline::Drain, and each hand-off that has to wait costs a kernel wake-up. The
three workers are busy 21-34 % of the time: the GPU idles because the chain of hand-offs is slow,
not because a thread runs out of CPU.
GpuMutexType::lock retries try_lock for a window before it blocks, the committer polls its item
count for the window before it waits on its condition variable, and Drain polls for twice the
window before it waits for the committer. A hand-off that completes within the window costs no
wake-up. The window is 150 us on machines with at least 8 logical processors and 0 below that;
APS5_HANDOFF_SPIN_US sets it (0 turns it off). Ordering and the lock's semantics are unchanged.
The test agc_handoff_spin covers the window policy, the spin helpers and contended locks, and
docs/dev/ARCHITECTURE.md describes the spin.
The stage-A pre-sync of a dispatch listed only the surfaces that stage B reads on the CPU. The DCC
keys of an image that a recorded kernel writes were not in the list, so stage B waited for that
kernel in WaitForKeyWriters with the GPU mutex held and every other worker queued behind the wait.
At the Minecraft Dungeons II title the pre-sync fired once in 296,000 dispatch checks while
WaitForKeyWriters synchronized 400 times per 10 s.
PresyncSurfaces now also returns the key range of every image binding (PresyncSurface::dccKeys).
presync waits for a key range only while recorded work writes it, the key-store memo does not
already give its value (DccKeyWritersPending) and, under the mutex it takes for a moment anyway, the
pending stores do not give the keys either (DccKeyWriterNeedsSync, the test WaitForKeyWriters makes
before it synchronizes). Which keys the dispatch reads, and when, is unchanged: the locked lookup runs
as before and finds nothing pending.
APS5_NO_DCC_PRESYNC=1 leaves the key ranges out. The [presync] line counts key ranges checked,
pre-syncs that only the keys required and the outcomes of WaitForKeyWriters. The test
agc_driver_dcc_key_presync covers a kernel that writes the keys, a recorded fill of the keys and
nothing pending, and skips with the reasons check_skips.py expects when the device has no timeline
semaphores or no host import.
A graphics pipeline the store builds or a compute pipeline a dispatch builds is
appended to pipelines.journal next to the shader cache entries, as the
materialized result of each stage (the specialized SPIR-V, bindings and push
constants) plus the recipe the pipeline was built from: the stages'
identities, vertex input layout, descriptor set layout keys and the state
fields the Pipeline constructor reads. When a VulkanDevice is created with a
window (a headless device waits for its first draw or dispatch, since the
window's device usually replaces it first) the journal is loaded and its
pipelines are built on a pool of threads while the title loads, so the first
draw or dispatch of a configuration seen on an earlier run finds its pipeline
resident instead of compiling it on the queue worker.
The pipeline stores key on RecompileResult::Identity, a hash of the module
words, specialization constants, subgroup width and bindings that is the same
on every run (SharedSpirv::ContentHash, computed once per shared module),
rather than on the process-lifetime variant ids; a generated stage is keyed
like any other. CachedPipeline builds outside the store's mutex and the
library parts and link are built outside the library store's mutex, so replay
threads and queue workers build in parallel; a concurrent build of the same
key keeps the first inserted. A replayed pipeline enters the store at its
least recently used end and a full store takes none, and its optimized relink
waits for the first draw that binds it, so the title's own pipelines come
first in both. The compute pipeline creation moves into
VulkanDevice::createComputePipeline, shared by dispatches and the replay. The
store key also covers the color component mapping, the depth target's sample
count and a depth bias clamp, which the constructor reads.
The file names the device it was written for (the properties the driver's
pipeline cache file is named by) and starts over for another; a torn tail is
cut at a record boundary; two runs sharing the directory take turns on a lock
file and keep each other's records; a read failure never starts the file
over. A device cancels its replay before it is replaced or destroyed, and a
replaced device destroys its pipelines then rather than at the driver's stop.
APS5_NO_PIPELINE_JOURNAL=1 disables it; APS5_PIPELINE_JOURNAL_THREADS,
APS5_PIPELINE_JOURNAL_ENTRIES (the first recorded), APS5_PIPELINE_STORE_ENTRIES
and APS5_PIPELINE_JOURNAL_MAX_MIB bound it; APS5_PROFILE_DRAW prints a
[pipeline-journal] summary. APS5_VERIFY_SET_LAYOUTS=1 now also checks a
dispatch's set layout against the one described from its shader. The lavapipe
test suite, which shares one cache directory, runs with the journal off except
for the journal's own tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01P6hw3kRAuQdfcU9YCZgaoK
Resolves#4943's draw relocation against this branch: tagVariantRuns, which now builds a draw variant's runs and words, also records the captured words it shares with a table's supplied blocks, per region as before. A variant whose runs relocation splits keeps sharing for the draw cache's word compare, and skips the vouched validation, which already requires one run per region.
The line said FormatVersion 36 while the code is at 37, as main's says 33 against 36. Without the number it no longer goes stale on each bump or conflicts between PRs that bump it.
A Windows import under 1 MiB returns to the write watch once its settle
read has completed, though main-queue work submitted before the read may
have used the import first, and an earlier import of the same pages may
not have run its first use yet. On the NVIDIA driver (617.42) neither
marks the pages once the read has completed, so the rewatch needs no wait
for the main queue. The docs said the marking lasts until the import's
first GPU use completes; they now say that a completed GPU use ends it,
the settle read included.
agc_import_settle_first_use submits 4096 copies of a 64 KiB import on
the main queue before its settle read is queued. The import must return
to the write watch while the copies still run, and the driver must mark
none of its pages afterwards.
agc_import_settle_earlier_import records 4096 copies of a 64 KiB window
import, replaces it with the import of its whole 128 KiB range, and
submits the copies only once the range is back in the write watch. The
driver must mark none of the range's pages while they run. A driver that
keeps marking fails either test.
Retiring a Windows import whose settle read was still running waited for
the read's batch under the imports mutex, so lookups on every thread
waited with it. The retired import is now kept with that batch and freed
once the batch has completed. Only a guest unmap of its memory still
waits for the batch, in the pin waiter, which waits for the main queue
there anyway.
The [buffers] profile line counts these retirements and the unmaps that
waited, apart from the imports of 1 MiB or more settled inside their
window, which the old wait counter mixed with the retirement waits.
agc_import_settle_retire replaces a 64 KiB window import with the import
of its whole range, 64 times, right after its settle read was submitted.
No replacement may wait, at least one must find the read running, and
the last import must return to the write watch.
ARCHITECTURE.md and the debt entry describe how imports under 1 MiB are
settled without waiting and return to the write watch, which the debt
entry listed as not implemented.
The logger thread, the bounded queue that blocks its caller, the LogSink
hook, APS5_LOG_SYNC and APS5_LOG_STATS, where the queue is flushed and
what stays outside it: direct fprintf(stderr) and printf calls are written
at once, so they can appear before older queued lines.
APS5_LAZY_SHADER_PREPARE=1 keeps registration validation and decoding but skips queueing the preparation. Draws and dispatches prepare a deferred snapshot on first use and cache it; explicit ABI resolution still prepares its stages. Titles that register thousands of shaders at boot no longer hold every prepared shader in memory before the first frame.
Cover lazy dispatch, metadata and failure cases in both asynchronous and synchronous modes.
Brings the series onto main 4e9a2a871 (#1193, #1201, #1202, #6060,
#6333 and the descriptor word tracking of 682b07eb0).
Resolutions:
- ShaderMemory: the column memo's supplied and spilled blocks now sit
on #1193's page vector, page pool and word masks. Pages are found by
binary search, since a spilled block can add many pages, and a block
placed into a lazily read page fills the rest of the 16-word blocks it
touches, so a later lazy read never overwrites placed words.
- ResourceMaterializer and SharedValues: the selected-record path and
the words index carry availableDwordMask, as the snapshot hash does.
A selected record's dwords past the end of its buffer count as read
zeros, which is what a scalar buffer load returns there, so a key past
the heap still selects a null image without reading the heap.
- DescriptorCache: kept table sets live beside #1201's recycled sets.
- Tests and CMake: both sides' tests are kept.
The draw cache sets each fresh stage capture against the variants it keeps (insertDrawEntry), and
the dispatch cache sets a fresh capture against its front variant, by comparing every captured word.
A 32,768-record table column is 262,144 of them, so each compare went through 1 MiB on each side
even when both captures had copied the column from the same shared block.
A variant now records the words it copied from a block the shader memory shares (SharedWords: where
they sit in the capture and in the block, and a weak reference to the block; consecutive pages of
one block join into one entry). SameCapturedWords takes the words both captures copied from the
same place of the same block as equal, since a block never changes, and compares the words around
them as before. Two blocks with equal words, words read into pages and a released block fall back
to the word compare. The reference is weak, so a kept variant holds no block alive, and a released
block's identity cannot pass to a new one: its control block stays until the last weak reference
goes.
A result memo miss resolves every image table of the snapshot again: ResolveImageTables decodes each
distinct T#, builds the element and poison indices and fills one map record per key, which is most
of a miss for a large table. A miss whose table did not change is common: another buffer, flat SRT
word or user data word changes the result key while the column is the one the column memo shares.
ResolveImageTablesShared keeps the last resolutions and hands one out by reference when the variant
(its id), the reducing state of the direct samplers paired with table images and every column match:
base, size, records, keys, fault, outside, and the column's codes and distinct T#s by identity. The
memo holds those shared values, so a block it compares cannot be freed and reused for other words;
a column read again is a new block and resolves again even when its words are equal.
It keeps at most 4 resolutions per variant and 256 in all within 8 Mi words, counting the columns
their keys keep alive, least recently used first out, under one mutex. A resolution is immutable
once made and is shared between threads read-only. APS5_NO_TABLE_RESOLUTION_REUSE=1 turns it off.
ImageTableReuseCounts gains the resolutions made and the ones shared.
When the write tracker vouched for an unwritten image table column, the capture still copied each of
its words into ShaderMemory's pages (supplyWords) so they would be in the captured regions, and
Regions() then found them again bit by bit. On a 32,768-record column that is a 1 MiB copy per
capture (about 120 us here) and a scan of 262,144 read bits per Regions() call (about 230 us), and
the draw path calls TakeRecentRegions() and Regions() once each per stage.
The memo now keeps a column's words in one immutable block (SrtGuestWords), and supplyWords hands
that block over. ShaderMemory keeps a reference and reports its runs page by page, the regions a
read of the same words reports, merged in address order with the pages and the initial regions; a
second stage that supplies the same block only marks it recent again. The words reach pages only
when they must: a later read in the same ShaderMemory that touches them first moves the whole block
into its pages (spilled), so a word is never reported twice, and a block that overlaps pages already
fetched or another block's runs is copied as before. A spilled block stays referenced until the
ShaderMemory goes, so the regions already taken from it stay valid. ShaderMemory::Supplies() counts
the words shared, copied and spilled.
A capture that reused a memoized image table column still copied it: each record's state and code
and the distinct T#s went from the memo into the snapshot, the snapshot rebuilt its codes from the
states, and the column's per-record vectors were allocated before the memo was even looked up. Every
Recompile then hashed the codes and the distinct T#s word by word for the result memo. On a
32,768-record column of distinct T#s that was about 155 us of copying per capture and 410 us of
hashing per Recompile here, for a column nothing wrote.
A column is now built once into immutable shared values (SharedValues: its codes, its distinct T#s
and its read ranges) and the memo keeps that column. A capture that reuses it takes a reference, so
two captures of an unchanged table share one column and copy nothing, and the per-record vectors
are allocated only when the column is walked. The codes index the column's own distinct T#s
(ImageTableColumnSnapshot::words) instead of a list shared by the shader's columns, so every column
of a shader is shared whole, not only the first. A column read again after a write is a new one:
the snapshots holding the old one keep it unchanged. The result memo hashes each shared value once
and keeps the hash in it; equal contents hash equal whether they are shared or not.
ImageTableReuseCounts gains the reuses that shared a column and the columns built.
The kernel maps a dma-buf import's pages for the GPU at the first submission that uses it. On RADV
that took about 2 s for a 3 GiB import, in the second frame of Hades (PPSA03355), and its staging
pool ran out while the frame was held.
Linux now makes the settle queue from #6060 when the device imports dma-buf memory, and settles
every dma-buf import of 1 MiB or more right after it is made with settleImport's 4-byte copy. The
settle queue needs neither the GPU mutex nor the main queue's work to finish. A dma-buf import's
buffer is shared with the settle queue's family instead of aliased, since a sparse import is bound
to many memory objects. Where the sparse queue already takes that family, the settle queue is a
second queue of it.
sceAgcDriverResolveShaderAbi and sceAgcDriverResolveGraphicsStagesAbi
prepare every stage when the title creates its pipeline, and threw when
the recompiler could not: Resident Evil Village stops there on a pixel
shader whose T# pointer is a phi of user data and a ballot, and a
compute shader that calls user-data pointers stopped the same way
although its dispatch links the callees.
ResolveShaderAbi and PrepareGraphicsStages now catch the failure inside
their own preparation transaction, so the ShaderPreparationScope that
libSceAgc opens around the call still commits. The shader is logged
once, marked deferred like a shader deferred at registration, and the
ABI key is recorded, so a repeated call prepares nothing. The draw or
dispatch that uses the stage prepares it as before and fails there if
it cannot. ResolvePreparedGraphics skips the rectangles of a deferred
stage that has no artifact yet, which also lets a stage deferred at
registration through sceAgcDriverResolveGraphicsAbi.
Driver failures, ProcessShutdown and a failure stored by the shader's
registration preparation still propagate.
Queue 0 now pipelines draw commits unless APS5_PIPELINED_DRAWS=0, and draw write ranges use the exact
color, DCC, CMASK, depth and stencil sizes unless APS5_EXACT_DRAW_WRITES=0. The base tests now run the
pipeline; agc_driver_pm4_failure_inline and agc_driver_draw_cache_reuse_inline keep the inline path
covered.
Co-Authored-By: Claude <[email protected]>
A kept column the runtime cannot vouch for is read and compared with its kept words. When they
differ, the fresh walk read the same runs again, so a capture that follows every change to a table
read each word twice: 1,048,576 words per capture for a 65,536-record column, 12% slower than
before the reuse with the write watch and 16% slower outside it.
The compare now leaves its words with the snapshotter, and the fresh walk takes them instead of
reading when its plan has the same runs: words the capture has already read read the same again.
Such a capture reads each word once, as before the reuse, and the 65,536-record column costs 7%
(12% outside the write watch) more than before the reuse; the rest is keeping the column. With 16
distinct T#s it costs 5 to 8% less. agc_shader_table_snapshot_reuse expects a changed table to be
read exactly once per capture.
Every read and store through GuestMemory calls the flush hook first. Recorder's FlushForAccess waits
for recorded GPU work over the range, records a queued label that writes it, publishes the unit
shadows over it and stores the storage image results kept on the GPU for it. A storage image result
is stamped only when it is stored (its write-back's MarkWritten), so ShaderMemory's stamp and check,
which never read the column, could vouch for words a read would have replaced: the column skipped
its read and the snapshot kept the T#s from before the result.
stampWrites and unchangedSince now call GuestMemory::FlushGpuWrites over the range after the pending
write query and before the tracker, as a read does, and as the dispatch cache stores pending images
before it validates an entry. The stamp then covers what the flush stored, so the capture after it
skips the read again.
agc_shader_table_snapshot_reuse installs a flush hook that stores a new T# into the table on its first
call over it. Without the flush in unchangedSince the reused snapshot keeps the old T#; without the
one in stampWrites the next capture compares the table instead of skipping its read.
Every capture of a shader with an image table read every record of each column and looked each
record up among the snapshot's distinct descriptors. A 32,768-record column of distinct T#s cost
6.1 ms per capture here (2.3 ms with 16 distinct T#s), and Resident Evil Village's title captures
such columns on nearly every draw.
TableSnapshotter now keeps the last 16 column snapshots per thread (at most 4 Mi words in all,
least recently used first out), keyed by heap base and size, record shape and key count: the runs
it read, their words, each record's state and code, and the distinct T#s. A narrowed, selected or
pointer-indirect column, or one with an unmapped record, is not kept. A capture of a kept column
takes the codes and the distinct T#s from it with no per-record work. When they are the snapshot's
first words they are taken as they are, and the words index is built only if a later column needs
it, so the result is the snapshot a fresh walk would produce, codes and words order included.
The runtime gets three optional callbacks: stampWrites gives a stamp for a range before any of it is
read, unchangedSince tells whether nothing wrote the range since, and supplyWords hands words that
equal guest memory to the capture as read, so they stay in the captured regions the draw cache
compares. ShaderMemory implements them with the write tracker. The stamp is CollectWrites's
generation, refused when this ShaderMemory already fetched a page of the range or recorded GPU work
writes it. A range is unchanged while no block of it is stamped after the stamp (collected CPU
stores, the driver's Write, WriteChanged and StoreOwnBytes, MarkWritten for GPU writes into
imports), no recorded GPU work writes it and the guest allocation generation is the stamp's. When
every run is vouched for, the column is not read at all; otherwise its runs are read and the
snapshot is reused only when they are equal word for word.
The 32,768-record column now costs 0.48 ms per capture with no word read when unwritten, and about
0.9 ms outside the write watch. APS5_NO_TABLE_SNAPSHOT_REUSE=1 turns the reuse off. The [image-table]
profile line counts the reuses, and ResourceMaterializer::TableReuse returns them with the words the
snapshotter read.
The second queue that settles imports is always created on Windows, and a device without one keeps
the comparisons. The other-family settle path is tested through the recorder test device built
with its settle queue on another family (agc_import_watch_other_family) instead of
APS5_SETTLE_OTHER_FAMILY.
Since #757, a T# loaded from a buffer heap at a runtime offset is an
image table column, and every capture snapshots the whole heap, up to
65,536 records, even when user data or an SRT read fixes the offset.
When the T#'s eight dwords and its offset are runtime values the SRT
walker evaluates, the column is now selected: the offset is kept as a
descriptor source, the shader indexes entry 0, and the snapshot reads
only the record at the evaluated offset, with the heap base moved to
that record.
The column keeps a table's semantics. An invalid T#, an unmapped record
or an offset that carries past 32 bits is poison that faults only when a
lane selects it; an offset the walker cannot read faults the table
instead of throwing; the record's mode is picked per entry, so no
pipeline variant is added; and a dispatch that writes the record faults
it as written by its dispatch. Dwords past the heap read zeros, as the
scalar loads would, and a record past the key cap is read rather than
poisoned.
A column stays a whole table when its offset is chosen per lane, or when
the shader may write the memory the offset is read from at run time: the
buffer it is read from (by V#), or any memory when the read or a write
goes through a pointer or a GPU-only descriptor. Samplers are unchanged:
an S# the walker evaluates is already a direct sampler. The plan stores
the selection, so the disk cache format is 36.
Five agc_shader_memory and agc_shader_bindless checks asserted the old
planning and now assert the selected record:
- "image table SRT key": a key read through the SRT narrowed a 4-key
table to records 0 and 2. It now selects record 2 alone, captures no
other record, and a changed key moves the capture without changing the
module or its disk key.
- "image table key union" and "image table late key": two keys of one
heap shared one column with two value key domains. Each key now
selects its own record, as two one-record tables.
- "image table unkeyed access": a read keyed through the SRT shared a
whole column with a read keyed per lane. The first now selects its
record; the second keeps the whole, unnarrowed table.
- "uniform table image": the T# at a user data offset was one key of a
full table. It is now the one selected record, and the record the user
data does not select is not read.
A value key domain only arises for a key the walker evaluates, so it no
longer narrows an image column; key buffer domains (keys loaded per
lane) still do.
Yunus Shaikh