From 766f6de920ab1639c64a97df8ae063b1cf61d4e8 Mon Sep 17 00:00:00 2001 From: Yuri Olive Date: Tue, 13 Oct 2026 20:51:48 -0300 Subject: [PATCH 01/27] fix(agc): read an unnormalized sampler without a mip filter at its base level A guest sampler with FORCE_UNNORMALIZED and a mip filter of none read a multi-level texture and the driver refused it: an unnormalized sampler is accepted on a view with several levels only when every use is image_sample_lz and the S# has no minimum LOD and no LOD bias. Kena: Bridge of Spirits (PPSA01746) binds such a sampler (implicit LOD, point filters, mip filter none, LOD bias 7) to a 9-level texture and stopped with "guest texture sampled with unnormalized coordinates is not a single-layer 1D or 2D view ...". With the mip filter off the sampler reads only level 0, so the base-level path the driver already has for unnormalized samplers is exact for it whatever its uses, minimum LOD or bias. A sampler with a point or linear mip filter is unchanged. Measured on a PS5 (gfx1013) through a sampler descriptor rewritten to force unnormalized coordinates, a 64x64 texture of four levels with a distinct marker per level, coordinates in texel units: - mip filter point, explicit LOD 0, 1, 2, 3: level 0, 1, 2, 3, 256 of 256 texels each, coordinates read in texel units of the selected level - mip filter point, implicit LOD from a quad derivative of 8 texels: level 3 - mip filter none with LOD bias 7: explicit LOD 2, explicit LOD 3 and the implicit LOD of 3 each read level 0, 256 of 256 texels Tested: Windows 11, MinGW-w64 GCC 15.2.0, Release, RTX 4070 SUPER. agc_shader_memory_tests passes with the change and fails without it on the new assertion; agc_driver_image_sample_unnormalized_tests, agc_shader_disk_cache_tests and agc_driver_graphics_tests pass. --- core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp | 10 ++++++++++ .../Optimization/src/DescriptorBindingBuilder.cpp | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp b/core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp index 9021647520..bb68c2e371 100644 --- a/core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp @@ -3447,6 +3447,16 @@ void verifyUnusedUnnormalizedSampler() { snapshot.samplers[0].dwords[2] = 0x05500180u; require(atBaseLevel(levelZero) == std::vector{false}, "unnormalized samplers: an S# with a LOD bias was flagged as exact at the base level"); snapshot.samplers[0].dwords[2] = 0x05500000u; + auto implicitUse = levelZero; + implicitUse.samplers[0].uses = SamplerUseImplicitLod; + snapshot.samplers[0].dwords[1] = 0x00fff100u; + snapshot.samplers[0].dwords[2] = 0x01500180u; + require(atBaseLevel(implicitUse) == std::vector{true}, "unnormalized samplers: an image read through an S# without a mip filter was not flagged as exact at its base level"); + require(atBaseLevel(selectedLod) == std::vector{true}, "unnormalized samplers: an image read at a selected LOD through an S# without a mip filter was not flagged as exact at its base level"); + snapshot.samplers[0].dwords[2] = 0x05500180u; + require(atBaseLevel(implicitUse) == std::vector{false}, "unnormalized samplers: an image read through an S# with a point mip filter was flagged as exact at its base level"); + snapshot.samplers[0].dwords[1] = 0x00fff000u; + snapshot.samplers[0].dwords[2] = 0x05500000u; snapshot.samplers[0].dwords[0] = 0x00000092u; const auto normalized = populate(info); require(normalized[0].imageUnnormalized == std::vector{false} && normalized[1].samplerUnnormalized == std::vector{false}, "unnormalized samplers: a normalized S# was flagged"); diff --git a/core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp b/core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp index 6081365137..d65fa13fbe 100644 --- a/core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp +++ b/core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp @@ -201,7 +201,7 @@ UnnormalizedProof ProveUnnormalized(const ShaderInfo& info, const ResourceSnapsh if (!exact) break; if (pair.image != resource) continue; const auto& words = snapshot.samplers.at(pair.sampler).dwords; - exact = proof.samplers[pair.sampler] && info.samplers.at(pair.sampler).uses == SamplerUseExplicitLod && (words[1] & 0xfffu) == 0u && (words[2] & 0x3fffu) == 0u; + exact = proof.samplers[pair.sampler] && (((words[2] >> 26u) & 3u) == 0u || (info.samplers.at(pair.sampler).uses == SamplerUseExplicitLod && (words[1] & 0xfffu) == 0u && (words[2] & 0x3fffu) == 0u)); } proof.imagesAtBaseLevel[resource] = exact; } From dfc71c84181d79169819f9fda3fa7de23dde15ee Mon Sep 17 00:00:00 2001 From: fdfawef324 <287205748+fdfawef324@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:52:20 -0400 Subject: [PATCH 02/27] fix(agc): fold bounded vertex lane masks before stencil replay Preserve replay refusal for observable lane-dependent values, non-vertex stages, unknown subgroup widths and varying-size modules. Validate the actual native vertex subgroup contract and retain compiled single-lane targets. Co-Authored-By: Codex --- .../libSceAgcDriver/tests/ConstantBitTest.cpp | 47 +++++++++++++++++++ .../include/Optimization/ConstantFolder.hpp | 2 + .../Optimization/src/ConstantFolder.cpp | 14 ++++++ core/shader/recompiler/Recompiler.cpp | 6 +++ 4 files changed, 69 insertions(+) diff --git a/core/libs/prx/libSceAgcDriver/tests/ConstantBitTest.cpp b/core/libs/prx/libSceAgcDriver/tests/ConstantBitTest.cpp index 38131c019d..0d425edf5d 100644 --- a/core/libs/prx/libSceAgcDriver/tests/ConstantBitTest.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/ConstantBitTest.cpp @@ -1,5 +1,6 @@ #include "IntermediateRepresentation/IrBuilder.hpp" #include "Optimization/ConstantFolder.hpp" +#include "Optimization/DeadCodeEliminator.hpp" #include #include #include @@ -44,6 +45,41 @@ bool reversesTo(std::uint32_t word, std::uint32_t expected) { return result->HasImmediate() && result->ImmediateBool(); } +bool vertexProjection(std::uint32_t low, std::uint32_t hostSubgroup, std::uint32_t version, bool expected, bool directLane = false, IrShaderStage stage = IrShaderStage::Vertex) { + IrProgram program; + program.Resources().stage = stage; + program.SetWaveSize(64u); + auto& block = program.CreateBlock(); + program.SetEntryBlock(block); + program.BlockOrder().push_back(&block); + IrBuilder builder(program); + builder.SetInsertionPoint(block); + auto& lane = builder.Emit(IrOpcode::LaneId, IrType::U32, {}); + auto& half = builder.Emit(IrOpcode::ULessThan32, IrType::Bool, {&lane, &builder.Constant(32u)}); + auto& word = builder.Emit(IrOpcode::SelectU32, IrType::U32, {&half, &builder.Constant(low), &builder.Constant(0u)}); + auto& amount = builder.Emit(IrOpcode::BitwiseAnd32, IrType::U32, {&lane, &builder.Constant(31u)}); + auto& shifted = builder.Emit(IrOpcode::ShiftRightLogical32, IrType::U32, {&word, &amount}); + auto& bit = builder.Emit(IrOpcode::BitwiseAnd32, IrType::U32, {&shifted, &builder.Constant(1u)}); + auto& test = builder.Emit(IrOpcode::INotEqual32, IrType::Bool, {&bit, &builder.Constant(0u)}); + auto& user = builder.Emit(IrOpcode::Reference, IrType::Void, {&test}); + if (directLane) static_cast(builder.Emit(IrOpcode::ReferenceU32, IrType::Void, {&lane})); + static_cast(builder.Emit(IrOpcode::Return, IrType::Void, {})); + ConstantFolder folder; + const bool folded = folder.FoldVertexLaneBounds(program, hostSubgroup, version); + const bool shouldFold = stage == IrShaderStage::Vertex && hostSubgroup == 32u && version >= 0x00010000u && version < 0x00010600u; + if (folded != shouldFold) return false; + if (!shouldFold && half.Opcode() != IrOpcode::ULessThan32) return false; + if (shouldFold && (!half.Resolve()->HasImmediate() || !half.Resolve()->ImmediateBool())) return false; + folder.Fold(program); + DeadCodeEliminator{}.RemoveIdentities(program); + DeadCodeEliminator{}.Eliminate(program); + const auto* result = user.Argument(0)->Resolve(); + if (expected) return result->HasImmediate() && result->ImmediateBool(); + bool retainedLane = false; + for (const auto* instruction : block.Instructions()) retainedLane = retainedLane || instruction->Opcode() == IrOpcode::LaneId; + return retainedLane; +} + bool run(const char* name, bool passed) { if (!passed) std::fprintf(stderr, "%s\n", name); return passed; @@ -58,6 +94,17 @@ int main() { passed &= run("an all-one word must fold the lane bit test to true", foldsTo(0xffffffffu, true, true)); passed &= run("a mixed word must keep the lane bit test", foldsTo(0x0000ffffu, false, true)); passed &= run("an unmasked lane number must keep the lane bit test", foldsTo(0xffffffffu, true, false)); + passed &= run("a fixed 32-lane vertex subgroup removes a constant lower-half export guard", vertexProjection(0xffffffffu, 32u, 0x00010300u, true)); + passed &= run("a single set bit remains dependent on the vertex lane", vertexProjection(1u, 32u, 0x00010300u, false)); + passed &= run("a partial mask remains dependent on the vertex lane", vertexProjection(0x0000ffffu, 32u, 0x00010300u, false)); + passed &= run("a real vertex lane output stays observable after folding its guard", vertexProjection(0xffffffffu, 32u, 0x00010300u, false, true)); + passed &= run("a 64-lane subgroup keeps a lower-half export guard", vertexProjection(0xffffffffu, 64u, 0x00010300u, false)); + passed &= run("an unknown subgroup keeps a lower-half export guard", vertexProjection(0xffffffffu, 0u, 0x00010300u, false)); + passed &= run("a single-lane request does not borrow the physical 32-lane proof", vertexProjection(0xffffffffu, 1u, 0x00010300u, false)); + passed &= run("SPIR-V 1.6 does not borrow a fixed subgroup-size proof", vertexProjection(0xffffffffu, 32u, 0x00010600u, false)); + passed &= run("an unknown module version keeps the lane guard", vertexProjection(0xffffffffu, 32u, 0u, false)); + passed &= run("the lane-bound optimization does not change compute shaders", vertexProjection(0xffffffffu, 32u, 0x00010300u, false, false, IrShaderStage::Compute)); + passed &= run("the lane-bound optimization does not change pixel shaders", vertexProjection(0xffffffffu, 32u, 0x00010300u, false, false, IrShaderStage::Pixel)); passed &= run("a reversed zero must fold to zero", reversesTo(0x00000000u, 0x00000000u)); passed &= run("a reversed single bit must fold to the opposite end", reversesTo(0x00000001u, 0x80000000u)); passed &= run("a reversed high bit must fold to bit zero", reversesTo(0x80000000u, 0x00000001u)); diff --git a/core/shader/recompiler/Optimization/include/Optimization/ConstantFolder.hpp b/core/shader/recompiler/Optimization/include/Optimization/ConstantFolder.hpp index d2df4ed3cc..6cbf62c460 100644 --- a/core/shader/recompiler/Optimization/include/Optimization/ConstantFolder.hpp +++ b/core/shader/recompiler/Optimization/include/Optimization/ConstantFolder.hpp @@ -14,6 +14,8 @@ public: void Fold(IrProgram& program, std::span blocks) const; + [[nodiscard]] bool FoldVertexLaneBounds(IrProgram& program, std::uint32_t hostSubgroupSize, std::uint32_t spirvVersion) const; + [[nodiscard]] bool ComposePackedAncillary(IrProgram& program) const; private: diff --git a/core/shader/recompiler/Optimization/src/ConstantFolder.cpp b/core/shader/recompiler/Optimization/src/ConstantFolder.cpp index 3505f640f4..bf9b33ac6b 100644 --- a/core/shader/recompiler/Optimization/src/ConstantFolder.cpp +++ b/core/shader/recompiler/Optimization/src/ConstantFolder.cpp @@ -846,6 +846,20 @@ void ConstantFolder::Fold(IrProgram& program) const { Fold(program, program.BlockOrder()); } +bool ConstantFolder::FoldVertexLaneBounds(IrProgram& program, std::uint32_t hostSubgroupSize, std::uint32_t spirvVersion) const { + if (program.Resources().stage != IrShaderStage::Vertex || hostSubgroupSize != 32u || spirvVersion < 0x00010000u || spirvVersion >= 0x00010600u) return false; + bool changed = false; + IrBuilder builder(program); + for (IrBlock* block : program.BlockOrder()) { + for (IrValue* value : block->Instructions()) { + if (value->Opcode() != IrOpcode::ULessThan32 || value->ArgumentCount() != 2u || !isLaneIndex(resolveArg(*value, 0)) || !isImmediateU32(resolveArg(*value, 1), 32u)) continue; + replaceWith(*value, builder.ConstantBool(true)); + changed = true; + } + } + return changed; +} + bool ConstantFolder::ComposePackedAncillary(IrProgram& program) const { std::vector words; for (IrBlock* block : program.BlockOrder()) { diff --git a/core/shader/recompiler/Recompiler.cpp b/core/shader/recompiler/Recompiler.cpp index 32dcb23ba2..ccd1b828ae 100644 --- a/core/shader/recompiler/Recompiler.cpp +++ b/core/shader/recompiler/Recompiler.cpp @@ -494,6 +494,12 @@ void prepareInterface(const RecompileRequest& request, IrProgram& program) { const bool nativeSampleOffsets = request.target.nonConstantImageOffsets && std::find(request.target.supportedCapabilities.begin(), request.target.supportedCapabilities.end(), spv::CapabilityImageGatherExtended) != request.target.supportedCapabilities.end(); resourceMaterializer.ApplyStaticInterface(program, nativeSampleOffsets); + constexpr ConstantFolder constantFolder; + if (request.shader.stage == ShaderStage::Vertex && constantFolder.FoldVertexLaneBounds(program, HostSubgroupSize(request), request.target.spirvVersion)) { + constantFolder.Fold(program); + ResolveControlFlowIdentities(program); + } + deadCodeEliminator.RemoveIdentities(program); constexpr RedundantValueEliminator redundantValueEliminator; static_cast(redundantValueEliminator.Eliminate(program)); From d353c6bccf2e95346a18ed892dd31d9acdedf8bf Mon Sep 17 00:00:00 2001 From: fdfawef324 <287205748+fdfawef324@users.noreply.github.com> Date: Wed, 14 Oct 2026 17:05:33 -0400 Subject: [PATCH 03/27] fix(shader): fold only constant lower-half lane words of vertex programs FoldVertexLaneBounds replaced every LaneId < 32 compare of a vertex program on a fixed 32-lane host subgroup with true. The half-wave scan matcher of the SPIR-V backend recognises a lane word by the select of its low and high words on that compare. A scan keyed by the entry EXEC ballot lost that shape, so its v_readlane_b32 of lane 63 was no longer read as a half-wave reduction and the recompile failed (agc_shader_memory). Fold the lane word select itself instead, and only when both words are constants. Lane words built from ballots or other values keep their lower-half test. Co-Authored-By: Claude Opus 5.5 --- .../libSceAgcDriver/tests/ConstantBitTest.cpp | 18 +++++++++++++----- .../Optimization/src/ConstantFolder.cpp | 8 +++++--- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/tests/ConstantBitTest.cpp b/core/libs/prx/libSceAgcDriver/tests/ConstantBitTest.cpp index 0d425edf5d..d660dc354d 100644 --- a/core/libs/prx/libSceAgcDriver/tests/ConstantBitTest.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/ConstantBitTest.cpp @@ -45,7 +45,7 @@ bool reversesTo(std::uint32_t word, std::uint32_t expected) { return result->HasImmediate() && result->ImmediateBool(); } -bool vertexProjection(std::uint32_t low, std::uint32_t hostSubgroup, std::uint32_t version, bool expected, bool directLane = false, IrShaderStage stage = IrShaderStage::Vertex) { +bool vertexProjection(std::uint32_t low, std::uint32_t hostSubgroup, std::uint32_t version, bool expected, bool directLane = false, IrShaderStage stage = IrShaderStage::Vertex, bool ballotWords = false) { IrProgram program; program.Resources().stage = stage; program.SetWaveSize(64u); @@ -56,7 +56,14 @@ bool vertexProjection(std::uint32_t low, std::uint32_t hostSubgroup, std::uint32 builder.SetInsertionPoint(block); auto& lane = builder.Emit(IrOpcode::LaneId, IrType::U32, {}); auto& half = builder.Emit(IrOpcode::ULessThan32, IrType::Bool, {&lane, &builder.Constant(32u)}); - auto& word = builder.Emit(IrOpcode::SelectU32, IrType::U32, {&half, &builder.Constant(low), &builder.Constant(0u)}); + IrValue* lowWord = &builder.Constant(low); + IrValue* highWord = &builder.Constant(0u); + if (ballotWords) { + auto& entry = builder.Emit(IrOpcode::Ballot, IrType::U32x4, {&builder.ConstantBool(true)}); + lowWord = &builder.Emit(IrOpcode::CompositeExtractU32x4, IrType::U32, {&entry, &builder.Constant(0u)}); + highWord = &builder.Emit(IrOpcode::CompositeExtractU32x4, IrType::U32, {&entry, &builder.Constant(1u)}); + } + auto& word = builder.Emit(IrOpcode::SelectU32, IrType::U32, {&half, lowWord, highWord}); auto& amount = builder.Emit(IrOpcode::BitwiseAnd32, IrType::U32, {&lane, &builder.Constant(31u)}); auto& shifted = builder.Emit(IrOpcode::ShiftRightLogical32, IrType::U32, {&word, &amount}); auto& bit = builder.Emit(IrOpcode::BitwiseAnd32, IrType::U32, {&shifted, &builder.Constant(1u)}); @@ -66,10 +73,10 @@ bool vertexProjection(std::uint32_t low, std::uint32_t hostSubgroup, std::uint32 static_cast(builder.Emit(IrOpcode::Return, IrType::Void, {})); ConstantFolder folder; const bool folded = folder.FoldVertexLaneBounds(program, hostSubgroup, version); - const bool shouldFold = stage == IrShaderStage::Vertex && hostSubgroup == 32u && version >= 0x00010000u && version < 0x00010600u; + const bool shouldFold = !ballotWords && stage == IrShaderStage::Vertex && hostSubgroup == 32u && version >= 0x00010000u && version < 0x00010600u; if (folded != shouldFold) return false; - if (!shouldFold && half.Opcode() != IrOpcode::ULessThan32) return false; - if (shouldFold && (!half.Resolve()->HasImmediate() || !half.Resolve()->ImmediateBool())) return false; + if (!shouldFold && (word.Opcode() != IrOpcode::SelectU32 || word.Argument(0)->Resolve() != &half || half.Opcode() != IrOpcode::ULessThan32)) return false; + if (shouldFold && (!word.Resolve()->HasImmediate() || word.Resolve()->ImmediateU32() != low)) return false; folder.Fold(program); DeadCodeEliminator{}.RemoveIdentities(program); DeadCodeEliminator{}.Eliminate(program); @@ -98,6 +105,7 @@ int main() { passed &= run("a single set bit remains dependent on the vertex lane", vertexProjection(1u, 32u, 0x00010300u, false)); passed &= run("a partial mask remains dependent on the vertex lane", vertexProjection(0x0000ffffu, 32u, 0x00010300u, false)); passed &= run("a real vertex lane output stays observable after folding its guard", vertexProjection(0xffffffffu, 32u, 0x00010300u, false, true)); + passed &= run("a lane word selected from ballot words keeps its lower-half test", vertexProjection(0xffffffffu, 32u, 0x00010300u, false, false, IrShaderStage::Vertex, true)); passed &= run("a 64-lane subgroup keeps a lower-half export guard", vertexProjection(0xffffffffu, 64u, 0x00010300u, false)); passed &= run("an unknown subgroup keeps a lower-half export guard", vertexProjection(0xffffffffu, 0u, 0x00010300u, false)); passed &= run("a single-lane request does not borrow the physical 32-lane proof", vertexProjection(0xffffffffu, 1u, 0x00010300u, false)); diff --git a/core/shader/recompiler/Optimization/src/ConstantFolder.cpp b/core/shader/recompiler/Optimization/src/ConstantFolder.cpp index bf9b33ac6b..52643c8990 100644 --- a/core/shader/recompiler/Optimization/src/ConstantFolder.cpp +++ b/core/shader/recompiler/Optimization/src/ConstantFolder.cpp @@ -849,11 +849,13 @@ void ConstantFolder::Fold(IrProgram& program) const { bool ConstantFolder::FoldVertexLaneBounds(IrProgram& program, std::uint32_t hostSubgroupSize, std::uint32_t spirvVersion) const { if (program.Resources().stage != IrShaderStage::Vertex || hostSubgroupSize != 32u || spirvVersion < 0x00010000u || spirvVersion >= 0x00010600u) return false; bool changed = false; - IrBuilder builder(program); for (IrBlock* block : program.BlockOrder()) { for (IrValue* value : block->Instructions()) { - if (value->Opcode() != IrOpcode::ULessThan32 || value->ArgumentCount() != 2u || !isLaneIndex(resolveArg(*value, 0)) || !isImmediateU32(resolveArg(*value, 1), 32u)) continue; - replaceWith(*value, builder.ConstantBool(true)); + if (value->Opcode() != IrOpcode::SelectU32 || value->ArgumentCount() != 3u) continue; + const auto& half = resolveArg(*value, 0); + auto& low = resolveArg(*value, 1); + if (half.Opcode() != IrOpcode::ULessThan32 || half.ArgumentCount() != 2u || !isLaneIndex(resolveArg(half, 0)) || !isImmediateU32(resolveArg(half, 1), 32u) || !isImmediate(low, IrType::U32) || !isImmediate(resolveArg(*value, 2), IrType::U32)) continue; + replaceWith(*value, low); changed = true; } } From 96070188b9f0d8dba01777d185807a9a51f0e113 Mon Sep 17 00:00:00 2001 From: Sam Date: Wed, 14 Oct 2026 07:55:29 +0100 Subject: [PATCH 04/27] perf(agc): allow shader preparation on first use APS5_LAZY_SHADER_PREPARE=1 keeps registration validation and decoding but skips queueing the preparation. Draws and dispatches prepare a deferred snapshot on first use and cache it; explicit ABI resolution still prepares its stages. Titles that register thousands of shaders at boot no longer hold every prepared shader in memory before the first frame. Cover lazy dispatch, metadata and failure cases in both asynchronous and synchronous modes. --- core/libs/prx/libSceAgcDriver/CMakeLists.txt | 9 +- .../src/Driver/Shaders/ShaderRegistry.cpp | 6 +- .../tests/execution/PreparedShaders.cpp | 97 ++++++++++++++++++- docs/dev/ARCHITECTURE.md | 1 + 4 files changed, 109 insertions(+), 4 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/CMakeLists.txt b/core/libs/prx/libSceAgcDriver/CMakeLists.txt index 051af9c6e5..66d89a4130 100644 --- a/core/libs/prx/libSceAgcDriver/CMakeLists.txt +++ b/core/libs/prx/libSceAgcDriver/CMakeLists.txt @@ -1730,7 +1730,7 @@ if(BUILD_TESTING) endif() endforeach() endforeach() - foreach(preparedMode IN ITEMS indirect uncached debug debug_failure deferred deferred_undecodable registered_workgroup registered_workgroup_sync) + foreach(preparedMode IN ITEMS indirect uncached debug debug_failure deferred deferred_undecodable registered_workgroup registered_workgroup_sync lazy lazy_sync) set(preparedTest "agc_driver_prepared_shaders_${preparedMode}") if(preparedMode STREQUAL "indirect") add_test(NAME ${preparedTest} COMMAND agc_driver_prepared_shaders_tests --indirect) @@ -1738,6 +1738,8 @@ if(BUILD_TESTING) add_test(NAME ${preparedTest} COMMAND agc_driver_prepared_shaders_tests --deferred) elseif(preparedMode STREQUAL "deferred_undecodable") add_test(NAME ${preparedTest} COMMAND agc_driver_prepared_shaders_tests --deferred-undecodable) + elseif(preparedMode STREQUAL "lazy" OR preparedMode STREQUAL "lazy_sync") + add_test(NAME ${preparedTest} COMMAND agc_driver_prepared_shaders_tests --lazy) elseif(preparedMode STREQUAL "registered_workgroup" OR preparedMode STREQUAL "registered_workgroup_sync") add_test(NAME ${preparedTest} COMMAND agc_driver_prepared_shaders_tests --registered-workgroup) elseif(preparedMode STREQUAL "debug_failure") @@ -1751,6 +1753,11 @@ if(BUILD_TESTING) set_property(TEST ${preparedTest} APPEND PROPERTY ENVIRONMENT "APS5_NO_DISPATCH_CACHE=1") elseif(preparedMode STREQUAL "registered_workgroup_sync") set_property(TEST ${preparedTest} APPEND PROPERTY ENVIRONMENT "APS5_SYNC_SHADER_PREPARE=1") + elseif(preparedMode STREQUAL "lazy" OR preparedMode STREQUAL "lazy_sync") + set_property(TEST ${preparedTest} APPEND PROPERTY ENVIRONMENT "APS5_LAZY_SHADER_PREPARE=1" "APS5_SUSPEND_DRAIN=1") + if(preparedMode STREQUAL "lazy_sync") + set_property(TEST ${preparedTest} APPEND PROPERTY ENVIRONMENT "APS5_SYNC_SHADER_PREPARE=1") + endif() elseif(preparedMode STREQUAL "debug" OR preparedMode STREQUAL "debug_failure") set_property(TEST ${preparedTest} APPEND PROPERTY ENVIRONMENT "APS5_PROFILE_DRAW=1" "APS5_NO_UNLOCKED_DEVICE=1") endif() diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp index b9a7b3ed9c..9d64334596 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp @@ -1514,7 +1514,11 @@ void Driver::RegisterShader(const Shader* shader) { registered.context = snapshot.registeredState->context; registered.userConfig = snapshot.registeredState->userConfig; std::shared_ptr preparation; - if (AsyncRegistrationPrepare()) { + static const bool lazy = std::getenv("APS5_LAZY_SHADER_PREPARE") != nullptr; + if (lazy) { + static_cast(PlanRegistered(snapshot, *localDevice, registered, true, &snapshot.prepared->deferred)); + snapshot.prepared->deferred = true; + } else if (AsyncRegistrationPrepare()) { preparation = PlanRegistered(snapshot, *localDevice, registered, true, &snapshot.prepared->deferred); if (preparation != nullptr) preparation->prebuildDevice = localDevice; if (preparation != nullptr && (snapshot.type == 0 || snapshot.type == 1)) BuildRegisteredAbiKey(registered, *localDevice, preparation->abiKey); diff --git a/core/libs/prx/libSceAgcDriver/tests/execution/PreparedShaders.cpp b/core/libs/prx/libSceAgcDriver/tests/execution/PreparedShaders.cpp index 777dd733f4..5109c06012 100644 --- a/core/libs/prx/libSceAgcDriver/tests/execution/PreparedShaders.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/execution/PreparedShaders.cpp @@ -5,6 +5,8 @@ #include "prx/libSceAgcDriver/Graphics/include/ShaderInputState.hpp" #include "prx/libSceAgcDriver/Execution/include/Driver.hpp" #include "prx/libSceAgcDriver/Execution/include/ShaderPreparation.hpp" +#include "../NarrowConstantStoreFixture.hpp" +#include "prx/libc/include/GuestAllocations.hpp" #include "prx/libSceAgcDriver/Submit/include/Acb.hpp" #include "prx/libSceAgcDriver/Submit/include/Dcb.hpp" #include @@ -27,6 +29,12 @@ #include #include #include +#ifdef _WIN32 +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#endif namespace { @@ -600,7 +608,7 @@ void RegisteredVertexWorkgroup(const ShaderRecompiler::SpirvTarget& target, bool header.specials.vgt_shader_stages_en.value = 0; ExpectFailure([&] { AgcDriverRegisterShader_nid_postfix(&header.shader); }, "missing AGC user-data header"); header.shader.user_data = &header.userData; - if (target.subgroupSize < 64u && std::getenv("APS5_SYNC_SHADER_PREPARE") != nullptr) { + if (target.subgroupSize < 64u && std::getenv("APS5_SYNC_SHADER_PREPARE") != nullptr && std::getenv("APS5_LAZY_SHADER_PREPARE") == nullptr) { ExpectFailure([&] { AgcDriverRegisterShader_nid_postfix(&header.shader); }, "v_readlane_b32 of lane 63 is outside"); } std::cout << "registered NGG preparation deferral passed; subgroup=" << target.subgroupSize << '\n'; @@ -612,6 +620,85 @@ void RegisteredVertexWorkgroup(const ShaderRecompiler::SpirvTarget& target, bool std::cout << "registered NGG wave64 preparation passed; subgroup=" << target.subgroupSize << '\n'; } +void LazyRegistration() { + using namespace NarrowConstantStoreFixture; + Require(std::getenv("APS5_LAZY_SHADER_PREPARE") != nullptr, "lazy preparation is not enabled"); + struct Header { + Shader shader{}; + std::array registers{}; + ShaderSpecialRegs specials{}; + } header; + const auto address = reinterpret_cast(Code.data()); + header.shader.file_header = 0x34333231u; + header.shader.version = 0x18; + header.shader.header_size = sizeof(header); + header.shader.shader_size = sizeof(Code); + header.shader.code = Code.data(); + header.shader.sh_registers = header.registers.data(); + header.shader.num_sh_registers = header.registers.size(); + header.shader.specials = &header.specials; + header.specials.dispatch_modifier = 0x8000; + header.registers = {{{0x20c, static_cast(address >> 8u)}, {0x20d, static_cast(address >> 40u)}, {0x207, Threads}, {0x208, 1}, {0x209, 1}, {0x212, 0}, {0x213, 4}}}; + AgcDriverRegisterShader_nid_postfix(&header.shader); + ++header.registers[0].value; + ExpectFailure([&] { AgcDriverRegisterShader_nid_postfix(&header.shader); }, "entry point is outside shader code"); + --header.registers[0].value; + header.registers[2].value = 0; + ExpectFailure([&] { AgcDriverRegisterShader_nid_postfix(&header.shader); }, "must be nonzero"); + header.registers[2].value = Threads; + header.shader.num_sh_registers = 255; + ExpectFailure([&] { AgcDriverRegisterShader_nid_postfix(&header.shader); }, "truncated shader metadata"); + header.shader.num_sh_registers = header.registers.size(); + constexpr std::size_t outputBytes = 65536; +#ifdef _WIN32 + auto* allocation = static_cast(VirtualAlloc(nullptr, outputBytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE)); +#else + auto* allocation = static_cast(std::aligned_alloc(outputBytes, outputBytes)); +#endif + Require(allocation != nullptr, "cannot allocate lazy shader test output"); + const auto release = [](std::uint8_t* block) { + GuestAllocations::Mutation().Remove(block); +#ifdef _WIN32 + VirtualFree(block, 0, MEM_RELEASE); +#else + std::free(block); +#endif + }; + const std::unique_ptr ownedOutput(allocation, release); + const std::span output(allocation, outputBytes); + GuestAllocations::Mutation().Add(output.data(), output.size(), true, true, true); + const auto outputAddress = reinterpret_cast(output.data()); + std::vector commands; + for (const auto reg : header.registers) commands.insert(commands.end(), {0xc0017600u, reg.offset, reg.value}); + commands.insert(commands.end(), {0xc0027600u, 0x240, static_cast(outputAddress), static_cast(outputAddress >> 32u), 0xc0031500u, 1, 1, 1, 0x8041}); + Packet packet{commands.data(), static_cast(commands.size()), 0, {}}; + for (unsigned iteration = 0; iteration < 2; ++iteration) { + std::ranges::fill(output, Fill); + sceAgcDriverSubmitAcb(0x20, &packet); + AgcDriverWaitIdle_nid_postfix(); + AgcDriverSuspendPoint_nid_postfix(); + AgcDriverWaitIdle_nid_postfix(); + for (std::size_t offset = 0; offset < output.size(); ++offset) { + const auto expected = offset < Threads * LaneBytes ? ExpectedLane[offset % LaneBytes] : Fill; + if (output[offset] != expected) throw std::runtime_error("lazy dispatch " + std::to_string(iteration) + " byte " + std::to_string(offset) + " is " + std::to_string(output[offset]) + ", expected " + std::to_string(expected)); + } + } + AgcDriverResolveShaderAbi_nid_postfix(&header.shader, {}, {}); + alignas(256) static const std::array invalid{0xbf9f0000u}; + const auto invalidAddress = reinterpret_cast(invalid.data()); + header.shader.code = invalid.data(); + header.shader.shader_size = sizeof(invalid); + header.registers[0].value = static_cast(invalidAddress >> 8u); + header.registers[1].value = static_cast(invalidAddress >> 40u); + AgcDriverRegisterShader_nid_postfix(&header.shader); + ExpectFailure([&] { AgcDriverResolveShaderAbi_nid_postfix(&header.shader, {}, {}); }, "s_code_end"); + commands[2] = header.registers[0].value; + commands[5] = header.registers[1].value; + sceAgcDriverSubmitAcb(0x20, &packet); + ExpectFailure([] { AgcDriverWaitIdle_nid_postfix(); }, "s_code_end"); + ExpectFailure([] { AgcDriverShutdown_nid_postfix(); }, "s_code_end"); +} + void DeferredRegistration() { alignas(256) std::array code{0xbe842104u, 0xbf810000u}; struct Header { @@ -672,7 +759,13 @@ void DeferredUndecodableRegistration() { int main(int argc, char** argv) { try { - Require(argc == 1 || (argc == 2 && (std::string_view(argv[1]) == "--indirect" || std::string_view(argv[1]) == "--fail-before-registration" || std::string_view(argv[1]) == "--deferred" || std::string_view(argv[1]) == "--deferred-undecodable" || std::string_view(argv[1]) == "--registered-workgroup")), "invalid test arguments"); + Require(argc == 1 || (argc == 2 && (std::string_view(argv[1]) == "--indirect" || std::string_view(argv[1]) == "--fail-before-registration" || std::string_view(argv[1]) == "--deferred" || std::string_view(argv[1]) == "--deferred-undecodable" || std::string_view(argv[1]) == "--registered-workgroup" || std::string_view(argv[1]) == "--lazy")), "invalid test arguments"); + if (argc == 2 && std::string_view(argv[1]) == "--lazy") { + if (!OpenVulkanTestDevice()) return VulkanTestSkipped; + LazyRegistration(); + std::cout << "lazy shader preparation tests passed\n"; + return 0; + } if (argc == 2 && std::string_view(argv[1]) == "--registered-workgroup") { const auto device = OpenVulkanTestDevice(); if (!device) return VulkanTestSkipped; diff --git a/docs/dev/ARCHITECTURE.md b/docs/dev/ARCHITECTURE.md index 54b4476de1..9370533583 100644 --- a/docs/dev/ARCHITECTURE.md +++ b/docs/dev/ARCHITECTURE.md @@ -100,6 +100,7 @@ flowchart LR - Queue 0's packet worker decodes, captures and prepares a draw and hands its recording (`Graphics::Draw` under the GPU lock) to a committer thread in [`DrawPipeline.cpp`](../../core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawPipeline.cpp), so the next packet is decoded while the previous draw is recorded. Each queued item carries the guest ranges it will write (`DrawWriteRanges`: written buffers and storage images, color targets with their DCC and CMASK, depth and stencil). A later packet that reads or writes one of those ranges, or that the pipeline can't order, drains it first; labels, submission ends, buffer fills and (with `APS5_PIPELINE_DMA=1`) DMA packets are queued as items instead. An exception thrown by a commit is rethrown on the worker at its next enqueue or drain, as an inline draw error would be. On by default; `APS5_PIPELINED_DRAWS=0` records every draw inline, and `APS5_EXACT_DRAW_WRITES=0` widens the color and depth ranges to estimates. - The shader registry maps a code address to every header registered for that code, in registration order. Static, graphics and rectangle ABI resolution pick the header at the bound address (or one equal to it, `SameHeader`, which also takes a hull header fused by `sceAgcFuseShaderHalves` (type 3) as the registered back half (type 7) it copies, the fused registers aside; graphics ABI resolution then uses the fused header's shader registers); draws and dispatches pick the program by the stage's binary type (`RegisteredProgram`), and a front entry matching both type 2 and type 4 throws. Within a type the last registration wins; re-registering an identical earlier header makes its snapshot current again without preparing it again, and changed code drops the older headers. - The snapshot is published as pending. `SourceHandleFor`, `InvocationFor` and `ShaderPreparationTransaction::Edit` / `Read` wait for the pending job before reading `PreparedShaders`, and rethrow its failure. `APS5_SYNC_SHADER_PREPARE=1` prepares on the registering thread instead. +- `APS5_LAZY_SHADER_PREPARE=1` retains registration validation and decoding but discards the temporary plan without queueing preparation. The snapshot is deferred, not pending; draws and dispatches prepare and cache it on first use. Explicit ABI resolution still prepares its requested stages. This opt-in mode takes precedence over synchronous registration preparation and trades background CPU/memory use for possible first-use stalls; disk caches and shader output are unchanged. - On devices with graphics pipeline libraries (`SpirvTarget::registrationModules`), a vertex or pixel artifact is emitted in registration form: the buffer V# words, image modes, swizzles and mip ranges, depth-compare state, export mappings and vertex component selectors it used to specialize on per draw are read at run time (image `ResourceMetadata` and a few state dwords in the shader data), descriptor heaps keep every element in place (an unused one gets a null or placeholder descriptor), and the pixel stage's bindings live in descriptor set 1 so each stage's library depends only on its own set layout (`INDEPENDENT_SETS`). An artifact above `registrationWordLimit` SPIR-V words is re-emitted specialized, and `CompiledShaderArtifact::registrationModule` records which form it has. For artifacts in registration form, `RegistrationResult` materializes the module once, and `PrebuildStageLibrary` builds its pre-rasterization or fragment-shader library on a separate one-thread pool (`APS5_PREBUILD_THREADS`) with a fixed default state, pairing it with a stub stage. A later draw with that state finds both libraries in the store and only links. - A draw or dispatch whose state matches no artifact prepared at registration (a title that builds its static ABI state itself, without `sceAgcDriverResolve*Abi`) gets one prepared at use by `SourceHandleFor` or `InvocationFor`, logged once per artifact and kept in the snapshot for the next use. - A compute shader whose `s_swappc_b64` calls a user-data pointer cannot be linked at registration, because the callee's address is only known at dispatch. `InvocationFor` in [`ShaderRegistry.cpp`](../../core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp) reads each callee from guest memory, appends one copy per call after the caller (`LinkUserDataCalls`) and prepares that linked code, so a different callee gets its own prepared entry. From c8bedff055a1643cfb3f03fa19f0f7491961d3aa Mon Sep 17 00:00:00 2001 From: Sam <85789460+Sam-mon7@users.noreply.github.com> Date: Thu, 15 Oct 2026 17:55:30 +0100 Subject: [PATCH 05/27] test(agc): check that lazy registration defers preparation The lazy mode of agc_driver_prepared_shaders only checked dispatch output and failure reporting, which eager preparation also produces. Read the registered snapshot through Driver::RegisteredShaders: registration leaves it deferred with no prepared artifact, and the first dispatch prepares it. --- .../Execution/include/Driver/Driver.hpp | 1 + .../src/Driver/Shaders/ShaderRegistry.cpp | 5 +++++ .../tests/execution/PreparedShaders.cpp | 18 ++++++++++++++++++ 3 files changed, 24 insertions(+) diff --git a/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Driver.hpp b/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Driver.hpp index 434f0d992b..6d440f6e31 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Driver.hpp +++ b/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Driver.hpp @@ -75,6 +75,7 @@ public: }; DispatchCacheFootprint DispatchCacheSize(); static std::uint64_t DispatchCacheBytes(); + std::shared_ptr RegisteredShaders(); private: friend class SampledReadScope; diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp index 9d64334596..ab8df700c0 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp @@ -1462,6 +1462,11 @@ ShaderSnapshot PrepareNullPixelProgram(const VulkanDevice& device) { return null; } +std::shared_ptr Driver::RegisteredShaders() { + std::lock_guard lock(mutex); + return shaders; +} + void Driver::RegisterShader(const Shader* shader) { PerformanceContext timingContext(FrameTiming::Preparation()); PerformanceTimer timing("Shader.Register"); diff --git a/core/libs/prx/libSceAgcDriver/tests/execution/PreparedShaders.cpp b/core/libs/prx/libSceAgcDriver/tests/execution/PreparedShaders.cpp index 5109c06012..e242f0a0b9 100644 --- a/core/libs/prx/libSceAgcDriver/tests/execution/PreparedShaders.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/execution/PreparedShaders.cpp @@ -4,6 +4,7 @@ #include "prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp" #include "prx/libSceAgcDriver/Graphics/include/ShaderInputState.hpp" #include "prx/libSceAgcDriver/Execution/include/Driver.hpp" +#include "prx/libSceAgcDriver/Execution/include/Driver/Driver.hpp" #include "prx/libSceAgcDriver/Execution/include/ShaderPreparation.hpp" #include "../NarrowConstantStoreFixture.hpp" #include "prx/libc/include/GuestAllocations.hpp" @@ -649,6 +650,19 @@ void LazyRegistration() { header.shader.num_sh_registers = 255; ExpectFailure([&] { AgcDriverRegisterShader_nid_postfix(&header.shader); }, "truncated shader metadata"); header.shader.num_sh_registers = header.registers.size(); + const auto registered = [&] { + const auto registry = AgcDriver::DriverDetail::Driver::Get().RegisteredShaders(); + Require(registry != nullptr, "lazy registration published no shader registry"); + auto snapshot = AgcDriver::DriverDetail::RegisteredHeader(*registry, &header.shader); + Require(snapshot != nullptr, "lazy registration published no snapshot"); + return snapshot; + }; + const auto lazySnapshot = registered(); + { + AgcDriver::DriverDetail::ShaderPreparationTransaction transaction; + const auto& state = transaction.Read(*lazySnapshot); + Require(state.deferred && state.entries.empty(), "lazy registration prepared the shader before its first dispatch"); + } constexpr std::size_t outputBytes = 65536; #ifdef _WIN32 auto* allocation = static_cast(VirtualAlloc(nullptr, outputBytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE)); @@ -683,6 +697,10 @@ void LazyRegistration() { if (output[offset] != expected) throw std::runtime_error("lazy dispatch " + std::to_string(iteration) + " byte " + std::to_string(offset) + " is " + std::to_string(output[offset]) + ", expected " + std::to_string(expected)); } } + { + AgcDriver::DriverDetail::ShaderPreparationTransaction transaction; + Require(!transaction.Read(*lazySnapshot).entries.empty(), "the first dispatch did not prepare the lazily registered shader"); + } AgcDriverResolveShaderAbi_nid_postfix(&header.shader, {}, {}); alignas(256) static const std::array invalid{0xbf9f0000u}; const auto invalidAddress = reinterpret_cast(invalid.data()); From a0347cfed51bc2807d6e771072e7013f1019340a Mon Sep 17 00:00:00 2001 From: billybobpicklio Date: Wed, 14 Oct 2026 09:22:03 -0400 Subject: [PATCH 06/27] feat(agc): draw line-strip output of NGG geometry programs as mesh lines A geometry program with VGT_GS_OUT_PRIM_TYPE 1 was refused as an unsupported output assembly. Each primitive export of an NGG line-strip program is one line whose indices are packed like a triangle's, so the mesh module now declares OutputLinesEXT with PrimitiveLineIndicesEXT for it. A subgroup gets VGT_GS_MAX_VERT_OUT - 1 lines per input primitive and at least GE_MAX_OUTPUT_PER_SUBGROUP line slots; fast launch keeps using PRIM_AMP_FACTOR. The output size is part of the cache, draw, validation and pipeline keys, and the request serialization version becomes 31. Triangle-strip geometry is unchanged. Co-Authored-By: Claude Opus 5.5 --- .../Execution/src/Driver/Draw/DrawLookup.cpp | 2 +- .../src/Driver/Draw/DrawRegisterKey.cpp | 2 +- .../prx/libSceAgcDriver/Graphics/src/Draw.cpp | 1 + .../libSceAgcDriver/Graphics/src/Pipeline.cpp | 1 + .../Graphics/src/ShaderValidation.cpp | 13 +++--- .../libSceAgcDriver/Graphics/src/State.cpp | 14 +++--- .../prx/libSceAgcDriver/tests/Graphics.cpp | 41 +++++++++++++++++ core/libs/prx/libSceAgcDriver/tests/Mesh.cpp | 44 +++++++++++++++++++ .../libSceAgcDriver/tests/ShaderMemory.cpp | 12 ++++- core/shader/recompiler/CacheKey.hpp | 2 +- .../ControlFlow/src/RequestSerializer.cpp | 6 ++- .../Optimization/ShaderStageInputInfo.hpp | 1 + core/shader/recompiler/Recompiler.hpp | 1 + .../src/SpirvMemory/SpirvModuleSetup.cpp | 2 +- .../SpirvBackend/src/SpirvModuleEmitter.cpp | 14 +++--- .../src/ShaderInputInfoBuilder.cpp | 3 +- .../tests/RequestSerializerTests.cpp | 14 +++--- docs/dev/debt/libSceAgcDriver.md | 1 + 18 files changed, 140 insertions(+), 34 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawLookup.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawLookup.cpp index 366793d496..c64a860a7a 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawLookup.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawLookup.cpp @@ -22,7 +22,7 @@ void Driver::lookupDraw(const Submission& submission, const std::shared_ptr& x, const std::optional& y) { if (x.has_value() != y.has_value()) return false; if (!x) return true; - return x->inputPrimitive == y->inputPrimitive && x->primitivesPerGroup == y->primitivesPerGroup && x->verticesPerGroup == y->verticesPerGroup && x->maxVertices == y->maxVertices && x->maxPrimitives == y->maxPrimitives && x->threadsPerGroup == y->threadsPerGroup && x->ldsSizeDwords == y->ldsSizeDwords && x->provokingVertex == y->provokingVertex && x->esgsItemSize == y->esgsItemSize; + return x->inputPrimitive == y->inputPrimitive && x->primitivesPerGroup == y->primitivesPerGroup && x->verticesPerGroup == y->verticesPerGroup && x->maxVertices == y->maxVertices && x->maxPrimitives == y->maxPrimitives && x->threadsPerGroup == y->threadsPerGroup && x->ldsSizeDwords == y->ldsSizeDwords && x->provokingVertex == y->provokingVertex && x->esgsItemSize == y->esgsItemSize && x->outputPrimitiveSize == y->outputPrimitiveSize; }; const auto sameTess = [](const std::optional& x, const std::optional& y) { if (x.has_value() != y.has_value()) return false; diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp index 7fb4593c6e..6e567b224f 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp @@ -658,6 +658,7 @@ bool ValidationKey(const Context& context, std::span shade add(mesh.threadsPerGroup); add(mesh.ldsSizeDwords); add(mesh.provokingVertex); + add(mesh.outputPrimitiveSize); } add(state.stages.tessellation.has_value()); if (state.stages.tessellation) { diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp index 9fd5790a24..0a10e1c8d6 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp @@ -872,6 +872,7 @@ void pipelineKey(std::vector& key, const Context& context, const Stat append(key, mesh.threadsPerGroup); append(key, mesh.ldsSizeDwords); append(key, mesh.provokingVertex); + append(key, mesh.outputPrimitiveSize); } append(key, state.stages.tessellation.has_value()); if (state.stages.tessellation) { diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp index 9c7dcb033e..a0c0001a35 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp @@ -47,7 +47,7 @@ struct Module { bool position = false; std::uint32_t clipDistances = 0; std::uint32_t cullDistances = 0; - bool primitiveIndices = false; + std::uint32_t primitiveIndices = 0; bool fragmentBarycentric = false; bool layerOutput = false; std::map> modes; @@ -134,8 +134,9 @@ struct Module { Require(input && ((value == spv::BuiltInPosition && signature == "f32x4") || (value == spv::BuiltInTessCoord && stage == Stage::TessellationEvaluation && signature == "f32x3") || ((value == spv::BuiltInInvocationId || value == spv::BuiltInPrimitiveId || value == spv::BuiltInPatchVertices) && (signature == "i32" || signature == "u32"))), "unsupported tessellation built-in"); return; } - if (!input && value == spv::BuiltInPrimitiveTriangleIndicesEXT) primitiveIndices = true; - Require((input && ((value == spv::BuiltInWorkgroupId || value == spv::BuiltInLocalInvocationId || value == spv::BuiltInGlobalInvocationId || value == spv::BuiltInNumWorkgroups) && signature == "u32x3")) || (input && value == spv::BuiltInLocalInvocationIndex && signature == "u32") || (!input && value == spv::BuiltInPrimitiveTriangleIndicesEXT && signature == "u32x3") || (!input && value == spv::BuiltInCullPrimitiveEXT && signature == "bool") || (!input && value == spv::BuiltInLayer && signature == "u32"), "unsupported mesh built-in"); + if (!input && value == spv::BuiltInPrimitiveLineIndicesEXT) primitiveIndices = 2u; + if (!input && value == spv::BuiltInPrimitiveTriangleIndicesEXT) primitiveIndices = 3u; + Require((input && ((value == spv::BuiltInWorkgroupId || value == spv::BuiltInLocalInvocationId || value == spv::BuiltInGlobalInvocationId || value == spv::BuiltInNumWorkgroups) && signature == "u32x3")) || (input && value == spv::BuiltInLocalInvocationIndex && signature == "u32") || (!input && value == spv::BuiltInPrimitiveLineIndicesEXT && signature == "u32x2") || (!input && value == spv::BuiltInPrimitiveTriangleIndicesEXT && signature == "u32x3") || (!input && value == spv::BuiltInCullPrimitiveEXT && signature == "bool") || (!input && value == spv::BuiltInLayer && signature == "u32"), "unsupported mesh built-in"); return; } const auto signature = Signature(type); @@ -437,7 +438,7 @@ Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysi mode(spv::ExecutionModeLocalSize, {paired ? config.threadsPerGroup / 2u : config.threadsPerGroup, 1, 1}); mode(spv::ExecutionModeOutputVertices, {config.maxVertices}); mode(spv::ExecutionModeOutputPrimitivesEXT, {config.maxPrimitives}); - mode(spv::ExecutionModeOutputTrianglesEXT, {}); + mode(config.outputPrimitiveSize == 2u ? spv::ExecutionModeOutputLinesEXT : spv::ExecutionModeOutputTrianglesEXT, {}); Require(module.modes.size() == 4, "unsupported mesh execution mode"); } else if (control) { Require(state.rectList || state.stages.tessellation.has_value(), "tessellation configuration is missing"); @@ -486,7 +487,7 @@ Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysi const auto length = module.constants.find(outer[3]); Require(length != module.constants.end() && length->second != 0, "invalid per-vertex interface array length"); if (mesh) { - const bool primitive = decoration.perPrimitive || (decoration.builtin && (*decoration.builtin == spv::BuiltInPrimitiveTriangleIndicesEXT || *decoration.builtin == spv::BuiltInCullPrimitiveEXT)); + const bool primitive = decoration.perPrimitive || (decoration.builtin && (*decoration.builtin == spv::BuiltInPrimitiveLineIndicesEXT || *decoration.builtin == spv::BuiltInPrimitiveTriangleIndicesEXT || *decoration.builtin == spv::BuiltInCullPrimitiveEXT)); Require(length->second == (primitive ? state.stages.mesh->maxPrimitives : state.stages.mesh->maxVertices), "mesh interface array disagrees with output limits"); } else if (geometry) { Require(length->second == 3u, "geometry interface array must hold the three vertices of a triangle"); @@ -594,7 +595,7 @@ Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysi Require(descriptors.size() == shader.bindings.size(), "recompiler binding metadata contains undeclared resources"); Require(push == (!shader.pushConstants.empty() || mesh || (control && !rectListControl)), "recompiler push constant metadata disagrees with SPIR-V"); for (const auto id : module.interface) Require(module.variables.contains(id), "entry point interface contains an unknown variable"); - if (mesh) Require(module.primitiveIndices, "mesh shader does not export primitive indices"); + if (mesh) Require(module.primitiveIndices == state.stages.mesh->outputPrimitiveSize, "mesh shader does not export primitive indices"); if (stage == Stage::Vertex || mesh || evaluation || geometry) Require(module.position, "vertex shader does not export position"); return module; } diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp index c16eb7847a..78fb6c50ea 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp @@ -876,7 +876,10 @@ ShaderStages DecodeShaderStages(const QueueState& queue, bool vertexWorkgroup) { validate(!fastLaunch || (!vertexWorkgroup && primitive == 1 && vertices == 1 && primitives == 1), "fast launch other than one point per subgroup is unsupported"); const auto maxVertices = read(queue.context, 0x1ff); const auto verticesPerPrimitive = vertexWorkgroup ? 3u : read(queue.context, 0x2ce); - validate((primitive == 1 || primitive == 2 || primitive == 4 || primitive == 5 || primitive == 6) && read(queue.context, 0x29b) == 2 && verticesPerPrimitive >= 3, "unsupported geometry input or output assembly"); + const auto outputType = read(queue.context, 0x29b); + validate(outputType == 2 || (outputType == 1 && !vertexWorkgroup), "VGT_GS_OUT_PRIM_TYPE other than a triangle strip, or a line strip from a geometry program, is unsupported"); + const auto outputSize = outputType == 1 ? 2u : 3u; + validate((primitive == 1 || primitive == 2 || primitive == 4 || primitive == 5 || primitive == 6) && verticesPerPrimitive >= outputSize, "unsupported geometry input or output assembly"); const auto inputSize = primitive == 1 ? 1u : primitive == 2 ? 2u : 3u; validate(vertices >= inputSize && maxVertices != 0 && (!vertexWorkgroup || maxVertices >= inputSize) && maxVertices <= 256 && verticesPerPrimitive <= 256, "invalid geometry subgroup output"); const auto inputStep = primitive == 5 || primitive == 6 ? 1u : inputSize; @@ -886,11 +889,11 @@ ShaderStages DecodeShaderStages(const QueueState& queue, bool vertexWorkgroup) { const auto resources = read(queue.shader, 0x8b, RegisterBank::Shader); const auto esgsItemSize = read(queue.context, 0x2ab); validate(esgsItemSize != 0 && esgsItemSize * vertices <= 0xffffu, "invalid VGT_ESGS_RING_ITEMSIZE"); - const auto amplification = fastLaunch ? read(queue.context, 0x2d3) & 0x1ffu : verticesPerPrimitive - 2u; + const auto amplification = fastLaunch ? read(queue.context, 0x2d3) & 0x1ffu : verticesPerPrimitive - outputSize + 1u; validate(amplification != 0, "fast launch subgroup amplifies to no primitives"); - const auto maxPrimitives = primitives * amplification; + const auto maxPrimitives = outputSize == 2u && !fastLaunch ? std::max(primitives * amplification, maxVertices) : primitives * amplification; const auto threads = std::max({(groupPrimitives - 1u) * inputStep + inputSize, primitives, maxVertices, maxPrimitives}); - result.mesh = ShaderRecompiler::MeshConfiguration{primitive, groupPrimitives, (groupPrimitives - 1u) * inputStep + inputSize, maxVertices, maxPrimitives, ((threads + result.vertexWaveSize - 1u) / result.vertexWaveSize) * result.vertexWaveSize, ((resources >> 19u) & 0xffu) * 128u, 0, esgsItemSize}; + result.mesh = ShaderRecompiler::MeshConfiguration{primitive, groupPrimitives, (groupPrimitives - 1u) * inputStep + inputSize, maxVertices, maxPrimitives, ((threads + result.vertexWaveSize - 1u) / result.vertexWaveSize) * result.vertexWaveSize, ((resources >> 19u) & 0xffu) * 128u, 0, esgsItemSize, outputSize}; } return result; } @@ -1054,7 +1057,7 @@ State DecodeState(const QueueState& queue, bool vertexWorkgroup) { const auto raster = read(cx, 0x205); // Bits 5-10 give the front/back polygon type (2 = filled triangles), which POLY_MODE (bit 3) turns on // explicitly; KEEP_TOGETHER_ENABLE (bit 24) only affects primitive distribution across the chip. - const bool pointsOrLines = result.topology == VK_PRIMITIVE_TOPOLOGY_POINT_LIST || result.topology == VK_PRIMITIVE_TOPOLOGY_LINE_LIST || result.topology == VK_PRIMITIVE_TOPOLOGY_LINE_STRIP; + const bool pointsOrLines = result.topology == VK_PRIMITIVE_TOPOLOGY_POINT_LIST || result.topology == VK_PRIMITIVE_TOPOLOGY_LINE_LIST || result.topology == VK_PRIMITIVE_TOPOLOGY_LINE_STRIP || (result.stages.mesh.has_value() && result.stages.mesh->outputPrimitiveSize == 2u); const auto lineOffset = (raster & 0x1800u) != 0 && !pointsOrLines ? 0x2000u : 0u; const auto rasterMode = raster & ~0x7u & ~(1u << 24u) & ~0x1800u & ~lineOffset & ~(1u << 19u) & ~(1u << 16u); if (rasterMode != 0 && rasterMode != 0x240u && rasterMode != 0x248u) throw std::runtime_error("AGC graphics: " + zeroMessage(0x205, raster, "polygon mode, depth bias or nonstandard rasterization")); @@ -1320,6 +1323,7 @@ std::array QuadSampleMasks(const State& state) { ShaderRecompiler::RasterPrimitive RasterPrimitiveOf(const State& state) { if (state.rectList) return ShaderRecompiler::RasterPrimitive::Rectangle; + if (state.stages.mesh.has_value() && state.stages.mesh->outputPrimitiveSize == 2u) return ShaderRecompiler::RasterPrimitive::Line; if (state.stages.mesh.has_value() || state.stages.tessellation.has_value()) return ShaderRecompiler::RasterPrimitive::Triangle; switch (state.topology) { case VK_PRIMITIVE_TOPOLOGY_POINT_LIST: return ShaderRecompiler::RasterPrimitive::Point; diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index c7d2efd679..bf2df04ece 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -816,6 +816,10 @@ void ShaderStageTests() { Require(state.stages.path == AgcDriver::Graphics::ShaderPath::Geometry && state.stages.vertexWaveSize == 32 && state.stages.mesh, "a fast-launch draw did not decode as wave32 geometry"); const auto& mesh = *state.stages.mesh; Require(mesh.inputPrimitive == 1 && mesh.primitivesPerGroup == 1 && mesh.verticesPerGroup == 1 && mesh.maxVertices == 32 && mesh.maxPrimitives == 64 && mesh.threadsPerGroup == 64 && mesh.ldsSizeDwords == 128, "a fast-launch subgroup did not decode as one point amplified by PRIM_AMP_FACTOR"); + fast.context[0x29b] = 1; + const auto fastLines = AgcDriver::Graphics::DecodeState(fast); + Require(fastLines.stages.mesh->outputPrimitiveSize == 2 && fastLines.stages.mesh->maxPrimitives == 64 && AgcDriver::Graphics::RasterPrimitiveOf(fastLines) == ShaderRecompiler::RasterPrimitive::Line, "a fast-launch line strip did not take its line count from PRIM_AMP_FACTOR"); + fast.context[0x29b] = 2; fast.context[0x2d5] = 0x502030; expectFailure([&] { AgcDriver::Graphics::DecodeState(fast); }, "unsupported geometry routing, fast launch or wave-ID state"); fast.context[0x2d5] = 0x482030; @@ -828,6 +832,40 @@ void ShaderStageTests() { fast.context[0x2d3] = 0; expectFailure([&] { AgcDriver::Graphics::DecodeState(fast); }, "amplifies to no primitives"); } + { + auto lines = makeState(); + lines.userConfig[0x242] = 1; + lines.context[0x2d5] = 0x2030; + lines.userConfig[0x25b] = 0x6633; + lines.context[0x1ff] = 255; + lines.context[0x2ce] = 5; + lines.context[0x29b] = 1; + lines.context[0x2ab] = 4; + lines.shader[0x8a] = 3u << 29u; + lines.shader[0x8b] = 3u << 16u; + auto state = AgcDriver::Graphics::DecodeState(lines); + Require(state.stages.path == AgcDriver::Graphics::ShaderPath::Geometry && state.stages.mesh && AgcDriver::Graphics::RasterPrimitiveOf(state) == ShaderRecompiler::RasterPrimitive::Line, "a geometry program that outputs line strips did not decode as geometry rasterized as lines"); + auto mesh = *state.stages.mesh; + Require(mesh.inputPrimitive == 1 && mesh.primitivesPerGroup == 51 && mesh.verticesPerGroup == 51 && mesh.maxVertices == 255 && mesh.maxPrimitives == 255 && mesh.threadsPerGroup == 256 && mesh.outputPrimitiveSize == 2, "a line-strip subgroup did not decode as 51 points with a line slot per output vertex"); + lines.context[0x29b] = 2; + state = AgcDriver::Graphics::DecodeState(lines); + mesh = *state.stages.mesh; + Require(mesh.maxPrimitives == 153 && mesh.threadsPerGroup == 256 && mesh.outputPrimitiveSize == 3 && AgcDriver::Graphics::RasterPrimitiveOf(state) == ShaderRecompiler::RasterPrimitive::Triangle, "the triangle-strip subgroup of the same draw changed"); + lines.context[0x29b] = 1; + lines.context[0x2ce] = 1; + expectFailure([&] { AgcDriver::Graphics::DecodeState(lines); }, "unsupported geometry input or output assembly"); + lines.context[0x2ce] = 5; + for (const auto output : {0u, 3u, 0x80000001u, 0x101u}) { + lines.context[0x29b] = output; + expectFailure([&] { AgcDriver::Graphics::DecodeState(lines); }, "VGT_GS_OUT_PRIM_TYPE other than a triangle strip, or a line strip from a geometry program, is unsupported"); + } + lines.userConfig[0x242] = 4; + lines.context[0x205] = 0x2a40; + lines.context[0x29b] = 2; + Require(AgcDriver::Graphics::DecodeState(lines).stages.mesh->primitivesPerGroup == 17, "triangle input with a triangle-strip output and a triangle polygon offset did not decode"); + lines.context[0x29b] = 1; + expectFailure([&] { AgcDriver::Graphics::DecodeState(lines); }, "polygon mode, depth bias or nonstandard rasterization"); + } queue.context[0x2ab] = 0; expectFailure([&] { AgcDriver::Graphics::DecodeState(queue); }, "invalid VGT_ESGS_RING_ITEMSIZE"); queue.context[0x2ab] = 4; @@ -947,6 +985,9 @@ void NggVertexWorkgroupTests() { queue.context[0x2d5] = stages; expectFailure([&] { AgcDriver::Graphics::DecodeState(queue, true); }, "requires PRIMGEN_EN without GS_EN, PRIMGEN_PASSTHRU_EN or tessellation"); } + queue.context[0x2d5] = 0x12010; + queue.context[0x29b] = 1; + expectFailure([&] { AgcDriver::Graphics::DecodeState(queue, true); }, "VGT_GS_OUT_PRIM_TYPE other than a triangle strip, or a line strip from a geometry program, is unsupported"); } constexpr std::array IdentityExports{0xe4u, 0xe4u, 0xe4u, 0xe4u, 0xe4u, 0xe4u, 0xe4u, 0xe4u}; diff --git a/core/libs/prx/libSceAgcDriver/tests/Mesh.cpp b/core/libs/prx/libSceAgcDriver/tests/Mesh.cpp index 9a9b548f49..1e59006cd6 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Mesh.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Mesh.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include namespace { @@ -41,6 +42,13 @@ alignas(256) constexpr std::array FastLaunchCode{ 0xbf810000, }; +alignas(256) constexpr std::array LineStripCode{ + 0xd7650000, 0x000100c1, 0x7e000505, 0x93018300, 0x4a020001, 0xe0382000, 0x80020201, 0xe0382010, + 0x80020601, 0xb07c2003, 0xbf900009, 0xbf8c3f70, 0x7d880083, 0xbefe036a, 0xf80008cf, 0x05040302, + 0xf800020f, 0x09080706, 0x7d880082, 0xbefe036a, 0x161400ff, 0x00000401, 0x4a1414ff, 0x00000400, + 0xf8000941, 0x0000000a, 0xbf810000, +}; + alignas(256) constexpr std::array PixelCode{ 0xc8020002, 0xc8060102, 0xc80a0202, 0xc80e0302, 0xf800180f, 0x03020100, 0xbf810000, }; @@ -85,6 +93,13 @@ alignas(256) std::array Indices{}; alignas(256) std::array Ordered{}; alignas(256) std::array Strip{}; +constexpr std::uint32_t LineStrips = 4; +alignas(256) std::array LineStripVertices{}; + +std::uint32_t LineStripRow(std::uint32_t strip) { + return 16u + 28u * strip; +} + constexpr std::uint32_t FanRim = 8; constexpr float FanStep = 6.28318530718f / FanRim; alignas(256) std::array Fan{}; @@ -279,6 +294,17 @@ int main() { const float y = i % 2u == 0u ? -1.0f : 0.0f; Strip[i] = {{x, y, 0.5f, 1.0f}, stripColor}; } + for (std::uint32_t strip = 0; strip < LineStrips; ++strip) { + const auto color = TriangleColor(strip); + const std::array rgba{color[0] / 255.0f, color[1] / 255.0f, color[2] / 255.0f, 1.0f}; + const float top = 1.0f - (static_cast(LineStripRow(strip)) + 0.5f) / (Height / 2.0f); + const float bottom = top - 20.0f / (Height / 2.0f); + const float left = 24.5f / (Width / 2.0f) - 1.0f; + const float right = 168.5f / (Width / 2.0f) - 1.0f; + LineStripVertices[3 * strip] = {{left, top, 0.5f, 1.0f}, rgba}; + LineStripVertices[3 * strip + 1] = {{right, top, 0.5f, 1.0f}, rgba}; + LineStripVertices[3 * strip + 2] = {{right, bottom, 0.5f, 1.0f}, rgba}; + } Fan[0] = {{0.0f, 0.0f, 0.5f, 1.0f}, {1.0f, 1.0f, 1.0f, 1.0f}}; for (std::uint32_t k = 1; k <= FanRim; ++k) { const auto color = TriangleColor(k); @@ -360,6 +386,24 @@ int main() { } Require(fastLaunchIndexedRefused, "an indexed fast-launch draw was drawn"); + ClearPixels(); + auto lineSubgroup = FastLaunchSubgroup; + lineSubgroup.maxPrimitives = 2u; + lineSubgroup.outputPrimitiveSize = 2u; + DrawMesh(device, {lineSubgroup, {0, LineStrips, 0, 1, 0, false}, VertexBufferDescriptor(LineStripVertices.data(), static_cast(LineStripVertices.size())), ShaderRecompiler::MeshDrawPushOffsetBytes, false, 32u, VK_CULL_MODE_NONE, LineStripCode, 0x482030u}); + const auto linePixel = [](std::uint32_t x, std::uint32_t y) { return (static_cast(y) * Width + x) * 4u; }; + for (std::uint32_t strip = 0; strip < LineStrips; ++strip) { + const auto row = LineStripRow(strip); + const auto color = TriangleColor(strip); + const auto where = "line strip " + std::to_string(strip) + ": "; + for (std::uint32_t x = 32; x <= 160; x += 8) Require(PixelIs(linePixel(x, row), color), where + "horizontal line pixel (" + std::to_string(x) + ", " + std::to_string(row) + ") is " + PixelText(linePixel(x, row))); + for (std::uint32_t y = row + 4u; y <= row + 16u; y += 4) Require(PixelIs(linePixel(168, y), color), where + "vertical line pixel (168, " + std::to_string(y) + ") is " + PixelText(linePixel(168, y))); + for (const auto& [x, y, what] : {std::tuple{96u, row - 3u, "above the horizontal line"}, std::tuple{96u, row + 3u, "below the horizontal line"}, std::tuple{150u, row + 5u, "inside the triangle of the strip's three vertices"}, std::tuple{96u, row + 10u, "on the edge closing the strip"}, std::tuple{96u, row + 24u, "between this strip and the next"}, std::tuple{8u, row, "left of the strip"}, std::tuple{184u, row + 10u, "right of the strip"}}) { + Require(PixelIs(linePixel(x, y), Background), where + "pixel (" + std::to_string(x) + ", " + std::to_string(y) + ") " + what + " is " + PixelText(linePixel(x, y))); + } + } + Require(PixelIs(0, Background) && PixelIs(Pixels.size() - 4u, Background), "line strips: the corners changed"); + constexpr std::size_t recordBlockBytes = 65536; auto* record = static_cast(::operator new(recordBlockBytes, std::align_val_t{65536})); { diff --git a/core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp b/core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp index a3294a30fe..5b7c940a01 100644 --- a/core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp @@ -2673,6 +2673,14 @@ void verifyMeshConfiguration() { other.graphics = GraphicsCompileContext{0u, {}, otherMesh, std::nullopt, {}}; RecompileCacheKey::Build(other, key); require(key != first && RecompileCacheKey::ContextHash(request) != RecompileCacheKey::ContextHash(other), "the cache keys ignore the mesh configuration"); + require(mesh.outputPrimitiveSize == 3u && replay.request.graphics->mesh->outputPrimitiveSize == 3u, "a mesh configuration does not output triangles by default"); + otherMesh = mesh; + otherMesh.outputPrimitiveSize = 2u; + other.graphics = GraphicsCompileContext{0u, {}, otherMesh, std::nullopt, {}}; + const auto lineReplay = RequestSerializer{}.Deserialize(RequestSerializer{}.Serialize(other)); + require(lineReplay.request.graphics.has_value() && lineReplay.request.graphics->mesh.has_value() && lineReplay.request.graphics->mesh->outputPrimitiveSize == 2u, "the mesh output primitive size was lost in serialization"); + RecompileCacheKey::Build(other, key); + require(key != first && RecompileCacheKey::ContextHash(request) != RecompileCacheKey::ContextHash(other), "the cache keys ignore the mesh output primitive size"); RecompileRequest hull{}; hull.shader = {ShaderStage::TessellationControl, 0x10000u, code, 0, {}}; hull.context.waveSize = 64; @@ -3049,7 +3057,7 @@ void verifyPixelRequestSerialization() { minimal.context.pixel->layerExported = true; minimal.context.pixel->ditheredAlphaToCoverage = true; const auto encoded = serializer.Serialize(minimal); - require(requestPrefix(encoded, 8u) == "NVNQQR4AAAA=", "new requests did not use serialization version 30"); + require(requestPrefix(encoded, 8u) == "NVNQQR8AAAA=", "new requests did not use serialization version 31"); const std::array fallbacks{0x40u, 0x1ac4u}; auto withFallbacks = minimal; withFallbacks.context.linkedCallFallbacks = fallbacks; @@ -3167,7 +3175,7 @@ void verifyPixelRequestSerialization() { for (std::size_t bytes = 0; bytes < 45u; ++bytes) { expectFailure([&] { static_cast(serializer.Deserialize(requestPrefix(encoded, mappingOffset + bytes))); }, "truncated data", "a truncated pixel mapping, packing, dual-source, alpha-to-coverage, stencil reference export flag, primitive class, layer export, dither or front-face flag, quad pixel mask or quad sample mask was accepted"); } - for (const auto unsupported : {"NVNQQQAAAAA=", "NVNQQR8AAAA="}) { + for (const auto unsupported : {"NVNQQQAAAAA=", "NVNQQSAAAAA="}) { expectFailure([&] { static_cast(serializer.Deserialize(unsupported)); }, "serialization version", "an unsupported request version was accepted"); } static constexpr std::string_view version18Request{ diff --git a/core/shader/recompiler/CacheKey.hpp b/core/shader/recompiler/CacheKey.hpp index 9a05433674..033a416692 100644 --- a/core/shader/recompiler/CacheKey.hpp +++ b/core/shader/recompiler/CacheKey.hpp @@ -104,7 +104,7 @@ private: const auto* mesh = request.graphics && request.graphics->mesh ? &*request.graphics->mesh : nullptr; append(key, mesh != nullptr); if (mesh == nullptr) return; - for (const auto value : {mesh->inputPrimitive, mesh->primitivesPerGroup, mesh->verticesPerGroup, mesh->maxVertices, mesh->maxPrimitives, mesh->threadsPerGroup, mesh->ldsSizeDwords, mesh->provokingVertex, mesh->esgsItemSize}) append(key, value); + for (const auto value : {mesh->inputPrimitive, mesh->primitivesPerGroup, mesh->verticesPerGroup, mesh->maxVertices, mesh->maxPrimitives, mesh->threadsPerGroup, mesh->ldsSizeDwords, mesh->provokingVertex, mesh->esgsItemSize, mesh->outputPrimitiveSize}) append(key, value); } static void appendTessellation(std::vector& key, const RecompileRequest& request) { diff --git a/core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp b/core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp index 20c4e29f51..3bee798ecb 100644 --- a/core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp +++ b/core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp @@ -518,6 +518,7 @@ void writeMeshConfiguration(Writer& writer, const MeshConfiguration& configurati writer.WriteU32(configuration.ldsSizeDwords); writer.WriteU32(configuration.provokingVertex); writer.WriteU32(configuration.esgsItemSize); + writer.WriteU32(configuration.outputPrimitiveSize); } MeshConfiguration readMeshConfiguration(Reader& reader, std::uint32_t version) { @@ -531,6 +532,7 @@ MeshConfiguration readMeshConfiguration(Reader& reader, std::uint32_t version) { configuration.ldsSizeDwords = reader.ReadU32(); configuration.provokingVertex = reader.ReadU32(); configuration.esgsItemSize = version >= 4u ? reader.ReadU32() : 4u; + if (version >= 31u) configuration.outputPrimitiveSize = reader.ReadU32(); return configuration; } @@ -776,7 +778,7 @@ std::string RequestSerializer::Serialize(const RecompileRequest& request) const std::string buffer; Writer writer(buffer); writer.WriteU32(0x41505335u); - writer.WriteU32(30u); + writer.WriteU32(31u); writeShaderBinary(writer, request.shader); writeGuestContext(writer, request.context); writeSpirvTarget(writer, request.target); @@ -809,7 +811,7 @@ DeserializedRequest RequestSerializer::Deserialize(std::string_view text) const Reader reader(decoded); if (reader.ReadU32() != 0x41505335u) throw std::runtime_error("invalid recompile request signature"); const auto version = reader.ReadU32(); - if (version < 1u || version > 30u) throw std::runtime_error("unsupported recompile request serialization version"); + if (version < 1u || version > 31u) throw std::runtime_error("unsupported recompile request serialization version"); DeserializedRequest result{}; result.request.shader = readShaderBinary(reader, result.shaderCode, result.shaderHeader); result.request.context = readGuestContext(reader, result, version); diff --git a/core/shader/recompiler/Optimization/include/Optimization/ShaderStageInputInfo.hpp b/core/shader/recompiler/Optimization/include/Optimization/ShaderStageInputInfo.hpp index 72fe2675ee..2ba98418bd 100644 --- a/core/shader/recompiler/Optimization/include/Optimization/ShaderStageInputInfo.hpp +++ b/core/shader/recompiler/Optimization/include/Optimization/ShaderStageInputInfo.hpp @@ -92,6 +92,7 @@ struct ShaderMeshInputInfo: ShaderWorkgroupInputInfo { std::uint32_t maxPrimitives = 0; std::uint32_t provokingVertex = 0; std::uint32_t esgsItemSize = 0; + std::uint32_t outputPrimitiveSize = 3; [[nodiscard]] std::uint32_t InputPrimitiveSize() const { return inputPrimitive == 1u ? 1u : inputPrimitive == 2u ? 2u : 3u; diff --git a/core/shader/recompiler/Recompiler.hpp b/core/shader/recompiler/Recompiler.hpp index 157ab11feb..7be06968e0 100644 --- a/core/shader/recompiler/Recompiler.hpp +++ b/core/shader/recompiler/Recompiler.hpp @@ -302,6 +302,7 @@ struct MeshConfiguration { std::uint32_t ldsSizeDwords; std::uint32_t provokingVertex; std::uint32_t esgsItemSize = 0; + std::uint32_t outputPrimitiveSize = 3; }; struct TessellationConfiguration { diff --git a/core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvModuleSetup.cpp b/core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvModuleSetup.cpp index 78de6340b4..e6efb53722 100644 --- a/core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvModuleSetup.cpp +++ b/core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvModuleSetup.cpp @@ -240,7 +240,7 @@ void DefineModule(SpirvEmitterState& state) { state.meshGuestFunc = state.module.AllocateId(); state.module.EmitCapability(spv::CapabilityMeshShadingEXT); state.module.EmitExtension("SPV_EXT_mesh_shader"); - state.module.AddExecutionMode(state.mainFunc, spv::ExecutionModeOutputTrianglesEXT); + state.module.AddExecutionMode(state.mainFunc, mesh.outputPrimitiveSize == 2u ? spv::ExecutionModeOutputLinesEXT : spv::ExecutionModeOutputTrianglesEXT); state.module.AddExecutionMode(state.mainFunc, spv::ExecutionModeOutputVertices, mesh.maxVertices); state.module.AddExecutionMode(state.mainFunc, spv::ExecutionModeOutputPrimitivesEXT, mesh.maxPrimitives); } diff --git a/core/shader/recompiler/SpirvBackend/src/SpirvModuleEmitter.cpp b/core/shader/recompiler/SpirvBackend/src/SpirvModuleEmitter.cpp index 774e16aa09..2c92bf42a0 100644 --- a/core/shader/recompiler/SpirvBackend/src/SpirvModuleEmitter.cpp +++ b/core/shader/recompiler/SpirvBackend/src/SpirvModuleEmitter.cpp @@ -767,11 +767,11 @@ void DefineMeshOutputs(SpirvEmitterState& state) { } state.meshAllocation = MeshArray(state, spv::StorageClassWorkgroup, TypeU32(state), 2u); state.meshPrimitiveData = MeshArray(state, spv::StorageClassPrivate, TypeU32(state), state.laneCount); - state.meshPrimitives = MeshArray(state, spv::StorageClassOutput, TypeU32Vector(state, 3u), mesh.maxPrimitives); + state.meshPrimitives = MeshArray(state, spv::StorageClassOutput, TypeU32Vector(state, mesh.outputPrimitiveSize), mesh.maxPrimitives); state.meshCull = MeshArray(state, spv::StorageClassOutput, TypeBool(state), mesh.maxPrimitives); state.interfaceVariables.push_back(state.meshPrimitives); state.interfaceVariables.push_back(state.meshCull); - state.module.AddAnnotation(spv::OpDecorate, state.meshPrimitives, spv::DecorationBuiltIn, spv::BuiltInPrimitiveTriangleIndicesEXT); + state.module.AddAnnotation(spv::OpDecorate, state.meshPrimitives, spv::DecorationBuiltIn, mesh.outputPrimitiveSize == 2u ? spv::BuiltInPrimitiveLineIndicesEXT : spv::BuiltInPrimitiveTriangleIndicesEXT); state.module.AddAnnotation(spv::OpDecorate, state.meshCull, spv::DecorationBuiltIn, spv::BuiltInCullPrimitiveEXT); state.module.AddAnnotation(spv::OpDecorate, state.meshCull, spv::DecorationPerPrimitiveEXT); if (state.numWorkgroupsVariable == 0u) { @@ -837,16 +837,16 @@ void EmitMeshEntryPoint(SpirvEmitterState& state) { EmitIfCondition(state, isPrimitive, [&]() { const auto packed = MeshLoad(state, state.meshPrimitiveData, spv::StorageClassPrivate, TypeU32(state), ConstantU32(state, half)); std::array vertex {}; - for (std::uint32_t component = 0; component < 3u; component++) { + for (std::uint32_t component = 0; component < mesh.outputPrimitiveSize; component++) { const auto field = state.module.AllocateId(); state.module.AddFunction(spv::OpBitFieldUExtract, TypeU32(state), field, packed, ConstantU32(state, component * 10u), ConstantU32(state, 9u)); vertex.at(component) = state.module.AllocateId(); state.module.AddFunction(spv::OpExtInst, TypeU32(state), vertex.at(component), GlslStd450(state), GLSLstd450UMin, field, lastVertex); } - const auto triangle = state.module.AllocateId(); - state.module.AddFunction(spv::OpCompositeConstruct, TypeU32Vector(state, 3u), triangle, vertex[0], vertex[1], vertex[2]); - const auto trianglePointer = MeshElement(state, state.meshPrimitives, spv::StorageClassOutput, TypeU32Vector(state, 3u), index); - state.module.AddFunction(spv::OpStore, trianglePointer, triangle); + const auto indices = state.module.AllocateId(); + state.module.AddFunction(spv::OpCompositeConstruct, TypeU32Vector(state, mesh.outputPrimitiveSize), indices, std::span(vertex.data(), mesh.outputPrimitiveSize)); + const auto indicesPointer = MeshElement(state, state.meshPrimitives, spv::StorageClassOutput, TypeU32Vector(state, mesh.outputPrimitiveSize), index); + state.module.AddFunction(spv::OpStore, indicesPointer, indices); const auto nullBit = EmitBinaryU32(state, spv::OpBitwiseAnd, packed, ConstantU32(state, 0x80000000u)); const auto culled = state.module.AllocateId(); state.module.AddFunction(spv::OpINotEqual, TypeBool(state), culled, nullBit, ConstantU32(state, 0u)); diff --git a/core/shader/recompiler/Translation/src/ShaderInputInfoBuilder.cpp b/core/shader/recompiler/Translation/src/ShaderInputInfoBuilder.cpp index 5de4543f7b..992dcf3388 100644 --- a/core/shader/recompiler/Translation/src/ShaderInputInfoBuilder.cpp +++ b/core/shader/recompiler/Translation/src/ShaderInputInfoBuilder.cpp @@ -217,7 +217,8 @@ ShaderStageInputInfo BuildShaderStageInputInfo(ShaderStageKind stage, const Gues target.maxPrimitives = mesh->maxPrimitives; target.provokingVertex = mesh->provokingVertex; target.esgsItemSize = mesh->esgsItemSize; - if (target.threadsNum[0] == 0u || target.maxVertices == 0u || target.maxPrimitives == 0u || target.provokingVertex > 2u) throw std::runtime_error("ShaderInputInfoBuilder: invalid mesh configuration"); + target.outputPrimitiveSize = mesh->outputPrimitiveSize; + if (target.threadsNum[0] == 0u || target.maxVertices == 0u || target.maxPrimitives == 0u || (target.outputPrimitiveSize != 2u && target.outputPrimitiveSize != 3u) || target.provokingVertex >= target.outputPrimitiveSize) throw std::runtime_error("ShaderInputInfoBuilder: invalid mesh configuration"); } ShaderStageInputInfo result; result.vertex = &vertexStorage; diff --git a/core/shader/recompiler/tests/RequestSerializerTests.cpp b/core/shader/recompiler/tests/RequestSerializerTests.cpp index b8d4b0e10e..54b249f255 100644 --- a/core/shader/recompiler/tests/RequestSerializerTests.cpp +++ b/core/shader/recompiler/tests/RequestSerializerTests.cpp @@ -206,7 +206,7 @@ int main() { Require(request.graphics->draw.indexAddress == 0xabcdef3400u && request.graphics->draw.indexCount == 57u && request.graphics->draw.indexElementBytes == 4u && request.graphics->draw.instanceCount == 9u); Require(request.target.float16DenormPreserve == preserve); auto upgraded = bytes; - upgraded[4] = 30u; + upgraded[4] = 31u; if (fixture.pixel) upgraded.insert(upgraded.begin() + 258u, {0u, 0u, 0u}); if (fixture.pixel) InsertPixelDefaultsAfterFrontFace(upgraded); if (fixture.tessellation) upgraded.insert(upgraded.end() - 35u, 5u, 0u); @@ -245,7 +245,7 @@ int main() { layerRejected = std::string_view(error.what()) == "RequestSerializer: truncated data"; } Require(layerRejected); - layered[4] = 30u; + layered[4] = 31u; layered.insert(layered.begin() + 259u, {0u, 0u}); InsertPixelDefaultsAfterFrontFace(layered); if (fixture.tessellation) layered.insert(layered.end() - 35u, 5u, 0u); @@ -270,7 +270,7 @@ int main() { Require(replay.request.context.pixel->rasterPrimitive == ShaderRecompiler::RasterPrimitive::Rectangle); Require(replay.request.context.vertex->paClVsOutCntl == 0x00050a05u); Require(replay.request.target.float16DenormPreserve == preserve); - bytes[4] = 30u; + bytes[4] = 31u; bytes.insert(bytes.begin() + 259u, {0u, 0u}); InsertPixelDefaultsAfterFrontFace(bytes); bytes.insert(bytes.end(), sizeof(std::uint64_t), 0u); @@ -394,7 +394,7 @@ int main() { Require(request.graphics->draw.indexAddress == 0xabcdef3400u && request.graphics->draw.indexCount == 57u && request.graphics->draw.indexElementBytes == 4u && request.graphics->draw.instanceCount == 9u); Require(request.target.float16DenormPreserve == preserve); auto upgraded = bytes; - upgraded[4] = 30u; + upgraded[4] = 31u; if (fixture.pixel) upgraded.insert(upgraded.begin() + 259u, {0u, 0u}); if (fixture.pixel) InsertPixelDefaultsAfterFrontFace(upgraded); upgraded.insert(upgraded.end(), sizeof(std::uint64_t), 0u); @@ -407,7 +407,7 @@ int main() { const auto replay25 = serializer.Deserialize(EncodeBase64(main25)); Require(!fixture.pixel || (replay25.request.context.pixel->layerExported == layer && replay25.request.context.pixel->ditheredAlphaToCoverage == dither && replay25.request.context.pixel->reversedBlend == 0u)); Require(replay25.request.target.float16DenormPreserve == preserve && (!fixture.tessellation || replay25.request.graphics->tessellation->primitiveGroupSize == 37u)); - main25[4] = 30u; + main25[4] = 31u; if (fixture.pixel) { main25.insert(main25.begin() + 260u, 0u); InsertPixelDefaultsAfterFrontFace(main25); @@ -426,9 +426,9 @@ int main() { const auto replay = serializer.Deserialize(encoded); Require(replay.request.target.float16DenormPreserve == preserve); auto bytes = DecodeBase64(encoded); - Require(bytes.size() >= 8u && bytes[4] == 30u && bytes[5] == 0u && bytes[6] == 0u && bytes[7] == 0u); + Require(bytes.size() >= 8u && bytes[4] == 31u && bytes[5] == 0u && bytes[6] == 0u && bytes[7] == 0u); auto future = bytes; - future[4] = 31u; + future[4] = 32u; bool futureRejected = false; try { static_cast(serializer.Deserialize(EncodeBase64(future))); diff --git a/docs/dev/debt/libSceAgcDriver.md b/docs/dev/debt/libSceAgcDriver.md index 09bd4b9432..b009e17be1 100644 --- a/docs/dev/debt/libSceAgcDriver.md +++ b/docs/dev/debt/libSceAgcDriver.md @@ -17,6 +17,7 @@ - [Indices far past every vertex fetch](../../../core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp) (libSceAgcDriver) - when a draw's highest index is at least 64Ki vertices past the largest per-vertex record count, every index past that count is moved to it in a copy of the indices, so the zero-padded vertex copies are not sized to the original index (PPSA04159 draws a list of `0xffffffff` with primitive restart off, which asked for 160 GiB). Each attribute fetched by vertex index reads zeros there as at the original index (out-of-bounds select 0 or 1); a draw with another out-of-bounds select keeps its indices. The vertex index the shader sees changes for the moved indices, which a shader that reads it directly would notice - [Depth clear draws](../../../core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp) (AGC driver) - a draw with `DB_RENDER_CONTROL.DEPTH_CLEAR_ENABLE` is taken to write `DB_DEPTH_CLEAR` to every pixel it covers whatever the depth test, depth bounds and depth bias say: it is drawn with an always-passing depth write and the viewport depth range pinned to the clear value. The draw is rasterized as submitted, so its coverage is the draw's own. Whether the DB still applies the depth test or HiZ to such a draw is not measured - [GS_FAST_LAUNCH geometry](../../../core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp) (AGC driver) - `VGT_SHADER_STAGES_EN.GS_FAST_LAUNCH = 1` with a point list and `GE_CNTL` groups of one vertex and one primitive decodes as one point per subgroup, with the primitive limit taken from `GE_NGG_SUBGRP_CNTL.PRIM_AMP_FACTOR` and the GS/ES VGPR counts not required. Follows how RADV drives mesh shaders on gfx10.3 (`ac_nir_lower_ngg.c`, the workgroup index comes from the vertex id of the first thread), not measured on the console. Fast launch 2, other group sizes or input primitives and indexed fast-launch draws still throw. PPSA03671's fast-launch draws decode, but the title stops earlier in a wave64 vertex program, so they have not reached the GPU yet. +- [Line-strip geometry output](../../../core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp) (AGC driver) - a geometry program with `VGT_GS_OUT_PRIM_TYPE = 1` (line strip) runs as a mesh shader with `OutputLinesEXT`: each primitive export is one line whose two vertex indices are packed like the three of a triangle, as Mesa's NGG lowering exports line strips (`ac_nir_lower_ngg_gs.c`), and each input primitive gets `VGT_GS_MAX_VERT_OUT - 1` lines where a triangle strip gets `- 2` triangles. A subgroup gets at least `GE_MAX_OUTPUT_PER_SUBGROUP` line slots, because Mesa's lowering exports one primitive, possibly null, per emitted vertex; whether the console's compiler does the same is not known, and triangle strips keep `- 2` per input. A fast-launch subgroup takes its line count from `GE_NGG_SUBGRP_CNTL.PRIM_AMP_FACTOR` as for triangles. Line output counts as lines for the polygon offset decode, as on the vertex path. The lines are drawn 1 pixel wide with Vulkan's default line rasterization, like lines on the vertex path: `PA_SU_LINE_CNTL` is not read, and neither its width nor the console's line rules were measured. Points, rectangle lists, per-stream types and line strips from a vertex program on the mesh path still throw. - [Alpha-to-coverage](../../../core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp) (AGC driver) - only `DB_ALPHA_TO_MASK = 0xaa01` is accepted: the enable bit with the four quad offsets at 2 and no `OFFSET_ROUND`, the undithered setting [radeonsi](https://gitlab.freedesktop.org/mesa/mesa/-/blob/bffd91947efc1932da99aca69fc540d6d4201b0c/src/gallium/drivers/radeonsi/si_state.c#L371-L381) programs. Draws are single-sampled, so the pixel shader [writes](../../../core/shader/recompiler/SpirvBackend/src/SpirvModuleEmitter.cpp) `SampleMask` 0 when its MRT0 alpha is below one half or NaN and all ones otherwise, and runs with late fragment tests. That rule was measured on a Radeon 660M (gfx1035, RADV with `RADV_DEBUG=noatocdithering`, which programs the same register) for 0, -0, 0.25, the two floats below one half, one half, the two above, 0.75, 1, 2, -1, both infinities and both NaNs, and is pinned in [AlphaToCoverage.cpp](../../../core/libs/prx/libSceAgcDriver/tests/execution/AlphaToCoverage.cpp); it is not measured on gfx1013. Vulkan's `alphaToCoverageEnable` is not used: the NVIDIA driver 615.78.08 dithers single-sample coverage (an alpha of 0.25 writes half the pixels of an RTX 3050 Laptop target). Other offsets, `OFFSET_ROUND`, a sample-mask export, and an MRT0 that is missing, swizzled, packed or exported without a floating-point alpha throw. A draw without a color target on MRT0 keeps alpha-to-coverage from its MRT0 export, as [radeonsi exports MRT0 as `32_AR` for alpha-to-coverage without a color buffer](https://gitlab.freedesktop.org/mesa/mesa/-/blob/mesa-24.0.0/src/gallium/drivers/radeonsi/si_state_shaders.cpp#L2425-L2431). A draw that enables alpha-to-coverage with no MRT0 export still throws: what the console covers then is not measured. Default dithered `DB_ALPHA_TO_MASK = 0x18701` additionally uses a per-quad sample-mask table [measured on gfx1036](https://github.com/boykopovar/AnyPS5/pull/4451#issuecomment-6124848921) for eight standard samples with FP16_ABGR or 32_ABGR exports; other sample counts, custom locations, EQAA combinations and dithered UNORM16/SNORM16 exports throw. The dithered rule is not measured on gfx1013. - [MSAA_ENABLE on a single-sample target](../../../core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp) (AGC driver) - `PA_SC_MODE_CNTL_0.MSAA_ENABLE` with `PA_SC_AA_CONFIG.MSAA_NUM_SAMPLES = 0` is drawn like the same state without the bit when sample 0 of the four `PA_SC_AA_SAMPLE_LOCS_PIXEL_*_0` registers is at the pixel center and the draw rasterizes filled triangles with alpha-to-coverage and conservative rasterization off. [RADV](https://gitlab.freedesktop.org/mesa/mesa/-/blob/745709a000482a4a24c91faef5c5a40bd653a863/src/amd/vulkan/radv_cmd_buffer.c#L13606-13609) programs this state for one-sample custom sample locations on GFX10 and later, where the bit makes the rasterizer use the programmed location of sample 0 ([061bc615](https://gitlab.freedesktop.org/mesa/mesa/-/commit/061bc6151a11597f720535f00e79a5752e24aff6)); that the bit changes nothing else for such a draw is inferred from that and is not measured on hardware. Other locations of sample 0, absent location registers, points, lines, rectangles, alpha-to-coverage and conservative rasterization throw - [sceAgcDriverGetEqContextId](../../../core/libs/prx/libSceAgcDriver/Eq/src/Query.cpp) (libSceAgcDriver) - returns `ident`: an end-of-pipe interrupt is delivered only to the registration whose id equals the raising queue, so `ident` is that context's id (titles register 0 or 1 for graphics and `0x20 + 8 * pipe + queue` for compute). The console's value is not known; KytyPS5 returns the event's `data`, which it fills with the interrupt's context id, whereas `data` here is the EV_CLEAR trigger count. Other event filters throw From 4aafda8e3b05cca9bebdab0ab647eed4b751ab4c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Wed, 14 Oct 2026 16:31:36 +0200 Subject: [PATCH 07/27] perf(agc): journal the pipelines of a run and build them ahead on the next one A graphics pipeline the store builds or a compute pipeline a dispatch builds is appended to pipelines.journal next to the shader cache entries, as the materialized result of each stage (the specialized SPIR-V, bindings and push constants) plus the recipe the pipeline was built from: the stages' identities, vertex input layout, descriptor set layout keys and the state fields the Pipeline constructor reads. When a VulkanDevice is created with a window (a headless device waits for its first draw or dispatch, since the window's device usually replaces it first) the journal is loaded and its pipelines are built on a pool of threads while the title loads, so the first draw or dispatch of a configuration seen on an earlier run finds its pipeline resident instead of compiling it on the queue worker. The pipeline stores key on RecompileResult::Identity, a hash of the module words, specialization constants, subgroup width and bindings that is the same on every run (SharedSpirv::ContentHash, computed once per shared module), rather than on the process-lifetime variant ids; a generated stage is keyed like any other. CachedPipeline builds outside the store's mutex and the library parts and link are built outside the library store's mutex, so replay threads and queue workers build in parallel; a concurrent build of the same key keeps the first inserted. A replayed pipeline enters the store at its least recently used end and a full store takes none, and its optimized relink waits for the first draw that binds it, so the title's own pipelines come first in both. The compute pipeline creation moves into VulkanDevice::createComputePipeline, shared by dispatches and the replay. The store key also covers the color component mapping, the depth target's sample count and a depth bias clamp, which the constructor reads. The file names the device it was written for (the properties the driver's pipeline cache file is named by) and starts over for another; a torn tail is cut at a record boundary; two runs sharing the directory take turns on a lock file and keep each other's records; a read failure never starts the file over. A device cancels its replay before it is replaced or destroyed, and a replaced device destroys its pipelines then rather than at the driver's stop. APS5_NO_PIPELINE_JOURNAL=1 disables it; APS5_PIPELINE_JOURNAL_THREADS, APS5_PIPELINE_JOURNAL_ENTRIES (the first recorded), APS5_PIPELINE_STORE_ENTRIES and APS5_PIPELINE_JOURNAL_MAX_MIB bound it; APS5_PROFILE_DRAW prints a [pipeline-journal] summary. APS5_VERIFY_SET_LAYOUTS=1 now also checks a dispatch's set layout against the one described from its shader. The lavapipe test suite, which shares one cache directory, runs with the journal off except for the journal's own tests. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01P6hw3kRAuQdfcU9YCZgaoK --- core/libs/prx/libSceAgcDriver/CMakeLists.txt | 14 + .../Execution/include/VulkanDevice.hpp | 14 + .../Execution/src/Driver/Lifecycle.cpp | 2 + .../Execution/src/VulkanDevice.cpp | 223 +++- .../Graphics/include/Pipeline.hpp | 29 +- .../Graphics/include/PipelineJournal.hpp | 104 ++ .../Graphics/include/PipelineLibrary.hpp | 7 + .../libSceAgcDriver/Graphics/src/Pipeline.cpp | 161 ++- .../Graphics/src/PipelineJournal.cpp | 1096 +++++++++++++++++ .../Graphics/src/PipelineLibrary.cpp | 29 +- .../prx/libSceAgcDriver/tests/Graphics.cpp | 257 +++- .../prx/libSceAgcDriver/tests/Recorder.cpp | 13 +- .../tests/execution/PipelineJournal.cpp | 431 +++++++ core/shader/recompiler/CMakeLists.txt | 6 + core/shader/recompiler/Recompiler.hpp | 109 +- core/shader/recompiler/ShaderDiskCache.cpp | 8 +- core/shader/recompiler/ShaderDiskCache.hpp | 2 +- docs/dev/ARCHITECTURE.md | 1 + docs/dev/debt/libSceAgcDriver.md | 1 + docs/user/USAGE.md | 4 + 20 files changed, 2367 insertions(+), 144 deletions(-) create mode 100644 core/libs/prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp create mode 100644 core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp create mode 100644 core/libs/prx/libSceAgcDriver/tests/execution/PipelineJournal.cpp diff --git a/core/libs/prx/libSceAgcDriver/CMakeLists.txt b/core/libs/prx/libSceAgcDriver/CMakeLists.txt index ebd508377d..fdaffbc3f8 100644 --- a/core/libs/prx/libSceAgcDriver/CMakeLists.txt +++ b/core/libs/prx/libSceAgcDriver/CMakeLists.txt @@ -206,6 +206,7 @@ set(agcDriverSources ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Pipeline.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/PipelineLibrary.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/PipelineCache.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/PipelineJournal.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Resources.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/PlaceholderImages.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Recorder.cpp @@ -1463,6 +1464,8 @@ target_include_directories(agc_driver_provoking_vertex_tests PRIVATE ${agcTestSh target_include_directories(agc_driver_prepared_exports_tests PRIVATE $) +target_include_directories(agc_driver_pipeline_journal_tests PRIVATE $) + target_include_directories(agc_driver_buffer_descriptor_runtime_tests PRIVATE $) target_include_directories(agc_driver_prepared_publication_tests PRIVATE $) @@ -1561,6 +1564,11 @@ if(BUILD_TESTING) set_tests_properties(agc_driver_native_depth_clears agc_driver_native_depth_clear_refusals PROPERTIES ENVIRONMENT_MODIFICATION "PATH=path_list_prepend:$") endif() set_tests_properties(agc_sparse_buffer_holes PROPERTIES TIMEOUT 30) + add_test(NAME agc_pipeline_journal COMMAND agc_driver_graphics_tests --pipeline-journal) + set_tests_properties(agc_pipeline_journal PROPERTIES TIMEOUT 30 WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}") + if(WIN32) + set_tests_properties(agc_pipeline_journal PROPERTIES ENVIRONMENT_MODIFICATION "PATH=path_list_prepend:$") + endif() set_tests_properties(agc_driver_sparse_buffer_holes PROPERTIES TIMEOUT 30) if(WIN32) set_tests_properties(agc_sparse_buffer_holes PROPERTIES ENVIRONMENT_MODIFICATION "PATH=path_list_prepend:$") @@ -2387,3 +2395,9 @@ if(BUILD_TESTING) "PATH=path_list_prepend:$") endif() endif() + +if(BUILD_TESTING) + get_directory_property(agcRegisteredTests TESTS) + list(FILTER agcRegisteredTests EXCLUDE REGEX "^(agc_pipeline_journal|agc_driver_pipeline_journal)$") + set_property(TEST ${agcRegisteredTests} APPEND PROPERTY ENVIRONMENT "APS5_NO_PIPELINE_JOURNAL=1") +endif() diff --git a/core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp b/core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp index 87800734a3..08de3994f0 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp +++ b/core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp @@ -30,6 +30,8 @@ struct ComputePipelineObjects; struct RecipeHit; // The record tail shared by an ordinary dispatch and a recipe hit (VulkanDevice::recordDispatch). struct RecordedDispatch; +// The module, layout and pipeline a compute dispatch binds (VulkanDevice::createComputePipeline). +struct ComputePipelineObjects; class VulkanDevice { public: @@ -37,6 +39,14 @@ public: struct CompileOnly {}; explicit VulkanDevice(CompileOnly); ComputePipelineReplayResult ReplayComputePipeline(const ComputePipelineCapture& capture, std::span initialCache = {}); + // Builds the pipeline of a journaled compute result ahead of its first dispatch (the journal + // replay's hook, on its threads): false when the device already holds a pipeline for the module. + bool ReplayComputePipeline(const ShaderRecompiler::RecompileResult& shader, bool pushConstants); + // Starts the journal replay if the device has not (a headless device starts it at its first + // draw or dispatch) and joins it (tests; the teardown joins it anyway). + void WaitForPipelineJournalReplay(); + // Compute pipelines this device built, by dispatches and by the journal replay (tests). + std::uint64_t ComputePipelinesCreated() const; ~VulkanDevice(); VulkanDevice(const VulkanDevice&) = delete; VulkanDevice& operator=(const VulkanDevice&) = delete; @@ -58,6 +68,7 @@ public: // three-step form below so the GPU wait happens without the mutex. void WaitIdle(); void PrepareForReplacement(); + void startJournalReplay(); bool ImportGuestMemory(const GuestAllocations::Mapped& ranges, std::uint64_t generation, bool adoptDevice); // Sends recorded work to the GPU without waiting for it. With `reapFirst` it first retires batches // that already finished, so the in-flight list stays short (APS5_NO_OPPORTUNISTIC_REAP=1 skips @@ -316,6 +327,9 @@ private: // The device's Graphics::Context: a copy of the one built at setup (its instance functions // resolved then, its function table filled then), or with APS5_NO_CONTEXT_CACHE=1 built anew. Graphics::Context graphicsContext() const; + VkPipelineCreateFlags computePipelineFlags() const; + std::uint32_t computeSubgroupSize(const ShaderRecompiler::RecompileResult& shader) const; + std::shared_ptr createComputePipeline(const Graphics::Context& context, const ShaderRecompiler::RecompileResult& shader, VkDescriptorSetLayout setLayout, std::span layoutKey, VkShaderStageFlags pushStages, std::uint64_t programAddress); ShaderRecompiler::SpirvTarget buildTarget() const; Graphics::Context buildContext() const; // Body of Dispatch and DispatchIndirect: `arguments` 0 dispatches x, y, z groups. diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Lifecycle.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Lifecycle.cpp index 6dc5164943..7aa349ce2d 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Lifecycle.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Lifecycle.cpp @@ -1,4 +1,5 @@ #include "prx/libSceAgcDriver/Execution/include/AsyncLog.hpp" +#include "prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp" #include "prx/libSceAgcDriver/Execution/include/Driver/Driver.hpp" #include "prx/libSceAgcDriver/Execution/include/Driver/Diagnostics.hpp" #include "prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp" @@ -119,6 +120,7 @@ void Driver::stop() { std::lock_guard gpuLock(GuestMemory::GpuMutex()); device.Reset(); replacedDevices.clear(); + Graphics::FlushPipelineJournal(); Graphics::ShutdownGuestBufferWorkers(); PerformanceTrace::StopOutput(); FlushAsyncLog_nid_no_patch(); diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp index 694152052f..661b90c564 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp @@ -6,6 +6,7 @@ #if ANYPS5_ENABLE_SPIRV_TOOLS #include "SpirvBackend/SpirvOptimizer.hpp" #endif +#include "prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp" #include "prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp" #include "prx/libSceAgcDriver/Execution/include/Driver.hpp" #include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp" @@ -215,6 +216,7 @@ struct ComputePipelineObjects { VkShaderModule module = VK_NULL_HANDLE; VkPipelineLayout layout = VK_NULL_HANDLE; VkPipeline pipeline = VK_NULL_HANDLE; + VkDescriptorSetLayout resourcesLayout = VK_NULL_HANDLE; ~ComputePipelineObjects() { if (pipeline != VK_NULL_HANDLE) destroyPipeline(device, pipeline, nullptr); if (layout != VK_NULL_HANDLE) destroyLayout(device, layout, nullptr); @@ -222,6 +224,16 @@ struct ComputePipelineObjects { } }; +// Pipelines are shared by dispatches of one module (the result's run-stable identity) with the +// same push constant use; descriptor set layouts built from the same bindings are compatible, so +// the pipeline layout of the first dispatch, or of the journal replay, serves them all. +struct ComputePipelineKey { + ShaderRecompiler::PipelineIdentity identity; + bool pushConstants = false; + auto operator<=>(const ComputePipelineKey&) const = default; + static ComputePipelineKey Of(const ShaderRecompiler::RecompileResult& shader, bool pushConstants) { return {shader.Identity(), pushConstants}; } +}; + constexpr std::uint64_t WorkgroupMemoryGranule = 1u << 20u; struct WorkgroupMemory { @@ -362,8 +374,12 @@ struct VulkanDevice::State { // dispatch's find-or-insert and the verify switch's lookups touch the map, recipes hold weak // references to its objects. std::mutex computePipelinesMutex; - std::map> computePipelines; - std::map>> computePipelinesBuilding; + std::map> computePipelines; + std::map>> computePipelinesBuilding; + // The journal replay building this device's pipelines on its own threads; drained first at teardown. + std::shared_ptr journalReplay; + std::atomic journalReplayPending{false}; + std::atomic computePipelinesCreated{0}; std::unique_ptr colorTransfer; std::shared_ptr bufferPool; std::unique_ptr emptyBuffer; @@ -671,6 +687,12 @@ struct VulkanDevice::State { } ~State() { + if (journalReplay != nullptr) { + journalReplay->Cancel(); + journalReplay->Wait(); + journalReplay.reset(); + Graphics::ReportPipelineJournal(); + } contextReady = false; if (deviceLostHook && Graphics::DeviceLostOwner() == this) { Graphics::DeviceLostHook() = nullptr; @@ -1928,10 +1950,53 @@ VulkanDevice::VulkanDevice(const PresentationWindow* window, bool compileOnly) : state->scaler = std::make_unique(graphicsContext(), VK_FORMAT_B8G8R8A8_UNORM, state->surfaceFormat); state->presentation = std::make_unique(graphicsContext(), state->presentSlots.size()); } + // A device without a window is usually replaced by the window's device before the title draws + // (Driver::AttachWindow): its replay starts at its first draw or dispatch instead, so the + // journal is replayed once, on the device the title draws with. + state->journalReplayPending.store(!compileOnly, std::memory_order_release); + if (window != nullptr) startJournalReplay(); +} + +void VulkanDevice::startJournalReplay() { + if (!state->journalReplayPending.exchange(false, std::memory_order_acq_rel)) return; + state->journalReplay = Graphics::StartPipelineJournalReplay(graphicsContext(), Graphics::PipelineJournalDeviceFingerprint(state->properties), {[this](const ShaderRecompiler::RecompileResult& shader, bool pushConstants) { return ReplayComputePipeline(shader, pushConstants); }}); } VulkanDevice::~VulkanDevice() = default; +VkPipelineCreateFlags VulkanDevice::computePipelineFlags() const { + return (graphicsContext().pipelineExecutableInfo ? VK_PIPELINE_CREATE_CAPTURE_STATISTICS_BIT_KHR : 0) | VK_PIPELINE_CREATE_DISPATCH_BASE_BIT; +} + +std::uint32_t VulkanDevice::computeSubgroupSize(const ShaderRecompiler::RecompileResult& shader) const { + return state->computeWave32 && (shader.hostSubgroupSize == 32u || shader.hostSubgroupSize > state->maxComputeSubgroupSize) ? std::min(shader.hostSubgroupSize, state->maxComputeSubgroupSize) : 0u; +} + +void VulkanDevice::WaitForPipelineJournalReplay() { + startJournalReplay(); + if (state->journalReplay != nullptr) state->journalReplay->Wait(); +} + +std::uint64_t VulkanDevice::ComputePipelinesCreated() const { + return state->computePipelinesCreated.load(std::memory_order_relaxed); +} + +bool VulkanDevice::ReplayComputePipeline(const ShaderRecompiler::RecompileResult& shader, bool pushConstants) { + ShaderRecompiler::RuntimeAbi::RequireVersion(shader.runtimeAbiVersion); + const auto key = ComputePipelineKey::Of(shader, pushConstants); + if (key.identity.Empty()) return false; + { + std::lock_guard pipelines(state->computePipelinesMutex); + if (state->computePipelines.contains(key) || state->computePipelinesBuilding.contains(key)) return false; + } + const auto context = graphicsContext(); + const std::array shaders{{{ShaderRecompiler::ShaderStage::Compute, &shader, 0}}}; + const auto layouts = Graphics::DescribeSetLayouts(context, shaders); + auto objects = createComputePipeline(context, shader, layouts.layouts.main, layouts.mainKey, pushConstants ? VK_SHADER_STAGE_COMPUTE_BIT : 0u, 0); + std::lock_guard pipelines(state->computePipelinesMutex); + return state->computePipelines.try_emplace(key, std::move(objects)).second; +} + bool VulkanDevice::ImportGuestMemory(const GuestAllocations::Mapped& ranges, std::uint64_t generation, bool adoptDevice) { if (adoptDevice) GuestMemory::AssertGpuLockHeld("VulkanDevice::ImportGuestMemory"); return Graphics::ImportMappedRanges(graphicsContext(), ranges, generation, adoptDevice); @@ -1940,10 +2005,21 @@ bool VulkanDevice::ImportGuestMemory(const GuestAllocations::Mapped& ranges, std void VulkanDevice::PrepareForReplacement() { GuestMemory::AssertGpuLockHeld("VulkanDevice::PrepareForReplacement"); require(state->recorder != nullptr && Graphics::Recorder::Active() == state->recorder.get(), "device replacement requires its active recorder"); + state->journalReplayPending.store(false, std::memory_order_release); + if (state->journalReplay != nullptr) { + state->journalReplay->Cancel(); + state->journalReplay->Wait(); + state->journalReplay.reset(); + } Graphics::FlushCachedTextures(state->device); Graphics::PublishAllShadows(state->context, Graphics::PublishReason::Teardown); WaitIdle(); Graphics::DestroyShadows(state->device); + // The replacement's draws never find this device's pipelines (the store keys on the device), and + // the device lives on until the driver stops: destroy them now rather than then. + Graphics::ClearCachedPipelines(state->device); + std::lock_guard pipelines(state->computePipelinesMutex); + state->computePipelines.clear(); } void VulkanDevice::WaitIdle() { @@ -3558,6 +3634,7 @@ bool VulkanDevice::PrebuildGraphicsLibraries(const Graphics::ShaderStages& stage } void VulkanDevice::Draw(const Graphics::State& graphics, const Pm4::DrawParameters& draw, std::span shaders, std::span snapshots, std::shared_ptr* recipe) { + startJournalReplay(); for (const auto& shader : shaders) { require(shader.program != nullptr, "missing compiled shader"); ShaderRecompiler::RuntimeAbi::RequireVersion(shader.program->runtimeAbiVersion); @@ -4417,11 +4494,74 @@ void VulkanDevice::recordDispatch(RecordedDispatch& record) { recordStep(PhaseRecordCompletion); } -std::shared_ptr VulkanDevice::computePipeline(const Graphics::Context& context, const ShaderRecompiler::RecompileResult& shader, VkDescriptorSetLayout setLayout, std::span layoutKey, VkShaderStageFlags pushStages, std::uint64_t programAddress) { +// The module, pipeline layout and pipeline of a compute result over `setLayout` (the set layout +// its bindings describe): what a dispatch or its preparation builds at the first use of a module, +// and what the journal replay builds on its own threads before that use. +std::shared_ptr VulkanDevice::createComputePipeline(const Graphics::Context& context, const ShaderRecompiler::RecompileResult& shader, VkDescriptorSetLayout setLayout, std::span layoutKey, VkShaderStageFlags pushStages, std::uint64_t programAddress) { static const bool profile = std::getenv("APS5_PROFILE_DRAW") != nullptr; - const std::uint64_t pipelineKey = shader.PipelineVariantId() != 0 ? (shader.PipelineVariantId() << 1u) | (pushStages != 0 ? 1u : 0u) : 0u; + const auto start = std::chrono::steady_clock::now(); + auto objects = std::make_shared(); + objects->device = state->device; + objects->destroyModule = state->DeviceFunction("vkDestroyShaderModule"); + objects->destroyLayout = state->DeviceFunction("vkDestroyPipelineLayout"); + objects->destroyPipeline = state->DeviceFunction("vkDestroyPipeline"); + VkShaderModuleCreateInfo moduleInfo{VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO}; + moduleInfo.codeSize = shader.spirv.size() * sizeof(std::uint32_t); + moduleInfo.pCode = shader.spirv.data(); + check(state->DeviceFunction("vkCreateShaderModule")(state->device, &moduleInfo, nullptr, &objects->module), "vkCreateShaderModule"); + objects->resourcesLayout = setLayout; + const std::array setLayouts{setLayout, shader.workgroupMemoryDwords != 0 ? state->workgroupMemoryLayout : VK_NULL_HANDLE}; + const VkPushConstantRange push{VK_SHADER_STAGE_COMPUTE_BIT, 0, Graphics::ComputePushConstantBytes}; + VkPipelineLayoutCreateInfo layoutInfo{VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO}; + layoutInfo.setLayoutCount = shader.workgroupMemoryDwords != 0 ? ShaderRecompiler::WorkgroupMemoryDescriptorSet + 1 : 1; + layoutInfo.pSetLayouts = setLayouts.data(); + layoutInfo.pushConstantRangeCount = pushStages != 0 ? 1 : 0; + layoutInfo.pPushConstantRanges = pushStages != 0 ? &push : nullptr; + check(state->DeviceFunction("vkCreatePipelineLayout")(state->device, &layoutInfo, nullptr, &objects->layout), "vkCreatePipelineLayout"); + const VkPipelineCreateFlags pipelineFlags = computePipelineFlags(); + const Graphics::PipelineSpecialization specialization(shader); + const auto subgroupSize = computeSubgroupSize(shader); + if (const char* directory = std::getenv("APS5_DUMP_COMPUTE_PIPELINES"); directory != nullptr && *directory != '\0' && CaptureComputePipeline(shader.spirv.size())) { + ComputePipelineCapture capture; + capture.spirv = shader.spirv.Copy(); + ComputeDescriptorSetLayout resourcesLayout; + const bool updateAfterBind = layoutKey.size() % 4u == 1u; + resourcesLayout.flags = updateAfterBind ? VK_DESCRIPTOR_SET_LAYOUT_CREATE_UPDATE_AFTER_BIND_POOL_BIT : 0; + for (std::size_t index = 0; index + 4u <= layoutKey.size(); index += 4u) { + const auto type = static_cast(layoutKey[index + 1u]); + resourcesLayout.bindings.push_back({layoutKey[index], type, layoutKey[index + 2u], layoutKey[index + 3u], updateAfterBind && type == VK_DESCRIPTOR_TYPE_STORAGE_BUFFER ? VK_DESCRIPTOR_BINDING_UPDATE_AFTER_BIND_BIT : 0u}); + } + capture.sets.push_back(std::move(resourcesLayout)); + if (shader.workgroupMemoryDwords != 0) capture.sets.push_back({0, {{0, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT, 0}}}); + if (pushStages != 0) capture.pushConstants.push_back(push); + for (const auto& value : shader.specialization) capture.specialization.push_back({value.id, value.value}); + capture.requiredSubgroupSize = subgroupSize; + capture.flags = pipelineFlags; + capture.deviceName = state->properties.deviceName; + capture.vendorId = state->properties.vendorID; + capture.deviceId = state->properties.deviceID; + capture.driverVersion = state->properties.driverVersion; + capture.apiVersion = state->properties.apiVersion; + std::copy_n(state->properties.pipelineCacheUUID, VK_UUID_SIZE, capture.pipelineCacheUuid.begin()); + const auto path = capture.Write(directory); + std::fprintf(stderr, "[compute-capture] %s\n", path.string().c_str()); + } + if (profile && shader.spirv.size() > 100000) std::fprintf(stderr, "[dispatch] creating a pipeline for %zu SPIR-V words (program 0x%llx)\n", shader.spirv.size(), static_cast(programAddress)); + CreateComputePipeline(state->device, state->DeviceFunction("vkCreateComputePipelines"), context.pipelineCache, objects->module, objects->layout, pipelineFlags, 0, specialization.Info(), subgroupSize, &objects->pipeline, context.pipelineCacheSynchronization); + objects->destroyModule(objects->device, objects->module, nullptr); + objects->module = VK_NULL_HANDLE; + computePipelinesCreated.fetch_add(1, std::memory_order_relaxed); + state->computePipelinesCreated.fetch_add(1, std::memory_order_relaxed); + if (GuestMemory::GpuMutex().HeldByThisThread()) computePipelinesCreatedLocked.fetch_add(1, std::memory_order_relaxed); + Graphics::LogPipelineStatistics_nid_no_patch(context, objects->pipeline); + if (profile && shader.spirv.size() > 100000) std::fprintf(stderr, "[dispatch] created a pipeline for %zu SPIR-V words in %.0f ms (program 0x%llx)\n", shader.spirv.size(), std::chrono::duration(std::chrono::steady_clock::now() - start).count(), static_cast(programAddress)); + return objects; +} + +std::shared_ptr VulkanDevice::computePipeline(const Graphics::Context& context, const ShaderRecompiler::RecompileResult& shader, VkDescriptorSetLayout setLayout, std::span layoutKey, VkShaderStageFlags pushStages, std::uint64_t programAddress) { + const auto pipelineKey = ComputePipelineKey::Of(shader, pushStages != 0); std::promise> building; - if (pipelineKey != 0) { + if (!pipelineKey.identity.Empty()) { std::shared_future> other; { std::lock_guard pipelines(state->computePipelinesMutex); @@ -4432,7 +4572,7 @@ std::shared_ptr VulkanDevice::computePipeline(const Grap if (other.valid()) return other.get(); } const auto finish = [&](std::shared_ptr built, std::exception_ptr error) { - if (pipelineKey == 0) return built; + if (pipelineKey.identity.Empty()) return built; { std::lock_guard pipelines(state->computePipelinesMutex); if (built != nullptr) built = state->computePipelines.try_emplace(pipelineKey, built).first->second; @@ -4444,60 +4584,8 @@ std::shared_ptr VulkanDevice::computePipeline(const Grap }; std::shared_ptr objects; try { - const auto start = std::chrono::steady_clock::now(); - objects = std::make_shared(); - objects->device = state->device; - objects->destroyModule = state->DeviceFunction("vkDestroyShaderModule"); - objects->destroyLayout = state->DeviceFunction("vkDestroyPipelineLayout"); - objects->destroyPipeline = state->DeviceFunction("vkDestroyPipeline"); - VkShaderModuleCreateInfo moduleInfo{VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO}; - moduleInfo.codeSize = shader.spirv.size() * sizeof(std::uint32_t); - moduleInfo.pCode = shader.spirv.data(); - check(state->DeviceFunction("vkCreateShaderModule")(state->device, &moduleInfo, nullptr, &objects->module), "vkCreateShaderModule"); - const std::array setLayouts{setLayout, shader.workgroupMemoryDwords != 0 ? state->workgroupMemoryLayout : VK_NULL_HANDLE}; - const VkPushConstantRange push{VK_SHADER_STAGE_COMPUTE_BIT, 0, Graphics::ComputePushConstantBytes}; - VkPipelineLayoutCreateInfo layoutInfo{VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO}; - layoutInfo.setLayoutCount = shader.workgroupMemoryDwords != 0 ? ShaderRecompiler::WorkgroupMemoryDescriptorSet + 1 : 1; - layoutInfo.pSetLayouts = setLayouts.data(); - layoutInfo.pushConstantRangeCount = pushStages != 0 ? 1 : 0; - layoutInfo.pPushConstantRanges = pushStages != 0 ? &push : nullptr; - check(state->DeviceFunction("vkCreatePipelineLayout")(state->device, &layoutInfo, nullptr, &objects->layout), "vkCreatePipelineLayout"); - const VkPipelineCreateFlags pipelineFlags = (context.pipelineExecutableInfo ? VK_PIPELINE_CREATE_CAPTURE_STATISTICS_BIT_KHR : 0) | VK_PIPELINE_CREATE_DISPATCH_BASE_BIT; - const Graphics::PipelineSpecialization specialization(shader); - const auto subgroupSize = state->computeWave32 && (shader.hostSubgroupSize == 32u || shader.hostSubgroupSize > state->maxComputeSubgroupSize) ? std::min(shader.hostSubgroupSize, state->maxComputeSubgroupSize) : 0u; - if (const char* directory = std::getenv("APS5_DUMP_COMPUTE_PIPELINES"); directory != nullptr && *directory != '\0' && CaptureComputePipeline(shader.spirv.size())) { - ComputePipelineCapture capture; - capture.spirv = shader.spirv.Copy(); - ComputeDescriptorSetLayout resourcesLayout; - const bool updateAfterBind = layoutKey.size() % 4u == 1u; - resourcesLayout.flags = updateAfterBind ? VK_DESCRIPTOR_SET_LAYOUT_CREATE_UPDATE_AFTER_BIND_POOL_BIT : 0; - for (std::size_t index = 0; index + 4u <= layoutKey.size(); index += 4u) { - const auto type = static_cast(layoutKey[index + 1u]); - resourcesLayout.bindings.push_back({layoutKey[index], type, layoutKey[index + 2u], layoutKey[index + 3u], updateAfterBind && type == VK_DESCRIPTOR_TYPE_STORAGE_BUFFER ? VK_DESCRIPTOR_BINDING_UPDATE_AFTER_BIND_BIT : 0u}); - } - capture.sets.push_back(std::move(resourcesLayout)); - if (shader.workgroupMemoryDwords != 0) capture.sets.push_back({0, {{0, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT, 0}}}); - if (pushStages != 0) capture.pushConstants.push_back(push); - for (const auto& value : shader.specialization) capture.specialization.push_back({value.id, value.value}); - capture.requiredSubgroupSize = subgroupSize; - capture.flags = pipelineFlags; - capture.deviceName = state->properties.deviceName; - capture.vendorId = state->properties.vendorID; - capture.deviceId = state->properties.deviceID; - capture.driverVersion = state->properties.driverVersion; - capture.apiVersion = state->properties.apiVersion; - std::copy_n(state->properties.pipelineCacheUUID, VK_UUID_SIZE, capture.pipelineCacheUuid.begin()); - const auto path = capture.Write(directory); - std::fprintf(stderr, "[compute-capture] %s\n", path.string().c_str()); - } - if (profile && shader.spirv.size() > 100000) std::fprintf(stderr, "[dispatch] creating a pipeline for %zu SPIR-V words (program 0x%llx)\n", shader.spirv.size(), static_cast(programAddress)); - CreateComputePipeline(state->device, state->DeviceFunction("vkCreateComputePipelines"), context.pipelineCache, objects->module, objects->layout, pipelineFlags, 0, specialization.Info(), subgroupSize, &objects->pipeline, context.pipelineCacheSynchronization); - objects->destroyModule(objects->device, objects->module, nullptr); - objects->module = VK_NULL_HANDLE; - computePipelinesCreated.fetch_add(1, std::memory_order_relaxed); - if (GuestMemory::GpuMutex().HeldByThisThread()) computePipelinesCreatedLocked.fetch_add(1, std::memory_order_relaxed); - Graphics::LogPipelineStatistics_nid_no_patch(context, objects->pipeline); - if (profile && shader.spirv.size() > 100000) std::fprintf(stderr, "[dispatch] created a pipeline for %zu SPIR-V words in %.0f ms (program 0x%llx)\n", shader.spirv.size(), std::chrono::duration(std::chrono::steady_clock::now() - start).count(), static_cast(programAddress)); + objects = createComputePipeline(context, shader, setLayout, layoutKey, pushStages, programAddress); + if (Graphics::PipelineJournalEnabled()) Graphics::RecordComputePipeline(shader, pushStages != 0); } catch (...) { finish(nullptr, std::current_exception()); throw; @@ -4506,6 +4594,7 @@ std::shared_ptr VulkanDevice::computePipeline(const Grap } VulkanDevice::IndirectOutcome VulkanDevice::dispatch(const ShaderRecompiler::RecompileResult& shader, std::uint32_t x, std::uint32_t y, std::uint32_t z, std::uint64_t arguments, std::span snapshots, std::uint64_t programAddress, std::shared_ptr prepared, std::shared_ptr* recipeOut) { + startJournalReplay(); state->recorder->BoundKeptBytes(); ShaderRecompiler::RuntimeAbi::RequireVersion(shader.runtimeAbiVersion); PerformanceTimer timing("Vulkan.Dispatch"); @@ -4525,11 +4614,9 @@ VulkanDevice::IndirectOutcome VulkanDevice::dispatch(const ShaderRecompiler::Rec } auto pushBytes = Graphics::AssemblePushConstants(shaders); const auto context = graphicsContext(); - // Pipelines are shared by dispatches of one compiled variant; descriptor set layouts built from the - // same bindings are compatible, so the pipeline layout of the first dispatch serves them all. - const std::uint64_t pipelineKey = shader.PipelineVariantId() != 0 ? (shader.PipelineVariantId() << 1u) | (pushStages != 0 ? 1u : 0u) : 0u; + const auto pipelineKey = ComputePipelineKey::Of(shader, pushStages != 0); std::shared_ptr objects = prepared != nullptr ? prepared->objects : nullptr; - if (objects == nullptr && pipelineKey != 0) { + if (objects == nullptr && !pipelineKey.identity.Empty()) { std::lock_guard pipelines(state->computePipelinesMutex); if (const auto found = state->computePipelines.find(pipelineKey); found != state->computePipelines.end()) objects = found->second; } @@ -4672,7 +4759,13 @@ VulkanDevice::IndirectOutcome VulkanDevice::dispatch(const ShaderRecompiler::Rec } timing.Mark("shader_resources"); timer.phase(PhaseResources); + static const bool verifySetLayouts = std::getenv("APS5_VERIFY_SET_LAYOUTS") != nullptr; + if (verifySetLayouts && objects != nullptr) require(objects->resourcesLayout == resources->Layout(), "the resident compute pipeline's set layout differs from the dispatch's"); if (objects == nullptr) { + if (verifySetLayouts && context.descriptorCache != nullptr) { + const auto described = Graphics::DescribeSetLayouts(context, shaders); + require(described.layouts.main == resources->Layout() && !described.layouts.split, "the set layout described from the compute shader differs from the dispatch's"); + } const auto layouts = resources->SetLayouts(); objects = computePipeline(context, shader, layouts.main, layouts.mainKey, pushStages, programAddress); timing.Mark("pipeline_create"); @@ -4806,7 +4899,7 @@ RecipeOutcome VulkanDevice::DispatchRecipe(const ShaderRecompiler::RecompileResu return RecipeOutcome::Rebuild; } std::shared_ptr mapped; - const auto pipelineKey = (shader.PipelineVariantId() << 1u) | (recipe.pushes ? 1u : 0u); + const auto pipelineKey = ComputePipelineKey::Of(shader, recipe.pushes); { std::lock_guard pipelines(state->computePipelinesMutex); if (const auto found = state->computePipelines.find(pipelineKey); found != state->computePipelines.end()) mapped = found->second; diff --git a/core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp b/core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp index a195283b06..55c7ba06c2 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp @@ -94,14 +94,29 @@ private: std::vector framebuffers; }; -// The pipeline for this configuration, shared across draws and frames: keyed by the stages' variant -// ids (the rect-list tessellation pair by the vertex and fragment ids that generate it), the vertex -// input layout, the descriptor set layout key, push constant stages and every pipeline state field. -// A configuration whose stage has no variant id gets a private pipeline. Entries are LRU-bounded and -// only evicted once no recorded draw holds them. Debug aid: APS5_NO_PIPELINE_CACHE=1 builds one per -// draw as before. +// The pipeline for this configuration, shared across draws and frames: keyed by the stages' +// identities (the rect-list tessellation pair and the geometry stage by the vertex and fragment +// results that generate them), the vertex input layout, the descriptor set layout key, push constant +// stages and every pipeline state field. A configuration whose stage has no module gets a private +// pipeline. A miss builds the pipeline outside the store's mutex (a concurrent build of the same +// key keeps the first inserted) and records its recipe in the pipeline journal. Entries are +// LRU-bounded (APS5_PIPELINE_STORE_ENTRIES, 1024) and only evicted once no recorded draw holds +// them. Debug aid: APS5_NO_PIPELINE_CACHE=1 builds one per draw as before. std::shared_ptr CachedPipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const ShaderResources& resources, std::span shaders, VkImageLayout attachmentLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL); -std::shared_ptr CachedPipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL); +// `created` reports whether this call built the pipeline (false: served from the store or private). +std::shared_ptr CachedPipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL, bool* created = nullptr); +// Cumulative pipeline store outcomes since start-up (the [pipecache] line reports 10 s windows). +struct PipelineStoreCounters { + std::uint64_t hits = 0; + std::uint64_t misses = 0; + std::uint64_t uncached = 0; + std::uint64_t evicted = 0; + std::uint64_t cached = 0; + std::uint64_t bound = 0; +}; +PipelineStoreCounters PipelineStoreTotals(); +// The pipeline store's key of a configuration (empty when the configuration gets a private pipeline). +void BuildPipelineStoreKey(std::vector& key, const Context& context, const State& state, const VertexInputLayout& vertexInput, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout); struct DescribedSetLayouts { PipelineSetLayouts layouts; std::vector mainKey; diff --git a/core/libs/prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp b/core/libs/prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp new file mode 100644 index 0000000000..c2cf90476e --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp @@ -0,0 +1,104 @@ +#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_PIPELINEJOURNAL_HPP +#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_PIPELINEJOURNAL_HPP + +#include "prx/libSceAgcDriver/Graphics/include/Context.hpp" +#include "prx/libSceAgcDriver/Graphics/include/Shaders.hpp" +#include "prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp" +#include "prx/libSceAgcDriver/Graphics/include/State.hpp" +#include "prx/libSceAgcDriver/Graphics/include/VertexInput.hpp" +#include "Recompiler.hpp" +#include +#include +#include +#include +#include +#include +#include + +namespace AgcDriver::Graphics { + +// What a graphics pipeline is built from besides the device: the stage results (named by their +// run-stable identity), the vertex input layout, the descriptor set layout keys and the state +// fields the Pipeline constructor reads, with every other State field cleared +// (CanonicalPipelineState). The pipeline store keys on exactly this, and the journal records it so +// the next run builds the same VkPipeline before the title asks for it. A generated stage (the +// rect-list control and evaluation pair, the barycentric geometry stage) is built from the vertex +// and fragment results and the device's limits; its identity is in the key like any other stage's, +// so a replayed pipeline built from another device's generated module is missed, not used. +struct GraphicsPipelineRecipe { + struct Stage { + ShaderRecompiler::ShaderStage stage; + ShaderRecompiler::PipelineIdentity identity; + std::uint32_t pushConstantOffset; + bool generated; + }; + VkImageLayout attachmentLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL; + std::vector stages; + VertexInputLayout vertexInput; + std::vector mainLayoutKey; + std::vector fragmentLayoutKey; + bool splitSets = false; + State state; +}; + +[[nodiscard]] State CanonicalPipelineState(const State& state); +[[nodiscard]] bool GeneratedPipelineStage(const State& state, ShaderRecompiler::ShaderStage stage); +[[nodiscard]] GraphicsPipelineRecipe MakeGraphicsPipelineRecipe(const State& state, const VertexInputLayout& vertexInput, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout); +[[nodiscard]] std::vector EncodeGraphicsPipelineRecipe(const GraphicsPipelineRecipe& recipe); +[[nodiscard]] bool DecodeGraphicsPipelineRecipe(std::span bytes, GraphicsPipelineRecipe& recipe); + +struct PipelineJournalCounters { + std::uint64_t loadedRecipes = 0; + std::uint64_t loadedModules = 0; + std::uint64_t loadedBytes = 0; + std::uint64_t droppedRecords = 0; + double loadMs = 0; + std::uint64_t replayedGraphics = 0; + std::uint64_t replayedCompute = 0; + std::uint64_t replayPresent = 0; + std::uint64_t replaySkipped = 0; + std::uint64_t replayRejected = 0; + std::uint64_t replayFailed = 0; + double replayMs = 0; + std::uint32_t replayThreads = 0; + std::uint64_t recordedGraphics = 0; + std::uint64_t recordedCompute = 0; + std::uint64_t recordedModules = 0; + std::uint64_t writtenBytes = 0; + std::uint64_t writeFailures = 0; +}; + +// A decimal switch of the pipeline store or journal, or its default when unset; anything else throws. +[[nodiscard]] std::uint64_t PipelineSwitch(const char* name, std::uint64_t fallback); +[[nodiscard]] bool PipelineJournalEnabled(); +// True on a replay thread while it builds an entry: what it builds is already in the file. +[[nodiscard]] bool PipelineJournalReplaying(); +[[nodiscard]] std::filesystem::path PipelineJournalPath(); +[[nodiscard]] PipelineJournalCounters PipelineJournalTotals(); +void RecordGraphicsPipeline(const GraphicsPipelineRecipe& recipe, std::span shaders); +void RecordComputePipeline(const ShaderRecompiler::RecompileResult& shader, bool pushConstants); +void FlushPipelineJournal(); +void ReportPipelineJournal(); + +// Cancel stops the threads taking further entries (the pipeline each is building completes); +// Wait joins them. A device cancels its replay before it is replaced or destroyed. +class PipelineJournalReplay { +public: + virtual ~PipelineJournalReplay() = default; + virtual void Cancel() = 0; + virtual void Wait() = 0; + [[nodiscard]] virtual bool Done() const = 0; +}; + +struct PipelineJournalHooks { + std::function compute; +}; + +// The journal belongs to the device that wrote it, named by the same properties as the driver's +// pipeline cache file; a replay for another device starts the file over. +[[nodiscard]] std::uint64_t PipelineJournalDeviceFingerprint(const VkPhysicalDeviceProperties& properties); +[[nodiscard]] std::shared_ptr StartPipelineJournalReplay(const Context& context, std::uint64_t deviceFingerprint, PipelineJournalHooks hooks); + +} + +#endif diff --git a/core/libs/prx/libSceAgcDriver/Graphics/include/PipelineLibrary.hpp b/core/libs/prx/libSceAgcDriver/Graphics/include/PipelineLibrary.hpp index b2ca74e468..a40cf15479 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/include/PipelineLibrary.hpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/include/PipelineLibrary.hpp @@ -4,6 +4,7 @@ #include "prx/libSceAgcDriver/Graphics/include/Context.hpp" #include #include +#include #include #include #include @@ -32,7 +33,13 @@ struct OptimizedPipeline { PFN_vkDestroyPipeline destroy = nullptr; std::atomic handle{VK_NULL_HANDLE}; std::atomic released{false}; + // The relink of a pipeline the journal replay built waits for the first draw that binds it + // (Schedule), so the relink thread serves the title's own pipelines first and a replayed + // pipeline the title never draws is never relinked. + std::atomic deferred{false}; + std::function relink; void Release() noexcept; + void Schedule(); }; VkPipeline LinkPipelineFromLibraries(const Context& context, const VkGraphicsPipelineCreateInfo& info, const VkPipelineRenderingCreateInfoKHR& rendering, const VkPipelineLayoutCreateInfo& layout, const PipelineLibraryKeys& keys, std::shared_ptr* optimized = nullptr, bool partsOnly = false, std::span modules = {}); diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp index 9fd5790a24..ec0251bcac 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp @@ -2,6 +2,7 @@ #include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp" #include "prx/libSceAgcDriver/Graphics/include/Pipeline.hpp" #include "prx/libSceAgcDriver/Graphics/include/PipelineLibrary.hpp" +#include "prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp" #include "prx/libSceAgcDriver/Graphics/include/DepthSurface.hpp" #include "prx/libSceAgcDriver/Graphics/include/PipelineSpecialization.hpp" #include "prx/libSceAgcDriver/Graphics/include/VertexInput.hpp" @@ -193,10 +194,12 @@ bool libraryKeys(PipelineLibraryKeys& keys, const Context& context, const State& if (!context.graphicsPipelineLibrary || context.pipelineExecutableInfo || state.depthBiasPerFace || state.rectList || SplitsAlternateAlpha(state)) return false; keys = {}; for (const auto& shader : shaders) { - if (shader.stage == Stage::Geometry || shader.program->PipelineVariantId() == 0) return false; + const auto identity = shader.program->Identity(); + if (shader.stage == Stage::Geometry || identity.Empty()) return false; auto& key = shader.stage == Stage::Fragment ? keys.fragmentShader : keys.preRasterization; appendKey(key, shader.stage); - appendKey(key, shader.program->PipelineVariantId()); + appendKey(key, identity.low); + appendKey(key, identity.high); appendKey(key, shader.pushConstantOffset); if (resources.split) { auto& sets = shader.stage == Stage::Fragment ? keys.fragmentSets : keys.preRasterizationSets; @@ -711,6 +714,7 @@ void Pipeline::Begin(VkCommandBuffer commands, const Framebuffer& framebuffer, V } void Pipeline::Continue(VkCommandBuffer commands, const State& state) const { + if (optimized != nullptr) optimized->Schedule(); const auto best = optimized != nullptr ? optimized->handle.load() : VK_NULL_HANDLE; context.Resolved(&DeviceFunctions::cmdBindPipeline, "vkCmdBindPipeline")(commands, VK_PIPELINE_BIND_POINT_GRAPHICS, best != VK_NULL_HANDLE ? best : pipeline); context.Resolved(&DeviceFunctions::cmdSetViewport, "vkCmdSetViewport")(commands, 0, 1, &state.viewport); @@ -779,25 +783,27 @@ void append(std::vector& key, const TValue& value) { key.insert(key.end(), bytes.begin(), bytes.end()); } -// Everything the Pipeline objects are built from, or empty when a stage's result has no variant id -// (the recompiler could not identify it, so nothing else may share its pipeline). The rect-list -// control and evaluation stages are generated from the vertex and fragment results, which the key -// already names, so they carry no id of their own. +// Everything the Pipeline objects are built from, or empty when a stage's result has no module +// (nothing else may share its pipeline). The stage identities and the state fields are the same on +// every run (CanonicalPipelineState keeps exactly the fields read here), which lets the journal's +// replay insert an entry a later draw finds; a generated stage (rect-list pair, geometry) is keyed +// by its identity like the others, so a replayed pipeline built from another device's generated +// module is missed rather than used. void pipelineKey(std::vector& key, const Context& context, const State& state, const VertexInputLayout& input, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout) { - using Stage = ShaderRecompiler::ShaderStage; key.clear(); append(key, context.device); append(key, attachmentLayout); append(key, shaders.size()); for (const auto& shader : shaders) { Require(shader.program != nullptr, "missing compiled shader"); - const bool generated = (state.rectList && (shader.stage == Stage::TessellationControl || shader.stage == Stage::TessellationEvaluation)) || shader.stage == Stage::Geometry; - if (!generated && shader.program->PipelineVariantId() == 0) { + const auto identity = shader.program->Identity(); + if (identity.Empty()) { key.clear(); return; } append(key, shader.stage); - append(key, generated ? std::uint64_t{0} : shader.program->PipelineVariantId()); + append(key, identity.low); + append(key, identity.high); // Where the stage's push constants sit in the block (AssemblePushConstants). append(key, shader.pushConstantOffset); } @@ -840,18 +846,23 @@ void pipelineKey(std::vector& key, const Context& context, const Stat append(key, state.sampleLocations); } append(key, state.colors.size()); - for (const auto& color : state.colors) append(key, color.format); + for (const auto& color : state.colors) { + append(key, color.format); + append(key, color.componentMapping); + } if (state.blends.size() != state.colors.size()) { for (const auto& color : state.colors) append(key, color.exportIndex); } append(key, state.depth.has_value()); if (state.depth) { append(key, state.depth->format); + append(key, state.depth->samples); append(key, state.depthTest); append(key, state.depthWrite); append(key, state.depthCompare); append(key, state.depthBoundsTest); append(key, state.depthBias); + append(key, state.depthBias && state.depthBiasClamp != 0.0f); append(key, state.depthBiasPerFace); append(key, state.stencilTest); auto front = state.stencilFront; @@ -903,7 +914,6 @@ struct PipelineStore { std::shared_ptr pipeline; }; std::mutex mutex; - std::vector scratchKey; // Least recently used first. std::list entries; std::unordered_map::iterator> index; @@ -911,6 +921,7 @@ struct PipelineStore { std::uint64_t misses = 0; std::uint64_t uncached = 0; std::uint64_t evicted = 0; + PipelineStoreCounters totals; std::chrono::steady_clock::time_point lastReport = std::chrono::steady_clock::now(); }; @@ -936,6 +947,11 @@ std::list::iterator abandon(PipelineStore& store, std::lis return store.entries.erase(it); } +std::size_t storeBound() { + static const auto bound = static_cast(std::max(1u, PipelineSwitch("APS5_PIPELINE_STORE_ENTRIES", 1024u))); + return bound; +} + void reportPipelines(PipelineStore& store) { static const bool profile = std::getenv("APS5_PROFILE_DRAW") != nullptr; if (!profile) return; @@ -953,57 +969,116 @@ std::shared_ptr CachedPipeline(const Context& context, const State& st return CachedPipeline(context, state, vertexInput, resources.SetLayouts(), shaders, attachmentLayout); } -std::shared_ptr CachedPipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout) { +std::shared_ptr CachedPipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout, bool* created) { static const bool disabled = std::getenv("APS5_NO_PIPELINE_CACHE") != nullptr; + if (created != nullptr) *created = false; if (disabled) return std::make_shared(context, state, vertexInput, resources, shaders, attachmentLayout); auto& store = Pipelines(); - std::lock_guard lock(store.mutex); - reportPipelines(store); - auto& key = store.scratchKey; + thread_local std::vector key; pipelineKey(key, context, state, vertexInput, resources, shaders, attachmentLayout); if (key.empty()) { + std::lock_guard lock(store.mutex); ++store.uncached; + ++store.totals.uncached; return std::make_shared(context, state, vertexInput, resources, shaders, attachmentLayout); } const auto hash = hashKey(key); - if (const auto found = store.index.find(hash); found != store.index.end()) { + // The store's entry of the key, or nullptr with `found` telling a miss (the key is new) from a + // collision (another configuration owns the hash, so this one stays private). + const auto lookup = [&](bool& collision) -> std::shared_ptr { + collision = false; + const auto found = store.index.find(hash); + if (found == store.index.end()) return nullptr; const auto it = found->second; - if (it->key == key) { - // The key names the device handle, which the loader may reuse for a device created after - // this one was destroyed without ClearCachedPipelines: such an entry is a miss. - if (alive(*it, context)) { - ++store.hits; - store.entries.splice(store.entries.end(), store.entries, it); - return it->pipeline; - } - abandon(store, it); - } else { + if (it->key != key) { + collision = true; + return nullptr; + } + // The key names the device handle, which the loader may reuse for a device created after + // this one was destroyed without ClearCachedPipelines: such an entry is a miss. + if (alive(*it, context)) { + store.entries.splice(store.entries.end(), store.entries, it); + return it->pipeline; + } + abandon(store, it); + return nullptr; + }; + { + std::lock_guard lock(store.mutex); + reportPipelines(store); + bool collision = false; + if (auto pipeline = lookup(collision)) { + ++store.hits; + ++store.totals.hits; + return pipeline; + } + if (collision) { // A different configuration with the same hash keeps the resident entry; this one stays private. ++store.uncached; + ++store.totals.uncached; return std::make_shared(context, state, vertexInput, resources, shaders, attachmentLayout); } + ++store.misses; + ++store.totals.misses; } - ++store.misses; - // Entries of another device belong to one the driver replaced (it does so under the GpuMutex - // before any draw reaches the new device), whose objects went with it: forget them. - for (auto it = store.entries.begin(); it != store.entries.end();) { - it = alive(*it, context) ? std::next(it) : abandon(store, it); - } + // Built outside the mutex: a draw that finds its pipeline resident is not held behind another + // thread's build (the journal replay builds here too), and a device's driver compiles in + // parallel when several threads build at once. auto pipeline = std::make_shared(context, state, vertexInput, resources, shaders, attachmentLayout); - store.entries.push_back({context.device, context.bufferPool, hash, key, pipeline}); - store.index[hash] = std::prev(store.entries.end()); - constexpr std::size_t bound = 1024; - while (store.entries.size() > bound) { - // Only an entry no recorded draw still holds may go (Kept keeps its shared_ptr until the fence). - const auto victim = std::find_if(store.entries.begin(), store.entries.end(), [](const PipelineStore::Entry& entry) { return entry.pipeline.use_count() == 1; }); - if (victim == store.entries.end()) break; - store.index.erase(victim->hash); - store.entries.erase(victim); - ++store.evicted; + const auto bound = storeBound(); + const bool replaying = PipelineJournalReplaying(); + { + std::lock_guard lock(store.mutex); + bool collision = false; + if (auto raced = lookup(collision)) return raced; + if (collision) { + ++store.uncached; + ++store.totals.uncached; + return pipeline; + } + // Entries of another device belong to one the driver replaced (it does so under the GpuMutex + // before any draw reaches the new device), whose objects went with it: forget them. + for (auto it = store.entries.begin(); it != store.entries.end();) { + it = alive(*it, context) ? std::next(it) : abandon(store, it); + } + // A replayed pipeline enters at the least recently used end and never displaces a draw's: + // the title's own pipelines evict it first, and a full store takes none. + if (replaying && store.entries.size() >= bound) return pipeline; + if (replaying) { + store.entries.push_front({context.device, context.bufferPool, hash, std::vector(key), pipeline}); + store.index[hash] = store.entries.begin(); + } else { + store.entries.push_back({context.device, context.bufferPool, hash, std::vector(key), pipeline}); + store.index[hash] = std::prev(store.entries.end()); + } + while (store.entries.size() > bound) { + // Only an entry no recorded draw still holds may go (Kept keeps its shared_ptr until the fence). + const auto victim = std::find_if(store.entries.begin(), store.entries.end(), [](const PipelineStore::Entry& entry) { return entry.pipeline.use_count() == 1; }); + if (victim == store.entries.end()) break; + store.index.erase(victim->hash); + store.entries.erase(victim); + ++store.evicted; + ++store.totals.evicted; + } } + if (created != nullptr) *created = true; + if (PipelineJournalEnabled() && !PipelineJournalReplaying()) RecordGraphicsPipeline(MakeGraphicsPipelineRecipe(state, vertexInput, resources, shaders, attachmentLayout), shaders); return pipeline; } +void BuildPipelineStoreKey(std::vector& key, const Context& context, const State& state, const VertexInputLayout& vertexInput, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout) { + pipelineKey(key, context, state, vertexInput, resources, shaders, attachmentLayout); +} + +PipelineStoreCounters PipelineStoreTotals() { + auto& store = Pipelines(); + std::lock_guard lock(store.mutex); + auto totals = store.totals; + totals.cached = store.entries.size(); + totals.bound = storeBound(); + return totals; +} + void ClearCachedPipelines(VkDevice device) { auto& store = Pipelines(); std::lock_guard lock(store.mutex); diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp new file mode 100644 index 0000000000..940ff93847 --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp @@ -0,0 +1,1096 @@ +#include "prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp" +#include "prx/libSceAgcDriver/Graphics/include/Pipeline.hpp" +#include "prx/libSceAgcDriver/Execution/include/ProfileOutput.hpp" +#include "ShaderCacheDirectory.hpp" +#include "ShaderDiskCache.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#ifdef _WIN32 +#include +#else +#include +#include +#include +#endif + +namespace AgcDriver::Graphics { + +namespace { + +static_assert(sizeof(VkPipelineColorBlendAttachmentState) == 32, "VkPipelineColorBlendAttachmentState changed: update the recipe codec"); +static_assert(sizeof(VkStencilOpState) == 28, "VkStencilOpState changed: update the recipe codec"); +static_assert(sizeof(VkVertexInputBindingDescription) == 12, "VkVertexInputBindingDescription changed: update the recipe codec"); +static_assert(sizeof(VkVertexInputAttributeDescription) == 16, "VkVertexInputAttributeDescription changed: update the recipe codec"); +#if defined(__linux__) && defined(__x86_64__) && defined(__GLIBCXX__) +static_assert(sizeof(State) == 664, "State changed: update CanonicalPipelineState and the recipe codec"); +static_assert(sizeof(ShaderStages) == 100, "ShaderStages changed: update the recipe codec"); +static_assert(sizeof(ShaderRecompiler::MeshConfiguration) == 36, "MeshConfiguration changed: update the recipe codec"); +static_assert(sizeof(ShaderRecompiler::TessellationConfiguration) == 32, "TessellationConfiguration changed: update the recipe codec"); +#endif + +constexpr std::uint32_t FileMagic = 0x4c4a5041u; +// Bumped when the recipe codec, CanonicalPipelineState or the store key change in a way the +// layout guards above do not catch (a field renumbered or reinterpreted at the same size). +constexpr std::uint32_t FileFormat = 1u; +constexpr std::uint32_t ModuleRecord = 1u; +constexpr std::uint32_t GraphicsRecord = 2u; +constexpr std::uint32_t ComputeRecord = 3u; +constexpr std::uint64_t RecordAlignment = 8u; + +struct FileHeader { + std::uint32_t magic; + std::uint32_t format; + std::uint32_t runtimeAbi; + std::uint32_t reserved; + std::uint64_t sourceVersion; + std::uint64_t device; +}; +static_assert(sizeof(FileHeader) == 32 && std::is_trivially_copyable_v); + +struct RecordHeader { + std::uint32_t kind; + std::uint32_t reserved; + std::uint64_t payloadBytes; + std::uint64_t payloadHash; +}; +static_assert(sizeof(RecordHeader) == 24 && std::is_trivially_copyable_v); + +class Writer { +public: + explicit Writer(std::vector& out) : out(out) {} + + template + void Raw(const TValue& value) requires std::is_trivially_copyable_v { + const auto offset = out.size(); + out.resize(offset + sizeof(value)); + std::memcpy(out.data() + offset, &value, sizeof(value)); + } + + void Bool(bool value) { Raw(value ? 1u : 0u); } + void U32(std::uint32_t value) { Raw(value); } + void U64(std::uint64_t value) { Raw(value); } + + template + void Raws(std::span values) requires std::is_trivially_copyable_v { + U64(values.size()); + for (const auto& value : values) Raw(value); + } + +private: + std::vector& out; +}; + +class Reader { +public: + explicit Reader(std::span bytes) : bytes(bytes) {} + + [[nodiscard]] bool Ok() const { return ok; } + [[nodiscard]] bool Done() const { return ok && position == bytes.size(); } + void Fail() { ok = false; } + + template + [[nodiscard]] TValue Raw() requires std::is_trivially_copyable_v { + TValue value{}; + if (!ok || bytes.size() - position < sizeof(value)) { + ok = false; + return value; + } + std::memcpy(&value, bytes.data() + position, sizeof(value)); + position += sizeof(value); + return value; + } + + [[nodiscard]] bool Bool() { + const auto value = Raw(); + if (value > 1u) ok = false; + return ok && value == 1u; + } + + [[nodiscard]] std::uint32_t U32() { return Raw(); } + [[nodiscard]] std::uint64_t U64() { return Raw(); } + + template + void Raws(std::vector& values, std::size_t limit) requires std::is_trivially_copyable_v { + const auto count = U64(); + if (!ok || count > limit || count > (bytes.size() - position) / sizeof(TValue)) { + ok = false; + values.clear(); + return; + } + values.resize(static_cast(count)); + for (auto& value : values) value = Raw(); + } + +private: + std::span bytes; + std::size_t position = 0; + bool ok = true; +}; + +template +void encodeOptional(Writer& writer, const std::optional& value, void (*encode)(Writer&, const TValue&)) { + writer.Bool(value.has_value()); + if (value) encode(writer, *value); +} + +void encodeMesh(Writer& writer, const ShaderRecompiler::MeshConfiguration& mesh) { + for (const auto value : {mesh.inputPrimitive, mesh.primitivesPerGroup, mesh.verticesPerGroup, mesh.maxVertices, mesh.maxPrimitives, mesh.threadsPerGroup, mesh.ldsSizeDwords, mesh.provokingVertex, mesh.esgsItemSize}) writer.U32(value); +} + +void decodeMesh(Reader& reader, ShaderRecompiler::MeshConfiguration& mesh) { + for (auto* value : {&mesh.inputPrimitive, &mesh.primitivesPerGroup, &mesh.verticesPerGroup, &mesh.maxVertices, &mesh.maxPrimitives, &mesh.threadsPerGroup, &mesh.ldsSizeDwords, &mesh.provokingVertex, &mesh.esgsItemSize}) *value = reader.U32(); +} + +void encodeTessellation(Writer& writer, const ShaderRecompiler::TessellationConfiguration& tessellation) { + for (const auto value : {tessellation.inputControlPoints, tessellation.outputControlPoints, tessellation.domain, tessellation.partitioning, tessellation.outputTopology, tessellation.patchesPerGroup, tessellation.primitiveGroupSize}) writer.U32(value); + writer.Bool(tessellation.breakAtInstance); +} + +void decodeTessellation(Reader& reader, ShaderRecompiler::TessellationConfiguration& tessellation) { + for (auto* value : {&tessellation.inputControlPoints, &tessellation.outputControlPoints, &tessellation.domain, &tessellation.partitioning, &tessellation.outputTopology, &tessellation.patchesPerGroup, &tessellation.primitiveGroupSize}) *value = reader.U32(); + tessellation.breakAtInstance = reader.Bool(); +} + +void encodeState(Writer& writer, const State& state) { + writer.U32(static_cast(state.stages.path)); + writer.U32(state.stages.vertexWaveSize); + writer.U32(state.stages.fragmentWaveSize); + encodeOptional(writer, state.stages.mesh, encodeMesh); + encodeOptional(writer, state.stages.tessellation, encodeTessellation); + writer.Bool(state.depth.has_value()); + if (state.depth) { + writer.U32(static_cast(state.depth->format)); + writer.U32(state.depth->samples); + } + writer.Bool(state.depthTest); + writer.Bool(state.depthWrite); + writer.U32(static_cast(state.depthCompare)); + writer.Bool(state.depthBoundsTest); + writer.Bool(state.depthBias); + writer.Raw(state.depthBiasClamp); + writer.Bool(state.depthBiasPerFace); + writer.Bool(state.stencilTest); + writer.Raw(state.stencilFront); + writer.Raw(state.stencilBack); + writer.U64(state.colors.size()); + for (const auto& color : state.colors) { + writer.U32(static_cast(color.format)); + writer.U32(color.exportIndex); + writer.Raw(color.componentMapping); + } + writer.Raws(std::span(state.blends)); + for (const auto value : state.blendConstants) writer.Raw(value); + writer.Bool(state.hasColorTarget); + writer.Bool(state.rectList); + writer.Bool(state.dualSourceBlend); + writer.U32(static_cast(state.topology)); + writer.Bool(state.primitiveRestart); + writer.Bool(state.negativeOneToOne); + writer.Bool(state.depthClamp); + writer.U32(static_cast(state.conservativeRasterization)); + writer.U32(state.cullMode); + writer.U32(static_cast(state.frontFace)); + writer.U32(static_cast(state.provokingVertexMode)); + writer.U32(state.samples); + for (const auto value : state.sampleLocations) writer.Raw(value); + writer.Bool(state.customSampleLocations); +} + +void decodeState(Reader& reader, State& state) { + state = State{}; + state.stages.path = static_cast(reader.U32()); + state.stages.vertexWaveSize = reader.U32(); + state.stages.fragmentWaveSize = reader.U32(); + if (reader.Bool()) decodeMesh(reader, state.stages.mesh.emplace()); + if (reader.Bool()) decodeTessellation(reader, state.stages.tessellation.emplace()); + if (reader.Bool()) { + auto& depth = state.depth.emplace(); + depth.format = static_cast(reader.U32()); + depth.samples = reader.U32(); + } + state.depthTest = reader.Bool(); + state.depthWrite = reader.Bool(); + state.depthCompare = static_cast(reader.U32()); + state.depthBoundsTest = reader.Bool(); + state.depthBias = reader.Bool(); + state.depthBiasClamp = reader.Raw(); + state.depthBiasPerFace = reader.Bool(); + state.stencilTest = reader.Bool(); + state.stencilFront = reader.Raw(); + state.stencilBack = reader.Raw(); + const auto colors = reader.U64(); + if (!reader.Ok() || colors > 8u) { + reader.Fail(); + state = State{}; + return; + } + for (std::uint64_t index = 0; index < colors; ++index) { + ColorTarget color{}; + color.format = static_cast(reader.U32()); + color.exportIndex = reader.U32(); + color.componentMapping = reader.Raw(); + state.colors.push_back(color); + } + reader.Raws(state.blends, 8u); + for (auto& value : state.blendConstants) value = reader.Raw(); + state.hasColorTarget = reader.Bool(); + state.rectList = reader.Bool(); + state.dualSourceBlend = reader.Bool(); + state.topology = static_cast(reader.U32()); + state.primitiveRestart = reader.Bool(); + state.negativeOneToOne = reader.Bool(); + state.depthClamp = reader.Bool(); + state.conservativeRasterization = static_cast(reader.U32()); + state.cullMode = reader.U32(); + state.frontFace = static_cast(reader.U32()); + state.provokingVertexMode = static_cast(reader.U32()); + state.samples = reader.U32(); + for (auto& value : state.sampleLocations) value = reader.Raw(); + state.customSampleLocations = reader.Bool(); +} + +void encodeRecipe(Writer& writer, const GraphicsPipelineRecipe& recipe) { + writer.U32(static_cast(recipe.attachmentLayout)); + writer.U64(recipe.stages.size()); + for (const auto& stage : recipe.stages) { + writer.U32(static_cast(stage.stage)); + writer.U64(stage.identity.low); + writer.U64(stage.identity.high); + writer.U32(stage.pushConstantOffset); + writer.Bool(stage.generated); + } + writer.Raws(std::span(recipe.vertexInput.bindings)); + writer.Raws(std::span(recipe.vertexInput.attributes)); + writer.Raws(std::span(recipe.mainLayoutKey)); + writer.Raws(std::span(recipe.fragmentLayoutKey)); + writer.Bool(recipe.splitSets); + encodeState(writer, recipe.state); +} + +bool decodeRecipe(Reader& reader, GraphicsPipelineRecipe& recipe) { + recipe = GraphicsPipelineRecipe{}; + recipe.attachmentLayout = static_cast(reader.U32()); + const auto stages = reader.U64(); + if (!reader.Ok() || stages > 8u) return false; + for (std::uint64_t index = 0; index < stages; ++index) { + GraphicsPipelineRecipe::Stage stage{}; + stage.stage = static_cast(reader.U32()); + stage.identity.low = reader.U64(); + stage.identity.high = reader.U64(); + stage.pushConstantOffset = reader.U32(); + stage.generated = reader.Bool(); + recipe.stages.push_back(stage); + } + reader.Raws(recipe.vertexInput.bindings, 64u); + reader.Raws(recipe.vertexInput.attributes, 64u); + reader.Raws(recipe.mainLayoutKey, 4096u); + reader.Raws(recipe.fragmentLayoutKey, 4096u); + recipe.splitSets = reader.Bool(); + decodeState(reader, recipe.state); + return reader.Done(); +} + +std::uint64_t padded(std::uint64_t bytes) { + return (bytes + RecordAlignment - 1u) / RecordAlignment * RecordAlignment; +} + +bool profiling() { + static const bool profile = std::getenv("APS5_PROFILE_DRAW") != nullptr; + return profile; +} + +thread_local bool replayingHere = false; + +std::uint64_t configured(const char* name, std::uint64_t fallback) { + const char* text = std::getenv(name); + if (text == nullptr || *text == '\0') return fallback; + char* end = nullptr; + const auto value = std::strtoull(text, &end, 10); + if (end == text || *end != '\0') throw std::runtime_error(std::string("AGC graphics: ") + name + " must be a decimal integer"); + return value; +} + +// Held while the journal is read, cut or appended: another process sharing the directory (a +// second run of the same install) takes the same lock file, so its appends and this run's never +// interleave and neither cuts a record the other is writing. Taken before the journal's mutex, +// never while holding it. +class FileLock { +public: + explicit FileLock(const std::filesystem::path& path) { +#ifdef _WIN32 + handle = CreateFileW(path.c_str(), GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, nullptr, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, nullptr); + if (handle == INVALID_HANDLE_VALUE) return; + OVERLAPPED overlapped{}; + if (!LockFileEx(handle, LOCKFILE_EXCLUSIVE_LOCK, 0, 1, 0, &overlapped)) { + CloseHandle(handle); + handle = INVALID_HANDLE_VALUE; + } +#else + descriptor = ::open(path.c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0644); + if (descriptor >= 0 && ::flock(descriptor, LOCK_EX) != 0) { + ::close(descriptor); + descriptor = -1; + } +#endif + } + + ~FileLock() { +#ifdef _WIN32 + if (handle == INVALID_HANDLE_VALUE) return; + OVERLAPPED overlapped{}; + UnlockFileEx(handle, 0, 1, 0, &overlapped); + CloseHandle(handle); +#else + if (descriptor < 0) return; + ::flock(descriptor, LOCK_UN); + ::close(descriptor); +#endif + } + + FileLock(const FileLock&) = delete; + FileLock& operator=(const FileLock&) = delete; + +private: +#ifdef _WIN32 + HANDLE handle = INVALID_HANDLE_VALUE; +#else + int descriptor = -1; +#endif +}; + +// The end of the last whole record in [from, size) of the file: records another process appended +// after this one loaded the file are kept, the torn tail of a record a crashed run left is cut. +std::uint64_t validTail(const std::filesystem::path& path, std::uint64_t from, std::uint64_t size); + +struct LoadedModule { + ShaderRecompiler::ShaderStage stage; + std::shared_ptr result; +}; + +struct LoadedEntry { + std::uint32_t kind; + GraphicsPipelineRecipe graphics; + ShaderRecompiler::PipelineIdentity compute; + bool pushConstants = false; +}; + +struct LoadedJournal { + std::map modules; + std::vector entries; + std::set indexModules; + std::set indexRecords; + std::uint64_t validBytes = 0; + std::uint64_t bytes = 0; + std::uint64_t dropped = 0; + bool headerValid = false; +}; + +class Journal { +public: + static Journal& Instance() { + static auto* journal = new Journal(); + return *journal; + } + + [[nodiscard]] bool Enabled() const { return !path.empty() && !mismatched.load(std::memory_order_relaxed); } + [[nodiscard]] const std::filesystem::path& Path() const { return path; } + + // The journal belongs to one device per process: the first device to replay binds its + // fingerprint, the file header names it and a file written by another device starts over. A + // later device with another fingerprint (the window's device is the headless device's GPU, so a + // run does not see one) turns the journal off for the rest of the run. + bool Bind(std::uint64_t fingerprint) { + std::lock_guard lock(mutex); + if (!bound.load(std::memory_order_relaxed)) { + device.store(fingerprint, std::memory_order_relaxed); + bound.store(true, std::memory_order_release); + return true; + } + if (device.load(std::memory_order_relaxed) == fingerprint) return true; + if (!mismatched.exchange(true)) std::fprintf(stderr, "[pipeline-journal] %s belongs to another device; the journal is off for this run\n", path.string().c_str()); + return false; + } + + // Reads the whole file (modules keep views into the buffer, which lives as long as they do), + // fills the dedupe index, and remembers how many bytes parsed so a torn tail is cut before the + // next append. + LoadedJournal Load() { + LoadedJournal loaded; + if (!Enabled() || !bound.load(std::memory_order_acquire)) return loaded; + const auto started = std::chrono::steady_clock::now(); + auto file = std::make_shared>(); + bool read = false; + { + FileLock fileLock(lockPath); + read = ShaderRecompiler::ReadWholeFile(path, *file); + } + if (read) parse(*file, file, loaded, true); + const auto ms = std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + { + std::lock_guard lock(mutex); + std::error_code error; + if (!indexed && (read || !std::filesystem::exists(path, error))) { + indexed = true; + validBytes = loaded.validBytes; + headerValid = loaded.headerValid; + writtenModules = loaded.indexModules; + writtenRecords = loaded.indexRecords; + } + counters.loadedRecipes.store(loaded.entries.size(), std::memory_order_relaxed); + counters.loadedModules.store(loaded.modules.size(), std::memory_order_relaxed); + counters.loadedBytes.store(loaded.bytes, std::memory_order_relaxed); + counters.droppedRecords.store(loaded.dropped, std::memory_order_relaxed); + loadMs.store(ms, std::memory_order_relaxed); + } + if (loaded.bytes != 0 || loaded.dropped != 0) std::fprintf(stderr, "[pipeline-journal] loaded %zu pipelines and %zu modules (%.1f MiB) from %s in %.0f ms%s\n", loaded.entries.size(), loaded.modules.size(), static_cast(loaded.bytes) / (1024.0 * 1024.0), path.string().c_str(), ms, !loaded.headerValid ? "; written by another build or device, starting over" : loaded.dropped != 0 ? "; the file's tail was unreadable and is dropped" : ""); + return loaded; + } + + void RecordGraphics(const GraphicsPipelineRecipe& recipe, std::span shaders) { + if (!Enabled() || !bound.load(std::memory_order_acquire)) return; + Pending pending; + pending.kind = GraphicsRecord; + for (std::size_t index = 0; index < shaders.size() && index < recipe.stages.size(); ++index) { + Require(shaders[index].program != nullptr, "a recorded pipeline stage has no program"); + pending.modules.push_back({recipe.stages[index].identity, recipe.stages[index].stage, std::make_shared(*shaders[index].program)}); + } + Writer writer(pending.payload); + encodeRecipe(writer, recipe); + enqueue(std::move(pending)); + } + + void RecordCompute(const ShaderRecompiler::RecompileResult& shader, bool pushConstants) { + if (!Enabled() || !bound.load(std::memory_order_acquire)) return; + const auto identity = shader.Identity(); + if (identity.Empty()) return; + Pending pending; + pending.kind = ComputeRecord; + pending.modules.push_back({identity, ShaderRecompiler::ShaderStage::Compute, std::make_shared(shader)}); + Writer writer(pending.payload); + writer.U64(identity.low); + writer.U64(identity.high); + writer.Bool(pushConstants); + enqueue(std::move(pending)); + } + + void Flush() { + std::unique_lock lock(mutex); + idle.wait(lock, [&] { return queue.empty() && !writing; }); + } + + PipelineJournalCounters Totals() const { + PipelineJournalCounters totals; + totals.loadedRecipes = counters.loadedRecipes.load(std::memory_order_relaxed); + totals.loadedModules = counters.loadedModules.load(std::memory_order_relaxed); + totals.loadedBytes = counters.loadedBytes.load(std::memory_order_relaxed); + totals.droppedRecords = counters.droppedRecords.load(std::memory_order_relaxed); + totals.loadMs = loadMs.load(std::memory_order_relaxed); + totals.recordedGraphics = counters.recordedGraphics.load(std::memory_order_relaxed); + totals.recordedCompute = counters.recordedCompute.load(std::memory_order_relaxed); + totals.recordedModules = counters.recordedModules.load(std::memory_order_relaxed); + totals.writtenBytes = counters.writtenBytes.load(std::memory_order_relaxed); + totals.writeFailures = counters.writeFailures.load(std::memory_order_relaxed); + return totals; + } + + struct Counters { + std::atomic loadedRecipes{0}, loadedModules{0}, loadedBytes{0}, droppedRecords{0}; + std::atomic recordedGraphics{0}, recordedCompute{0}, recordedModules{0}, writtenBytes{0}, writeFailures{0}; + }; + +private: + struct PendingModule { + ShaderRecompiler::PipelineIdentity identity; + ShaderRecompiler::ShaderStage stage; + std::shared_ptr result; + }; + + struct Pending { + std::uint32_t kind = 0; + std::vector modules; + std::vector payload; + }; + + Journal() { + const char* off = std::getenv("APS5_NO_PIPELINE_JOURNAL"); + if ((off != nullptr && std::strcmp(off, "0") != 0) || !ShaderRecompiler::ShaderDiskCache::Enabled()) return; + path = ShaderRecompiler::ShaderDiskCache::EntryDirectory() / "pipelines.journal"; + lockPath = ShaderRecompiler::ShaderDiskCache::EntryDirectory() / "pipelines.journal.lock"; + maxBytes = configured("APS5_PIPELINE_JOURNAL_MAX_MIB", 1024u) << 20u; + std::thread([this] { run(); }).detach(); + } + + // Parses the file: with `loaded` set every record is decoded, otherwise only the index (the + // module identities and record hashes) is built. Stops at the first record that does not + // parse; the bytes before it stay valid. + void parse(const std::vector& file, const std::shared_ptr>& owner, LoadedJournal& loaded, bool decode) { + std::uint64_t valid = 0; + bool header = false; + std::uint64_t dropped = 0; + const auto bytes = std::span(file); + if (bytes.size() >= sizeof(FileHeader)) { + FileHeader fileHeader; + std::memcpy(&fileHeader, bytes.data(), sizeof(fileHeader)); + header = fileHeader.magic == FileMagic && fileHeader.format == FileFormat && fileHeader.runtimeAbi == ShaderRecompiler::RuntimeAbi::Version && fileHeader.sourceVersion == ShaderRecompiler::ShaderDiskCache::SourceVersion() && fileHeader.device == device.load(std::memory_order_relaxed); + } + if (header) { + valid = padded(sizeof(FileHeader)); + std::uint64_t cursor = valid; + while (cursor + sizeof(RecordHeader) <= bytes.size()) { + RecordHeader record; + std::memcpy(&record, bytes.data() + cursor, sizeof(record)); + const auto payloadOffset = cursor + sizeof(RecordHeader); + if (record.payloadBytes > bytes.size() - payloadOffset) break; + const auto end = padded(payloadOffset + record.payloadBytes); + if (end > bytes.size()) break; + const auto payload = bytes.subspan(static_cast(payloadOffset), static_cast(record.payloadBytes)); + if (ShaderRecompiler::HashBytes(payload) != record.payloadHash) break; + if (record.kind == ModuleRecord) { + if (payload.size() < 20u) break; + ShaderRecompiler::PipelineIdentity identity; + std::memcpy(&identity.low, payload.data(), sizeof(identity.low)); + std::memcpy(&identity.high, payload.data() + 8, sizeof(identity.high)); + std::uint32_t stage = 0; + std::memcpy(&stage, payload.data() + 16, sizeof(stage)); + bool decoded = !decode; + if (decode && !loaded.indexModules.contains(identity)) { + auto result = std::make_shared(); + decoded = ShaderRecompiler::ShaderDiskCache::DecodeResult(payload.subspan(20), *result, owner) && result->Identity() == identity; + if (decoded) loaded.modules.emplace(identity, LoadedModule{static_cast(stage), std::move(result)}); + else ++dropped; + } + if (decoded) loaded.indexModules.insert(identity); + } else if (record.kind == GraphicsRecord || record.kind == ComputeRecord) { + if (decode && !loaded.indexRecords.contains(record.payloadHash)) { + LoadedEntry entry; + entry.kind = record.kind; + bool decoded = false; + if (record.kind == GraphicsRecord) { + Reader reader(payload); + decoded = decodeRecipe(reader, entry.graphics); + } else { + Reader reader(payload); + entry.compute.low = reader.U64(); + entry.compute.high = reader.U64(); + entry.pushConstants = reader.Bool(); + decoded = reader.Done(); + } + if (decoded) loaded.entries.push_back(std::move(entry)); + else ++dropped; + } + loaded.indexRecords.insert(record.payloadHash); + } else { + break; + } + valid = end; + cursor = end; + } + } + if (valid < bytes.size()) ++dropped; + loaded.validBytes = valid; + loaded.bytes = bytes.size(); + loaded.dropped = dropped; + loaded.headerValid = header; + } + + // The index the writer dedupes against, replaced from the file: what it holds and where it + // ends. Under the file lock. False when the file exists but could not be read, so nothing is + // written until it can be (the next record tries again) and the file is never started over for + // a read failure. + bool indexLocked() { + std::vector file; + LoadedJournal index; + std::error_code error; + const bool exists = std::filesystem::exists(path, error); + if (exists && !ShaderRecompiler::ReadWholeFile(path, file)) return false; + if (exists) parse(file, nullptr, index, false); + std::lock_guard lock(mutex); + indexed = true; + validBytes = index.validBytes; + headerValid = index.headerValid; + writtenModules = std::move(index.indexModules); + writtenRecords = std::move(index.indexRecords); + return true; + } + + bool ensureIndexed() { + { + std::lock_guard lock(mutex); + if (indexed) return true; + } + FileLock fileLock(lockPath); + { + std::lock_guard lock(mutex); + if (indexed) return true; + } + return indexLocked(); + } + + void enqueue(Pending pending) { + { + std::lock_guard lock(mutex); + queue.push_back(std::move(pending)); + } + wake.notify_one(); + } + + void run() { + for (;;) { + Pending pending; + { + std::unique_lock lock(mutex); + wake.wait(lock, [&] { return !queue.empty(); }); + pending = std::move(queue.front()); + queue.pop_front(); + writing = true; + } + try { + write(pending); + } catch (const std::exception& error) { + counters.writeFailures.fetch_add(1, std::memory_order_relaxed); + static std::atomic reported{0}; + if (reported.fetch_add(1) < 4) std::fprintf(stderr, "[pipeline-journal] cannot write %s: %s\n", path.string().c_str(), error.what()); + } + { + std::lock_guard lock(mutex); + writing = false; + if (queue.empty()) idle.notify_all(); + } + } + } + + // One record per module the file lacks, then the pipeline record; a record whose payload the + // file already holds is skipped. The file is opened for appending per batch, after the torn + // tail of a crashed run (bytes past the last parsed record) was cut. + void write(const Pending& pending) { + if (!ensureIndexed()) throw std::runtime_error("the journal could not be read"); + for (unsigned attempt = 0; attempt < 2; ++attempt) { + if (append(pending)) return; + } + throw std::runtime_error("the journal keeps changing under this run"); + } + + // One attempt: false when the file was cut or replaced by someone else since it was indexed + // (it is indexed again, and the batch is rebuilt against that index). + bool append(const Pending& pending) { + const auto recordHash = ShaderRecompiler::HashBytes(pending.payload); + std::vector batch; + std::vector newModules; + { + std::lock_guard lock(mutex); + if (writtenRecords.contains(recordHash)) return true; + for (const auto& module : pending.modules) { + if (!writtenModules.contains(module.identity) && std::ranges::find(newModules, module.identity) == newModules.end()) newModules.push_back(module.identity); + } + } + const auto appendRecord = [&](std::uint32_t kind, std::span payload) { + const RecordHeader header{kind, 0u, payload.size(), ShaderRecompiler::HashBytes(payload)}; + Writer writer(batch); + writer.Raw(header); + batch.insert(batch.end(), payload.begin(), payload.end()); + batch.resize(static_cast(padded(batch.size()))); + }; + for (const auto& identity : newModules) { + const auto module = std::ranges::find_if(pending.modules, [&](const PendingModule& candidate) { return candidate.identity == identity; }); + std::vector payload; + Writer writer(payload); + writer.U64(identity.low); + writer.U64(identity.high); + writer.U32(static_cast(module->stage)); + ShaderRecompiler::ShaderDiskCache::EncodeResult(*module->result, payload); + appendRecord(ModuleRecord, payload); + } + appendRecord(pending.kind, pending.payload); + std::error_code error; + std::filesystem::create_directories(path.parent_path(), error); + FileLock fileLock(lockPath); + bool fresh = false; + { + std::lock_guard lock(mutex); + fresh = !headerValid; + if (!fresh) { + const auto size = std::filesystem::file_size(path, error); + if (error) fresh = true; + else if (size > validBytes) { + const auto tail = validTail(path, validBytes, size); + if (tail < size) std::filesystem::resize_file(path, tail, error); + if (error) fresh = true; + else validBytes = tail; + } + } + if (!fresh && validBytes + batch.size() > maxBytes) { + static std::atomic reported{false}; + if (!reported.exchange(true)) std::fprintf(stderr, "[pipeline-journal] %s reached APS5_PIPELINE_JOURNAL_MAX_MIB; later pipelines are not recorded\n", path.string().c_str()); + counters.writeFailures.fetch_add(1, std::memory_order_relaxed); + return true; + } + } + if (!fresh) { + const auto size = std::filesystem::file_size(path, error); + if (!error && size < validBytes) { + if (!indexLocked()) throw std::runtime_error("the journal could not be read"); + return false; + } + } + std::vector prefix; + if (fresh) { + const FileHeader header{FileMagic, FileFormat, ShaderRecompiler::RuntimeAbi::Version, 0u, ShaderRecompiler::ShaderDiskCache::SourceVersion(), device.load(std::memory_order_relaxed)}; + Writer writer(prefix); + writer.Raw(header); + prefix.resize(static_cast(padded(prefix.size()))); + } + std::ofstream file(path, fresh ? std::ios::binary | std::ios::trunc : std::ios::binary | std::ios::app); + if (!file) throw std::runtime_error("cannot open the journal for writing"); + if (!prefix.empty()) file.write(reinterpret_cast(prefix.data()), static_cast(prefix.size())); + file.write(reinterpret_cast(batch.data()), static_cast(batch.size())); + file.flush(); + if (file.fail()) throw std::runtime_error("the journal write failed"); + std::lock_guard lock(mutex); + headerValid = true; + validBytes = (fresh ? padded(sizeof(FileHeader)) : validBytes) + batch.size(); + for (const auto& identity : newModules) writtenModules.insert(identity); + writtenRecords.insert(recordHash); + counters.recordedModules.fetch_add(newModules.size(), std::memory_order_relaxed); + if (pending.kind == GraphicsRecord) counters.recordedGraphics.fetch_add(1, std::memory_order_relaxed); + else counters.recordedCompute.fetch_add(1, std::memory_order_relaxed); + counters.writtenBytes.fetch_add(prefix.size() + batch.size(), std::memory_order_relaxed); + return true; + } + + std::filesystem::path path; + std::filesystem::path lockPath; + std::uint64_t maxBytes = 0; + std::atomic device{0}; + std::atomic bound{false}; + std::atomic mismatched{false}; + std::mutex mutex; + std::condition_variable wake; + std::condition_variable idle; + std::deque queue; + bool writing = false; + bool indexed = false; + bool headerValid = false; + std::uint64_t validBytes = 0; + std::set writtenModules; + std::set writtenRecords; + Counters counters; + std::atomic loadMs{0}; +}; + +std::uint64_t validTail(const std::filesystem::path& path, std::uint64_t from, std::uint64_t size) { + if (size <= from) return from; + std::ifstream file(path, std::ios::binary); + if (!file) return from; + file.seekg(static_cast(from)); + std::vector bytes(static_cast(size - from)); + file.read(reinterpret_cast(bytes.data()), static_cast(bytes.size())); + if (static_cast(file.gcount()) != bytes.size()) return from; + const auto span = std::span(bytes); + std::uint64_t cursor = 0; + std::uint64_t valid = 0; + while (cursor + sizeof(RecordHeader) <= span.size()) { + RecordHeader record; + std::memcpy(&record, span.data() + cursor, sizeof(record)); + const auto payloadOffset = cursor + sizeof(RecordHeader); + if (record.kind == 0 || record.kind > ComputeRecord || record.payloadBytes > span.size() - payloadOffset) break; + const auto end = padded(payloadOffset + record.payloadBytes); + if (end > span.size()) break; + if (ShaderRecompiler::HashBytes(span.subspan(static_cast(payloadOffset), static_cast(record.payloadBytes))) != record.payloadHash) break; + valid = end; + cursor = end; + } + return from + valid; +} + +struct ReplayCounters { + std::atomic graphics{0}, compute{0}, present{0}, skipped{0}, rejected{0}, failed{0}; + std::atomic threads{0}; + std::atomic ms{0}; +}; + +ReplayCounters& replayCounters() { + static ReplayCounters counters; + return counters; +} + +// Builds the journal's pipelines for one device on a pool: the first thread loads the file, every +// thread then takes the next entry. A pipeline the store already holds counts as present; an entry +// whose modules are missing or whose set layouts differ from what its modules describe is rejected. +class Replay : public PipelineJournalReplay { +public: + // The context keeps the device's buffer pool: a store entry is alive for a draw only while its + // pool is the draw's (the pool tells device instances apart), so the entries built here must + // name it. The pool is released with the replay, before the device's teardown resets it. + Replay(Context context, PipelineJournalHooks hooks, unsigned threads) : context(std::move(context)), hooks(std::move(hooks)), active(threads) { + started = std::chrono::steady_clock::now(); + replayCounters().threads.store(threads, std::memory_order_relaxed); + workers.reserve(threads); + for (unsigned index = 0; index < threads; ++index) workers.emplace_back([this] { run(); }); + } + + ~Replay() override { + Cancel(); + Wait(); + } + + void Cancel() override { cancelled.store(true, std::memory_order_relaxed); } + + void Wait() override { + for (auto& worker : workers) { + if (worker.joinable()) worker.join(); + } + workers.clear(); + } + + [[nodiscard]] bool Done() const override { return finished.load(std::memory_order_acquire); } + +private: + void run() { + std::call_once(loadOnce, [&] { + try { + auto journal = Journal::Instance().Load(); + const auto limit = configured("APS5_PIPELINE_JOURNAL_ENTRIES", 1024u); + if (journal.entries.size() > limit) journal.entries.resize(static_cast(limit)); + loaded = std::move(journal); + } catch (const std::exception& error) { + std::fprintf(stderr, "[pipeline-journal] cannot load %s: %s\n", Journal::Instance().Path().string().c_str(), error.what()); + } + }); + for (;;) { + if (cancelled.load(std::memory_order_relaxed)) break; + const auto index = cursor.fetch_add(1, std::memory_order_relaxed); + if (index >= loaded.entries.size()) break; + try { + replay(loaded.entries[index]); + } catch (const std::exception& error) { + replayCounters().failed.fetch_add(1, std::memory_order_relaxed); + static std::atomic reported{0}; + if (reported.fetch_add(1) < 8) std::fprintf(stderr, "[pipeline-journal] replay failed: %s\n", error.what()); + } + } + if (active.fetch_sub(1, std::memory_order_acq_rel) == 1) { + loaded = LoadedJournal{}; + const auto ms = std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + replayCounters().ms.store(ms, std::memory_order_relaxed); + finished.store(true, std::memory_order_release); + ReportPipelineJournal(); + } + } + + void replay(const LoadedEntry& entry) { + struct Replaying { + Replaying() { replayingHere = true; } + ~Replaying() { replayingHere = false; } + } replaying; + auto& counters = replayCounters(); + if (entry.kind == ComputeRecord) { + const auto module = loaded.modules.find(entry.compute); + if (module == loaded.modules.end() || module->second.stage != ShaderRecompiler::ShaderStage::Compute || !hooks.compute) { + counters.rejected.fetch_add(1, std::memory_order_relaxed); + return; + } + (hooks.compute(*module->second.result, entry.pushConstants) ? counters.compute : counters.present).fetch_add(1, std::memory_order_relaxed); + return; + } + std::vector shaders; + for (const auto& stage : entry.graphics.stages) { + const auto module = loaded.modules.find(stage.identity); + if (module == loaded.modules.end() || module->second.stage != stage.stage) { + counters.rejected.fetch_add(1, std::memory_order_relaxed); + return; + } + shaders.push_back({stage.stage, module->second.result.get(), stage.pushConstantOffset}); + } + if (shaders.empty() || context.descriptorCache == nullptr) { + counters.rejected.fetch_add(1, std::memory_order_relaxed); + return; + } + const auto store = PipelineStoreTotals(); + if (store.cached >= store.bound) { + counters.skipped.fetch_add(1, std::memory_order_relaxed); + return; + } + const auto layouts = DescribeSetLayouts(context, shaders); + if (!std::ranges::equal(layouts.mainKey, entry.graphics.mainLayoutKey) || !std::ranges::equal(layouts.fragmentKey, entry.graphics.fragmentLayoutKey) || layouts.layouts.split != entry.graphics.splitSets) { + counters.rejected.fetch_add(1, std::memory_order_relaxed); + return; + } + bool created = false; + static_cast(CachedPipeline(context, entry.graphics.state, entry.graphics.vertexInput, layouts.View(), shaders, entry.graphics.attachmentLayout, &created)); + (created ? counters.graphics : counters.present).fetch_add(1, std::memory_order_relaxed); + } + + Context context; + PipelineJournalHooks hooks; + std::vector workers; + std::once_flag loadOnce; + LoadedJournal loaded; + std::atomic cursor{0}; + std::atomic active{0}; + std::atomic finished{false}; + std::atomic cancelled{false}; + std::chrono::steady_clock::time_point started; +}; + +} + +State CanonicalPipelineState(const State& state) { + State canonical{}; + canonical.stages.path = state.stages.path; + canonical.stages.vertexWaveSize = state.stages.vertexWaveSize; + canonical.stages.fragmentWaveSize = state.stages.fragmentWaveSize; + canonical.stages.mesh = state.stages.mesh; + canonical.stages.tessellation = state.stages.tessellation; + if (state.depth) { + DepthTarget depth{}; + depth.format = state.depth->format; + depth.samples = state.depth->samples; + canonical.depth = depth; + } + canonical.depthTest = state.depthTest; + canonical.depthWrite = state.depthWrite; + canonical.depthCompare = state.depthCompare; + canonical.depthBoundsTest = state.depthBoundsTest; + canonical.depthBias = state.depthBias; + canonical.depthBiasClamp = state.depthBiasClamp; + canonical.depthBiasPerFace = state.depthBiasPerFace; + canonical.stencilTest = state.stencilTest; + canonical.stencilFront = state.stencilFront; + canonical.stencilBack = state.stencilBack; + for (const auto& color : state.colors) { + ColorTarget canonicalColor{}; + canonicalColor.format = color.format; + canonicalColor.exportIndex = color.exportIndex; + canonicalColor.componentMapping = color.componentMapping; + canonical.colors.push_back(canonicalColor); + } + canonical.blends = state.blends; + canonical.blendConstants = state.blendConstants; + canonical.hasColorTarget = state.hasColorTarget; + canonical.rectList = state.rectList; + canonical.dualSourceBlend = state.dualSourceBlend; + canonical.topology = state.topology; + canonical.primitiveRestart = state.primitiveRestart; + canonical.negativeOneToOne = state.negativeOneToOne; + canonical.depthClamp = state.depthClamp; + canonical.conservativeRasterization = state.conservativeRasterization; + canonical.cullMode = state.cullMode; + canonical.frontFace = state.frontFace; + canonical.provokingVertexMode = state.provokingVertexMode; + canonical.samples = state.samples; + canonical.sampleLocations = state.sampleLocations; + canonical.customSampleLocations = state.customSampleLocations; + return canonical; +} + +bool GeneratedPipelineStage(const State& state, ShaderRecompiler::ShaderStage stage) { + using Stage = ShaderRecompiler::ShaderStage; + return (state.rectList && (stage == Stage::TessellationControl || stage == Stage::TessellationEvaluation)) || stage == Stage::Geometry; +} + +GraphicsPipelineRecipe MakeGraphicsPipelineRecipe(const State& state, const VertexInputLayout& vertexInput, const PipelineSetLayouts& resources, std::span shaders, VkImageLayout attachmentLayout) { + GraphicsPipelineRecipe recipe; + recipe.attachmentLayout = attachmentLayout; + for (const auto& shader : shaders) { + Require(shader.program != nullptr, "missing compiled shader"); + recipe.stages.push_back({shader.stage, shader.program->Identity(), shader.pushConstantOffset, GeneratedPipelineStage(state, shader.stage)}); + } + recipe.vertexInput = vertexInput; + recipe.mainLayoutKey.assign(resources.mainKey.begin(), resources.mainKey.end()); + recipe.fragmentLayoutKey.assign(resources.fragmentKey.begin(), resources.fragmentKey.end()); + recipe.splitSets = resources.split; + recipe.state = CanonicalPipelineState(state); + return recipe; +} + +std::vector EncodeGraphicsPipelineRecipe(const GraphicsPipelineRecipe& recipe) { + std::vector bytes; + Writer writer(bytes); + encodeRecipe(writer, recipe); + return bytes; +} + +bool DecodeGraphicsPipelineRecipe(std::span bytes, GraphicsPipelineRecipe& recipe) { + Reader reader(bytes); + return decodeRecipe(reader, recipe); +} + +std::uint64_t PipelineSwitch(const char* name, std::uint64_t fallback) { + return configured(name, fallback); +} + +bool PipelineJournalEnabled() { + return Journal::Instance().Enabled(); +} + +bool PipelineJournalReplaying() { + return replayingHere; +} + +std::filesystem::path PipelineJournalPath() { + return Journal::Instance().Path(); +} + +PipelineJournalCounters PipelineJournalTotals() { + auto totals = Journal::Instance().Totals(); + const auto& replay = replayCounters(); + totals.replayedGraphics = replay.graphics.load(std::memory_order_relaxed); + totals.replayedCompute = replay.compute.load(std::memory_order_relaxed); + totals.replayPresent = replay.present.load(std::memory_order_relaxed); + totals.replaySkipped = replay.skipped.load(std::memory_order_relaxed); + totals.replayRejected = replay.rejected.load(std::memory_order_relaxed); + totals.replayFailed = replay.failed.load(std::memory_order_relaxed); + totals.replayMs = replay.ms.load(std::memory_order_relaxed); + totals.replayThreads = replay.threads.load(std::memory_order_relaxed); + return totals; +} + +void RecordGraphicsPipeline(const GraphicsPipelineRecipe& recipe, std::span shaders) { + Journal::Instance().RecordGraphics(recipe, shaders); +} + +void RecordComputePipeline(const ShaderRecompiler::RecompileResult& shader, bool pushConstants) { + Journal::Instance().RecordCompute(shader, pushConstants); +} + +void FlushPipelineJournal() { + if (Journal::Instance().Enabled()) Journal::Instance().Flush(); +} + +void ReportPipelineJournal() { + if (!profiling() || !Journal::Instance().Enabled()) return; + const auto totals = PipelineJournalTotals(); + AgcDriver::ProfilePrint_nid_no_patch("[pipeline-journal] loaded %llu pipelines, %llu modules (%.1f MiB, %.0f ms); replay on %u threads: %llu graphics and %llu compute built, %llu present, %llu skipped (store full), %llu rejected, %llu failed in %.0f ms; recorded this run: %llu graphics, %llu compute, %llu modules (%.1f MiB), %llu write failures\n", + static_cast(totals.loadedRecipes), static_cast(totals.loadedModules), static_cast(totals.loadedBytes) / (1024.0 * 1024.0), totals.loadMs, totals.replayThreads, + static_cast(totals.replayedGraphics), static_cast(totals.replayedCompute), static_cast(totals.replayPresent), static_cast(totals.replaySkipped), static_cast(totals.replayRejected), static_cast(totals.replayFailed), totals.replayMs, + static_cast(totals.recordedGraphics), static_cast(totals.recordedCompute), static_cast(totals.recordedModules), static_cast(totals.writtenBytes) / (1024.0 * 1024.0), static_cast(totals.writeFailures)); +} + +std::uint64_t PipelineJournalDeviceFingerprint(const VkPhysicalDeviceProperties& properties) { + std::array words{properties.vendorID, properties.deviceID, properties.driverVersion, 0u, 0u, 0u, 0u}; + static_assert(VK_UUID_SIZE == 16); + std::memcpy(words.data() + 3, properties.pipelineCacheUUID, VK_UUID_SIZE); + return ShaderRecompiler::HashSpirvWords(words, 0x6a6f75726e616c01ull); +} + +std::shared_ptr StartPipelineJournalReplay(const Context& context, std::uint64_t deviceFingerprint, PipelineJournalHooks hooks) { + if (!Journal::Instance().Enabled() || !Journal::Instance().Bind(deviceFingerprint)) return nullptr; + const auto threads = static_cast(configured("APS5_PIPELINE_JOURNAL_THREADS", std::max(1u, std::thread::hardware_concurrency() / 2u))); + if (threads == 0) return nullptr; + return std::make_shared(context, std::move(hooks), threads); +} + +} diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp index bd2a12ed32..7e8fec28b6 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp @@ -1,4 +1,5 @@ #include "prx/libSceAgcDriver/Graphics/include/PipelineLibrary.hpp" +#include "prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp" #include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp" #include #include @@ -204,13 +205,16 @@ VkPipeline LinkPipelineFromLibraries(const Context& context, const VkGraphicsPip break; } VkPipeline handle = VK_NULL_HANDLE; - if (partsOnly) lock.unlock(); + // Built without the store's mutex: a draw whose parts are resident is not held behind + // another thread's build, and builds from several threads (the registration prebuild, the + // journal replay) run at once; a part another thread built meanwhile is kept, this one dropped. + lock.unlock(); VkResult created; { auto pipelineCacheLock = context.LockPipelineCacheForUse(); created = context.Function("vkCreateGraphicsPipelines")(context.device, context.pipelineCache, 1, &create, nullptr, &handle); } - if (partsOnly) lock.lock(); + lock.lock(); Check(created, "vkCreateGraphicsPipelines library"); if (const auto raced = cache.find(key); raced != cache.end()) { device.destroyPipeline(context.device, handle, nullptr); @@ -238,10 +242,12 @@ VkPipeline LinkPipelineFromLibraries(const Context& context, const VkGraphicsPip VkGraphicsPipelineCreateInfo create{VK_STRUCTURE_TYPE_GRAPHICS_PIPELINE_CREATE_INFO, &linked}; create.layout = info.layout; VkPipeline pipeline = VK_NULL_HANDLE; + lock.unlock(); { auto pipelineCacheLock = context.LockPipelineCacheForUse(); Check(context.Function("vkCreateGraphicsPipelines")(context.device, context.pipelineCache, 1, &create, nullptr, &pipeline), "vkCreateGraphicsPipelines link"); } + lock.lock(); ++device.counters.linked; if (optimize && optimized != nullptr) { auto slot = std::make_shared(); @@ -252,7 +258,7 @@ VkPipeline LinkPipelineFromLibraries(const Context& context, const VkGraphicsPip const auto cache = context.pipelineCache; const auto cacheSynchronization = context.pipelineCacheSynchronization; const auto libraryLayout = fullLayout; - enqueue(store, context.device, [slot, createPipelines, cache, cacheSynchronization, libraryLayout, libraries, &store, devicePointer = &device] { + std::function relink = [slot, createPipelines, cache, cacheSynchronization, libraryLayout, libraries, &store, devicePointer = &device] { if (slot->released.load()) return; VkPipelineLibraryCreateInfoKHR linked{VK_STRUCTURE_TYPE_PIPELINE_LIBRARY_CREATE_INFO_KHR}; linked.libraryCount = static_cast(libraries.size()); @@ -272,7 +278,13 @@ VkPipeline LinkPipelineFromLibraries(const Context& context, const VkGraphicsPip } std::lock_guard lock(store.mutex); ++devicePointer->counters.optimized; - }); + }; + if (PipelineJournalReplaying()) { + slot->relink = std::move(relink); + slot->deferred.store(true, std::memory_order_release); + } else { + enqueue(store, context.device, std::move(relink)); + } } return pipeline; } @@ -282,6 +294,15 @@ void OptimizedPipeline::Release() noexcept { if (const auto pipeline = handle.exchange(VK_NULL_HANDLE)) destroy(device, pipeline, nullptr); } +void OptimizedPipeline::Schedule() { + if (!deferred.load(std::memory_order_acquire)) return; + auto& store = Libraries(); + std::lock_guard lock(store.mutex); + if (!deferred.exchange(false, std::memory_order_acq_rel)) return; + if (relink && !released.load()) enqueue(store, device, std::move(relink)); + relink = nullptr; +} + void WaitForOptimizedPipelines(VkDevice device) { auto& store = Libraries(); std::unique_lock lock(store.mutex); diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 2e1d26e11f..6a67b64204 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -3,6 +3,7 @@ #include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp" #include "prx/libSceAgcDriver/Graphics/include/Recorder.hpp" #include "prx/libSceAgcDriver/Graphics/include/Pipeline.hpp" +#include "prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp" #include "prx/libSceAgcDriver/Graphics/include/TextureDetiler.hpp" #include "prx/libSceAgcDriver/Graphics/include/VertexInput.hpp" #include "prx/libSceAgcDriver/Graphics/include/Draw.hpp" @@ -34,6 +35,8 @@ #include #include #include +#include +#include #include #include #include @@ -3626,7 +3629,15 @@ VKAPI_ATTR VkResult VKAPI_CALL mockCreatePipelineLayout(VkDevice, const VkPipeli } VKAPI_ATTR VkResult VKAPI_CALL mockCreateGraphicsPipelines(VkDevice, VkPipelineCache, std::uint32_t count, const VkGraphicsPipelineCreateInfo* infos, const VkAllocationCallbacks*, VkPipeline* pipelines) { - Require(count == 1 && (infos[0].flags & VK_PIPELINE_CREATE_LIBRARY_BIT_KHR) != 0, "mock expects a graphics pipeline library"); + Require(count == 1, "mock expects exactly one graphics pipeline per call"); + if ((infos[0].flags & VK_PIPELINE_CREATE_LIBRARY_BIT_KHR) == 0) { + const auto* linked = static_cast(infos[0].pNext); + Require(linked != nullptr && linked->sType == VK_STRUCTURE_TYPE_PIPELINE_LIBRARY_CREATE_INFO_KHR && linked->libraryCount != 0, "mock expects a graphics pipeline library or a pipeline linked from libraries"); + *pipelines = makeHandle(); + ++mock.pipelineCreateCount; + ++mock.live; + return VK_SUCCESS; + } const auto* library = static_cast(infos[0].pNext); Require(library != nullptr && library->sType == VK_STRUCTURE_TYPE_GRAPHICS_PIPELINE_LIBRARY_CREATE_INFO_EXT, "missing graphics pipeline library description"); mock.libraryLayouts[library->flags] = infos[0].layout; @@ -7251,6 +7262,245 @@ void ResourceContentKeyTests(bool benchmark) { } } +// The pipeline journal without a GPU: the recipe codec and the store key agree with each other and +// with the canonical state, identities follow the module and the specialization, the journal +// records each pipeline once, and a replay into the recording context (the lavapipe test replays +// into a second device) rebuilds the recorded graphics pipeline so the same configuration hits the +// store, hands the compute result to its hook, recovers a tail torn inside a record's padding and +// stops when cancelled. +void pipelineJournalTests() { + using namespace AgcDriver::Graphics; + using Stage = ShaderRecompiler::ShaderStage; + const auto directory = std::filesystem::temp_directory_path() / ("aps5-pipeline-journal-mock-" + std::to_string(std::random_device{}())); + std::filesystem::create_directories(directory); + struct Cleanup { + std::filesystem::path directory; + ~Cleanup() { + std::error_code error; + std::filesystem::remove_all(directory, error); + } + } cleanup{directory}; +#ifdef _WIN32 + _putenv_s("ANYPS5_NO_SHADER_CACHE", "0"); + _putenv_s("ANYPS5_SHADER_CACHE_DIR", directory.string().c_str()); + _putenv_s("APS5_PIPELINE_JOURNAL_THREADS", "2"); +#else + setenv("ANYPS5_NO_SHADER_CACHE", "0", 1); + setenv("ANYPS5_SHADER_CACHE_DIR", directory.string().c_str(), 1); + setenv("APS5_PIPELINE_JOURNAL_THREADS", "2", 1); +#endif + Require(PipelineJournalEnabled(), "the journal is disabled with a shader cache directory set"); + const auto module = [](std::uint32_t salt) { + ShaderRecompiler::RecompileResult result; + result.spirv = std::vector{0x07230203u, 0x00010000u, 0u, 8u, 0u, 0x00020011u, 1u, 0x00020011u, 2u + salt}; + result.runtimeAbiVersion = ShaderRecompiler::RuntimeAbi::Version; + return result; + }; + { + auto first = module(1); + const auto& constFirst = first; + auto copy = first; + const auto& constCopy = copy; + Require(!constFirst.Identity().Empty() && constFirst.Identity() == constCopy.Identity() && constFirst.spirv.data() == constCopy.spirv.data(), "a copied result did not share its module and identity"); + copy.spirv[8] = 99; + Require(!(constCopy.Identity() == constFirst.Identity()) && constCopy.spirv.data() != constFirst.spirv.data() && constFirst.Identity() == module(1).Identity(), "writing a copied module did not give it an identity of its own"); + auto specialized = module(1); + specialized.specialization.push_back({3, 4}); + Require(!(specialized.Identity() == constFirst.Identity()), "specialization constants did not enter the identity"); + auto subgroup = module(1); + subgroup.hostSubgroupSize = 64; + Require(!(subgroup.Identity() == constFirst.Identity()), "the host subgroup size did not enter the identity"); + Require(ShaderRecompiler::RecompileResult{}.Identity().Empty(), "a result without a module has an identity"); + } + mock = MockVulkan{}; + auto context = mockContext(); + context.device = makeHandle(); + context.graphicsPipelineLibrary = true; + context.tessellationShader = true; + context.limits.maxTessellationPatchSize = 32; + context.limits.maxColorAttachments = 8; + context.limits.maxPushConstantsSize = 256; + context.limits.maxVertexInputBindings = 16; + context.limits.maxVertexInputAttributes = 16; + context.limits.maxFramebufferWidth = 4096; + context.limits.maxFramebufferHeight = 4096; + context.limits.maxFramebufferLayers = 1; + std::uint64_t replayedComputeIdentityLow = 0; + { + DescriptorCache cache(context); + context.descriptorCache = &cache; + auto vertex = module(1); + vertex.bindings.push_back(makeBinding(Role::GuestBuffers, 0, 1, std::vector(4))); + auto fragment = module(2); + fragment.pushConstants.resize(8); + fragment.bindings.push_back(makeBinding(Role::ShaderData, 0, 1, std::vector(4))); + fragment.bindings.back().descriptorSet = ShaderRecompiler::RuntimeAbi::FragmentDescriptorSet; + const std::array shaders{{{Stage::Vertex, &vertex, 0}, {Stage::Fragment, &fragment, PipelinePushSlotBytes}}}; + State state{}; + state.stages = {ShaderPath::Vertex, 7u, 64u, 32u, std::nullopt, std::nullopt}; + state.stages.tessellation = ShaderRecompiler::TessellationConfiguration{3, 3, 2, 1, 4, 128, 16, true}; + ColorTarget color{}; + color.address = 0x1000; + color.extent = {64, 32}; + color.format = VK_FORMAT_R8G8B8A8_UNORM; + color.bytes = 64 * 32 * 4; + color.componentMapping = 0xec; + state.colors = {color, color}; + state.colors[1].format = VK_FORMAT_B8G8R8A8_UNORM; + state.colors[1].exportIndex = 1; + state.color = color; + state.hasColorTarget = true; + VkPipelineColorBlendAttachmentState blend{}; + blend.srcColorBlendFactor = VK_BLEND_FACTOR_SRC_ALPHA; + blend.dstColorBlendFactor = VK_BLEND_FACTOR_ONE_MINUS_SRC_ALPHA; + blend.colorWriteMask = 0xf; + state.blends = {blend, blend}; + state.blend = blend; + state.blendConstants = {0.25f, 0.5f, 0.75f, 1.0f}; + DepthTarget depth{}; + depth.address = 0x2000; + depth.format = VK_FORMAT_D32_SFLOAT; + depth.samples = 1; + state.depth = depth; + state.depthTest = true; + state.depthWrite = true; + state.depthCompare = VK_COMPARE_OP_LESS_OR_EQUAL; + state.stencilTest = true; + state.stencilFront = {VK_STENCIL_OP_KEEP, VK_STENCIL_OP_REPLACE, VK_STENCIL_OP_KEEP, VK_COMPARE_OP_ALWAYS, 0xff, 0xff, 1}; + state.stencilBack = state.stencilFront; + state.renderExtent = {64, 32}; + state.topology = VK_PRIMITIVE_TOPOLOGY_TRIANGLE_STRIP; + state.viewport = {0, 32, 64, -32, 0, 1}; + state.scissor = {{0, 0}, {64, 32}}; + state.cullMode = VK_CULL_MODE_BACK_BIT; + state.frontFace = VK_FRONT_FACE_CLOCKWISE; + state.samples = 1; + const VertexInputLayout input{{{0, 16, VK_VERTEX_INPUT_RATE_VERTEX}}, {{0, 0, VK_FORMAT_R32G32B32A32_SFLOAT, 0}}}; + const auto layouts = DescribeSetLayouts(context, shaders); + Require(layouts.layouts.split, "a fragment stage with the fragment set did not split the set layouts"); + const auto recipe = MakeGraphicsPipelineRecipe(state, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + Require(recipe.stages.size() == 2 && recipe.stages[0].identity == vertex.Identity() && recipe.stages[1].identity == fragment.Identity() && !recipe.stages[0].generated && recipe.stages[1].pushConstantOffset == PipelinePushSlotBytes && recipe.splitSets, "the recipe does not name its stages"); + Require(recipe.state.colors.size() == 2 && recipe.state.colors[0].address == 0 && recipe.state.colors[0].componentMapping == 0xec && recipe.state.depth && recipe.state.depth->address == 0 && recipe.state.viewport.width == 0 && recipe.state.stages.registerValue == 0 && recipe.state.stages.tessellation && recipe.state.stages.tessellation->breakAtInstance, "the canonical state kept or dropped the wrong fields"); + const auto bytes = EncodeGraphicsPipelineRecipe(recipe); + GraphicsPipelineRecipe decoded; + Require(DecodeGraphicsPipelineRecipe(bytes, decoded), "the recipe does not decode"); + Require(EncodeGraphicsPipelineRecipe(decoded) == bytes, "the recipe does not round-trip byte for byte"); + for (std::size_t length = 0; length < bytes.size(); length += 7) { + GraphicsPipelineRecipe scratch; + Require(!DecodeGraphicsPipelineRecipe(std::span(bytes).first(length), scratch), "a truncated recipe decoded"); + } + std::vector original, canonical, replayed, moved, mapped; + BuildPipelineStoreKey(original, context, state, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + BuildPipelineStoreKey(canonical, context, recipe.state, recipe.vertexInput, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + const PipelineSetLayouts decodedLayouts{layouts.layouts.main, layouts.layouts.fragment, decoded.splitSets, decoded.mainLayoutKey, decoded.fragmentLayoutKey}; + BuildPipelineStoreKey(replayed, context, decoded.state, decoded.vertexInput, decodedLayouts, shaders, decoded.attachmentLayout); + Require(!original.empty() && original == canonical && original == replayed, "the canonical or the decoded state keys another pipeline"); + auto movedState = state; + movedState.viewport.x = 5; + movedState.colors[0].address = 0x9000; + movedState.renderLayers = 4; + BuildPipelineStoreKey(moved, context, movedState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + Require(moved == original, "a field the pipeline does not read changed the key"); + auto mappedState = state; + mappedState.colors[0].componentMapping = 0x24; + BuildPipelineStoreKey(mapped, context, mappedState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + Require(mapped != original, "the component mapping did not enter the key"); + std::vector unclamped, clamped, biased; + auto clampedState = state; + clampedState.depthBiasClamp = 1.0f; + BuildPipelineStoreKey(clamped, context, clampedState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + Require(clamped == original, "a depth bias clamp without depth bias changed the key"); + clampedState.depthBias = true; + BuildPipelineStoreKey(clamped, context, clampedState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + clampedState.depthBiasClamp = 0.0f; + BuildPipelineStoreKey(unclamped, context, clampedState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + Require(unclamped != original && clamped != unclamped && CanonicalPipelineState(clampedState).depthBias && CanonicalPipelineState(mappedState).depthBiasClamp == 0.0f, "the depth bias clamp did not enter the key"); + clampedState.depthBiasClamp = 2.0f; + BuildPipelineStoreKey(biased, context, clampedState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + Require(biased == clamped, "the depth bias clamp's value entered the key"); + auto compute = module(3); + compute.bindings.push_back(makeBinding(Role::GuestBuffers, 0, 2, std::vector(8))); + compute.pushConstants.resize(16); + compute.hostSubgroupSize = 32; + constexpr std::uint64_t fingerprint = 0x5eed'0000'0000'0001ull; + { + auto bound = StartPipelineJournalReplay(context, fingerprint, {}); + Require(bound != nullptr, "the first replay did not start"); + bound->Wait(); + Require(bound->Done() && PipelineJournalTotals().loadedRecipes == 0, "an absent journal loaded pipelines"); + } + const auto before = PipelineJournalTotals(); + RecordGraphicsPipeline(recipe, shaders); + RecordComputePipeline(compute, true); + FlushPipelineJournal(); + const auto recorded = PipelineJournalTotals(); + Require(recorded.recordedGraphics == before.recordedGraphics + 1 && recorded.recordedCompute == before.recordedCompute + 1 && recorded.recordedModules == before.recordedModules + 3 && recorded.writeFailures == 0, "the journal did not record one graphics and one compute pipeline with three modules"); + RecordGraphicsPipeline(recipe, shaders); + RecordComputePipeline(compute, true); + FlushPipelineJournal(); + const auto repeated = PipelineJournalTotals(); + Require(repeated.recordedGraphics == recorded.recordedGraphics && repeated.recordedCompute == recorded.recordedCompute && repeated.recordedModules == recorded.recordedModules, "a repeated pipeline was recorded again"); + Require(std::filesystem::file_size(PipelineJournalPath()) > 0, "the journal file is empty"); + const auto storeBefore = PipelineStoreTotals(); + std::vector receivedWords; + ShaderRecompiler::RecompileResult received; + bool receivedPush = false; + std::uint32_t computeCalls = 0; + PipelineJournalHooks hooks{[&](const ShaderRecompiler::RecompileResult& shader, bool pushConstants) { + ++computeCalls; + received = shader; + receivedPush = pushConstants; + return true; + }}; + auto replay = StartPipelineJournalReplay(context, fingerprint, hooks); + Require(replay != nullptr, "the replay did not start"); + replay->Wait(); + Require(replay->Done(), "the replay did not finish"); + const auto totals = PipelineJournalTotals(); + Require(totals.loadedRecipes == 2 && totals.loadedModules == 3 && totals.droppedRecords == 0, "the replay did not load the two recorded pipelines and three modules"); + Require(totals.replayedGraphics == 1 && totals.replayedCompute == 1 && totals.replayRejected == 0 && totals.replayFailed == 0, "the replay did not build one graphics and one compute pipeline"); + Require(computeCalls == 1 && receivedPush && received.Identity() == compute.Identity() && received.spirv == compute.spirv && received.bindings.size() == 1 && received.bindings[0].count == 2 && received.pushConstants.size() == 16 && received.hostSubgroupSize == 32, "the compute hook did not receive the recorded result"); + replayedComputeIdentityLow = received.Identity().low; + const auto storeReplayed = PipelineStoreTotals(); + Require(storeReplayed.misses == storeBefore.misses + 1 && storeReplayed.cached == storeBefore.cached + 1, "the replayed graphics pipeline did not enter the store"); + const auto created = mock.pipelineCreateCount; + bool built = true; + const auto pipeline = CachedPipeline(context, state, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL, &built); + const auto storeHit = PipelineStoreTotals(); + Require(pipeline != nullptr && !built && mock.pipelineCreateCount == created && storeHit.hits == storeReplayed.hits + 1 && storeHit.misses == storeReplayed.misses, "the draw's configuration did not find the replayed pipeline"); + { + const auto journalPath = PipelineJournalPath(); + const auto whole = std::filesystem::file_size(journalPath); + Require(whole % 8 == 0 && whole > 3, "the journal file is not record aligned"); + std::filesystem::resize_file(journalPath, whole - 3); + auto later = module(4); + later.bindings.push_back(makeBinding(Role::GuestBuffers, 0, 1, std::vector(4))); + RecordComputePipeline(later, false); + FlushPipelineJournal(); + Require(std::filesystem::file_size(journalPath) % 8 == 0 && PipelineJournalTotals().writeFailures == 0, "an append after a tail torn inside a record's padding left the file unaligned"); + auto torn = StartPipelineJournalReplay(context, fingerprint, hooks); + Require(torn != nullptr, "the replay after the torn tail did not start"); + torn->Wait(); + const auto reloaded = PipelineJournalTotals(); + Require(reloaded.droppedRecords == 0 && reloaded.loadedModules == 4 && reloaded.loadedRecipes == 2 && reloaded.replayFailed == 0, "the journal did not recover from a tail torn inside a record's padding"); + RecordComputePipeline(compute, true); + FlushPipelineJournal(); + Require(PipelineJournalTotals().recordedCompute == reloaded.recordedCompute + 1 && PipelineJournalTotals().recordedModules == reloaded.recordedModules, "the pipeline whose record was torn was not recorded again, or its module was"); + } + { + auto cancelled = StartPipelineJournalReplay(context, fingerprint, hooks); + Require(cancelled != nullptr, "the third replay did not start"); + cancelled->Cancel(); + cancelled->Wait(); + const auto after = PipelineJournalTotals(); + Require(cancelled->Done() && after.replayFailed == 0 && after.replayRejected == 0 && after.replayedGraphics + after.replayPresent <= totals.replayedGraphics + totals.replayPresent + 1, "a cancelled replay did not stop cleanly"); + } + ClearCachedPipelines(context.device); + } + Require(replayedComputeIdentityLow != 0, "the replayed compute identity is empty"); + Require(mock.live == 0, "the pipeline journal test leaked Vulkan objects"); +} + int main(int argc, char** argv) { #ifdef _WIN32 _putenv_s("APS5_PIN_WAIT_MS", "200"); @@ -7264,6 +7514,11 @@ int main(int argc, char** argv) { stencilReferenceUnusedTests(); return 0; } + if (argc == 2 && std::string_view(argv[1]) == "--pipeline-journal") { + pipelineJournalTests(); + std::cout << "pipeline journal recipe, record and replay tests passed\n"; + return 0; + } if (argc == 2 && std::string_view(argv[1]) == "--sparse-buffer-holes") { bool passed = false; { diff --git a/core/libs/prx/libSceAgcDriver/tests/Recorder.cpp b/core/libs/prx/libSceAgcDriver/tests/Recorder.cpp index ca6b3dad35..e132d8db4e 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Recorder.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Recorder.cpp @@ -7314,10 +7314,13 @@ void pipelineCacheTests(const Device& device, bool benchmark) { input = {}; Require(lookup(input) == first, "pipeline cache retained bytes from a larger key"); vertex.variantId = 0; - const auto unidentified = lookup(input); - Require(lookup(input) != unidentified, "pipeline cache retained an unidentified shader"); + Require(lookup(input) == first, "pipeline cache keyed on the variant id instead of the module"); + vertex.spirv[2] ^= 1u; + const auto other = lookup(input); + Require(other != first && lookup(input) == other, "pipeline cache ignored another module"); + vertex.spirv[2] ^= 1u; vertex.variantId = 1; - Require(lookup(input) == first, "an unidentified shader disturbed the retained pipeline"); + Require(lookup(input) == first, "another module disturbed the retained pipeline"); ClearCachedPipelines(context.device); Require(lookup(input) != first, "clearing the pipeline cache retained its old entry"); std::cout << "Pipeline cache key and retained lifetime tests passed\n"; @@ -7789,13 +7792,13 @@ void pipelineLibraryTests(const Device& device) { Require(StagePushOffset(88, ShaderRecompiler::ShaderStage::Fragment, true) == PipelinePushSlotBytes && StagePushOffset(88, ShaderRecompiler::ShaderStage::Fragment, false) == 88 && StagePushOffset(40, ShaderRecompiler::ShaderStage::TessellationEvaluation, true) == 40, "a stage got the wrong push offset"); const auto beforeSlots = counters().built; ShaderRecompiler::RecompileResult otherVertex = vertex; - otherVertex.variantId = 14; + otherVertex.spirv[2] ^= 1u; std::array slotted{CompiledShader{ShaderRecompiler::ShaderStage::Vertex, &vertex, 0}, CompiledShader{ShaderRecompiler::ShaderStage::Fragment, &fragment, PipelinePushSlotBytes}}; static_cast(CachedPipeline(context, state, input, resources, slotted, VK_IMAGE_LAYOUT_GENERAL)); slotted.front().program = &otherVertex; static_cast(CachedPipeline(context, state, input, resources, slotted, VK_IMAGE_LAYOUT_GENERAL)); Require(counters().built[2] == beforeSlots[2] + 1u && counters().built[1] == beforeSlots[1] + 1u, "a pixel shader in its fixed slot built its library again for another vertex shader"); - fragment.variantId = 13; + fragment.spirv[2] ^= 1u; static_cast(lookup()); Require(counters().built == std::array{2, 2, 3, 2}, "a new pixel shader rebuilt more than the fragment shader library"); state.colors.front().format = VK_FORMAT_R16G16B16A16_SFLOAT; diff --git a/core/libs/prx/libSceAgcDriver/tests/execution/PipelineJournal.cpp b/core/libs/prx/libSceAgcDriver/tests/execution/PipelineJournal.cpp new file mode 100644 index 0000000000..16d3742761 --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/tests/execution/PipelineJournal.cpp @@ -0,0 +1,431 @@ +#include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp" +#include "prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp" +#include "prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp" +#include "prx/libSceAgcDriver/Graphics/include/ColorTargetLayout.hpp" +#include "prx/libSceAgcDriver/Graphics/include/Draw.hpp" +#include "prx/libSceAgcDriver/Graphics/include/Pipeline.hpp" +#include "prx/libSceAgcDriver/Graphics/include/PipelineJournal.hpp" +#include "prx/libSceAgcDriver/Graphics/include/ShaderInputState.hpp" +#include "prx/libSceAgcDriver/Graphics/include/State.hpp" +#include "prx/libSceAgcDriver/Graphics/include/Texture.hpp" +#include "prx/libc/include/GuestAllocations.hpp" +#include "prx/libc/include/GuestArena.hpp" +#include "prx/libc/include/GuestWriteWatch.hpp" +#include "../NarrowConstantStoreFixture.hpp" +#include "Recompiler.hpp" +#include "Optimization/ResourceProgram.hpp" +#include "VulkanTestDevice.hpp" +#ifdef _WIN32 +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#else +#include +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +using AgcDriver::Graphics::ColorTargetLayout; +using AgcDriver::Graphics::ColorTileMode; +using AgcDriver::Graphics::Require; +using ShaderRecompiler::ShaderStage; + +constexpr std::uint32_t Width = 64; +constexpr std::uint32_t Height = 32; +constexpr std::size_t BlockBytes = 65536; +constexpr std::uint32_t Color1555Info = 0x8044u; +constexpr std::uint32_t LinearAttrib3 = 0x09000000u; +constexpr std::uint32_t Float32Export = 9; + +alignas(256) constexpr std::array VertexCode{ + 0xe0382000, 0x80000005, 0xbf8c3f70, 0xf80008cf, 0x03020100, 0xbf810000, +}; + +constexpr std::array, 3> Triangle{{ + {-1.0f, -1.0f, 0.5f, 1.0f}, {3.0f, -1.0f, 0.5f, 1.0f}, {-1.0f, 3.0f, 0.5f, 1.0f} +}}; + +alignas(256) std::array PixelProgram{}; +std::size_t pixelProgramWords = 0; + +std::span Program(const std::array& color) { + std::size_t at = 0; + for (std::uint32_t channel = 0; channel < 4; ++channel) { + PixelProgram[at++] = 0x7e0002ffu | ((4u + channel) << 17u); + PixelProgram[at++] = std::bit_cast(color[channel]); + } + PixelProgram[at++] = 0xf800180fu; + PixelProgram[at++] = 0x07060504u; + PixelProgram[at++] = 0xbf810000u; + pixelProgramWords = at; + return std::span(PixelProgram.data(), at); +} + +class Block { +public: + explicit Block(std::size_t bytes) : bytes(bytes), watched(AgcDriver::GuestMemory::WriteWatched()) { +#ifdef _WIN32 + if (watched) { + data = static_cast(GuestArena::GuestArenaAllocate_nid_postfix(bytes, BlockBytes)); + GuestArena::GuestArenaCommit_nid_postfix(data, bytes, PAGE_READWRITE, bytes); + } else { + data = static_cast(VirtualAlloc(nullptr, bytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE)); + } +#else + if (watched) { + void* raw = mmap(nullptr, bytes + BlockBytes, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + Require(raw != MAP_FAILED, "cannot map the guest block"); + const auto begin = reinterpret_cast(raw); + const auto aligned = (begin + BlockBytes - 1) & ~(static_cast(BlockBytes) - 1); + if (aligned != begin) munmap(raw, aligned - begin); + if (aligned + bytes != begin + bytes + BlockBytes) munmap(reinterpret_cast(aligned + bytes), begin + BlockBytes - aligned); + data = reinterpret_cast(aligned); + GuestWriteWatch::GuestWriteWatchRegister_nid_postfix(data, bytes); + } else { + data = static_cast(std::aligned_alloc(BlockBytes, bytes)); + } +#endif + Require(data != nullptr, "cannot allocate the guest block"); + GuestAllocations::Mutation mutation; + mutation.Add(data, bytes, true, true, true); + } + + ~Block() { + AgcDriver::Graphics::StorageTexture::FlushPending(Address(), bytes, nullptr, "test release"); + { + GuestAllocations::Mutation mutation; + mutation.Remove(data); + } +#ifdef _WIN32 + if (watched) { + GuestArena::GuestArenaReset_nid_postfix(data, bytes); + GuestArena::GuestArenaRelease_nid_postfix(data, bytes); + } else { + VirtualFree(data, 0, MEM_RELEASE); + } +#else + if (watched) { + munmap(data, bytes); + GuestWriteWatch::GuestWriteWatchUnregister_nid_postfix(data, bytes); + } else { + std::free(data); + } +#endif + } + + Block(const Block&) = delete; + Block& operator=(const Block&) = delete; + std::uint64_t Address() const { return reinterpret_cast(data); } + std::size_t bytes; + bool watched; + std::uint8_t* data = nullptr; +}; + +AgcDriver::Graphics::ColorTarget DecodeTarget(const Block& block) { + AgcDriver::Registers cx; + cx[0x318] = static_cast(block.Address() >> 8u); + cx[0x31b] = 0; + cx[0x31c] = Color1555Info; + cx[0x31d] = 0; + cx[0x3b0] = ((Width - 1u) << 14u) | (Height - 1u); + cx[0x3b8] = LinearAttrib3; + cx[0x390] = static_cast(block.Address() >> 40u); + auto color = AgcDriver::Graphics::DecodeColorBuffer(cx, 0); + Require(color.format == VK_FORMAT_A1R5G5B5_UNORM_PACK16 && color.bytes == block.bytes, "the test target did not decode as an A1R5G5B5 surface"); + return color; +} + +void Draw(AgcDriver::VulkanDevice& device, const AgcDriver::Graphics::ColorTarget& color, std::span pixelCode) { + const auto target = device.Target(); + constexpr std::uint32_t waveSize = 64; + std::vector vertexUserData(4, 0u); + const auto address = reinterpret_cast(Triangle.data()); + const std::array vertexBuffer{static_cast(address), static_cast((address >> 32u) & 0xffffu) | (16u << 16u), static_cast(Triangle.size()), 0x01016facu}; + std::copy(vertexBuffer.begin(), vertexBuffer.end(), vertexUserData.begin()); + const std::array vertexMemory{{{reinterpret_cast(VertexCode.data()), std::as_bytes(std::span(VertexCode))}}}; + ShaderRecompiler::RecompileRequest vertex{ + {ShaderStage::Vertex, reinterpret_cast(VertexCode.data()), VertexCode, 0, {}}, + {waveSize, 0, vertexUserData, std::nullopt, std::nullopt, ShaderRecompiler::ShaderVertexStageInfo{}, vertexMemory}, + target, + {0, 0, 0, 64} + }; + vertex.useCache = false; + const auto vertexResult = ShaderRecompiler::Recompile(vertex); + const auto vertexPush = static_cast(vertexResult.pushConstants.size()); + + AgcDriver::Graphics::State state{}; + state.stages = {AgcDriver::Graphics::ShaderPath::Vertex, 0u, waveSize, waveSize, std::nullopt, std::nullopt}; + state.colors = {color}; + state.color = color; + state.hasColorTarget = true; + state.renderExtent = {Width, Height}; + state.topology = VK_PRIMITIVE_TOPOLOGY_TRIANGLE_LIST; + state.viewport = {0, static_cast(Height), static_cast(Width), -static_cast(Height), 0, 1}; + state.negativeOneToOne = false; + state.scissor = {{0, 0}, {Width, Height}}; + state.cullMode = VK_CULL_MODE_NONE; + state.frontFace = VK_FRONT_FACE_COUNTER_CLOCKWISE; + VkPipelineColorBlendAttachmentState blend{}; + blend.colorWriteMask = 0xf; + state.blends = {blend}; + state.blend = blend; + state.blendConstants = {}; + + const auto codeAddress = reinterpret_cast(pixelCode.data()); + const AgcDriver::Registers context{{0x1b6, 0x8000}, {0x1b3, 0}, {0x1b4, 0}, {0x203, 0}, {0x1c5, Float32Export}}; + const std::array pixelMemory{{{codeAddress, std::as_bytes(pixelCode)}}}; + ShaderRecompiler::RecompileRequest fragment{ + {ShaderStage::Fragment, codeAddress, pixelCode, 0, {}}, + {waveSize, 0, {}, std::nullopt, AgcDriver::Graphics::DecodePixelStageInfo(context, {}), std::nullopt, pixelMemory}, + target, + PixelPushLayout(vertexPush, target) + }; + const auto handle = ShaderRecompiler::PrepareShader(fragment); + AgcDriver::DriverDetail::ShaderSnapshot snapshot{codeAddress, 0, 1, {pixelCode.begin(), pixelCode.end()}, {}}; + snapshot.prepared->entries.push_back({0, handle}); + fragment.shader.code = snapshot.code; + fragment.context.pixel = AgcDriver::Graphics::DecodePixelStageInfo(context, AgcDriver::Graphics::ExportMappings(state)); + const ShaderRecompiler::SrtRuntime runtime{}; + const auto invocation = AgcDriver::DriverDetail::InvocationFor(snapshot, 0, fragment); + const auto pixelResult = invocation.Materialize(*invocation.Capture(runtime)); + const std::array shaders{{ + {ShaderStage::Vertex, &vertexResult, 0}, + {ShaderStage::Fragment, pixelResult.get(), PixelPushOffset(vertexPush, target)} + }}; + const AgcDriver::Pm4::DrawParameters draw{0, static_cast(Triangle.size()), 0, 1, 0, false}; + device.Draw(state, draw, shaders); +} + +void ExpectTexel(AgcDriver::VulkanDevice& device, const Block& block, std::uint16_t texel, const char* what) { + AgcDriver::Graphics::StorageTexture::FlushPending(block.Address(), block.bytes, nullptr, "test read-back"); + device.WaitIdle(); + std::vector stored(block.bytes / 2u); + AgcDriver::GuestMemory::Read(block.Address(), std::as_writable_bytes(std::span(stored)), 1); + const ColorTargetLayout layout(Width, Height, ColorTileMode::Linear, 2); + for (std::uint32_t y = 0; y < Height; ++y) { + for (std::uint32_t x = 0; x < Width; ++x) { + const auto index = layout.Offset(x, y) / 2u; + if (stored[index] == texel) continue; + char detail[160]; + std::snprintf(detail, sizeof(detail), ": pixel (%u, %u) holds 0x%04x, expected 0x%04x", x, y, stored[index], texel); + throw std::runtime_error(std::string(what) + detail); + } + } +} + +// The constant-store compute program with its two stored literals replaced: `salt` 0 is the +// fixture's program, every other value a program of its own with its own module. +std::vector ComputeCode(std::uint32_t salt) { + std::vector code(NarrowConstantStoreFixture::Code.begin(), NarrowConstantStoreFixture::Code.end()); + if (salt != 0) { + code[3] = 0x00a10200u ^ (salt * 0x01010101u); + code[5] = 0x0000fec3u ^ (salt * 0x00010001u); + } + return code; +} + +ShaderRecompiler::RecompileResult CompileCompute(AgcDriver::VulkanDevice& device, const std::vector& code, std::span userData, std::uint32_t waveSize) { + using namespace ShaderRecompiler; + const std::span words(code); + const std::array memory{{{reinterpret_cast(words.data()), std::as_bytes(words)}}}; + const ShaderComputeStageInfo compute{{NarrowConstantStoreFixture::Threads, 1, 1}, 0, {false, false, false}, false, 1, {}}; + RecompileRequest request{ + {ShaderStage::Compute, reinterpret_cast(words.data()), words, 0, {}}, + {waveSize, 0, userData, compute, std::nullopt, std::nullopt, memory}, + device.Target(), {0, 0, 0, 128}, std::nullopt, false, + }; + return Recompile(request); +} + +void ExpectConstantStore(const Block& block, const char* what) { + using namespace NarrowConstantStoreFixture; + for (std::size_t offset = 0; offset < Threads * LaneBytes; ++offset) { + const auto expected = ExpectedLane[offset % LaneBytes]; + Require(block.data[offset] == expected, std::string(what) + ": byte " + std::to_string(offset) + " is " + std::to_string(block.data[offset]) + ", expected " + std::to_string(expected)); + } +} + +std::array AddressWords(const Block& block) { + const auto address = block.Address(); + return {static_cast(address), static_cast(address >> 32u)}; +} + +void SetEnvironment(const char* name, const std::string& value) { +#ifdef _WIN32 + _putenv_s(name, value.c_str()); +#else + setenv(name, value.c_str(), 1); +#endif +} + +struct Totals { + AgcDriver::Graphics::PipelineJournalCounters journal; + AgcDriver::Graphics::PipelineStoreCounters store; +}; + +Totals Snapshot() { + return {AgcDriver::Graphics::PipelineJournalTotals(), AgcDriver::Graphics::PipelineStoreTotals()}; +} + +void JournalTests() { + using namespace NarrowConstantStoreFixture; + Require(AgcDriver::Graphics::PipelineJournalEnabled(), "the pipeline journal is disabled although the shader cache directory is set"); + const auto journalPath = AgcDriver::Graphics::PipelineJournalPath(); + Block compute(BlockBytes); + Block target(ColorTargetLayout(Width, Height, ColorTileMode::Linear, 2).Bytes()); + const auto computeCode = ComputeCode(0); + const std::array red{1.0f, 0.0f, 0.0f, 1.0f}; + constexpr std::uint16_t redTexel = 0x801f; + { + const auto device = OpenVulkanTestDevice(); + Require(device != nullptr, "the first device is required once the second opened"); + device->WaitForPipelineJournalReplay(); + const auto before = Snapshot(); + Require(before.journal.loadedRecipes == 0 && before.journal.replayedGraphics == 0 && before.journal.replayedCompute == 0, "an empty journal replayed pipelines"); + std::memset(compute.data, Fill, compute.bytes); + const auto words = AddressWords(compute); + const auto result = CompileCompute(*device, computeCode, words, 32); + device->Dispatch(result, 1, 1, 1, {}, reinterpret_cast(computeCode.data())); + device->WaitIdle(); + ExpectConstantStore(compute, "first run: the constant store dispatch"); + std::memset(target.data, 0, target.bytes); + Draw(*device, DecodeTarget(target), Program(red)); + ExpectTexel(*device, target, redTexel, "first run: the draw"); + Require(device->ComputePipelinesCreated() == 1, "the first run did not build exactly one compute pipeline"); + AgcDriver::Graphics::FlushPipelineJournal(); + const auto after = Snapshot(); + Require(after.store.misses == before.store.misses + 1 && after.store.hits == before.store.hits, "the first draw did not miss the pipeline store exactly once"); + Require(after.journal.recordedGraphics == 1 && after.journal.recordedCompute == 1 && after.journal.recordedModules == 3 && after.journal.writeFailures == 0, "the first run did not record one graphics and one compute pipeline with three modules"); + Draw(*device, DecodeTarget(target), Program(red)); + device->WaitIdle(); + AgcDriver::Graphics::FlushPipelineJournal(); + const auto repeated = Snapshot(); + Require(repeated.store.hits == after.store.hits + 1 && repeated.store.misses == after.store.misses, "a repeated draw did not hit the pipeline store"); + Require(repeated.journal.recordedGraphics == 1 && repeated.journal.recordedModules == 3, "a repeated draw recorded its pipeline again"); + } + Require(std::filesystem::exists(journalPath) && std::filesystem::file_size(journalPath) > 0, "the journal file was not written"); + { + const auto device = OpenVulkanTestDevice(); + Require(device != nullptr, "the second device could not be opened"); + device->WaitForPipelineJournalReplay(); + const auto replayed = Snapshot(); + Require(replayed.journal.loadedRecipes == 2 && replayed.journal.loadedModules == 3 && replayed.journal.droppedRecords == 0, "the second run did not load the two recorded pipelines and their three modules"); + Require(replayed.journal.replayedGraphics == 1 && replayed.journal.replayedCompute == 1 && replayed.journal.replayRejected == 0 && replayed.journal.replayFailed == 0 && replayed.journal.replayPresent == 0, "the replay did not build one graphics and one compute pipeline"); + Require(device->ComputePipelinesCreated() == 1, "the replay did not build the compute pipeline on the new device"); + std::memset(compute.data, Fill, compute.bytes); + const auto words = AddressWords(compute); + const auto result = CompileCompute(*device, computeCode, words, 32); + device->Dispatch(result, 1, 1, 1, {}, reinterpret_cast(computeCode.data())); + device->WaitIdle(); + ExpectConstantStore(compute, "second run: the constant store dispatch"); + Require(device->ComputePipelinesCreated() == 1, "the second run's dispatch built a compute pipeline the replay had built"); + std::memset(target.data, 0, target.bytes); + Draw(*device, DecodeTarget(target), Program(red)); + ExpectTexel(*device, target, redTexel, "second run: the draw through the replayed pipeline"); + const auto drawn = Snapshot(); + Require(drawn.store.hits == replayed.store.hits + 1 && drawn.store.misses == replayed.store.misses, "the second run's draw did not find the replayed pipeline"); + AgcDriver::Graphics::FlushPipelineJournal(); + const auto final = Snapshot(); + Require(final.journal.recordedGraphics == 1 && final.journal.recordedCompute == 1 && final.journal.recordedModules == 3, "the second run recorded pipelines the journal already held"); + } +} + +// --benchmark [count]: `count` distinct compute programs, dispatched for the first time on three +// devices in turn. The first device starts without the driver's pipeline cache file (a driver with +// an on-disk shader cache of its own, lavapipe's Mesa cache here, still serves from it unless that +// is disabled too, MESA_SHADER_CACHE_DISABLE=true, where the driver's compile dominates); it leaves +// the driver's cache and the journal warm. The second replays the journal before its dispatches. +// The third dispatches with the driver's cache warm but the journal file removed: what a warm run +// pays at each first use without the journal. Wall times only; no pass threshold. +void Benchmark(std::uint32_t count) { + using namespace NarrowConstantStoreFixture; + Require(AgcDriver::Graphics::PipelineJournalEnabled(), "the pipeline journal is disabled"); + Block compute(BlockBytes); + const auto words = AddressWords(compute); + std::vector> codes; + for (std::uint32_t salt = 1; salt <= count; ++salt) codes.push_back(ComputeCode(salt)); + struct Run { + double openMs = 0; + double dispatchMs = 0; + std::uint64_t created = 0; + AgcDriver::Graphics::PipelineJournalCounters journal; + }; + const auto run = [&](const char* what) { + Run measured; + const auto openedAt = std::chrono::steady_clock::now(); + const auto device = OpenVulkanTestDevice(); + Require(device != nullptr, std::string("no device for the benchmark's ") + what); + device->WaitForPipelineJournalReplay(); + measured.openMs = std::chrono::duration(std::chrono::steady_clock::now() - openedAt).count(); + measured.journal = AgcDriver::Graphics::PipelineJournalTotals(); + std::vector results; + for (const auto& code : codes) results.push_back(CompileCompute(*device, code, words, 32)); + const auto createdBefore = device->ComputePipelinesCreated(); + const auto started = std::chrono::steady_clock::now(); + for (std::size_t index = 0; index < results.size(); ++index) device->Dispatch(results[index], 1, 1, 1, {}, reinterpret_cast(codes[index].data())); + device->WaitIdle(); + measured.dispatchMs = std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + measured.created = device->ComputePipelinesCreated() - createdBefore; + AgcDriver::Graphics::FlushPipelineJournal(); + return measured; + }; + const auto cold = run("cold run"); + const auto replayed = run("replayed run"); + std::error_code error; + std::filesystem::remove(AgcDriver::Graphics::PipelineJournalPath(), error); + Require(!error, "cannot remove the journal for the benchmark's last run"); + const auto warm = run("warm run without the journal"); + std::printf("pipeline journal benchmark: %u compute programs, %u replay threads\n", count, replayed.journal.replayThreads); + std::printf(" cold driver cache, no journal: %llu pipelines built at their dispatches, %u dispatches in %.1f ms (%.2f ms each)\n", static_cast(cold.created), count, cold.dispatchMs, cold.dispatchMs / count); + std::printf(" warm driver cache, no journal: %llu pipelines built at their dispatches, %u dispatches in %.1f ms (%.2f ms each)\n", static_cast(warm.created), count, warm.dispatchMs, warm.dispatchMs / count); + std::printf(" warm driver cache, journal: %llu pipelines built by the replay in %.1f ms of replay work (%.1f ms from the device opening), then %llu built at their dispatches, %u dispatches in %.1f ms (%.2f ms each)\n", static_cast(replayed.journal.replayedCompute), replayed.journal.replayMs, replayed.openMs, static_cast(replayed.created), count, replayed.dispatchMs, replayed.dispatchMs / count); +} + +} + +int main(int argc, char** argv) { + try { + const auto directory = std::filesystem::temp_directory_path() / ("aps5-pipeline-journal-test-" + std::to_string(std::random_device{}())); + std::filesystem::create_directories(directory); + SetEnvironment("ANYPS5_NO_SHADER_CACHE", "0"); + SetEnvironment("ANYPS5_SHADER_CACHE_DIR", directory.string()); + SetEnvironment("APS5_NO_PIPELINE_JOURNAL", "0"); + SetEnvironment("APS5_VERIFY_SET_LAYOUTS", "1"); + SetEnvironment("APS5_PIPELINE_JOURNAL_THREADS", "2"); + struct Cleanup { + std::filesystem::path directory; + ~Cleanup() { + std::error_code error; + std::filesystem::remove_all(directory, error); + } + } cleanup{directory}; + { + const auto device = OpenVulkanTestDevice(); + if (!device) return VulkanTestSkipped; + } + if (argc >= 2 && std::string(argv[1]) == "--benchmark") { + Benchmark(argc >= 3 ? static_cast(std::strtoul(argv[2], nullptr, 10)) : 24u); + return 0; + } + JournalTests(); + std::puts("pipeline journal record and replay passed"); + return 0; + } catch (const std::exception& error) { + std::cerr << error.what() << '\n'; + return 1; + } +} diff --git a/core/shader/recompiler/CMakeLists.txt b/core/shader/recompiler/CMakeLists.txt index 18cea782dc..a14e40ce48 100644 --- a/core/shader/recompiler/CMakeLists.txt +++ b/core/shader/recompiler/CMakeLists.txt @@ -255,6 +255,12 @@ function(add_shader_recompiler target runtimeTarget) SetDebugProbeActive ShaderCacheDirectory ShaderStage + HashSpirvWords + ShaderDiskCache::DecodeResult + ShaderDiskCache::Enabled + ShaderDiskCache::EncodeResult + ShaderDiskCache::EntryDirectory + ShaderDiskCache::SourceVersion SrgbDecodeBit SrgbUnormFormat FindUserDataTail diff --git a/core/shader/recompiler/Recompiler.hpp b/core/shader/recompiler/Recompiler.hpp index ff93352beb..b66f1cd695 100644 --- a/core/shader/recompiler/Recompiler.hpp +++ b/core/shader/recompiler/Recompiler.hpp @@ -5,6 +5,7 @@ #include "PipelineSpecialization.hpp" #include #include +#include #include #include #include @@ -468,25 +469,64 @@ struct BarycentricEmulationLayout { [[nodiscard]] BarycentricEmulationLayout LayoutBarycentricEmulation(std::span parameters, const BarycentricEmulation& emulation); +// A 128-bit hash of a module's words (SharedSpirv::ContentHash), zero for an empty module. The +// mixing is pure integer arithmetic over the words in order, so the value is the same on every run, +// platform and compiler: a pipeline journal names modules by it across runs. +struct SpirvContentHash { + std::uint64_t low = 0; + std::uint64_t high = 0; + bool operator==(const SpirvContentHash&) const = default; +}; + +inline std::uint64_t HashSpirvWords(std::span words, std::uint64_t seed) { + constexpr std::uint64_t multiplier = 0x9e3779b97f4a7c15ull; + std::uint64_t hash = seed ^ (static_cast(words.size()) * multiplier); + const auto mix = [&](std::uint64_t chunk) { + chunk *= 0xff51afd7ed558ccdull; + chunk ^= chunk >> 32u; + hash = (hash ^ chunk) * multiplier; + hash = (hash << 27u) | (hash >> 37u); + }; + std::size_t index = 0; + for (; index + 2 <= words.size(); index += 2) mix(words[index] | (static_cast(words[index + 1]) << 32u)); + if (index < words.size()) mix(words[index] ^ 0x8000000000000000ull); + hash ^= hash >> 33u; + hash *= 0xff51afd7ed558ccdull; + hash ^= hash >> 33u; + hash *= 0xc4ceb9fe1a85ec53ull; + hash ^= hash >> 33u; + return hash; +} + // Compiled SPIR-V shared between a cached variant and every result materialized from it: results // are copied per dispatch and draw, so the words are reference counted and only duplicated when a // holder writes to them (tests and tools patch modules in place). Reads look like a vector. // The non-const data(), operator[], begin() and end() count as writes: on a shared holder (every // cached result) they clone the module, so a consumer that only reads takes the result by const // reference, or the per-dispatch copy this class removes comes back without a compiler hint. +// ContentHash is computed once per shared holder, on first use, and dropped by a write. class SharedSpirv { public: SharedSpirv() = default; - SharedSpirv(std::vector words) : words(std::make_shared>(std::move(words))) {} - SharedSpirv(std::shared_ptr owner, std::span view) : owner(std::move(owner)), view(view) {} + SharedSpirv(std::vector words) : holder(std::make_shared(std::move(words), nullptr, std::span{})) {} + SharedSpirv(std::shared_ptr owner, std::span view) : holder(std::make_shared(std::vector{}, std::move(owner), view)) {} SharedSpirv& operator=(std::vector other) { - words = std::make_shared>(std::move(other)); - owner.reset(); - view = {}; + holder = std::make_shared(std::move(other), nullptr, std::span{}); return *this; } - [[nodiscard]] std::span Words() const { return words ? std::span(*words) : view; } + [[nodiscard]] std::span Words() const { return holder ? holder->Words() : std::span{}; } + [[nodiscard]] SpirvContentHash ContentHash() const { + if (holder == nullptr) return {}; + if (holder->hashed.load(std::memory_order_acquire)) return {holder->hashLow.load(std::memory_order_relaxed), holder->hashHigh.load(std::memory_order_relaxed)}; + const auto words = holder->Words(); + if (words.empty()) return {}; + const SpirvContentHash hash{HashSpirvWords(words, 0x5eed5a1d00000001ull), HashSpirvWords(words, 0x5eed5a1d00000002ull)}; + holder->hashLow.store(hash.low, std::memory_order_relaxed); + holder->hashHigh.store(hash.high, std::memory_order_relaxed); + holder->hashed.store(true, std::memory_order_release); + return hash; + } [[nodiscard]] std::vector Copy() const { return {Words().begin(), Words().end()}; } [[nodiscard]] std::size_t size() const { return Words().size(); } [[nodiscard]] bool empty() const { return Words().empty(); } @@ -503,18 +543,27 @@ public: friend bool operator==(const SharedSpirv& left, const SharedSpirv& right) { return std::ranges::equal(left.Words(), right.Words()); } private: + struct Holder { + Holder(std::vector words, std::shared_ptr owner, std::span view) : words(std::move(words)), owner(std::move(owner)), view(view) {} + [[nodiscard]] std::span Words() const { return owner ? view : std::span(words); } + std::vector words; + std::shared_ptr owner; + std::span view; + mutable std::atomic hashLow{0}; + mutable std::atomic hashHigh{0}; + mutable std::atomic hashed{false}; + }; + std::vector& Mutable() { - if (words == nullptr || words.use_count() != 1) { - words = std::make_shared>(Copy()); - owner.reset(); - view = {}; + if (holder == nullptr || holder.use_count() != 1 || holder->owner != nullptr) { + holder = std::make_shared(Copy(), nullptr, std::span{}); + } else { + holder->hashed.store(false, std::memory_order_relaxed); } - return *words; + return holder->words; } - std::shared_ptr> words; - std::shared_ptr owner; - std::span view; + std::shared_ptr holder; }; struct VertexInput { @@ -576,11 +625,43 @@ struct ShaderInvocation { std::uint32_t imageTableFaults = 0; }; +// What a Vulkan pipeline built from a result depends on besides the fixed-function state and the +// descriptor set layouts: the module words, the specialization constants handed to the driver, the +// subgroup width the module was laid out for, the workgroup memory slice and the bindings' shape. +// Two results with equal identities build equal pipelines, in this run or another: unlike the +// process-lifetime variant ids, the identity is the same on every run, so the pipeline stores key on +// it and a pipeline journal can name a stage across runs. Empty for a result without a module. +struct PipelineIdentity { + std::uint64_t low = 0; + std::uint64_t high = 0; + [[nodiscard]] bool Empty() const { return low == 0 && high == 0; } + bool operator==(const PipelineIdentity&) const = default; + auto operator<=>(const PipelineIdentity&) const = default; +}; + struct RecompileResult : CompiledShaderArtifact, ShaderInvocation { bool cacheHit = false; std::uint32_t workgroupMemoryDwords = 0; std::vector shaderDataUserWords; [[nodiscard]] std::uint64_t PipelineVariantId() const { return specializationId != 0 ? specializationId : variantId; } + [[nodiscard]] PipelineIdentity Identity() const { + const auto hash = spirv.ContentHash(); + if (hash == SpirvContentHash{}) return {}; + PipelineIdentity identity{hash.low, hash.high}; + const auto mix = [&](std::uint64_t value) { + identity.low = (identity.low ^ value) * 0x9e3779b97f4a7c15ull; + identity.low ^= identity.low >> 29u; + identity.high = (identity.high ^ (value + 0x7f4a7c159e3779b9ull)) * 0xc2b2ae3d27d4eb4full; + identity.high ^= identity.high >> 31u; + }; + mix(specialization.size()); + for (const auto& constant : specialization) mix(constant.id | (static_cast(constant.value) << 32u)); + mix(hostSubgroupSize | (static_cast(workgroupMemoryDwords) << 32u)); + mix(bindings.size()); + for (const auto& binding : bindings) mix(binding.descriptorSet | (static_cast(binding.binding) << 16u) | (static_cast(binding.kind) << 40u) | (static_cast(binding.count) << 48u)); + if (identity.Empty()) identity.low = 1; + return identity; + } }; [[nodiscard]] RecompileResult Recompile(const RecompileRequest& request); diff --git a/core/shader/recompiler/ShaderDiskCache.cpp b/core/shader/recompiler/ShaderDiskCache.cpp index de5f1e4f0a..3c288e94b6 100644 --- a/core/shader/recompiler/ShaderDiskCache.cpp +++ b/core/shader/recompiler/ShaderDiskCache.cpp @@ -84,9 +84,9 @@ std::filesystem::path ShaderCacheDirectory() { namespace ShaderRecompiler::ShaderDiskCache { #if defined(__linux__) && defined(__x86_64__) && defined(__GLIBCXX__) -static_assert(sizeof(CompiledShaderArtifact) == 264, "CompiledShaderArtifact changed: update the artifact encoder"); +static_assert(sizeof(CompiledShaderArtifact) == 232, "CompiledShaderArtifact changed: update the artifact encoder"); static_assert(sizeof(ShaderInvocation) == 200, "ShaderInvocation changed: update the invocation encoder"); -static_assert(sizeof(RecompileResult) == 496, "RecompileResult changed: update EncodeResult and DecodeResult"); +static_assert(sizeof(RecompileResult) == 464, "RecompileResult changed: update EncodeResult and DecodeResult"); static_assert(sizeof(DescriptorBinding) == 592, "DescriptorBinding changed: update the binding encoder"); static_assert(sizeof(VertexAttribute) == 32, "VertexAttribute changed: update the attribute encoder"); static_assert(sizeof(VertexInput) == 16, "VertexInput changed: update the vertex input encoder"); @@ -1697,8 +1697,8 @@ void EncodeResult(const RecompileResult& result, std::vector& out) { writer.Values(std::span(result.shaderDataUserWords)); } -bool DecodeResult(std::span bytes, RecompileResult& result) { - Reader reader(bytes); +bool DecodeResult(std::span bytes, RecompileResult& result, std::shared_ptr owner) { + Reader reader(bytes, std::move(owner)); decodeArtifact(reader, result); decodeInvocation(reader, result); reader.Value(result.workgroupMemoryDwords); diff --git a/core/shader/recompiler/ShaderDiskCache.hpp b/core/shader/recompiler/ShaderDiskCache.hpp index b03d85a809..7322efa496 100644 --- a/core/shader/recompiler/ShaderDiskCache.hpp +++ b/core/shader/recompiler/ShaderDiskCache.hpp @@ -53,7 +53,7 @@ void BuildKey(const RecompileRequest& request, std::uint32_t hostSubgroupSize, s [[nodiscard]] std::string EntryName(std::span key); void EncodeResult(const RecompileResult& result, std::vector& out); -[[nodiscard]] bool DecodeResult(std::span bytes, RecompileResult& result); +[[nodiscard]] bool DecodeResult(std::span bytes, RecompileResult& result, std::shared_ptr owner = nullptr); [[nodiscard]] std::vector EncodeEntry(std::span key, const CompiledVariant& variant); [[nodiscard]] LoadStatus DecodeEntry(std::span file, std::span key, CompiledVariant& variant, std::shared_ptr owner = nullptr); diff --git a/docs/dev/ARCHITECTURE.md b/docs/dev/ARCHITECTURE.md index 0a9a669545..d3f5deabad 100644 --- a/docs/dev/ARCHITECTURE.md +++ b/docs/dev/ARCHITECTURE.md @@ -95,6 +95,7 @@ flowchart LR - [`Recompiler.cpp`](../../core/shader/recompiler/Recompiler.cpp) runs the stages in this order. With `ANYPS5_ENABLE_SPIRV_TOOLS`, the SPIR-V is also validated and optimized with SPIRV-Tools. - `ShaderRecompiler::Recompile` keeps compiled variants in memory, and `ShaderDiskCache` stores them on disk so later runs reuse them. Before the writable cache, a miss is also looked up in read-only prebuilt packs and loose entry folders under `shader_cache/prebuilt`; pack entries are ordinary encoded cache entries behind a sorted key-hash index, validated by magic, compiler source version and index checksum when mapped, and a miss or rejected entry falls through to the writable cache and the compiler. Cache format 35 also stores the finite stencil-reference set, its replay-safety proof and reserved push-data word in the artifact and binding layouts. Entries from formats 31 to 34 are rejected. +- The Vulkan pipelines are journaled as well ([`PipelineJournal.cpp`](../../core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp)): a graphics pipeline the store builds (`CachedPipeline`) or a compute pipeline a dispatch builds is appended to `pipelines.journal` next to the shader cache entries, as the materialized result of each stage (the specialized SPIR-V, bindings and push constants, through `ShaderDiskCache::EncodeResult`) plus the recipe the pipeline was built from (`GraphicsPipelineRecipe`: the stages' identities, vertex input layout, descriptor set layout keys and the state fields the `Pipeline` constructor reads, `CanonicalPipelineState`). When a `VulkanDevice` is created, the journal is loaded and its pipelines are built on a pool of `hardware_concurrency() / 2` threads (`APS5_PIPELINE_JOURNAL_THREADS`) while the title loads, so the first draw or dispatch of a configuration seen on an earlier run finds its pipeline resident. The pipeline stores key on `RecompileResult::Identity`, a hash of the module words, specialization constants, subgroup width and bindings that is the same on every run (`SharedSpirv::ContentHash`, computed once per shared module), rather than on the process-lifetime variant ids, and a graphics pipeline is built outside the store's mutex, so the replay threads and the queue workers build in parallel. A draw that arrives before its replayed entry builds the pipeline itself; the first inserted wins. A replayed entry enters the store at its least recently used end and a full store takes none, so the title's own pipelines are never evicted for a replayed one, and the optimized relink of a replayed pipeline (`OptimizedPipeline::Schedule`) waits for the first draw that binds it, so the relink thread serves the title's pipelines first. A device created without a window starts its replay at its first draw or dispatch rather than at creation, since `Driver::AttachWindow` usually replaces it before the title draws. The journal records a pipeline once (modules and recipes are deduplicated by content), keeps at most `APS5_PIPELINE_JOURNAL_MAX_MIB` (1024), replays the first `APS5_PIPELINE_JOURNAL_ENTRIES` recorded (1024, the pipeline store's bound, `APS5_PIPELINE_STORE_ENTRIES`; a run reaches those first), and is disabled with `APS5_NO_PIPELINE_JOURNAL=1` or without a shader cache directory. The file header names the device (`PipelineJournalDeviceFingerprint`, the properties the driver's pipeline cache file is named by), so a journal written for another GPU or driver is started over rather than replayed; appends and reads hold `pipelines.journal.lock` against another process sharing the directory, and a device cancels its replay before it is replaced (`PrepareForReplacement`) or destroyed, so a replaced headless device never inserts entries the window's device then sweeps. `APS5_PROFILE_DRAW=1` prints a `[pipeline-journal]` summary when the replay finishes and at teardown. - A compute shader whose LDS plus its lock dword exceeds the host's `maxComputeSharedMemorySize` is not given a `Workgroup` array. `Recompile` sets `ShaderInfo::sharedMemoryBytes` for it and the SPIR-V backend addresses the LDS in a coherent storage buffer at descriptor set 1 (`WorkgroupMemoryDescriptorSet`), with a slice of `RecompileResult::workgroupMemoryDwords` per workgroup, indexed by `WorkgroupID` and `NumWorkGroups`. `VulkanDevice` keeps one such buffer, sized from the dispatch dimensions, and binds it after the resource set; programs that fit keep using shared memory. - `Driver::RegisterShader` splits shader preparation in two. `PlanRegistered` runs on the registering guest thread: it validates the registers and entry point, decodes the code and builds the recompile request, so invalid registrations still fail there. The RDNA to IR translation and resource plan (`PrepareShader`) run on a background pool of `hardware_concurrency() / 2` detached threads (at least 2, 64 MiB stacks). - Queue 0's packet worker decodes, captures and prepares a draw and hands its recording (`Graphics::Draw` under the GPU lock) to a committer thread in [`DrawPipeline.cpp`](../../core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawPipeline.cpp), so the next packet is decoded while the previous draw is recorded. Each queued item carries the guest ranges it will write (`DrawWriteRanges`: written buffers and storage images, color targets with their DCC and CMASK, depth and stencil). A later packet that reads or writes one of those ranges, or that the pipeline can't order, drains it first; labels, submission ends, buffer fills and (with `APS5_PIPELINE_DMA=1`) DMA packets are queued as items instead. An exception thrown by a commit is rethrown on the worker at its next enqueue or drain, as an inline draw error would be. On by default; `APS5_PIPELINED_DRAWS=0` records every draw inline, and `APS5_EXACT_DRAW_WRITES=0` widens the color and depth ranges to estimates. diff --git a/docs/dev/debt/libSceAgcDriver.md b/docs/dev/debt/libSceAgcDriver.md index 7e4ffe410b..d5c0f17655 100644 --- a/docs/dev/debt/libSceAgcDriver.md +++ b/docs/dev/debt/libSceAgcDriver.md @@ -64,6 +64,7 @@ - [Covered-centroid raster state](../../../core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp) (libSceAgcDriver) accepts PA_SC_AA_CONFIG.COVERED_CENTROID_IS_CENTER only when the pixel program is skipped or SPI_PS_INPUT_ENA and SPI_PS_INPUT_ADDR have no active perspective or linear centroid input. Bit 29 follows [PS5_Mesa GFX10.3 registers](https://github.com/mihawk-99/PS5_Mesa/blob/0b2d6d1a61d9bbf89cf8beb88a696144f67c61f8/src/amd/registers/gfx103.json); [RADV](https://github.com/mihawk-99/PS5_Mesa/blob/0b2d6d1a61d9bbf89cf8beb88a696144f67c61f8/src/amd/vulkan/radv_cmd_buffer.c) sets it on GFX10.3 and later. Active centroid interpolation and unspecified input registers still throw. Console covered-pixel centroid results were not measured. - [Compute LDS beyond the device's shared-memory limit](../../../core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp) (libSceAgcDriver) lives in a coherent storage buffer with one slice of `LDS bytes / 4 + 1` dwords per workgroup (the last dword is the atomic lock), shared by all such dispatches and grown in 1 MiB steps up to `maxStorageBufferRange`; a dispatch whose slices do not fit throws. It is slower than shared memory and its cost was not measured. An indirect dispatch of such a program always reads its group counts on the CPU. Its results are pinned against the in-limit semantics in `agc_driver_lds_beyond_device_limit`, not measured on hardware. - [AGC buffer allocations](../../../core/libs/prx/libSceAgcDriver/Graphics/src/BufferPool.cpp) retry host/device out-of-memory once after reclaiming idle pool buffers and fully unused slab blocks. Live buffers and blocks with live slots are retained. Live images and other retained resources have no global memory bound; allocation failure still throws when reclaim cannot satisfy the request. +- [Pipeline journal](../../../core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp) (libSceAgcDriver): the journal is append-only and never compacted, so a module a later run no longer builds pipelines from stays in the file until it is deleted, and the whole file is read into memory while the replay runs (its modules reference that buffer until the last replayed pipeline was built). Only the first `APS5_PIPELINE_JOURNAL_ENTRIES` pipelines recorded are replayed (a record is never rewritten, so a pipeline past the bound stays past it), and the pipeline store evicts the least recently used entry past `APS5_PIPELINE_STORE_ENTRIES` (1024 each): a title with more configurations rebuilds the rest at the draw as before. A compute pipeline is replayed over the set layout `DescribeSetLayouts` derives from the result's bindings, which a dispatch that finds it resident checks against its `ShaderResources` layout only under `APS5_VERIFY_SET_LAYOUTS=1`. Recording and replay are best effort: a record whose write fails is lost, an entry whose build fails is skipped, and both are counted in the `[pipeline-journal]` line rather than stopping the run. A replayed pipeline that the title never asks for again costs its build time and its driver memory (its optimized relink is deferred to its first draw, so a run whose first frames bind many replayed pipelines queues their relinks then); nothing ages recipes out of the file. Two runs sharing the file take turns under its lock file, but each only knows the modules it loaded or wrote itself, so a module the other run appended meanwhile may be appended again (the loader keeps the first copy). Records a run appends after another run's load are replayed by that run only after its next start. The file header's format number is the only guard against a change of the recipe codec, `CanonicalPipelineState` or the store key that keeps the structures' sizes (the recompiler's source version does not cover the driver's sources). - [Graphics pipeline libraries](../../../core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp) (libSceAgcDriver): a pipeline is fast-linked at the draw and relinked with `VK_PIPELINE_CREATE_LINK_TIME_OPTIMIZATION_BIT_EXT` on a background thread, and binds the optimized one from the next draw on. Draws recorded before that use the fast link, whose vertex stage keeps the outputs the fragment stage does not read: on RADV (Radeon 660M, Mesa 26.2.4) 346 of 483 Astro's Playroom vertex shaders paired with a fragment stage that reads none of them used more VGPRs (median 8 more, up to 2.5 times), and on an RTX 4090 13 of 15 PPSA36082 vertex shaders up to 68 instead of 38 (#2242). The optimized link compiles to the same code as a whole pipeline - [Prebuilt pipeline libraries](../../../core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp) (libSceAgcDriver): each registered vertex or pixel shader in registration form gets its pre-rasterization or fragment-shader library built on one background thread, with a fixed state the draw may not use: triangle list, counter-clockwise front face, no depth clamp, depth bias or conservative rasterization, first provoking vertex, one sample. Libraries are kept until the device is torn down; on Astro's Playroom 1948 of them added 0.44 GB of peak memory on RADV. - [`APS5_THREAD_PRIORITY=rt`](../../../core/libs/prx/libSceAgcDriver/Execution/src/ThreadPriority.cpp) (libSceAgcDriver) lowers the process's hard `RLIMIT_RTTIME` when it is above `APS5_THREAD_RTTIME_US` or rtkit's `RTTimeUSecMax`, because [rtkit](https://github.com/heftig/rtkit/blob/c295fa849f52b487be6433e69e08b46251950399/rtkit-daemon.c#L592-L637) (`verify_process_rttime`) reads the hard limit from `/proc//limits` and refuses real-time scheduling when it is higher. An unprivileged process cannot raise the hard limit again, so it stays lowered for the process and its threads even when rtkit then refuses the request (burst limit, user or start-time checks) and the thread throws. rtkit's `MaxRealtimePriority` and `RTTimeUSecMax` are read first, so an unreachable rtkit or a priority above its maximum leaves the limit untouched. The lowered limit only matters for threads that later run under a real-time policy diff --git a/docs/user/USAGE.md b/docs/user/USAGE.md index aa8af21f3b..88f4b7e449 100644 --- a/docs/user/USAGE.md +++ b/docs/user/USAGE.md @@ -179,6 +179,10 @@ HDR output is available on Windows only. Titles see it when the monitor showing Recompiled shaders and the Vulkan pipeline cache are kept in a `shader_cache/` directory beside the output executable, so later runs skip recompilation. Set `ANYPS5_SHADER_CACHE_DIR` to use another directory, or set `ANYPS5_NO_SHADER_CACHE` to any non-empty value other than `0` to disable both caches. Shaders are stored in a subdirectory named after the recompiler's source version; subdirectories of other versions unused for 14 days are removed the first time a run looks up a shader. The pipeline cache is one `vk-pipelines-*.bin` file per GPU and driver version. Deleting the directory is safe: it is rebuilt on the next run. +### Pipeline journal + +Besides the recompiled shaders, the shader cache directory holds `pipelines.journal`: every Vulkan graphics pipeline the pipeline store kept and every compute pipeline a run built, with the specialized shader modules they were built from. The next run of the title builds those pipelines on background threads while it loads, so a draw or dispatch the earlier run already made does not compile its pipeline when it is first reached. The journal belongs to the GPU and driver that wrote it, like the driver's pipeline cache file next to it: a run on another GPU or driver version starts it over, and the driver's own pipeline cache decides how fast each pipeline is rebuilt. Two runs sharing the directory take turns on `pipelines.journal.lock`. `APS5_NO_PIPELINE_JOURNAL=1` disables recording and replay; `APS5_PIPELINE_JOURNAL_THREADS` sets the replay threads (half the CPU threads by default; `0` records without replaying), `APS5_PIPELINE_JOURNAL_ENTRIES` how many pipelines are replayed (the first recorded, which a run reaches first) and `APS5_PIPELINE_STORE_ENTRIES` how many graphics pipelines stay resident (1024 each), and `APS5_PIPELINE_JOURNAL_MAX_MIB` (1024) the file size at which recording stops. Deleting the file loses nothing but the head start. With `APS5_PROFILE_DRAW=1` a `[pipeline-journal]` line reports what was loaded, replayed and recorded. + ### Storing GPU results at each flip GPU results in storage images and render targets stay on the GPU until something reads their memory. A title that frees such an image after waiting for its GPU work and reuses the memory can have its new data overwritten when the results are stored later ([TechnicalDebt](../dev/TechnicalDebt.md)). `APS5_STORE_AT_FLIP=1` stores every pending result when a display buffer is flipped, at the cost of one write-back per pending image and frame. Any other value stops the title at its first flip. From addcdc7c41ba0f8c0ce3339c766a840a7add249d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Wed, 14 Oct 2026 17:56:32 +0200 Subject: [PATCH 08/27] fix(agc): journal the sample location grid and per-sample shading of a pipeline #5861 added both to the pipeline state and its store key; the recipe dropped them, so a replayed pipeline keyed another configuration. Co-Authored-By: Claude --- .../Graphics/src/PipelineJournal.cpp | 8 +++++++- core/libs/prx/libSceAgcDriver/tests/Graphics.cpp | 13 +++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp index 940ff93847..c0475c6044 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp @@ -35,7 +35,7 @@ static_assert(sizeof(VkStencilOpState) == 28, "VkStencilOpState changed: update static_assert(sizeof(VkVertexInputBindingDescription) == 12, "VkVertexInputBindingDescription changed: update the recipe codec"); static_assert(sizeof(VkVertexInputAttributeDescription) == 16, "VkVertexInputAttributeDescription changed: update the recipe codec"); #if defined(__linux__) && defined(__x86_64__) && defined(__GLIBCXX__) -static_assert(sizeof(State) == 664, "State changed: update CanonicalPipelineState and the recipe codec"); +static_assert(sizeof(State) == 688, "State changed: update CanonicalPipelineState and the recipe codec"); static_assert(sizeof(ShaderStages) == 100, "ShaderStages changed: update the recipe codec"); static_assert(sizeof(ShaderRecompiler::MeshConfiguration) == 36, "MeshConfiguration changed: update the recipe codec"); static_assert(sizeof(ShaderRecompiler::TessellationConfiguration) == 32, "TessellationConfiguration changed: update the recipe codec"); @@ -207,6 +207,8 @@ void encodeState(Writer& writer, const State& state) { writer.U32(state.samples); for (const auto value : state.sampleLocations) writer.Raw(value); writer.Bool(state.customSampleLocations); + writer.Bool(state.sampleLocationGrid); + writer.Bool(state.perSampleShading); } void decodeState(Reader& reader, State& state) { @@ -260,6 +262,8 @@ void decodeState(Reader& reader, State& state) { state.samples = reader.U32(); for (auto& value : state.sampleLocations) value = reader.Raw(); state.customSampleLocations = reader.Bool(); + state.sampleLocationGrid = reader.Bool(); + state.perSampleShading = reader.Bool(); } void encodeRecipe(Writer& writer, const GraphicsPipelineRecipe& recipe) { @@ -993,6 +997,8 @@ State CanonicalPipelineState(const State& state) { canonical.samples = state.samples; canonical.sampleLocations = state.sampleLocations; canonical.customSampleLocations = state.customSampleLocations; + canonical.sampleLocationGrid = state.sampleLocationGrid; + canonical.perSampleShading = state.perSampleShading; return canonical; } diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 6a67b64204..da6f687548 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -7395,6 +7395,19 @@ void pipelineJournalTests() { const PipelineSetLayouts decodedLayouts{layouts.layouts.main, layouts.layouts.fragment, decoded.splitSets, decoded.mainLayoutKey, decoded.fragmentLayoutKey}; BuildPipelineStoreKey(replayed, context, decoded.state, decoded.vertexInput, decodedLayouts, shaders, decoded.attachmentLayout); Require(!original.empty() && original == canonical && original == replayed, "the canonical or the decoded state keys another pipeline"); + auto gridState = state; + gridState.samples = 2; + gridState.customSampleLocations = true; + gridState.sampleLocationGrid = true; + gridState.perSampleShading = true; + gridState.sampleLocations[31] = 0x5a; + GraphicsPipelineRecipe gridDecoded; + Require(DecodeGraphicsPipelineRecipe(EncodeGraphicsPipelineRecipe(MakeGraphicsPipelineRecipe(gridState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL)), gridDecoded), "the sample grid recipe does not decode"); + Require(gridDecoded.state.sampleLocationGrid && gridDecoded.state.perSampleShading && gridDecoded.state.sampleLocations[31] == 0x5a, "the recipe dropped the sample grid, per-sample shading or the last sample location"); + std::vector grid, gridReplayed; + BuildPipelineStoreKey(grid, context, gridState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + BuildPipelineStoreKey(gridReplayed, context, gridDecoded.state, gridDecoded.vertexInput, layouts.View(), shaders, gridDecoded.attachmentLayout); + Require(grid != original && grid == gridReplayed, "the decoded sample grid state keys another pipeline"); auto movedState = state; movedState.viewport.x = 5; movedState.colors[0].address = 0x9000; From 67f5350e251d1d42f93d1219ac108574725f0de3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Thu, 15 Oct 2026 16:03:50 +0200 Subject: [PATCH 09/27] fix(agc): keep the pipeline key buffer in a host thread-local slot MinGW's emulated TLS runs thread_local destructors from the module's TLS callback, which corrupts the heap when a replay thread or queue worker that ran CachedPipeline exits (0xC0000374, prx_thread_local_destructors). Co-Authored-By: Claude --- core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp index ec0251bcac..e79504441a 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp @@ -6,6 +6,7 @@ #include "prx/libSceAgcDriver/Graphics/include/DepthSurface.hpp" #include "prx/libSceAgcDriver/Graphics/include/PipelineSpecialization.hpp" #include "prx/libSceAgcDriver/Graphics/include/VertexInput.hpp" +#include "prx/libc/include/HostThreadLocal.hpp" #include #include #include @@ -974,7 +975,8 @@ std::shared_ptr CachedPipeline(const Context& context, const State& st if (created != nullptr) *created = false; if (disabled) return std::make_shared(context, state, vertexInput, resources, shaders, attachmentLayout); auto& store = Pipelines(); - thread_local std::vector key; + struct PipelineKeyStorage {}; + auto& key = HostThreadLocal, PipelineKeyStorage>(); pipelineKey(key, context, state, vertexInput, resources, shaders, attachmentLayout); if (key.empty()) { std::lock_guard lock(store.mutex); From 57ff6a63b22463fc7419fd1ecd0ad4ee4962b4ad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Thu, 15 Oct 2026 16:03:50 +0200 Subject: [PATCH 10/27] fix(agc): record nothing in the pipeline journal without its file lock Co-Authored-By: Claude --- .../libSceAgcDriver/Graphics/src/PipelineJournal.cpp | 12 ++++++++++++ docs/user/USAGE.md | 2 +- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp index c0475c6044..9882003802 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp @@ -367,6 +367,12 @@ public: FileLock(const FileLock&) = delete; FileLock& operator=(const FileLock&) = delete; +#ifdef _WIN32 + bool Held() const { return handle != INVALID_HANDLE_VALUE; } +#else + bool Held() const { return descriptor >= 0; } +#endif + private: #ifdef _WIN32 HANDLE handle = INVALID_HANDLE_VALUE; @@ -721,6 +727,12 @@ private: std::error_code error; std::filesystem::create_directories(path.parent_path(), error); FileLock fileLock(lockPath); + if (!fileLock.Held()) { + static std::atomic reported{false}; + if (!reported.exchange(true)) std::fprintf(stderr, "[pipeline-journal] cannot lock %s; pipelines are not recorded while it cannot be locked\n", lockPath.string().c_str()); + counters.writeFailures.fetch_add(1, std::memory_order_relaxed); + return true; + } bool fresh = false; { std::lock_guard lock(mutex); diff --git a/docs/user/USAGE.md b/docs/user/USAGE.md index 88f4b7e449..72772bf384 100644 --- a/docs/user/USAGE.md +++ b/docs/user/USAGE.md @@ -181,7 +181,7 @@ Recompiled shaders and the Vulkan pipeline cache are kept in a `shader_cache/` d ### Pipeline journal -Besides the recompiled shaders, the shader cache directory holds `pipelines.journal`: every Vulkan graphics pipeline the pipeline store kept and every compute pipeline a run built, with the specialized shader modules they were built from. The next run of the title builds those pipelines on background threads while it loads, so a draw or dispatch the earlier run already made does not compile its pipeline when it is first reached. The journal belongs to the GPU and driver that wrote it, like the driver's pipeline cache file next to it: a run on another GPU or driver version starts it over, and the driver's own pipeline cache decides how fast each pipeline is rebuilt. Two runs sharing the directory take turns on `pipelines.journal.lock`. `APS5_NO_PIPELINE_JOURNAL=1` disables recording and replay; `APS5_PIPELINE_JOURNAL_THREADS` sets the replay threads (half the CPU threads by default; `0` records without replaying), `APS5_PIPELINE_JOURNAL_ENTRIES` how many pipelines are replayed (the first recorded, which a run reaches first) and `APS5_PIPELINE_STORE_ENTRIES` how many graphics pipelines stay resident (1024 each), and `APS5_PIPELINE_JOURNAL_MAX_MIB` (1024) the file size at which recording stops. Deleting the file loses nothing but the head start. With `APS5_PROFILE_DRAW=1` a `[pipeline-journal]` line reports what was loaded, replayed and recorded. +Besides the recompiled shaders, the shader cache directory holds `pipelines.journal`: every Vulkan graphics pipeline the pipeline store kept and every compute pipeline a run built, with the specialized shader modules they were built from. The next run of the title builds those pipelines on background threads while it loads, so a draw or dispatch the earlier run already made does not compile its pipeline when it is first reached. The journal belongs to the GPU and driver that wrote it, like the driver's pipeline cache file next to it: a run on another GPU or driver version starts it over, and the driver's own pipeline cache decides how fast each pipeline is rebuilt. Two runs sharing the directory take turns on `pipelines.journal.lock`; a run that cannot lock it records nothing. `APS5_NO_PIPELINE_JOURNAL=1` disables recording and replay; `APS5_PIPELINE_JOURNAL_THREADS` sets the replay threads (half the CPU threads by default; `0` records without replaying), `APS5_PIPELINE_JOURNAL_ENTRIES` how many pipelines are replayed (the first recorded, which a run reaches first) and `APS5_PIPELINE_STORE_ENTRIES` how many graphics pipelines stay resident (1024 each), and `APS5_PIPELINE_JOURNAL_MAX_MIB` (1024) the file size at which recording stops. Deleting the file loses nothing but the head start. With `APS5_PROFILE_DRAW=1` a `[pipeline-journal]` line reports what was loaded, replayed and recorded. ### Storing GPU results at each flip From 52d8d6ce28543a8118d19281bf20bcae04967318 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Thu, 15 Oct 2026 16:09:17 +0200 Subject: [PATCH 11/27] fix(agc): journal the pipeline state fields main added The replayed stencil reference, the reversed blend, the quad sample masks and the tessellation instance patches and draw parameters change the pipeline a draw creates, so the recipe carries them; the file format is bumped and the layout guards follow the new sizes. Co-Authored-By: Claude --- .../Graphics/src/PipelineJournal.cpp | 21 +++++++++++++++---- .../prx/libSceAgcDriver/tests/Graphics.cpp | 14 +++++++++++++ 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp index 9882003802..7bffdeffdd 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp @@ -35,16 +35,16 @@ static_assert(sizeof(VkStencilOpState) == 28, "VkStencilOpState changed: update static_assert(sizeof(VkVertexInputBindingDescription) == 12, "VkVertexInputBindingDescription changed: update the recipe codec"); static_assert(sizeof(VkVertexInputAttributeDescription) == 16, "VkVertexInputAttributeDescription changed: update the recipe codec"); #if defined(__linux__) && defined(__x86_64__) && defined(__GLIBCXX__) -static_assert(sizeof(State) == 688, "State changed: update CanonicalPipelineState and the recipe codec"); -static_assert(sizeof(ShaderStages) == 100, "ShaderStages changed: update the recipe codec"); +static_assert(sizeof(State) == 752, "State changed: update CanonicalPipelineState and the recipe codec"); +static_assert(sizeof(ShaderStages) == 108, "ShaderStages changed: update the recipe codec"); static_assert(sizeof(ShaderRecompiler::MeshConfiguration) == 36, "MeshConfiguration changed: update the recipe codec"); -static_assert(sizeof(ShaderRecompiler::TessellationConfiguration) == 32, "TessellationConfiguration changed: update the recipe codec"); +static_assert(sizeof(ShaderRecompiler::TessellationConfiguration) == 40, "TessellationConfiguration changed: update the recipe codec"); #endif constexpr std::uint32_t FileMagic = 0x4c4a5041u; // Bumped when the recipe codec, CanonicalPipelineState or the store key change in a way the // layout guards above do not catch (a field renumbered or reinterpreted at the same size). -constexpr std::uint32_t FileFormat = 1u; +constexpr std::uint32_t FileFormat = 2u; constexpr std::uint32_t ModuleRecord = 1u; constexpr std::uint32_t GraphicsRecord = 2u; constexpr std::uint32_t ComputeRecord = 3u; @@ -157,11 +157,15 @@ void decodeMesh(Reader& reader, ShaderRecompiler::MeshConfiguration& mesh) { void encodeTessellation(Writer& writer, const ShaderRecompiler::TessellationConfiguration& tessellation) { for (const auto value : {tessellation.inputControlPoints, tessellation.outputControlPoints, tessellation.domain, tessellation.partitioning, tessellation.outputTopology, tessellation.patchesPerGroup, tessellation.primitiveGroupSize}) writer.U32(value); writer.Bool(tessellation.breakAtInstance); + writer.U32(tessellation.instancePatches); + writer.Bool(tessellation.drawParameters); } void decodeTessellation(Reader& reader, ShaderRecompiler::TessellationConfiguration& tessellation) { for (auto* value : {&tessellation.inputControlPoints, &tessellation.outputControlPoints, &tessellation.domain, &tessellation.partitioning, &tessellation.outputTopology, &tessellation.patchesPerGroup, &tessellation.primitiveGroupSize}) *value = reader.U32(); tessellation.breakAtInstance = reader.Bool(); + tessellation.instancePatches = reader.U32(); + tessellation.drawParameters = reader.Bool(); } void encodeState(Writer& writer, const State& state) { @@ -185,6 +189,7 @@ void encodeState(Writer& writer, const State& state) { writer.Bool(state.stencilTest); writer.Raw(state.stencilFront); writer.Raw(state.stencilBack); + writer.Bool(state.stencilReplayReference.has_value()); writer.U64(state.colors.size()); for (const auto& color : state.colors) { writer.U32(static_cast(color.format)); @@ -196,6 +201,7 @@ void encodeState(Writer& writer, const State& state) { writer.Bool(state.hasColorTarget); writer.Bool(state.rectList); writer.Bool(state.dualSourceBlend); + writer.U32(state.reversedBlend); writer.U32(static_cast(state.topology)); writer.Bool(state.primitiveRestart); writer.Bool(state.negativeOneToOne); @@ -209,6 +215,7 @@ void encodeState(Writer& writer, const State& state) { writer.Bool(state.customSampleLocations); writer.Bool(state.sampleLocationGrid); writer.Bool(state.perSampleShading); + for (const auto mask : state.quadSampleMask) writer.Raw(mask); } void decodeState(Reader& reader, State& state) { @@ -233,6 +240,7 @@ void decodeState(Reader& reader, State& state) { state.stencilTest = reader.Bool(); state.stencilFront = reader.Raw(); state.stencilBack = reader.Raw(); + if (reader.Bool()) state.stencilReplayReference = 0u; const auto colors = reader.U64(); if (!reader.Ok() || colors > 8u) { reader.Fail(); @@ -251,6 +259,7 @@ void decodeState(Reader& reader, State& state) { state.hasColorTarget = reader.Bool(); state.rectList = reader.Bool(); state.dualSourceBlend = reader.Bool(); + state.reversedBlend = reader.U32(); state.topology = static_cast(reader.U32()); state.primitiveRestart = reader.Bool(); state.negativeOneToOne = reader.Bool(); @@ -264,6 +273,7 @@ void decodeState(Reader& reader, State& state) { state.customSampleLocations = reader.Bool(); state.sampleLocationGrid = reader.Bool(); state.perSampleShading = reader.Bool(); + for (auto& mask : state.quadSampleMask) mask = reader.Raw(); } void encodeRecipe(Writer& writer, const GraphicsPipelineRecipe& recipe) { @@ -986,6 +996,7 @@ State CanonicalPipelineState(const State& state) { canonical.stencilTest = state.stencilTest; canonical.stencilFront = state.stencilFront; canonical.stencilBack = state.stencilBack; + if (state.stencilReplayReference) canonical.stencilReplayReference = 0u; for (const auto& color : state.colors) { ColorTarget canonicalColor{}; canonicalColor.format = color.format; @@ -998,6 +1009,7 @@ State CanonicalPipelineState(const State& state) { canonical.hasColorTarget = state.hasColorTarget; canonical.rectList = state.rectList; canonical.dualSourceBlend = state.dualSourceBlend; + canonical.reversedBlend = state.reversedBlend; canonical.topology = state.topology; canonical.primitiveRestart = state.primitiveRestart; canonical.negativeOneToOne = state.negativeOneToOne; @@ -1011,6 +1023,7 @@ State CanonicalPipelineState(const State& state) { canonical.customSampleLocations = state.customSampleLocations; canonical.sampleLocationGrid = state.sampleLocationGrid; canonical.perSampleShading = state.perSampleShading; + canonical.quadSampleMask = state.quadSampleMask; return canonical; } diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index da6f687548..139c5247d3 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -7408,6 +7408,20 @@ void pipelineJournalTests() { BuildPipelineStoreKey(grid, context, gridState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); BuildPipelineStoreKey(gridReplayed, context, gridDecoded.state, gridDecoded.vertexInput, layouts.View(), shaders, gridDecoded.attachmentLayout); Require(grid != original && grid == gridReplayed, "the decoded sample grid state keys another pipeline"); + auto fourSampleState = state; + fourSampleState.samples = 4; + auto maskState = fourSampleState; + maskState.quadSampleMask = {0x3u, 0x3u, 0x3u, 0x3u}; + maskState.stencilReplayReference = 7u; + maskState.reversedBlend = 1u; + GraphicsPipelineRecipe maskDecoded; + Require(DecodeGraphicsPipelineRecipe(EncodeGraphicsPipelineRecipe(MakeGraphicsPipelineRecipe(maskState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL)), maskDecoded), "the sample mask recipe does not decode"); + Require(maskDecoded.state.quadSampleMask == maskState.quadSampleMask && maskDecoded.state.stencilReplayReference.has_value() && maskDecoded.state.reversedBlend == 1u, "the recipe dropped the quad sample masks, the replayed stencil reference or the reversed blend"); + std::vector fourSamples, masked, maskReplayed; + BuildPipelineStoreKey(fourSamples, context, fourSampleState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + BuildPipelineStoreKey(masked, context, maskState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); + BuildPipelineStoreKey(maskReplayed, context, maskDecoded.state, maskDecoded.vertexInput, layouts.View(), shaders, maskDecoded.attachmentLayout); + Require(masked != fourSamples && masked == maskReplayed, "the decoded sample mask state keys another pipeline"); auto movedState = state; movedState.viewport.x = 5; movedState.colors[0].address = 0x9000; From 1ad590fc054db4c767c411c56ce8566e40b8b1bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Thu, 15 Oct 2026 16:27:15 +0200 Subject: [PATCH 12/27] test(agc): give the new program of compute_pipeline_outside_gpu_mutex its own module Pipelines are now shared by every program that recompiles to the same module, and a leading s_nop recompiles to the same one, so the second dispatch reused the first pipeline. The new program also stores to the second word. Co-Authored-By: Claude --- .../tests/execution/ComputePipelineOutsideGpuMutex.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/tests/execution/ComputePipelineOutsideGpuMutex.cpp b/core/libs/prx/libSceAgcDriver/tests/execution/ComputePipelineOutsideGpuMutex.cpp index 95513fdc4b..dd464ab921 100644 --- a/core/libs/prx/libSceAgcDriver/tests/execution/ComputePipelineOutsideGpuMutex.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/execution/ComputePipelineOutsideGpuMutex.cpp @@ -21,7 +21,7 @@ void Require(bool condition, const std::string& message) { } alignas(256) std::array ReaderCode{0xf4040100u, 0xfa000000u, 0xf4080201u, 0xfa000000u, 0xbf8cc07fu, 0xf4000182u, 0xfa000000u, 0xbf8cc07fu, 0x7e000206u, 0xe0700000u, 0x80020000u, 0xbf810000u}; -alignas(256) std::array NewReaderCode{0xbf800000u, 0xf4040100u, 0xfa000000u, 0xf4080201u, 0xfa000000u, 0xbf8cc07fu, 0xf4000182u, 0xfa000000u, 0xbf8cc07fu, 0x7e000206u, 0xe0700000u, 0x80020000u, 0xbf810000u}; +alignas(256) std::array NewReaderCode{0xf4040100u, 0xfa000000u, 0xf4080201u, 0xfa000000u, 0xbf8cc07fu, 0xf4000182u, 0xfa000000u, 0xbf8cc07fu, 0x7e000206u, 0xe0700000u, 0x80020000u, 0xe0700004u, 0x80020000u, 0xbf810000u}; struct Slot { alignas(256) std::array value{}; @@ -104,7 +104,7 @@ int main() { Require(Slots[1].done == 0, "the dispatch finished while another thread held the GPU mutex"); } WaitDone(Slots[1], "the dispatch of the new program did not finish"); - Require(Slots[1].output[0] == 0x5678u, "the dispatch of the new program stored " + std::to_string(Slots[1].output[0])); + Require(Slots[1].output[0] == 0x5678u && Slots[1].output[1] == 0x5678u, "the dispatch of the new program stored " + std::to_string(Slots[1].output[0]) + " and " + std::to_string(Slots[1].output[1])); const auto after = VulkanDevice::ComputePipelineCounts(); Require(after.underGpuMutex == before.underGpuMutex, "a compute pipeline was created under the GPU mutex"); std::cout << "compute pipeline outside GPU mutex test passed\n"; From fa30a9d5d296855f39678a251d79e0f7e2156a60 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Fri, 16 Oct 2026 11:19:37 +0200 Subject: [PATCH 13/27] fix(agc): do not create the pipeline journal at driver shutdown Flushing at shutdown built the singleton and its writer thread on a run that never recorded anything; on Windows that throws std::system_error while the process is exiting. --- .../Graphics/src/PipelineJournal.cpp | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp index 7bffdeffdd..094b1ed9bd 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp @@ -421,10 +421,12 @@ struct LoadedJournal { class Journal { public: static Journal& Instance() { - static auto* journal = new Journal(); + static auto* journal = Create(); return *journal; } + static Journal* Existing() { return created().load(std::memory_order_acquire); } + [[nodiscard]] bool Enabled() const { return !path.empty() && !mismatched.load(std::memory_order_relaxed); } [[nodiscard]] const std::filesystem::path& Path() const { return path; } @@ -544,6 +546,17 @@ private: std::vector payload; }; + static std::atomic& created() { + static std::atomic pointer{nullptr}; + return pointer; + } + + static Journal* Create() { + auto* journal = new Journal(); + created().store(journal, std::memory_order_release); + return journal; + } + Journal() { const char* off = std::getenv("APS5_NO_PIPELINE_JOURNAL"); if ((off != nullptr && std::strcmp(off, "0") != 0) || !ShaderRecompiler::ShaderDiskCache::Enabled()) return; @@ -1098,7 +1111,8 @@ void RecordComputePipeline(const ShaderRecompiler::RecompileResult& shader, bool } void FlushPipelineJournal() { - if (Journal::Instance().Enabled()) Journal::Instance().Flush(); + auto* journal = Journal::Existing(); + if (journal != nullptr && journal->Enabled()) journal->Flush(); } void ReportPipelineJournal() { From 7041ba8d9a9fdf8b30895fe1da2aec3cd68d0718 Mon Sep 17 00:00:00 2001 From: abnergtz <63434566+abnergtz@users.noreply.github.com> Date: Sat, 10 Oct 2026 23:52:43 -0300 Subject: [PATCH 14/27] fix(recompiler): terminate pixel invocation on BDA fault when supported Pixel invocations stopped by a BDA fault now end with OpTerminateInvocation when the target offers SPV_KHR_terminate_invocation, and keep OpKill otherwise. OpKill is not enough on AMD driver branch 21045, where a guest atomic after the kill still lands once per faulting pixel. Based on the approach in poonasor/AnyPS5@73969879. Refs #2009 --- .../prx/libSceAgcDriver/tests/Graphics.cpp | 23 +++++++++++++++---- .../SpirvBackend/src/SpirvBdaFault.cpp | 10 +++++++- 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 2e1d26e11f..f90bcdee3b 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -5592,7 +5592,7 @@ std::string interlockShape(std::span words) { return shape; } -std::string interlockExits(std::span words, std::size_t& kills) { +std::string interlockExits(std::span words, std::size_t& kills, std::size_t& terminates) { struct Block { std::vector ops; std::vector next; @@ -5610,7 +5610,7 @@ std::string interlockExits(std::span words, std::size_t& ki if (op == spv::OpLabel) label = words[at + 1]; if (op == spv::OpLabel && function == entry && first == 0) first = label; if (function != entry) { - if (op == spv::OpBeginInvocationInterlockEXT || op == spv::OpEndInvocationInterlockEXT || op == spv::OpKill) return "an interlock instruction or OpKill outside the entry point"; + if (op == spv::OpBeginInvocationInterlockEXT || op == spv::OpEndInvocationInterlockEXT || op == spv::OpKill || op == spv::OpTerminateInvocation) return "an interlock instruction, OpKill or OpTerminateInvocation outside the entry point"; continue; } if (label == 0) continue; @@ -5634,6 +5634,7 @@ std::string interlockExits(std::span words, std::size_t& ki if (op == spv::OpEndInvocationInterlockEXT && stage++ != 1) return "block " + std::to_string(current) + " ends the interlock outside it"; if ((op == spv::OpKill || op == spv::OpReturn || op == spv::OpReturnValue || op == spv::OpUnreachable || op == spv::OpTerminateInvocation) && stage != 2) return "an exit in block " + std::to_string(current) + " skips OpEndInvocationInterlockEXT"; if (op == spv::OpKill) ++kills; + if (op == spv::OpTerminateInvocation) ++terminates; } for (const auto next : blocks[current].next) { const auto [known, added] = state.emplace(next, stage); @@ -5682,9 +5683,21 @@ void orderedPixelShaderTests() { auto faulting = orderedPixelRequest(0x30600u, bdaInterlock, OrderedBdaPixelCode); faulting.target.supportedExtensions = bdaExtensions; faulting.target.bdaAbiVersion = ShaderRecompiler::BdaAbi::Version; - std::size_t kills = 0; - const auto exits = interlockExits(ShaderRecompiler::Recompile(faulting).spirv.Words(), kills); - Require(exits.empty() && kills == 6u, "a primitive-ordered pixel shader with BDA reads: " + (exits.empty() ? std::to_string(kills) + " OpKill, not the valid-mask kill and 5 BDA fault kills" : exits)); + std::size_t kills = 0, terminates = 0; + const auto exits = interlockExits(ShaderRecompiler::Recompile(faulting).spirv.Words(), kills, terminates); + Require(exits.empty() && kills == 6u && terminates == 0u, "a primitive-ordered pixel shader with BDA reads: " + (exits.empty() ? std::to_string(kills) + " OpKill and " + std::to_string(terminates) + " OpTerminateInvocation, not the valid-mask kill and 5 BDA fault kills" : exits)); + const std::array terminateExtensions{"SPV_KHR_physical_storage_buffer", "SPV_KHR_8bit_storage", "SPV_KHR_terminate_invocation"}; + auto terminating = faulting; + terminating.target.supportedExtensions = terminateExtensions; + const auto terminated = ShaderRecompiler::Recompile(terminating).spirv.Words(); + std::size_t terminatedKills = 0, terminatedStops = 0; + const auto terminatedExits = interlockExits(terminated, terminatedKills, terminatedStops); + Require(terminatedExits.empty() && terminatedKills == 1u && terminatedStops == 5u, "a primitive-ordered pixel shader with BDA reads and SPV_KHR_terminate_invocation: " + (terminatedExits.empty() ? std::to_string(terminatedKills) + " OpKill and " + std::to_string(terminatedStops) + " OpTerminateInvocation, not the valid-mask kill and 5 BDA fault terminations" : terminatedExits)); + std::size_t terminateDeclarations = 0; + for (std::size_t at = 5; at < terminated.size() && (terminated[at] >> 16u) != 0; at += terminated[at] >> 16u) { + if ((terminated[at] & 0xffffu) == spv::OpExtension && std::string_view(reinterpret_cast(&terminated[at + 1])) == "SPV_KHR_terminate_invocation") ++terminateDeclarations; + } + Require(terminateDeclarations == 1u, "the module declared SPV_KHR_terminate_invocation " + std::to_string(terminateDeclarations) + " times"); expectFailure([&] { static_cast(ShaderRecompiler::Recompile(orderedPixelRequest(0x30600u, plain))); }, "fragmentShaderPixelInterlock"); } diff --git a/core/shader/recompiler/SpirvBackend/src/SpirvBdaFault.cpp b/core/shader/recompiler/SpirvBackend/src/SpirvBdaFault.cpp index 4b0d60d921..3ae3e3744e 100644 --- a/core/shader/recompiler/SpirvBackend/src/SpirvBdaFault.cpp +++ b/core/shader/recompiler/SpirvBackend/src/SpirvBdaFault.cpp @@ -1,6 +1,7 @@ #include "SpirvBackend/SpirvBda.hpp" #include "SpirvBackend/SpirvMemory/SpirvTypes.hpp" #include "SpirvBackend/SpirvMemory/SpirvConstants.hpp" +#include #include namespace ShaderRecompiler { @@ -115,7 +116,14 @@ void StopBdaInvocationIf(SpirvEmitterState& state, std::uint32_t condition) { if (OrderedPixelShader(state)) { state.module.AddFunction(spv::OpEndInvocationInterlockEXT); } - state.module.AddFunction(state.program.Resources().stage == IrShaderStage::Pixel ? spv::OpKill : spv::OpReturn); + if (state.program.Resources().stage != IrShaderStage::Pixel) { + state.module.AddFunction(spv::OpReturn); + } else if (std::ranges::find(state.supportedExtensions, "SPV_KHR_terminate_invocation") != state.supportedExtensions.end()) { + state.module.EmitExtension("SPV_KHR_terminate_invocation"); + state.module.AddFunction(spv::OpTerminateInvocation); + } else { + state.module.AddFunction(spv::OpKill); + } } EmitLabel(state, next); } From 42cd3f2ec4f28e99c2ac40766a34e82c7b181ebe Mon Sep 17 00:00:00 2001 From: abnergtz <63434566+abnergtz@users.noreply.github.com> Date: Sat, 10 Oct 2026 23:53:56 -0300 Subject: [PATCH 15/27] feat(agc): gate SPV_KHR_terminate_invocation on shaderTerminateInvocation Shader validation accepts the extension only in fragment shaders and only when the device enabled shaderTerminateInvocation. The flag is part of the validation memo key. Refs #2009 --- .../Graphics/include/Context.hpp | 1 + .../Graphics/include/Pipeline.hpp | 2 +- .../prx/libSceAgcDriver/Graphics/src/Draw.cpp | 3 +- .../Graphics/src/ShaderValidation.cpp | 31 +++++++++---------- .../prx/libSceAgcDriver/tests/Graphics.cpp | 11 ++++++- 5 files changed, 28 insertions(+), 20 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp b/core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp index cba5321c23..886e67680c 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp @@ -232,6 +232,7 @@ struct Context { VkSampleCountFlags sampleLocationSampleCounts = 0; VkSampleCountFlags sampleLocationGridSampleCounts = 0; PipelineCacheSynchronization* pipelineCacheSynchronization = nullptr; + bool shaderTerminateInvocation = false; PipelineCacheSynchronization::PipelineUseLock LockPipelineCacheForUse() const { return pipelineCacheSynchronization != nullptr ? pipelineCacheSynchronization->LockForPipelineUse() : PipelineCacheSynchronization::PipelineUseLock{}; diff --git a/core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp b/core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp index a195283b06..e588f58b67 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp @@ -126,7 +126,7 @@ bool ShadersWriteMemory(std::span shaders); bool DeclaresEarlyFragmentTests(const ShaderRecompiler::RecompileResult& program); void ValidateSampleMask(const Context& context, const State& state, std::span shaders); // Returns the color attachment locations the pixel shader writes. -std::set ValidateShaders(std::span shaders, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric, bool descriptorIndexing = false, bool imageInt64Atomics = false, bool geometryShader = false, bool sampleRateShading = false, bool bufferInt64Atomics = false, bool shaderClipDistance = false, bool shaderCullDistance = false, bool viewportIndexLayer = false, bool runtimeDescriptorArray = false, bool demoteToHelperInvocation = false); +std::set ValidateShaders(std::span shaders, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric, bool descriptorIndexing = false, bool imageInt64Atomics = false, bool geometryShader = false, bool sampleRateShading = false, bool bufferInt64Atomics = false, bool shaderClipDistance = false, bool shaderCullDistance = false, bool viewportIndexLayer = false, bool runtimeDescriptorArray = false, bool demoteToHelperInvocation = false, bool shaderTerminateInvocation = false); } diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp index 5ad7489d14..7eb39c07a6 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/Draw.cpp @@ -646,6 +646,7 @@ bool ValidationKey(const Context& context, std::span shade add(context.shaderClipDistance); add(context.shaderCullDistance); add(context.viewportIndexLayer); + add(context.shaderTerminateInvocation); add(state.paClVsOutCntl); add(state.stages.mesh.has_value()); if (state.stages.mesh) { @@ -701,7 +702,7 @@ std::set CachedFragmentOutputs(const Context& context, std::span< } std::set outputs; try { - outputs = ValidateShaders(shaders, state, context.subgroup, context.fragmentShaderBarycentric, context.descriptorIndexing, context.imageInt64Atomics, context.geometryShader, context.sampleRateShading, context.bufferInt64Atomics, context.shaderClipDistance, context.shaderCullDistance, context.viewportIndexLayer, context.runtimeDescriptorArray, context.demoteToHelperInvocation); + outputs = ValidateShaders(shaders, state, context.subgroup, context.fragmentShaderBarycentric, context.descriptorIndexing, context.imageInt64Atomics, context.geometryShader, context.sampleRateShading, context.bufferInt64Atomics, context.shaderClipDistance, context.shaderCullDistance, context.viewportIndexLayer, context.runtimeDescriptorArray, context.demoteToHelperInvocation, context.shaderTerminateInvocation); } catch (const std::exception& error) { if (keyed) { std::lock_guard lock(validationMutex()); diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp index 9c7dcb033e..0862785621 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp @@ -151,7 +151,7 @@ struct Module { } }; -Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric, bool descriptorIndexing, bool imageInt64Atomics, bool geometryShader, bool sampleRateShading, bool bufferInt64Atomics, bool shaderClipDistance, bool shaderCullDistance, bool viewportIndexLayer, bool runtimeDescriptorArray, bool demoteToHelperInvocation) { +Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric, bool descriptorIndexing, bool imageInt64Atomics, bool geometryShader, bool sampleRateShading, bool bufferInt64Atomics, bool shaderClipDistance, bool shaderCullDistance, bool viewportIndexLayer, bool runtimeDescriptorArray, bool demoteToHelperInvocation, bool shaderTerminateInvocation) { using Stage = ShaderRecompiler::ShaderStage; Require(compiled.program != nullptr, "missing compiled shader"); const auto& shader = *compiled.program; @@ -185,6 +185,7 @@ Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysi std::uint32_t entryPoint = 0; std::set executionModeTargets; bool upperLeft = false; + bool terminateExtension = false; for (std::size_t cursor = 5; cursor < words.size();) { const auto count = words[cursor] >> 16u; Require(count != 0 && count <= words.size() - cursor, "truncated SPIR-V instruction"); @@ -331,9 +332,19 @@ Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysi Require(fragment && fragmentShaderBarycentric, "SPV_KHR_fragment_shader_barycentric requires enabled fragmentShaderBarycentric in a fragment shader"); break; } + if (extension == "SPV_KHR_terminate_invocation") { + Require(fragment && shaderTerminateInvocation, "SPV_KHR_terminate_invocation requires enabled shaderTerminateInvocation in a fragment shader"); + terminateExtension = true; + break; + } Require(extension == "SPV_KHR_float_controls" || (imageInt64Atomics && extension == "SPV_EXT_shader_image_int64") || (viewportIndexLayer && extension == "SPV_EXT_shader_viewport_index_layer") || (fragment && demoteToHelperInvocation && extension == "SPV_EXT_demote_to_helper_invocation") || (mesh && (extension == "SPV_EXT_mesh_shader" || extension == "SPV_KHR_physical_storage_buffer")) || ((descriptorIndexing || runtimeDescriptorArray) && extension == "SPV_EXT_descriptor_indexing") || (shader.bdaAbiVersion == ShaderRecompiler::BdaAbi::Version && (extension == "SPV_KHR_physical_storage_buffer" || extension == "SPV_KHR_8bit_storage")), "unsupported SPIR-V extension"); break; } + case spv::OpTerminateInvocation: + Require(count == 1, "invalid OpTerminateInvocation instruction"); + Require(fragment && shaderTerminateInvocation, "OpTerminateInvocation requires enabled shaderTerminateInvocation in a fragment shader"); + Require(terminateExtension, "OpTerminateInvocation requires SPV_KHR_terminate_invocation"); + break; case spv::OpDecorateId: case spv::OpDecorationGroup: case spv::OpGroupDecorate: @@ -601,21 +612,7 @@ Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysi } -bool DeclaresEarlyFragmentTests(const ShaderRecompiler::RecompileResult& program) { - const auto& words = program.spirv.Words(); - Require(words.size() >= 5 && words[0] == spv::MagicNumber, "the fragment shader is not a SPIR-V module"); - for (std::size_t cursor = 5; cursor < words.size();) { - const auto count = words[cursor] >> 16u; - Require(count != 0 && count <= words.size() - cursor, "truncated SPIR-V instruction"); - const auto op = static_cast(words[cursor] & 0xffffu); - if (op == spv::OpFunction) return false; - if (op == spv::OpExecutionMode && count >= 3 && words[cursor + 2] == spv::ExecutionModeEarlyFragmentTests) return true; - cursor += count; - } - return false; -} - -std::set ValidateShaders(std::span shaders, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric, bool descriptorIndexing, bool imageInt64Atomics, bool geometryShader, bool sampleRateShading, bool bufferInt64Atomics, bool shaderClipDistance, bool shaderCullDistance, bool viewportIndexLayer, bool runtimeDescriptorArray, bool demoteToHelperInvocation) { +std::set ValidateShaders(std::span shaders, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric, bool descriptorIndexing, bool imageInt64Atomics, bool geometryShader, bool sampleRateShading, bool bufferInt64Atomics, bool shaderClipDistance, bool shaderCullDistance, bool viewportIndexLayer, bool runtimeDescriptorArray, bool demoteToHelperInvocation, bool shaderTerminateInvocation) { using Stage = ShaderRecompiler::ShaderStage; const bool tessellation = state.stages.path == ShaderPath::Tessellation; const bool mesh = state.stages.path == ShaderPath::Geometry; @@ -636,7 +633,7 @@ std::set ValidateShaders(std::span shaders, Require(shaders[i].program != nullptr, "missing compiled shader"); Require(shaders[i].stage == expected, "graphics stage order disagrees"); for (const auto& binding : shaders[i].program->bindings) Require(binding.descriptorSet == ShaderRecompiler::RuntimeAbi::DescriptorSet || (shaders[i].stage == ShaderRecompiler::ShaderStage::Fragment && binding.descriptorSet == ShaderRecompiler::RuntimeAbi::FragmentDescriptorSet), "graphics resource uses a descriptor set other than its stage's"); - const auto current = Inspect(shaders[i], state, subgroup, fragmentShaderBarycentric, descriptorIndexing, imageInt64Atomics, geometryShader, sampleRateShading, bufferInt64Atomics, shaderClipDistance, shaderCullDistance, viewportIndexLayer, runtimeDescriptorArray, demoteToHelperInvocation); + const auto current = Inspect(shaders[i], state, subgroup, fragmentShaderBarycentric, descriptorIndexing, imageInt64Atomics, geometryShader, sampleRateShading, bufferInt64Atomics, shaderClipDistance, shaderCullDistance, viewportIndexLayer, runtimeDescriptorArray, demoteToHelperInvocation, shaderTerminateInvocation); if (i != 0) { for (const auto& [location, signature] : current.inputs) { const auto output = previous.outputs.find(location); diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index f90bcdee3b..9452795be1 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -4783,6 +4783,7 @@ struct ModuleShape { std::uint32_t clipDistanceLength = 0; std::uint32_t cullDistanceLength = 0; bool layerOutput = false; + bool terminateInvocation = false; }; void emit(std::vector& out, spv::Op op, std::initializer_list operands) { @@ -4977,7 +4978,7 @@ std::vector makeModule(const ModuleShape& shape) { } emit(function, spv::OpFunction, {voidType, main, 0, functionType}); emit(function, spv::OpLabel, {label}); - emit(function, spv::OpReturn, {}); + emit(function, shape.terminateInvocation ? spv::OpTerminateInvocation : spv::OpReturn, {}); emit(function, spv::OpFunctionEnd, {}); std::vector words{spv::MagicNumber, 0x10300, 0, next, 0}; emit(words, spv::OpCapability, {spv::CapabilityShader}); @@ -5003,6 +5004,14 @@ std::vector makeModule(const ModuleShape& shape) { words.resize(start + count, 0); for (std::size_t i = 0; i < extension.size(); ++i) words[start + i / 4] |= static_cast(static_cast(extension[i])) << ((i % 4) * 8); } + if (shape.terminateInvocation) { + const std::string extension = "SPV_KHR_terminate_invocation"; + const auto count = (extension.size() + 4) / 4; + words.push_back((static_cast(count + 1) << 16u) | spv::OpExtension); + const auto start = words.size(); + words.resize(start + count, 0); + for (std::size_t i = 0; i < extension.size(); ++i) words[start + i / 4] |= static_cast(static_cast(extension[i])) << ((i % 4) * 8); + } emit(words, spv::OpMemoryModel, {spv::AddressingModelLogical, spv::MemoryModelGLSL450}); const auto entryPointOffset = words.size(); if (shape.vertexInput) emit(words, spv::OpEntryPoint, {spv::ExecutionModelVertex, main, 0x6e69616du, 0, output, input}); From 0e8ce286e1830a07320a57ea7f03372d1425db54 Mon Sep 17 00:00:00 2001 From: abnergtz <63434566+abnergtz@users.noreply.github.com> Date: Sun, 11 Oct 2026 00:27:35 -0300 Subject: [PATCH 16/27] fix(agc): validate OpTerminateInvocation itself, not only its extension The validator only checked SPV_KHR_terminate_invocation when it saw the OpExtension, so a module using OpTerminateInvocation without declaring the extension, or in a non-fragment stage, was accepted. Require the fragment stage, the enabled shaderTerminateInvocation feature and the declared extension at the instruction itself. Refs #2009 --- .../prx/libSceAgcDriver/tests/Graphics.cpp | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 9452795be1..5faea9186b 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -4784,6 +4784,7 @@ struct ModuleShape { std::uint32_t cullDistanceLength = 0; bool layerOutput = false; bool terminateInvocation = false; + bool omitTerminateExtension = false; }; void emit(std::vector& out, spv::Op op, std::initializer_list operands) { @@ -5004,7 +5005,7 @@ std::vector makeModule(const ModuleShape& shape) { words.resize(start + count, 0); for (std::size_t i = 0; i < extension.size(); ++i) words[start + i / 4] |= static_cast(static_cast(extension[i])) << ((i % 4) * 8); } - if (shape.terminateInvocation) { + if (shape.terminateInvocation && !shape.omitTerminateExtension) { const std::string extension = "SPV_KHR_terminate_invocation"; const auto count = (extension.size() + 4) / 4; words.push_back((static_cast(count + 1) << 16u) | spv::OpExtension); @@ -6699,6 +6700,23 @@ void validationTests() { vertex.spirv = makeModule({.barycentric = true}); pixel.spirv = makeModule({.fragment = true}); expectFailure([&] { AgcDriver::Graphics::ValidateShaders(shaders, state, subgroup, true); }, "requires a fragment shader"); + const auto validateTerminate = [&](bool enabled) { + return AgcDriver::Graphics::ValidateShaders(shaders, state, subgroup, false, false, false, false, false, false, false, false, false, enabled); + }; + vertex.spirv = makeModule({}); + pixel.spirv = makeModule({.fragment = true, .terminateInvocation = true}); + validateTerminate(true); + expectFailure([&] { validateTerminate(false); }, "shaderTerminateInvocation"); + vertex.spirv = makeModule({.terminateInvocation = true}); + pixel.spirv = makeModule({.fragment = true}); + expectFailure([&] { validateTerminate(true); }, "fragment shader"); + vertex.spirv = makeModule({}); + pixel.spirv = makeModule({.fragment = true, .terminateInvocation = true, .omitTerminateExtension = true}); + expectFailure([&] { validateTerminate(true); }, "OpTerminateInvocation requires SPV_KHR_terminate_invocation"); + expectFailure([&] { validateTerminate(false); }, "OpTerminateInvocation requires enabled shaderTerminateInvocation"); + vertex.spirv = makeModule({.terminateInvocation = true, .omitTerminateExtension = true}); + pixel.spirv = makeModule({.fragment = true}); + expectFailure([&] { validateTerminate(true); }, "OpTerminateInvocation requires enabled shaderTerminateInvocation in a fragment shader"); } { ShaderRecompiler::RecompileResult vertex; From 0fe8527a0a2e0e56bbf1aaa2483fdcdbff3f9683 Mon Sep 17 00:00:00 2001 From: abnergtz <63434566+abnergtz@users.noreply.github.com> Date: Wed, 14 Oct 2026 11:45:21 +0000 Subject: [PATCH 17/27] test(agc): cover terminate invocation specialization, stage boundaries and disabled feature --- .../prx/libSceAgcDriver/tests/Graphics.cpp | 47 +++++++++++++++++++ .../tests/ShaderDeviceProfile.cpp | 11 +++++ 2 files changed, 58 insertions(+) diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 5faea9186b..df69286a9c 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -25,6 +25,7 @@ #include "prx/libc/include/general/VabiMacros.hpp" #include "prx/libc/include/GuestHeap.hpp" #include "ImageTableAbi.hpp" +#include "SpirvBackend/SpirvSpecialization.hpp" #include #include #include @@ -5708,6 +5709,41 @@ void orderedPixelShaderTests() { if ((terminated[at] & 0xffffu) == spv::OpExtension && std::string_view(reinterpret_cast(&terminated[at + 1])) == "SPV_KHR_terminate_invocation") ++terminateDeclarations; } Require(terminateDeclarations == 1u, "the module declared SPV_KHR_terminate_invocation " + std::to_string(terminateDeclarations) + " times"); + const auto countOps = [](std::span words) { + std::map counts; + for (std::size_t at = 5; at < words.size() && (words[at] >> 16u) != 0; at += words[at] >> 16u) ++counts[static_cast(words[at] & 0xffffu)]; + return counts; + }; + const std::array bdaCapabilities{spv::CapabilityShader, spv::CapabilityInt64, spv::CapabilityPhysicalStorageBufferAddresses, spv::CapabilityStorageBuffer8BitAccess}; + auto unordered = orderedPixelRequest(0x20600u, bdaCapabilities, OrderedBdaPixelCode); + unordered.target.bdaAbiVersion = ShaderRecompiler::BdaAbi::Version; + unordered.target.supportedExtensions = terminateExtensions; + const auto unorderedOps = countOps(ShaderRecompiler::Recompile(unordered).spirv.Words()); + Require(!unorderedOps.contains(spv::OpBeginInvocationInterlockEXT), "a pixel shader without PRIMITIVE_ORDERED_PIXEL_SHADER began the interlock"); + Require(unorderedOps.at(spv::OpTerminateInvocation) == 5u && unorderedOps.at(spv::OpKill) == 1u, "a pixel shader with BDA reads and SPV_KHR_terminate_invocation did not terminate on a fault"); + unordered.target.supportedExtensions = bdaExtensions; + const auto unorderedKills = countOps(ShaderRecompiler::Recompile(unordered).spirv.Words()); + Require(!unorderedKills.contains(spv::OpTerminateInvocation) && unorderedKills.at(spv::OpKill) == 6u, "a pixel shader with BDA reads and no SPV_KHR_terminate_invocation did not kill on a fault"); + constexpr std::array vertexBdaCode{0x7e040f00u, 0x7e060280u, 0xdc208001u, 0x047d0002u, 0xdc308008u, 0x057d0002u, 0xbf8c3f70u, 0x4a080b04u, 0x7e0e0d04u, 0xf80008cfu, 0x07070707u, 0xbf810000u}; + const auto vertexWords = [&](std::span extensions) { + ShaderRecompiler::RecompileRequest request{}; + request.shader = {ShaderRecompiler::ShaderStage::Vertex, 0x10000u, vertexBdaCode, 0, {}}; + request.context.waveSize = 64; + request.context.vertex = ShaderRecompiler::ShaderVertexStageInfo{}; + request.target.vulkanVersion = 0x00401000u; + request.target.spirvVersion = 0x00010300u; + request.target.subgroupSize = 64; + request.target.supportedCapabilities = bdaCapabilities; + request.target.supportedExtensions = extensions; + request.target.bdaAbiVersion = ShaderRecompiler::BdaAbi::Version; + request.layout.pushConstantSizeBytes = 128; + request.useCache = false; + return ShaderRecompiler::Recompile(request).spirv.Words(); + }; + const auto vertexStops = vertexWords(bdaExtensions); + const auto vertexTerminate = vertexWords(terminateExtensions); + const auto vertexOps = countOps(vertexTerminate); + Require(vertexStops == vertexTerminate && !vertexOps.contains(spv::OpTerminateInvocation) && !vertexOps.contains(spv::OpKill) && vertexOps.at(spv::OpReturn) > 1u, "a vertex shader with BDA reads changed its fault exit when SPV_KHR_terminate_invocation was advertised"); expectFailure([&] { static_cast(ShaderRecompiler::Recompile(orderedPixelRequest(0x30600u, plain))); }, "fragmentShaderPixelInterlock"); } @@ -6717,6 +6753,17 @@ void validationTests() { vertex.spirv = makeModule({.terminateInvocation = true, .omitTerminateExtension = true}); pixel.spirv = makeModule({.fragment = true}); expectFailure([&] { validateTerminate(true); }, "OpTerminateInvocation requires enabled shaderTerminateInvocation in a fragment shader"); + vertex.spirv = makeModule({}); + pixel.spirv = ShaderRecompiler::SpecializeSpirv(makeModule({.fragment = true, .terminateInvocation = true})); + std::size_t terminates = 0; + std::size_t declarations = 0; + for (std::size_t at = 5; at < pixel.spirv.size(); at += pixel.spirv[at] >> 16u) { + const auto op = pixel.spirv[at] & 0xffffu; + if (op == spv::OpTerminateInvocation) ++terminates; + if (op == spv::OpExtension && std::string_view(reinterpret_cast(&pixel.spirv[at + 1])) == "SPV_KHR_terminate_invocation") ++declarations; + } + Require(terminates == 1u && declarations == 1u, "specialization dropped OpTerminateInvocation or its extension declaration"); + validateTerminate(true); } { ShaderRecompiler::RecompileResult vertex; diff --git a/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp b/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp index 1143e50b0d..ca31e4ab4d 100644 --- a/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp @@ -106,6 +106,17 @@ void CheckProfile() { enabled.indexing.shaderStorageImageArrayNonUniformIndexing = VK_TRUE; const AgcDriver::ShaderDeviceProfile accepted(enabled.Target(), enabled.device, enabled.limits); Require(accepted.Target().supportedCapabilities.size() == 6u, "enabled descriptor indexing was rejected"); + ProfileInput terminating; + VkPhysicalDeviceShaderTerminateInvocationFeaturesKHR terminate{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_TERMINATE_INVOCATION_FEATURES_KHR, &terminating.bda, VK_TRUE}; + terminating.device.pNext = &terminate; + terminating.extensions.push_back("SPV_KHR_terminate_invocation"); + const AgcDriver::ShaderDeviceProfile terminated(terminating.Target(), terminating.device, terminating.limits); + Require(std::ranges::find(terminated.Target().supportedExtensions, "SPV_KHR_terminate_invocation") != terminated.Target().supportedExtensions.end(), "enabled shaderTerminateInvocation was rejected"); + ProfileInput disabled; + VkPhysicalDeviceShaderTerminateInvocationFeaturesKHR disabledTerminate{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_TERMINATE_INVOCATION_FEATURES_KHR, &disabled.bda, VK_FALSE}; + disabled.device.pNext = &disabledTerminate; + disabled.extensions.push_back("SPV_KHR_terminate_invocation"); + Reject([&] { AgcDriver::ShaderDeviceProfile rejected(disabled.Target(), disabled.device, disabled.limits); }, "shaderTerminateInvocation"); } void CheckAbi() { From 54e95dfa76e23b9cf464f1cd6d32feae09e3275e Mon Sep 17 00:00:00 2001 From: abnergtz <63434566+abnergtz@users.noreply.github.com> Date: Sat, 10 Oct 2026 23:55:14 -0300 Subject: [PATCH 18/27] feat(agc): enable VK_KHR_shader_terminate_invocation on the device The device queries and enables shaderTerminateInvocation when available, advertises SPV_KHR_terminate_invocation to the recompiler and validation, and the shader device profile rejects a target that advertises the extension without the enabled feature. Based on the approach in poonasor/AnyPS5@73969879. Refs #2009 (cherry picked from commit 1866a3f70e7ac1c34c8cd87fc79a973bd9ba6947) --- .../Execution/src/ShaderDeviceProfile.cpp | 4 ++-- .../Execution/src/VulkanDevice.cpp | 17 +++++++++++++++++ .../tests/ShaderDeviceProfile.cpp | 1 + 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp index 7dc7a4640c..45bbf6c978 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp @@ -30,7 +30,7 @@ ShaderDeviceProfile::ShaderDeviceProfile(const ShaderRecompiler::SpirvTarget& ta const auto* barycentric = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FRAGMENT_SHADER_BARYCENTRIC_FEATURES_KHR); const auto* mesh = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MESH_SHADER_FEATURES_EXT); const auto* clock = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_CLOCK_FEATURES_KHR); - const auto* drawParameters = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_DRAW_PARAMETERS_FEATURES); + const auto* terminate = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_TERMINATE_INVOCATION_FEATURES_KHR); const auto* maintenance8 = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MAINTENANCE_8_FEATURES_KHR); Require(target.bdaAbiVersion == ShaderRecompiler::BdaAbi::Version, "shader device profile has an incompatible BDA ABI"); Require(bda != nullptr && bda->bufferDeviceAddress == VK_TRUE && core.shaderInt64 == VK_TRUE, "shader runtime requires enabled bufferDeviceAddress and shaderInt64"); @@ -65,7 +65,6 @@ ShaderDeviceProfile::ShaderDeviceProfile(const ShaderRecompiler::SpirvTarget& ta const bool meshEnabled = mesh != nullptr && mesh->meshShader == VK_TRUE; checkFeature(spv::CapabilityFragmentBarycentricKHR, barycentricEnabled, "fragmentShaderBarycentric"); checkFeature(spv::CapabilityMeshShadingEXT, meshEnabled, "meshShader"); - checkFeature(spv::CapabilityDrawParameters, drawParameters != nullptr && drawParameters->shaderDrawParameters == VK_TRUE, "shaderDrawParameters"); checkFeature(spv::CapabilityShaderClockKHR, clock != nullptr && clock->shaderSubgroupClock == VK_TRUE && clock->shaderDeviceClock == VK_TRUE, "shader clock"); Require(!target.fragmentShaderBarycentricEnabled || barycentricEnabled, "shader device profile exposes disabled barycentrics"); Require(!target.mesh || meshEnabled, "shader device profile exposes disabled mesh shaders"); @@ -76,6 +75,7 @@ ShaderDeviceProfile::ShaderDeviceProfile(const ShaderRecompiler::SpirvTarget& ta extensions.erase(std::unique(extensions.begin(), extensions.end()), extensions.end()); for (const auto& extension : extensions) extensionViews.emplace_back(extension); Require(std::ranges::find(extensionViews, "SPV_KHR_physical_storage_buffer") != extensionViews.end() && std::ranges::find(extensionViews, "SPV_KHR_8bit_storage") != extensionViews.end(), "shader device profile is missing runtime extensions"); + Require(std::ranges::find(extensionViews, "SPV_KHR_terminate_invocation") == extensionViews.end() || (terminate != nullptr && terminate->shaderTerminateInvocation == VK_TRUE), "shader device profile advertises a disabled feature: shaderTerminateInvocation"); this->target.supportedCapabilities = capabilities; this->target.supportedExtensions = extensionViews; } diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp index 694152052f..3174261ad5 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp @@ -277,6 +277,7 @@ struct VulkanDevice::State { bool tessellationShader = false; bool meshShader = false; bool fragmentShaderBarycentric = false; + bool shaderTerminateInvocation = false; bool geometryShader = false; bool sampleRateShading = false; bool shaderClipDistance = false; @@ -1133,6 +1134,12 @@ VulkanDevice::VulkanDevice(const PresentationWindow* window, bool compileOnly) : state->fragmentShaderBarycentric = FragmentShaderBarycentricUsable(driver.driverID, state->fragmentShaderBarycentric); } } + VkPhysicalDeviceShaderTerminateInvocationFeaturesKHR terminateFeatures{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_TERMINATE_INVOCATION_FEATURES_KHR}; + if (hasExtension(VK_KHR_SHADER_TERMINATE_INVOCATION_EXTENSION_NAME)) { + VkPhysicalDeviceFeatures2 features{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2, &terminateFeatures}; + state->InstanceFunction("vkGetPhysicalDeviceFeatures2")(selected, &features); + state->shaderTerminateInvocation = terminateFeatures.shaderTerminateInvocation == VK_TRUE; + } VkPhysicalDeviceFragmentShaderInterlockFeaturesEXT interlockFeatures{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FRAGMENT_SHADER_INTERLOCK_FEATURES_EXT}; if (hasExtension(VK_EXT_FRAGMENT_SHADER_INTERLOCK_EXTENSION_NAME)) { VkPhysicalDeviceFeatures2 features{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2, &interlockFeatures}; @@ -1179,6 +1186,10 @@ VulkanDevice::VulkanDevice(const PresentationWindow* window, bool compileOnly) : state->capabilities.push_back(spv::CapabilityFragmentBarycentricKHR); state->spirvExtensions.push_back("SPV_KHR_fragment_shader_barycentric"); } + if (state->shaderTerminateInvocation) { + deviceExtensions.push_back(VK_KHR_SHADER_TERMINATE_INVOCATION_EXTENSION_NAME); + state->spirvExtensions.push_back("SPV_KHR_terminate_invocation"); + } if (state->fragmentShaderPixelInterlock) { deviceExtensions.push_back(VK_EXT_FRAGMENT_SHADER_INTERLOCK_EXTENSION_NAME); state->capabilities.push_back(spv::CapabilityFragmentShaderPixelInterlockEXT); @@ -1651,6 +1662,11 @@ VulkanDevice::VulkanDevice(const PresentationWindow* window, bool compileOnly) : barycentricFeatures.pNext = byteFeatures.pNext; byteFeatures.pNext = &barycentricFeatures; } + if (state->shaderTerminateInvocation) { + terminateFeatures.shaderTerminateInvocation = VK_TRUE; + terminateFeatures.pNext = byteFeatures.pNext; + byteFeatures.pNext = &terminateFeatures; + } if (state->fragmentShaderPixelInterlock) { interlockFeatures.pNext = byteFeatures.pNext; byteFeatures.pNext = &interlockFeatures; @@ -3490,6 +3506,7 @@ Graphics::Context VulkanDevice::buildContext() const { context.sampleRateShading = state->sampleRateShading; context.shaderClipDistance = state->shaderClipDistance; context.shaderCullDistance = state->shaderCullDistance; + context.shaderTerminateInvocation = state->shaderTerminateInvocation; context.nullDescriptors = state->shaderProfile != nullptr && state->shaderProfile->NullDescriptors(); context.bufferInt64Atomics = state->bufferInt64Atomics; context.placeholderImages = state->placeholderImages.get(); diff --git a/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp b/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp index ca31e4ab4d..470a8074e2 100644 --- a/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp @@ -99,6 +99,7 @@ void CheckProfile() { invalid([](auto& value) { value.capabilities.push_back(spv::CapabilityStorageImageMultisample); }, "shaderStorageImageMultisample"); invalid([](auto& value) { value.capabilities.push_back(spv::CapabilityFragmentBarycentricKHR); }, "fragmentShaderBarycentric"); invalid([](auto& value) { value.capabilities.push_back(spv::CapabilityMeshShadingEXT); }, "meshShader"); + invalid([](auto& value) { value.extensions.push_back("SPV_KHR_terminate_invocation"); }, "shaderTerminateInvocation"); ProfileInput enabled; enabled.capabilities.push_back(spv::CapabilitySampledImageArrayDynamicIndexing); enabled.capabilities.push_back(spv::CapabilityStorageImageArrayNonUniformIndexing); From 502a972f2d5c2390c66b4f3b68ecf49611462dd3 Mon Sep 17 00:00:00 2001 From: MarcZX100 Date: Thu, 15 Oct 2026 14:51:11 +0000 Subject: [PATCH 19/27] test(agc): update terminate checks for current APIs --- .../Graphics/src/ShaderValidation.cpp | 14 ++++++++++++++ core/libs/prx/libSceAgcDriver/tests/Graphics.cpp | 4 ++-- .../libSceAgcDriver/tests/ShaderDeviceProfile.cpp | 7 ++++--- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp index 0862785621..1463b56251 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/ShaderValidation.cpp @@ -612,6 +612,20 @@ Module Inspect(const CompiledShader& compiled, const State& state, const VkPhysi } +bool DeclaresEarlyFragmentTests(const ShaderRecompiler::RecompileResult& program) { + const auto words = program.spirv.Words(); + Require(words.size() >= 5 && words[0] == spv::MagicNumber, "the fragment shader is not a SPIR-V module"); + for (std::size_t cursor = 5; cursor < words.size();) { + const auto count = words[cursor] >> 16u; + Require(count != 0 && count <= words.size() - cursor, "truncated SPIR-V instruction"); + const auto op = static_cast(words[cursor] & 0xffffu); + if (op == spv::OpFunction) return false; + if (op == spv::OpExecutionMode && count >= 3 && words[cursor + 2] == spv::ExecutionModeEarlyFragmentTests) return true; + cursor += count; + } + return false; +} + std::set ValidateShaders(std::span shaders, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric, bool descriptorIndexing, bool imageInt64Atomics, bool geometryShader, bool sampleRateShading, bool bufferInt64Atomics, bool shaderClipDistance, bool shaderCullDistance, bool viewportIndexLayer, bool runtimeDescriptorArray, bool demoteToHelperInvocation, bool shaderTerminateInvocation) { using Stage = ShaderRecompiler::ShaderStage; const bool tessellation = state.stages.path == ShaderPath::Tessellation; diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index df69286a9c..5131ceaf2a 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -5743,7 +5743,7 @@ void orderedPixelShaderTests() { const auto vertexStops = vertexWords(bdaExtensions); const auto vertexTerminate = vertexWords(terminateExtensions); const auto vertexOps = countOps(vertexTerminate); - Require(vertexStops == vertexTerminate && !vertexOps.contains(spv::OpTerminateInvocation) && !vertexOps.contains(spv::OpKill) && vertexOps.at(spv::OpReturn) > 1u, "a vertex shader with BDA reads changed its fault exit when SPV_KHR_terminate_invocation was advertised"); + Require(std::ranges::equal(vertexStops, vertexTerminate) && !vertexOps.contains(spv::OpTerminateInvocation) && !vertexOps.contains(spv::OpKill) && vertexOps.at(spv::OpReturn) > 1u, "a vertex shader with BDA reads changed its fault exit when SPV_KHR_terminate_invocation was advertised"); expectFailure([&] { static_cast(ShaderRecompiler::Recompile(orderedPixelRequest(0x30600u, plain))); }, "fragmentShaderPixelInterlock"); } @@ -6737,7 +6737,7 @@ void validationTests() { pixel.spirv = makeModule({.fragment = true}); expectFailure([&] { AgcDriver::Graphics::ValidateShaders(shaders, state, subgroup, true); }, "requires a fragment shader"); const auto validateTerminate = [&](bool enabled) { - return AgcDriver::Graphics::ValidateShaders(shaders, state, subgroup, false, false, false, false, false, false, false, false, false, enabled); + return AgcDriver::Graphics::ValidateShaders(shaders, state, subgroup, false, false, false, false, false, false, false, false, false, false, enabled); }; vertex.spirv = makeModule({}); pixel.spirv = makeModule({.fragment = true, .terminateInvocation = true}); diff --git a/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp b/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp index 470a8074e2..53f43075d8 100644 --- a/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp @@ -9,6 +9,7 @@ #include #include #include +#include namespace { @@ -16,8 +17,8 @@ using AgcDriver::Graphics::Require; struct ProfileInput { std::vector capabilities{spv::CapabilityShader, spv::CapabilityInt64, spv::CapabilityPhysicalStorageBufferAddresses, spv::CapabilityStorageBuffer8BitAccess}; - std::array extensionStrings{"SPV_KHR_physical_storage_buffer", "SPV_KHR_8bit_storage"}; - std::array extensions{extensionStrings[0], extensionStrings[1]}; + std::vector extensionStrings{"SPV_KHR_physical_storage_buffer", "SPV_KHR_8bit_storage"}; + std::vector extensions{extensionStrings[0], extensionStrings[1]}; VkPhysicalDeviceRobustness2FeaturesEXT robustness{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_ROBUSTNESS_2_FEATURES_EXT, nullptr, VK_FALSE, VK_FALSE, VK_TRUE}; VkPhysicalDeviceDescriptorIndexingFeatures indexing{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DESCRIPTOR_INDEXING_FEATURES, &robustness}; VkPhysicalDevice8BitStorageFeatures bytes{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_8BIT_STORAGE_FEATURES, &indexing, VK_TRUE}; @@ -54,7 +55,7 @@ void Reject(TAction action, const char* expected) { Require(std::string(error.what()).find(expected) != std::string::npos, "unexpected profile validation error"); return; } - throw std::runtime_error("invalid shader device profile was accepted"); + throw std::runtime_error(std::string("invalid shader device profile was accepted: ") + expected); } void CheckProfile() { From 8a82811d9b02fc18cb40a281bc9fe2b3e07ca659 Mon Sep 17 00:00:00 2001 From: MarcZX100 Date: Fri, 16 Oct 2026 05:40:22 +0000 Subject: [PATCH 20/27] fix(agc): preserve draw parameter profile validation Restore the shaderDrawParameters feature guard while adding shaderTerminateInvocation support. Cover enabled and disabled DrawParameters profiles, and pass the termination feature to the correct shader validation argument. --- .../libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp | 2 ++ core/libs/prx/libSceAgcDriver/tests/Graphics.cpp | 2 +- .../libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp | 7 +++++++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp index 45bbf6c978..df5ad7ec42 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/ShaderDeviceProfile.cpp @@ -30,6 +30,7 @@ ShaderDeviceProfile::ShaderDeviceProfile(const ShaderRecompiler::SpirvTarget& ta const auto* barycentric = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FRAGMENT_SHADER_BARYCENTRIC_FEATURES_KHR); const auto* mesh = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MESH_SHADER_FEATURES_EXT); const auto* clock = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_CLOCK_FEATURES_KHR); + const auto* drawParameters = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_DRAW_PARAMETERS_FEATURES); const auto* terminate = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_TERMINATE_INVOCATION_FEATURES_KHR); const auto* maintenance8 = findFeatures(deviceInfo.pNext, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MAINTENANCE_8_FEATURES_KHR); Require(target.bdaAbiVersion == ShaderRecompiler::BdaAbi::Version, "shader device profile has an incompatible BDA ABI"); @@ -65,6 +66,7 @@ ShaderDeviceProfile::ShaderDeviceProfile(const ShaderRecompiler::SpirvTarget& ta const bool meshEnabled = mesh != nullptr && mesh->meshShader == VK_TRUE; checkFeature(spv::CapabilityFragmentBarycentricKHR, barycentricEnabled, "fragmentShaderBarycentric"); checkFeature(spv::CapabilityMeshShadingEXT, meshEnabled, "meshShader"); + checkFeature(spv::CapabilityDrawParameters, drawParameters != nullptr && drawParameters->shaderDrawParameters == VK_TRUE, "shaderDrawParameters"); checkFeature(spv::CapabilityShaderClockKHR, clock != nullptr && clock->shaderSubgroupClock == VK_TRUE && clock->shaderDeviceClock == VK_TRUE, "shader clock"); Require(!target.fragmentShaderBarycentricEnabled || barycentricEnabled, "shader device profile exposes disabled barycentrics"); Require(!target.mesh || meshEnabled, "shader device profile exposes disabled mesh shaders"); diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 5131ceaf2a..30078a407a 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -6737,7 +6737,7 @@ void validationTests() { pixel.spirv = makeModule({.fragment = true}); expectFailure([&] { AgcDriver::Graphics::ValidateShaders(shaders, state, subgroup, true); }, "requires a fragment shader"); const auto validateTerminate = [&](bool enabled) { - return AgcDriver::Graphics::ValidateShaders(shaders, state, subgroup, false, false, false, false, false, false, false, false, false, false, enabled); + return AgcDriver::Graphics::ValidateShaders(shaders, state, subgroup, false, false, false, false, false, false, false, false, false, false, false, enabled); }; vertex.spirv = makeModule({}); pixel.spirv = makeModule({.fragment = true, .terminateInvocation = true}); diff --git a/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp b/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp index 53f43075d8..f5014a3cb9 100644 --- a/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/ShaderDeviceProfile.cpp @@ -100,6 +100,7 @@ void CheckProfile() { invalid([](auto& value) { value.capabilities.push_back(spv::CapabilityStorageImageMultisample); }, "shaderStorageImageMultisample"); invalid([](auto& value) { value.capabilities.push_back(spv::CapabilityFragmentBarycentricKHR); }, "fragmentShaderBarycentric"); invalid([](auto& value) { value.capabilities.push_back(spv::CapabilityMeshShadingEXT); }, "meshShader"); + invalid([](auto& value) { value.capabilities.push_back(spv::CapabilityDrawParameters); }, "shaderDrawParameters"); invalid([](auto& value) { value.extensions.push_back("SPV_KHR_terminate_invocation"); }, "shaderTerminateInvocation"); ProfileInput enabled; enabled.capabilities.push_back(spv::CapabilitySampledImageArrayDynamicIndexing); @@ -108,6 +109,12 @@ void CheckProfile() { enabled.indexing.shaderStorageImageArrayNonUniformIndexing = VK_TRUE; const AgcDriver::ShaderDeviceProfile accepted(enabled.Target(), enabled.device, enabled.limits); Require(accepted.Target().supportedCapabilities.size() == 6u, "enabled descriptor indexing was rejected"); + ProfileInput drawParameters; + drawParameters.capabilities.push_back(spv::CapabilityDrawParameters); + VkPhysicalDeviceShaderDrawParametersFeatures drawFeatures{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_DRAW_PARAMETERS_FEATURES, &drawParameters.bda, VK_TRUE}; + drawParameters.device.pNext = &drawFeatures; + const AgcDriver::ShaderDeviceProfile drawParametersAccepted(drawParameters.Target(), drawParameters.device, drawParameters.limits); + Require(std::ranges::find(drawParametersAccepted.Target().supportedCapabilities, spv::CapabilityDrawParameters) != drawParametersAccepted.Target().supportedCapabilities.end(), "enabled shaderDrawParameters was rejected"); ProfileInput terminating; VkPhysicalDeviceShaderTerminateInvocationFeaturesKHR terminate{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_TERMINATE_INVOCATION_FEATURES_KHR, &terminating.bda, VK_TRUE}; terminating.device.pNext = &terminate; From 00fdf5b4c4c6513416e1a7e119d89fb0c7dafdaa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Fri, 16 Oct 2026 11:47:50 +0200 Subject: [PATCH 21/27] test(agc): give the vertex terminate test the ClipDistance capability --- core/libs/prx/libSceAgcDriver/tests/Graphics.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 30078a407a..210b55680d 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -5725,6 +5725,7 @@ void orderedPixelShaderTests() { const auto unorderedKills = countOps(ShaderRecompiler::Recompile(unordered).spirv.Words()); Require(!unorderedKills.contains(spv::OpTerminateInvocation) && unorderedKills.at(spv::OpKill) == 6u, "a pixel shader with BDA reads and no SPV_KHR_terminate_invocation did not kill on a fault"); constexpr std::array vertexBdaCode{0x7e040f00u, 0x7e060280u, 0xdc208001u, 0x047d0002u, 0xdc308008u, 0x057d0002u, 0xbf8c3f70u, 0x4a080b04u, 0x7e0e0d04u, 0xf80008cfu, 0x07070707u, 0xbf810000u}; + const std::array vertexCapabilities{spv::CapabilityShader, spv::CapabilityInt64, spv::CapabilityPhysicalStorageBufferAddresses, spv::CapabilityStorageBuffer8BitAccess, spv::CapabilityClipDistance}; const auto vertexWords = [&](std::span extensions) { ShaderRecompiler::RecompileRequest request{}; request.shader = {ShaderRecompiler::ShaderStage::Vertex, 0x10000u, vertexBdaCode, 0, {}}; @@ -5733,7 +5734,7 @@ void orderedPixelShaderTests() { request.target.vulkanVersion = 0x00401000u; request.target.spirvVersion = 0x00010300u; request.target.subgroupSize = 64; - request.target.supportedCapabilities = bdaCapabilities; + request.target.supportedCapabilities = vertexCapabilities; request.target.supportedExtensions = extensions; request.target.bdaAbiVersion = ShaderRecompiler::BdaAbi::Version; request.layout.pushConstantSizeBytes = 128; From b85146dbf596eae40cfbec116b380938ee4bf0a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Fri, 16 Oct 2026 11:54:41 +0200 Subject: [PATCH 22/27] test(agc): keep the recompiled words alive in the terminate tests --- core/libs/prx/libSceAgcDriver/tests/Graphics.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 210b55680d..abc9744ce7 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -5700,7 +5700,7 @@ void orderedPixelShaderTests() { const std::array terminateExtensions{"SPV_KHR_physical_storage_buffer", "SPV_KHR_8bit_storage", "SPV_KHR_terminate_invocation"}; auto terminating = faulting; terminating.target.supportedExtensions = terminateExtensions; - const auto terminated = ShaderRecompiler::Recompile(terminating).spirv.Words(); + const auto terminated = ShaderRecompiler::Recompile(terminating).spirv.Copy(); std::size_t terminatedKills = 0, terminatedStops = 0; const auto terminatedExits = interlockExits(terminated, terminatedKills, terminatedStops); Require(terminatedExits.empty() && terminatedKills == 1u && terminatedStops == 5u, "a primitive-ordered pixel shader with BDA reads and SPV_KHR_terminate_invocation: " + (terminatedExits.empty() ? std::to_string(terminatedKills) + " OpKill and " + std::to_string(terminatedStops) + " OpTerminateInvocation, not the valid-mask kill and 5 BDA fault terminations" : terminatedExits)); @@ -5739,7 +5739,7 @@ void orderedPixelShaderTests() { request.target.bdaAbiVersion = ShaderRecompiler::BdaAbi::Version; request.layout.pushConstantSizeBytes = 128; request.useCache = false; - return ShaderRecompiler::Recompile(request).spirv.Words(); + return ShaderRecompiler::Recompile(request).spirv.Copy(); }; const auto vertexStops = vertexWords(bdaExtensions); const auto vertexTerminate = vertexWords(terminateExtensions); From a18d6e9a6b87bab0fe9a4007ede97f05ef1f25bf Mon Sep 17 00:00:00 2001 From: ps54all Date: Wed, 14 Oct 2026 16:22:42 +0200 Subject: [PATCH 23/27] feat(shader): store D16 data to SINT storage images image_store with D16 data to a storage image of a SINT format threw "stores 16-bit data to an image of a SINT format", both when the descriptor is decoded and in the SPIR-V image emitter, and the runtime image modes left out the SINT conversions for such images. God of War Ragnarok (PPSA08330) stops on it in compute shader 0x1052a17900 right after its MFSR setup. Each 16-bit half is now converted like the D16 format buffer stores (EmitD16StoreComponent): sign-extended, then saturated to the component width as for 32-bit data. The buffer form was measured on RDNA2 in #331; the image form is not measured (TechnicalDebt). agc_driver_image_store_sint now stores D16 data with DMASK 0xf and 0x5 to all eight SINT formats and after an image_atomic_add on 32_SINT, instead of expecting the refusal. Co-Authored-By: Claude Opus 5.5 --- .../tests/execution/ImageStoreSint.cpp | 35 ++++++++++++++----- .../Optimization/src/ResourceMaterializer.cpp | 3 +- .../SpirvBackend/src/SpirvImageEmitter.cpp | 5 +-- docs/dev/debt/shader.md | 2 +- 4 files changed, 29 insertions(+), 16 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/tests/execution/ImageStoreSint.cpp b/core/libs/prx/libSceAgcDriver/tests/execution/ImageStoreSint.cpp index a178c8c162..8daddabd90 100644 --- a/core/libs/prx/libSceAgcDriver/tests/execution/ImageStoreSint.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/execution/ImageStoreSint.cpp @@ -55,9 +55,15 @@ alignas(256) constexpr std::array LoadCode{ 0xbf810000, }; -alignas(256) constexpr std::array D16Code{0x7e3c0300, 0x7e3e0280, 0xf0201108, 0x8001141e, 0xbf810000}; +alignas(256) constexpr std::array D16Code{ + 0x34020084, 0xe0381000, 0x80001401, 0xbf8c3f70, 0x7e3c0300, 0x7e3e0280, 0xf0201f08, 0x8001141e, + 0x7e3e0281, 0xf0201508, 0x8001141e, 0xbf810000, +}; alignas(256) constexpr std::array PackedCode{0x7e3c0300, 0x7e3e0280, 0xf0281108, 0x0001141e, 0xbf810000}; -alignas(256) constexpr std::array AtomicD16Code{0x7e3c0300, 0x7e3e0280, 0xf0441108, 0x0001141e, 0xf0201108, 0x8001141e, 0xbf810000}; +alignas(256) constexpr std::array AtomicD16Code{ + 0x34020084, 0xe0381000, 0x80001401, 0xbf8c3f70, 0x7e3c0300, 0x7e3e0280, 0xf0441108, 0x0001141e, + 0xf0201108, 0x8001141e, 0xbf810000, +}; alignas(256) constexpr std::array PackedWideCode{0x34020084, 0xe0381000, 0x80001401, 0xbf8c3f70, 0x7e3c0300, 0x7e3e0280, 0xf0281308, 0x0001141e, 0xbf810000}; alignas(256) constexpr std::array AtomicCode{0x7e3c0300, 0x7e3e0280, 0xf0441108, 0x0001141e, 0xbf810000}; @@ -181,13 +187,21 @@ std::uint32_t StoredComponent(const Format& format, std::uint32_t tid, std::uint return Saturate(Value(tid, index), format.bits); } -std::vector Expected(const Format& format, std::span writes) { +std::uint32_t StoredD16Component(const Format& format, std::uint32_t tid, std::uint32_t dmask, std::uint32_t component) { + if (((dmask >> component) & 1u) == 0u) return 0u; + std::uint32_t index = 0; + for (std::uint32_t lower = 0; lower < component; ++lower) index += (dmask >> lower) & 1u; + const auto half = (Value(tid, index / 2u) >> (16u * (index & 1u))) & 0xffffu; + return Saturate(SignExtend(half, 16u), format.bits); +} + +std::vector Expected(const Format& format, std::span writes, bool d16 = false) { auto bytes = Initial(); for (std::uint32_t tid = 0; tid < Threads; ++tid) { for (const auto& write : writes) { auto* texel = bytes.data() + TexelOffset(format, tid, write.row); for (std::uint32_t component = 0; component < format.components; ++component) { - const auto value = StoredComponent(format, tid, write.dmask, component); + const auto value = d16 ? StoredD16Component(format, tid, write.dmask, component) : StoredComponent(format, tid, write.dmask, component); std::memcpy(texel + component * format.bits / 8u, &value, format.bits / 8u); } } @@ -244,8 +258,8 @@ void Store(AgcDriver::VulkanDevice& device, std::uint8_t* texels, std::span(Input.size() * 4u))); } -void CheckTexels(const std::uint8_t* texels, const Format& format, std::span writes, const std::string& what) { - const auto expected = Expected(format, writes); +void CheckTexels(const std::uint8_t* texels, const Format& format, std::span writes, const std::string& what, bool d16 = false) { + const auto expected = Expected(format, writes, d16); for (std::uint32_t y = 0; y < Height; ++y) { for (std::uint32_t x = 0; x < Width; ++x) { const auto offset = TexelOffset(format, x, y); @@ -297,8 +311,10 @@ int main() { } Store(*device, texels, TitleCode, Formats[0].format, TitleSwizzle); CheckTexels(texels, Formats[0], TitleWrites, "image_store dmask:0x1 glc with DST_SEL X,1,1,1"); - RequireRefused(*device, texels, D16Code, 12u, "stores 16-bit data to an image of a SINT format", "image_store d16 of 16_SINT"); - RequireRefused(*device, texels, D16Code, 21u, "stores 16-bit data to an image of a SINT format", "image_store d16 of 32_SINT"); + for (const auto& format : Formats) { + Store(*device, texels, D16Code, format.format, SwizzleOf(format)); + CheckTexels(texels, format, StoreWrites, "image_store d16", true); + } RequireRefused(*device, texels, PackedCode, 12u, "storage image descriptor uses an unsupported format", "image_store_pck of 16_SINT"); RequireRefused(*device, texels, AtomicCode, 12u, "atomic image descriptor uses an unsupported format 12", "image_atomic_add of 16_SINT"); RequireRefused(*device, texels, StoreCode, 73u, "storage image descriptor uses an unsupported format", "image_store of 32_32_32_SINT"); @@ -306,7 +322,8 @@ int main() { Store(*device, texels, PackedWideCode, Formats[index].format, SwizzleOf(Formats[index])); CheckTexels(texels, Formats[index], PackedWideWrites, "image_store_pck dmask:0x3"); } - RequireRefused(*device, texels, AtomicD16Code, 21u, "stores 16-bit data to an image of a SINT format", "image_store d16 of an atomically updated 32_SINT image"); + Store(*device, texels, AtomicD16Code, Formats[2].format, IdentitySwizzle); + CheckTexels(texels, Formats[2], TitleWrites, "image_store d16 after image_atomic_add", true); std::puts("image store sint tests passed"); return 0; } catch (const std::exception& error) { diff --git a/core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp b/core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp index 43aad3402d..ff8686cb2f 100644 --- a/core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp +++ b/core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp @@ -260,7 +260,6 @@ ImageDecode inspectImageDescriptor(const DescriptorValue& descriptor, const Imag decoded.numericClass = IrTextureNumericClass::Uint; } if ((rawSintStorage || (base.atomic && format == IrBufferFormat::Format32SInt)) && !base.packed) { - if (!base.depthBitsCompatible) return rejectImage(PoisonReason::InvalidFormat, "stores 16-bit data to an image of a SINT format"); decoded.conversionFormat = format; } } else if (decoded.numericClass == IrTextureNumericClass::Unsupported || (base.depthCompare && decoded.numericClass != IrTextureNumericClass::Float)) { @@ -1318,7 +1317,7 @@ std::vector ResourceMaterializer::RuntimeImageModes(const ImageRe } } } - if (storage && image.depthBitsCompatible && !image.packed && !image.atomic64 && ((image.written && !image.read) || image.atomic)) { + if (storage && !image.packed && !image.atomic64 && ((image.written && !image.read) || image.atomic)) { constexpr std::array formats{IrBufferFormat::Format32SInt, IrBufferFormat::Format32_32SInt, IrBufferFormat::Format32_32_32_32SInt, IrBufferFormat::Format16SInt, IrBufferFormat::Format8_8SInt, IrBufferFormat::Format16_16SInt, IrBufferFormat::Format8_8_8_8SInt, IrBufferFormat::Format16_16_16_16SInt}; for (const auto format : formats) { if (image.atomic && format != IrBufferFormat::Format32SInt) continue; diff --git a/core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp b/core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp index db0ca5d84b..d2ce577837 100644 --- a/core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp +++ b/core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp @@ -741,9 +741,6 @@ std::uint32_t StoreTexel(SpirvValueEmitContext& ctx, const ImageEmitAccess& acce auto& state = ctx.state; const auto& mem = access.mem; const auto sint = SintStorageFormat(access.image); - if (sint.format != IrBufferFormat::Invalid && mem.dataBits == 16u) { - ctx.Fail(access.inst, "stores 16-bit data to an image of a SINT format"); - } std::uint32_t values[4] = {}; const auto dmask = EffectiveDmask(mem); for (std::uint32_t component = 0; component < 4u; component++) { @@ -766,7 +763,7 @@ std::uint32_t StoreTexel(SpirvValueEmitContext& ctx, const ImageEmitAccess& acce } values[component] = integer ? raw : mem.dataBits == 16u ? EmitF16BitsToF32(state, raw) : Unary(state, spv::OpBitcast, TypeF32(state), raw); if (component < sint.componentCount) { - values[component] = EmitFormatStoreComponent(state, sint, component, values[component]); + values[component] = mem.dataBits == 16u ? EmitD16StoreComponent(state, sint, component, values[component]) : EmitFormatStoreComponent(state, sint, component, values[component]); } } const auto texel = state.module.AllocateId(); diff --git a/docs/dev/debt/shader.md b/docs/dev/debt/shader.md index 9a5d65d1aa..ac5340dcb2 100644 --- a/docs/dev/debt/shader.md +++ b/docs/dev/debt/shader.md @@ -86,7 +86,7 @@ - [v_interp_p1ll_f16, v_interp_p1lv_f16 and v_interp_p2_f16](../../../core/shader/recompiler/Translation/src/AttributeInstructions.cpp) read the 16-bit halves of each vertex's parameter through `VK_KHR_fragment_shader_barycentric` and throw without it, as fixed-function interpolation cannot split a 32-bit attribute. P10 and P20 are the vertex differences rounded to f16 to nearest even, which is not confirmed on hardware. v_interp_p1ll_f16 and v_interp_p1lv_f16 round the exact `P10 * I + P0` once to f32 and v_interp_p2_f16 rounds the exact `P20 * J + S2` once to f16, as measured on RDNA2 (#697); the exact sums are computed in f64, so these instructions need the Float64 capability. They follow the shader's float mode, measured on RDNA2 (gfx1035) in all eight combinations of `IEEE_MODE` and kept/flushed f32 and f16 denormals over 768 special and 1,024 random rows. With IEEE=1 a NaN operand is returned quieted; with IEEE=0 it is returned unquieted, except that v_interp_p2_f16's f16 result is always quiet. Flushed f32 denormals flush I/J, v_interp_p2_f16's f32 S2 and the f32 result; flushed f16 denormals flush P0, P10/P20, v_interp_p1lv_f16's f16 S2 and the f16 result. Denormal modes 1 and 2 (flush only inputs or only outputs) and rounding other than to nearest even are not measured and throw. An unknown mode follows IEEE=0 with denormals kept. Lavapipe has no barycentrics, so the draw test is skipped in CI; `agc_driver_interpolation_modes` checks the arithmetic in every mode against the hardware rows on any device with Float64. - Without `VK_KHR_fragment_shader_barycentric`, a pixel shader whose I/J VGPRs [only feed `v_interp_p1_f32`/`v_interp_p2_f32` of their own pair](../../../core/shader/recompiler/Optimization/src/HostInterpolationChecker.cpp) reads the attribute Vulkan interpolated. Any other read of them or of a raw vertex parameter takes the weights and the three vertex values from a [generated geometry stage](../../../core/shader/recompiler/SpirvBackend/src/BarycentricGeometryShader.cpp), which needs `geometryShader` and a vertex shader before it: tessellation, mesh and rect-list pipelines, primitives other than triangles, more than 32 fragment input locations and interfaces beyond the device's geometry limits throw. - [v_fma_f64 and v_div_fmas_f64](../../../core/shader/recompiler/SpirvBackend/src/SpirvAlu/SpirvAluEmitterFloat64.cpp) with a subnormal operand or result round once, as IEEE 754 fusedMultiplyAdd does with f64 denormals kept, measured on RDNA2 (gfx1035) at IEEE=0 and IEEE=1, `v_div_fmas_f64` with VCC clear and set. The f16/f64 denormal field of `FLOAT_MODE` (bits 7:6) is not read: when it flushes, the hardware flushes the operands and the result of `v_fma_f64` (tininess before rounding) and only the result of `v_div_fmas_f64`, while these keep them. Shaders with f64 instructions declare `DenormPreserve` for 64-bit floats only when `shaderDenormPreserveFloat64` is true and Vulkan reports `denormBehaviorIndependence` as `ALL`; other independence modes are not handled by the current backend because they require coupled execution modes. See [VkShaderFloatControlsIndependence](https://docs.vulkan.org/refpages/latest/refpages/source/VkShaderFloatControlsIndependence.html). Elsewhere (including lavapipe), the driver may flush f64 subnormals, and `agc_driver_f64_fma_subnormals` is skipped. -- [SINT storage images](../../../core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp) of 8- and 16-bit components saturate each stored 32-bit value to the component range, as adriadam10 measured for image stores on RDNA2 (gfx1035) in #944 (448 rows match); the PS5's gfx1013 is not measured. +- [SINT storage images](../../../core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp) of 8- and 16-bit components saturate each stored 32-bit value to the component range, as adriadam10 measured for image stores on RDNA2 (gfx1035) in #944 (448 rows match); the PS5's gfx1013 is not measured. A D16 store sign-extends each 16-bit half before that saturation, as the D16 format buffer stores measured in #331 do; D16 image stores to SINT surfaces themselves are not measured. - [`StoreTexel`](../../../core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp) writes channel 1 of a two-channel format from the data slot whose DST_SEL is Y, for SINT and UINT alike. With DST_SEL X,Y,Z,W, gfx1035 writes it from the W slot, as adriadam10 measured in #944. - [v_ldexp_f16](../../../core/shader/recompiler/Translation/src/FloatInstructions.cpp) follows the `FLOAT_MODE` f16 denormal field: bit 0 preserves the half input and bit 1 the half result. Otherwise a denormal input is flushed to its signed zero, and a result whose exact scaled value is below the minimum normal (2^-14) is flushed to its signed zero before rounding, so `v_ldexp_f16 0x07ff, -1` gives +0 instead of 0x0400. Measured on RDNA2 (gfx1036) with round to nearest, IEEE 0 and 1 and denorm16 0..3, over every half magnitude 0x0000..0x0fff of both signs scaled by 2^-1 and 2^0 (16384 rows per mode): no mismatch. NaN and infinity handling and destination high-half preservation are unchanged. Shaders without a known float mode keep the previous lowering. - [v_mullit_f32](../../../core/shader/recompiler/Translation/src/FloatInstructions.cpp) follows the hardware measured with the MODE register at IEEE=1 and denormals kept (a signaling NaN src0 is quieted only with IEEE=1, as measured at IEEE=0 too; a positive denormal src2 counts as positive); with f32 denormals flushed it follows the f32 denormals entry below, and denormal modes 1 and 2 throw. From d8c2b71d12ccd54b5676f028563055b452cbe156 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Fri, 16 Oct 2026 13:18:43 +0200 Subject: [PATCH 24/27] docs(debt): D16 SINT image stores measured on RDNA2 --- docs/dev/debt/shader.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/debt/shader.md b/docs/dev/debt/shader.md index ac5340dcb2..dea88ed54f 100644 --- a/docs/dev/debt/shader.md +++ b/docs/dev/debt/shader.md @@ -86,7 +86,7 @@ - [v_interp_p1ll_f16, v_interp_p1lv_f16 and v_interp_p2_f16](../../../core/shader/recompiler/Translation/src/AttributeInstructions.cpp) read the 16-bit halves of each vertex's parameter through `VK_KHR_fragment_shader_barycentric` and throw without it, as fixed-function interpolation cannot split a 32-bit attribute. P10 and P20 are the vertex differences rounded to f16 to nearest even, which is not confirmed on hardware. v_interp_p1ll_f16 and v_interp_p1lv_f16 round the exact `P10 * I + P0` once to f32 and v_interp_p2_f16 rounds the exact `P20 * J + S2` once to f16, as measured on RDNA2 (#697); the exact sums are computed in f64, so these instructions need the Float64 capability. They follow the shader's float mode, measured on RDNA2 (gfx1035) in all eight combinations of `IEEE_MODE` and kept/flushed f32 and f16 denormals over 768 special and 1,024 random rows. With IEEE=1 a NaN operand is returned quieted; with IEEE=0 it is returned unquieted, except that v_interp_p2_f16's f16 result is always quiet. Flushed f32 denormals flush I/J, v_interp_p2_f16's f32 S2 and the f32 result; flushed f16 denormals flush P0, P10/P20, v_interp_p1lv_f16's f16 S2 and the f16 result. Denormal modes 1 and 2 (flush only inputs or only outputs) and rounding other than to nearest even are not measured and throw. An unknown mode follows IEEE=0 with denormals kept. Lavapipe has no barycentrics, so the draw test is skipped in CI; `agc_driver_interpolation_modes` checks the arithmetic in every mode against the hardware rows on any device with Float64. - Without `VK_KHR_fragment_shader_barycentric`, a pixel shader whose I/J VGPRs [only feed `v_interp_p1_f32`/`v_interp_p2_f32` of their own pair](../../../core/shader/recompiler/Optimization/src/HostInterpolationChecker.cpp) reads the attribute Vulkan interpolated. Any other read of them or of a raw vertex parameter takes the weights and the three vertex values from a [generated geometry stage](../../../core/shader/recompiler/SpirvBackend/src/BarycentricGeometryShader.cpp), which needs `geometryShader` and a vertex shader before it: tessellation, mesh and rect-list pipelines, primitives other than triangles, more than 32 fragment input locations and interfaces beyond the device's geometry limits throw. - [v_fma_f64 and v_div_fmas_f64](../../../core/shader/recompiler/SpirvBackend/src/SpirvAlu/SpirvAluEmitterFloat64.cpp) with a subnormal operand or result round once, as IEEE 754 fusedMultiplyAdd does with f64 denormals kept, measured on RDNA2 (gfx1035) at IEEE=0 and IEEE=1, `v_div_fmas_f64` with VCC clear and set. The f16/f64 denormal field of `FLOAT_MODE` (bits 7:6) is not read: when it flushes, the hardware flushes the operands and the result of `v_fma_f64` (tininess before rounding) and only the result of `v_div_fmas_f64`, while these keep them. Shaders with f64 instructions declare `DenormPreserve` for 64-bit floats only when `shaderDenormPreserveFloat64` is true and Vulkan reports `denormBehaviorIndependence` as `ALL`; other independence modes are not handled by the current backend because they require coupled execution modes. See [VkShaderFloatControlsIndependence](https://docs.vulkan.org/refpages/latest/refpages/source/VkShaderFloatControlsIndependence.html). Elsewhere (including lavapipe), the driver may flush f64 subnormals, and `agc_driver_f64_fma_subnormals` is skipped. -- [SINT storage images](../../../core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp) of 8- and 16-bit components saturate each stored 32-bit value to the component range, as adriadam10 measured for image stores on RDNA2 (gfx1035) in #944 (448 rows match); the PS5's gfx1013 is not measured. A D16 store sign-extends each 16-bit half before that saturation, as the D16 format buffer stores measured in #331 do; D16 image stores to SINT surfaces themselves are not measured. +- [SINT storage images](../../../core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp) of 8- and 16-bit components saturate each stored 32-bit value to the component range, as adriadam10 measured for image stores on RDNA2 (gfx1035) in #944 (448 rows match); the PS5's gfx1013 is not measured. A D16 store sign-extends each 16-bit half before that saturation, as measured on RDNA2 (gfx1035) for `image_store d16` to all eight SINT formats with DMASK 0xf, 0x7, 0x5, 0x3, 0x2, 0xa and 0x1 (1792 rows match). - [`StoreTexel`](../../../core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp) writes channel 1 of a two-channel format from the data slot whose DST_SEL is Y, for SINT and UINT alike. With DST_SEL X,Y,Z,W, gfx1035 writes it from the W slot, as adriadam10 measured in #944. - [v_ldexp_f16](../../../core/shader/recompiler/Translation/src/FloatInstructions.cpp) follows the `FLOAT_MODE` f16 denormal field: bit 0 preserves the half input and bit 1 the half result. Otherwise a denormal input is flushed to its signed zero, and a result whose exact scaled value is below the minimum normal (2^-14) is flushed to its signed zero before rounding, so `v_ldexp_f16 0x07ff, -1` gives +0 instead of 0x0400. Measured on RDNA2 (gfx1036) with round to nearest, IEEE 0 and 1 and denorm16 0..3, over every half magnitude 0x0000..0x0fff of both signs scaled by 2^-1 and 2^0 (16384 rows per mode): no mismatch. NaN and infinity handling and destination high-half preservation are unchanged. Shaders without a known float mode keep the previous lowering. - [v_mullit_f32](../../../core/shader/recompiler/Translation/src/FloatInstructions.cpp) follows the hardware measured with the MODE register at IEEE=1 and denormals kept (a signaling NaN src0 is quieted only with IEEE=1, as measured at IEEE=0 too; a positive denormal src2 counts as positive); with f32 denormals flushed it follows the f32 denormals entry below, and denormal modes 1 and 2 throw. From b1e88d5995795b56b00df928280bdec345bd722c Mon Sep 17 00:00:00 2001 From: milescoviello Date: Fri, 16 Oct 2026 11:36:24 +0000 Subject: [PATCH 25/27] fix(agc): journal the mesh output primitive size #5884 added MeshConfiguration::outputPrimitiveSize, and #5875's pipeline journal guards the layouts it encodes. The two merged separately, so on Linux with libstdc++ PipelineJournal.cpp no longer compiles: MeshConfiguration is 40 bytes, ShaderStages (which holds an optional one) 112. The guards now name those sizes, and the recipe codec writes and reads the new field, so a replayed line-strip geometry pipeline keeps its two-vertex primitives. Co-Authored-By: Claude Opus 5.5 --- .../prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp | 8 ++++---- core/libs/prx/libSceAgcDriver/tests/Graphics.cpp | 5 +++++ 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp index 094b1ed9bd..8729ac4d9d 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp @@ -36,8 +36,8 @@ static_assert(sizeof(VkVertexInputBindingDescription) == 12, "VkVertexInputBindi static_assert(sizeof(VkVertexInputAttributeDescription) == 16, "VkVertexInputAttributeDescription changed: update the recipe codec"); #if defined(__linux__) && defined(__x86_64__) && defined(__GLIBCXX__) static_assert(sizeof(State) == 752, "State changed: update CanonicalPipelineState and the recipe codec"); -static_assert(sizeof(ShaderStages) == 108, "ShaderStages changed: update the recipe codec"); -static_assert(sizeof(ShaderRecompiler::MeshConfiguration) == 36, "MeshConfiguration changed: update the recipe codec"); +static_assert(sizeof(ShaderStages) == 112, "ShaderStages changed: update the recipe codec"); +static_assert(sizeof(ShaderRecompiler::MeshConfiguration) == 40, "MeshConfiguration changed: update the recipe codec"); static_assert(sizeof(ShaderRecompiler::TessellationConfiguration) == 40, "TessellationConfiguration changed: update the recipe codec"); #endif @@ -147,11 +147,11 @@ void encodeOptional(Writer& writer, const std::optional& value, void (*e } void encodeMesh(Writer& writer, const ShaderRecompiler::MeshConfiguration& mesh) { - for (const auto value : {mesh.inputPrimitive, mesh.primitivesPerGroup, mesh.verticesPerGroup, mesh.maxVertices, mesh.maxPrimitives, mesh.threadsPerGroup, mesh.ldsSizeDwords, mesh.provokingVertex, mesh.esgsItemSize}) writer.U32(value); + for (const auto value : {mesh.inputPrimitive, mesh.primitivesPerGroup, mesh.verticesPerGroup, mesh.maxVertices, mesh.maxPrimitives, mesh.threadsPerGroup, mesh.ldsSizeDwords, mesh.provokingVertex, mesh.esgsItemSize, mesh.outputPrimitiveSize}) writer.U32(value); } void decodeMesh(Reader& reader, ShaderRecompiler::MeshConfiguration& mesh) { - for (auto* value : {&mesh.inputPrimitive, &mesh.primitivesPerGroup, &mesh.verticesPerGroup, &mesh.maxVertices, &mesh.maxPrimitives, &mesh.threadsPerGroup, &mesh.ldsSizeDwords, &mesh.provokingVertex, &mesh.esgsItemSize}) *value = reader.U32(); + for (auto* value : {&mesh.inputPrimitive, &mesh.primitivesPerGroup, &mesh.verticesPerGroup, &mesh.maxVertices, &mesh.maxPrimitives, &mesh.threadsPerGroup, &mesh.ldsSizeDwords, &mesh.provokingVertex, &mesh.esgsItemSize, &mesh.outputPrimitiveSize}) *value = reader.U32(); } void encodeTessellation(Writer& writer, const ShaderRecompiler::TessellationConfiguration& tessellation) { diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index 5acb0bb165..dcc4ea5ae9 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -7537,6 +7537,11 @@ void pipelineJournalTests() { BuildPipelineStoreKey(grid, context, gridState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL); BuildPipelineStoreKey(gridReplayed, context, gridDecoded.state, gridDecoded.vertexInput, layouts.View(), shaders, gridDecoded.attachmentLayout); Require(grid != original && grid == gridReplayed, "the decoded sample grid state keys another pipeline"); + auto lineState = state; + lineState.stages.mesh = ShaderRecompiler::MeshConfiguration{1, 2, 2, 4, 4, 32, 64, 1, 4, 2}; + GraphicsPipelineRecipe lineDecoded; + Require(DecodeGraphicsPipelineRecipe(EncodeGraphicsPipelineRecipe(MakeGraphicsPipelineRecipe(lineState, input, layouts.View(), shaders, VK_IMAGE_LAYOUT_GENERAL)), lineDecoded), "the line strip mesh recipe does not decode"); + Require(lineDecoded.state.stages.mesh && lineDecoded.state.stages.mesh->outputPrimitiveSize == 2u && lineDecoded.state.stages.mesh->esgsItemSize == 4u, "the recipe dropped the mesh output primitive size"); auto fourSampleState = state; fourSampleState.samples = 4; auto maskState = fourSampleState; From 2d5f761cb59239996a905915c3f20cce8f1a30ac Mon Sep 17 00:00:00 2001 From: milescoviello Date: Fri, 16 Oct 2026 11:45:11 +0000 Subject: [PATCH 26/27] fix(agc): discard pipeline journals written before the mesh field The layout guards only build on Linux with libstdc++, so a Windows or macOS build of main after #5884 wrote journals whose mesh records lack outputPrimitiveSize under the new shader-cache source version. The journal format is now 3, so such a file is discarded instead of being read with the new codec. MarcZX100 pointed this out. Co-Authored-By: Claude Opus 5.5 --- .../Graphics/src/PipelineJournal.cpp | 2 +- core/libs/prx/libSceAgcDriver/tests/Graphics.cpp | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp index 8729ac4d9d..77d4566132 100644 --- a/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp +++ b/core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp @@ -44,7 +44,7 @@ static_assert(sizeof(ShaderRecompiler::TessellationConfiguration) == 40, "Tessel constexpr std::uint32_t FileMagic = 0x4c4a5041u; // Bumped when the recipe codec, CanonicalPipelineState or the store key change in a way the // layout guards above do not catch (a field renumbered or reinterpreted at the same size). -constexpr std::uint32_t FileFormat = 2u; +constexpr std::uint32_t FileFormat = 3u; constexpr std::uint32_t ModuleRecord = 1u; constexpr std::uint32_t GraphicsRecord = 2u; constexpr std::uint32_t ComputeRecord = 3u; diff --git a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp index dcc4ea5ae9..d6c6af2d83 100644 --- a/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp +++ b/core/libs/prx/libSceAgcDriver/tests/Graphics.cpp @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -7656,6 +7657,19 @@ void pipelineJournalTests() { const auto after = PipelineJournalTotals(); Require(cancelled->Done() && after.replayFailed == 0 && after.replayRejected == 0 && after.replayedGraphics + after.replayPresent <= totals.replayedGraphics + totals.replayPresent + 1, "a cancelled replay did not stop cleanly"); } + { + const std::uint32_t meshWithoutOutputSize = 2u; + std::fstream file(PipelineJournalPath(), std::ios::in | std::ios::out | std::ios::binary); + file.seekp(4); + file.write(reinterpret_cast(&meshWithoutOutputSize), sizeof(meshWithoutOutputSize)); + file.close(); + Require(file.good(), "the journal header could not be rewritten"); + auto stale = StartPipelineJournalReplay(context, fingerprint, hooks); + Require(stale != nullptr, "the replay of a journal in the previous format did not start"); + stale->Wait(); + const auto discarded = PipelineJournalTotals(); + Require(discarded.loadedRecipes == 0 && discarded.loadedModules == 0, "a journal in format 2, whose mesh records lack the output primitive size, was loaded"); + } ClearCachedPipelines(context.device); } Require(replayedComputeIdentityLow != 0, "the replayed compute identity is empty"); From 37db6a80b74eadc38cb6945b0f9bcda2dc3fd5ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A0=20Franch?= Date: Fri, 16 Oct 2026 14:30:15 +0200 Subject: [PATCH 27/27] fix(agc): bound the async log's exit flush On Windows the atexit handlers of a DLL run after the process has ended its other threads, so the logger thread can be gone with lines still queued and the unbounded Flush waits forever (guest_shader_link timed out on 4 of the last 7 completed main builds since #6371). Run the bounded FlushAtEnd there instead. Co-Authored-By: Claude --- core/libs/prx/libSceAgcDriver/Execution/include/AsyncLog.hpp | 2 +- docs/dev/ARCHITECTURE.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/core/libs/prx/libSceAgcDriver/Execution/include/AsyncLog.hpp b/core/libs/prx/libSceAgcDriver/Execution/include/AsyncLog.hpp index 383e554f61..8af0c00d39 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/include/AsyncLog.hpp +++ b/core/libs/prx/libSceAgcDriver/Execution/include/AsyncLog.hpp @@ -192,7 +192,7 @@ private: lastStats = Clock::now(); if (!asynchronous) return; LogSink().store(&AsyncLog::Sink, std::memory_order_release); - std::atexit([] { Get().Flush(); }); + std::atexit(&AsyncLog::FlushAtEnd); std::thread([this] { Run(); }).detach(); LibcSetEndFlush_nid_no_patch(&AsyncLog::FlushAtEnd); } diff --git a/docs/dev/ARCHITECTURE.md b/docs/dev/ARCHITECTURE.md index c51b0407c7..6876b67f68 100644 --- a/docs/dev/ARCHITECTURE.md +++ b/docs/dev/ARCHITECTURE.md @@ -125,7 +125,7 @@ flowchart LR - **Queue.** One logger thread drains a queue that holds at most 16 MiB of waiting lines; the batch the logger is writing comes on top. A caller that finds the queue full blocks until the logger has taken the pending batch, so no line is dropped. The logger writes entries in queue order and joins consecutive entries of one stream; it writes them with one `fwrite` and flush when the text reaches 64 KiB, the stream changes or the batch ends. `AGC_LOG_DEFERRED` queues the format string with numeric, enum and `LogLiteral` arguments and formats them on the logger thread; `APS5_LOG_TIMESTAMPS` stamps a line when it is queued. - **Order.** Lines that pass through the sink, on stdout and stderr alike, appear in call order. A direct `fprintf(stderr, ...)` or `printf` does not pass through it and is written at once, so it can appear before older queued lines; the coalesced `deferred write-back failed` notes are of this kind. `APS5_LOG_SYNC=1` keeps one order for all of them. -- **Flush.** `Flush` waits without a limit until every line queued before the call has been written, including while another thread holds the stream. It runs when the driver fails, in `Driver::stop` and at exit. `AsyncLog` also registers a bounded flush (`FlushFor`, 2 s) with `LibcSetEndFlush_nid_no_patch`; libc runs it where the process ends without those: `abort()` and `std::terminate` (a `SIGABRT` handler that the first registration chains in front of the previous one, so `raise(SIGABRT)` still ends the process), guest `abort()`, the Windows crash report of a fatal exception and guest `_Exit`. A title that installs its own `SIGABRT` disposition replaces that handler. The bounded flush polls two counters and takes no lock, so a thread that dies holding the queue lock or a stream cannot stall it. After 2 s it gives up, later calls return at once, and the lines still queued are lost. +- **Flush.** `Flush` waits without a limit until every line queued before the call has been written, including while another thread holds the stream. It runs when the driver fails and in `Driver::stop`. At exit `AsyncLog` runs a bounded flush (`FlushFor`, 2 s) instead: on Windows the `atexit` handlers of a DLL run after the process has ended its other threads, so the logger thread may be gone with lines still queued, and a `Flush` there would wait forever. It also registers that bounded flush with `LibcSetEndFlush_nid_no_patch`; libc runs it where the process ends without `atexit`: `abort()` and `std::terminate` (a `SIGABRT` handler that the first registration chains in front of the previous one, so `raise(SIGABRT)` still ends the process), guest `abort()`, the Windows crash report of a fatal exception and guest `_Exit`. A title that installs its own `SIGABRT` disposition replaces that handler. The bounded flush polls two counters and takes no lock, so a thread that dies holding the queue lock or a stream cannot stall it. After 2 s it gives up, later calls return at once, and the lines still queued are lost. - **Statistics.** A `[log]` line every 10 s in which lines were logged reports lines, bytes, the time spent by the callers and by the logger, and the queue peak; `APS5_LOG_STATS=0` hides it. Identical `deferred write-back failed` notes print three times per 10 s and are then counted in one `repeated N more times` line. `agc_async_log` checks the order of deferred and written lines, long lines, the repeat limit, a `Flush` that waits for a held stream, a bounded flush that gives up, and a full queue that blocks its caller. `agc_async_log_fatal` starts the test executable as a child that queues lines behind a held stream and then calls `abort()`, calls `std::terminate()`, raises an access violation (Windows) or `SIGABRT` (elsewhere), or aborts without ever releasing the stream; the first four must leave every line in the file in order, the last must still end after the bound.