fix(shader): keep a v_writelane lane known across computed-index window stores

fdfawef324 notes that current LLVM spills SGPRs into VGPRs it allocates like any other
register, so one window register can be an ordinary register early in a
program and hold spill lanes later. ReadLaneEliminator only followed a
spilled lane across M0-indexed window stores for slots never used as
ordinary registers, so RESIDENT EVIL 4 (PPSA07411) and LEGO Horizon
Adventures (PPSA14632) kept their reloaded descriptors as run-time
values and stopped. A known lane can only come from a v_writelane chain,
so keep it across a computed-index store instead; a constant-index store
of the slot still ends the spill. This is fdfawef324's suggestion on
#6277.

That such a store misses the spill cannot be proven for these programs,
so it is checked at run time where a replaced read depends on it: a
backward liveness over each slot and lane whose known value a replaced
read used puts a WindowLaneGuard before every computed-index store at
which that lane is live, and the guard records a WindowLaneSpill fault
when the store changes that lane of the slot. The driver reads the fault
buffer back for every program with such a guard and reports the dispatch
or draw as failed, so a broken assumption is no longer silent. This also
covers the pure spill slots main already folds.

Co-authored-by: fdfawef324 <[email protected]>
Co-Authored-By: Claude <[email protected]>

core/libs/prx/libSceAgcDriver/Graphics/src/BdaResources.cpp

@@ -282,6 +282,10 @@ void BdaResources::CheckFault() const {
         message << "shader stage " << report.stage << " called address 0x" << std::hex << report.address << " at program counter 0x" << report.instruction << ", which no linked callee of that call matches";
         throw std::runtime_error(message.str());
     }
+    if (report.reason == ShaderRecompiler::BdaAbi::FaultReason::WindowLaneSpill) {
+        message << "shader stage " << report.stage << ": a relative write stored " << report.bytes << " in lane " << (report.address >> 32u) << " of vector window slot " << (report.address & 0xffffffffu) << ", which holds a scalar spilled with v_writelane_b32 that a later v_readlane_b32 was replaced with";
+        throw std::runtime_error(message.str());
+    }
     message << "BDA access failed: address=0x" << std::hex << report.address << " instruction=0x" << report.instruction << std::dec << " bytes=" << report.bytes << " stage=" << report.stage << " reason=" << static_cast<std::uint32_t>(report.reason);
     if (report.reason == ShaderRecompiler::BdaAbi::FaultReason::Permission && table != nullptr) {
         const auto bytes = table->Bytes();

core/libs/prx/libSceAgcDriver/Graphics/src/ShaderAddressResources.cpp

@@ -134,7 +134,7 @@ void ShaderResources::prepareAddressBindings(std::span<const CompiledShader> sha
         if (tables != 0 && storesThroughAddressesCached(shader.program->spirv)) bdaReadOnly = false;
         usesFaultBuffer = usesFaultBuffer || faults != 0;
         const bool tableFaultsOnly = std::any_of(shader.program->bindings.begin(), shader.program->bindings.end(), [](const ShaderRecompiler::DescriptorBinding& binding) { return binding.role == ShaderRecompiler::DescriptorRole::ImageTableMap; });
-        faultChecks = faultChecks || tables != 0 || (faults != 0 && (!tableFaultsOnly || shader.program->imageTableFaults != 0 || shader.program->poisonedSrtReads != 0 || loopGuarded));
+        faultChecks = faultChecks || tables != 0 || (faults != 0 && (!tableFaultsOnly || shader.program->imageTableFaults != 0 || shader.program->poisonedSrtReads != 0 || shader.program->windowLaneGuards != 0 || loopGuarded));
     }
     if (usesBda) {
         Require(context.bufferDeviceAddress, "buffer device address is not enabled");

core/libs/prx/libSceAgcDriver/tests/VectorWindow.cpp

@@ -40,6 +40,13 @@ constexpr std::uint32_t SampleV8V2S0S8 = 0x00400802u;
 constexpr std::uint32_t AddV8V1V8 = 0x4a101101u;
 constexpr std::uint32_t BufferStoreDword = 0xe0700000u;
 constexpr std::uint32_t StoreV8WithS12 = 0x80030800u;
+constexpr std::uint32_t AndM0GroupXOne = 0x877c8104u;
+constexpr std::uint32_t MovreldV2V0 = 0x7e048500u;
+constexpr std::uint32_t MoveS16Two = 0xbe900382u;
+constexpr std::uint32_t BranchForwardTwo = 0xbf820002u;
+constexpr std::uint32_t SubtractS16One = 0x80908110u;
+constexpr std::uint32_t CompareS16NotZero = 0xbf078010u;
+constexpr std::uint32_t BranchBackFourOnScc = 0xbf85fffcu;
 
 struct Shape {
     std::size_t functionArrays = 0;
@@ -306,6 +313,35 @@ int main(int argc, char** argv) {
         return shape.functionArrays == 0 && shape.accessChains == 0;
     });
 
+    passed &= run("a sampler dword spilled into a window register with ordinary writes keeps its definition past a relative write", [] {
+        const std::array<std::uint32_t, 16> userData{};
+        const std::array<std::uint32_t, 14> spilled{AndM0GroupXOne, MoveV2Zero, MoveV3Zero, MoveV4One, WritelaneV4, FromS8Lane0, MovreldV2V0, ReadlaneS8, FromV4Lane0, ImageSampleLz2d, SampleV8V2S0S8, BufferStoreDword, StoreV8WithS12, EndProgram};
+        return Scan(RecompileProgram(spilled, userData)).functionArrays == 1;
+    });
+
+    passed &= run("a sampler dword spilled into a window register and overwritten by an ordinary write before its v_readlane stays a run-time value", [] {
+        const std::array<std::uint32_t, 16> userData{};
+        const std::array<std::uint32_t, 14> overwritten{AndM0GroupXOne, MoveV2Zero, MoveV3Zero, WritelaneV4, FromS8Lane0, MovreldV2V0, MoveV4One, ReadlaneS8, FromV4Lane0, ImageSampleLz2d, SampleV8V2S0S8, BufferStoreDword, StoreV8WithS12, EndProgram};
+        try {
+            RecompileProgram(overwritten, userData);
+        } catch (const std::exception& error) {
+            return std::string_view(error.what()).find("GetSamplerResource dword 0 is not a valid runtime value; chain: ReadLane(LoadVectorWindow(Immediate), Immediate)") != std::string_view::npos;
+        }
+        return false;
+    });
+
+    passed &= run("a sampler dword spilled into a window register before a loop entered at its condition keeps its definition past the loop's relative write", [] {
+        const std::array<std::uint32_t, 16> userData{};
+        const std::array<std::uint32_t, 18> code{AndM0GroupXOne, MoveV2Zero, MoveV3Zero, WritelaneV4, FromS8Lane0, MoveS16Two, BranchForwardTwo, MovreldV2V0, SubtractS16One, CompareS16NotZero, BranchBackFourOnScc, ReadlaneS8, FromV4Lane0, ImageSampleLz2d, SampleV8V2S0S8, BufferStoreDword, StoreV8WithS12, EndProgram};
+        return Scan(RecompileProgram(code, userData)).functionArrays == 1;
+    });
+
+    passed &= run("a sampler dword spilled into a window register with ordinary writes before a loop entered at its condition keeps its definition past the loop's relative write", [] {
+        const std::array<std::uint32_t, 16> userData{};
+        const std::array<std::uint32_t, 19> code{AndM0GroupXOne, MoveV2Zero, MoveV3Zero, MoveV4One, WritelaneV4, FromS8Lane0, MoveS16Two, BranchForwardTwo, MovreldV2V0, SubtractS16One, CompareS16NotZero, BranchBackFourOnScc, ReadlaneS8, FromV4Lane0, ImageSampleLz2d, SampleV8V2S0S8, BufferStoreDword, StoreV8WithS12, EndProgram};
+        return Scan(RecompileProgram(code, userData)).functionArrays == 1;
+    });
+
     passed &= run("a sampler dword spilled into a relatively read register keeps its definition through v_readlane", [] {
         const std::array<std::uint32_t, 14> code{MoveM0GroupX, MoveV2Zero, MoveV3Zero, WritelaneV4, FromS8Lane0, MovrelsV1V4, ReadlaneS8, FromV4Lane0, ImageSampleLz2d, SampleV8V2S0S8, AddV8V1V8, BufferStoreDword, StoreV8WithS12, EndProgram};
         const std::array<std::uint32_t, 16> userData{};

core/libs/prx/libSceAgcDriver/tests/execution/WindowLaneSpill.cpp

@@ -7,7 +7,13 @@
 #include <iostream>
 #include <span>
 #include <string>
+#include <string_view>
 #include <vector>
+#ifdef _WIN32
+#include <io.h>
+#else
+#include <unistd.h>
+#endif
 
 namespace {
 
@@ -15,7 +21,8 @@ using AgcDriver::Graphics::Require;
 using ShaderRecompiler::ShaderStage;
 
 constexpr std::uint32_t Threads = 32;
-alignas(256) std::array<std::uint32_t, Threads> Output{};
+constexpr std::uint32_t WideThreads = 64;
+alignas(256) std::array<std::uint32_t, WideThreads> Output{};
 
 alignas(256) constexpr std::array<std::uint32_t, 26> Code{
     0x7e020300, 0xbefc030c, 0x7e048500, 0xd7610003, 0x00010004, 0xd7610003, 0x00010205, 0xd7610003, 0x00010406,
@@ -29,28 +36,111 @@ alignas(256) constexpr std::array<std::uint32_t, 27> IndexedStoreCode{
     0xd7600009, 0x00010303, 0xd760000a, 0x00010503, 0xd760000b, 0x00010703, 0xe0702000, 0x80020001, 0xbf810000,
 };
 
+alignas(256) constexpr std::array<std::uint32_t, 28> ReusedRegisterCode{
+    0x7e060300, 0x7e020300, 0xbefc030c, 0x7e048500, 0xd7610003, 0x00010004, 0xd7610003, 0x00010205, 0xd7610003,
+    0x00010406, 0xd7610003, 0x00010607, 0x7e048500, 0xbe840380, 0xbe850380, 0xbe860380, 0xbe870380, 0xd7600008,
+    0x00010103, 0xd7600009, 0x00010303, 0xd760000a, 0x00010503, 0xd760000b, 0x00010703, 0xe0702000, 0x80020001,
+    0xbf810000,
+};
+
+alignas(256) constexpr std::array<std::uint32_t, 14> OverwrittenSpillCode{
+    0x877c8104, 0x7e040280, 0x7e060280, 0xd7610004, 0x00010008, 0x7e048500, 0x7e0802f2, 0xd7600008, 0x00010104,
+    0xf09c0f08, 0x00400802, 0xe0700000, 0x80030800, 0xbf810000,
+};
+
+alignas(256) constexpr std::array<std::uint32_t, 13> AliasingStoreCode{
+    0x7e020300, 0xbefc030c, 0x7e048500, 0xd7610003, 0x0001000d, 0x7e040287, 0x7e048500,
+    0xd7600008, 0x00010103, 0x7e040208, 0xe0702000, 0x80010201, 0xbf810000,
+};
+
+alignas(256) constexpr std::array<std::uint32_t, 13> WideAliasingStoreCode{
+    0x7e020300, 0xbefc030c, 0x7e048500, 0xd7610003, 0x0001500d, 0x7e040287, 0x7e048500,
+    0xd7600008, 0x00015103, 0x7e040208, 0xe0702000, 0x80010201, 0xbf810000,
+};
+
+alignas(256) constexpr std::array<std::uint32_t, 12> StoreAfterReadCode{
+    0x7e020300, 0xbefc030c, 0x7e048500, 0xd7610003, 0x0001000d, 0xd7600008,
+    0x00010103, 0x7e048500, 0x7e040208, 0xe0702000, 0x80010201, 0xbf810000,
+};
+
+constexpr std::uint32_t Spilled = 0x1234u;
+
+class StderrCapture {
+public:
+    StderrCapture() {
+        std::fflush(stderr);
+        file = std::tmpfile();
+        Require(file != nullptr, "cannot open a temporary file for stderr");
+#ifdef _WIN32
+        saved = _dup(_fileno(stderr));
+        _dup2(_fileno(file), _fileno(stderr));
+#else
+        saved = dup(fileno(stderr));
+        dup2(fileno(file), fileno(stderr));
+#endif
+    }
+    std::string Finish() {
+        std::fflush(stderr);
+#ifdef _WIN32
+        _dup2(saved, _fileno(stderr));
+        _close(saved);
+#else
+        dup2(saved, fileno(stderr));
+        close(saved);
+#endif
+        std::rewind(file);
+        std::string text;
+        char buffer[4096];
+        for (std::size_t read = 0; (read = std::fread(buffer, 1, sizeof(buffer), file)) != 0;) text.append(buffer, read);
+        std::fclose(file);
+        std::fwrite(text.data(), 1, text.size(), stderr);
+        return text;
+    }
+
+private:
+    std::FILE* file = nullptr;
+    int saved = -1;
+};
+
 std::array<std::uint32_t, 4> BufferDescriptor(const void* data, std::uint32_t count) {
     const auto address = reinterpret_cast<std::uintptr_t>(data);
     return {static_cast<std::uint32_t>(address), static_cast<std::uint32_t>((address >> 32u) & 0xffffu) | (4u << 16u), count, 0x11016facu};
 }
 
-void Run(AgcDriver::VulkanDevice& device, std::span<const std::uint32_t> code) {
-    Output.fill(0xdeadbeefu);
-    std::vector<std::uint32_t> userData(16, 0u);
-    const auto output = BufferDescriptor(Output.data(), static_cast<std::uint32_t>(Output.size()));
-    std::copy(output.begin(), output.end(), userData.begin() + 4);
+ShaderRecompiler::RecompileResult Compile(AgcDriver::VulkanDevice& device, std::span<const std::uint32_t> code, std::span<const std::uint32_t> userData, std::uint32_t waveSize = Threads) {
     const std::array<ShaderRecompiler::MemoryRegion, 1> memory{{{reinterpret_cast<std::uintptr_t>(code.data()), std::as_bytes(code)}}};
-    const ShaderRecompiler::ShaderComputeStageInfo compute{{Threads, 1, 1}, 0, {false, false, false}, false, 1};
+    const ShaderRecompiler::ShaderComputeStageInfo compute{{waveSize, 1, 1}, 0, {false, false, false}, false, 1};
     ShaderRecompiler::RecompileRequest request{
         {ShaderStage::Compute, reinterpret_cast<std::uintptr_t>(code.data()), code, 0, {}},
-        {32, 0, userData, compute, std::nullopt, std::nullopt, memory},
+        {waveSize, 0, userData, compute, std::nullopt, std::nullopt, memory},
         device.Target(),
         {0, 0, 0, 128}
     };
     request.useCache = false;
-    const auto result = ShaderRecompiler::Recompile(request);
+    return ShaderRecompiler::Recompile(request);
+}
+
+std::string Run(AgcDriver::VulkanDevice& device, std::span<const std::uint32_t> code, std::uint32_t relativeIndex = 0, std::uint32_t waveSize = Threads) {
+    Output.fill(0xdeadbeefu);
+    std::vector<std::uint32_t> userData(16, 0u);
+    const auto output = BufferDescriptor(Output.data(), static_cast<std::uint32_t>(Output.size()));
+    std::copy(output.begin(), output.end(), userData.begin() + 4);
+    userData[12] = relativeIndex;
+    userData[13] = Spilled;
+    const auto result = Compile(device, code, userData, waveSize);
+    StderrCapture log;
     device.Dispatch(result, 1, 1, 1, {}, reinterpret_cast<std::uintptr_t>(code.data()));
     device.WaitIdle();
+    return log.Finish();
+}
+
+constexpr std::string_view Overwritten = "which holds a scalar spilled with v_writelane_b32 that a later v_readlane_b32 was replaced with";
+
+void CheckSpilled(const std::string& log, const std::string& name, std::uint32_t lanes = Threads) {
+    Require(log.find(Overwritten) == std::string::npos, name + ": reported an overwritten spill: " + log);
+    for (std::uint32_t tid = 0; tid < lanes; ++tid) {
+        Require(Output[tid] == Spilled, name + ": lane " + std::to_string(tid) + " stored " + std::to_string(Output[tid]) + ", expected the spilled " + std::to_string(Spilled));
+    }
 }
 
 void Check(const std::string& name) {
@@ -59,6 +149,17 @@ void Check(const std::string& name) {
     }
 }
 
+void CheckRefused(AgcDriver::VulkanDevice& device, std::span<const std::uint32_t> code, const std::string& name) {
+    const std::vector<std::uint32_t> userData(16, 0u);
+    std::string error;
+    try {
+        static_cast<void>(Compile(device, code, userData));
+    } catch (const std::exception& thrown) {
+        error = thrown.what();
+    }
+    Require(error.find("GetSamplerResource dword 0 is not a valid runtime value") != std::string::npos, name + ": " + (error.empty() ? std::string("recompiled") : error));
+}
+
 }
 
 int main() {
@@ -73,6 +174,20 @@ int main() {
         Check("window lane spill");
         Run(*device, IndexedStoreCode);
         Check("window lane spill past an indexed store");
+        CheckRefused(*device, OverwrittenSpillCode, "window lane spill overwritten by an ordinary write before its v_readlane");
+        Run(*device, ReusedRegisterCode);
+        Check("window lane spill into a register with ordinary writes past an indexed store");
+        CheckSpilled(Run(*device, AliasingStoreCode, 0), "window lane spill past an indexed store with m0 = 0");
+        const auto aliased = Run(*device, AliasingStoreCode, 1);
+        Require(aliased.find(Overwritten) != std::string::npos, "window lane spill overwritten by an indexed store with m0 = 1 was not reported: " + aliased);
+        CheckSpilled(Run(*device, StoreAfterReadCode, 1), "window lane spill overwritten by an indexed store with m0 = 1 after its v_readlane");
+        if (device->Target().subgroupSize >= WideThreads) {
+            CheckSpilled(Run(*device, WideAliasingStoreCode, 0, WideThreads), "wave64 window lane spill in lane 40 past an indexed store with m0 = 0", WideThreads);
+            const auto wide = Run(*device, WideAliasingStoreCode, 1, WideThreads);
+            Require(wide.find("stored 40 in lane 40 of vector window slot 1") != std::string::npos, "wave64 window lane spill in lane 40 overwritten by an indexed store with m0 = 1 was not reported: " + wide);
+        } else {
+            std::printf("wave64 cases skipped, subgroup size %u\n", device->Target().subgroupSize);
+        }
         std::puts("window lane spill tests passed");
         return 0;
     } catch (const std::exception& error) {

core/shader/recompiler/BdaAbi.hpp

@@ -35,7 +35,7 @@ inline bool RangesOrdered(const Range* ranges, std::size_t count) {
 }
 
 enum class FaultState : std::uint32_t { Empty, Writing, Ready };
-enum class FaultReason : std::uint32_t { Unmapped = 1, Permission, Overflow, InvalidTable, InvalidRectangle, LoopLimit, Unaligned, InvalidDescriptor, ImageTable, UnlinkedCall };
+enum class FaultReason : std::uint32_t { Unmapped = 1, Permission, Overflow, InvalidTable, InvalidRectangle, LoopLimit, Unaligned, InvalidDescriptor, ImageTable, UnlinkedCall, WindowLaneSpill };
 
 struct Fault {
     FaultState state;

core/shader/recompiler/IntermediateRepresentation/include/IntermediateRepresentation/IrOpcode.hpp

@@ -74,6 +74,7 @@ enum class IrOpcode : std::uint16_t {
     Reference,
     ReferenceU32,
     UnlinkedCall,
+    WindowLaneGuard,
     GetUserData,
     GetShaderBase,
     ShaderClock,

core/shader/recompiler/IntermediateRepresentation/src/IrOpcode.cpp

@@ -115,6 +115,7 @@ constexpr std::array<OpcodeMeta, static_cast<std::size_t>(IrOpcode::Count)> Meta
     makeMeta("Reference", Void, U1),
     makeMeta("ReferenceU32", Void, U32),
     makeMeta("UnlinkedCall", Void, U32, U32, U32),
+    makeMeta("WindowLaneGuard", Void, U32, U32, U32, U32, U32),
     makeMeta("GetUserData", U32, ScalarReg),
     makeMeta("GetShaderBase", U64),
     makeMeta("ShaderClock", U64),
@@ -894,6 +895,7 @@ bool IrOpcodeHasSideEffects(IrOpcode opcode) {
         case IrOpcode::Reference:
         case IrOpcode::ReferenceU32:
         case IrOpcode::UnlinkedCall:
+        case IrOpcode::WindowLaneGuard:
         case IrOpcode::SetTessellationAttribute:
         case IrOpcode::InterpolateHostP2:
         case IrOpcode::SetThreadBitScalarRegister:

core/shader/recompiler/Optimization/include/Optimization/ReadLaneEliminator.hpp

@@ -8,6 +8,7 @@ namespace ShaderRecompiler {
 
 struct ReadLaneEliminationStats {
     std::uint32_t rewrittenReads = 0;
+    std::uint32_t guardedStores = 0;
 };
 
 class ReadLaneEliminator {

core/shader/recompiler/Optimization/src/ReadLaneEliminator.cpp

@@ -1,6 +1,8 @@
 #include "Optimization/ReadLaneEliminator.hpp"
 #include <algorithm>
 #include <list>
+#include <map>
+#include <set>
 #include <queue>
 #include <stdexcept>
 #include <unordered_map>
@@ -52,44 +54,13 @@ struct LaneFlow {
 
 struct WindowFlows {
     const IrProgram* program = nullptr;
-    std::unordered_set<std::uint32_t> spillSlots;
     std::unordered_map<std::uint64_t, LaneFlow> flows;
     std::unordered_set<std::uint64_t> computing;
+    std::vector<std::pair<IrValue*, std::uint32_t>>* folded = nullptr;
 };
 
 thread_local WindowFlows* windowFlows = nullptr;
 
-std::unordered_set<std::uint32_t> collectLaneSpillSlots(const IrProgram& program) {
-    std::unordered_set<std::uint32_t> candidates;
-    std::unordered_set<std::uint32_t> rejected;
-    const auto immediateSlot = [](const IrValue& inst, std::uint32_t& slot) {
-        const IrValue* index = inst.Argument(0)->Resolve();
-        if (!isImmediateU32(*index)) return false;
-        slot = index->ImmediateU32();
-        return true;
-    };
-    for (const auto& block : program.Blocks()) {
-        for (const IrValue* inst : block->Instructions()) {
-            std::uint32_t slot = 0;
-            if (inst->Opcode() == IrOpcode::StoreVectorWindow) {
-                if (!immediateSlot(*inst, slot)) continue;
-                candidates.insert(slot);
-                if (inst->Argument(1)->Resolve()->Opcode() != IrOpcode::WriteLane) rejected.insert(slot);
-            } else if (inst->Opcode() == IrOpcode::LoadVectorWindow) {
-                if (!immediateSlot(*inst, slot)) continue;
-                candidates.insert(slot);
-                for (const IrUse& use : inst->OperandUses()) {
-                    const bool laneRead = use.user->Opcode() == IrOpcode::ReadLane && use.operand == 0u;
-                    const bool chained = use.user->Opcode() == IrOpcode::WriteLane && use.operand == 2u;
-                    if (!laneRead && !chained) rejected.insert(slot);
-                }
-            }
-        }
-    }
-    for (const auto slot : rejected) candidates.erase(slot);
-    return candidates;
-}
-
 LaneState windowLoadLane(const IrValue* load, std::uint32_t lane, std::uint32_t waveSize);
 
 LaneState storedLane(IrValue* value, std::uint32_t slot, std::uint32_t lane, std::uint32_t waveSize, const LaneFlow& flow) {
@@ -117,7 +88,6 @@ const LaneFlow* laneFlow(std::uint32_t slot, std::uint32_t lane, std::uint32_t w
     if (const auto found = flows.flows.find(key); found != flows.flows.end()) return &found->second;
     if (!flows.computing.insert(key).second) return nullptr;
     LaneFlow flow;
-    const bool spill = flows.spillSlots.contains(slot);
     const auto& blocks = flows.program->Blocks();
     constexpr int MaxPasses = 64;
     bool settled = false;
@@ -144,11 +114,7 @@ const LaneFlow* laneFlow(std::uint32_t slot, std::uint32_t lane, std::uint32_t w
                     }
                 } else if (inst->Opcode() == IrOpcode::StoreVectorWindow) {
                     const IrValue* index = inst->Argument(0)->Resolve();
-                    if (!isImmediateU32(*index)) {
-                        if (!spill) running = {LaneState::Kind::Unknown, nullptr};
-                        continue;
-                    }
-                    if (index->ImmediateU32() != slot) continue;
+                    if (!isImmediateU32(*index) || index->ImmediateU32() != slot) continue;
                     running = storedLane(inst->Argument(1), slot, lane, waveSize, flow);
                 }
             }
@@ -177,8 +143,98 @@ LaneState windowLoadLane(const IrValue* load, std::uint32_t lane, std::uint32_t
 
 WindowLane windowLane(IrValue* load, std::uint32_t lane, std::uint32_t waveSize) {
     const LaneState state = windowLoadLane(load, lane, waveSize);
-    if (state.kind == LaneState::Kind::Known) return {state.value, true};
-    return {};
+    if (state.kind != LaneState::Kind::Known) return {};
+    if (windowFlows != nullptr && windowFlows->folded != nullptr) windowFlows->folded->emplace_back(load, lane);
+    return {state.value, true};
+}
+
+IrValue* laneSource(IrValue* value, std::uint32_t lane, std::uint32_t waveSize) {
+    for (;;) {
+        value = value->Resolve();
+        if (value->Opcode() == IrOpcode::WriteLane) {
+            const IrValue* selector = value->Argument(1)->Resolve();
+            if (!isImmediateU32(*selector) || selector->ImmediateU32() % waveSize == lane) return nullptr;
+            value = value->Argument(2);
+            continue;
+        }
+        if (value->Opcode() != IrOpcode::LoadVectorWindow || !isImmediateU32(*value->Argument(0)->Resolve())) return nullptr;
+        return value;
+    }
+}
+
+IrValue& immediate(IrProgram& program, std::uint32_t value) {
+    IrValue& constant = program.CreateValue(IrOpcode::Void, IrType::U32);
+    constant.SetImmediateU32(value);
+    return constant;
+}
+
+std::size_t guardSpilledLanes(IrProgram& program, const std::vector<std::pair<IrValue*, std::uint32_t>>& folded, std::uint32_t waveSize) {
+    const auto keyOf = [](std::uint32_t slot, std::uint32_t lane) { return (static_cast<std::uint64_t>(slot) << 32u) | lane; };
+    std::map<std::uint64_t, std::unordered_set<const IrValue*>> used;
+    std::vector<std::uint64_t> work;
+    const auto use = [&](IrValue* load, std::uint32_t lane) {
+        const auto key = keyOf(load->Argument(0)->Resolve()->ImmediateU32(), lane);
+        if (!used[key].insert(load).second) return false;
+        work.push_back(key);
+        return true;
+    };
+    for (const auto& [load, lane] : folded) use(load, lane);
+    std::unordered_map<const IrValue*, std::set<std::uint64_t>> guarded;
+    while (!work.empty()) {
+        const auto key = work.back();
+        work.pop_back();
+        const auto slot = static_cast<std::uint32_t>(key >> 32u);
+        const auto lane = static_cast<std::uint32_t>(key);
+        const auto& usedLoads = used[key];
+        std::unordered_map<const IrBlock*, bool> liveIn;
+        for (bool changed = true; changed;) {
+            changed = false;
+            for (auto block = program.Blocks().rbegin(); block != program.Blocks().rend(); ++block) {
+                bool live = false;
+                for (const IrBlock* successor : (*block)->Successors()) live = live || liveIn[successor];
+                for (auto inst = (*block)->Instructions().rbegin(); inst != (*block)->Instructions().rend(); ++inst) {
+                    IrValue* value = *inst;
+                    if (value->Opcode() == IrOpcode::LoadVectorWindow) {
+                        live = live || usedLoads.contains(value);
+                        continue;
+                    }
+                    if (value->Opcode() != IrOpcode::StoreVectorWindow) continue;
+                    const IrValue* index = value->Argument(0)->Resolve();
+                    if (!isImmediateU32(*index)) {
+                        if (live) guarded[value].insert(key);
+                        continue;
+                    }
+                    if (index->ImmediateU32() != slot) continue;
+                    if (live) {
+                        if (IrValue* source = laneSource(value->Argument(1), lane, waveSize); source != nullptr && use(source, lane)) changed = true;
+                    }
+                    live = false;
+                }
+                if (const auto found = liveIn.find(block->get()); found == liveIn.end() || found->second != live) {
+                    changed = changed || live || found != liveIn.end();
+                    liveIn[block->get()] = live;
+                }
+            }
+        }
+    }
+    std::size_t guards = 0;
+    for (const auto& block : program.Blocks()) {
+        for (IrValue* store : std::vector<IrValue*>(block->Instructions().begin(), block->Instructions().end())) {
+            const auto found = guarded.find(store);
+            if (found == guarded.end()) continue;
+            IrValue& current = program.CreateValue(IrOpcode::LoadVectorWindow, IrType::U32);
+            current.AddArgument(store->Argument(0));
+            block->InsertInstructionBefore(store, &current);
+            for (const auto key : found->second) {
+                IrValue& guard = program.CreateValue(IrOpcode::WindowLaneGuard, IrType::Void);
+                for (IrValue* argument : {store->Argument(0), store->Argument(1), &current, &immediate(program, static_cast<std::uint32_t>(key >> 32u)), &immediate(program, static_cast<std::uint32_t>(key))}) guard.AddArgument(argument);
+                block->InsertInstructionBefore(store, &guard);
+                ++guards;
+            }
+        }
+    }
+    if (guards != 0u) program.Info().usesFaultBuffer = true;
+    return guards;
 }
 
 ChainResult searchChain(IrValue* value, std::uint32_t lane, std::uint32_t waveSize) {
@@ -362,7 +418,7 @@ ReadLaneEliminationStats ReadLaneEliminator::Eliminate(IrProgram& program, std::
     }
     WindowFlows flows;
     flows.program = &program;
-    flows.spillSlots = collectLaneSpillSlots(program);
+    std::vector<std::pair<IrValue*, std::uint32_t>> folded;
     struct FlowScope {
         ~FlowScope() { windowFlows = nullptr; }
     } flowScope;
@@ -381,6 +437,7 @@ ReadLaneEliminationStats ReadLaneEliminator::Eliminate(IrProgram& program, std::
             const std::uint32_t lane = selector->ImmediateU32() % waveSize;
             const ChainResult chain = searchChain(inst->Argument(0), lane, waveSize);
             if (chain.write != nullptr) {
+                if (chain.value->Opcode() == IrOpcode::LoadVectorWindow) folded.emplace_back(chain.value, lane);
                 inst->ReplaceUsesWith(chain.scalar, true);
                 stats.rewrittenReads++;
                 continue;
@@ -397,10 +454,14 @@ ReadLaneEliminationStats ReadLaneEliminator::Eliminate(IrProgram& program, std::
             }
 
             PhiMap phiMap;
-            inst->ReplaceUsesWith(getRealValue(program, phiMap, chain.value, lane, waveSize), true);
+            flows.folded = &folded;
+            IrValue* real = getRealValue(program, phiMap, chain.value, lane, waveSize);
+            flows.folded = nullptr;
+            inst->ReplaceUsesWith(real, true);
             stats.rewrittenReads++;
         }
     }
+    stats.guardedStores = static_cast<std::uint32_t>(guardSpilledLanes(program, folded, waveSize));
     return stats;
 }
 

core/shader/recompiler/Recompiler.cpp

@@ -666,6 +666,8 @@ CompiledVariant emitVariant(const RecompileRequest& request, IrProgram& program)
     }
 
     result.bdaAbiVersion = program.Info().usesDma || program.Info().usesFaultBuffer ? request.target.bdaAbiVersion : 0u;
+    result.windowLaneGuards = 0;
+    for (const auto& block : program.Blocks()) result.windowLaneGuards += static_cast<std::uint32_t>(std::ranges::count_if(block->Instructions(), [](const IrValue* value) { return value->Opcode() == IrOpcode::WindowLaneGuard; }));
     result.memoryOffsetDword = bindings.layout.memoryOffsetDword;
     result.shaderDataDwords = bindings.layout.ShaderDataDwords();
     result.imageMetadataDword = bindings.layout.ImageMetadataDword();

core/shader/recompiler/Recompiler.hpp

@@ -653,6 +653,7 @@ struct CompiledShaderArtifact {
     std::vector<std::uint32_t> runtimeImageResources;
     bool registrationModule = false;
     std::uint32_t bdaAbiVersion = 0;
+    std::uint32_t windowLaneGuards = 0;
     std::uint32_t runtimeAbiVersion = RuntimeAbi::Version;
     std::vector<VertexInput> vertexInputs;
     std::int32_t vertexOffsetSgpr = -1;

core/shader/recompiler/ShaderDiskCache.cpp

@@ -469,6 +469,7 @@ void encodeArtifact(Writer& writer, const CompiledShaderArtifact& result) {
     writer.Value(result.runtimeAbiVersion);
     writer.Values(std::span<const std::uint32_t>(result.spirv.Words()));
     writer.Value(result.bdaAbiVersion);
+    writer.Value(result.windowLaneGuards);
     writer.Value(result.memoryOffsetDword);
     writer.Value(result.shaderDataDwords);
     writer.Value(result.imageMetadataDword);
@@ -527,6 +528,7 @@ void decodeArtifact(Reader& reader, CompiledShaderArtifact& result) {
     reader.Value(result.runtimeAbiVersion);
     result.spirv = reader.Spirv();
     reader.Value(result.bdaAbiVersion);
+    reader.Value(result.windowLaneGuards);
     reader.Value(result.memoryOffsetDword);
     reader.Value(result.shaderDataDwords);
     reader.Value(result.imageMetadataDword);

core/shader/recompiler/ShaderDiskCache.hpp

@@ -13,7 +13,7 @@
 
 namespace ShaderRecompiler::ShaderDiskCache {
 
-inline constexpr std::uint32_t FormatVersion = 44;
+inline constexpr std::uint32_t FormatVersion = 45;
 
 enum class LoadStatus {
     Loaded,

core/shader/recompiler/SpirvBackend/include/SpirvBackend/SpirvEmitterInstructions.hpp

@@ -221,6 +221,7 @@ std::uint32_t EmitFPCmpClass32(SpirvEmitterState& state, std::uint32_t value, st
 void EmitReference(SpirvValueEmitContext& context);
 void EmitReferenceU32(SpirvValueEmitContext& context);
 void EmitUnlinkedCall(SpirvValueEmitContext& ctx, const IrValue& inst, std::uint32_t low, std::uint32_t high, std::uint32_t programCounter);
+void EmitWindowLaneGuard(SpirvValueEmitContext& ctx, const IrValue& inst, std::uint32_t index, std::uint32_t value, std::uint32_t current, std::uint32_t slot, std::uint32_t lane);
 void EmitControlNop(SpirvValueEmitContext& context);
 void EmitWaitcnt(SpirvValueEmitContext& context);
 void EmitSendmsg(SpirvValueEmitContext& context);

core/shader/recompiler/SpirvBackend/src/SpirvFlowEmitter.cpp

@@ -308,6 +308,7 @@ void EmitDirectInstruction(SpirvValueEmitContext& ctx, const IrValue& inst) {
         case IrOpcode::Reference: return Invoke(EmitReference, ctx, inst);
         case IrOpcode::ReferenceU32: return Invoke(EmitReferenceU32, ctx, inst);
         case IrOpcode::UnlinkedCall: return Invoke(EmitUnlinkedCall, ctx, inst);
+        case IrOpcode::WindowLaneGuard: return Invoke(EmitWindowLaneGuard, ctx, inst);
         case IrOpcode::GetUserData: return Invoke(EmitGetUserData, ctx, inst);
         case IrOpcode::GetShaderBase: return Invoke(EmitGetShaderBase, ctx, inst);
         case IrOpcode::ShaderClock: return Invoke(EmitShaderClock, ctx, inst);
@@ -920,6 +921,15 @@ void EmitUnlinkedCall(SpirvValueEmitContext& ctx, const IrValue& inst, std::uint
     StopBdaInvocationIf(state, ConstantBool(state, true));
 }
 
+void EmitWindowLaneGuard(SpirvValueEmitContext& ctx, const IrValue& inst, std::uint32_t index, std::uint32_t value, std::uint32_t current, std::uint32_t slot, std::uint32_t lane) {
+    auto& state = ctx.state;
+    if (state.faultBufferVariable == 0) ctx.Fail(inst, "checks a spilled lane without a fault buffer");
+    const auto boolean = TypeBool(state);
+    const auto ownLane = Binary(state, spv::OpIEqual, boolean, EmitSubgroupLocalInvocationId(state), Binary(state, spv::OpBitwiseAnd, TypeU32(state), lane, ConstantU32(state, state.program.WaveSize() - 1u)));
+    const auto overwritten = Binary(state, spv::OpLogicalAnd, boolean, Binary(state, spv::OpIEqual, boolean, index, slot), Binary(state, spv::OpINotEqual, boolean, value, current));
+    EmitIfCondition(state, Binary(state, spv::OpLogicalAnd, boolean, ownLane, overwritten), [&] { RecordBdaFaultWords(state, slot, lane, value, ConstantU32(state, 0u), BdaAbi::FaultReason::WindowLaneSpill); });
+}
+
 void EmitControlNop(SpirvValueEmitContext& context) {
     EmitVoid(context);
 }

docs/dev/debt/shader.md

@@ -166,6 +166,7 @@
 - [Static scalar calls](../../../core/shader/recompiler/ControlFlow/src/GraphBuilder.cpp) (`s_call_b64` and `s_swappc_b64`) require an ordinary even-aligned SGPR link pair (s0:s1 through s104:s105) and a later `s_setpc_b64` return reading that exact pair. Outside every region of its calls the pair is free to use wherever no return address can reach: a dataflow over the decoded instructions follows each dword of the pair from the calls that write it until a scalar load, SALU or VALU result is known to overwrite it, and an instruction there that may read a dword still holding a return address throws; SOPC compares are exempt, as they only set SCC. A VALU mask covers one SGPR in wave32 and a pair in wave64. When the graph is built without the wave size, the dataflow runs for both: a program that escapes under both throws there, and one that escapes under only one throws when it is translated at that size.
 - [Static scalar call link pairs](../../../core/shader/recompiler/ControlFlow/src/GraphBuilder.cpp) each serve one call, or several calls whose regions are disjoint, each call closed by the first return reading the pair after its target. Regions must nest or be disjoint; recursion, other shared links, escaping link values (including overlapping tuples and resource descriptors), invalid targets, interleaved regions, cross-region branches or fall-through, relative SGPR reads and dynamic `s_setpc_b64` in a program with calls throw. The analysis checks all decoded instructions, including unreachable code. Arbitrary indirect calls and general call frames are not implemented.
 - [Relative vector addressing](../../../core/shader/recompiler/Translation/src/ControlFlowInstructions.cpp) (`v_movrels_b32`, `v_movreld_b32`, `v_movrelsd_b32`, `v_movrelsd_2_b32`, `v_swaprel_b32`) keeps every vector register from the lowest base register the program addresses relatively up to the last register the program names in zero-initialised function-scope storage per invocation (`LoadVectorWindow`/`StoreVectorWindow`, one element per register), so an indexed access is one load or store instead of a compare-and-select chain over the window. [`IndexedWindowSlots`](../../../core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvModuleSetup.cpp) bounds each relative index through the IR (bit-field extracts, masks, shifts, adds, selects, phis, `ReadFirstLane` and the window stores that reach a load); the storage is an array of the slots below the largest bound that is under half the window, and one function variable per slot after it. A constant index uses the array element or the slot's variable; a relative access whose index has no such bound, or lands past the array at run time, indexes the array below its size and otherwise branches through an `OpSwitch` over the slot variables, so the bound only decides speed. When no index has a bound under half the window, the whole window is the array, as with `APS5_NO_WINDOW_NARROWING=1` (a recompiler switch in the shader cache key). Measured on an NVIDIA GeForce RTX 3060 (driver 615.78.08, Linux) with a compute program whose 167-register window an ALU loop reads and writes in full and whose relative accesses take `m0 = s8 & 3`: a 2048-group dispatch took 70.2 ms in wave32 and 123.3 ms in wave64 with the whole window in the array (as before narrowing) and 1.9-2.1 ms and 7.8 ms narrowed; with a second relative access whose index has no bound and lands past the array at run time (index 50 or 166), 113 ms and 5.8-6.0 ms in wave32. In the PPSA08427 (Killing Floor 3) hub it made no measurable difference to presents or to the GPU time of the eight most expensive programs. Whether the host compiler keeps the array and the slot variables in registers or spills them to scratch is not measured. A register in the window the program never wrote reads 0, as an unwritten SSA register does; the console's value is undefined. M0 is unsigned: an index at or past the window reads `v0` and drops its write, and `v_swaprel_b32` writes neither side when either is out of range, as measured on RDNA2 (gfx1035, `agc_driver_relative_indexing`). The window ends at the last register the program names rather than at the shader's VGPR allocation, which the recompiler does not receive, so a relative write past the named registers but inside the allocation is dropped where the hardware keeps it for a later relative read. Killzone: Liberation's PSP vertex program (PPSA11142, 3176 ISA words, 104 relative accesses over 154 registers) went from 649,515 to 392,333 SPIR-V words with this. Pixel programs whose relative span contains the packed ancillary input instead keep that span in SSA and implement indexed operations with compare/select chains. This keeps ancillary definitions visible to field lowering and masked-write proofs; storing the unmodelled whole word in a private array would require composing unknown bits. A fold after masked-select elimination lowers fields whose consumers become visible only after redundant copies are removed. A whole-word use that the masked-select pass keeps gets the composed word described under the packed pixel ancillary word. Both representations include implicit entry inputs in the inferred register limit and retain unsigned M0, inactive-write and DPP update rules; wave32/wave64 native Vulkan readbacks cover relative reads, writes, split-offset swaps, inactive writes, DPP masks and out-of-range accesses, but console comparison and the performance cost of selecting over this span are unverified.
+- [Lane spills read back with `v_readlane_b32`](../../../core/shader/recompiler/Optimization/src/ReadLaneEliminator.cpp) (shader recompiler) - not verified for the guest compiler: a relative write (`v_movreld_b32`, `v_movrelsd_b32`, `v_movrelsd_2_b32`, `v_swaprel_b32`) with a computed index is taken to miss a register between a `v_writelane_b32` of a lane and a later `v_readlane_b32` of it, so the read becomes the spilled scalar. That holds when the compiler keeps spill registers out of its relatively indexed arrays. Each such write that a replaced read depends on is checked at run time: one that changes that lane of that register records a fault, and the dispatch or draw fails with `which holds a scalar spilled with v_writelane_b32`, where the console would read the new value ([m0 = 1 counterexample](https://github.com/boykopovar/AnyPS5/pull/6277#issuecomment-6215244790)). An inactive or out-of-range lane writes nothing. A constant-index write of the register still ends the spill.
 - [s_call_b64](../../../core/shader/recompiler/Translation/src/ScalarInstructions.cpp) saves `PC + 4` over `GetShaderBase` and branches to `PC + 4 + signext(SIMM16) * 4`, following [AMD RDNA2 ISA](https://gpuopen.com/download/rdna2-shader-instruction-set-architecture.pdf) sections 3.6.3, 12.2 and 12.3. Forward callees after `s_endpgm` are decoded. Measured on RDNA2 (gfx1036) in wave32 and wave64 with SIMM16 = 3: the link pair receives `PC + 4`, the branch goes to `PC + 4 + signext(SIMM16) * 4`, and `s_setpc_b64` of the link returns to the instruction after the call. Execution on the PS5 (gfx1013) and title compatibility for these bounded scalar calls remain unverified; `s_rfe_b64` still throws in translation.
 - [BDA anchor halfword reads](../../../core/shader/recompiler/SpirvBackend/src/SpirvBdaRead.cpp) retain volatile byte loads even for noncoherent memory. On RX 6650 XT / RADV Mesa 26.2.4, the valid SPIR-V for two adjacent nonvolatile byte loads returned `0x80a0` for bytes `0xa0, 0xa1`, where lookup-based reads returned `0xa1a0`. Adding the volatile operand makes the same GPU exactness fixture pass, including mixed direct/fallback lanes. The driver transformation responsible has not been isolated; removing this restriction needs equivalent readback coverage on the affected driver. Single-byte reads keep their original access mask.
 - [BDA span reads](../../../core/shader/recompiler/SpirvBackend/src/SpirvBdaRead.cpp) (shader recompiler) - `read_bda_span` is one selection: one arm calls `load_bda_span`, a `DontInline` function that holds the in-place dword loads and calls nothing, and the other arm is the per-dword byte fallback. Two other shapes of the same reads each hang a group of NVIDIA drivers, where a dispatch never finishes and the device is lost. With the loads emitted in the arm itself, a dispatch whose lanes took both arms hung on an RTX 3060 and an RTX 4060 Laptop GPU (Windows, driver 591.86) and an RTX 5060 (Linux, driver 590.48.01) ([#3291](https://github.com/boykopovar/AnyPS5/issues/3291)). With the loads and the fallback as two selections, the first under the span's condition and the second under its negation, it hung on an RTX 4070 Laptop GPU (Windows, drivers 595.97 and 596.36), an RTX 5080 (Windows, driver 610.62) and an RTX 4090 (Linux, driver 610.57.04) ([#4975](https://github.com/boykopovar/AnyPS5/issues/4975)). The current shape is measured on the RTX 4090 with driver 610.57.04 only, where it does not hang with or without SPIRV-Tools; a shape with the loads in their own `DontInline` function was reported passing on driver 591.86 in #3291. The cause inside the driver is not known.