Merge upstream main into fix/audit-2026-09

Resolve conflicts with boykopovar/AnyPS5#10 (sce module loading) and
later upstream changes.

- Pthread Cond/Mutex: take upstream. It already releases mutex
  ownership across condition waits, returns ETIMEDOUT 0x8002003C, and
  rejects unlocks from non-owners for every mutex type.
- Pthread Thread: keep upstream's lifecycle, Windows adoption and job
  affinity; re-apply the POSIX fixes on top: reference-counted
  release on join/detach/exit, self-join rejection on both platforms,
  and scePthreadSelf adoption for threads not created by
  scePthreadCreate.
- Semaphore: keep upstream's TimedWait::Condition and wait tracing;
  re-apply sceKernelDeleteSema waking waiters with EACCES.
- Relinker: keep --windows-gui alongside upstream's --skip-sce-module,
  trampoline-aware Patch signature and new Python tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018emyZ4Rk8yoSJvxTFBe4DL

.gitignore

@@ -32,6 +32,7 @@ _SaveData/
 _sd/
 
 build/
+build_tools/
 b/
 Build/
 build-*/

CMakeLists.txt

@@ -23,7 +23,7 @@ set(SDL_WAYLAND OFF CACHE BOOL "" FORCE)
 set(SDL_OPENGL OFF CACHE BOOL "" FORCE)
 set(SDL_OPENGLES OFF CACHE BOOL "" FORCE)
 set(SDL_VULKAN ON CACHE BOOL "" FORCE)
-set(SDL_JOYSTICK OFF CACHE BOOL "" FORCE)
+set(SDL_JOYSTICK ON CACHE BOOL "" FORCE)
 set(SDL_HAPTIC OFF CACHE BOOL "" FORCE)
 set(SDL_HIDAPI OFF CACHE BOOL "" FORCE)
 set(SDL_SENSOR OFF CACHE BOOL "" FORCE)

README.md

@@ -4,13 +4,17 @@ Tool for automatic executables porting to Linux and Windows.
 
 Includes a [relinker](core/relinker) that converts executable to the target system's native format and implementations of [system prx libraries](core/libs/prx) suitable for dynamic linking. No emulation or separate runtime process.
 
+`relinker --to-intel` rewrites the Zen 2-only instructions a title may contain (EXTRQ, INSERTQ, MOVNTSS, MOVNTSD) at relink time, in place when an equal-length Intel encoding exists and otherwise through a jump to a generated stub section, so Intel hosts need no runtime emulation.
+
+By default the relinker also converts the title's own `sce_module`/`sce_modules` prx files into guest modules that take priority at link time (they require eager binding). A title whose `sce_module` ships its own `libc.prx`, such as Demon's Souls (PPSA01342), is relinked with `--skip-sce-module`: its libc would replace the project's [libc.prx](core/libs/prx/libc) (guest memory arena, heap, crash reporting) and imports kernel functions that are not implemented, so the executable fails to bind at startup.
+
 Releases will be published after the first full successful launch of at least one game.
 
 ## Status
 
 Execution reaches `_start`, [stack unwinding](core/libs/prx/libc/src/exception/Unwind.cpp) and exception handling tables are built, reaches main. Unsupported or unexpected states strictly throw `std::runtime_error`. `what()` is printed to stderr and the process terminates.
 
-The [shader recompiler](core/shader/recompiler/Recompiler.cpp) successfully produces validated via [Spirv-Tools](3rdparty/SPIRV-Tools) SPIR-V.
+The [shader recompiler](core/shader/recompiler/Recompiler.cpp) successfully produces SPIR-V, validated via [Spirv-Tools](3rdparty/SPIRV-Tools) when built with `ANYPS5_ENABLE_SPIRV_TOOLS`.
 
 The real game reaches the logo, main menu, and [gameplay](https://gist.github.com/user-attachments/assets/81d28e9b-c237-4545-b2ca-720071129816) with audio.
 
@@ -24,6 +28,10 @@ The relinker uses only the C++20 standard library and should build with any conf
 
 The project targets maximum compiler portability. Support for additional compilers will be addressed after the first successful game launch.
 
+## Input mapping
+
+SDL-mapped game controllers are supported, including analog sticks and triggers. Keyboard and mouse controls can be configured with an `anyps5-input.ini` file. See [input mapping](docs/INPUT_MAPPING.md) for the supported devices and configuration format.
+
 ## Disclaimer
 
 This project is intended for interoperability, research, preservation, and compatibility purposes. It does not include, distribute, or require copyrighted software, firmware, cryptographic keys, or proprietary libraries. Users are responsible for ensuring that any binaries used with this project are obtained and used in accordance with applicable laws and their respective license terms.

core/libs/CMakeLists.txt

@@ -109,6 +109,34 @@ endforeach()
 add_custom_target(libs DEPENDS ${PATCHED_OUTPUTS})
 
 if(BUILD_TESTING)
+    add_library(host_thread_local_test_library SHARED tests/HostThreadLocalLibrary.cpp)
+    target_include_directories(host_thread_local_test_library PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    add_executable(host_thread_local_tests tests/HostThreadLocal.cpp)
+    target_link_libraries(host_thread_local_tests PRIVATE host_thread_local_test_library)
+    if(MINGW)
+        target_compile_options(host_thread_local_test_library PRIVATE -fasynchronous-unwind-tables)
+        target_compile_options(host_thread_local_tests PRIVATE -fasynchronous-unwind-tables)
+    endif()
+    add_test(NAME host_thread_local COMMAND host_thread_local_tests)
+    set_tests_properties(host_thread_local PROPERTIES TIMEOUT 15)
+    add_executable(texture_format_lookup_tests tests/TextureFormatLookup.cpp)
+    target_include_directories(texture_format_lookup_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_SOURCE_DIR}/3rdparty/Vulkan-Headers/include)
+    target_link_libraries(texture_format_lookup_tests PRIVATE libSceAgcDriver libc)
+    configure_windows_unwind(texture_format_lookup_tests)
+    add_test(NAME texture_format_lookup COMMAND texture_format_lookup_tests)
+    if(WIN32)
+        set_tests_properties(texture_format_lookup PROPERTIES ENVIRONMENT_MODIFICATION
+            "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
+    endif()
+    add_executable(linear_display_buffer_tests tests/LinearDisplayBuffer.cpp)
+    target_include_directories(linear_display_buffer_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_link_libraries(linear_display_buffer_tests PRIVATE libSceAgcDriver libc)
+    configure_windows_unwind(linear_display_buffer_tests)
+    add_test(NAME linear_display_buffer COMMAND linear_display_buffer_tests)
+    if(WIN32)
+        set_tests_properties(linear_display_buffer PROPERTIES ENVIRONMENT_MODIFICATION
+            "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
+    endif()
     add_executable(guest_rendering_wait_tests tests/GuestRenderingWait.cpp)
     target_include_directories(guest_rendering_wait_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
     target_link_libraries(guest_rendering_wait_tests PRIVATE libSceAgcDriver libc)
@@ -127,6 +155,23 @@ if(BUILD_TESTING)
         set_tests_properties(guest_buffer_reuse PROPERTIES ENVIRONMENT_MODIFICATION
             "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
     endif()
+    add_executable(guest_indexed_draw_tests tests/GuestIndexedDraw.cpp)
+    add_executable(shader_buffer_address_tests tests/ShaderBufferAddress.cpp)
+    target_link_libraries(shader_buffer_address_tests PRIVATE shader_recompiler_agc_driver libc)
+    configure_windows_unwind(shader_buffer_address_tests)
+    add_test(NAME shader_buffer_address COMMAND shader_buffer_address_tests)
+    if(WIN32)
+        set_tests_properties(shader_buffer_address PROPERTIES ENVIRONMENT_MODIFICATION
+            "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
+    endif()
+    target_include_directories(guest_indexed_draw_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_link_libraries(guest_indexed_draw_tests PRIVATE libSceAgc libSceAgcDriver libc)
+    configure_windows_unwind(guest_indexed_draw_tests)
+    add_test(NAME guest_indexed_draw COMMAND guest_indexed_draw_tests)
+    if(WIN32)
+        set_tests_properties(guest_indexed_draw PROPERTIES ENVIRONMENT_MODIFICATION
+            "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
+    endif()
     add_executable(guest_shader_link_tests tests/GuestShaderLink.cpp)
     target_include_directories(guest_shader_link_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
     target_link_libraries(guest_shader_link_tests PRIVATE libSceAgc libc)
@@ -330,6 +375,22 @@ if(BUILD_TESTING)
         set_tests_properties(guest_rtc PROPERTIES ENVIRONMENT_MODIFICATION
             "PATH=path_list_prepend:$<TARGET_FILE_DIR:libSceRtc>")
     endif()
+    add_executable(guest_json_tests tests/GuestJson.cpp)
+    target_include_directories(guest_json_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_link_libraries(guest_json_tests PRIVATE libSceJson2)
+    add_test(NAME guest_json COMMAND guest_json_tests)
+    if(WIN32)
+        set_tests_properties(guest_json PROPERTIES ENVIRONMENT_MODIFICATION
+            "PATH=path_list_prepend:$<TARGET_FILE_DIR:libSceJson2>")
+    endif()
+    add_executable(fiber_tests tests/Fiber.cpp)
+    target_include_directories(fiber_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_link_libraries(fiber_tests PRIVATE libSceFiber)
+    add_test(NAME fiber COMMAND fiber_tests)
+    if(WIN32)
+        set_tests_properties(fiber PROPERTIES ENVIRONMENT_MODIFICATION
+            "PATH=path_list_prepend:$<TARGET_FILE_DIR:libSceFiber>")
+    endif()
     add_executable(mspace_tests tests/Mspace.cpp)
     target_include_directories(mspace_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
     target_link_libraries(mspace_tests PRIVATE libc)
@@ -348,6 +409,12 @@ if(BUILD_TESTING)
         set_tests_properties(guest_memory PROPERTIES ENVIRONMENT_MODIFICATION
             "PATH=path_list_prepend:$<TARGET_FILE_DIR:libkernel>")
     endif()
+    if(NOT WIN32)
+        add_executable(guest_directory_entry_tests tests/GuestDirectoryEntries.cpp)
+        target_include_directories(guest_directory_entry_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+        target_link_libraries(guest_directory_entry_tests PRIVATE libkernel libc)
+        add_test(NAME guest_directory_entries COMMAND guest_directory_entry_tests)
+    endif()
     add_executable(guest_search_tests tests/GuestSearch.cpp)
     target_include_directories(guest_search_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
     target_link_libraries(guest_search_tests PRIVATE libc)
@@ -366,11 +433,29 @@ if(BUILD_TESTING)
         set_tests_properties(guest_strings PROPERTIES ENVIRONMENT_MODIFICATION
             "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
     endif()
+    if(UNIX)
+        add_executable(exception_runtime_tests tests/ExceptionRuntime.cpp)
+        target_include_directories(exception_runtime_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+        target_link_libraries(exception_runtime_tests PRIVATE libc)
+        add_test(NAME exception_runtime COMMAND exception_runtime_tests)
+    endif()
 endif()
 
-add_executable(application_heap_tests EXCLUDE_FROM_ALL tests/ApplicationHeap.cpp prx/libc/src/ApplicationHeap.cpp)
+add_executable(application_heap_tests EXCLUDE_FROM_ALL tests/ApplicationHeap.cpp)
 target_include_directories(application_heap_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+target_link_libraries(application_heap_tests PRIVATE libc)
+configure_windows_unwind(application_heap_tests)
 set_target_properties(application_heap_tests PROPERTIES CXX_EXTENSIONS OFF)
+if(BUILD_TESTING)
+    set_target_properties(application_heap_tests PROPERTIES EXCLUDE_FROM_ALL FALSE)
+    add_test(NAME application_heap_replacement COMMAND application_heap_tests)
+    add_test(NAME application_heap_invalid_metadata COMMAND application_heap_tests invalid)
+    add_test(NAME application_heap_default COMMAND application_heap_tests default)
+    if(WIN32)
+        set_tests_properties(application_heap_replacement application_heap_invalid_metadata application_heap_default
+            PROPERTIES ENVIRONMENT_MODIFICATION "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
+    endif()
+endif()
 
 add_executable(guest_locale_tests EXCLUDE_FROM_ALL tests/GuestLocale.cpp)
 target_include_directories(guest_locale_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})

core/libs/SceTypes.hpp

@@ -173,9 +173,9 @@ using PthreadCondattr = PthreadCondattrPrivate*;
 using PthreadKey = int;
 using pthread_entry_func_t = void* (*)(void*);
 using pthread_key_destructor_func_t = void (*)(void*);
-using thread_dtors_func_t = void (*)();
-using get_thread_atexit_count_func_t = int (*)(KernelModule);
-using thread_atexit_report_func_t = void (*)(KernelModule);
+using thread_dtors_func_t = void (APS5_VABI *)();
+using get_thread_atexit_count_func_t = int (APS5_VABI *)(KernelModule);
+using thread_atexit_report_func_t = void (APS5_VABI *)(KernelModule);
 
 struct FileStat {
     std::uint32_t st_dev;

core/libs/nid/include/nid/NidPatcherUtils.hpp

@@ -44,7 +44,7 @@ inline std::string StripNidPostfix(const std::string& name) {
     std::string result = name;
     if (result.size() >= kNidPostfixLen &&
         result.compare(result.size() - kNidPostfixLen, kNidPostfixLen, kNidPostfix) == 0) {
-        result = result.substr(0u, result.size() - kNidPostfixLen);
+        result.resize(result.size() - kNidPostfixLen);
     }
     const auto disambigPos = result.rfind(kNidDisambigMarker);
     if (disambigPos != std::string::npos) {
@@ -54,7 +54,7 @@ inline std::string StripNidPostfix(const std::string& name) {
             [](unsigned char c) { return std::isdigit(c) != 0; }
         );
         if (allDigits && suffixStart < result.size())
-            result = result.substr(0u, disambigPos);
+            result.resize(disambigPos);
     }
     return result;
 }

core/libs/prx/libRenoirCore.PS5/Export.cpp

@@ -0,0 +1,145 @@
+#include <cstdint>
+#include "SceTypes.hpp"
+#include "prx/libc/include/General.hpp"
+
+extern "C" {
+
+APS5_EXPORT("0DsDTr534gc", renoirStub00);
+uint64_t APS5_VABI renoirStub00(void) {
+ NotImplemented_nid_no_patch("0DsDTr534gc");
+ return 0;
+}
+
+APS5_EXPORT("2T56HgbPW08", renoirStub01);
+uint64_t APS5_VABI renoirStub01(void) {
+ NotImplemented_nid_no_patch("2T56HgbPW08");
+ return 0;
+}
+
+APS5_EXPORT("3HnCozh8Az8", renoirStub02);
+uint64_t APS5_VABI renoirStub02(void) {
+ NotImplemented_nid_no_patch("3HnCozh8Az8");
+ return 0;
+}
+
+APS5_EXPORT("5Ysce+s-CWU", renoirStub03);
+uint64_t APS5_VABI renoirStub03(void) {
+ NotImplemented_nid_no_patch("5Ysce+s-CWU");
+ return 0;
+}
+
+APS5_EXPORT("AdlSqA4lGS8", renoirStub04);
+uint64_t APS5_VABI renoirStub04(void) {
+ NotImplemented_nid_no_patch("AdlSqA4lGS8");
+ return 0;
+}
+
+APS5_EXPORT("bC37hk7rHqs", renoirStub05);
+uint64_t APS5_VABI renoirStub05(void) {
+ NotImplemented_nid_no_patch("bC37hk7rHqs");
+ return 0;
+}
+
+APS5_EXPORT("dfPK9R6M0qM", renoirStub06);
+uint64_t APS5_VABI renoirStub06(void) {
+ NotImplemented_nid_no_patch("dfPK9R6M0qM");
+ return 0;
+}
+
+APS5_EXPORT("fZiBbVa2ZDU", renoirStub07);
+uint64_t APS5_VABI renoirStub07(void) {
+ NotImplemented_nid_no_patch("fZiBbVa2ZDU");
+ return 0;
+}
+
+APS5_EXPORT("hkWBH8-WfW4", renoirStub08);
+uint64_t APS5_VABI renoirStub08(void) {
+ NotImplemented_nid_no_patch("hkWBH8-WfW4");
+ return 0;
+}
+
+APS5_EXPORT("hUxvGdi64Io", renoirStub09);
+uint64_t APS5_VABI renoirStub09(void) {
+ NotImplemented_nid_no_patch("hUxvGdi64Io");
+ return 0;
+}
+
+APS5_EXPORT("NxqsZxGisrA", renoirStub10);
+uint64_t APS5_VABI renoirStub10(void) {
+ NotImplemented_nid_no_patch("NxqsZxGisrA");
+ return 0;
+}
+
+APS5_EXPORT("OgZVIkaNmb0", renoirStub11);
+uint64_t APS5_VABI renoirStub11(void) {
+ NotImplemented_nid_no_patch("OgZVIkaNmb0");
+ return 0;
+}
+
+APS5_EXPORT("ojLQdRUnVgA", renoirStub12);
+uint64_t APS5_VABI renoirStub12(void) {
+ NotImplemented_nid_no_patch("ojLQdRUnVgA");
+ return 0;
+}
+
+APS5_EXPORT("s0qDfgsAgVg", renoirStub13);
+uint64_t APS5_VABI renoirStub13(void) {
+ NotImplemented_nid_no_patch("s0qDfgsAgVg");
+ return 0;
+}
+
+APS5_EXPORT("sgrd-r1Nw60", renoirStub14);
+uint64_t APS5_VABI renoirStub14(void) {
+ NotImplemented_nid_no_patch("sgrd-r1Nw60");
+ return 0;
+}
+
+APS5_EXPORT("sjTRb8YD7dQ", renoirStub15);
+uint64_t APS5_VABI renoirStub15(void) {
+ NotImplemented_nid_no_patch("sjTRb8YD7dQ");
+ return 0;
+}
+
+APS5_EXPORT("tG86XGc2VtE", renoirStub16);
+uint64_t APS5_VABI renoirStub16(void) {
+ NotImplemented_nid_no_patch("tG86XGc2VtE");
+ return 0;
+}
+
+APS5_EXPORT("umlGxOcNzgc", renoirStub17);
+uint64_t APS5_VABI renoirStub17(void) {
+ NotImplemented_nid_no_patch("umlGxOcNzgc");
+ return 0;
+}
+
+APS5_EXPORT("vPxA9OFW0os", renoirStub18);
+uint64_t APS5_VABI renoirStub18(void) {
+ NotImplemented_nid_no_patch("vPxA9OFW0os");
+ return 0;
+}
+
+APS5_EXPORT("X2qEBlOLkNo", renoirStub19);
+uint64_t APS5_VABI renoirStub19(void) {
+ NotImplemented_nid_no_patch("X2qEBlOLkNo");
+ return 0;
+}
+
+APS5_EXPORT("ynh4E+BIaqY", renoirStub20);
+uint64_t APS5_VABI renoirStub20(void) {
+ NotImplemented_nid_no_patch("ynh4E+BIaqY");
+ return 0;
+}
+
+APS5_EXPORT("YXe6ABbHZPs", renoirStub21);
+uint64_t APS5_VABI renoirStub21(void) {
+ NotImplemented_nid_no_patch("YXe6ABbHZPs");
+ return 0;
+}
+
+APS5_EXPORT("z0jsXZVz75I", renoirStub22);
+uint64_t APS5_VABI renoirStub22(void) {
+ NotImplemented_nid_no_patch("z0jsXZVz75I");
+ return 0;
+}
+
+}

core/libs/prx/libSceAcm/Export.cpp

@@ -2,47 +2,55 @@
 #include <cstddef>
 #include "SceTypes.hpp"
 #include "prx/libc/include/General.hpp"
+#include <atomic>
+
+// Audio co-processor batches complete immediately without producing output; audio is silent.
+static constexpr int SCE_ACM_ERROR_INVALID_PARAMETER = static_cast<int>(0x80E30002);
+static std::atomic<uint32_t> g_nextId{1};
 
 extern "C" {
 
 int APS5_VABI sceAcmBatchStartBuffer(AcmContextId context, const void* batch_commands, size_t batch_size, AcmBatchError* batch_error, AcmBatchId* batch) {
- (void)context;
- (void)batch_commands;
- (void)batch_size;
- (void)batch_error;
- (void)batch;
- NotImplemented_nid_no_patch(__func__);
- return 0;
+    (void)context;
+    (void)batch_commands;
+    (void)batch_size;
+    (void)batch_error;
+    if (!batch) return SCE_ACM_ERROR_INVALID_PARAMETER;
+    *batch = g_nextId.fetch_add(1, std::memory_order_relaxed);
+    return 0;
 }
 
 int APS5_VABI sceAcmBatchStartBuffers(AcmContextId context, uint32_t batch_info_count, const AcmBatchInfo* const batch_info[], AcmBatchError* batch_error, AcmBatchId* batch) {
- (void)context;
- (void)batch_info_count;
- (void)batch_info;
- (void)batch_error;
- (void)batch;
- NotImplemented_nid_no_patch(__func__);
- return 0;
+    (void)context;
+    (void)batch_info_count;
+    (void)batch_info;
+    (void)batch_error;
+    if (!batch) return SCE_ACM_ERROR_INVALID_PARAMETER;
+    *batch = g_nextId.fetch_add(1, std::memory_order_relaxed);
+    return 0;
 }
 
 int APS5_VABI sceAcmBatchWait(AcmContextId context, AcmBatchId batch, uint32_t timeout) {
- (void)context;
- (void)batch;
- (void)timeout;
- NotImplemented_nid_no_patch(__func__);
- return 0;
+    (void)context;
+    (void)batch;
+    (void)timeout;
+    return 0;
 }
 
 int APS5_VABI sceAcmContextCreate(AcmContextId* context) {
- (void)context;
- NotImplemented_nid_no_patch(__func__);
- return 0;
+    if (!context) return SCE_ACM_ERROR_INVALID_PARAMETER;
+    *context = g_nextId.fetch_add(1, std::memory_order_relaxed);
+    return 0;
 }
 
 int APS5_VABI sceAcmContextDestroy(AcmContextId context) {
- (void)context;
- NotImplemented_nid_no_patch(__func__);
- return 0;
+    (void)context;
+    return 0;
+}
+
+// Builds a convolution-reverb command into a batch; batches execute as no-ops, so nothing is encoded.
+int APS5_VABI sceAcm_ConvReverb_SharedInput(void) {
+    return 0;
 }
 
 }

core/libs/prx/libSceAgc/Acb/src/Control.cpp

@@ -2,6 +2,7 @@
 
 #include "prx/libSceAgc/Command/include/Memory.hpp"
 #include "prx/libSceAgc/Command/include/Packet.hpp"
+#include "prx/libSceAgc/DcbState/include/Marker.hpp"
 #include <cstdint>
 #include <cstddef>
 #include "SceTypes.hpp"
@@ -30,11 +31,16 @@ uint32_t APS5_VABI sceAgcAcbJumpGetSize(void) {
  return 0;
 }
 
-uint32_t* APS5_VABI sceAgcAcbResetQueue(CommandBuffer* buf, uint32_t op) {
- (void)buf;
- (void)op;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+// Encoded as the custom DISPATCH_RESET NOP packet the driver consumes on compute queues.
+uint32_t* APS5_VABI sceAgcAcbResetQueue(CommandBuffer* buf, uint32_t op, uint32_t value) {
+    (void)value;
+    Agc::Command::Require(buf != nullptr, __func__, "null command buffer");
+    constexpr std::uint32_t OpcodeNop = 0x10;
+    constexpr std::uint32_t CustomDispatchReset = 0x09;
+    auto* packet = Agc::Command::Allocate(buf, 2, __func__);
+    packet[0] = Agc::Command::Header(OpcodeNop, 2, CustomDispatchReset << 2);
+    packet[1] = op;
+    return packet;
 }
 
 std::uint32_t* APS5_VABI sceAgcAcbRewind(CommandBuffer* buf, std::uint32_t initialState) {
@@ -68,25 +74,19 @@ std::uint32_t* APS5_VABI sceAgcAcbSetFlip(CommandBuffer* buf, std::uint32_t vide
 }
 
 uint32_t* APS5_VABI sceAgcAcbPushMarker(CommandBuffer* buf, const char* str, uint32_t color) {
- (void)buf;
- (void)str;
- (void)color;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    (void)color;
+    return Agc::Marker::Push(buf, str, __func__);
 }
 
 uint32_t* APS5_VABI sceAgcAcbPopMarker(CommandBuffer* buf) {
- (void)buf;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    return Agc::Marker::Pop(buf, __func__);
 }
 
 uint32_t* APS5_VABI sceAgcAcbSetMarker(CommandBuffer* buf, const char* str, uint32_t color) {
- (void)buf;
- (void)str;
- (void)color;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    (void)color;
+    auto* packet = Agc::Marker::Push(buf, str, __func__);
+    Agc::Marker::Pop(buf, __func__);
+    return packet;
 }
 
 }

core/libs/prx/libSceAgc/Acb/src/Dispatch.cpp

@@ -10,11 +10,11 @@
 extern "C" {
 
 uint32_t* APS5_VABI sceAgcAcbDispatchIndirect(CommandBuffer* buf, const volatile void* indirect_args, uint32_t modifier) {
- (void)buf;
- (void)indirect_args;
- (void)modifier;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    // Compute queues take the argument address directly (DISPATCH_INDIRECT with a 64-bit address).
+    const auto address = reinterpret_cast<std::uintptr_t>(indirect_args);
+    Agc::Command::CheckAddress(address, 4, __func__);
+    Agc::Command::CheckBits(modifier, 0xa038u | 0x41u, __func__);
+    return Agc::Command::Emit(buf, 0x16u, {static_cast<std::uint32_t>(address), static_cast<std::uint32_t>(address >> 32u), modifier | 0x41u}, __func__);
 }
 
 std::uint32_t APS5_VABI sceAgcAcbDispatchIndirectGetSize() {

core/libs/prx/libSceAgc/CMakeLists.txt

@@ -1,4 +1,5 @@
 add_library(libSceAgc SHARED EXCLUDE_FROM_ALL
+        ${CMAKE_CURRENT_SOURCE_DIR}/Unimplemented.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Acb/src/Control.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Acb/src/Dispatch.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Acb/src/Memory.cpp

core/libs/prx/libSceAgc/Cb/src/Dispatch.cpp

@@ -9,7 +9,7 @@
 extern "C" {
 
 std::uint32_t* APS5_VABI sceAgcCbDispatch(CommandBuffer* buf, std::uint32_t threadGroupX, std::uint32_t threadGroupY, std::uint32_t threadGroupZ, std::uint32_t modifier) {
-    Agc::Command::CheckBits(modifier, 0xa038u, __func__);
+    Agc::Command::CheckBits(modifier, 0xa038u | 0x41u, __func__);
     return Agc::Command::Emit(buf, 0x15u, {threadGroupX, threadGroupY, threadGroupZ, modifier | 0x41u}, __func__);
 }
 

core/libs/prx/libSceAgc/Cb/src/Sync.cpp

@@ -1,6 +1,8 @@
 #include "prx/libSceAgc/Cb/include/Sync.hpp"
 
 #include "prx/libSceAgc/Command/include/Packet.hpp"
+#include <cstdio>
+#include <cstdlib>
 #include <cstdint>
 #include <cstddef>
 #include "SceTypes.hpp"
@@ -40,7 +42,8 @@ std::uint32_t* APS5_VABI sceAgcCbReleaseMem(CommandBuffer* buf, std::uint8_t act
     } else {
         Agc::Command::Require(gdsOffset == 0 && gdsSize <= 1, __func__, "GDS parameters supplied for a non-GDS release");
     }
-    if (dataSelect != 0 && interrupt != 4) {
+    // A null label is accepted: games emit releases whose destination is never read.
+    if (dataSelect != 0 && interrupt != 4 && guestAddress != 0) {
         Agc::Command::CheckAddress(guestAddress, dataSelect == 2 || dataSelect == 3 ? 8 : 4, __func__);
     }
     if (dataSelect == 1) {
@@ -51,7 +54,10 @@ std::uint32_t* APS5_VABI sceAgcCbReleaseMem(CommandBuffer* buf, std::uint8_t act
         control |= 0x200u;
     }
     const auto eventIndex = action >= 0x2fu ? 6u : 5u;
-    return Agc::Command::Emit(buf, 0x49u, {action | (eventIndex << 8u) | (control << 12u) | (static_cast<std::uint32_t>(cachePolicy) << 25u), (static_cast<std::uint32_t>(dst) << 16u) | (static_cast<std::uint32_t>(interrupt) << 24u) | (static_cast<std::uint32_t>(dataSelect) << 29u), static_cast<std::uint32_t>(guestAddress), static_cast<std::uint32_t>(guestAddress >> 32u), static_cast<std::uint32_t>(value), static_cast<std::uint32_t>(value >> 32u), interruptContextId}, __func__);
+    auto* packet = Agc::Command::Emit(buf, 0x49u, {action | (eventIndex << 8u) | (control << 12u) | (static_cast<std::uint32_t>(cachePolicy) << 25u), (static_cast<std::uint32_t>(dst) << 16u) | (static_cast<std::uint32_t>(interrupt) << 24u) | (static_cast<std::uint32_t>(dataSelect) << 29u), static_cast<std::uint32_t>(guestAddress), static_cast<std::uint32_t>(guestAddress >> 32u), static_cast<std::uint32_t>(value), static_cast<std::uint32_t>(value >> 32u), interruptContextId}, __func__);
+    static const std::uint64_t traced = [] { const char* value = std::getenv("APS5_TRACE_LABEL"); return value ? std::strtoull(value, nullptr, 16) : 0ull; }();
+    if (traced != 0 && guestAddress == traced) std::fprintf(stderr, "[agc] RELEASE_MEM for 0x%llx recorded at %p"  "\n", static_cast<unsigned long long>(guestAddress), static_cast<void*>(packet));
+    return packet;
 }
 
 }

core/libs/prx/libSceAgc/Command/src/Memory.cpp

@@ -53,11 +53,13 @@ std::uint32_t* WriteWait(CommandBuffer* buffer, std::uint8_t size, std::uint8_t
     CheckBits(cachePolicy, 3, function);
     Require((pollCycles >> 4u) <= 0xffffu, function, "wait poll interval overflow");
     if (size == 0) {
-        CheckBits(reference, 0xffffffffu, function);
-        CheckBits(mask, 0xffffffffu, function);
+        // 32-bit waits only use the low halves; guests commonly pass a full 64-bit mask.
+        reference &= 0xffffffffu;
+        mask &= 0xffffffffu;
     }
     const auto guestAddress = reinterpret_cast<std::uintptr_t>(address);
-    CheckAddress(guestAddress, size == 0 ? 4 : 8, function);
+    // The address may be left null and filled in later with sceAgcWaitRegMemPatchAddress.
+    if (guestAddress != 0) CheckAddress(guestAddress, size == 0 ? 4 : 8, function);
     CheckBits(guestAddress, 0xffffffffffffull, function);
     const auto waitSize = size == 0 ? 7u : 9u;
     auto* packet = Allocate(buffer, waitSize + 7u, function);

core/libs/prx/libSceAgc/Command/src/Packet.cpp

@@ -1,6 +1,7 @@
 #include "prx/libSceAgc/Command/include/Packet.hpp"
 
 #include <algorithm>
+#include <cstdio>
 #include <limits>
 #include <stdexcept>
 #include <string>
@@ -15,11 +16,17 @@ void Require(bool condition, const char* function, const char* reason) {
 }
 
 void CheckBits(std::uint64_t value, std::uint64_t mask, const char* function) {
-    Require((value & ~mask) == 0, function, "reserved bits are set");
+    if ((value & ~mask) == 0) return;
+    char reason[96];
+    std::snprintf(reason, sizeof(reason), "reserved bits are set (value 0x%llx, allowed 0x%llx)", static_cast<unsigned long long>(value), static_cast<unsigned long long>(mask));
+    Require(false, function, reason);
 }
 
 void CheckAddress(std::uint64_t address, std::uint32_t alignment, const char* function) {
-    Require(address != 0 && (address & (alignment - 1u)) == 0, function, "null or misaligned address");
+    if (address != 0 && (address & (alignment - 1u)) == 0) return;
+    char message[64];
+    std::snprintf(message, sizeof(message), "null or misaligned address 0x%llx", static_cast<unsigned long long>(address));
+    Require(false, function, message);
 }
 
 std::uint32_t Header(std::uint32_t opcode, std::uint32_t count, std::uint32_t flags) {

core/libs/prx/libSceAgc/DcbDraw/src/DrawIndexed.cpp

@@ -9,13 +9,10 @@
 
 extern "C" {
 
-uint32_t* APS5_VABI sceAgcDcbDrawIndex(CommandBuffer* buf, uint32_t index_count, const volatile void* index_addr, uint64_t modifier) {
- (void)buf;
- (void)index_count;
- (void)index_addr;
- (void)modifier;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+std::uint32_t* APS5_VABI sceAgcDcbDrawIndex(CommandBuffer* buf, std::uint32_t indexCount, const volatile void* indexAddress, std::uint64_t modifier) {
+    const auto address = reinterpret_cast<std::uintptr_t>(indexAddress);
+    Agc::Command::CheckAddress(address, 1, __func__);
+    return Agc::Command::Emit(buf, 0x27u, {indexCount, static_cast<std::uint32_t>(address), static_cast<std::uint32_t>(address >> 32u), indexCount, Agc::Command::DrawInitiator(modifier, true, __func__)}, __func__);
 }
 
 std::uint32_t APS5_VABI sceAgcDcbDrawIndexGetSize() {
@@ -40,11 +37,9 @@ uint32_t APS5_VABI sceAgcDcbDrawIndexOffsetGetSize(void) {
 }
 
 uint32_t* APS5_VABI sceAgcDcbDrawIndexIndirect(CommandBuffer* buf, uint32_t data_offset_in_bytes, uint64_t modifier) {
- (void)buf;
- (void)data_offset_in_bytes;
- (void)modifier;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    Agc::Command::Require((data_offset_in_bytes & 3u) == 0, __func__, "misaligned indirect argument offset");
+    const auto offsets = Agc::Command::DrawPatchOffsets(modifier, __func__);
+    return Agc::Command::Emit(buf, 0x25u, {data_offset_in_bytes, static_cast<std::uint32_t>(offsets), static_cast<std::uint32_t>(offsets >> 32u), Agc::Command::DrawInitiator(modifier, true, __func__)}, __func__);
 }
 
 std::uint32_t APS5_VABI sceAgcDcbDrawIndexIndirectGetSize() {
@@ -52,15 +47,18 @@ std::uint32_t APS5_VABI sceAgcDcbDrawIndexIndirectGetSize() {
 }
 
 uint32_t* APS5_VABI sceAgcDcbDrawIndexIndirectMulti(CommandBuffer* buf, uint32_t data_offset_in_bytes, uint32_t count_indirect, uint32_t max_count_or_count, const volatile void* count_addr, uint32_t stride_in_bytes, uint64_t modifier) {
- (void)buf;
- (void)data_offset_in_bytes;
- (void)count_indirect;
- (void)max_count_or_count;
- (void)count_addr;
- (void)stride_in_bytes;
- (void)modifier;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    Agc::Command::CheckBits(count_indirect, 1, __func__);
+    Agc::Command::Require((data_offset_in_bytes & 3u) == 0 && (stride_in_bytes & 3u) == 0 && stride_in_bytes >= 20, __func__, "invalid indirect draw offset or stride");
+    const auto address = reinterpret_cast<std::uintptr_t>(count_addr);
+    if (count_indirect != 0) {
+        Agc::Command::CheckAddress(address, 4, __func__);
+    } else {
+        Agc::Command::Require(address == 0, __func__, "count address supplied for a direct draw count");
+    }
+    const auto offsets = Agc::Command::DrawPatchOffsets(modifier, __func__);
+    const auto low = static_cast<std::uint32_t>(modifier);
+    const auto control = Agc::Command::DrawIndexLocation(modifier) | ((low & 0x10u) << 23u) | (count_indirect << 30u) | ((low & 8u) << 28u);
+    return Agc::Command::Emit(buf, 0x38u, {data_offset_in_bytes, static_cast<std::uint32_t>(offsets), static_cast<std::uint32_t>(offsets >> 32u), control, max_count_or_count, static_cast<std::uint32_t>(address), static_cast<std::uint32_t>(address >> 32u), stride_in_bytes, Agc::Command::DrawInitiator(modifier, true, __func__)}, __func__);
 }
 
 std::uint32_t APS5_VABI sceAgcDcbDrawIndexIndirectMultiGetSize() {

core/libs/prx/libSceAgc/DcbDraw/src/IndexBuffer.cpp

@@ -9,7 +9,7 @@
 extern "C" {
 
 std::uint32_t* APS5_VABI sceAgcDcbSetIndexBuffer(CommandBuffer* buf, std::uint64_t indexAddress) {
-    Agc::Command::CheckAddress(indexAddress, 2, __func__);
+    if (indexAddress != 0) Agc::Command::CheckAddress(indexAddress, 2, __func__);
     return Agc::Command::Emit(buf, 0x26u, {static_cast<std::uint32_t>(indexAddress), static_cast<std::uint32_t>(indexAddress >> 32u)}, __func__);
 }
 
@@ -18,16 +18,12 @@ std::uint32_t APS5_VABI sceAgcDcbSetIndexBufferGetSize() {
     return 0;
 }
 
-uint32_t* APS5_VABI sceAgcDcbSetIndexCount(CommandBuffer* buf, uint32_t index_count) {
- (void)buf;
- (void)index_count;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+std::uint32_t* APS5_VABI sceAgcDcbSetIndexCount(CommandBuffer* buf, std::uint32_t indexCount) {
+    return Agc::Command::Emit(buf, 0x13u, {indexCount}, __func__);
 }
 
 std::uint32_t APS5_VABI sceAgcDcbSetIndexCountGetSize() {
-    NotImplemented_nid_no_patch(__func__);
-    return 0;
+    return 8;
 }
 
 std::uint32_t* APS5_VABI sceAgcDcbSetIndexSize(CommandBuffer* buf, std::uint8_t indexSize, std::uint8_t cachePolicy) {

core/libs/prx/libSceAgc/DcbDraw/src/Instancing.cpp

@@ -17,11 +17,17 @@ std::uint32_t APS5_VABI sceAgcDcbSetNumInstancesGetSize() {
 }
 
 uint32_t* APS5_VABI sceAgcDcbSetBaseIndirectArgs(CommandBuffer* buf, uint32_t shader_type, const volatile void* indirect_base_addr) {
- (void)buf;
- (void)shader_type;
- (void)indirect_base_addr;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    // SET_BASE with base index 1; header bit 1 selects the dispatch (compute) base over the draw base.
+    Agc::Command::CheckBits(shader_type, 1, __func__);
+    const auto address = reinterpret_cast<std::uintptr_t>(indirect_base_addr);
+    Agc::Command::CheckAddress(address, 8, __func__);
+    Agc::Command::CheckBits(address, 0xffffffffffffull, __func__);
+    auto* packet = Agc::Command::Allocate(buf, 4, __func__);
+    packet[0] = Agc::Command::Header(0x11u, 4, shader_type != 0 ? 2u : 0u);
+    packet[1] = 1;
+    packet[2] = static_cast<std::uint32_t>(address);
+    packet[3] = static_cast<std::uint32_t>(address >> 32u);
+    return packet;
 }
 
 }

core/libs/prx/libSceAgc/DcbDraw/src/Stats.cpp

@@ -15,9 +15,10 @@ std::uint32_t* APS5_VABI sceAgcDcbGetLodStats(CommandBuffer* buf, std::uint8_t c
     Agc::Command::CheckBits(reportAndReset, 1, __func__);
     Agc::Command::CheckBits(reportingIntervalIn100kClocks, 0xffu, __func__);
     const auto address = reinterpret_cast<std::uintptr_t>(buffer);
-    Agc::Command::CheckAddress(address, 64, __func__);
+    if (address != 0) Agc::Command::CheckAddress(address, 64, __func__);
     const auto control = (static_cast<std::uint32_t>(cachePolicy) << 28u) | (static_cast<std::uint32_t>(reportAndReset) << 19u) | (static_cast<std::uint32_t>(forceReset) << 18u) | (resetCount << 10u) | (reportingIntervalIn100kClocks << 2u);
-    return Agc::Command::Emit(buf, 0x8eu, {bufferSizeInBytes, static_cast<std::uint32_t>(address), static_cast<std::uint32_t>(address >> 32u), control}, __func__);
+    // LOD statistics are not collected; a same-sized NOP keeps the packet layout without producing data.
+    return Agc::Command::Emit(buf, 0x10u, {bufferSizeInBytes, static_cast<std::uint32_t>(address), static_cast<std::uint32_t>(address >> 32u), control}, __func__);
 }
 
 std::uint32_t APS5_VABI sceAgcDcbBeginOcclusionQueryGetSize() {

core/libs/prx/libSceAgc/DcbFlow/src/Control.cpp

@@ -51,11 +51,9 @@ uint32_t APS5_VABI sceAgcDcbRewindGetSize(void) {
 }
 
 uint32_t* APS5_VABI sceAgcDcbWaitUntilSafeForRendering(CommandBuffer* buf, uint32_t video_out_handle, uint32_t display_buffer_index) {
- (void)buf;
- (void)video_out_handle;
- (void)display_buffer_index;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    // Presentation copies the display buffer when the flip executes, so the buffer is always safe to
+    // render into; keep the arguments in a NOP for command-buffer dumps.
+    return Agc::Command::Emit(buf, 0x10u, {video_out_handle, display_buffer_index}, __func__);
 }
 
 }

core/libs/prx/libSceAgc/DcbFlow/src/Dispatch.cpp

@@ -10,11 +10,9 @@
 extern "C" {
 
 uint32_t* APS5_VABI sceAgcDcbDispatchIndirect(CommandBuffer* buf, uint32_t data_offset_in_bytes, uint32_t flags) {
- (void)buf;
- (void)data_offset_in_bytes;
- (void)flags;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    // Graphics queues read the arguments at an offset from the base set by SetBaseDispatchIndirectArgs.
+    Agc::Command::CheckBits(flags, 0xa038u | 0x41u, __func__);
+    return Agc::Command::Emit(buf, 0x16u, {data_offset_in_bytes, flags | 0x41u}, __func__);
 }
 
 std::uint32_t APS5_VABI sceAgcDcbDispatchIndirectGetSize() {

core/libs/prx/libSceAgc/DcbFlow/src/Memory.cpp

@@ -13,10 +13,13 @@ std::uint32_t* APS5_VABI sceAgcDcbAcquireMem(CommandBuffer* buf, std::uint8_t en
     Agc::Command::CheckBits(engine, 1, __func__);
     Agc::Command::CheckBits(cbDbOp, 0x7fffffffu, __func__);
     Agc::Command::CheckBits(gcrControl, 0x7ffffu, __func__);
-    const auto address = reinterpret_cast<std::uintptr_t>(base);
-    Agc::Command::Require((address & 0xffu) == 0 && (address >> 40u) == 0, __func__, "invalid acquire memory base address");
-    const auto wholeAddressSpace = sizeBytes == 0xffffffffffffffffull;
-    Agc::Command::Require(wholeAddressSpace || ((sizeBytes & 0xffu) == 0 && (sizeBytes >> 40u) == 0), __func__, "invalid acquire memory range size");
+    // The range only scopes cache maintenance, so round it outward to the 256-byte granularity the
+    // packet encodes instead of rejecting unaligned ranges.
+    const auto requested = reinterpret_cast<std::uintptr_t>(base);
+    auto address = requested & ~static_cast<std::uintptr_t>(0xffu);
+    const auto wholeAddressSpace = sizeBytes == 0xffffffffffffffffull || (address >> 40u) != 0 || sizeBytes > (1ull << 40u);
+    if (wholeAddressSpace) address = 0;
+    else sizeBytes = (sizeBytes + (requested - address) + 0xffu) & ~0xffull;
     Agc::Command::Require(pollCycles / 40u <= 0xffffu, __func__, "acquire poll interval overflow");
     return Agc::Command::Emit(buf, 0x58u, {(static_cast<std::uint32_t>(engine) << 31u) | cbDbOp, wholeAddressSpace ? 0u : static_cast<std::uint32_t>(sizeBytes >> 8u), 0, static_cast<std::uint32_t>(address >> 8u), 0, pollCycles / 40u, gcrControl}, __func__);
 }

core/libs/prx/libSceAgc/DcbState/include/Marker.hpp

@@ -1,4 +1,14 @@
 #ifndef CORE_LIBS_PRX_LIBSCEAGC_DCBSTATE_INCLUDE_MARKER_HPP
 #define CORE_LIBS_PRX_LIBSCEAGC_DCBSTATE_INCLUDE_MARKER_HPP
 
+#include <cstdint>
+#include "SceTypes.hpp"
+
+namespace Agc::Marker {
+
+std::uint32_t* Push(CommandBuffer* buf, const char* str, const char* function);
+std::uint32_t* Pop(CommandBuffer* buf, const char* function);
+
+}
+
 #endif

core/libs/prx/libSceAgc/DcbState/src/Marker.cpp

@@ -3,31 +3,61 @@
 #include "prx/libSceAgc/Command/include/Packet.hpp"
 #include <cstdint>
 #include <cstddef>
+#include <cstring>
 #include "SceTypes.hpp"
 #include "prx/libc/include/General.hpp"
 
+// Debug markers are custom NOP packets: PUSH carries the NUL-terminated text, POP has no payload.
+// Marker colors are only meaningful to capture tools and are not encoded.
+namespace {
+
+constexpr std::uint32_t OpcodeNop = 0x10;
+constexpr std::uint32_t CustomPushMarker = 0x0b;
+constexpr std::uint32_t CustomPopMarker = 0x0c;
+
+}
+
+namespace Agc::Marker {
+
+std::uint32_t* Push(CommandBuffer* buf, const char* str, const char* function) {
+    Agc::Command::Require(buf != nullptr, function, "null command buffer");
+    const char* text = str ? str : "";
+    const std::size_t bytes = std::strlen(text) + 1;
+    const auto payload = static_cast<std::uint32_t>((bytes + 3) / 4);
+    Agc::Command::Require(payload <= 0x4000u, function, "marker text too long");
+    auto* packet = Agc::Command::Allocate(buf, payload + 1, function);
+    packet[0] = Agc::Command::Header(OpcodeNop, payload + 1, CustomPushMarker << 2);
+    std::memset(packet + 1, 0, payload * sizeof(std::uint32_t));
+    std::memcpy(packet + 1, text, bytes);
+    return packet;
+}
+
+std::uint32_t* Pop(CommandBuffer* buf, const char* function) {
+    Agc::Command::Require(buf != nullptr, function, "null command buffer");
+    auto* packet = Agc::Command::Allocate(buf, 2, function);
+    packet[0] = Agc::Command::Header(OpcodeNop, 2, CustomPopMarker << 2);
+    packet[1] = 0;
+    return packet;
+}
+
+}
+
 extern "C" {
 
 uint32_t* APS5_VABI sceAgcDcbSetMarker(CommandBuffer* buf, const char* str, uint32_t color) {
- (void)buf;
- (void)str;
- (void)color;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    (void)color;
+    auto* packet = Agc::Marker::Push(buf, str, __func__);
+    Agc::Marker::Pop(buf, __func__);
+    return packet;
 }
 
 uint32_t* APS5_VABI sceAgcDcbPopMarker(CommandBuffer* buf) {
- (void)buf;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    return Agc::Marker::Pop(buf, __func__);
 }
 
 uint32_t* APS5_VABI sceAgcDcbPushMarker(CommandBuffer* buf, const char* str, uint32_t color) {
- (void)buf;
- (void)str;
- (void)color;
- NotImplemented_nid_no_patch(__func__);
- return nullptr;
+    (void)color;
+    return Agc::Marker::Push(buf, str, __func__);
 }
 
 }

core/libs/prx/libSceAgc/Patch/src/WaitRegMem.cpp

@@ -30,9 +30,17 @@ int APS5_VABI sceAgcWaitRegMemPatchReference(std::uint32_t* cmd, std::uint64_t r
 }
 
 int APS5_VABI sceAgcWaitRegMemPatchCompareFunction(std::uint32_t* cmd, std::uint8_t compareFunction) {
-    (void)cmd;
-    (void)compareFunction;
-    NotImplemented_nid_no_patch(__func__);
+    auto* wait = Agc::Command::ValidateWait(cmd, __func__);
+    Agc::Command::Require(compareFunction <= 6, __func__, "invalid wait comparison");
+    wait[1] = (wait[1] & ~7u) | compareFunction;
+    return 0;
+}
+
+int APS5_VABI sceAgcWriteDataPatchSetAddressOrOffset(std::uint32_t* cmd, std::uint64_t address) {
+    Agc::Command::CheckAddress(reinterpret_cast<std::uintptr_t>(cmd), 4, __func__);
+    Agc::Command::Require(((cmd[0] >> 8u) & 0xffu) == 0x37u, __func__, "not a WRITE_DATA packet");
+    cmd[2] = static_cast<std::uint32_t>(address);
+    cmd[3] = static_cast<std::uint32_t>(address >> 32u);
     return 0;
 }
 

core/libs/prx/libSceAgc/Unimplemented.cpp

@@ -0,0 +1,172 @@
+#include <cstdint>
+#include <cstddef>
+#include "SceTypes.hpp"
+#include "prx/libc/include/General.hpp"
+
+extern "C" {
+
+int APS5_VABI sceAgcAcbPushMarkerSpan() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcAcbSetMarkerSpan() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcAcbSetWorkloadComplete() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcAcbSetWorkloadStreamInactive() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcAcbSetWorkloadsActive() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcAcquireMemSetEngine() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcDcbPushMarkerSpan() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcDcbSetMarkerSpan() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcDcbSetWorkloadStreamInactive() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcDebugRaiseException() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcGetDataPacketPayloadRange() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcGetDefaultCxStateFlat() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcGetGsOversubscription() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcGetIsTrinityMode() {
+    return 0;
+}
+
+int APS5_VABI sceAgcGetSemaphoreLabel() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+int APS5_VABI sceAgcSetAmmSemaphoreMemory() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("-KRzWekV120", sceAgcUnknown__MKRzWekV120);
+// Unnamed DCB state toggle: games call it as (cb, enable, 0, 0) and ignore the result. It sits
+// between SetBaseIndirectArgs and SetIndexBuffer in the import table; treated as a no-op.
+std::uint32_t* APS5_VABI sceAgcUnknown__MKRzWekV120(void* buf, std::uint32_t enable, std::uint64_t a2, std::uint64_t a3) {
+    static bool logged = false;
+    if (!logged) {
+        logged = true;
+        std::fprintf(stderr, "[agc] -KRzWekV120 treated as no-op (enable=%u args=0x%llx,0x%llx)" "\n", enable, static_cast<unsigned long long>(a2), static_cast<unsigned long long>(a3));
+    }
+    (void)buf;
+    return nullptr;
+}
+
+APS5_EXPORT("6nths4DHNrs", sceAgcUnknown_6nths4DHNrs);
+int APS5_VABI sceAgcUnknown_6nths4DHNrs() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("7Wa3aeJgeVU", sceAgcUnknown_7Wa3aeJgeVU);
+int APS5_VABI sceAgcUnknown_7Wa3aeJgeVU() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("EJBA4dbmvfg", sceAgcUnknown_EJBA4dbmvfg);
+int APS5_VABI sceAgcUnknown_EJBA4dbmvfg() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("ICkECTBxrMw", sceAgcUnknown_ICkECTBxrMw);
+int APS5_VABI sceAgcUnknown_ICkECTBxrMw() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("Ikfdt-rIqCE", sceAgcUnknown_Ikfdt_MrIqCE);
+int APS5_VABI sceAgcUnknown_Ikfdt_MrIqCE() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("RTpj-tIlvZc", sceAgcUnknown_RTpj_MtIlvZc);
+int APS5_VABI sceAgcUnknown_RTpj_MtIlvZc() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("SwI6QxqwAC0", sceAgcUnknown_SwI6QxqwAC0);
+int APS5_VABI sceAgcUnknown_SwI6QxqwAC0() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("d4NZIlguzv0", sceAgcUnknown_d4NZIlguzv0);
+int APS5_VABI sceAgcUnknown_d4NZIlguzv0() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("eAy8eGNsCuU", sceAgcUnknown_eAy8eGNsCuU);
+int APS5_VABI sceAgcUnknown_eAy8eGNsCuU() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("rP5xLdOf26k", sceAgcUnknown_rP5xLdOf26k);
+int APS5_VABI sceAgcUnknown_rP5xLdOf26k() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("tmy-+rBpspY", sceAgcUnknown_tmy_M_PrBpspY);
+int APS5_VABI sceAgcUnknown_tmy_M_PrBpspY() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+APS5_EXPORT("y5K5tPktiL8", sceAgcUnknown_y5K5tPktiL8);
+int APS5_VABI sceAgcUnknown_y5K5tPktiL8() {
+ NotImplemented_nid_no_patch(__func__);
+ return 0;
+}
+
+}

core/libs/prx/libSceAgc/tests/Commands.cpp

@@ -7,6 +7,7 @@
 #include <array>
 #include <cstdio>
 #include <stdexcept>
+#include <cstring>
 #include <string>
 
 extern "C" std::uint32_t* APS5_VABI sceAgcDcbResetQueue(CommandBuffer* buf, std::uint32_t op, std::uint32_t state);
@@ -14,10 +15,19 @@ extern "C" std::uint32_t* APS5_VABI sceAgcDcbSetFlip(CommandBuffer* buf, std::ui
 extern "C" int APS5_VABI sceAgcSuspendPoint();
 extern "C" int APS5_VABI sceAgcInit(std::uint32_t* state, std::uint32_t version);
 extern "C" std::uint32_t* APS5_VABI sceAgcDcbDrawIndexAuto(CommandBuffer* buf, std::uint32_t indexCount, std::uint64_t modifier);
+extern "C" std::uint32_t* APS5_VABI sceAgcDcbDrawIndexIndirect(CommandBuffer* buf, std::uint32_t dataOffsetInBytes, std::uint64_t modifier);
+extern "C" std::uint32_t* APS5_VABI sceAgcDcbDrawIndexIndirectMulti(CommandBuffer* buf, std::uint32_t dataOffsetInBytes, std::uint32_t countIndirect, std::uint32_t maxCountOrCount, const volatile void* countAddress, std::uint32_t strideInBytes, std::uint64_t modifier);
 extern "C" int APS5_VABI sceAgcWaitRegMemPatchReference(std::uint32_t* cmd, std::uint64_t reference);
 extern "C" int APS5_VABI sceAgcGetDataPacketPayloadAddressUnk(std::uint32_t** addr, std::uint32_t* cmd, int type);
 extern "C" std::uint32_t* APS5_VABI sceAgcCbSetShRegisterRangeDirect(CommandBuffer* buf, std::uint32_t offset, const std::uint32_t* values, std::uint32_t numValues);
 extern "C" std::uint32_t* APS5_VABI sceAgcDcbContextStateAnotherOp(CommandBuffer* buf, std::uint32_t operation);
+extern "C" std::uint32_t* APS5_VABI sceAgcDcbPushMarker(CommandBuffer* buf, const char* str, std::uint32_t color);
+extern "C" std::uint32_t* APS5_VABI sceAgcDcbPopMarker(CommandBuffer* buf);
+extern "C" std::uint32_t* APS5_VABI sceAgcDcbSetMarker(CommandBuffer* buf, const char* str, std::uint32_t color);
+extern "C" std::uint32_t* APS5_VABI sceAgcAcbPushMarker(CommandBuffer* buf, const char* str, std::uint32_t color);
+extern "C" std::uint32_t* APS5_VABI sceAgcAcbPopMarker(CommandBuffer* buf);
+extern "C" std::uint32_t* APS5_VABI sceAgcAcbSetMarker(CommandBuffer* buf, const char* str, std::uint32_t color);
+extern "C" std::uint32_t* APS5_VABI sceAgcDcbSetIndexBuffer(CommandBuffer* buf, std::uint64_t indexAddress);
 
 namespace {
 
@@ -96,6 +106,64 @@ bool APS5_VABI growContext(CommandBuffer* buffer, std::uint32_t count, void* use
     return true;
 }
 
+void testIndexedIndirectDraws() {
+    Storage storage;
+    const auto* single = sceAgcDcbDrawIndexIndirect(&storage.buffer, 0x40, 0);
+    const std::array<std::uint32_t, 5> expectedSingle{0xc0032500u, 0x40, 0x280, 0x280, 0};
+    check(single == storage.words.data() && std::equal(expectedSingle.begin(), expectedSingle.end(), single), "indexed indirect draw packet mismatch");
+    alignas(4) std::uint32_t count = 0;
+    const auto* multi = sceAgcDcbDrawIndexIndirectMulti(&storage.buffer, 0x80, 1, 8, &count, 20, 0);
+    const auto address = reinterpret_cast<std::uintptr_t>(&count);
+    const std::array<std::uint32_t, 10> expectedMulti{0xc0083800u, 0x80, 0x280, 0x280, 0x40000280u, 8, static_cast<std::uint32_t>(address), static_cast<std::uint32_t>(address >> 32u), 20, 0};
+    check(multi == storage.words.data() + expectedSingle.size() && std::equal(expectedMulti.begin(), expectedMulti.end(), multi), "indexed indirect multi draw packet mismatch");
+    check(storage.buffer.cursor_up == storage.words.data() + expectedSingle.size() + expectedMulti.size(), "incorrect indexed indirect cursor advance");
+    const auto before = storage.words;
+    expectFailure([&] { sceAgcDcbDrawIndexIndirect(&storage.buffer, 2, 0); });
+    expectFailure([&] { sceAgcDcbDrawIndexIndirectMulti(&storage.buffer, 0, 1, 8, &count, 16, 0); });
+    expectFailure([&] { sceAgcDcbDrawIndexIndirectMulti(&storage.buffer, 0, 0, 8, &count, 20, 0); });
+    check(storage.words == before, "invalid indexed indirect draw modified packet memory");
+}
+
+void testMarkers() {
+    Storage dcb;
+    Storage acb;
+    const auto* dcbPush = sceAgcDcbPushMarker(&dcb.buffer, "frame", 0xff0000u);
+    const auto* acbPush = sceAgcAcbPushMarker(&acb.buffer, "frame", 0x00ff00u);
+    check(acbPush == acb.words.data() && acbPush[0] == Agc::Command::Header(0x10, 3, 0x0bu << 2u), "ACB push marker header mismatch");
+    check(std::strcmp(reinterpret_cast<const char*>(acbPush + 1), "frame") == 0, "ACB push marker text mismatch");
+    const auto* acbPop = sceAgcAcbPopMarker(&acb.buffer);
+    check(acbPop == acb.words.data() + 3 && acbPop[0] == Agc::Command::Header(0x10, 2, 0x0cu << 2u) && acbPop[1] == 0, "ACB pop marker mismatch");
+    sceAgcDcbPopMarker(&dcb.buffer);
+    const auto* acbSet = sceAgcAcbSetMarker(&acb.buffer, nullptr, 0);
+    const auto* dcbSet = sceAgcDcbSetMarker(&dcb.buffer, nullptr, 0);
+    check(acbSet == acb.words.data() + 5 && dcbSet == dcb.words.data() + 5, "set marker did not return its push packet");
+    check(acbSet[0] == Agc::Command::Header(0x10, 2, 0x0bu << 2u) && acbSet[1] == 0 && acbSet[2] == Agc::Command::Header(0x10, 2, 0x0cu << 2u), "ACB set marker is not a push and pop pair");
+    check(dcbPush == dcb.words.data() && dcb.words == acb.words, "ACB and DCB markers differ");
+    check(acb.buffer.cursor_up == acb.words.data() + 9, "incorrect ACB marker cursor advance");
+    expectFailure([] { sceAgcAcbPushMarker(nullptr, "frame", 0); });
+    expectFailure([] { sceAgcAcbPopMarker(nullptr); });
+    expectFailure([] { sceAgcAcbSetMarker(nullptr, "frame", 0); });
+}
+
+void testIndexBuffer() {
+    Storage storage;
+    alignas(4) std::uint16_t indices[2]{};
+    const auto address = reinterpret_cast<std::uintptr_t>(indices);
+    const auto* bound = sceAgcDcbSetIndexBuffer(&storage.buffer, address);
+    check(bound[1] == static_cast<std::uint32_t>(address) && bound[2] == static_cast<std::uint32_t>(address >> 32u), "index buffer address mismatch");
+    const auto* unbound = sceAgcDcbSetIndexBuffer(&storage.buffer, 0);
+    check(unbound == bound + 3 && unbound[0] == bound[0] && unbound[1] == 0 && unbound[2] == 0, "index buffer was not unbound");
+    const auto before = storage.words;
+    try {
+        sceAgcDcbSetIndexBuffer(&storage.buffer, 0x1001);
+    } catch (const std::runtime_error& error) {
+        check(std::string(error.what()).find("0x1001") != std::string::npos, "misaligned index buffer error omits the address");
+        check(storage.words == before, "misaligned index buffer modified packet memory");
+        return;
+    }
+    throw std::runtime_error("misaligned index buffer was accepted");
+}
+
 void testContextState() {
     const std::array<std::array<std::uint32_t, 6>, 4> sizes{{{5}, {5, 8, 9, 3, 2}, {3, 5, 8, 9, 2}, {5, 8, 9, 3, 2, 5}}};
     const std::array<std::array<std::uint32_t, 4>, 4> reservations{{{5}, {22, 3, 2}, {3, 22, 2}, {22, 3, 2, 5}}};
@@ -272,6 +340,9 @@ void testDefaults() {
 int main() {
     try {
         testPackets();
+        testIndexedIndirectDraws();
+        testMarkers();
+        testIndexBuffer();
         testContextState();
         testFlip();
         testRegisters();

core/libs/prx/libSceAgcDriver/CMakeLists.txt

@@ -47,14 +47,15 @@ add_custom_command(
 )
 
 add_library(libSceAgcDriver SHARED EXCLUDE_FROM_ALL
+        ${CMAKE_CURRENT_SOURCE_DIR}/Unimplemented.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/DisplayBuffer.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/Driver.cpp
+        ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/WorkerSampler.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/GuestMemory.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/ShaderMemory.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/BdaFeatures.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/Pm4.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/VulkanDevice.cpp
-        ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/SwapchainState.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/AspectFit.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/PresentationScaler.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Draw.cpp
@@ -62,6 +63,7 @@ add_library(libSceAgcDriver SHARED EXCLUDE_FROM_ALL
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/ColorTargetTransfer.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Pipeline.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Resources.cpp
+        ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Recorder.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/ShaderInputState.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/ShaderResources.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/ShaderAddressResources.cpp
@@ -73,9 +75,11 @@ add_library(libSceAgcDriver SHARED EXCLUDE_FROM_ALL
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/GuestTextureResource.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/GuestSamplerResource.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/TextureFormat.cpp
+        ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/DccMetadata.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/TextureTiling.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/TextureDetiler.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Texture.cpp
+        ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/UnitShadow.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Graphics/src/Sampler.cpp
         ${agcTextureDetileShaderHeader}
         ${CMAKE_CURRENT_SOURCE_DIR}/Eq/src/Event.cpp
@@ -169,6 +173,25 @@ target_link_libraries(agc_driver_flip_tests PRIVATE
 set_target_properties(agc_driver_flip_tests PROPERTIES CXX_EXTENSIONS OFF)
 configure_windows_unwind(agc_driver_flip_tests)
 
+add_executable(agc_driver_recorder_tests EXCLUDE_FROM_ALL
+        tests/Recorder.cpp
+)
+
+target_include_directories(agc_driver_recorder_tests PRIVATE
+        ${LIBS_INCLUDE_DIR}
+        ${CMAKE_SOURCE_DIR}/core/shader/recompiler
+        ${CMAKE_SOURCE_DIR}/3rdparty/Vulkan-Headers/include
+)
+
+target_link_libraries(agc_driver_recorder_tests PRIVATE
+        libSceAgcDriver
+        SDL2-static
+        libc
+)
+
+set_target_properties(agc_driver_recorder_tests PROPERTIES CXX_EXTENSIONS OFF)
+configure_windows_unwind(agc_driver_recorder_tests)
+
 add_executable(agc_driver_pm4_tests EXCLUDE_FROM_ALL
         tests/Pm4.cpp
 )
@@ -195,6 +218,8 @@ add_executable(agc_driver_graphics_tests EXCLUDE_FROM_ALL
         tests/GuestTextureResource.cpp
         tests/GuestSamplerResource.cpp
         tests/TextureDetiler.cpp
+        Graphics/src/Draw.cpp
+        Graphics/src/Pipeline.cpp
         Graphics/src/State.cpp
         Graphics/src/ColorTargetLayout.cpp
         Graphics/src/ColorTargetTransfer.cpp
@@ -208,9 +233,11 @@ add_executable(agc_driver_graphics_tests EXCLUDE_FROM_ALL
         Graphics/src/GuestTextureResource.cpp
         Graphics/src/GuestSamplerResource.cpp
         Graphics/src/TextureFormat.cpp
+        Graphics/src/DccMetadata.cpp
         Graphics/src/TextureTiling.cpp
         Graphics/src/TextureDetiler.cpp
         Graphics/src/Texture.cpp
+        Graphics/src/UnitShadow.cpp
         Graphics/src/Sampler.cpp
         ${agcTextureDetileShaderHeader}
         Execution/src/GuestMemory.cpp
@@ -228,6 +255,7 @@ target_include_directories(agc_driver_graphics_tests PRIVATE
 )
 
 target_link_libraries(agc_driver_graphics_tests PRIVATE
+        shader_recompiler_agc_driver
         libc
 )
 
@@ -253,6 +281,18 @@ target_link_libraries(agc_driver_bda_device_tests PRIVATE SDL2-static libc shade
 set_target_properties(agc_driver_bda_device_tests PROPERTIES CXX_EXTENSIONS OFF)
 configure_windows_unwind(agc_driver_bda_device_tests)
 
+if(BUILD_TESTING)
+    foreach(agcTest IN ITEMS agc_driver_tests agc_shader_memory_tests agc_driver_flip_tests agc_driver_pm4_tests)
+        string(REGEX REPLACE "_tests$" "" agcTestName "${agcTest}")
+        set_target_properties(${agcTest} PROPERTIES EXCLUDE_FROM_ALL OFF)
+        add_test(NAME ${agcTestName} COMMAND ${agcTest})
+        if(WIN32)
+            set_tests_properties(${agcTestName} PROPERTIES ENVIRONMENT_MODIFICATION
+                "PATH=path_list_prepend:$<TARGET_FILE_DIR:libc>")
+        endif()
+    endforeach()
+endif()
+
 option(AGC_BUILD_VISUAL_TEST "Enable the standalone AGC SPIR-V visual test" OFF)
 
 if(AGC_BUILD_VISUAL_TEST)
@@ -282,7 +322,6 @@ if(AGC_BUILD_VISUAL_TEST)
     add_executable(agc_driver_visual_test EXCLUDE_FROM_ALL
             tests/Visual.cpp
             Execution/src/VulkanDevice.cpp
-            Execution/src/SwapchainState.cpp
             Execution/src/AspectFit.cpp
             Execution/src/PresentationScaler.cpp
             Execution/src/GuestMemory.cpp
@@ -301,9 +340,11 @@ if(AGC_BUILD_VISUAL_TEST)
             Graphics/src/GuestTextureResource.cpp
             Graphics/src/GuestSamplerResource.cpp
             Graphics/src/TextureFormat.cpp
+            Graphics/src/DccMetadata.cpp
             Graphics/src/TextureTiling.cpp
             Graphics/src/TextureDetiler.cpp
             Graphics/src/Texture.cpp
+            Graphics/src/UnitShadow.cpp
             Graphics/src/Sampler.cpp
             ${agcTextureDetileShaderHeader}
             ${visualShaders}
@@ -363,3 +404,14 @@ if(AGC_BUILD_VISUAL_TEST)
 endif()
 
 include(PostCompile.cmake)
+# Replays shader_*.req files the driver writes when a compute dispatch cannot be recompiled.
+add_executable(agc_shader_replay EXCLUDE_FROM_ALL
+        tools/ShaderReplay.cpp
+)
+target_include_directories(agc_shader_replay PRIVATE ${LIBS_INCLUDE_DIR})
+target_link_libraries(agc_shader_replay PRIVATE shader_recompiler_agc_driver libc)
+if(ANYPS5_ENABLE_SPIRV_TOOLS)
+    target_link_libraries(agc_shader_replay PRIVATE SPIRV-Tools-static)
+endif()
+set_target_properties(agc_shader_replay PROPERTIES CXX_EXTENSIONS OFF)
+configure_windows_unwind(agc_shader_replay)

core/libs/prx/libSceAgcDriver/Execution/include/DisplayBuffer.hpp

@@ -13,6 +13,8 @@ struct DisplayBuffer {
     std::uint64_t pixelFormat;
     std::uint32_t width;
     std::uint32_t height;
+    std::uint32_t tilingMode = 0;
+    std::uint32_t pitchInPixel = 0;
 };
 
 std::size_t DisplayBufferSize(const DisplayBuffer& buffer);

core/libs/prx/libSceAgcDriver/Execution/include/GuestMemory.hpp

@@ -1,16 +1,224 @@
 #ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_GUESTMEMORY_HPP
 #define CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_GUESTMEMORY_HPP
 
+#include <atomic>
 #include <cstddef>
 #include <cstdint>
+#include <mutex>
 #include <span>
+#include <utility>
+#include <vector>
 
 namespace AgcDriver::GuestMemory {
 
 void CheckRange(const void* pointer, std::size_t bytes, std::size_t alignment, bool writable = false);
-void CheckGpuRange(const void* pointer, std::size_t bytes, std::size_t alignment, bool writable = false);
+// CheckRange without throwing: whether the whole range is mapped with the requested access.
+bool Accessible(const void* pointer, std::size_t bytes, bool writable = false);
+// The accessible parts [begin, end) of a range, in address order. GPU heaps are often bound whole while
+// the guest commits their pages on demand, so a binding can cover reserved but uncommitted pages.
+std::vector<std::pair<std::uint64_t, std::uint64_t>> CommittedRanges(std::uint64_t address, std::size_t bytes, bool writable = false);
+// Both facts from one walk of the page tables: the accessible parts of a range and whether they cover
+// all of it (`whole`, what Accessible answers). Reserved pages are queried each time, so asking twice
+// costs two VirtualQuery per run.
+struct Commitment {
+    std::vector<std::pair<std::uint64_t, std::uint64_t>> ranges;
+    bool whole = false;
+};
+Commitment DescribeCommitted(std::uint64_t address, std::size_t bytes, bool writable = false);
 void Read(std::uint64_t address, std::span<std::byte> destination, std::size_t alignment = 1);
+// Variants for resources in heaps the guest commits on demand (partially resident textures, GPU
+// scratch heaps): uncommitted pages read as zeros, are not compared, and are never stored.
+void ReadCommitted(std::uint64_t address, std::span<std::byte> destination);
+bool EqualsCommitted(std::uint64_t address, std::span<const std::byte> bytes);
+// EqualsCommitted without the flush hook: the caller has decided that the recorded GPU work
+// writing the range need not be waited for (the dispatch-cache validation in Driver.cpp).
+bool EqualsCommittedUnsynced(std::uint64_t address, std::span<const std::byte> bytes);
+// EqualsCommittedUnsynced for a caller that treats an unmapped page as a miss (the dispatch-cache
+// validation's per-run compare): one walk of the page states, one memcmp per mapped run, no profile
+// counter and no reason string. Unmapped: some page of the range is not readable (or the address
+// space cannot be queried).
+enum class Compare : std::uint8_t { Equal, Differs, Unmapped };
+Compare CompareMapped(std::uint64_t address, std::span<const std::byte> bytes);
+// CompareMapped's walk with a copy in place of the compare: the bytes at `address` are copied into
+// `out` (Equal), or Unmapped as CompareMapped answers it (the dispatch-cache validation's masked
+// re-read of a differing run).
+Compare CopyMapped(std::uint64_t address, std::span<std::byte> out);
+void WriteChangedCommitted(std::uint64_t address, std::span<const std::byte> current, std::span<const std::byte> original);
 void Write(std::uint64_t address, std::span<const std::byte> source, std::size_t alignment = 1);
+// Stores the parts of `current` that differ from `original` (the guest bytes the GPU started from), so
+// guest writes made meanwhile to untouched bytes survive. Compares in 256-byte blocks.
+void WriteChanged(std::uint64_t address, std::span<const std::byte> current, std::span<const std::byte> original);
+
+// Write tracking. The guest arena is reserved with page write watching, so CollectWrites learns which
+// pages of a range the CPU wrote since they were last collected, stamps their 64 KiB blocks with a new
+// generation and returns it; a snapshot of the range taken after that call is current while
+// UnchangedSince(range, that generation) holds. Every validation collects its own range first, so
+// resources sharing pages see each other's writes. GPU writes into host-imported memory bypass the
+// page tables and are reported with MarkWritten. Without write watching CollectWrites returns 0 and
+// UnchangedSince is always false, so callers fall back to comparing bytes.
+std::uint64_t CollectWrites(std::uint64_t address, std::size_t bytes);
+bool UnchangedSince(std::uint64_t address, std::size_t bytes, std::uint64_t generation);
+// UnchangedSince for several ranges under one tracker lock: true only when every one holds.
+struct UnchangedQuery {
+    std::uint64_t address;
+    std::size_t bytes;
+    std::uint64_t generation;
+};
+bool UnchangedSinceAll(std::span<const UnchangedQuery> queries);
+// Returns the generation the blocks were stamped with (0 when the arena is not write-watched):
+// UnchangedSince(range, it) holds until the next store over the range.
+std::uint64_t MarkWritten(std::uint64_t address, std::size_t bytes);
+// Collect epoch: within one epoch a range already collected is not walked again, CollectWrites
+// returns the current generation instead. A guest write landing between two collects of the same
+// epoch is seen by the next epoch, which is the ordering real hardware gives a CPU write made while a
+// submission runs. The driver's own writes (Write, WriteChanged, MarkWritten) stamp their blocks, so
+// they are never hidden. APS5_NO_COLLECT_MEMO=1 disables the memo. CollectWritesUncached always walks:
+// for decisions about which blocks the CPU wrote while a GPU result was pending (a stale answer there
+// stores old texels over reused memory).
+// The epoch is per queue worker: BumpCollectEpoch advances the calling thread's epoch to a value no
+// other thread ever holds (one worker's walk never satisfies another's check) at the hardware
+// ordering points of its queue: the start of a submission, a WAIT_REG_MEM the CPU may have satisfied
+// (at entry, polled, timed out, or by another queue's recorded label; a label this queue recorded
+// itself orders no CPU write), a device drain, and before reaping completions outside a submission.
+// A thread that never bumps (the presenter, game threads inside the flush hook) gets a fresh epoch
+// for every collect: nothing is reused without an ordering point. APS5_PACKET_EPOCH=1 makes the
+// workers bump before every packet as before. CollectEpochBumps counts the bumps ([guestmem] line).
+void BumpCollectEpoch();
+std::uint64_t CollectEpochBumps();
+std::uint64_t CollectWritesUncached(std::uint64_t address, std::size_t bytes);
+// The tracker's current generation (every collect and MarkWritten bumps it): a stamp taken after
+// a set of driver stores, for UnchangedSinceCollected to compare against later. Read under the
+// tracker mutex, so every collect that bumped before it has finished its walk and every later
+// collect stamps newer.
+std::uint64_t TrackerGeneration();
+// Whether no CPU write touched the range since `generation`: an uncached resetting collect of the
+// range's pages first (a game store not collected yet becomes a block stamp), then the compare over
+// the stamps collects make (dirty pages), not the MarkWritten stamps of the driver's own GPU label
+// records, which share the title's label blocks. False when the arena is not write-watched.
+bool UnchangedSinceCollected(std::uint64_t address, std::size_t bytes, std::uint64_t generation);
+// Per 64 KiB tracker block of [address, address + bytes) (block 0 holds `address`): `changed[k]`
+// receives whether the block was stamped after generations[k] (UnchangedSince over that block
+// alone), `cpu[k]`, when given, whether a collect stamped it after (a CPU store, as
+// UnchangedSinceCollected tells them apart from the driver's MarkWritten stamps, but without its
+// walk: the caller collected the range first). One tracker lock for the whole range. Outside the
+// watched arena, or for a generation of 0, a block reads as changed.
+void ChangedBlocks(std::uint64_t address, std::size_t bytes, std::span<const std::uint64_t> generations, std::span<std::uint8_t> changed, std::span<std::uint8_t> cpu = {});
+
+// Serializes device work: draws, dispatches, presentation and the deferred write-backs below. The
+// mutex is recursive; it is wrapped so every acquisition (std::lock_guard at any site) measures how
+// long it waited: the graphics worker's frame is mostly spent waiting for this lock behind the
+// compute workers, and that wait has to stay visible. Under APS5_PROFILE_DRAW each thread reports its
+// waits in a [lock] line every 10 s (APS5_NO_LOCK_PROFILE=1 turns the timing off); a queue worker tags
+// its thread with TagGpuLockThread so the line names the queue.
+class GpuMutexType {
+public:
+    void lock();
+    bool try_lock();
+    void unlock();
+    // Whether the calling thread holds the mutex (at any depth): the APS5_ASSERT_GPU_LOCK checks
+    // (AssertGpuLockHeld) and code that must know whether it runs inside the device lock.
+    bool HeldByThisThread() const;
+    // The calling thread's recursion depth (0 when it does not hold the mutex): 1 means its
+    // current acquisition is the outermost, so code that releases the mutex around a wait
+    // (Recorder::SyncThrough for the flush hook) gives up no caller's hold.
+    std::uint32_t DepthOnThisThread() const;
+
+private:
+    void acquired();
+    std::recursive_mutex mutex;
+    // The holder's thread token and recursion depth, written by the holder only; another thread
+    // reads the owner just to see that it is not itself.
+    std::atomic<const void*> owner{nullptr};
+    std::uint32_t depth = 0;
+};
+GpuMutexType& GpuMutex();
+// Called on the thread that just gave up its outermost hold, right after the mutex was released:
+// work the hold deferred to run without it (the recorder's release of the objects completed batches
+// kept). One hook, set once; it must not take GpuMutex itself.
+void SetGpuUnlockHook(void (*hook)());
+// Debug aid (APS5_ASSERT_GPU_LOCK=1): aborts with a [lock] line naming `where` when the calling
+// thread does not hold GpuMutex. Recording, submits, command batches and the detiler's pools must
+// only be reached under it (a resource build's stage A runs without it, see ShaderResources).
+void AssertGpuLockHeld(const char* where);
+void TagGpuLockThread(std::uint32_t queue);
+// The presenter stays untagged (its tag is a queue id to the recorder) but takes its own column
+// of the [lock] holder matrix, apart from the game threads.
+void MarkPresenterThread();
+// The queue tag of the calling thread (0xffffffff when untagged), for reports that name the thread.
+std::uint32_t GpuLockThreadTag();
+// Lock sites (APS5_PROFILE_DRAW): a caller names the site of its next lock() so the [lock] line
+// splits acquisitions and waits per site (whose holds queue 0 waits behind is then measurable). The
+// tag is consumed by that acquisition, so a nested lock inside the flush hook is attributed to the
+// hook, and an acquisition nobody tagged counts as "other". Try-only acquisitions are not counted
+// as acquisitions; the hold they start is timed under "try". Holds are timed per site as well (the
+// outermost acquisition of a thread until its unlock; APS5_NO_HOLD_PROFILE=1 turns that off), so
+// the line shows who holds the mutex, not only who waits for it.
+enum class GpuLockSite : std::uint8_t { Other = 0, Dispatch, Indirect, Draw, Hook, Wait, Label, Flush, Present, Fill, Copy, End, Try, Count };
+void TagGpuLockSite(GpuLockSite site);
+// A GPU wait (a fence or timeline wait that found its work unfinished) made while the calling
+// thread holds GpuMutex: counted on the thread's [lock] line as 'locked GPU waits' (APS5_PROFILE_DRAW).
+void NoteLockedGpuWait(double ms);
+// Bytes the calling thread's write-watch collects walked so far (0 without the memory profile):
+// a caller reads the difference across a span of its own work.
+std::uint64_t ThreadCollectedBytes();
+// A code address (a return address) as an offset into its module, in the form of the [guestmem]
+// callers (symbolized with nm against the driver's image). Takes the loader lock: for reports only.
+unsigned long long CodeOffset(const void* address);
+
+// Deferred GPU writes. The graphics layer keeps the results of storage images on the GPU until guest
+// memory is read; it registers a hook that stores every pending result overlapping a range, which
+// every read and store through this interface calls first. Code that reads guest memory directly
+// calls FlushGpuWrites itself.
+void SetFlushHook(void (*hook)(std::uint64_t address, std::size_t bytes));
+void FlushGpuWrites(std::uint64_t address, std::size_t bytes);
+// Moves with every ForgetPages call (the libc invalidator: memory unmapped or re-registered): odd
+// while the call stores its page states, moved past it after, so a reader that loads an even
+// value before consulting the page states and reading directly, and the same value after, knows
+// the states it consulted were current and no page it read changed its mapping meanwhile.
+std::uint64_t ForgetSerial();
+
+// Attribution of guest memory accesses (the [hooksync] line in Recorder.cpp and the read-site
+// counts of the [guestmem] line, APS5_PROFILE_DRAW): a sync the flush hook makes for an access has
+// to be charged to the PM4 packet and the driver code that made the access. A queue worker names
+// the packet it executes at the top of its packet loop (SetCurrentPacket; opcode 0xffff is the
+// flip); a thread that runs no packets (the presenter, game threads) reads back NoPacket. The main
+// read paths name themselves with a ReadSiteScope; an access made by none of them is attributed by
+// its return addresses instead (CaptureCallerOffsets).
+struct PacketTag {
+    std::uint32_t opcode;
+    std::uint32_t queue;
+};
+constexpr std::uint32_t NoPacket = 0xfffffffeu;
+void SetCurrentPacket(std::uint32_t opcode, std::uint32_t queue);
+PacketTag CurrentPacket();
+
+// Store: a CPU write (Write/WriteChanged) rather than a read; a hook sync made for one orders a
+// write after a pending GPU write, so whether the bytes change says nothing about its necessity.
+// MirrorRefresh: an address-based build bringing an image mirror up to date with guest memory
+// (GuestBufferMemory.cpp: the flush before a writable mirror's compare, a new mirror's fill).
+// DrawCache: a draw entry's per-stage value validation (Driver.cpp drawCache), where a draw
+// capture's hook waits reappear on a hit.
+// Every enumerator needs its name in ReadSiteName's table (GuestMemory.cpp).
+enum class ReadSite : std::uint8_t { Unknown = 0, Capture, DispatchCache, TextureCompare, TextureRead, BufferUpload, IndexBuffer, VertexBuffer, Registers, IndirectArguments, Wait, Label, Scanout, Store, MirrorRefresh, DrawCache, Count };
+const char* ReadSiteName(ReadSite site);
+// Sets the calling thread's read site and returns the previous one (nested scopes restore it).
+ReadSite SetReadSite(ReadSite site);
+ReadSite CurrentReadSite();
+class ReadSiteScope {
+public:
+    explicit ReadSiteScope(ReadSite site) : previous(SetReadSite(site)) {}
+    ~ReadSiteScope() { SetReadSite(previous); }
+    ReadSiteScope(const ReadSiteScope&) = delete;
+    ReadSiteScope& operator=(const ReadSiteScope&) = delete;
+
+private:
+    ReadSite previous;
+};
+// Return addresses above the caller as offsets into the driver's module (the same form as the
+// [guestmem] callers, symbolized with nm against the driver's image; an address outside the module
+// is written as 0), newest first, skipping `skip` frames above the caller; returns how many were
+// written. Stack unwinding, so only for rare, profiled events.
+std::size_t CaptureCallerOffsets(std::span<unsigned long long> frames, unsigned skip);
 
 }
 

core/libs/prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp

@@ -67,6 +67,45 @@ public:
         flipReached = reached;
     }
 
+    Clock::time_point FlipReached() {
+        std::lock_guard lock(mutex);
+        return flipReached;
+    }
+
+    // Recorder batches at the flip packet (after its submit) and how many had not signaled.
+    void NoteFlipBatches(std::uint64_t atFlip, std::uint64_t unsignaled) {
+        std::lock_guard lock(mutex);
+        batchesAtFlip = atFlip;
+        unsignaledAtFlip = unsignaled;
+    }
+
+    // 0 until the flip packet sampled the recorder (the drain paths never do).
+    std::uint64_t BatchesAtFlip() {
+        std::lock_guard lock(mutex);
+        return batchesAtFlip;
+    }
+
+    // At the presentation's blit: the batches ahead of it, the newest batch's vkQueueSubmit time
+    // and the blit's own, both measured from the flip packet.
+    void NoteBlit(std::uint64_t atBlit, Clock::time_point lastSubmitted, Clock::time_point blitSubmitted) {
+        std::lock_guard lock(mutex);
+        batchesAtBlit = atBlit;
+        if (flipReached == Clock::time_point{}) return;
+        blitSubmitAfterFlip = blitSubmitted - flipReached;
+        lastSubmitAfterFlip = std::max(lastSubmitted, blitSubmitted) - flipReached;
+    }
+
+    // A presentation retired during this present (with APS5_FLIP_INFLIGHT=1 the previous frame's):
+    // the batches submitted behind its blit; and, for a retired presentation whose batches all
+    // have their completion record by this present (VulkanDevice's SettleRetired: usually one
+    // present later than the retire), the GPU's busy time and idle gaps over those batches.
+    void NoteRetired(std::uint64_t afterFlip, double busyMs, double gapMs) {
+        std::lock_guard lock(mutex);
+        batchesAfterFlip += afterFlip;
+        gpuBusyMs += busyMs;
+        gpuGapMs += gapMs;
+    }
+
     void Print(std::uint32_t outputHandle, std::int32_t buffer, std::int64_t argument, Clock::time_point finished, Clock::duration interval) {
         std::lock_guard lock(mutex);
         if (firstSerial == 0 || flipSerial == 0) throw std::runtime_error("Frame timing: incomplete submission lineage");
@@ -89,6 +128,9 @@ public:
         }
         output << " worker_unattributed_ms=" << milliseconds(flipReached - executionStart - accounted);
         if (interval != Clock::duration::zero()) output << " flip_interval_ms=" << milliseconds(interval);
+        output << " batches_at_flip=" << batchesAtFlip << " unsignaled_at_flip=" << unsignaledAtFlip << " batches_at_blit=" << batchesAtBlit << " batches_after_flip=" << batchesAfterFlip;
+        output << " gpu_busy_ms=" << gpuBusyMs << " gpu_gap_ms=" << gpuGapMs;
+        output << " last_submit_after_flip_ms=" << milliseconds(lastSubmitAfterFlip) << " blit_submit_after_flip_ms=" << milliseconds(blitSubmitAfterFlip);
         output << " metrics=inclusive(count,sum_ms,max_ms[,bytes])";
         for (const auto& [key, metric] : metrics) {
             if (metric.count == 0) continue;
@@ -117,6 +159,14 @@ private:
     Clock::time_point executionStart{};
     Clock::time_point flipReceived{};
     Clock::time_point flipReached{};
+    std::uint64_t batchesAtFlip = 0;
+    std::uint64_t unsignaledAtFlip = 0;
+    std::uint64_t batchesAtBlit = 0;
+    std::uint64_t batchesAfterFlip = 0;
+    double gpuBusyMs = 0;
+    double gpuGapMs = 0;
+    Clock::duration lastSubmitAfterFlip{};
+    Clock::duration blitSubmitAfterFlip{};
 };
 
 class PerformanceContext {

core/libs/prx/libSceAgcDriver/Execution/include/Pm4.hpp

@@ -3,6 +3,9 @@
 
 #include "prx/libSceAgcDriver/Execution/include/QueueState.hpp"
 #include "prx/libSceAgcDriver/Execution/include/Pm4Opcodes.hpp"
+#include <cstddef>
+#include <vector>
+#include <optional>
 #include <array>
 #include <span>
 #include <string>
@@ -18,14 +21,114 @@ struct DrawParameters {
     bool indexed = true;
     std::uint32_t firstVertex = 0;
     std::uint32_t firstInstance = 0;
+    // An indirect draw (DRAW_INDIRECT 0x24, DRAW_INDEX_INDIRECT 0x25 and their multi forms 0x2c and
+    // 0x38): the CP reads {vertexCount, instanceCount, startVertex, startInstance} (indexed:
+    // {indexCount, instanceCount, firstIndex, vertexOffset, startInstance}) per record at
+    // `arguments` + record * stride, writes startVertex / vertexOffset, startInstance and the draw
+    // index into the SH registers the packet names (0x280: nowhere) and takes the instance count
+    // from the record. indexCount and instanceCount above are then unknown (indexed draws bind the
+    // declared INDEX_BUFFER_SIZE range), firstVertex holds GE_INDX_OFFSET. The driver decides per
+    // dimension how the GPU reproduces the CP's register write for the fixed-function fetch (see
+    // Driver.cpp): InPlace draws from the record's dword, Constant rewrites it with the constant.
+    struct IndirectDraw {
+        std::uint64_t arguments;
+        std::uint32_t opcode;
+        std::uint32_t recordBytes;
+        std::uint32_t stride;
+        std::uint32_t count;
+        bool countIndirect;
+        std::uint64_t countAddress;
+        std::uint32_t baseVertexLocation;
+        std::uint32_t startInstanceLocation;
+        std::uint32_t drawIndexLocation;
+        bool drawIndexEnabled;
+        std::uint32_t indxOffset;
+        enum class Rule : std::uint8_t { InPlace, Constant };
+        Rule vertexRule = Rule::Constant;
+        Rule instanceRule = Rule::Constant;
+        std::uint32_t vertexConstant = 0;
+        std::uint32_t instanceConstant = 0;
+        std::int32_t baseVertexSgpr = -1;
+        std::int32_t startInstanceSgpr = -1;
+        std::int32_t drawIndexSgpr = -1;
+        // Bytes of the record range: (count - 1) * stride + recordBytes, or 0 for no records.
+        std::uint64_t RangeBytes() const { return count == 0 ? 0 : static_cast<std::uint64_t>(count - 1) * stride + recordBytes; }
+        std::uint32_t VertexDwordOffset() const { return recordBytes == 20 ? 12u : 8u; }
+        std::uint32_t InstanceDwordOffset() const { return recordBytes == 20 ? 16u : 12u; }
+    };
+    std::optional<IndirectDraw> indirect;
 };
 
+// One indirect draw record as the CP reads it (firstVertexOrIndex is startVertex for the
+// non-indexed layout and firstIndex for the indexed one; vertexOffset exists only in the latter).
+struct DrawArguments {
+    std::uint32_t count;
+    std::uint32_t instances;
+    std::uint32_t firstVertexOrIndex;
+    std::uint32_t vertexOffset;
+    std::uint32_t firstInstance;
+};
+// Reads record `record` of an indirect draw, or its count dword, through the checked guest memory
+// path (which waits for recorded GPU work that writes them).
+DrawArguments ReadDrawArguments(const DrawParameters::IndirectDraw& indirect, std::uint32_t record);
+std::uint32_t ReadDrawCount(const DrawParameters::IndirectDraw& indirect);
+inline bool IndirectDrawOpcode(std::uint32_t opcode) { return opcode == 0x24 || opcode == 0x25 || opcode == 0x2c || opcode == 0x38; }
+inline bool DrawOpcode(std::uint32_t opcode) { return opcode == 0x27 || opcode == 0x2d || opcode == 0x35 || IndirectDrawOpcode(opcode); }
+
 std::string Name(std::uint32_t header);
+// A PM4 type-2 packet is a one-dword filler (command-buffer padding); type 3 and type 0 carry a
+// dword count in bits 29:16. Type 1 is undefined.
+inline bool FillerPacket(std::uint32_t header) { return (header >> 30u) == 2u; }
+inline std::size_t PacketWords(std::uint32_t header) { return FillerPacket(header) ? 1u : static_cast<std::size_t>((header >> 16u) & 0x3fffu) + 2u; }
 std::string_view UnsupportedReason(std::uint32_t header);
 void Validate(std::span<const std::uint32_t> packet, std::uint32_t queue);
 void Execute(std::span<const std::uint32_t> packet, QueueState& queue);
 bool AccessesMemory(std::uint32_t header);
+// Whether an ACQUIRE_MEM packet asks only for GPU cache actions (no CPU-visible memory
+// synchronization): such a packet needs a pipeline barrier, not a device drain.
 bool UsesGpuCacheBarrier(std::span<const std::uint32_t> packet);
+bool IsTagMarker(std::span<const std::uint32_t> packet);
+bool WaitSatisfied(std::span<const std::uint32_t> packet);
+// WaitSatisfied for a polling loop: the caller has validated the address once, so the value is read
+// directly instead of through the checked guest memory path.
+bool WaitSatisfiedUnchecked(std::span<const std::uint32_t> packet);
+// Whether a WAIT_REG_MEM's compare holds for `value` (the 4 or 8 bytes a label the recorder still
+// holds will store), without reading memory.
+bool WaitComparesValue(std::span<const std::uint32_t> packet, std::uint64_t value);
+// A label write (RELEASE_MEM with a data select, WRITE_DATA to memory): the destination and the
+// bytes it stores, so the write can be recorded on the GPU behind the work it signals. Packets
+// without a memory destination decode to nothing. No allocation per label (tens of thousands per
+// second): WRITE_DATA's bytes are a view of the packet (valid while the packet is), a RELEASE_MEM
+// value is held inline.
+struct LabelWrite {
+    std::uint64_t address;
+    std::span<const std::byte> packetBytes;
+    std::array<std::byte, 8> inlineBytes{};
+    std::size_t inlineSize = 0;
+    std::span<const std::byte> Bytes() const { return inlineSize != 0 ? std::span<const std::byte>(inlineBytes).first(inlineSize) : packetBytes; }
+};
+std::optional<LabelWrite> DecodeLabelWrite(std::span<const std::uint32_t> packet);
+// A memory store the CPU can resolve before the GPU runs it (COPY_DATA and DMA_DATA to memory,
+// DUMP_CONST_RAM): the destination and the bytes it stores, so the driver can record the store on
+// the GPU like a label instead of draining the device and storing on the CPU. Immediate and constant
+// RAM bytes are known at once; a memory source is read here through the checked guest memory path
+// (which waits only for recorded GPU work that writes the source). Nothing when the packet is not
+// such a store, when it stores more than `limit` bytes, or when its destination or size is not a
+// multiple of 4 (the GPU store needs both; the source is then not read: the caller keeps its CPU
+// path). A store of zero bytes resolves with empty bytes.
+struct StoreWrite {
+    std::uint64_t address;
+    std::span<const std::byte> viewBytes;
+    std::vector<std::byte> ownedBytes;
+    std::span<const std::byte> Bytes() const { return ownedBytes.empty() ? viewBytes : std::span<const std::byte>(ownedBytes); }
+};
+std::optional<StoreWrite> ResolveStore(std::span<const std::uint32_t> packet, const QueueState& queue, std::size_t limit);
+// A DISPATCH_INDIRECT's arguments: the guest address of its three group-count dwords (no memory
+// access, so the GPU can read them in place: VulkanDevice::DispatchIndirect), the DISPATCH_DIRECT
+// packet made by reading them there (through the checked guest memory path, which waits for
+// recorded GPU work that writes them), and both steps in one for the packet.
+std::uint64_t DispatchArgumentAddress(std::span<const std::uint32_t> packet, const QueueState& queue);
+std::array<std::uint32_t, 5> ReadDispatchArguments(std::uint64_t arguments, std::uint32_t initiator);
 std::array<std::uint32_t, 5> ResolveDispatch(std::span<const std::uint32_t> packet, const QueueState& queue);
 DrawParameters ResolveDraw(std::span<const std::uint32_t> packet, const QueueState& queue);
 

core/libs/prx/libSceAgcDriver/Execution/include/PresentationScaler.hpp

@@ -18,7 +18,18 @@ public:
     void EnsureSourceImage(std::uint32_t width, std::uint32_t height);
     void RecordUpload(VkCommandBuffer commands, VkBuffer uploadBuffer);
     void RecordImage(VkCommandBuffer commands, VkImage image);
+    // Fills the source image from another image of the same extent (any blittable format, layout
+    // unchanged), leaving the source image ready to blit or read back.
+    void RecordBlitInto(VkCommandBuffer commands, VkImage image, VkImageLayout layout, VkFilter filter);
+    // Copies the source image's texels (tightly packed rows of its format) into a host-readable buffer
+    // and makes them visible to the host once the submission completes.
+    void RecordReadback(VkCommandBuffer commands, VkBuffer destination);
     void RecordBlit(VkCommandBuffer commands, VkImage destinationImage, std::uint32_t destinationWidth, std::uint32_t destinationHeight);
+    // The aspect-fitting blit of RecordBlit from any image: presenting a resident render target needs
+    // no copy into the source image first.
+    static void RecordBlitFrom(const Graphics::Context& context, VkCommandBuffer commands, VkImage image, VkImageLayout layout, std::uint32_t width, std::uint32_t height, VkFilter filter, VkImage destinationImage, std::uint32_t destinationWidth, std::uint32_t destinationHeight);
+    std::uint32_t SourceWidth() const { return sourceWidth; }
+    std::uint32_t SourceHeight() const { return sourceHeight; }
 
 private:
     void release() noexcept;

core/libs/prx/libSceAgcDriver/Execution/include/Recipe.hpp

@@ -0,0 +1,91 @@
+#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_RECIPE_HPP
+#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_RECIPE_HPP
+
+#define VK_NO_PROTOTYPES
+#include <vulkan/vulkan.h>
+#include "prx/libSceAgcDriver/Graphics/include/GuestBufferMemory.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/Pipeline.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/Shaders.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/State.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/Texture.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/VertexInput.hpp"
+#include <array>
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+#include <optional>
+#include <set>
+#include <utility>
+#include <vector>
+
+namespace AgcDriver {
+
+// A compute dispatch's Vulkan objects (VulkanDevice.cpp), shared by every dispatch of one compiled
+// variant.
+struct ComputePipelineObjects;
+
+// What a value-equal dispatch-cache hit needs to record its dispatch without preparing, finding or
+// comparing anything again (design_cpu_final M4): the template the resource cache served the
+// entry's compiled result with, the pipeline objects of its variant, the push constants and data
+// words the compiled result fixes, and the pre-sync answer. Built at the end of a successful device
+// call of a reusable object and attached to the dispatch-cache variant; immutable after that. It
+// holds weak references only: the resource cache owns the template (its 1024-entry bound stays
+// what pins textures and storage images) and the device's pipeline map owns the objects, so a dead
+// reference is a recipe miss, never a leak past the caches' budgets.
+struct Recipe {
+    // R10: unusable on any other device.
+    VkDevice device = VK_NULL_HANDLE;
+    std::weak_ptr<Graphics::ShaderResources> templateRef;
+    // The template's content key: touches the cache's LRU on a hit.
+    Graphics::ResourceCache::Key key;
+    std::weak_ptr<ComputePipelineObjects> objects;
+    std::array<std::byte, Graphics::PipelinePushConstantBytes> pushBytes{};
+    bool pushes = false;
+    // FNV over the compiled result's ShaderData/FlattenedSrt words (ShaderResources::DataWordsHash):
+    // the template's data buffers need a refresh iff its own hash differs.
+    std::uint64_t dataWordsHash = 0;
+    // The template's PresyncSurfaces() as of the attach and the host-import identity they were
+    // computed under (a surface is CPU-read only while its memory has no import).
+    std::vector<std::pair<std::uint64_t, std::uint64_t>> presyncSurfaces;
+    Graphics::HostImportsProof presyncProof;
+    // Both false for every recipe (only reusable objects get one); kept for the asserts.
+    bool needsCompletion = false;
+    bool holdsLease = false;
+};
+
+// Recorded: the dispatch went into the open batch from the recipe; Rebuild: the proof failed or an
+// object went away, the caller runs the ordinary path (the recipe is re-attached after it).
+enum class RecipeOutcome { Recorded, Rebuild };
+
+// What a recorded draw needs to record again from its draw-cache hit without looking anything up
+// (design_cpu_final M8, step 8b): the resource-cache template the draw's stages were served with
+// (weak, as a dispatch recipe's), its pipeline and framebuffer (weak, R10: the pipeline store owns
+// the pipeline and destroys it with its device (ClearCachedPipelines), the pipeline owns its
+// framebuffer list; a recipe outlives a device replacement in the draw cache, so an owning
+// reference would destroy the objects against a dead VkDevice), the resident targets (weak: a hit
+// proves them on the stored objects while they are still Cached()) with their attachment views,
+// and the per-draw derivations of the compiled stages and the state: the vertex input layout, the
+// push constant block, the state with the unwritten outputs masked, the fragment outputs and the
+// pipeline stages. Built at the end of a recorded, cacheable, reusable, non-indirect draw and
+// attached to the draw entry's matched stage variants; immutable.
+struct DrawRecipe {
+    VkDevice device = VK_NULL_HANDLE;
+    std::weak_ptr<Graphics::ShaderResources> templateRef;
+    Graphics::ResourceCache::Key key;
+    std::weak_ptr<Graphics::Pipeline> pipeline;
+    std::weak_ptr<Graphics::Framebuffer> framebuffer;
+    std::vector<std::weak_ptr<Graphics::StorageTexture>> targets;
+    std::vector<VkImageView> targetViews;
+    std::uint64_t passKey = 0;
+    Graphics::VertexInputLayout vertexInput;
+    std::array<std::byte, Graphics::PipelinePushConstantBytes> pushBytes{};
+    VkShaderStageFlags pushStages = 0;
+    std::optional<Graphics::State> masked;
+    std::set<std::uint32_t> fragmentOutputs;
+    VkPipelineStageFlags shaderStages = 0;
+};
+
+}
+
+#endif

core/libs/prx/libSceAgcDriver/Execution/include/ShaderMemory.hpp

@@ -2,21 +2,112 @@
 #define CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_SHADERMEMORY_HPP
 
 #include "Recompiler.hpp"
+#include <array>
+#include <bitset>
+#include <cstddef>
+#include <cstdint>
 #include <map>
+#include <memory>
+#include <span>
+#include <utility>
+#include <vector>
+
+namespace ShaderRecompiler {
+struct SourceHandle;
+}
 
 namespace AgcDriver {
 
+// Records the guest words a shader's resource analysis reads, as the memory regions its recompile
+// request carries. Guest memory is fetched a 4 KiB page at a time; the regions report exactly the
+// dwords read, so cache keys built from them do not change with unrelated bytes nearby. A page that
+// `pendingWrite` reports as written by recorded GPU work is read word by word instead, so only a
+// dword the capture needs waits for that work through the flush hook, not the whole page; a dword
+// the query knows to be left unchanged by that work (Driver.cpp's written-element evidence) is read
+// raw, without the hook. Debug aid: APS5_NO_WORDWISE_CAPTURE=1 fetches every page whole.
 class ShaderMemory {
 public:
-    explicit ShaderMemory(std::span<const ShaderRecompiler::MemoryRegion> initial);
-    void Capture(const ShaderRecompiler::RecompileRequest& request);
+    // The driver's answer for a range: nothing recorded writes it; recorded work writes it and the
+    // read must go through the flush hook; recorded work writes it but leaves it unchanged, so the
+    // bytes are read raw; VerifyRaw reads raw and through the hook, counting a difference;
+    // KnownValue: recorded work writes it with bytes the driver already knows (a copy HLE's
+    // destination, Driver.cpp's known-value ring entries), copied into `known` when the caller
+    // passes a span of exactly `bytes` (empty: the answer alone); VerifyKnownValue serves them and
+    // reads through the hook too, counting a difference.
+    enum class PendingWrite : std::uint8_t { None, Sync, Raw, VerifyRaw, KnownValue, VerifyKnownValue };
+    using PendingWriteQuery = PendingWrite (*)(std::uint64_t address, std::size_t bytes, std::span<std::byte> known);
+    // Words served from known values by every capture, and the verified ones and their mismatches
+    // (APS5_VERIFY_KNOWN_VALUES=1), for Driver.cpp's [copy] line.
+    struct KnownValueCounts {
+        std::uint64_t served, verified, mismatches;
+    };
+    static KnownValueCounts KnownValues();
+    // Told, for a dword read that went through the hook, whether the bytes read before the hook's
+    // wait were still there after it (the driver's evidence for later raw reads).
+    using PendingWriteObserver = void (*)(std::uint64_t address, bool unchanged);
+    // The calling thread's count of hook syncs that waited for unfinished GPU work: a read is only
+    // observed when the count moved across it (against finished work both reads see the GPU's
+    // bytes, and "unchanged" would be no evidence). Null observes every read that went through
+    // the hook.
+    using HookWaitCounter = std::uint64_t (*)();
+    // The calling thread's GPU waits so far in milliseconds (Recorder::ThreadWaitedMs), set once
+    // by the driver: the [capture] line charges the waits made inside a capture to it.
+    using WaitedMsProvider = double (*)();
+    static void SetWaitedMsProvider(WaitedMsProvider provider);
+    explicit ShaderMemory(std::span<const ShaderRecompiler::MemoryRegion> initial, PendingWriteQuery pendingWrite = nullptr, PendingWriteObserver observe = nullptr, HookWaitCounter hookWaits = nullptr);
+    // Returns what the capture resolved (plan, snapshot, specialization) for
+    // ShaderRecompiler::Recompile(request, capture), which then skips its own materialization. One
+    // result per call: the draw path captures several stages on one ShaderMemory. With `handle`
+    // (the driver's memoized ShaderRecompiler::ResolveSource result) the capture skips the source
+    // resolution.
+    std::shared_ptr<const ShaderRecompiler::ResourceCapture> Capture(const ShaderRecompiler::RecompileRequest& request, const ShaderRecompiler::SourceHandle* handle = nullptr);
     [[nodiscard]] std::vector<ShaderRecompiler::MemoryRegion> Regions() const;
+    // The page regions read since the previous call (or construction), a word read again
+    // included, the initial regions excluded: one stage's own reads on the draw path's shared
+    // ShaderMemory (Regions() stays the union). The spans point into the pages, as Regions()'s do.
+    [[nodiscard]] std::vector<ShaderRecompiler::MemoryRegion> TakeRecentRegions();
+    // The driver's source handle memo outcomes, for the [capture] line (APS5_PROFILE_DRAW).
+    static void CountHandleMemo(bool hit);
 
 private:
+    static constexpr std::size_t PageBytes = 4096;
+    static constexpr std::size_t PageWords = PageBytes / sizeof(std::uint32_t);
+
+    struct Page {
+        std::array<std::uint32_t, PageWords> words{};
+        std::bitset<PageWords> valid;
+        std::bitset<PageWords> read;
+        std::bitset<PageWords> recent;
+        bool wordwise = false;
+    };
+
     static bool read(void* context, std::uint64_t address, std::uint32_t* value);
-    std::map<std::uint64_t, std::vector<std::byte>> regions;
+    Page& page(std::uint64_t base);
+
+    // Regions given at construction (the registered shader's code and header), referenced as given:
+    // the caller keeps them alive for as long as the capture is used.
+    std::map<std::uint64_t, std::span<const std::byte>> initial;
+    std::map<std::uint64_t, Page> pages;
+    PendingWriteQuery pendingWrite = nullptr;
+    PendingWriteObserver observe = nullptr;
+    HookWaitCounter hookWaits = nullptr;
 };
 
+// The positions, among a dispatch-cache variant's stored words, of the pure flat-SRT leaves a
+// capture read (ShaderRecompiler::SrtReadTrace), for a hit that differs only there. `runs` are the
+// variant's [begin, end) byte ranges in address order and `words` their dwords in that order;
+// `leaves` are (flat offset, address) and `otherReads` the sorted addresses of every other read.
+// A leaf is skipped when its address is among the other reads (a walk read aliases the dword,
+// counted `aliased`), is not dword-aligned or lies in no run (counted `unmapped`), or its stored
+// word differs from the flattened SRT's at that offset (counted `mismatched`). `positions`
+// (sorted, may repeat) and `slots` (the flat offset of each) are parallel.
+struct DataWordPositionCounts {
+    std::uint64_t unmapped = 0;
+    std::uint64_t mismatched = 0;
+    std::uint64_t aliased = 0;
+};
+DataWordPositionCounts DataWordPositions(std::span<const std::pair<std::uint64_t, std::uint64_t>> runs, std::span<const std::pair<std::uint32_t, std::uint64_t>> leaves, std::span<const std::uint64_t> otherReads, std::span<const std::uint32_t> words, std::span<const std::uint32_t> flattenedSrt, std::vector<std::uint32_t>& positions, std::vector<std::uint32_t>& slots);
+
 }
 
 #endif

core/libs/prx/libSceAgcDriver/Execution/include/SwapchainState.hpp

@@ -1,20 +0,0 @@
-#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_SWAPCHAINSTATE_HPP
-#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_SWAPCHAINSTATE_HPP
-
-#include <vulkan/vulkan.h>
-
-namespace AgcDriver {
-
-class SwapchainState {
-public:
-    bool NeedsRecreation() const;
-    void Recreated();
-    bool ProcessResult(VkResult result, const char* operation);
-
-private:
-    bool outdated = false;
-};
-
-}
-
-#endif

core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp

@@ -6,11 +6,26 @@
 #include "Recompiler.hpp"
 #include "prx/libSceAgcDriver/Execution/include/Presentation.hpp"
 #include "prx/libSceAgcDriver/Execution/include/DisplayBuffer.hpp"
+#include "prx/libSceAgcDriver/Execution/include/Recipe.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/Draw.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/Recorder.hpp"
+#include <functional>
 #include <memory>
+#include <optional>
 
 namespace AgcDriver {
 
+namespace Graphics {
+class StorageTexture;
+}
+
+// Stage A of a dispatch's resource build (see VulkanDevice::PrepareDispatch); opaque to callers.
+struct PreparedDispatch;
+// The unlocked pre-checks of a recipe hit (see VulkanDevice::PrepareRecipe); opaque to callers.
+struct RecipeHit;
+// The record tail shared by an ordinary dispatch and a recipe hit (VulkanDevice::recordDispatch).
+struct RecordedDispatch;
+
 class VulkanDevice {
 public:
     explicit VulkanDevice(const PresentationWindow* window = nullptr);
@@ -18,25 +33,278 @@ public:
     VulkanDevice(const VulkanDevice&) = delete;
     VulkanDevice& operator=(const VulkanDevice&) = delete;
     ShaderRecompiler::SpirvTarget Target() const;
+    // Distinguishes this device from every earlier one in the process (a memo keyed by the device
+    // cannot rely on the pointer, which a replacement may reuse).
+    std::uint64_t Serial() const { return serial; }
+    // The Vulkan device handle (a Recipe names the device it was built on).
+    VkDevice Device() const;
+    // Drains everything under the caller's GpuMutex: recorder Sync plus vkDeviceWaitIdle. For suspend,
+    // resize, device replacement, CPU fill fallback and APS5_DRAIN_ALL; the packet-loop drains use the
+    // three-step form below so the GPU wait happens without the mutex.
     void WaitIdle();
-    void WaitDraws();
-    void AcquireGpuMemory();
+    void PrepareForReplacement();
+    // Sends recorded work to the GPU without waiting for it. With `reapFirst` it first retires batches
+    // that already finished, so the in-flight list stays short (APS5_NO_OPPORTUNISTIC_REAP=1 skips
+    // that). A reap runs completion actions, and a write-back can wait for a later batch under the
+    // mutex through the flush hook, so a thread submitting on another queue's behalf (a WAIT_REG_MEM
+    // poller, a compute worker's between-packet flush) passes false: it must only submit, never wait.
+    void SubmitRecorded(bool reapFirst = true);
+    // Unlocked drain: SubmitAndEpoch (under the mutex) returns the serial covering every recorded
+    // batch (0: nothing in flight); WaitRecorded(serial) waits for it WITHOUT the mutex (the caller
+    // holds a shared_ptr to this device); ReapRecorded(serial) (under the mutex) runs the completions
+    // of the batches up to it. CanWaitUnlocked is false without timeline semaphores: use WaitIdle.
+    bool CanWaitUnlocked() const;
+    std::uint64_t SubmitAndEpoch();
+    void WaitRecorded(std::uint64_t serial);
+    void ReapRecorded(std::uint64_t serial);
+    // Completes the batches that already finished (non-blocking); under the mutex.
+    void ReapRecorded();
+    // Records a store of `bytes` at a guest address behind the recorded work, so the value appears
+    // once that work completed; the batch is submitted by the queue worker's flush rules (or at once
+    // with APS5_LABEL_SUBMIT_NOW=1). `stamp` is the record-order stamp and `queue` the recording
+    // queue for the recorder's pending-label table. Returns 0 when recorded on the GPU, 5 when kept
+    // as a completion action behind pending write-backs and 6 when kept as one because the memory is
+    // not host-imported so the GPU cannot store it (both land when their batch is reaped), else why
+    // the CPU must write it:
+    // 1 nothing recorded (the write is already ordered), 2 write-backs pending and 3 memory not
+    // imported (both only with APS5_DRAIN_COMPLETION_LABELS=1), 4 unsuitable size or alignment.
+    // `reapFirst` retires finished batches before the checks; a caller recording a group of labels
+    // under one lock passes it for the first label only.
+    int WriteLabelOnGpu(std::uint64_t address, std::span<const std::byte> bytes, std::uint64_t stamp, std::uint32_t queue, bool reapFirst = true);
+    // Pending-label table lookup and open-batch overlap test for WAIT_REG_MEM (see Recorder).
+    std::optional<Graphics::Recorder::LabelHit> PendingLabel(std::uint64_t address, std::size_t bytes, std::uint64_t afterStamp, Graphics::Recorder::LabelRefusal* refusal = nullptr) const;
+    bool OpenWriteOverlaps(std::uint64_t address, std::size_t bytes) const;
+    // Fills [address, address + bytes) of host-imported guest memory with a repeating 16-byte pattern,
+    // recorded behind the open batch; false when the range is not imported (the caller stores it).
+    // Recorded GPU stores over the range are ordered before the fill by its barrier; finished
+    // batches are retired first, and it waits only for stores a batch's completion makes on the
+    // CPU (a copied buffer's write-back, a deferred label). Debug aid: APS5_FILL_SYNC=1 waits for
+    // every recorded store over the range.
+    bool FillBuffer(std::uint64_t address, std::size_t bytes, std::span<const std::uint32_t, 4> pattern);
+    // Copies `bytes` of guest memory from `source` to `destination` (disjoint ranges) in place of the
+    // engine's memcpy kernel (Driver.cpp copyBuffer). `path` 0: copied on the CPU at once, when
+    // every test of the rule holds (each a pure query, nothing flushed or recorded before the
+    // decision): at most `cpuMax` bytes; no storage image with results pending over either range;
+    // the source settled (no recorded write, or every batch writing it signaled and no completion
+    // store or label pending in it); no recorded write and no label (recorded or queued) in the
+    // destination; and no unexecuted batch reads the destination in place (Recorder read tracking;
+    // with APS5_COPY_READ_TRACKING=0 the recorder must be idle instead). The bytes are then settled:
+    // later CPU readers never wait for a batch, and no GPU work sees them early. 1: recorded as a
+    // transfer between the host imports behind the open batch, ordered like FillBuffer (recorded GPU
+    // stores by its barrier, a completion's CPU store over either range by a wait, counted in
+    // `synced`/`syncMs`), with the source noted as an in-place read, only for at most `gpuMax`
+    // bytes (APS5_COPY_HLE_GPU_MAX; a larger copy answers 8 and the caller runs the kernel: its
+    // dispatch costs the GPU the same as the transfer); 2: a range is not host-imported (the
+    // caller runs the kernel). `reason` says which test refused the CPU path.
+    // `queue` is the calling queue (the wait experiment below never waits on queue 0).
+    // `noteWriter` is called for paths 1 and 3 right BEFORE the destination's pending write is
+    // noted (the caller's evidence ring entry then precedes the note, as a dispatch's elements do)
+    // with the destination's post-copy bytes when they are known now (path 1, at most `knownMax`
+    // bytes, and the source CPU-current at the note: no recorded write, pending image, unpublished
+    // shadow, pending label or copied writer over it; 0 disables) and the write-watch generation
+    // MarkWritten(destination) stamped, else an empty span and 0.
+    // `programAddress` keys the transfer's [gputime] range. Debug aids: APS5_COPY_SYNC=1 waits for
+    // every recorded store over either range; APS5_COPY_WAIT_SOURCE=1 waits (unlocked) for a
+    // source whose only unfinished writer is the oldest in-flight batch and decides again
+    // (`waitedForSource`/`waitMs`); APS5_COPY_VERIFY=1 follows every CPU copy with a full sync and
+    // byte compares (CopyVerifyCounts).
+    struct CopyOutcome {
+        // 0 a CPU copy, 1 a transfer between the host imports, 2 refused (not imported), 3 a
+        // device copy between the surfaces' storage images (StorageTexture::CopyFrom; the
+        // destination's write-back is deferred; APS5_NO_COPY_ALIAS=1 disables), 8 refused (over
+        // `gpuMax`; Driver.cpp's CopyOverGpuMax).
+        int path;
+        bool synced;
+        double syncMs;
+        // DestinationPending: an unsignaled batch writes the destination; DestinationUnsettled:
+        // every writer signaled but a completion store of one is still to land (as for the source).
+        // Shadow: a unit shadow holds unpublished results over either range (the GPU path
+        // publishes them; see UnitShadow.hpp).
+        enum Refusal { None = 0, Size, Image, SourcePending, SourceUnsettled, DestinationPending, DestinationUnsettled, Label, Reader, Shadow, Refusals };
+        int reason;
+        // Path 0 taken although a batch had written the source: every such batch had signaled.
+        bool sourceSettledBySignal;
+        bool waitedForSource;
+        double waitMs;
+        // The reader that refused (reason Reader): batch serial, its queue, the reader kind
+        // (Graphics::Recorder::ReadKind) and whether it is the open batch.
+        std::uint64_t readerSerial;
+        std::uint32_t readerQueue;
+        int readerKind;
+        bool readerOpen;
+    };
+    using CopyWriterNote = std::function<void(std::span<const std::byte> value, std::uint64_t generation)>;
+    CopyOutcome CopyBuffer(std::uint64_t destination, std::uint64_t source, std::size_t bytes, std::size_t cpuMax, std::size_t gpuMax, std::size_t knownMax, std::uint64_t programAddress, std::uint32_t queue, const CopyWriterNote& noteWriter);
+    // APS5_COPY_VERIFY=1 totals: CPU copies verified, whose source bytes changed over the sync (a
+    // producer the rule missed), whose destination bytes were overwritten (a store the rule
+    // missed), and whose destination an unsignaled batch read in place (the rule's own answer
+    // re-checked without the fence shortcut).
+    struct CopyVerification {
+        std::uint64_t verified, sourceChanged, destinationChanged, readerMissed;
+    };
+    static CopyVerification CopyVerifyCounts();
+    // Aliased copies (path 3): made, destination images created for them, and the transfers
+    // taken instead because the source had no live image, the destination's image differed in
+    // shape, or the image copy refused (keys, not imported, released).
+    struct CopyAliasing {
+        std::uint64_t aliased, created, noSource, shape, refused;
+    };
+    static CopyAliasing CopyAliasCounts();
     void ResolveMemory(std::uint64_t address, std::size_t bytes, bool writable);
     void* Window() const;
     void Resize(std::uint32_t width, std::uint32_t height);
     bool Presentable() const;
-    void PresentClear(std::uint32_t width, std::uint32_t height, bool opaque);
+    // A presentation is a few steps so the presenter holds GuestMemory::GpuMutex only while it
+    // touches the queue. Presentations are slots (FlipInFlight() + 1, each with its own command
+    // buffer, fence, kept resident image and dump buffer): RetirePresents(keep) (no mutex) retires
+    // the slots whose fence signaled, oldest first, and waits for the oldest ones until at most
+    // `keep` are in flight (the only CPU wait of the flip path; it returns the time spent waiting);
+    // AcquireImage takes the next swapchain image (no mutex needed: the swapchain and its fences
+    // are the presenter's own, and in FIFO mode this is where a frame waits for the vblank);
+    // PresentClear/PresentDisplayBuffer record and submit the frame into the next slot (under the
+    // mutex; the blit follows the frame's batches in queue order, so the displayed image is always
+    // complete) and return whether one was submitted; QueuePresent hands the image to the swapchain
+    // (under the mutex). FinishPresent = RetirePresents(0): the render fence wait of the
+    // synchronous paths (APS5_FLIP_INFLIGHT=0, APS5_SYNC_FLIP, PresentPixels). AcquireImage returns
+    // false when the swapchain is out of date (the frame is dropped; the next Resize recreates it);
+    // a present without a prior AcquireImage acquires itself, one whose slot is in flight waits for
+    // it. PresentPixels does all steps itself. Retiring a slot releases its kept image and writes
+    // its frame dump; WaitIdle, Resize and the destructor retire every slot before destroying what
+    // an in-flight blit reads.
+    // APS5_FLIP_INFLIGHT=<n>: presentations whose fence may be unsignaled when the next blit is
+    // submitted (0: today's render fence wait before the flip completes, 1 default, 2; larger
+    // values are refused; 0 when APS5_SYNC_FLIP is set).
+    static std::size_t FlipInFlight();
+    double RetirePresents(std::size_t keepInFlight);
+    // Whether presenting `buffer` will use the single scaler source, staging or upload objects (a
+    // GPU frame dump, the guest-memory path, a source-size change) while an in-flight blit still
+    // reads them, i.e. wait for every slot: the presenter then retires them before taking the
+    // mutex (the same wait under it is only the fallback when the answer changes in between).
+    bool PresentWaitsForSlots(const DisplayBuffer* buffer) const;
+    bool AcquireImage();
+    bool PresentClear(std::uint32_t width, std::uint32_t height, bool opaque);
     void PresentPixels(std::uint32_t width, std::uint32_t height, std::span<const std::byte> pixels);
-    void PresentDisplayBuffer(const DisplayBuffer& buffer);
-    void Dispatch(const ShaderRecompiler::RecompileResult& shader, std::uint32_t x, std::uint32_t y, std::uint32_t z, std::span<const Graphics::GuestMemorySnapshot> snapshots = {});
-    void Draw(const Graphics::State& graphics, const Pm4::DrawParameters& draw, std::span<const Graphics::CompiledShader> shaders, std::span<const Graphics::GuestMemorySnapshot> snapshots = {});
+    bool PresentDisplayBuffer(const DisplayBuffer& buffer);
+    double FinishPresent();
+    void QueuePresent();
+    // At the flip packet (under the mutex): the recorder's submissions so far and, under
+    // APS5_PROFILE_DRAW, how many in flight have not signaled (the frame record's batches_at_flip).
+    void FlipBatches(std::uint64_t& submissions, std::uint64_t& unsignaled) const;
+    // Cumulative presentation counters for the [present] line (APS5_PROFILE_DRAW): presentations
+    // submitted; over the batches the retired blits waited for: GPU busy time, idle gaps and the
+    // batches whose completion record was missing; sums of the batches submitted between the flip
+    // packet and the blit (ahead of it) and behind the blit (after_flip), of the last-submit and
+    // blit-submit times after the flip packet; noted in-place reads checked
+    // (APS5_FLIP_READ_CHECK=1) and found overwritten before the blit's fence.
+    struct PresentStatistics {
+        std::uint64_t presents, gpuUnread, batchesAheadOfBlit, batchesAfterFlip, readsChecked, readsOverwritten;
+        double gpuBusyMs, gpuGapMs, lastSubmitAfterFlipMs, blitSubmitAfterFlipMs;
+    };
+    static PresentStatistics PresentCounts();
+    // Stage A of a dispatch's resource build, run WITHOUT GuestMemory::GpuMutex before Dispatch or
+    // DispatchIndirect: the binding plan, data buffers, imports already serving the guest buffers,
+    // read-only copies and the descriptor set (see ShaderResources). The dispatch completes it
+    // under the mutex (stage B: texture lookups, the rest of the upload, descriptor writes) and
+    // records. Null when nothing is prepared: the resource cache may serve the dispatch (its
+    // Revalidate stays under the mutex), or APS5_LOCKED_BUILD=1 keeps the whole build under it as
+    // before. `shader` and `snapshots` must outlive the dispatch.
+    std::shared_ptr<PreparedDispatch> PrepareDispatch(const ShaderRecompiler::RecompileResult& shader, std::span<const Graphics::GuestMemorySnapshot> snapshots);
+    // APS5_PROFILE_DRAW: the parts of a PrepareDispatch in milliseconds, in the order key, find,
+    // precollect, presync, stage A (the driver's 'prepare:' rows).
+    static std::span<const double, 5> PreparePhaseMs(const PreparedDispatch& prepared);
+    // `recipe`, when given, receives the Recipe a successful call built for its dispatch-cache
+    // variant (design_cpu_final M4): only when the resource cache served or took the object
+    // (reusable, cacheable) and recipes are on (APS5_NO_DISPATCH_RECIPE=1 builds none); else null.
+    void Dispatch(const ShaderRecompiler::RecompileResult& shader, std::uint32_t x, std::uint32_t y, std::uint32_t z, std::span<const Graphics::GuestMemorySnapshot> snapshots = {}, std::uint64_t programAddress = 0, std::shared_ptr<PreparedDispatch> prepared = nullptr, std::shared_ptr<const Recipe>* recipe = nullptr);
+    // A dispatch whose group counts are the three dwords at `arguments` in guest memory
+    // (DISPATCH_INDIRECT): the GPU reads them in place from the host import, ordered after everything
+    // recorded before, so the CPU never waits for the shader that wrote them. When the GPU could not
+    // see the current bytes the counts are read on the CPU instead (through the flush hook, as the
+    // driver resolved every indirect dispatch before) and the dispatch is recorded as a direct one:
+    // cpuReason 1 storage-image results were pending over them, 2 a recorded dispatch writes them
+    // through a copied buffer (its CPU write-back lands only when the batch is reaped) or a label
+    // over those dwords is pending, 3 the memory is not host-imported; 0 when recorded GPU-side. argumentReadMs is
+    // what that CPU read (its sync) took. The device limit on group counts is not checked GPU-side.
+    // Debug aid: APS5_NO_GPU_INDIRECT=1 (in the driver) keeps every indirect dispatch on the CPU path.
+    struct IndirectOutcome {
+        int cpuReason;
+        double argumentReadMs;
+    };
+    IndirectOutcome DispatchIndirect(const ShaderRecompiler::RecompileResult& shader, std::uint64_t arguments, std::span<const Graphics::GuestMemorySnapshot> snapshots = {}, std::uint64_t programAddress = 0, std::shared_ptr<PreparedDispatch> prepared = nullptr, std::shared_ptr<const Recipe>* recipe = nullptr);
+    // A value-equal dispatch-cache hit with a recipe (design_cpu_final M4), in two steps.
+    // PrepareRecipe, WITHOUT GuestMemory::GpuMutex: the pre-checks (recipes on, the recipe's
+    // device is this one, its template and pipeline objects still alive), the template's
+    // precollect and the pre-sync from the recipe's surface list (recomputed from the template when
+    // the host-import identity it was made under moved). Null when a pre-check fails (counted on
+    // the [recipe] line by reason): the caller takes the ordinary path. `indirect` files the
+    // counters under the indirect rows.
+    std::shared_ptr<RecipeHit> PrepareRecipe(const std::shared_ptr<const Recipe>& recipe, bool indirect);
+    // DispatchRecipe, under the mutex after the packet's labels: reaps the pre-synced batches,
+    // proves the template current (ShaderResources::ProveCurrent, the authoritative check),
+    // touches the resource cache's LRU, decides an indirect dispatch's argument path as
+    // DispatchIndirect does, and records from the recipe's objects (the template refreshed by
+    // vkCmdUpdateBuffer only when its data-word hash differs from the recipe's). Rebuild when the
+    // proof failed (the template is removed from the cache and kept by the batch) or the device
+    // changed: nothing was recorded, the caller releases the mutex and restarts at
+    // PrepareDispatch. `arguments` non-zero is a DISPATCH_INDIRECT (`outcome` as DispatchIndirect
+    // reports it). `verify` (APS5_VERIFY_RECIPE=1): the PrepareDispatch made beside the hit, whose
+    // found object, the pipeline map's objects and the per-word RefreshData decision must agree
+    // with the recipe's, else the process aborts. `refreshByWords`: the driver's data-only hit
+    // (`shader` carries live flat-SRT words the recipe's hash does not name), so the template's
+    // data refresh is decided by the per-word compare instead of the hash.
+    RecipeOutcome DispatchRecipe(const ShaderRecompiler::RecompileResult& shader, std::uint32_t x, std::uint32_t y, std::uint32_t z, std::uint64_t arguments, std::uint64_t programAddress, const std::shared_ptr<RecipeHit>& hit, IndirectOutcome& outcome, const std::shared_ptr<PreparedDispatch>& verify = nullptr, bool refreshByWords = false);
+    // Whether dispatch-cache hits use recipes (APS5_NO_DISPATCH_RECIPE unset) and whether every
+    // hit is verified (APS5_VERIFY_RECIPE=1).
+    static bool DispatchRecipes();
+    static bool VerifyRecipes();
+    // Whether compute templates refresh their data buffers with a hit's words instead of keying
+    // them (APS5_NO_TEMPLATE_DATA_REFRESH unset); the driver's data-only hits require it.
+    static bool TemplateDataRefresh();
+    // The driver's recipe events for the [recipe] line: a restart after Rebuild, an attach; the
+    // rows are the dispatch, indirect and draw kinds. NoteDrawRecipeMiss files a draw hit's
+    // pre-check miss made in the driver (no recipe for the matched variants, or a device mismatch).
+    enum class RecipeEvent { Restart, Attach };
+    enum class RecipeKind : std::size_t { Dispatch, Indirect, Draw };
+    static void NoteRecipe(RecipeEvent event, bool indirect);
+    static void NoteRecipe(RecipeEvent event, RecipeKind kind);
+    enum class DrawRecipePrecheck : std::size_t { NoRecipe, Device };
+    static void NoteDrawRecipeMiss(DrawRecipePrecheck miss);
+    // `recipe`, when given, receives the DrawRecipe a recorded, cacheable, reusable, direct draw
+    // built for its draw-cache entry (design_cpu_final M8); null otherwise.
+    void Draw(const Graphics::State& graphics, const Pm4::DrawParameters& draw, std::span<const Graphics::CompiledShader> shaders, std::span<const Graphics::GuestMemorySnapshot> snapshots = {}, std::shared_ptr<const DrawRecipe>* recipe = nullptr);
+    // A draw-cache hit recorded from its recipe (Graphics::DrawWithRecipe), under the mutex after
+    // the packet's labels; Rebuild when the recipe's device is not this one or DrawWithRecipe
+    // missed (nothing recorded: the caller runs Draw with the hit's stages and re-attaches).
+    RecipeOutcome DrawFromRecipe(const Graphics::State& graphics, const Pm4::DrawParameters& draw, std::span<const Graphics::CompiledShader> shaders, std::span<const Graphics::GuestMemorySnapshot> snapshots, const std::shared_ptr<const DrawRecipe>& recipe);
+    // The indirect draw features the device enabled, for the driver's decision whether an indirect
+    // draw's records can be read by the GPU (Graphics::Draw) or must be resolved on the CPU.
+    struct IndirectDrawSupport {
+        bool firstInstance;
+        bool multi;
+        bool count;
+    };
+    IndirectDrawSupport DrawIndirectSupport() const;
     void EnqueueDraw(const Graphics::State& graphics, const Pm4::DrawParameters& draw, std::span<const Graphics::CompiledShader> shaders, std::span<const Graphics::GuestMemorySnapshot> snapshots = {});
 
 private:
+    // The device's Graphics::Context: a copy of the one built at setup (its instance functions
+    // resolved then, its function table filled then), or with APS5_NO_CONTEXT_CACHE=1 built anew.
     Graphics::Context graphicsContext() const;
-    void present(std::uint32_t width, std::uint32_t height, bool opaque, std::span<const std::byte> pixels, const DisplayBuffer* display = nullptr);
+    Graphics::Context buildContext() const;
+    // Body of Dispatch and DispatchIndirect: `arguments` 0 dispatches x, y, z groups.
+    IndirectOutcome dispatch(const ShaderRecompiler::RecompileResult& shader, std::uint32_t x, std::uint32_t y, std::uint32_t z, std::uint64_t arguments, std::span<const Graphics::GuestMemorySnapshot> snapshots, std::uint64_t programAddress, std::shared_ptr<PreparedDispatch> prepared, std::shared_ptr<const Recipe>* recipe);
+    // The stage-A pre-sync over `surfaces` (see PrepareDispatch): the serial waited for, 0 none.
+    std::uint64_t presync(std::span<const std::pair<std::uint64_t, std::uint64_t>> surfaces);
+    // A DISPATCH_INDIRECT's argument path (see DispatchIndirect): GPU-side from the host import, or
+    // the counts read on the CPU (the record then continues as a direct dispatch).
+    void decideIndirect(RecordedDispatch& record, IndirectOutcome& outcome, char* groupsText);
+    // The tail of a dispatch's device call from the open batch's command buffer to the completion
+    // registration: keeps, the template data refresh, barriers, bind, push, dispatch, marks.
+    void recordDispatch(RecordedDispatch& record);
+    bool present(std::uint32_t width, std::uint32_t height, bool opaque, std::span<const std::byte> pixels, const DisplayBuffer* display = nullptr, const std::shared_ptr<Graphics::StorageTexture>& resident = nullptr, VkFilter residentFilter = VK_FILTER_LINEAR, bool dumpFrame = false);
     struct State;
     std::unique_ptr<State> state;
+    std::uint64_t serial;
 };
 
 }

core/libs/prx/libSceAgcDriver/Execution/include/WorkerSampler.hpp

@@ -0,0 +1,11 @@
+#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_WORKERSAMPLER_HPP
+#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_WORKERSAMPLER_HPP
+
+namespace AgcDriver {
+
+// Starts sampling the calling thread when APS5_SAMPLE_WORKER names an output file.
+void StartWorkerSampler();
+
+}
+
+#endif

core/libs/prx/libSceAgcDriver/Execution/src/DeviceSynchronization.cpp

@@ -1,17 +0,0 @@
-#include "prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp"
-#include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
-#include "prx/libSceAgcDriver/Graphics/include/DrawQueue.hpp"
-#include "prx/libc/include/GuestMemoryTracking.hpp"
-
-namespace AgcDriver {
-
-void VulkanDevice::AcquireGpuMemory() {
-    std::lock_guard memoryLock(GuestMemoryTracking::GuestMemoryTrackingMutex_nid_postfix());
-    PerformanceTimer timing("Vulkan.AcquireGpuMemory");
-    const auto context = graphicsContext();
-    Graphics::Require(context.drawQueue != nullptr, "GPU memory barrier requires a draw queue");
-    context.drawQueue->RecordMemoryBarrier(context);
-    timing.Mark("barrier_record");
-}
-
-}

core/libs/prx/libSceAgcDriver/Execution/src/DisplayBuffer.cpp

@@ -1,6 +1,9 @@
 #include "prx/libSceAgcDriver/Execution/include/DisplayBuffer.hpp"
 #include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp"
 #include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
+#include <chrono>
+#include <cstdio>
+#include <cstdlib>
 #include <cstring>
 #include <limits>
 #include <stdexcept>
@@ -12,6 +15,15 @@ void require(bool condition, const char* reason) {
     if (!condition) throw std::runtime_error(reason);
 }
 
+// Bit 56 selects the 10-bit (A2B10G10R10) variant of the format.
+constexpr std::uint64_t PixelFormatUnormBit = 0x0100000000000000ull;
+constexpr std::uint64_t PixelFormatB8G8R8A8 = 0x8000000000000000ull;
+constexpr std::uint64_t PixelFormatR8G8B8A8 = 0x8000000022000000ull;
+
+std::uint64_t baseFormat(std::uint64_t pixelFormat) {
+    return pixelFormat & ~PixelFormatUnormBit;
+}
+
 std::uint32_t tileOffset(std::uint32_t x, std::uint32_t y) {
     return ((y << 4u) & 0x0070u) ^ ((y << 5u) & 0x0f00u) ^ ((y << 9u) & 0x1000u) ^ ((y << 8u) & 0x4000u) ^ ((x << 2u) & 0x000cu) ^ ((x << 5u) & 0x0380u) ^ ((x << 4u) & 0x0400u) ^ ((x << 6u) & 0x0800u) ^ ((x << 9u) & 0xa000u);
 }
@@ -20,24 +32,50 @@ std::uint32_t tileOffset(std::uint32_t x, std::uint32_t y) {
 
 std::size_t DisplayBufferSize(const DisplayBuffer& buffer) {
     require(buffer.width != 0 && buffer.height != 0 && buffer.width <= 16384 && buffer.height <= 16384, "VideoOut: invalid display buffer dimensions");
-    require(buffer.pixelFormat == 0x8000000000000000ull || buffer.pixelFormat == 0x8000000022000000ull, "VideoOut: unsupported display pixel format");
+    if (baseFormat(buffer.pixelFormat) != PixelFormatB8G8R8A8 && baseFormat(buffer.pixelFormat) != PixelFormatR8G8B8A8) {
+        char message[96];
+        std::snprintf(message, sizeof(message), "VideoOut: unsupported display pixel format 0x%016llx", static_cast<unsigned long long>(buffer.pixelFormat));
+        throw std::runtime_error(message);
+    }
     require(buffer.address != 0 && (buffer.address & 65535u) == 0, "VideoOut: display buffer requires 64 KiB alignment");
-    const auto size = static_cast<std::uint64_t>((buffer.width + 127u) / 128u) * ((buffer.height + 127u) / 128u) * 65536u;
+    require(buffer.tilingMode <= 1, "VideoOut: unsupported display tiling mode");
+    require(buffer.tilingMode != 0 || buffer.pitchInPixel == 0, "VideoOut: tiled display pitch is unsupported");
+    const auto pitch = buffer.pitchInPixel == 0 ? buffer.width : buffer.pitchInPixel;
+    require(pitch >= buffer.width && pitch <= 16384, "VideoOut: invalid linear display pitch");
+    const auto size = buffer.tilingMode == 1 ? static_cast<std::uint64_t>(pitch) * buffer.height * 4u
+        : static_cast<std::uint64_t>((buffer.width + 127u) / 128u) * ((buffer.height + 127u) / 128u) * 65536u;
     require(size <= std::numeric_limits<std::size_t>::max() && size <= std::numeric_limits<std::uintptr_t>::max() - buffer.address, "VideoOut: display buffer range overflow");
     return static_cast<std::size_t>(size);
 }
 
 std::vector<std::byte> DecodeDisplayBuffer(const DisplayBuffer& buffer, std::span<const std::byte> source) {
     PerformanceTimer timing("DisplayBuffer.Decode");
-    require(source.size() == DisplayBufferSize(buffer), "VideoOut: invalid tiled display buffer size");
+    require(source.size() == DisplayBufferSize(buffer), "VideoOut: invalid display buffer size");
     std::vector<std::byte> pixels(static_cast<std::size_t>(buffer.width) * buffer.height * 4);
     const auto blocksPerRow = (buffer.width + 127u) / 128u;
-    const bool rgba = buffer.pixelFormat == 0x8000000022000000ull;
+    const bool rgba = baseFormat(buffer.pixelFormat) == PixelFormatR8G8B8A8;
+    // Bit 56 marks the 10-bit variant: A2B10G10R10 with red in the low bits (the scanout draws render
+    // COLOR_2_10_10_10 into it). Presented as 8-bit BGRA by dropping the low two bits of each channel.
+    const bool tenBit = (buffer.pixelFormat & PixelFormatUnormBit) != 0;
     timing.Mark("validate_allocate");
     for (std::uint32_t y = 0; y < buffer.height; ++y) {
         for (std::uint32_t x = 0; x < buffer.width; ++x) {
-            const auto tiled = (static_cast<std::size_t>(y / 128u) * blocksPerRow + x / 128u) * 65536u + tileOffset(x, y);
+            const auto pitch = buffer.pitchInPixel == 0 ? buffer.width : buffer.pitchInPixel;
+            const auto tiled = buffer.tilingMode == 1 ? (static_cast<std::size_t>(y) * pitch + x) * 4u
+                : (static_cast<std::size_t>(y / 128u) * blocksPerRow + x / 128u) * 65536u + tileOffset(x, y);
             const auto linear = (static_cast<std::size_t>(y) * buffer.width + x) * 4;
+            if (tenBit) {
+                std::uint32_t word = 0;
+                std::memcpy(&word, source.data() + tiled, 4);
+                const auto red = static_cast<std::byte>((word >> 2u) & 0xffu);
+                const auto green = static_cast<std::byte>((word >> 12u) & 0xffu);
+                const auto blue = static_cast<std::byte>((word >> 22u) & 0xffu);
+                pixels[linear] = rgba ? blue : red;
+                pixels[linear + 1] = green;
+                pixels[linear + 2] = rgba ? red : blue;
+                pixels[linear + 3] = std::byte{0xff};
+                continue;
+            }
             pixels[linear] = source[tiled + (rgba ? 2 : 0)];
             pixels[linear + 1] = source[tiled + 1];
             pixels[linear + 2] = source[tiled + (rgba ? 0 : 2)];
@@ -50,6 +88,12 @@ std::vector<std::byte> DecodeDisplayBuffer(const DisplayBuffer& buffer, std::spa
 
 std::vector<std::byte> ReadDisplayBuffer(const DisplayBuffer& buffer) {
     const auto size = DisplayBufferSize(buffer);
+    static const bool traceGpu = std::getenv("APS5_TRACE_GPU") != nullptr;
+    if (traceGpu) {
+        static const auto start = std::chrono::steady_clock::now();
+        std::fprintf(stderr, "[gpu] scanout display buffer 0x%llx %ux%u format 0x%016llx at %.1f s\n", static_cast<unsigned long long>(buffer.address), buffer.width, buffer.height, static_cast<unsigned long long>(buffer.pixelFormat), std::chrono::duration<double>(std::chrono::steady_clock::now() - start).count());
+    }
+    GuestMemory::FlushGpuWrites(buffer.address, size);
     GuestMemory::CheckRange(reinterpret_cast<const void*>(buffer.address), size, 65536);
     const auto* source = reinterpret_cast<const std::byte*>(buffer.address);
     return DecodeDisplayBuffer(buffer, {source, size});

core/libs/prx/libSceAgcDriver/Execution/src/GuestGpuRange.cpp

@@ -1,18 +0,0 @@
-#include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp"
-#include "prx/libSceAgcDriver/Execution/include/MemoryAccessScope.hpp"
-#include "prx/libc/include/GuestMemoryTracking.hpp"
-#include <limits>
-#include <stdexcept>
-
-namespace AgcDriver::GuestMemory {
-
-void CheckGpuRange(const void* pointer, std::size_t bytes, std::size_t alignment, bool writable) {
-    const auto address = reinterpret_cast<std::uintptr_t>(pointer);
-    if (alignment == 0 || address == 0 || address % alignment != 0 || bytes > std::numeric_limits<std::uintptr_t>::max() - address) throw std::invalid_argument("invalid guest GPU memory range");
-    const MemoryAccessScope suspended(nullptr, nullptr);
-    GuestMemoryTracking::GuestMemoryTrackingValidate_nid_postfix(address, bytes, [writable](std::uint64_t first, std::size_t count) {
-        CheckRange(reinterpret_cast<const void*>(first), count, 1, writable);
-    });
-}
-
-}

core/libs/prx/libSceAgcDriver/Execution/src/Pm4.cpp

@@ -2,14 +2,23 @@
 #include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp"
 #include "prx/libc/include/General.hpp"
 #include <algorithm>
+#include <thread>
+#include <chrono>
 #include <cstdio>
 #include <cstring>
 #include <limits>
 #include <stdexcept>
+#include <string>
 
 namespace AgcDriver::Pm4 {
 namespace {
 
+std::string ToHex(std::uint32_t value) {
+    char buffer[16];
+    std::snprintf(buffer, sizeof(buffer), "%x", value);
+    return buffer;
+}
+
 void require(bool condition, const char* reason) {
     if (!condition) throw std::runtime_error(reason);
 }
@@ -25,6 +34,27 @@ std::uint32_t registerOffset(std::uint32_t value) {
     return offset;
 }
 
+// Debug aid: APS5_TRACE_CONTEXT_STATE logs context-state operations and the context registers
+// written while a context is pushed. Read once: getenv scans the environment under a lock.
+bool TraceContextState() {
+    static const bool value = std::getenv("APS5_TRACE_CONTEXT_STATE") != nullptr;
+    return value;
+}
+
+// Kill switch: APS5_NO_INDIRECT_DRAW=1 rejects the indirect draw packets at submission validation as
+// before they were implemented (the driver then dumps the submission as it did).
+bool IndirectDrawsDisabled() {
+    static const bool value = std::getenv("APS5_NO_INDIRECT_DRAW") != nullptr;
+    return value;
+}
+
+// An SH register an indirect draw packet may name for the CP's patch: the sentinel 0x280 (write
+// nowhere) or a user-data register of the vertex-side stage banks (0x8c.. for the vertex / geometry
+// programs, 0x10c.. for the local / hull programs).
+bool drawLocation(std::uint32_t value) {
+    return value == 0x280u || value - 0x8cu < 32u || value - 0x10cu < 32u;
+}
+
 Registers& registersFor(QueueState& queue, std::uint32_t opcode) {
     if (opcode == 0x69 || opcode == 0x9f) return queue.context;
     if (opcode == 0x76 || opcode == 0x63) return queue.shader;
@@ -32,7 +62,10 @@ Registers& registersFor(QueueState& queue, std::uint32_t opcode) {
 }
 
 void writeRegister(QueueState& queue, std::uint32_t opcode, std::uint32_t offset, std::uint32_t value) {
+    if ((opcode == 0x69 || opcode == 0x9f) && (offset == 0x8e || offset == 0x8f || offset == 0x318 || offset == 0x31b || offset == 0x31c || offset == 0x31d || offset == 0x390 || offset == 0x3b0 || offset == 0x3b8))
+        APS5_LOG_OUT_DEBUG("CONTEXT WRITE opcode=0x%x offset=0x%x value=0x%x", opcode, offset, value);
     registersFor(queue, opcode).insert_or_assign(offset, value);
+    if (TraceContextState() && (opcode == 0x69 || opcode == 0x9f) && ((offset >= 0x318 && offset < 0x318 + 8 * 0xf && (offset - 0x318) % 0xf == 0) || offset == 0x8e)) std::fprintf(stderr, "[context]   write %x = %08x (0x%x)\n", offset, value, opcode);
     if ((opcode == 0x64 || opcode == 0x79 || opcode == 0x7a) && offset == 0x243) queue.indexType = value & 3u;
 }
 
@@ -48,9 +81,9 @@ std::uint32_t dmaDestination(std::span<const std::uint32_t> packet) {
     return ((packet[1] >> 20u) & 3u) | ((packet[6] >> 25u) & 4u) | ((packet[6] >> 26u) & 8u);
 }
 
-void copyMemory(std::uint64_t source, std::uint64_t destination, std::size_t bytes, bool immediate) {
-    if (bytes == 0) return;
-    GuestMemory::CheckRange(reinterpret_cast<void*>(destination), bytes, 1, true);
+// The bytes a COPY_DATA or DMA_DATA stores: an immediate is a repeating 32-bit pattern, a memory
+// source is read through the checked path (it waits for recorded GPU work that writes it).
+std::vector<std::byte> copySource(std::uint64_t source, std::size_t bytes, bool immediate) {
     std::vector<std::byte> data(bytes);
     if (immediate) {
         const auto value = static_cast<std::uint32_t>(source);
@@ -58,12 +91,44 @@ void copyMemory(std::uint64_t source, std::uint64_t destination, std::size_t byt
     } else {
         GuestMemory::Read(source, data);
     }
-    GuestMemory::Write(destination, data);
+    return data;
+}
+
+constexpr std::uint32_t DmaSelectGds = 1;
+constexpr std::size_t GdsBytes = 0x10000;
+
+struct GdsStorage {
+    std::mutex mutex;
+    std::array<std::byte, GdsBytes> bytes{};
+};
+
+GdsStorage& Gds() {
+    static GdsStorage storage;
+    return storage;
+}
+
+bool gdsRange(std::uint64_t offset, std::size_t bytes) {
+    return offset <= GdsBytes && bytes <= GdsBytes - offset;
+}
+
+std::vector<std::byte> dmaSourceBytes(std::span<const std::uint32_t> packet) {
+    const std::size_t bytes = packet[6] & 0x3ffffffu;
+    if (dmaSource(packet) != DmaSelectGds) return copySource(address(packet[2], packet[3]), bytes, dmaSource(packet) == 2);
+    auto& gds = Gds();
+    std::lock_guard lock(gds.mutex);
+    return {gds.bytes.begin() + packet[2], gds.bytes.begin() + packet[2] + bytes};
+}
+
+void copyMemory(std::uint64_t source, std::uint64_t destination, std::size_t bytes, bool immediate) {
+    if (bytes == 0) return;
+    GuestMemory::CheckRange(reinterpret_cast<void*>(destination), bytes, 1, true);
+    GuestMemory::Write(destination, copySource(source, bytes, immediate));
 }
 
 }
 
 std::string Name(std::uint32_t header) {
+    if (FillerPacket(header)) return "PM4_FILLER";
     const auto opcode = (header >> 8u) & 0xffu;
     if (opcode == 0x10 && (header & 0xfcu) != 0) {
         switch ((header >> 2u) & 0x3fu) {
@@ -97,20 +162,25 @@ std::string_view UnsupportedReason(std::uint32_t header) {
         }
     }
     switch (opcode) {
-        case 0x11: case 0x12: case 0x13: case 0x15: case 0x16: case 0x26:
+        case 0x11: case 0x12: case 0x13: case 0x15: case 0x16: case 0x26: case 0x27:
         case 0x2a: case 0x2d: case 0x2f: case 0x35: case 0x37: case 0x40: case 0x42: case 0x46: case 0x50:
         case 0x58: case 0x63: case 0x64: case 0x69: case 0x76: case 0x79: case 0x7a:
         case 0x81: case 0x83: case 0x9f: return {};
-        case 0x24: case 0x25: case 0x27: case 0x2c: case 0x38: case 0x3a: case 0x8d:
+        case 0x24: case 0x25: case 0x2c: case 0x38:
+            if (IndirectDrawsDisabled()) return "graphics draw, shader stages and guest render-target materialization are not implemented";
+            return {};
+        case 0x3a: case 0x8d:
             return "graphics draw, shader stages and guest render-target materialization are not implemented";
         case 0x20: return "GPU query predication is not implemented";
         case 0x22: return "conditional command execution and conditional flip reservation are not implemented";
         case 0x33: case 0x3f: return "nested command buffers, branching and nested flip reservation are not implemented";
-        case 0x39: case 0x3c: case 0x59: case 0x93:
+        case 0x3c: case 0x93: return {};
+        case 0x39: case 0x59:
             return "cooperative command-queue waits are not implemented";
         case 0x84: case 0x85: case 0x86: case 0x88:
             return "separate CE/DE execution and counter synchronization are not implemented";
-        case 0x43: case 0x47: case 0x48: case 0x49:
+        case 0x49: return {};
+        case 0x43: case 0x47: case 0x48:
             return "guest cache actions, GPU events and interrupt delivery are not implemented";
         case 0x8e: return "GPU LOD statistics are not implemented; synthetic results are forbidden";
         case 0x28: case 0x41: case 0x68: case 0x78:
@@ -119,9 +189,17 @@ std::string_view UnsupportedReason(std::uint32_t header) {
     }
 }
 
+constexpr std::uint32_t DmaSourceCachePolicy = 3u << 13u;
+constexpr std::uint32_t DmaDestinationCachePolicy = 3u << 25u;
+
 void Validate(std::span<const std::uint32_t> packet, std::uint32_t queue) {
-    require(packet.size() >= 2, "truncated PM4 header or payload");
+    require(!packet.empty(), "truncated PM4 header");
     const auto header = packet[0];
+    if (FillerPacket(header)) {
+        require(packet.size() == 1, "invalid PM4 filler size");
+        return;
+    }
+    require(packet.size() >= 2, "truncated PM4 header or payload");
     require((header & 0xc0000000u) == 0xc0000000u, "unsupported PM4 packet type");
     require(packet.size() == ((header >> 16u) & 0x3fffu) + 2u, "invalid PM4 packet size");
     const auto opcode = (header >> 8u) & 0xffu;
@@ -153,7 +231,13 @@ void Validate(std::span<const std::uint32_t> packet, std::uint32_t queue) {
         }
         return;
     }
-    require((header & 0xffu) == 0 || (opcode == 0x11 && (header & 0xffu) == 2), "PM4 header flags are not implemented");
+    // Header bit 1 selects the compute shader type and bit 2 (register writes) resets the filter CAM;
+    // neither changes what the packet writes. Predication (bit 0) is not implemented.
+    const auto flags = header & 0xffu;
+    const bool registerWrite = opcode == 0x69 || opcode == 0x76 || opcode == 0x79 || opcode == 0x7a;
+    auto allowedFlags = opcode == 0x11 ? 2u : registerWrite ? 6u : opcode == 0x3c || opcode == 0x93 ? 2u : 0u;
+    if (IsTagMarker(packet)) allowedFlags |= 1u;
+    if ((flags & ~allowedFlags) != 0) throw std::runtime_error("PM4 header flags 0x" + ToHex(flags) + " are not implemented");
     switch (opcode) {
         case 0x11:
             size(4);
@@ -163,6 +247,13 @@ void Validate(std::span<const std::uint32_t> packet, std::uint32_t queue) {
         case 0x12: graphics(); size(2); require((packet[1] & ~0xfu) == 0, "unsupported CLEAR_STATE payload bits"); break;
         case 0x13: case 0x2f: graphics(); size(2); break;
         case 0x26: graphics(); size(3); break;
+        case 0x27:
+            graphics();
+            size(6);
+            require(packet[3] <= 0xffffu, "unsupported index address bits");
+            require(packet[4] <= packet[1], "index count exceeds maximum index size");
+            require((packet[5] & ~0x20u) == 0, "unsupported indexed draw flags");
+            break;
         case 0x2a: graphics(); size(2); require(packet[1] <= 3, "unsupported index-type modifiers"); break;
         case 0x2d:
             graphics();
@@ -175,10 +266,30 @@ void Validate(std::span<const std::uint32_t> packet, std::uint32_t queue) {
             require(packet[3] <= packet[1], "index count exceeds maximum index size");
             require((packet[4] & ~0x20u) == 0, "unsupported indexed draw flags");
             break;
-        case 0x15: size(5); require((packet[4] & ~0x8000u) == 0x41u, "dispatch modifiers are not implemented"); break;
+        case 0x24: case 0x25:
+            graphics();
+            size(5);
+            require((packet[1] & 3u) == 0, "misaligned indirect draw offset");
+            require((packet[2] >> 16u) == 0 && (packet[3] >> 16u) == 0, "start-index location semantics are not implemented");
+            require(drawLocation(packet[2]) && drawLocation(packet[3]), "invalid indirect draw register location");
+            require((packet[4] & ~0x20u) == (opcode == 0x24 ? 2u : 0u), "unsupported indirect draw initiator");
+            break;
+        case 0x2c: case 0x38:
+            graphics();
+            size(10);
+            require((packet[1] & 3u) == 0, "misaligned indirect draw offset");
+            require((packet[2] >> 16u) == 0 && (packet[3] >> 16u) == 0, "start-index location semantics are not implemented");
+            require(drawLocation(packet[2]) && drawLocation(packet[3]) && drawLocation(packet[4] & 0xffffu), "invalid indirect draw register location");
+            require((packet[4] & ~(0xffffu | (1u << 30u) | (1u << 31u))) == 0, "indirect multi-draw control bits are not implemented");
+            if ((packet[4] & (1u << 30u)) != 0) require((packet[6] & 3u) == 0 && address(packet[6], packet[7]) != 0, "invalid indirect draw count address");
+            else require(address(packet[6], packet[7]) == 0, "count address without an indirect draw count");
+            require((packet[8] & 3u) == 0 && packet[8] >= (opcode == 0x2c ? 16u : 20u), "invalid indirect draw stride");
+            require((packet[9] & ~0x20u) == (opcode == 0x2c ? 2u : 0u), "unsupported indirect draw initiator");
+            break;
+        case 0x15: size(5); if ((packet[4] & ~0x8020u) != 0x41u) throw std::runtime_error("dispatch modifiers 0x" + ToHex(packet[4]) + " are not implemented"); break;
         case 0x16:
             require(packet.size() == 3 || packet.size() == 4, "invalid indirect dispatch size");
-            require((packet.back() & ~0x8000u) == 0x41u, "indirect dispatch modifiers are not implemented");
+            require((packet.back() & ~0x8020u) == 0x41u, "indirect dispatch modifiers are not implemented");
             break;
         case 0x42: size(2); require(packet[1] == 0, "unsupported PFP_SYNC_ME payload"); break;
         case 0x46: {
@@ -222,6 +333,7 @@ void Validate(std::span<const std::uint32_t> packet, std::uint32_t queue) {
             require((packet[1] & 3u) == 0 && packet[3] == 0x80000000u && packet[4] <= 0x3fffu, "unsupported indirect-register address or control fields");
             break;
         case 0x69: case 0x76: case 0x79: case 0x7a: {
+            if (IsTagMarker(packet)) break;
             if (opcode != 0x76) graphics();
             require(packet.size() >= 3, "register packet has no values");
             if (opcode == 0x7a) require((packet[1] & 0xf0000000u) == 0 || (packet.size() == 3 && packet[1] == 0x20000243u), "indexed register bank selection is not implemented");
@@ -229,6 +341,20 @@ void Validate(std::span<const std::uint32_t> packet, std::uint32_t queue) {
             require(packet.size() - 2 <= 0x10000u - offset, "register range overflow");
             break;
         }
+        case 0x3c: case 0x93: {
+            size(opcode == 0x3c ? 7 : 9);
+            require((packet[1] & 0x10u) != 0, "register-space WAIT_REG_MEM is not implemented");
+            require((packet[1] & 7u) <= 6u, "invalid WAIT_REG_MEM compare function");
+            require((packet[2] & (opcode == 0x3c ? 3u : 7u)) == 0, "misaligned WAIT_REG_MEM address");
+            break;
+        }
+        case 0x49: {
+            size(8);
+            const auto dataSelect = packet[2] >> 29u;
+            require(dataSelect <= 3, "GDS RELEASE_MEM data is not implemented");
+            require(dataSelect == 0 || address(packet[3], packet[4]) == 0 || (packet[3] & (dataSelect == 1 ? 3u : 7u)) == 0, "misaligned RELEASE_MEM destination");
+            break;
+        }
         case 0x81:
             graphics();
             require(packet[1] <= 0xbffcu && (packet[1] & 3u) == 0 && packet.size() - 2 <= 0x3000u - packet[1] / 4u, "constant RAM write range overflow or misalignment");
@@ -257,15 +383,122 @@ void Validate(std::span<const std::uint32_t> packet, std::uint32_t queue) {
         }
         case 0x50:
             size(7);
-            require((packet[1] & ~0xe0300001u) == 0, "DMA_DATA cache or reserved fields are not implemented");
-            require(memorySelector(dmaDestination(packet)), "DMA_DATA register, GDS or prefetch destination is not implemented");
-            require(memorySelector(dmaSource(packet)) || dmaSource(packet) == 2, "DMA_DATA register or GDS source is not implemented");
+            require((packet[1] & ~(0xe0300001u | DmaSourceCachePolicy | DmaDestinationCachePolicy)) == 0, "DMA_DATA reserved fields are not implemented");
+            require(memorySelector(dmaDestination(packet)) || dmaDestination(packet) == DmaSelectGds, "DMA_DATA register or prefetch destination is not implemented");
+            require(memorySelector(dmaSource(packet)) || dmaSource(packet) == 2 || dmaSource(packet) == DmaSelectGds, "DMA_DATA register source is not implemented");
+            require(dmaSource(packet) != DmaSelectGds || (packet[3] == 0 && gdsRange(packet[2], packet[6] & 0x3ffffffu)), "DMA_DATA GDS source range exceeds the GDS");
+            require(dmaDestination(packet) != DmaSelectGds || (packet[5] == 0 && gdsRange(packet[4], packet[6] & 0x3ffffffu)), "DMA_DATA GDS destination range exceeds the GDS");
             require(dmaSource(packet) != 2 || packet[3] == 0, "DMA_DATA immediate exceeds 32 bits");
             break;
         default: throw std::runtime_error("known packet has no validator");
     }
 }
 
+bool IsTagMarker(std::span<const std::uint32_t> packet) {
+    // libSceAgc brackets waits and draws with predicated SET_UCONFIG_REG writes of a tag value to
+    // 0x342; they carry no register state.
+    return packet.size() >= 3 && ((packet[0] >> 8u) & 0xffu) == 0x79u && (packet[0] & 1u) != 0 && packet[1] == 0x342u;
+}
+
+namespace {
+
+bool waitCompares(std::span<const std::uint32_t> packet, bool wide, std::uint64_t value) {
+    const std::uint64_t reference = wide ? address(packet[4], packet[5]) : packet[4];
+    const std::uint64_t mask = wide ? address(packet[6], packet[7]) : packet[5];
+    const auto masked = value & mask;
+    switch (packet[1] & 7u) {
+        case 0: return true;
+        case 1: return masked < reference;
+        case 2: return masked <= reference;
+        case 3: return masked == reference;
+        case 4: return masked != reference;
+        case 5: return masked >= reference;
+        case 6: return masked > reference;
+        default: return false;
+    }
+}
+
+}
+
+bool WaitSatisfied(std::span<const std::uint32_t> packet) {
+    const bool wide = ((packet[0] >> 8u) & 0xffu) == 0x93u;
+    const auto source = address(packet[2], packet[3]);
+    std::uint64_t value = 0;
+    // Named for the [hooksync] attribution (the read goes through the flush hook).
+    const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::Wait);
+    GuestMemory::Read(source, std::as_writable_bytes(std::span(&value, 1)).first(wide ? 8 : 4), wide ? 8 : 4);
+    return waitCompares(packet, wide, value);
+}
+
+bool WaitSatisfiedUnchecked(std::span<const std::uint32_t> packet) {
+    const bool wide = ((packet[0] >> 8u) & 0xffu) == 0x93u;
+    const auto source = address(packet[2], packet[3]);
+    const std::uint64_t value = wide ? *reinterpret_cast<const volatile std::uint64_t*>(source) : *reinterpret_cast<const volatile std::uint32_t*>(source);
+    return waitCompares(packet, wide, value);
+}
+
+bool WaitComparesValue(std::span<const std::uint32_t> packet, std::uint64_t value) {
+    const bool wide = ((packet[0] >> 8u) & 0xffu) == 0x93u;
+    return waitCompares(packet, wide, value);
+}
+
+std::optional<LabelWrite> DecodeLabelWrite(std::span<const std::uint32_t> packet) {
+    const auto opcode = (packet[0] >> 8u) & 0xffu;
+    if (opcode == 0x49 && packet.size() >= 7) {
+        const auto dataSelect = packet[2] >> 29u;
+        const auto destination = address(packet[3], packet[4]);
+        if (dataSelect == 0 || dataSelect > 3) return std::nullopt;
+        if (destination == 0) return std::nullopt;
+        std::uint64_t value = address(packet[5], packet[6]);
+        if (dataSelect == 3) value = static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::steady_clock::now().time_since_epoch()).count() / 10);
+        LabelWrite label{destination, {}};
+        label.inlineSize = dataSelect == 1 ? 4 : 8;
+        std::memcpy(label.inlineBytes.data(), &value, label.inlineSize);
+        return label;
+    }
+    if (opcode == 0x37 && packet.size() > 4) {
+        const auto destination = address(packet[2], packet[3]);
+        if (destination == 0) return std::nullopt;
+        // WR_ONE_ADDR stores every value at the same address: the last one remains.
+        const auto words = (packet[1] & 0x10000u) != 0 ? packet.last(1) : packet.subspan(4);
+        return LabelWrite{destination, std::as_bytes(words)};
+    }
+    return std::nullopt;
+}
+
+std::optional<StoreWrite> ResolveStore(std::span<const std::uint32_t> packet, const QueueState& queue, std::size_t limit) {
+    // The same decode as Execute's cases below (a packet Validate accepted): a store that is not
+    // resolved here still runs there. The size and alignment tests precede any memory read: a store
+    // the GPU path would reject anyway must not read (and sync for) its source here and again there.
+    const auto fits = [limit](std::uint64_t destination, std::size_t bytes) { return bytes <= limit && destination % 4 == 0 && bytes % 4 == 0; };
+    switch ((packet[0] >> 8u) & 0xffu) {
+        case 0x40: {
+            if (packet.size() < 6) return std::nullopt;
+            const auto source = ((packet[1] & 0xfu) << 1u) | ((packet[1] >> 30u) & 1u);
+            const std::size_t bytes = (packet[1] & 0x10000u) != 0 ? 8 : 4;
+            const auto destination = address(packet[4], packet[5]);
+            if (!fits(destination, bytes)) return std::nullopt;
+            return StoreWrite{destination, {}, copySource(address(packet[2], packet[3]), bytes, source >= 10)};
+        }
+        case 0x50: {
+            if (packet.size() < 7 || dmaDestination(packet) == DmaSelectGds) return std::nullopt;
+            const std::size_t bytes = packet[6] & 0x3ffffffu;
+            const auto destination = address(packet[4], packet[5]);
+            if (!fits(destination, bytes)) return std::nullopt;
+            return StoreWrite{destination, {}, dmaSourceBytes(packet)};
+        }
+        case 0x83: {
+            if (packet.size() < 5 || packet[1] / 4 > queue.constantRam.size() || packet[2] > queue.constantRam.size() - packet[1] / 4) return std::nullopt;
+            const auto words = std::span<const std::uint32_t>(queue.constantRam).subspan(packet[1] / 4, packet[2]);
+            if (words.size_bytes() > limit) return std::nullopt;
+            // A view of the queue's constant RAM: valid until the next CONST_RAM write on this queue,
+            // which is a later packet of the same worker.
+            return StoreWrite{address(packet[3], packet[4]), std::as_bytes(words), {}};
+        }
+        default: return std::nullopt;
+    }
+}
+
 bool UsesGpuCacheBarrier(std::span<const std::uint32_t> packet) {
     require(!packet.empty() && ((packet[0] >> 8u) & 0xffu) == 0x58, "cache barrier requires ACQUIRE_MEM");
     Validate(packet, 0);
@@ -274,26 +507,107 @@ bool UsesGpuCacheBarrier(std::span<const std::uint32_t> packet) {
 
 bool AccessesMemory(std::uint32_t header) {
     switch ((header >> 8u) & 0xffu) {
-        case 0x16: case 0x2d: case 0x35: case 0x37: case 0x40: case 0x50: case 0x63: case 0x64: case 0x83: case 0x9f: return true;
+        case 0x27: case 0x3c: case 0x93: case 0x49: case 0x16: case 0x2d: case 0x35: case 0x24: case 0x25: case 0x2c: case 0x38: case 0x37: case 0x40: case 0x50: case 0x63: case 0x64: case 0x83: case 0x9f: return true;
         default: return false;
     }
 }
 
-std::array<std::uint32_t, 5> ResolveDispatch(std::span<const std::uint32_t> packet, const QueueState& queue) {
-    std::uint64_t source = 0;
-    if (packet.size() == 4) source = address(packet[1], packet[2]);
-    else {
-        require(queue.dispatchIndirectBase != 0, "indirect dispatch base has not been set");
-        require(packet[1] <= std::numeric_limits<std::uint64_t>::max() - queue.dispatchIndirectBase, "indirect dispatch address overflow");
-        source = queue.dispatchIndirectBase + packet[1];
-    }
-    std::array<std::uint32_t, 5> result{0xc0031500u, 0, 0, 0, packet.back()};
-    GuestMemory::Read(source, std::as_writable_bytes(std::span(result).subspan(1, 3)), 4);
+std::uint64_t DispatchArgumentAddress(std::span<const std::uint32_t> packet, const QueueState& queue) {
+    if (packet.size() == 4) return address(packet[1], packet[2]);
+    require(queue.dispatchIndirectBase != 0, "indirect dispatch base has not been set");
+    require(packet[1] <= std::numeric_limits<std::uint64_t>::max() - queue.dispatchIndirectBase, "indirect dispatch address overflow");
+    return queue.dispatchIndirectBase + packet[1];
+}
+
+std::array<std::uint32_t, 5> ReadDispatchArguments(std::uint64_t arguments, std::uint32_t initiator) {
+    std::array<std::uint32_t, 5> result{0xc0031500u, 0, 0, 0, initiator};
+    // Named for the [hooksync] attribution (the read goes through the flush hook).
+    const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::IndirectArguments);
+    GuestMemory::Read(arguments, std::as_writable_bytes(std::span(result).subspan(1, 3)), 4);
     return result;
 }
 
+std::array<std::uint32_t, 5> ResolveDispatch(std::span<const std::uint32_t> packet, const QueueState& queue) {
+    return ReadDispatchArguments(DispatchArgumentAddress(packet, queue), packet.back());
+}
+
+namespace {
+
+// The index buffer state an indexed draw binds: the index size for the journaled INDEX_TYPE and the
+// checks every indexed draw makes on INDEX_BASE.
+std::uint32_t indexSizeOf(const QueueState& queue) {
+    require(queue.indexType <= 2, "unsupported index type");
+    const std::uint32_t indexSize = queue.indexType == 0 ? 2 : queue.indexType == 1 ? 4 : 1;
+    require(queue.indexBase != 0 && queue.indexBase % indexSize == 0, "null or misaligned index base");
+    return indexSize;
+}
+
+DrawParameters resolveIndirectDraw(std::span<const std::uint32_t> packet, const QueueState& queue) {
+    const auto opcode = (packet[0] >> 8u) & 0xffu;
+    const bool indexed = opcode == 0x25 || opcode == 0x38;
+    const bool multi = opcode == 0x2c || opcode == 0x38;
+    require(queue.drawIndirectBase != 0, "indirect draw base has not been set");
+    require(packet[1] <= std::numeric_limits<std::uint64_t>::max() - queue.drawIndirectBase, "indirect draw address overflow");
+    DrawParameters::IndirectDraw indirect{};
+    indirect.arguments = queue.drawIndirectBase + packet[1];
+    indirect.opcode = opcode;
+    indirect.recordBytes = indexed ? 20 : 16;
+    indirect.stride = multi ? packet[8] : indirect.recordBytes;
+    indirect.count = multi ? packet[5] : 1;
+    indirect.countIndirect = multi && (packet[4] & (1u << 30u)) != 0;
+    indirect.countAddress = multi ? address(packet[6], packet[7]) : 0;
+    indirect.baseVertexLocation = packet[2];
+    indirect.startInstanceLocation = packet[3];
+    indirect.drawIndexLocation = multi ? packet[4] & 0xffffu : 0x280u;
+    indirect.drawIndexEnabled = multi && (packet[4] >> 31u) != 0;
+    indirect.indxOffset = 0;
+    require(indirect.count <= (std::numeric_limits<std::uint64_t>::max() - indirect.arguments - indirect.recordBytes) / indirect.stride, "indirect draw record range overflow");
+    // The counts are unknown here (the CP takes them from the record; NUM_INSTANCES is ignored).
+    DrawParameters draw{0, 0, 0, 0, packet.back() & 0x20u, indexed, 0, 0};
+    if (!indexed) {
+        const auto offset = queue.userConfig.find(0x24a);
+        require(offset != queue.userConfig.end(), "missing GE_INDX_OFFSET register");
+        draw.firstVertex = offset->second;
+        indirect.indxOffset = offset->second;
+    } else {
+        // The CP clamps firstIndex + indexCount to INDEX_BUFFER_SIZE: the declared range is the
+        // hardware's view of the index buffer, so that is what gets bound.
+        const auto indexSize = indexSizeOf(queue);
+        require(queue.indexBufferSize != 0, "INDEX_BUFFER_SIZE has not been set");
+        const auto bytes = static_cast<std::uint64_t>(queue.indexBufferSize) * indexSize;
+        require(bytes <= std::numeric_limits<std::size_t>::max() && bytes <= std::numeric_limits<std::uint64_t>::max() - queue.indexBase, "index range size overflow");
+        GuestMemory::CheckRange(reinterpret_cast<const void*>(queue.indexBase), static_cast<std::size_t>(bytes), indexSize);
+        draw.indexAddress = queue.indexBase;
+        draw.indexCount = queue.indexBufferSize;
+        draw.indexSize = indexSize;
+    }
+    draw.indirect = indirect;
+    return draw;
+}
+
+}
+
+DrawArguments ReadDrawArguments(const DrawParameters::IndirectDraw& indirect, std::uint32_t record) {
+    require(record < indirect.count, "indirect draw record index exceeds the packet's count");
+    std::array<std::uint32_t, 5> words{};
+    // Named for the [hooksync] attribution (the read goes through the flush hook).
+    const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::IndirectArguments);
+    GuestMemory::Read(indirect.arguments + static_cast<std::uint64_t>(record) * indirect.stride, std::as_writable_bytes(std::span(words)).first(indirect.recordBytes), 4);
+    if (indirect.recordBytes == 20) return {words[0], words[1], words[2], words[3], words[4]};
+    return {words[0], words[1], words[2], 0, words[3]};
+}
+
+std::uint32_t ReadDrawCount(const DrawParameters::IndirectDraw& indirect) {
+    require(indirect.countIndirect && indirect.countAddress != 0, "indirect draw has no count address");
+    std::uint32_t count = 0;
+    const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::IndirectArguments);
+    GuestMemory::Read(indirect.countAddress, std::as_writable_bytes(std::span(&count, 1)), 4);
+    return count;
+}
+
 DrawParameters ResolveDraw(std::span<const std::uint32_t> packet, const QueueState& queue) {
     Validate(packet, 0);
+    if (IndirectDrawOpcode((packet[0] >> 8u) & 0xffu)) return resolveIndirectDraw(packet, queue);
     if (((packet[0] >> 8u) & 0xffu) == 0x2d) {
         const auto offset = queue.userConfig.find(0x24a);
         require(offset != queue.userConfig.end(), "missing GE_INDX_OFFSET register");
@@ -301,17 +615,21 @@ DrawParameters ResolveDraw(std::span<const std::uint32_t> packet, const QueueSta
         require(packet[1] == 0 || firstVertex <= std::numeric_limits<std::uint32_t>::max() - (packet[1] - 1u), "auto draw vertex range overflow");
         return {0, packet[1], 0, queue.instanceCount, packet[2] & 0x20u, false, firstVertex, 0};
     }
-    require(((packet[0] >> 8u) & 0xffu) == 0x35, "expected DRAW_INDEX_OFFSET_2 packet");
+    const bool explicitAddress = ((packet[0] >> 8u) & 0xffu) == 0x27;
+    require(explicitAddress || ((packet[0] >> 8u) & 0xffu) == 0x35, "expected indexed draw packet");
     require(queue.indexType <= 2, "unsupported index type");
     const std::uint32_t indexSize = queue.indexType == 0 ? 2 : queue.indexType == 1 ? 4 : 1;
-    require(queue.indexBase != 0 && queue.indexBase % indexSize == 0, "null or misaligned index base");
-    const auto offset = static_cast<std::uint64_t>(packet[2]) * indexSize;
-    require(offset <= std::numeric_limits<std::uint64_t>::max() - queue.indexBase, "index address overflow");
-    const auto address = queue.indexBase + offset;
-    const auto bytes = static_cast<std::uint64_t>(packet[3]) * indexSize;
+    const auto base = explicitAddress ? address(packet[2], packet[3]) : queue.indexBase;
+    const auto count = explicitAddress ? packet[4] : packet[3];
+    require(base != 0 && base % indexSize == 0, "null or misaligned index base");
+    const auto offset = explicitAddress ? 0 : static_cast<std::uint64_t>(packet[2]) * indexSize;
+    require(offset <= std::numeric_limits<std::uint64_t>::max() - base, "index address overflow");
+    const auto address = base + offset;
+    const auto bytes = static_cast<std::uint64_t>(count) * indexSize;
     require(bytes <= std::numeric_limits<std::size_t>::max(), "index range size overflow");
     GuestMemory::CheckRange(reinterpret_cast<const void*>(address), static_cast<std::size_t>(bytes), indexSize);
-    return {address, packet[3], indexSize, queue.instanceCount, packet[4]};
+    APS5_LOG_OUT_DEBUG("ResolveDraw context targetMask=0x%x shaderMask=0x%x indexCount=%u indexType=%u instances=%u", queue.context.contains(0x8e) ? queue.context.at(0x8e) : 0u, queue.context.contains(0x8f) ? queue.context.at(0x8f) : 0u, count, queue.indexType, queue.instanceCount);
+    return {address, count, indexSize, queue.instanceCount, packet.back()};
 }
 
 void Execute(std::span<const std::uint32_t> packet, QueueState& queue) {
@@ -327,6 +645,8 @@ void Execute(std::span<const std::uint32_t> packet, QueueState& queue) {
                     queue.markers.pop_back();
                     return;
                 case 0x1a:
+                    if (TraceContextState()) std::fprintf(stderr, "[context] op %u: %zu registers, saved %d, cb0 %x info %x mask %x\n", packet[1], queue.context.size(), queue.savedContext.has_value() ? 1 : 0, queue.context.contains(0x318) ? queue.context.at(0x318) : 0u, queue.context.contains(0x31c) ? queue.context.at(0x31c) : 0u, queue.context.contains(0x8e) ? queue.context.at(0x8e) : 0u);
+                    APS5_LOG_OUT_DEBUG("CONTEXT_STATE operation=%u targetMaskBefore=0x%x shaderMaskBefore=0x%x", packet[1], queue.context.contains(0x8e) ? queue.context.at(0x8e) : 0u, queue.context.contains(0x8f) ? queue.context.at(0x8f) : 0u);
                     switch (packet[1]) {
                         case 0: queue.ClearContext(); break;
                         case 1: case 3:
@@ -340,6 +660,7 @@ void Execute(std::span<const std::uint32_t> packet, QueueState& queue) {
                             queue.savedContext.reset();
                             break;
                     }
+                    APS5_LOG_OUT_DEBUG("CONTEXT_STATE done operation=%u targetMaskAfter=0x%x shaderMaskAfter=0x%x", packet[1], queue.context.contains(0x8e) ? queue.context.at(0x8e) : 0u, queue.context.contains(0x8f) ? queue.context.at(0x8f) : 0u);
                     return;
                 default: throw std::runtime_error("custom packet requires driver execution");
             }
@@ -347,6 +668,7 @@ void Execute(std::span<const std::uint32_t> packet, QueueState& queue) {
             ((packet[0] & 2u) == 0 ? queue.drawIndirectBase : queue.dispatchIndirectBase) = address(packet[2], packet[3]);
             return;
         case 0x12:
+            APS5_LOG_OUT_DEBUG("CLEAR_STATE targetMaskBefore=0x%x shaderMaskBefore=0x%x", queue.context.contains(0x8e) ? queue.context.at(0x8e) : 0u, queue.context.contains(0x8f) ? queue.context.at(0x8f) : 0u);
             queue.ClearContext(); return;
         case 0x13: queue.indexBufferSize = packet[1]; return;
         case 0x26: queue.indexBase = address(packet[1], packet[2]); return;
@@ -354,12 +676,47 @@ void Execute(std::span<const std::uint32_t> packet, QueueState& queue) {
         case 0x2f: queue.instanceCount = packet[1]; return;
         case 0x63: case 0x64: case 0x9f: {
             std::vector<std::uint32_t> pairs(static_cast<std::size_t>(packet[4]) * 2);
+            // Named for the [hooksync] attribution (the read goes through the flush hook).
+            const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::Registers);
             GuestMemory::Read(address(packet[1], packet[2]), std::as_writable_bytes(std::span(pairs)), 4);
             for (std::size_t i = 0; i < pairs.size(); i += 2) registerOffset(pairs[i]);
             for (std::size_t i = 0; i < pairs.size(); i += 2) writeRegister(queue, opcode, registerOffset(pairs[i]), pairs[i + 1]);
+            if (queue.savedContext.has_value() && TraceContextState()) {
+                std::fprintf(stderr, "[context]   indirect 0x%x:", opcode);
+                for (std::size_t i = 0; i < pairs.size(); i += 2) std::fprintf(stderr, " %x", registerOffset(pairs[i]));
+                std::fprintf(stderr, "\n");
+            }
+            return;
+        }
+        case 0x3c: case 0x93: {
+            // The waited-on value is written by the CPU or another queue; poll it like the CP would.
+            const auto start = std::chrono::steady_clock::now();
+            bool warned = false;
+            while (!WaitSatisfied(packet)) {
+                std::this_thread::sleep_for(std::chrono::microseconds(50));
+                if (!warned && std::chrono::steady_clock::now() - start > std::chrono::seconds(5)) {
+                    warned = true;
+                    std::fprintf(stderr, "[gpu] WAIT_REG_MEM at 0x%llx still waiting after 5s (function %u ref 0x%x mask 0x%x)\n",
+                                 static_cast<unsigned long long>(address(packet[2], packet[3])), packet[1] & 7u, packet[4], packet[5]);
+                }
+            }
+            return;
+        }
+        case 0x49: {
+            // Earlier work has drained by the time the driver reaches this packet, so the end-of-pipe
+            // write can happen immediately. Interrupt delivery is not modeled.
+            const auto dataSelect = packet[2] >> 29u;
+            const auto destination = address(packet[3], packet[4]);
+            if (dataSelect == 0 || destination == 0) return;
+            std::uint64_t value = address(packet[5], packet[6]);
+            if (dataSelect == 3) {
+                value = static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::steady_clock::now().time_since_epoch()).count() / 10);
+            }
+            GuestMemory::Write(destination, std::as_bytes(std::span(&value, 1)).first(dataSelect == 1 ? 4 : 8), dataSelect == 1 ? 4 : 8);
             return;
         }
         case 0x69: case 0x76: case 0x79: case 0x7a: {
+            if (IsTagMarker(packet)) return;
             const auto offset = registerOffset(packet[1]);
             for (std::size_t i = 2; i < packet.size(); ++i) writeRegister(queue, opcode, offset + static_cast<std::uint32_t>(i - 2), packet[i]);
             return;
@@ -382,9 +739,21 @@ void Execute(std::span<const std::uint32_t> packet, QueueState& queue) {
             copyMemory(address(packet[2], packet[3]), address(packet[4], packet[5]), (packet[1] & 0x10000u) != 0 ? 8 : 4, source >= 10);
             return;
         }
-        case 0x50:
-            copyMemory(address(packet[2], packet[3]), address(packet[4], packet[5]), packet[6] & 0x3ffffffu, dmaSource(packet) == 2);
+        case 0x50: {
+            const std::size_t bytes = packet[6] & 0x3ffffffu;
+            if (bytes == 0) return;
+            auto data = dmaSourceBytes(packet);
+            if (dmaDestination(packet) == DmaSelectGds) {
+                auto& gds = Gds();
+                std::lock_guard lock(gds.mutex);
+                std::copy(data.begin(), data.end(), gds.bytes.begin() + packet[4]);
+                return;
+            }
+            const auto destination = address(packet[4], packet[5]);
+            GuestMemory::CheckRange(reinterpret_cast<void*>(destination), bytes, 1, true);
+            GuestMemory::Write(destination, data);
             return;
+        }
         default: throw std::runtime_error("packet requires driver execution");
     }
 }

core/libs/prx/libSceAgcDriver/Execution/src/PresentationImage.cpp

@@ -1,29 +0,0 @@
-#include "prx/libSceAgcDriver/Execution/include/PresentationScaler.hpp"
-
-namespace AgcDriver {
-
-void PresentationScaler::RecordImage(VkCommandBuffer commands, VkImage image) {
-    Graphics::Require(sourceImage != VK_NULL_HANDLE && image != VK_NULL_HANDLE, "presentation image is unavailable");
-    const auto pipelineBarrier = context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier");
-    VkImageMemoryBarrier barrier{VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER};
-    barrier.dstAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT;
-    barrier.oldLayout = VK_IMAGE_LAYOUT_UNDEFINED;
-    barrier.newLayout = VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL;
-    barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
-    barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
-    barrier.image = sourceImage;
-    barrier.subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1};
-    pipelineBarrier(commands, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, nullptr, 0, nullptr, 1, &barrier);
-    VkImageCopy copy{};
-    copy.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1};
-    copy.dstSubresource = copy.srcSubresource;
-    copy.extent = {sourceWidth, sourceHeight, 1};
-    context.Function<PFN_vkCmdCopyImage>("vkCmdCopyImage")(commands, image, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, sourceImage, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &copy);
-    barrier.srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT;
-    barrier.dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT;
-    barrier.oldLayout = VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL;
-    barrier.newLayout = VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL;
-    pipelineBarrier(commands, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, nullptr, 0, nullptr, 1, &barrier);
-}
-
-}

core/libs/prx/libSceAgcDriver/Execution/src/PresentationScaler.cpp

@@ -1,5 +1,6 @@
 #include "prx/libSceAgcDriver/Execution/include/PresentationScaler.hpp"
 #include "prx/libSceAgcDriver/Execution/include/AspectFit.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/Resources.hpp"
 
 namespace AgcDriver {
 
@@ -77,17 +78,56 @@ void PresentationScaler::RecordUpload(VkCommandBuffer commands, VkBuffer uploadB
     pipelineBarrier(commands, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, nullptr, 0, nullptr, 1, &barrier);
 }
 
+void PresentationScaler::RecordBlitInto(VkCommandBuffer commands, VkImage image, VkImageLayout layout, VkFilter filter) {
+    Graphics::Require(sourceImage != VK_NULL_HANDLE && image != VK_NULL_HANDLE, "presentation image is unavailable");
+    const auto pipelineBarrier = context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier");
+    VkImageMemoryBarrier barrier{VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER};
+    barrier.dstAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT;
+    barrier.oldLayout = VK_IMAGE_LAYOUT_UNDEFINED;
+    barrier.newLayout = VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL;
+    barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
+    barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
+    barrier.image = sourceImage;
+    barrier.subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1};
+    pipelineBarrier(commands, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, nullptr, 0, nullptr, 1, &barrier);
+    VkImageBlit blit{};
+    blit.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1};
+    blit.srcOffsets[1] = {static_cast<std::int32_t>(sourceWidth), static_cast<std::int32_t>(sourceHeight), 1};
+    blit.dstSubresource = blit.srcSubresource;
+    blit.dstOffsets[1] = blit.srcOffsets[1];
+    context.Function<PFN_vkCmdBlitImage>("vkCmdBlitImage")(commands, image, layout, sourceImage, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &blit, filter);
+    barrier.srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT;
+    barrier.dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT;
+    barrier.oldLayout = VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL;
+    barrier.newLayout = VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL;
+    pipelineBarrier(commands, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, nullptr, 0, nullptr, 1, &barrier);
+}
+
+void PresentationScaler::RecordReadback(VkCommandBuffer commands, VkBuffer destination) {
+    Graphics::Require(sourceImage != VK_NULL_HANDLE && destination != VK_NULL_HANDLE, "presentation readback is unavailable");
+    VkBufferImageCopy copy{};
+    copy.imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1};
+    copy.imageExtent = {sourceWidth, sourceHeight, 1u};
+    context.Function<PFN_vkCmdCopyImageToBuffer>("vkCmdCopyImageToBuffer")(commands, sourceImage, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, destination, 1, &copy);
+    Graphics::RecordMemoryBarrier(context, commands, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_HOST_BIT, VK_ACCESS_TRANSFER_WRITE_BIT, VK_ACCESS_HOST_READ_BIT);
+}
+
 void PresentationScaler::RecordBlit(VkCommandBuffer commands, VkImage destinationImage, std::uint32_t destinationWidth, std::uint32_t destinationHeight) {
     Graphics::Require(sourceImage != VK_NULL_HANDLE, "presentation source image has not been created");
-    const auto rect = ComputeContainRect_nid_postfix(sourceWidth, sourceHeight, destinationWidth, destinationHeight);
+    RecordBlitFrom(context, commands, sourceImage, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, sourceWidth, sourceHeight, VK_FILTER_LINEAR, destinationImage, destinationWidth, destinationHeight);
+}
+
+void PresentationScaler::RecordBlitFrom(const Graphics::Context& context, VkCommandBuffer commands, VkImage image, VkImageLayout layout, std::uint32_t width, std::uint32_t height, VkFilter filter, VkImage destinationImage, std::uint32_t destinationWidth, std::uint32_t destinationHeight) {
+    Graphics::Require(image != VK_NULL_HANDLE && width != 0 && height != 0, "presentation blit source is unavailable");
+    const auto rect = ComputeContainRect_nid_postfix(width, height, destinationWidth, destinationHeight);
     VkImageBlit blit{};
     blit.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1};
     blit.srcOffsets[0] = {0, 0, 0};
-    blit.srcOffsets[1] = {static_cast<std::int32_t>(sourceWidth), static_cast<std::int32_t>(sourceHeight), 1};
+    blit.srcOffsets[1] = {static_cast<std::int32_t>(width), static_cast<std::int32_t>(height), 1};
     blit.dstSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1};
     blit.dstOffsets[0] = {rect.x, rect.y, 0};
     blit.dstOffsets[1] = {rect.x + static_cast<std::int32_t>(rect.width), rect.y + static_cast<std::int32_t>(rect.height), 1};
-    context.Function<PFN_vkCmdBlitImage>("vkCmdBlitImage")(commands, sourceImage, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, destinationImage, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &blit, VK_FILTER_LINEAR);
+    context.Function<PFN_vkCmdBlitImage>("vkCmdBlitImage")(commands, image, layout, destinationImage, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &blit, filter);
 }
 
 }

core/libs/prx/libSceAgcDriver/Execution/src/ShaderMemory.cpp

@@ -3,69 +3,328 @@
 #include "Optimization/RequestMemoryView.hpp"
 #include "Optimization/ResourceMaterializer.hpp"
 #include "Optimization/ResourceProgram.hpp"
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <cstdio>
+#include <cstdlib>
 #include <cstring>
-#include <iterator>
 #include <limits>
 #include <stdexcept>
-#include <utility>
+#include <vector>
 
 namespace AgcDriver {
+namespace {
 
-ShaderMemory::ShaderMemory(std::span<const ShaderRecompiler::MemoryRegion> initial) {
-    const ShaderRecompiler::RequestMemoryView validated(initial);
-    for (const auto& region : initial) {
-        regions.emplace(region.guestAddress, std::vector<std::byte>(region.bytes.begin(), region.bytes.end()));
+struct CaptureProfile {
+    std::atomic<std::uint64_t> captures{0};
+    std::atomic<std::uint64_t> reads{0};
+    std::atomic<std::uint64_t> pages{0};
+    // The whole CaptureResources call: the plan lookup and the materialization are one call.
+    std::atomic<std::uint64_t> captureNanoseconds{0};
+    // Pages read word by word because recorded GPU work writes them, and the word reads of those
+    // pages that waited (a read taking longer than WordWaitNanoseconds went through a hook sync),
+    // were read raw on the driver's evidence, or were read both ways (verify) and differed.
+    std::atomic<std::uint64_t> pagesWordwise{0};
+    std::atomic<std::uint64_t> wordWaits{0};
+    std::atomic<std::uint64_t> wordWaitNanoseconds{0};
+    std::atomic<std::uint64_t> wordsRaw{0};
+    std::atomic<std::uint64_t> wordsVerified{0};
+    std::atomic<std::uint64_t> wordMismatches{0};
+    // Words served from the driver's known values (never profile-gated: the [copy] line reads them).
+    std::atomic<std::uint64_t> wordsKnown{0};
+    std::atomic<std::uint64_t> wordsKnownVerified{0};
+    std::atomic<std::uint64_t> wordsKnownMismatches{0};
+    // Hook reads not reported to the observer because no GPU wait happened across them.
+    std::atomic<std::uint64_t> observationsSkipped{0};
+    // Sub-phases of a capture: the source lookup (the stage inputs, the recompiler's key over the
+    // code and the plan, timed by CaptureResources itself: ResourceCapture::sourceNanoseconds),
+    // whole-page fetches, per-word fetches (their hook waits included) and the GPU waits the flush
+    // hook made anywhere inside the capture (Recorder::ThreadWaitedMs). The walk itself is what
+    // remains of the capture time after the lookup, the fetches and the specialization
+    // (ResourceMaterializer).
+    std::atomic<std::uint64_t> resolveNanoseconds{0};
+    std::atomic<std::uint64_t> pageReadNanoseconds{0};
+    std::atomic<std::uint64_t> wordReadNanoseconds{0};
+    std::atomic<std::uint64_t> hookWaitNanoseconds{0};
+    // The driver's per-shader source handle memo: captures served from it, and resolves it made.
+    std::atomic<std::uint64_t> handleHits{0};
+    std::atomic<std::uint64_t> handleMisses{0};
+};
+
+constexpr std::uint64_t WordWaitNanoseconds = 50000;
+
+bool WordwisePages() {
+    static const bool wordwise = std::getenv("APS5_NO_WORDWISE_CAPTURE") == nullptr;
+    return wordwise;
+}
+
+bool CaptureProfiled() {
+    static const bool profile = std::getenv("APS5_PROFILE_DRAW") != nullptr;
+    return profile;
+}
+
+std::uint64_t NanosecondsSince(std::chrono::steady_clock::time_point started) {
+    return static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::steady_clock::now() - started).count());
+}
+
+CaptureProfile& CaptureTotals() {
+    static CaptureProfile profile;
+    return profile;
+}
+
+std::atomic<ShaderMemory::WaitedMsProvider> waitedMsProvider{nullptr};
+
+double WaitedMs() {
+    const auto provider = waitedMsProvider.load(std::memory_order_acquire);
+    return provider != nullptr ? provider() : 0.0;
+}
+
+}
+
+void ShaderMemory::SetWaitedMsProvider(WaitedMsProvider provider) {
+    waitedMsProvider.store(provider, std::memory_order_release);
+}
+
+ShaderMemory::ShaderMemory(std::span<const ShaderRecompiler::MemoryRegion> regions, PendingWriteQuery pendingWrite, PendingWriteObserver observe, HookWaitCounter hookWaits) : pendingWrite(pendingWrite), observe(observe), hookWaits(hookWaits) {
+    const ShaderRecompiler::RequestMemoryView validated(regions);
+    for (const auto& region : regions) {
+        initial.emplace(region.guestAddress, region.bytes);
     }
 }
 
+ShaderMemory::Page& ShaderMemory::page(std::uint64_t base) {
+    const auto found = pages.find(base);
+    if (found != pages.end()) return found->second;
+    auto& page = pages[base];
+    ++CaptureTotals().pages;
+    // A page is either mapped whole or read word by word where the guest mapped less than a page,
+    // or where recorded GPU work writes into it (a whole-page read would wait for that work).
+    if (GuestMemory::Accessible(reinterpret_cast<const void*>(base), PageBytes)) {
+        if (WordwisePages() && pendingWrite != nullptr && pendingWrite(base, PageBytes, {}) != PendingWrite::None) {
+            page.wordwise = true;
+            ++CaptureTotals().pagesWordwise;
+            return page;
+        }
+        const auto started = CaptureProfiled() ? std::chrono::steady_clock::now() : std::chrono::steady_clock::time_point{};
+        GuestMemory::Read(base, std::as_writable_bytes(std::span(page.words)), sizeof(std::uint32_t));
+        if (CaptureProfiled()) CaptureTotals().pageReadNanoseconds += NanosecondsSince(started);
+        page.valid.set();
+    }
+    return page;
+}
+
 bool ShaderMemory::read(void* context, std::uint64_t address, std::uint32_t* value) {
     auto& self = *static_cast<ShaderMemory*>(context);
     if (address % sizeof(*value) != 0 || address > std::numeric_limits<std::uint64_t>::max() - sizeof(*value)) {
         throw std::runtime_error("AGC driver: invalid shader memory read address");
     }
-    const auto next = self.regions.upper_bound(address);
-    if (next != self.regions.begin()) {
-        const auto previous = std::prev(next);
-        const auto offset = address - previous->first;
-        if (offset < previous->second.size()) {
-            if (previous->second.size() - offset < sizeof(*value)) {
-                throw std::runtime_error("AGC driver: shader memory read crosses a snapshot boundary");
+    ++CaptureTotals().reads;
+    if (!self.initial.empty()) {
+        const auto next = self.initial.upper_bound(address);
+        if (next != self.initial.begin()) {
+            const auto previous = std::prev(next);
+            const auto offset = address - previous->first;
+            if (offset < previous->second.size()) {
+                if (previous->second.size() - offset < sizeof(*value)) throw std::runtime_error("AGC driver: shader memory read crosses a snapshot boundary");
+                std::memcpy(value, previous->second.data() + offset, sizeof(*value));
+                return true;
             }
-            std::memcpy(value, previous->second.data() + offset, sizeof(*value));
-            return true;
         }
+        if (next != self.initial.end() && next->first - address < sizeof(*value)) throw std::runtime_error("AGC driver: shader memory read overlaps a snapshot boundary");
     }
-    if (next != self.regions.end() && next->first - address < sizeof(*value)) {
-        throw std::runtime_error("AGC driver: shader memory read overlaps a snapshot boundary");
+    auto& page = self.page(address & ~static_cast<std::uint64_t>(PageBytes - 1));
+    const auto index = static_cast<std::size_t>((address % PageBytes) / sizeof(*value));
+    if (!page.valid.test(index)) {
+        std::uint32_t word = 0;
+        auto policy = PendingWrite::Sync;
+        if (page.wordwise) {
+            policy = self.pendingWrite(address, sizeof(word), std::as_writable_bytes(std::span(&word, 1)));
+            if (policy != PendingWrite::None && policy != PendingWrite::Sync && !GuestMemory::Accessible(reinterpret_cast<const void*>(address), sizeof(word))) policy = PendingWrite::Sync;
+        }
+        // A read is evidence only when the hook waited for unfinished GPU work across it.
+        const auto hookWaits = [&] { return self.hookWaits != nullptr ? self.hookWaits() : 0; };
+        const auto report = [&](std::uint64_t waitsBefore, bool unchanged) {
+            if (self.hookWaits == nullptr || self.hookWaits() != waitsBefore) self.observe(address, unchanged);
+            else ++CaptureTotals().observationsSkipped;
+        };
+        const auto fetchStarted = CaptureProfiled() ? std::chrono::steady_clock::now() : std::chrono::steady_clock::time_point{};
+        if (policy == PendingWrite::KnownValue || policy == PendingWrite::VerifyKnownValue) {
+            // The query stored the known dword in `word`.
+            ++CaptureTotals().wordsKnown;
+            if (policy == PendingWrite::VerifyKnownValue) {
+                std::uint32_t waited = 0;
+                const auto waitsBefore = hookWaits();
+                GuestMemory::Read(address, std::as_writable_bytes(std::span(&waited, 1)), alignof(std::uint32_t));
+                ++CaptureTotals().wordsKnownVerified;
+                if (waited != word) ++CaptureTotals().wordsKnownMismatches;
+                if (self.observe != nullptr) report(waitsBefore, waited == word);
+                word = waited;
+            }
+        } else if (policy == PendingWrite::Raw || policy == PendingWrite::VerifyRaw) {
+            std::memcpy(&word, reinterpret_cast<const void*>(address), sizeof(word));
+            ++CaptureTotals().wordsRaw;
+            if (policy == PendingWrite::VerifyRaw) {
+                std::uint32_t waited = 0;
+                const auto waitsBefore = hookWaits();
+                GuestMemory::Read(address, std::as_writable_bytes(std::span(&waited, 1)), alignof(std::uint32_t));
+                ++CaptureTotals().wordsVerified;
+                if (waited != word) ++CaptureTotals().wordMismatches;
+                if (self.observe != nullptr) report(waitsBefore, waited == word);
+                word = waited;
+            }
+        } else {
+            // On a word-wise page the bytes before the hook's wait are kept for the observer.
+            const bool observed = page.wordwise && policy == PendingWrite::Sync && self.observe != nullptr && GuestMemory::Accessible(reinterpret_cast<const void*>(address), sizeof(word));
+            std::uint32_t before = 0;
+            if (observed) std::memcpy(&before, reinterpret_cast<const void*>(address), sizeof(before));
+            const auto waitsBefore = observed ? hookWaits() : 0;
+            const auto started = page.wordwise ? std::chrono::steady_clock::now() : std::chrono::steady_clock::time_point{};
+            GuestMemory::Read(address, std::as_writable_bytes(std::span(&word, 1)), alignof(std::uint32_t));
+            if (page.wordwise) {
+                const auto nanoseconds = static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::steady_clock::now() - started).count());
+                if (nanoseconds >= WordWaitNanoseconds) {
+                    ++CaptureTotals().wordWaits;
+                    CaptureTotals().wordWaitNanoseconds += nanoseconds;
+                }
+            }
+            if (observed) report(waitsBefore, before == word);
+        }
+        if (CaptureProfiled()) CaptureTotals().wordReadNanoseconds += NanosecondsSince(fetchStarted);
+        page.words[index] = word;
+        page.valid.set(index);
     }
-    std::vector<std::byte> bytes(sizeof(*value));
-    GuestMemory::Read(address, bytes, alignof(std::uint32_t));
-    std::memcpy(value, bytes.data(), sizeof(*value));
-    self.regions.emplace(address, std::move(bytes));
+    page.read.set(index);
+    page.recent.set(index);
+    *value = page.words[index];
     return true;
 }
 
-void ShaderMemory::Capture(const ShaderRecompiler::RecompileRequest& request) {
-    const auto plan = ShaderRecompiler::GetResourcePlan(request);
-    constexpr ShaderRecompiler::ResourceMaterializer materializer;
+ShaderMemory::KnownValueCounts ShaderMemory::KnownValues() {
+    auto& totals = CaptureTotals();
+    return {totals.wordsKnown.load(std::memory_order_relaxed), totals.wordsKnownVerified.load(std::memory_order_relaxed), totals.wordsKnownMismatches.load(std::memory_order_relaxed)};
+}
+
+void ShaderMemory::CountHandleMemo(bool hit) {
+    if (!CaptureProfiled()) return;
+    (hit ? CaptureTotals().handleHits : CaptureTotals().handleMisses).fetch_add(1, std::memory_order_relaxed);
+}
+
+std::shared_ptr<const ShaderRecompiler::ResourceCapture> ShaderMemory::Capture(const ShaderRecompiler::RecompileRequest& request, const ShaderRecompiler::SourceHandle* handle) {
+    // The capture's word and page reads (through `read`) are attributed to it ([hooksync], [guestmem]).
+    const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::Capture);
+    const bool profile = CaptureProfiled();
+    auto& totals = CaptureTotals();
+    // The hook's GPU waits made anywhere inside the capture.
+    const auto waitedBefore = profile ? WaitedMs() : 0.0;
+    const auto started = std::chrono::steady_clock::now();
     ShaderRecompiler::SrtRuntime runtime;
     runtime.userData = request.context.userData;
     runtime.shaderBase = request.shader.codeAddress;
     runtime.userContext = this;
     runtime.readMemory = &read;
     runtime.readSpecializationMemory = &read;
-    ShaderRecompiler::ResourceSnapshot snapshot;
-    ShaderRecompiler::ResourceSpecialization specialization;
-    materializer.Materialize(*plan, runtime, snapshot, specialization);
+    auto capture = handle != nullptr ? ShaderRecompiler::CaptureResources(request, runtime, *handle) : ShaderRecompiler::CaptureResources(request, runtime);
+    if (profile) {
+        totals.captureNanoseconds += NanosecondsSince(started);
+        totals.resolveNanoseconds += capture->sourceNanoseconds;
+        totals.hookWaitNanoseconds += static_cast<std::uint64_t>((WaitedMs() - waitedBefore) * 1e6);
+        std::uint64_t initialBytes = 0;
+        for (const auto& [address, bytes] : initial) initialBytes += bytes.size();
+        if (++totals.captures % 500 == 0) {
+            const auto captures = static_cast<double>(totals.captures.load());
+            const auto captureNs = totals.captureNanoseconds.load();
+            const auto resolveNs = totals.resolveNanoseconds.load();
+            const auto pageNs = totals.pageReadNanoseconds.load();
+            const auto wordNs = totals.wordReadNanoseconds.load();
+            const auto specializationNs = ShaderRecompiler::ResourceMaterializer::SpecializationNanoseconds();
+            const auto hookNs = totals.hookWaitNanoseconds.load();
+            const auto walkNs = captureNs > resolveNs + pageNs + wordNs + specializationNs ? captureNs - resolveNs - pageNs - wordNs - specializationNs : 0;
+            std::fprintf(stderr, "[capture] %llu captures: %llu word reads, %llu pages fetched (%llu read word by word over pending GPU writes: %llu words waited %.2f s, %llu read raw on evidence, %llu verified with %llu mismatches, %llu served from known values, %llu hook reads not observed: no GPU wait), capture (plan lookup + materialize) %.1f s, %llu KiB registered code this capture; sub-phases (s, us per capture): source resolve %.2f/%.1f, walk %.2f/%.1f, specialization (every materialize) %.2f/%.1f, page reads %.2f/%.1f, word reads %.2f/%.1f, hook GPU waits inside %.2f/%.1f; source handle memo %llu hits, %llu resolves\n", static_cast<unsigned long long>(totals.captures.load()), static_cast<unsigned long long>(totals.reads.load()), static_cast<unsigned long long>(totals.pages.load()), static_cast<unsigned long long>(totals.pagesWordwise.load()), static_cast<unsigned long long>(totals.wordWaits.load()), totals.wordWaitNanoseconds.load() / 1e9, static_cast<unsigned long long>(totals.wordsRaw.load()), static_cast<unsigned long long>(totals.wordsVerified.load()), static_cast<unsigned long long>(totals.wordMismatches.load()), static_cast<unsigned long long>(totals.wordsKnown.load()), static_cast<unsigned long long>(totals.observationsSkipped.load()), captureNs / 1e9, static_cast<unsigned long long>(initialBytes / 1024), resolveNs / 1e9, resolveNs / captures / 1e3, walkNs / 1e9, walkNs / captures / 1e3, specializationNs / 1e9, specializationNs / captures / 1e3, pageNs / 1e9, pageNs / captures / 1e3, wordNs / 1e9, wordNs / captures / 1e3, hookNs / 1e9, hookNs / captures / 1e3, static_cast<unsigned long long>(totals.handleHits.load()), static_cast<unsigned long long>(totals.handleMisses.load()));
+        }
+    }
+    return capture;
 }
 
 std::vector<ShaderRecompiler::MemoryRegion> ShaderMemory::Regions() const {
     std::vector<ShaderRecompiler::MemoryRegion> result;
-    result.reserve(regions.size());
-    for (const auto& [address, bytes] : regions) {
-        result.push_back({address, bytes});
+    result.reserve(initial.size() + pages.size());
+    auto next = initial.begin();
+    // Both maps are ordered by address and never overlap, so a merge keeps the result sorted.
+    for (const auto& [base, page] : pages) {
+        while (next != initial.end() && next->first < base) {
+            result.push_back({next->first, next->second});
+            ++next;
+        }
+        for (std::size_t index = 0; index < PageWords;) {
+            if (!page.read.test(index)) {
+                ++index;
+                continue;
+            }
+            const auto first = index;
+            while (index < PageWords && page.read.test(index)) ++index;
+            result.push_back({base + first * sizeof(std::uint32_t), std::as_bytes(std::span(page.words).subspan(first, index - first))});
+        }
+    }
+    for (; next != initial.end(); ++next) result.push_back({next->first, next->second});
+    return result;
+}
+
+std::vector<ShaderRecompiler::MemoryRegion> ShaderMemory::TakeRecentRegions() {
+    std::vector<ShaderRecompiler::MemoryRegion> result;
+    for (auto& [base, page] : pages) {
+        if (page.recent.none()) continue;
+        for (std::size_t index = 0; index < PageWords;) {
+            if (!page.recent.test(index)) {
+                ++index;
+                continue;
+            }
+            const auto first = index;
+            while (index < PageWords && page.recent.test(index)) ++index;
+            result.push_back({base + first * sizeof(std::uint32_t), std::as_bytes(std::span(page.words).subspan(first, index - first))});
+        }
+        page.recent.reset();
     }
     return result;
 }
 
+DataWordPositionCounts DataWordPositions(std::span<const std::pair<std::uint64_t, std::uint64_t>> runs, std::span<const std::pair<std::uint32_t, std::uint64_t>> leaves, std::span<const std::uint64_t> otherReads, std::span<const std::uint32_t> words, std::span<const std::uint32_t> flattenedSrt, std::vector<std::uint32_t>& positions, std::vector<std::uint32_t>& slots) {
+    DataWordPositionCounts counts;
+    positions.clear();
+    slots.clear();
+    std::vector<std::size_t> prefix(runs.size() + 1, 0);
+    for (std::size_t i = 0; i < runs.size(); ++i) prefix[i + 1] = prefix[i] + static_cast<std::size_t>((runs[i].second - runs[i].first) / sizeof(std::uint32_t));
+    std::vector<std::pair<std::uint32_t, std::uint32_t>> found;
+    for (const auto& [slot, address] : leaves) {
+        if ((address & 3u) != 0) {
+            ++counts.unmapped;
+            continue;
+        }
+        if (std::binary_search(otherReads.begin(), otherReads.end(), address)) {
+            ++counts.aliased;
+            continue;
+        }
+        auto run = std::upper_bound(runs.begin(), runs.end(), address, [](std::uint64_t value, const std::pair<std::uint64_t, std::uint64_t>& candidate) { return value < candidate.first; });
+        if (run == runs.begin() || address + sizeof(std::uint32_t) > (run - 1)->second) {
+            ++counts.unmapped;
+            continue;
+        }
+        --run;
+        const auto position = prefix[static_cast<std::size_t>(run - runs.begin())] + static_cast<std::size_t>((address - run->first) / sizeof(std::uint32_t));
+        if (position >= words.size() || slot >= flattenedSrt.size() || words[position] != flattenedSrt[slot]) {
+            ++counts.mismatched;
+            continue;
+        }
+        found.emplace_back(static_cast<std::uint32_t>(position), slot);
+    }
+    std::sort(found.begin(), found.end());
+    positions.reserve(found.size());
+    slots.reserve(found.size());
+    for (const auto& [position, slot] : found) {
+        positions.push_back(position);
+        slots.push_back(slot);
+    }
+    return counts;
+}
+
 }

core/libs/prx/libSceAgcDriver/Execution/src/SwapchainState.cpp

@@ -1,28 +0,0 @@
-#include "prx/libSceAgcDriver/Execution/include/SwapchainState.hpp"
-#include <stdexcept>
-#include <string>
-
-namespace AgcDriver {
-
-bool SwapchainState::NeedsRecreation() const {
-    return outdated;
-}
-
-void SwapchainState::Recreated() {
-    outdated = false;
-}
-
-bool SwapchainState::ProcessResult(VkResult result, const char* operation) {
-    if (result == VK_ERROR_OUT_OF_DATE_KHR) {
-        outdated = true;
-        return false;
-    }
-    if (result == VK_SUBOPTIMAL_KHR) {
-        outdated = true;
-        return true;
-    }
-    if (result != VK_SUCCESS) throw std::runtime_error(std::string(operation) + ": Vulkan result " + std::to_string(result));
-    return true;
-}
-
-}

core/libs/prx/libSceAgcDriver/Execution/src/WorkerSampler.cpp

@@ -0,0 +1,181 @@
+#include "prx/libSceAgcDriver/Execution/include/WorkerSampler.hpp"
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#ifdef _WIN32
+#include <windows.h>
+#include <tlhelp32.h>
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <map>
+#include <mutex>
+#include <string>
+#include <thread>
+#include <vector>
+#endif
+
+namespace AgcDriver {
+
+#ifdef _WIN32
+namespace {
+
+// Debug aid: APS5_SAMPLE_WORKER=<file> samples the calling thread every millisecond and, every 20 s,
+// writes "module+offset self inclusive" lines to <file> for offline symbolization with nm.
+struct Sampler {
+    HANDLE target = nullptr;
+    std::string path;
+    std::mutex mutex;
+    std::map<std::uint64_t, std::pair<std::uint64_t, std::uint64_t>> counts;
+    std::uint64_t samples = 0;
+
+    void sample() {
+        CONTEXT context{};
+        context.ContextFlags = CONTEXT_FULL;
+        if (SuspendThread(target) == static_cast<DWORD>(-1)) return;
+        const bool captured = GetThreadContext(target, &context) != 0;
+        std::vector<std::uint64_t> frames;
+        if (captured) {
+            // Unwinding reads the target's stack while it is suspended; frames without unwind data
+            // (JIT/guest code) end the walk.
+            for (int depth = 0; depth < 12 && context.Rip != 0; ++depth) {
+                frames.push_back(context.Rip);
+                DWORD64 imageBase = 0;
+                auto* entry = RtlLookupFunctionEntry(context.Rip, &imageBase, nullptr);
+                if (entry == nullptr) break;
+                PVOID handler = nullptr;
+                DWORD64 establisher = 0;
+                RtlVirtualUnwind(UNW_FLAG_NHANDLER, imageBase, context.Rip, entry, &context, &handler, &establisher, nullptr);
+            }
+        }
+        ResumeThread(target);
+        std::lock_guard lock(mutex);
+        ++samples;
+        for (std::size_t i = 0; i < frames.size(); ++i) {
+            auto& count = counts[frames[i]];
+            if (i == 0) ++count.first;
+            ++count.second;
+        }
+    }
+
+    void write() {
+        std::lock_guard lock(mutex);
+        std::FILE* file = std::fopen(path.c_str(), "w");
+        if (file == nullptr) return;
+        std::fprintf(file, "# %llu samples\n", static_cast<unsigned long long>(samples));
+        for (const auto& [address, count] : counts) {
+            HMODULE module = nullptr;
+            char name[MAX_PATH] = "?";
+            if (GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, reinterpret_cast<LPCSTR>(address), &module)) GetModuleFileNameA(module, name, sizeof(name));
+            const char* base = std::strrchr(name, '\\');
+            std::fprintf(file, "%s 0x%llx %llu %llu\n", base ? base + 1 : name, static_cast<unsigned long long>(address - reinterpret_cast<std::uint64_t>(module)), static_cast<unsigned long long>(count.first), static_cast<unsigned long long>(count.second));
+        }
+        std::fclose(file);
+    }
+};
+
+// Debug aid: APS5_SAMPLE_THREADS=<file> samples every thread in the process every 2 ms and, every
+// 20 s, writes each busy thread's hottest frames ("module+offset self inclusive") to <file>.
+struct ProcessSampler {
+    std::string path;
+    DWORD self = 0;
+    std::map<DWORD, Sampler> threads;
+    std::uint64_t rounds = 0;
+
+    std::chrono::steady_clock::time_point enumerated{};
+
+    // The system-wide thread snapshot is slow (tens of ms); the thread list is refreshed every 5 s and
+    // the rounds in between only sample the known threads.
+    void enumerate() {
+        const HANDLE snapshot = CreateToolhelp32Snapshot(TH32CS_SNAPTHREAD, 0);
+        if (snapshot == INVALID_HANDLE_VALUE) return;
+        THREADENTRY32 entry{};
+        entry.dwSize = sizeof(entry);
+        const DWORD process = GetCurrentProcessId();
+        for (BOOL more = Thread32First(snapshot, &entry); more; more = Thread32Next(snapshot, &entry)) {
+            if (entry.th32OwnerProcessID != process || entry.th32ThreadID == self) continue;
+            auto& thread = threads[entry.th32ThreadID];
+            if (thread.target == nullptr) thread.target = OpenThread(THREAD_SUSPEND_RESUME | THREAD_GET_CONTEXT | THREAD_QUERY_INFORMATION, FALSE, entry.th32ThreadID);
+        }
+        CloseHandle(snapshot);
+        enumerated = std::chrono::steady_clock::now();
+    }
+
+    void sample() {
+        if (threads.empty() || std::chrono::steady_clock::now() - enumerated > std::chrono::seconds(5)) enumerate();
+        for (auto& [id, thread] : threads) {
+            if (thread.target != nullptr) thread.sample();
+        }
+        ++rounds;
+    }
+
+    void write() {
+        std::FILE* file = std::fopen(path.c_str(), "w");
+        if (file == nullptr) return;
+        std::fprintf(file, "# %llu rounds\n", static_cast<unsigned long long>(rounds));
+        for (auto& [id, thread] : threads) {
+            std::lock_guard lock(thread.mutex);
+            std::vector<std::pair<std::uint64_t, std::pair<std::uint64_t, std::uint64_t>>> hot(thread.counts.begin(), thread.counts.end());
+            std::sort(hot.begin(), hot.end(), [](const auto& a, const auto& b) { return a.second.second > b.second.second; });
+            std::uint64_t leaves = 0;
+            for (const auto& [address, count] : hot) leaves += count.first;
+            if (leaves == 0) continue;
+            std::fprintf(file, "thread %lu samples %llu\n", static_cast<unsigned long>(id), static_cast<unsigned long long>(thread.samples));
+            for (std::size_t i = 0; i < hot.size() && i < 40; ++i) {
+                const auto address = hot[i].first;
+                HMODULE module = nullptr;
+                char name[MAX_PATH] = "?";
+                if (GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, reinterpret_cast<LPCSTR>(address), &module)) GetModuleFileNameA(module, name, sizeof(name));
+                const char* base = std::strrchr(name, '\\');
+                std::fprintf(file, "  %s 0x%llx %llu %llu\n", base ? base + 1 : name, static_cast<unsigned long long>(address - reinterpret_cast<std::uint64_t>(module)), static_cast<unsigned long long>(hot[i].second.first), static_cast<unsigned long long>(hot[i].second.second));
+            }
+        }
+        std::fclose(file);
+    }
+};
+
+void StartProcessSampler() {
+    const char* path = std::getenv("APS5_SAMPLE_THREADS");
+    if (path == nullptr) return;
+    std::thread([path = std::string(path)] {
+        auto* sampler = new ProcessSampler();
+        sampler->path = path;
+        sampler->self = GetCurrentThreadId();
+        auto flushed = std::chrono::steady_clock::now();
+        for (;;) {
+            std::this_thread::sleep_for(std::chrono::milliseconds(2));
+            sampler->sample();
+            if (std::chrono::steady_clock::now() - flushed > std::chrono::seconds(20)) {
+                sampler->write();
+                flushed = std::chrono::steady_clock::now();
+            }
+        }
+    }).detach();
+}
+
+}
+
+void StartWorkerSampler() {
+    StartProcessSampler();
+    const char* path = std::getenv("APS5_SAMPLE_WORKER");
+    if (path == nullptr) return;
+    auto* sampler = new Sampler();
+    sampler->path = path;
+    DuplicateHandle(GetCurrentProcess(), GetCurrentThread(), GetCurrentProcess(), &sampler->target, THREAD_ALL_ACCESS, FALSE, 0);
+    std::thread([sampler] {
+        auto flushed = std::chrono::steady_clock::now();
+        for (;;) {
+            std::this_thread::sleep_for(std::chrono::milliseconds(1));
+            sampler->sample();
+            if (std::chrono::steady_clock::now() - flushed > std::chrono::seconds(20)) {
+                sampler->write();
+                flushed = std::chrono::steady_clock::now();
+            }
+        }
+    }).detach();
+}
+#else
+void StartWorkerSampler() {}
+#endif
+
+}

core/libs/prx/libSceAgcDriver/Graphics/include/BdaResources.hpp

@@ -12,9 +12,33 @@ public:
     VkDescriptorBufferInfo Table() const;
     VkDescriptorBufferInfo Fault() const;
     void CheckFault() const;
+    // APS5_PROFILE_DRAW: page tables served from the per-device cache and built anew (cumulative),
+    // and how many recent tables the cache holds right now.
+    struct TableCacheStats {
+        std::uint64_t hits = 0;
+        std::uint64_t misses = 0;
+        std::size_t held = 0;
+        // Hits served by the cached address space's serial alone (no hash, no compare; see
+        // GuestBufferMemory::CachedAddressTable), counted in `hits` too.
+        std::uint64_t spaceTables = 0;
+        // Why the most recently used table did not serve a build ([bda-table] first-differing entry
+        // class): its owners were gone (expired weak reference), or its word hash differed, split by
+        // whether the first differing range begins below 0x10000000 (a low mapping: the exe image,
+        // the system heaps) or in the guest heap above; `sameHash` is a hash match whose ranges
+        // still differed, `empty` a lookup with no entry held.
+        std::uint64_t firstExpired = 0;
+        std::uint64_t firstDiffersLow = 0;
+        std::uint64_t firstDiffersHeap = 0;
+        std::uint64_t firstSameHash = 0;
+        std::uint64_t firstEmpty = 0;
+    };
+    static TableCacheStats TableCacheCounters();
 
 private:
-    std::unique_ptr<Buffer> table;
+    // The page table is read-only to the shader, so consecutive builds mapping the same ranges to the
+    // same device addresses share one buffer while one of them is alive (see the table cache in
+    // BdaResources.cpp, which refers to it weakly).
+    std::shared_ptr<Buffer> table;
     std::unique_ptr<Buffer> fault;
     std::size_t tableBytes = 0;
 };

core/libs/prx/libSceAgcDriver/Graphics/include/BufferPool.hpp

@@ -2,10 +2,11 @@
 #define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_BUFFERPOOL_HPP
 
 #include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
-#include <array>
+#include <cstdint>
 #include <memory>
 #include <mutex>
 #include <optional>
+#include <vector>
 
 namespace AgcDriver::Graphics {
 
@@ -15,46 +16,89 @@ struct BufferAllocation {
     void* mapping;
     VkDeviceAddress address;
     VkDeviceSize allocationBytes;
+    // Size the VkBuffer was created with (see BufferPool::Capacity), not the size a user asked for.
     std::size_t bytes;
     VkBufferUsageFlags usage;
     VkMemoryPropertyFlags properties = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT;
 };
 
+// Released buffer allocations kept for reuse, since creating, binding and mapping one costs tens of
+// microseconds and a build makes several. Small requests are served by size class (the next power of
+// two), so the many differently sized data and copied-region buffers of consecutive builds hit;
+// requests of a MiB and more keep their exact size (the multi-MiB registered-range snapshots repeat
+// exactly, and rounding them would waste pinned host memory). Retained allocations are evicted least
+// recently used under a byte budget and a slot count, so a burst of small buffers between two large
+// builds no longer sweeps the large ones out. Reuse is safe because a Buffer is only released once
+// the GPU work using it completed (kept until the batch fence, or after a CommandBatch wait).
+// APS5_NO_BUFFER_CLASSES=1 matches exact sizes only, as before; APS5_BUFFER_POOL_SLOTS=64 restores
+// the old slot count (the bound applies to each tier below, so twice that many slots in all).
+//
+// The size classes and the exact-size allocations are retained in two tiers with a budget each:
+// the multi-MiB texture scratch buffers that come back from a batch (GPU-direct uploads and
+// write-backs of storage images) filled the one budget, so every one returned evicted hundreds of
+// the 256-byte copied-region buffers to fit, and those then missed on every build (81869 misses
+// and 80850 evictions per run, each a Vulkan create or destroy under the pool mutex, 3.5 s per
+// run). APS5_BUFFER_POOL_SHARED=1 keeps one tier as before.
+//
+// Device-local allocations (the detiler's scratch buffers and the staging shadows of written guest
+// buffers, see GuestBufferMemory) are retained in a third tier with a budget of their own, video
+// memory instead of pinned host memory: APS5_STAGING_POOL_MIB (default 512); 0 keeps them in the
+// two host tiers as before.
 class BufferPool {
 public:
     explicit BufferPool(const Context& context);
     ~BufferPool();
     BufferPool(const BufferPool&) = delete;
     BufferPool& operator=(const BufferPool&) = delete;
+    // The size a buffer for `bytes` is created with: its size class, or `bytes` itself when large.
+    static std::size_t Capacity(std::size_t bytes);
     std::optional<BufferAllocation> Take(std::size_t bytes, VkBufferUsageFlags usage, VkMemoryPropertyFlags properties = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT);
     void Put(const BufferAllocation& allocation) noexcept;
 
 private:
-    static constexpr std::size_t capacity = 1024;
-    static constexpr std::size_t bucketCount = 256;
-    static constexpr std::size_t none = capacity;
     struct Slot {
-        std::optional<BufferAllocation> allocation;
-        std::size_t next = none;
-        std::size_t older = none;
-        std::size_t newer = none;
+        BufferAllocation allocation;
+        std::uint64_t lastUse;
     };
-    static std::size_t bucket(std::size_t bytes, VkBufferUsageFlags usage, VkMemoryPropertyFlags properties) noexcept;
-    BufferAllocation remove(std::size_t index) noexcept;
+    // One retention tier: its slots, their bytes, the byte budget they are evicted under and its
+    // counters (APS5_PROFILE_DRAW, reported every 10 s from Take).
+    struct Tier {
+        std::vector<Slot> free;
+        VkDeviceSize retainedBytes = 0;
+        VkDeviceSize budget = 0;
+        std::uint64_t hits = 0;
+        std::uint64_t misses = 0;
+        std::uint64_t evictions = 0;
+    };
+    // The tier a buffer of `capacity` and `properties` is retained in (the large one for everything
+    // when shared; the device tier for device-local memory while it has a budget).
+    Tier& tierFor(std::size_t capacity, VkMemoryPropertyFlags properties);
+    // The device tier's budget (APS5_STAGING_POOL_MIB), read once.
+    static VkDeviceSize DeviceBudget();
     void destroy(const BufferAllocation& allocation) noexcept;
+    // Moves the tier's least recently used slot to `evicted`; the caller destroys those after
+    // releasing the mutex, so builds taking buffers on other threads do not wait behind the
+    // Vulkan destroy calls. Nothing changes when the vector cannot grow.
+    void evictOldest(Tier& tier, std::vector<BufferAllocation>& evicted);
+    // The retained-slot bound of each tier (APS5_BUFFER_POOL_SLOTS, default `defaultSlots`), read once.
+    static std::size_t MaxSlots();
     VkDevice device;
     PFN_vkUnmapMemory unmap;
     PFN_vkDestroyBuffer destroyBuffer;
     PFN_vkFreeMemory freeMemory;
     std::mutex mutex;
-    std::array<Slot, capacity> slots;
-    std::array<std::size_t, bucketCount> buckets;
-    std::array<std::size_t, capacity> freeSlots;
-    std::size_t freeCount = capacity;
-    std::size_t oldest = none;
-    std::size_t newest = none;
-    VkDeviceSize retainedBytes = 0;
+    // Not `small`/`large`: <rpcndr.h> (via <windows.h>) defines `small` as a macro.
+    Tier smallTier;
+    Tier largeTier;
+    Tier deviceTier;
+    std::uint64_t clock = 0;
     static constexpr VkDeviceSize budget = 512ull * 1024 * 1024;
+    // The small tier's own budget (512 slots of at most half a MiB each): pinned host memory the
+    // large tier's budget does not count.
+    static constexpr VkDeviceSize smallBudget = 64ull * 1024 * 1024;
+    // Requests of this size and more keep their exact size and go to the large tier.
+    static constexpr std::size_t classLimit = std::size_t{1} << 20u;
+    static constexpr std::size_t defaultSlots = 512;
 };
 
 std::shared_ptr<BufferPool> GetBufferPool(const Context& context);

core/libs/prx/libSceAgcDriver/Graphics/include/ColorTargetLayout.hpp

@@ -4,6 +4,7 @@
 #include <cstddef>
 #include <cstdint>
 #include <span>
+#include <vector>
 
 namespace AgcDriver::Graphics {
 
@@ -16,10 +17,11 @@ ColorTileMode DecodeColorTileMode(std::uint32_t attrib3);
 
 class ColorTargetLayout {
 public:
-    ColorTargetLayout(std::uint32_t width, std::uint32_t height, ColorTileMode mode);
+    ColorTargetLayout(std::uint32_t width, std::uint32_t height, ColorTileMode mode, std::uint32_t bytesPerElement = 4);
     std::size_t Bytes() const { return bytes; }
-    std::size_t LinearBytes() const { return static_cast<std::size_t>(width) * height * 4u; }
+    std::size_t LinearBytes() const { return static_cast<std::size_t>(width) * height * elementBytes; }
     std::size_t Alignment() const { return mode == ColorTileMode::Linear ? 256u : 65536u; }
+    std::uint32_t BlocksPerRow() const { return pitch / blockWidth; }
     std::size_t Offset(std::uint32_t x, std::uint32_t y) const;
     void Detile(std::span<const std::byte> source, std::span<std::byte> destination) const;
     void Tile(std::span<const std::byte> source, std::span<std::byte> destination) const;
@@ -31,6 +33,12 @@ private:
     std::uint32_t pitch;
     ColorTileMode mode;
     std::size_t bytes;
+    std::uint32_t elementBytes;
+    std::uint32_t blockWidth = 1;
+    std::uint32_t blockHeight = 1;
+    // The XOR swizzle is linear over GF(2), so a block offset is xOffsets[x] ^ yOffsets[y].
+    std::vector<std::uint32_t> xOffsets;
+    std::vector<std::uint32_t> yOffsets;
 };
 
 }

core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp

@@ -9,6 +9,7 @@
 #include <memory>
 #include <stdexcept>
 #include <string>
+#include <vector>
 
 namespace AgcDriver::Graphics {
 
@@ -19,8 +20,10 @@ class TextureCache;
 class RenderCache;
 class DrawQueue;
 class GraphicsPipelineCache;
+class Recorder;
 class DescriptorCache;
 class SamplerCache;
+class ShaderResources;
 
 inline void Require(bool condition, const std::string& reason) {
     if (!condition) throw std::runtime_error("AGC graphics: " + reason);
@@ -34,6 +37,43 @@ inline void Require(bool condition, const char* reason) {
     if (!condition) throw std::runtime_error(std::string("AGC graphics: ") + reason);
 }
 
+// Device entry points resolved once per device (VulkanDevice's State fills it after setup): the
+// loader's vkGetDeviceProcAddr is a name lookup under its global mutex per call, paid at every
+// record site otherwise. Null members (tests, APS5_NO_PROC_TABLE=1) resolve per call (Resolved).
+struct DeviceFunctions {
+    PFN_vkCmdPipelineBarrier cmdPipelineBarrier = nullptr;
+    PFN_vkCmdCopyBuffer cmdCopyBuffer = nullptr;
+    PFN_vkCmdUpdateBuffer cmdUpdateBuffer = nullptr;
+    PFN_vkCmdFillBuffer cmdFillBuffer = nullptr;
+    PFN_vkCmdBindPipeline cmdBindPipeline = nullptr;
+    PFN_vkCmdBindDescriptorSets cmdBindDescriptorSets = nullptr;
+    PFN_vkCmdPushConstants cmdPushConstants = nullptr;
+    PFN_vkCmdDispatch cmdDispatch = nullptr;
+    PFN_vkCmdDispatchIndirect cmdDispatchIndirect = nullptr;
+    PFN_vkCmdBeginRenderPass cmdBeginRenderPass = nullptr;
+    PFN_vkCmdEndRenderPass cmdEndRenderPass = nullptr;
+    PFN_vkCmdSetViewport cmdSetViewport = nullptr;
+    PFN_vkCmdSetScissor cmdSetScissor = nullptr;
+    PFN_vkCmdBindVertexBuffers cmdBindVertexBuffers = nullptr;
+    PFN_vkCmdBindIndexBuffer cmdBindIndexBuffer = nullptr;
+    PFN_vkCmdDraw cmdDraw = nullptr;
+    PFN_vkCmdDrawIndexed cmdDrawIndexed = nullptr;
+    PFN_vkCmdDrawIndirect cmdDrawIndirect = nullptr;
+    PFN_vkCmdDrawIndexedIndirect cmdDrawIndexedIndirect = nullptr;
+    PFN_vkCmdCopyBufferToImage cmdCopyBufferToImage = nullptr;
+    PFN_vkCmdCopyImageToBuffer cmdCopyImageToBuffer = nullptr;
+    PFN_vkCmdClearColorImage cmdClearColorImage = nullptr;
+    PFN_vkUpdateDescriptorSets updateDescriptorSets = nullptr;
+    PFN_vkAllocateDescriptorSets allocateDescriptorSets = nullptr;
+    PFN_vkGetFenceStatus getFenceStatus = nullptr;
+};
+
+// Per-thread count of vkGetDeviceProcAddr lookups made through Context::Function (the [vk] line).
+inline std::uint64_t& DeviceProcLookups() {
+    thread_local std::uint64_t count = 0;
+    return count;
+}
+
 struct Context {
     VkDevice device;
     VkPhysicalDevice physical;
@@ -59,20 +99,49 @@ struct Context {
     mutable std::shared_ptr<BufferPool> bufferPool;
     TextureCache* textureCache = nullptr;
     VkPipelineCache pipelineCache = VK_NULL_HANDLE;
+    bool depthClamp = false;
+    // Nonzero when VK_EXT_external_memory_host is enabled: the required host pointer alignment.
+    VkDeviceSize hostImportAlignment = 0;
     RenderCache* renderCache = nullptr;
     DrawQueue* drawQueue = nullptr;
     GraphicsPipelineCache* graphicsPipelines = nullptr;
-    mutable std::shared_ptr<DescriptorCache> descriptorCache;
-    mutable std::shared_ptr<SamplerCache> samplerCache;
+    // Batches GPU work across guest commands (see Recorder); null before the device finished setup.
+    Recorder* recorder = nullptr;
+    // Per-device caches of the immutable descriptor objects a ShaderResources build needs (set
+    // layouts, descriptor pools, samplers); null (tests) means every build makes and destroys its own.
+    DescriptorCache* descriptorCache = nullptr;
+    SamplerCache* samplerCache = nullptr;
+    // Indirect draw features of the device: records with a non-zero first instance, several records
+    // per call, and a GPU-side draw count (VK_KHR_draw_indirect_count).
+    bool drawIndirectFirstInstance = false;
+    bool multiDrawIndirect = false;
+    bool drawIndirectCount = false;
+    // The device's list of recorded dispatches whose copied written buffers await a CPU write-back
+    // (VulkanDevice's State::copiedWriters; the draw counterpart is DrawCopiedWriters): an indirect
+    // draw whose records one of them produces reads them on the CPU. Null in tests.
+    const std::vector<std::shared_ptr<ShaderResources>>* copiedWriters = nullptr;
+    // The device's resolved entry points (see DeviceFunctions); null until the device set up.
+    const DeviceFunctions* functions = nullptr;
+    // VK_EXT_descriptor_indexing with non-uniform sampled/storage image array indexing enabled
+    // (bindless image tables in graphics stages).
+    bool descriptorIndexing = false;
 
     template<typename TFunction>
     TFunction Function(const char* name) const {
         Require(deviceProc != nullptr, "missing Vulkan device function resolver");
+        ++DeviceProcLookups();
         const auto function = reinterpret_cast<TFunction>(deviceProc(device, name));
         if (function == nullptr) throw std::runtime_error(std::string("AGC graphics: missing Vulkan function: ") + name);
         return function;
     }
 
+    // The table's entry point, or a per-call lookup while the table is absent or lacks it.
+    template<typename TFunction>
+    TFunction Resolved(TFunction DeviceFunctions::*member, const char* name) const {
+        if (functions != nullptr && functions->*member != nullptr) return functions->*member;
+        return Function<TFunction>(name);
+    }
+
     std::uint32_t MemoryType(std::uint32_t mask, VkMemoryPropertyFlags flags) const {
         for (std::uint32_t i = 0; i < memory.memoryTypeCount; ++i) {
             if ((mask & (1u << i)) != 0 && (memory.memoryTypes[i].propertyFlags & flags) == flags) return i;

core/libs/prx/libSceAgcDriver/Graphics/include/DccMetadata.hpp

@@ -0,0 +1,77 @@
+#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_DCCMETADATA_HPP
+#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_DCCMETADATA_HPP
+
+#ifndef VK_NO_PROTOTYPES
+#define VK_NO_PROTOTYPES
+#endif
+#include <vulkan/vulkan.h>
+#include "prx/libSceAgcDriver/Graphics/include/GuestTextureResource.hpp"
+#include <cstddef>
+#include <cstdint>
+#include <span>
+
+namespace AgcDriver::Graphics {
+
+struct Context;
+
+// Delta color compression keeps one key byte per 256 bytes of a color surface. Surfaces are written
+// uncompressed here, so the keys that matter are the fast-clear codes a title writes into the metadata
+// (the surface then reads as a constant whatever its texels hold) and "uncompressed", which the driver
+// stores after it writes a surface so later reads see the texels.
+enum class DccKeys { Uncompressed, Clear0000, Clear0001, Clear1110, Clear1111, ClearRegister, Mixed, Unreadable };
+
+const char* DccKeysName(DccKeys keys);
+// The keys covering a surface of `surfaceBytes`, when they all agree.
+DccKeys ReadDccKeys(std::uint64_t metaAddress, std::uint64_t surfaceBytes);
+bool IsDccClear(DccKeys keys);
+// Stores "uncompressed" keys over the surface's metadata on the CPU (a guest memory write: it waits
+// for recorded GPU work that writes the keys first).
+void MarkDccUncompressed(std::uint64_t metaAddress, std::uint64_t surfaceBytes);
+// The same after a write-back recorded under GuestMemory::GpuMutex: when the keys are in host-imported
+// memory and a recorder is active, the store is a fill recorded into the open batch (ordered behind
+// the title's key-writing kernels like the write-back itself, nothing waits on the CPU) and the range
+// reads as uncompressed from ReadDccKeys while that batch is pending; otherwise the CPU store above.
+// APS5_CPU_DCC_KEYS=1 always stores on the CPU.
+void MarkDccUncompressed(const Context& context, std::uint64_t metaAddress, std::uint64_t surfaceBytes);
+// Fills `bytes` with the texel a 0000/0001/1110/1111 clear code stands for ("1" is 1.0 or the integer
+// maximum; the alpha channel is the last one in memory when alphaOnMsb, else the first, and 3-channel
+// formats have none). False when the format has no encoding here.
+bool FillDccClear(VkFormat format, DccKeys keys, bool alphaOnMsb, std::span<std::byte> bytes);
+
+// Where a color target's metadata puts alpha: the last channel in memory unless the component swap is
+// reversed (single-channel formats: only with the alternate reversed swap).
+bool DccAlphaOnMsb(VkFormat format, std::uint32_t componentSwap);
+
+// The keys of a texture's DCC surface when they fast-clear it to a value encodable here, else
+// Uncompressed (the texels are read as stored; other keys are reported once).
+DccKeys TextureClearKeys(const GuestTextureResource& resource, std::uint64_t guestBytes);
+// The last stable key scan of one surface's metadata (ProvedClearKeys): the keys read and the write
+// generation the key range was collected at before the scan. Held by the image or texture of the
+// surface, read and written under GuestMemory::GpuMutex only.
+struct DccKeyProof {
+    DccKeys keys = DccKeys::Uncompressed;
+    std::uint64_t generation = 0;
+};
+// TextureClearKeys with the scan skipped while the key range is unchanged since `proof` was taken
+// (CollectWrites over the keys, then UnchangedSince its generation), so a surface's keys are scanned
+// once per change instead of on every use. A scan becomes the proof only when it read the bytes
+// (not the pending-store memo) and no recorded work still writes them: a recorded key store stamps
+// the range at its note, before it lands, so a proof taken meanwhile would outlive the change. Every
+// other key writer stamps the range too (CPU writes through the page write watch, GPU writes at
+// their record), which makes the proof exactly as sound as UnchangedSince over the texels.
+// APS5_NO_KEY_FAST_PATH=1 scans on every call and stores nothing.
+DccKeys ProvedClearKeys(const GuestTextureResource& resource, std::uint64_t guestBytes, DccKeyProof& proof);
+bool KeyFastPath();
+// Cumulative outcomes of ProvedClearKeys: calls answered by a proof, scans made, and scans not kept.
+struct DccKeyProofCounts {
+    std::uint64_t proved;
+    std::uint64_t scanned;
+    std::uint64_t unstable;
+};
+DccKeyProofCounts KeyProofCounts();
+// A surface's texels as a read sees them: the guest bytes, or the clear value of fast-cleared keys.
+void ReadTextureSurface(const GuestTextureResource& resource, DccKeys keys, std::span<std::byte> bytes);
+
+}
+
+#endif

core/libs/prx/libSceAgcDriver/Graphics/include/DescriptorCache.hpp

@@ -1,46 +0,0 @@
-#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_DESCRIPTORCACHE_HPP
-#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_DESCRIPTORCACHE_HPP
-
-#include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
-#include <array>
-#include <mutex>
-#include <span>
-#include <vector>
-
-namespace AgcDriver::Graphics {
-
-class DescriptorAllocation {
-public:
-    DescriptorAllocation(const Context& context, std::span<const std::uint32_t> key, std::span<const VkDescriptorSetLayoutBinding> bindings, std::span<const VkDescriptorPoolSize> sizes);
-    ~DescriptorAllocation();
-    DescriptorAllocation(const DescriptorAllocation&) = delete;
-    DescriptorAllocation& operator=(const DescriptorAllocation&) = delete;
-    VkDescriptorSetLayout Layout() const { return layout; }
-    VkDescriptorSet Set() const { return set; }
-    bool Matches(std::span<const std::uint32_t> candidate) const;
-
-private:
-    void release() noexcept;
-    VkDevice device;
-    PFN_vkDestroyDescriptorPool destroyPool;
-    PFN_vkDestroyDescriptorSetLayout destroyLayout;
-    std::vector<std::uint32_t> key;
-    VkDescriptorSetLayout layout = VK_NULL_HANDLE;
-    VkDescriptorSet set = VK_NULL_HANDLE;
-    VkDescriptorPool pool = VK_NULL_HANDLE;
-};
-
-class DescriptorCache {
-public:
-    std::unique_ptr<DescriptorAllocation> Take(std::span<const std::uint32_t> key);
-    void Put(std::unique_ptr<DescriptorAllocation> allocation);
-
-private:
-    std::mutex mutex;
-    std::array<std::unique_ptr<DescriptorAllocation>, 256> available;
-    std::size_t cursor = 0;
-};
-
-}
-
-#endif

core/libs/prx/libSceAgcDriver/Graphics/include/Draw.hpp

@@ -3,10 +3,72 @@
 
 #include "prx/libSceAgcDriver/Graphics/include/Pipeline.hpp"
 #include "prx/libSceAgcDriver/Execution/include/Pm4.hpp"
+#include "prx/libSceAgcDriver/Execution/include/Recipe.hpp"
+#include <memory>
+#include <vector>
 
 namespace AgcDriver::Graphics {
 
-void Draw(const Context& context, const State& state, const Pm4::DrawParameters& draw, std::span<const CompiledShader> shaders, std::span<const GuestMemorySnapshot> snapshots = {});
+// `recipe`, when given, receives the DrawRecipe a recorded, cacheable, reusable, non-indirect draw
+// built for its draw-cache entry (design_cpu_final M8; null otherwise, and always under
+// APS5_NO_DRAW_RECIPE=1).
+void Draw(const Context& context, const State& state, const Pm4::DrawParameters& draw, std::span<const CompiledShader> shaders, std::span<const GuestMemorySnapshot> snapshots = {}, std::shared_ptr<const DrawRecipe>* recipe = nullptr);
+
+// Why DrawWithRecipe did not record from the recipe (the caller then runs Draw): the draw is not
+// recordable (no recorder, APS5_SYNC_DRAWS, APS5_DUMP_TARGETS), a resident target is gone from the
+// storage cache, the template is gone, the pipeline is gone from the pipeline store, or the
+// template's proof failed (ShaderResources::ProveCurrent; the template is removed from the
+// resource cache and kept by the batch).
+enum class DrawRecipeMiss : std::uint8_t { None, NotRecordable, TargetGone, TemplateGone, ObjectsGone, Proof, Count };
+const char* DrawRecipeMissName(DrawRecipeMiss miss);
+struct DrawRecipeOutcome {
+    bool recorded = false;
+    DrawRecipeMiss miss = DrawRecipeMiss::None;
+    ShaderResources::ProofReport proof;
+    // APS5_PROFILE_DRAW: the proof's and the record's time.
+    double proofUs = 0;
+    double recordUs = 0;
+};
+// A draw-cache hit recorded from its recipe, under GuestMemory::GpuMutex after the packet's labels
+// (design_cpu_final M8, step 8b): the validation with the recipe's stored outputs, the index and
+// vertex work as Draw's, the resident-target proof on the stored images (Refresh while Cached()),
+// the template's proof (ProveCurrent) with the alias checks repeated, the pipeline and framebuffer
+// from the recipe, then the record as Draw's (pass continued or begun, Kept, MarkGpuWrites). A
+// miss records nothing.
+DrawRecipeOutcome DrawWithRecipe(const Context& context, const State& state, const Pm4::DrawParameters& draw, std::span<const CompiledShader> shaders, std::span<const GuestMemorySnapshot> snapshots, const DrawRecipe& recipe);
+// Whether recorded draws build recipes and draw-cache hits use them (APS5_NO_DRAW_RECIPE unset).
+bool DrawRecipes();
+
+// How an indirect draw's records were read, for the [draws] line (APS5_PROFILE_DRAW): Gpu is
+// vkCmdDrawIndirect from the host import; every other value names why the CPU read them instead,
+// either decided by the driver before the draw (its patched-SGPR fold rules, the shader path, the
+// draw index, the vertex ranges, the device features, APS5_NO_GPU_INDIRECT_DRAW) or under the device
+// lock in Draw (the memory state of the records). `readMs` is the CPU read's time (its sync);
+// `rewritten` counts a GPU-side draw whose records were copied and patched with a constant.
+enum class IndirectDrawPath : std::uint8_t { Gpu, NotFolded, FetchUnknown, NonVertexPath, IndxOffset, DrawIndex, VertexRange, FeatureGap, PendingImage, PendingLabelOrCopy, NotImported, Disabled, Count };
+void CountIndirectDraw(IndirectDrawPath path, double readMs, bool rewritten = false);
+const char* IndirectDrawPathName(IndirectDrawPath path);
+
+// Draw packets that drew nothing, for the [draws] line: Nothing (an empty count, or no color writes
+// and no pixel shader), Prechecked (State.hpp's DrawRejection refused it before the decode) and
+// Thrown (the decode or the build threw); `us` is the packet's time in the driver.
+enum class DrawSkip : std::uint8_t { Nothing, Prechecked, Thrown, Count };
+void CountDrawSkip(DrawSkip kind, double us);
+
+// Recorded draws whose written guest buffers were copied (their results reach guest memory by a CPU
+// write-back when their batch completes), listed until that write-back ran. It is the draw
+// counterpart of VulkanDevice's State::copiedWriters for dispatches, which VulkanDevice.cpp consults
+// in two places: DispatchIndirect (an indirect dispatch whose arguments such a writer produced reads
+// them on the CPU, behind the write-back, instead of from the host import on the GPU) and the fill
+// HLE (a vkCmdFillBuffer over a range a listed write-back will land on must wait for it, or the
+// write-back overwrites the fill). Draws are listed here only under APS5_RECORD_COPIED_DRAWS=1,
+// which is safe only once BOTH of those checks also test this list, i.e. each extends its
+// `std::any_of(state->copiedWriters ...)` with
+// `|| std::any_of(Graphics::DrawCopiedWriters()->begin(), Graphics::DrawCopiedWriters()->end(), [&](const auto& writer) { return writer->WritesOverlap(<its range>); })`;
+// cleaner still, State::copiedWriters can become this list (one registry for both producers), after
+// which the switch can turn into APS5_NO_RECORD_COPIED_DRAWS. Until then such draws are recorded and
+// waited for at once. Read and written under GuestMemory::GpuMutex, like the device's list.
+std::shared_ptr<std::vector<std::shared_ptr<ShaderResources>>> DrawCopiedWriters();
 
 }
 

core/libs/prx/libSceAgcDriver/Graphics/include/DrawQueue.hpp

@@ -1,39 +0,0 @@
-#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_DRAWQUEUE_HPP
-#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_DRAWQUEUE_HPP
-
-#include "prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp"
-
-namespace AgcDriver::Graphics {
-
-class DrawQueue {
-public:
-    ~DrawQueue();
-    VkCommandBuffer Begin(const Context& context);
-    void Enqueue(std::shared_ptr<ShaderResources> resources, std::shared_ptr<void> storage);
-    void Flush();
-    void Resolve(std::uint64_t address, std::size_t bytes);
-    void Wait();
-    void WaitGpu();
-    void Collect();
-    void RecordMemoryBarrier(const Context& context);
-
-private:
-    struct Entry {
-        std::shared_ptr<void> storage;
-        std::shared_ptr<ShaderResources> resources;
-    };
-    struct Batch {
-        std::vector<Entry> entries;
-        std::unique_ptr<CommandBatch> commands;
-        bool hasBarrier = false;
-    };
-    void retire(Batch batch);
-    Batch recording;
-    std::vector<Batch> pending;
-    std::vector<std::unique_ptr<CommandBatch>> available;
-    std::size_t drawCount = 0;
-};
-
-}
-
-#endif

core/libs/prx/libSceAgcDriver/Graphics/include/GpuColorTransfer.hpp

@@ -13,18 +13,15 @@ public:
     GpuColorTransfer(const GpuColorTransfer&) = delete;
     GpuColorTransfer& operator=(const GpuColorTransfer&) = delete;
     void Upload(std::uint64_t address, std::uint32_t width, std::uint32_t height, ColorTileMode mode);
-    void Detile(VkCommandBuffer commands, bool swapRedBlue = false);
+    void Detile(VkCommandBuffer commands, bool swapRedBlue = false, bool tenBit = false);
     void Tile(VkCommandBuffer commands);
     void WriteBack(std::uint64_t address);
-    void WriteBackTracked(std::uint64_t address);
-    bool MatchesGuest(std::uint64_t address);
     VkBuffer LinearBuffer() const;
     RenderTarget& Target(const ColorTarget& color, bool blending);
 
 private:
-    void writeBack(std::uint64_t address, bool tracked);
     void prepare(std::uint32_t width, std::uint32_t height, ColorTileMode mode);
-    void convert(VkCommandBuffer commands, bool toTiled, bool swapRedBlue);
+    void convert(VkCommandBuffer commands, bool toTiled, bool swapRedBlue, bool tenBit = false);
     void release() noexcept;
     Context context;
     VkDescriptorSetLayout descriptorLayout = VK_NULL_HANDLE;
@@ -32,10 +29,10 @@ private:
     VkDescriptorSet descriptorSet = VK_NULL_HANDLE;
     VkPipelineLayout pipelineLayout = VK_NULL_HANDLE;
     VkPipeline pipeline = VK_NULL_HANDLE;
+    // Guest bytes in host memory, and the device-local copies the conversion shader works on.
     std::unique_ptr<Buffer> tiled;
-    std::unique_ptr<Buffer> linear;
-    std::unique_ptr<Buffer> readback;
-    std::unique_ptr<Buffer> upload;
+    std::unique_ptr<DeviceBuffer> tiledDevice;
+    std::unique_ptr<DeviceBuffer> linear;
     std::uint32_t width = 0;
     std::uint32_t height = 0;
     ColorTileMode mode = ColorTileMode::Linear;

core/libs/prx/libSceAgcDriver/Graphics/include/GraphicsPipelineCache.hpp

@@ -1,30 +0,0 @@
-#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_GRAPHICSPIPELINECACHE_HPP
-#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_GRAPHICSPIPELINECACHE_HPP
-
-#include "prx/libSceAgcDriver/Graphics/include/Pipeline.hpp"
-#include "prx/libSceAgcDriver/Graphics/include/RenderCache.hpp"
-#include <list>
-#include <map>
-#include <string>
-
-namespace AgcDriver::Graphics {
-
-class GraphicsPipelineCache {
-public:
-    explicit GraphicsPipelineCache(const Context& context) : context(context) {}
-    std::shared_ptr<Pipeline> Get(const State& state, const std::shared_ptr<ResidentColor>& target, const ShaderResources& resources, std::span<const CompiledShader> shaders);
-
-private:
-    struct Entry {
-        std::string key;
-        std::shared_ptr<ResidentColor> target;
-        std::shared_ptr<Pipeline> pipeline;
-    };
-    Context context;
-    std::list<Entry> entries;
-    std::map<std::string, std::list<Entry>::iterator> lookup;
-};
-
-}
-
-#endif

core/libs/prx/libSceAgcDriver/Graphics/include/GuestBufferMemory.hpp

@@ -5,6 +5,8 @@
 #include "BdaAbi.hpp"
 #include "prx/libc/include/GuestAllocations.hpp"
 #include <memory>
+#include <optional>
+#include <utility>
 #include <vector>
 
 namespace AgcDriver::Graphics {
@@ -14,17 +16,195 @@ struct GuestMemorySnapshot {
     std::span<const std::byte> bytes;
 };
 
+// A guest allocation imported with VK_EXT_external_memory_host, usable by the GPU in place.
+struct HostImport {
+    std::uint64_t base;
+    std::uint64_t bytes;
+    VkBuffer buffer;
+    VkDeviceMemory memory;
+    VkDeviceAddress address;
+    // Identity for the life of this import (see HostImportSerial); 0 until first asked for.
+    std::uint64_t serial = 0;
+};
+
+// The host import of the registered allocation containing [address, address + bytes), made on demand
+// (alignment and budget permitting), or null. Bytes at `address` are at `address - import->base` in
+// the import's buffer.
+const HostImport* HostImportFor(const Context& context, std::uint64_t address, std::size_t bytes);
+// Whether an existing import covers [address, address + bytes), without reconciling the imports
+// with the registry or making one (HostImportFor may take a registry lease): a hint for choices
+// made outside the device lock (a sampled texture's path, a dispatch's pre-sync); the path taken
+// re-checks with HostImportFor when it binds the import.
+bool HostImportCovers(const Context& context, std::uint64_t address, std::size_t bytes);
+// Whether a readable registered allocation contains [address, address + bytes) right now (one
+// registry lease): a storage image whose memory was freed or re-registered has nothing to store to.
+bool RegisteredReadableCovers(std::uint64_t address, std::size_t bytes);
+
+// A persistent device copy of one registered range of the main guest image (which cannot be host
+// imported); see GuestBufferMemory.cpp.
+struct ImageMirror;
+class Recorder;
+
+// Identity of the import serving [address, address + bytes): a serial unique for the life of one
+// import, 0 when no import serves the range. An equal serial later means the same VkBuffer still
+// backs the range (a re-import gets a new serial), which is what a descriptor set written against it
+// needs to stay valid (see the ShaderResources cache). With `reconcile`, imports whose registered
+// range changed are dropped first, as an upload does; without it (right after an upload, before the
+// work using the import is recorded) the imports are left as they are.
+std::uint64_t HostImportSerial(const Context& context, std::uint64_t address, std::size_t bytes, bool reconcile);
+
+// The import table's identity (the fast Revalidate): serials proved under one identity stand
+// while the table still has it, i.e. the same device, the same epoch (bumped by every retire)
+// and the registry generation it was reconciled with, which must still be the live one.
+struct HostImportsProof {
+    VkDevice device = VK_NULL_HANDLE;
+    std::uint64_t epoch = 0;
+    std::uint64_t refreshedGeneration = 0;
+};
+// Whether the table still matches `proof` (one lock); false for an empty or stale proof.
+bool HostImportsUnchanged(const Context& context, const HostImportsProof& proof);
+// The table's identity when it is reconciled with the live registry (one lock), else empty.
+HostImportsProof HostImportsIdentity(const Context& context);
+
+// Identity of the read-only image mirror serving [address, address + bytes) (the counterpart of
+// HostImportSerial for main-image ranges, which no import serves): a serial unique for the life of
+// one mirror with its top bit set, 0 when no read-only mirror whose registered range still exists
+// covers the range. A read-only mirror's bytes and VkBuffer never change, so a descriptor set written
+// against it stays valid while the serial is unchanged.
+std::uint64_t ImageMirrorSerial(const Context& context, std::uint64_t address, std::size_t bytes);
+
+// Deferred lease release (see GuestBufferMemory.cpp). An address-based build leases every readable
+// registered allocation until its write-back, which runs when its batch completed; the guest's
+// unmap/mprotect/free of a leased allocation waits for that through the registry's pin waiter, which
+// submits the recorder and finishes its batches up to the newest one recorded with a lease. SyncLeaseWork
+// says whether the build's work must instead be synced as soon as it is recorded
+// (APS5_SYNC_LEASE_DISPATCH=1, the behaviour before deferral). The driver and the draw path count
+// their choice with CountLeaseOutcome right after the lease-holding resources were kept by the open
+// batch: `batchSerial` is that batch's serial (Recorder::Submissions() + 1 under the device lock), 0
+// when synced, and the waiter targets the newest of them. Under APS5_PROFILE_DRAW CountLeaseOutcome
+// prints the [address-sync] leases line every 10 s from LeaseCounters (cumulative).
+bool SyncLeaseWork();
+void CountLeaseOutcome(bool synced, std::uint64_t batchSerial);
+struct LeaseStats {
+    std::uint64_t deferred = 0;
+    std::uint64_t synced = 0;
+    // Pin-contention waits made by guest threads (the waiter calls), how many of them finished the
+    // newest lease batch, how many had to drain the whole recorder instead (no lease batch noted),
+    // how many only dropped the cached address space (no GPU wait, see AddressSpaceStats), and
+    // their total time.
+    std::uint64_t contentionWaits = 0;
+    std::uint64_t contentionSyncs = 0;
+    std::uint64_t contentionDrains = 0;
+    std::uint64_t cacheDrops = 0;
+    double contentionMs = 0;
+};
+LeaseStats LeaseCounters();
+
+// The cached address space of address-based builds (see GuestBufferMemory.cpp): builds served by
+// the cached space, rebuilds by reason, builds whose space could not be published (an import
+// retired while it was built) and the pin waiter's drops (cumulative). `enabled` is false under
+// APS5_NO_ADDRESS_SPACE_CACHE=1, when every build takes the per-build path.
+struct AddressSpaceStats {
+    bool enabled = true;
+    std::uint64_t hits = 0;
+    std::uint64_t rebuiltFirst = 0;
+    std::uint64_t rebuiltGeneration = 0;
+    std::uint64_t rebuiltEpoch = 0;
+    std::uint64_t rebuiltDevice = 0;
+    std::uint64_t rebuiltWaiterDrop = 0;
+    std::uint64_t unpublished = 0;
+    // Builds that left the space for the per-build path: a region partially overlapping one of
+    // its regions, or an import retired under it before the upload.
+    std::uint64_t dissolvedOverlap = 0;
+    std::uint64_t dissolvedImports = 0;
+    std::uint64_t waiterDrops = 0;
+};
+AddressSpaceStats AddressSpaceCounters();
+
 class GuestBufferMemory {
 public:
     explicit GuestBufferMemory(const Context& context);
+    ~GuestBufferMemory();
     void AcquireRegistered();
-    void AddWritable(std::uint64_t address, std::size_t bytes);
+    // APS5_PROFILE_DRAW: the parts of one address-based build ([address] line every 10 s): the
+    // registry lease, the imports pass, the mirror preparation and the mirror compare (blocks and
+    // copied blocks) are timed by AcquireRegistered on the calling thread; the caller adds the
+    // snapshot compares it made after it (ShaderResources::prepareAddressBindings).
+    static void CountAddressBuild(double snapshotsUs);
+    // A guest range bound through a descriptor. Both read live guest memory at upload and bind the
+    // same way (a storage buffer); AddWritable also notes the range in Writes(), so it gets the
+    // write-back's reference copy, a write-back, the recorder's pending-write note and the
+    // direct-write marks. AddReadable is for an element the shader is proved never to store to
+    // (DescriptorBinding::bufferWritten): the CPU never has to wait for it. `atomic` marks an
+    // element the shader updates atomically (DescriptorBinding::bufferAtomic), staged in device
+    // memory whatever its size (see AllowDeviceStaging).
+    void AddWritable(std::uint64_t address, std::size_t bytes, bool atomic = false);
+    void AddReadable(std::uint64_t address, std::size_t bytes);
+    // Device-local staging of written and atomic elements inside host imports (see
+    // GuestBufferMemory.cpp): allowed only for a build whose every use records its work and then
+    // calls RecordCopyBacks (a dispatch), since a staged region's results reach guest memory by
+    // that copy alone. Call before Upload.
+    void AllowDeviceStaging() { stagingAllowed = true; }
+    // Records the copy-in of every staged region anew for another use of this upload (a resource
+    // cache hit, from ShaderResources::Revalidate, under GuestMemory::GpuMutex, once the imports
+    // were confirmed unchanged): the previous use's copy-back left the shadow behind, and the next
+    // RecordCopyBacks copies it back again. Nothing to do without staged regions.
+    void RecordStagingCopies(Recorder& recorder);
     void AddSnapshot(const GuestMemorySnapshot& snapshot);
+    // Upload is the two stages below back to back. UploadPrepare needs no device lock: it merges the
+    // regions, binds the image mirrors and host imports that already serve them (an import pointer is
+    // re-checked against the registry epoch later) and copies read-only regions. UploadFinish runs
+    // under GuestMemory::GpuMutex: it reconciles and makes imports (retiring one hands its buffer to
+    // the recorder), refreshes writable mirrors and flushes pending results (both may wait for
+    // recorded work) and copies the regions a descriptor may write, so the window between that copy
+    // and the recorder's pending-write note stays closed.
     void Upload(bool addressable);
+    void UploadPrepare(bool addressable);
+    void UploadFinish(bool addressable);
     VkDescriptorBufferInfo Descriptor(std::uint64_t address, std::size_t bytes) const;
     std::vector<ShaderRecompiler::BdaAbi::Range> AddressRanges() const;
+    // The BDA table of the cached address space when it serves this upload alone (an address-based
+    // build with no region outside it): its ranges, immutable while the space lives, and the
+    // space's serial, by which BdaResources reuses the table buffer built for it without a
+    // compare. Nothing when a region of this build is not in the space (AddressRanges then merges).
+    struct CachedTable {
+        std::uint64_t serial;
+        const std::vector<ShaderRecompiler::BdaAbi::Range>* ranges;
+    };
+    std::optional<CachedTable> CachedAddressTable() const;
+    // The registered memory an address-based build maps, shared by consecutive builds (see
+    // GuestBufferMemory.cpp): its lease and the sorted, immutable regions of the ranges served in
+    // place by imports and mirrors. Opaque outside GuestBufferMemory.cpp.
+    struct AddressSpace;
     void WriteBack();
     bool WritesOverlap(std::uint64_t address, std::size_t bytes) const;
+    // Guest ranges the shader may write through descriptors, as [begin, end).
+    const std::vector<std::pair<std::uint64_t, std::uint64_t>>& Writes() const { return writes; }
+    // Whether any written range lives in a copied buffer, so a write-back must run once the GPU is done.
+    bool HasCopiedWrites() const;
+    // Reports writes into host-imported memory (made by the GPU in place, or copied back into it by
+    // RecordCopyBacks) to the write tracking now.
+    void MarkDirectWrites() const;
+    // Records, into the recorder's open batch, the copy of every written sub-range of a region the
+    // GPU copied out of a host import (see Region::gpuCopy) back into the import: what the shader
+    // wrote lands in guest memory by the GPU, ordered after the recorded work, so the region needs
+    // no CPU write-back (HasCopiedWrites no longer counts it) and the caller notes the ranges as
+    // pending writes and marks them as MarkDirectWrites does. Called right after the work using the
+    // regions was recorded (ShaderResources::MarkGpuWrites), under GuestMemory::GpuMutex. A use that
+    // never calls it (a synchronous draw) stores the staging bytes from the CPU in WriteBack.
+    void RecordCopyBacks(Recorder& recorder);
+    // Whether registered allocations are pinned until write-back (address-based shaders): by this
+    // build's own lease, or by the cached address space it holds.
+    bool HoldsLease() const { return !lease.empty() || space != nullptr; }
+    // Every uploaded region as [begin, end) when all of them are served by host imports, in place or
+    // through a device-local staging copy of the import (nothing was copied through the CPU, so the
+    // upload can serve a later identical build), else nothing.
+    std::optional<std::vector<std::pair<std::uint64_t, std::uint64_t>>> DirectRegions() const;
+    // Every region the recorded work reads in place through a host import, as [begin, end): the
+    // regions bound in place (`direct`), an address-based build's leased heaps included. A region
+    // the GPU copies out of an import (gpuCopy) notes its read itself when the copy is recorded.
+    // For the recorder's read tracking (ShaderResources::MarkGpuWrites); nothing once committed.
+    std::vector<std::pair<std::uint64_t, std::uint64_t>> InPlaceReads() const;
 
 private:
     struct Region {
@@ -32,14 +212,106 @@ private:
         std::uint64_t end;
         bool writable;
         std::vector<std::byte> snapshot;
-        std::unique_ptr<Buffer> buffer;
+        // Shared so a recorded GPU copy into it (see gpuCopy) can keep it in the batch itself, before
+        // the caller keeps the whole resources: a throw between the two would otherwise return it to
+        // the pool under an unsubmitted copy command.
+        std::shared_ptr<Buffer> buffer;
+        // Guest bytes as uploaded; write-back only stores bytes the GPU changed.
+        std::vector<std::byte> uploaded {};
+        // Registered allocation that is imported: its bytes are read from live guest memory, not a snapshot.
+        bool hostBacked = false;
+        // Set when the region is served by an imported allocation; nothing is copied or written back.
+        const HostImport* direct = nullptr;
+        // Set when the region covers uncommitted pages: only the `backed` parts (possibly none) are guest
+        // memory; the rest reads as zeros and is never stored.
+        bool sparse = false;
+        std::vector<std::pair<std::uint64_t, std::uint64_t>> backed {};
+        // Set when the region is served by an image mirror: nothing is copied; writable mirrors are
+        // written back by comparing with the mirror's shadow. Kept alive here for recorded work.
+        std::shared_ptr<ImageMirror> mirror {};
+        // The mirror is a descriptor sub-range one found by UploadPrepare (not a lease mirror), which
+        // UploadFinish confirms still has its Range before the region binds it.
+        bool subrangeMirror = false;
+        // Set by UploadPrepare when UploadFinish still has device-lock work for the region: a copy a
+        // descriptor writes, a writable mirror's refresh, an import to reconcile or make, or the
+        // pending-results flush of an import it took.
+        bool pending = false;
+        // The region lies inside a host import but is misaligned for binding in place: `buffer` is
+        // filled by a vkCmdCopyBuffer from the import recorded into the batch (no CPU read, so no
+        // flush-hook wait), from `copySource` (the import's VkBuffer, alive until the batch
+        // completed, with the guest address of its first byte). `copiedBack` once RecordCopyBacks
+        // recorded the written sub-ranges' copies back into the import.
+        bool gpuCopy = false;
+        VkBuffer copySource = VK_NULL_HANDLE;
+        std::uint64_t copySourceBase = 0;
+        bool copiedBack = false;
+        // An element the shader updates atomically lies inside (AddWritable's `atomic`).
+        bool atomic = false;
+        // The gpuCopy buffer is a device-local staging shadow (see stagingEligible): no host
+        // mapping, so nothing is ever stored from it by the CPU, and the region is taken even when
+        // the import could bind it in place.
+        bool deviceLocal = false;
+        // The device refused the shadow (memory or allocations exhausted): the region takes the
+        // path it would take without staging, in both upload stages.
+        bool unstaged = false;
     };
 
+    // How [begin, end) lies against the space's base regions.
+    enum class BaseOverlap { None, Inside, Partial };
+    BaseOverlap baseOverlap(std::uint64_t begin, std::uint64_t end, const Region** owner) const;
+    // Copies the space's base regions into this build's own regions (today's per-build form) and
+    // drops the space: for a region that partially overlaps a base region, and for a build whose
+    // imports changed under the space (`resolve`: the direct regions are re-resolved by UploadFinish).
+    void dissolveSpace(bool resolve);
+    // The region containing `address`: this build's own regions (sorted after UploadPrepare) or
+    // the space's base regions, or null.
+    const Region* owner(std::uint64_t address) const;
+    // Builds this build's regions from a lease as the per-build path always did: `base` gets the
+    // ranges served in place (imports, mirrors), `copied` the ones copied per build.
+    struct CopiedRange {
+        std::uint64_t begin;
+        std::uint64_t end;
+        bool writable;
+    };
+    void addCopiedRange(const CopiedRange& range);
+    // The BDA table entry of an uploaded region.
+    static ShaderRecompiler::BdaAbi::Range addressRange(const Region& region);
     void validate(std::uint64_t address, std::size_t bytes) const;
+    // The region of a descriptor-bound range (AddWritable/AddReadable), committed pages only. A
+    // range inside a base region of the space adds nothing: the region serves it, as today's merge
+    // of the two did.
+    void addDescriptorRegion(std::uint64_t address, std::size_t bytes, bool atomic);
+    // Gives a region a buffer of its own with its bytes (guest memory for host-backed and writable
+    // ranges, plus the write-back's reference copy for a range a descriptor writes; else its snapshot).
+    void copyRegion(Region& region, bool addressable);
+    // Whether a region inside a host import that cannot be bound in place is copied by the GPU
+    // instead of the CPU (see Region::gpuCopy): live guest bytes, not sparse, at most the size
+    // APS5_GPU_COPY_MAX_KIB allows, and APS5_CPU_COPIES unset.
+    bool gpuCopyEligible(const Region& region) const;
+    // Whether a region inside a host import is staged in device memory instead of bound in place
+    // (see GuestBufferMemory.cpp): staging allowed, not address-based, a written element inside,
+    // and an atomic element or a size within the written-shadow window. Independent of the import,
+    // so UploadPrepare and UploadFinish decide alike.
+    bool stagingEligible(const Region& region, bool addressable) const;
+    // Records the import-to-buffer copies of the given gpuCopy regions into the open batch, with
+    // the barriers that order them after earlier recorded writes and before the shaders reading them.
+    void recordGpuCopies(std::span<Region* const> copies, bool addressable);
     Context context;
+    bool stagingAllowed = false;
     GuestAllocations::Lease lease;
+    // The cached address space this build maps through (its lease pins the ranges); `regions` then
+    // holds only the regions outside it (V#s, snapshots, ranges copied per build).
+    std::shared_ptr<const AddressSpace> space;
+    // Import registry epoch when `direct` pointers were taken at acquire time; they are reused while
+    // no import was destroyed since.
+    std::uint64_t importsEpoch = 0;
     std::vector<Region> regions;
+    // Whether `regions` is in ascending address order (true right after AcquireRegistered, whose
+    // regions follow the registry's order), so AddSnapshot can search instead of scanning.
+    bool regionsSorted = false;
     std::vector<std::pair<std::uint64_t, std::uint64_t>> writes;
+    // UploadPrepare ran (regions are frozen); `uploaded` once UploadFinish ran.
+    bool prepared = false;
     bool uploaded = false;
     bool committed = false;
 };

core/libs/prx/libSceAgcDriver/Graphics/include/GuestTextureResource.hpp

@@ -12,14 +12,32 @@ enum class TextureTileMode {
     kStandard256B,
     kStandard4KB,
     kStandard64KB,
-    RenderTarget64KB
+    // 64 KiB XOR swizzles; their address equations live in TextureSwizzleEquations.hpp.
+    kZ64KBX,
+    kS64KBX,
+    kD64KBX,
+    kR64KBX,
+    // Upstream's name for SW_64KB_R_X (tile mode 0x1b).
+    RenderTarget64KB = kR64KBX
 };
 
+// The hardware SW_MODE of an XOR swizzle tile mode, or 0 for the modes addressed without an equation table.
+constexpr std::uint32_t XorSwizzleMode(TextureTileMode mode) {
+    switch (mode) {
+        case TextureTileMode::kZ64KBX: return 24u;
+        case TextureTileMode::kS64KBX: return 25u;
+        case TextureTileMode::kD64KBX: return 26u;
+        case TextureTileMode::kR64KBX: return 27u;
+        default: return 0u;
+    }
+}
+
 enum class TextureDimension {
     k1D,
     k2D,
     k2DArray,
-    kCube
+    kCube,
+    k3D
 };
 
 struct GuestTextureResource {
@@ -37,6 +55,11 @@ struct GuestTextureResource {
     std::uint8_t dstSelY;
     std::uint8_t dstSelZ;
     std::uint8_t dstSelW;
+    // Last mip level the view exposes; the surface itself holds mipCount levels.
+    std::uint32_t lastLevel = 0;
+    // DCC metadata of a compressed surface, or 0 (see DccMetadata.hpp).
+    std::uint64_t dccAddress = 0;
+    bool dccAlphaOnMsb = false;
 };
 
 GuestTextureResource DecodeTextureResource(std::span<const std::uint32_t> words);

core/libs/prx/libSceAgcDriver/Graphics/include/Pipeline.hpp

@@ -2,31 +2,93 @@
 #define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_PIPELINE_HPP
 
 #include "prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp"
+#include <set>
 
 namespace AgcDriver::Graphics {
 
+struct VertexInputLayout;
+
+// The attachments of one render pass instance. Work recorded with it references the handle until
+// the batch completes, so a recorded draw keeps the object (Recorder::Keep) like its pipeline.
+class Framebuffer {
+public:
+    Framebuffer(const Context& context, VkRenderPass renderPass, std::span<const VkImageView> targets, VkExtent2D extent);
+    ~Framebuffer();
+    Framebuffer(const Framebuffer&) = delete;
+    Framebuffer& operator=(const Framebuffer&) = delete;
+    VkFramebuffer Handle() const { return framebuffer; }
+    // Forgets the handle without destroying it (the device it belongs to is already gone).
+    void Abandon() noexcept;
+
+private:
+    Context context;
+    VkFramebuffer framebuffer = VK_NULL_HANDLE;
+};
+
+// The shader modules, layout, render pass and VkPipeline of one draw configuration. Viewport and
+// scissor are dynamic state set at Begin, so pipelines are shared by draws that differ only there
+// (see CachedPipeline).
 class Pipeline {
 public:
-    Pipeline(const Context& context, const State& state, const RenderTarget* target, const ShaderResources& resources, std::span<const CompiledShader> shaders);
+    // `state` carries the blend states with the outputs the pixel shader lacks already masked and
+    // `vertexInput` the layout of the draw's vertex descriptors; the shaders were validated by the
+    // caller (ValidateShaders). `attachmentLayout` is the layout the color attachments are in
+    // before, during and after the pass (GENERAL for resident targets, which then need no
+    // transitions).
+    Pipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const ShaderResources& resources, std::span<const CompiledShader> shaders, VkImageLayout attachmentLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL);
     ~Pipeline();
     Pipeline(const Pipeline&) = delete;
     Pipeline& operator=(const Pipeline&) = delete;
     VkPipelineLayout Layout() const;
-    void Begin(VkCommandBuffer commands, VkExtent2D extent) const;
+    // The framebuffer of these attachment views. A resident target's view (`owners[i]` set) is stable
+    // while its StorageTexture lives, so framebuffers made of resident views are kept in the pipeline
+    // and reused when every owner is still the same object; any other set is built per call.
+    std::shared_ptr<Framebuffer> AcquireFramebuffer(std::span<const VkImageView> targets, std::span<const std::shared_ptr<StorageTexture>> owners, VkExtent2D extent);
+    // Begins the render pass on the framebuffer, binds the pipeline and sets viewport and scissor.
+    void Begin(VkCommandBuffer commands, const Framebuffer& framebuffer, VkExtent2D extent, const VkViewport& viewport, const VkRect2D& scissor) const;
+    // The same inside a render pass another pipeline of the same attachments began (compatible by
+    // construction: the attachment formats alone decide).
+    void Continue(VkCommandBuffer commands, const VkViewport& viewport, const VkRect2D& scissor) const;
     void PushConstants(VkCommandBuffer commands, std::span<const CompiledShader> shaders) const;
+    // The same from an assembled block (a draw recipe's, made by AssemblePushConstants once).
+    void PushConstants(VkCommandBuffer commands, VkShaderStageFlags stages, std::span<const std::byte, PipelinePushConstantBytes> bytes) const;
+    // Forgets the Vulkan objects without destroying them: for entries of a device that is already gone.
+    void Abandon() noexcept;
 
 private:
+    struct CachedFramebuffer {
+        std::vector<VkImageView> views;
+        std::vector<std::weak_ptr<StorageTexture>> owners;
+        VkExtent2D extent;
+        std::shared_ptr<Framebuffer> framebuffer;
+    };
     void release() noexcept;
     Context context;
     std::vector<VkShaderModule> _modules;
     VkPipelineLayout layout = VK_NULL_HANDLE;
     VkRenderPass renderPass = VK_NULL_HANDLE;
-    VkFramebuffer framebuffer = VK_NULL_HANDLE;
     VkPipeline pipeline = VK_NULL_HANDLE;
+    std::size_t attachments = 0;
+    std::vector<CachedFramebuffer> framebuffers;
 };
 
+// The pipeline for this configuration, shared across draws and frames: keyed by the stages' variant
+// ids (the rect-list tessellation pair by the vertex and fragment ids that generate it), the vertex
+// input layout, the descriptor set layout key, push constant stages and every pipeline state field.
+// A configuration whose stage has no variant id gets a private pipeline. Entries are LRU-bounded and
+// only evicted once no recorded draw holds them. Debug aid: APS5_NO_PIPELINE_CACHE=1 builds one per
+// draw as before.
+std::shared_ptr<Pipeline> CachedPipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const ShaderResources& resources, std::span<const CompiledShader> shaders, VkImageLayout attachmentLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL);
+// Destroys the cached pipelines of a device; to be called before the device goes away. Without it,
+// entries of a gone device are recognised by their buffer pool (made and reset with the device, so
+// it tells device instances apart when the loader reuses a VkDevice handle) and forgotten unused.
+void ClearCachedPipelines(VkDevice device);
+// Device limit checks of the viewport, which is dynamic state and so no longer checked by Pipeline.
+void ValidateViewport(const Context& context, const VkViewport& viewport);
+
 void ValidateShaderPair(const ShaderRecompiler::RecompileResult& vertex, const ShaderRecompiler::RecompileResult& fragment);
-void ValidateShaders(std::span<const CompiledShader> shaders, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric);
+// Returns the color attachment locations the pixel shader writes.
+std::set<std::uint32_t> ValidateShaders(std::span<const CompiledShader> shaders, const State& state, const VkPhysicalDeviceSubgroupProperties& subgroup, bool fragmentShaderBarycentric, bool descriptorIndexing = false);
 
 }
 

core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp

@@ -0,0 +1,688 @@
+#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_RECORDER_HPP
+#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_RECORDER_HPP
+
+#include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
+#include <atomic>
+#include <chrono>
+#include <cstddef>
+#include <cstdint>
+#include <array>
+#include <deque>
+#include <mutex>
+#include <functional>
+#include <map>
+#include <memory>
+#include <optional>
+#include <span>
+#include <tuple>
+#include <utility>
+#include <vector>
+
+namespace AgcDriver::Graphics {
+
+// Accumulates GPU work across guest commands so the CPU does not wait for each one. Dispatches and the
+// copies that feed them record into one open batch; Submit sends it to the queue without waiting and
+// Sync waits for every batch, then runs its completion actions (write-backs) in order. Objects handed
+// to Keep live until the batch that recorded them completed. Guest memory ranges the recorded work
+// will write are noted, so a CPU access to such a range (the GuestMemory flush hook) syncs first.
+// Every call happens under GuestMemory::GpuMutex, except WaitSerial and the static lock-free readers
+// below. The noted ranges are also published as an immutable snapshot (rebuilt under the mutex at
+// every note and after a batch's completions ran) that the flush hook reads without the mutex, so
+// accesses overlapping nothing never wait behind device work.
+//
+// Submissions are numbered (serials) and, when the device has timeline semaphores, every Submit
+// signals a timeline semaphore with its serial: a thread can then wait for "everything submitted
+// up to serial S" WITHOUT the mutex (WaitSerial), because the monotonic value cannot be reset or
+// recycled the way the pooled batch fences are, and afterwards run the completions under the mutex
+// (FinishUpTo). Without timeline semaphores callers use the locked Sync as before.
+class Recorder {
+public:
+    // `timelineSemaphores`: the device enabled VK_KHR_timeline_semaphore (WaitSerial is usable).
+    explicit Recorder(const Context& context, bool timelineSemaphores = false);
+    ~Recorder();
+    Recorder(const Recorder&) = delete;
+    Recorder& operator=(const Recorder&) = delete;
+
+    // The open batch's command buffer, starting a batch when none is open. `coveredAccess`, when
+    // given, receives the destination access mask of the last recorded command's trailing barrier
+    // when that barrier had ALL_COMMANDS as its destination stage (MarkCovered), else 0: every
+    // write recorded so far is then available and visible to those accesses of any later stage,
+    // so a dispatch, fill or copy whose leading barrier asks for a subset of them may skip it
+    // (the [barriers] line counts the merges). The call clears the mask (whatever the caller
+    // records is assumed uncovered until it marks); a new batch starts uncovered.
+    VkCommandBuffer Commands(VkAccessFlags* coveredAccess = nullptr);
+    // After a trailing barrier with ALL_COMMANDS as its destination stage: `access` is its
+    // destination access mask.
+    void MarkCovered(VkAccessFlags access);
+    void MarkShaderReadsCovered() { MarkCovered(VK_ACCESS_SHADER_READ_BIT | VK_ACCESS_SHADER_WRITE_BIT); }
+    // A recorded draw's render pass (Draw.cpp) is left open after the draw: the next draw of the
+    // same attachments (`key`: the views and the extent) continues it when nothing was recorded in
+    // between and the earlier draw allowed it (`continuable`: it wrote nothing but its
+    // attachments, so no barrier is owed inside the pass), and anything else recorded first ends
+    // it (vkCmdEndRenderPass, one trailing barrier for the pass, the end of the draw class range
+    // `timing`): Commands(), RecordStore and Submit end it; CommandsInRenderPass hands a
+    // continuing draw the command buffer without ending it.
+    bool ContinuesRenderPass(std::uint64_t key) const;
+    VkCommandBuffer CommandsInRenderPass();
+    void LeaveRenderPassOpen(std::uint64_t key, std::uint32_t timing, bool continuable);
+    // DCC "uncompressed" key stores (DccMetadata.cpp StoreUncompressedOnGpu): queued on the open
+    // batch and recorded as one run (one barrier pair for every queued fill) at Submit, before a
+    // label store (RecordStore), or before a command that writes or reads a queued range (the
+    // fill must precede a later key writer or reader in the batch: FlushKeyStores by the callers
+    // that know their ranges). `seed` (kept) supplies an
+    // unaligned head or tail; [begin, end) is the guest range. Debug aid: APS5_DCC_KEYS_EACH=1
+    // records every store at once, as before (QueueKeyStore then records and returns).
+    void QueueKeyStore(VkBuffer buffer, VkDeviceSize first, VkDeviceSize last, std::shared_ptr<void> seed, std::uint64_t begin, std::uint64_t end);
+    bool HasQueuedKeyStores() const { return open != nullptr && !open->keyStores.empty(); }
+    bool QueuedKeyStoreOverlaps(std::uint64_t address, std::size_t bytes) const;
+    // Whether `overlaps(begin, end)` holds for a queued store's range.
+    bool AnyQueuedKeyStore(const std::function<bool(std::uint64_t, std::uint64_t)>& overlaps) const;
+    void FlushKeyStores();
+    void FlushKeyStoresOverlapping(std::uint64_t address, std::size_t bytes) { if (HasQueuedKeyStores() && QueuedKeyStoreOverlaps(address, bytes)) FlushKeyStores(); }
+    bool Recording() const { return open != nullptr; }
+    bool Idle() const { return open == nullptr && inFlight.empty(); }
+    // Whether recorded work still has completion actions (write-backs the CPU must see) to run.
+    bool HasCompletions() const;
+    // The object lives at least until the batch open now completed; it is then destroyed AFTER the
+    // finishing thread released GuestMemory::GpuMutex, on a release thread of its own (never under
+    // a hold, see Recorder.cpp ReleaseDeferredKeeps; the finishing thread destroys it itself when
+    // the thread's queue is full, APS5_RELEASE_QUEUE_MAX batches, or with APS5_RELEASE_ON_UNLOCK=1;
+    // APS5_RELEASE_UNDER_LOCK=1 destroys it in the reap as before), so its destructor must need
+    // neither the mutex nor the recorder nor a particular thread. ~Recorder joins the release
+    // thread and waits for every release in progress before the device goes.
+    void Keep(std::shared_ptr<void> object);
+    void OnComplete(std::function<void()> action);
+    void NotePendingWrite(std::uint64_t address, std::size_t bytes);
+    // Notes several [begin, end) ranges and publishes the snapshot once (a dispatch writes many buffers).
+    void NotePendingWrites(std::span<const std::pair<std::uint64_t, std::uint64_t>> ranges);
+    bool PendingWriteOverlaps(std::uint64_t address, std::size_t bytes) const;
+    // Whether no unfinished batch writes the range any more: no open-batch overlap, and every
+    // overlapping in-flight batch's fence has signaled (its in-place GPU stores are final in host
+    // memory; its completion stores are the caller's business, see VulkanDevice::CopyBuffer).
+    bool PendingWriteSettled(std::uint64_t address, std::size_t bytes) const;
+    // Batch read tracking: the guest ranges the recorded work reads IN PLACE through a host import
+    // and the GPU has not executed yet (a V# element bound in place, a region the GPU copies out of
+    // an import, an address-based build's leased heaps, indirect arguments, a GPU-direct storage
+    // upload, the copy HLE's source). A CPU store into such a range before the batch ran would be
+    // seen by the reader early (WAR); VulkanDevice::CopyBuffer refuses its CPU path on a hit. Noted
+    // on the open batch, no snapshot: the only reader holds the mutex. `kind` names the reader for
+    // the [recorder] hit counters. APS5_COPY_READ_TRACKING=0 notes nothing (ReadTracking() is then
+    // false and the CPU copy falls back to Idle()).
+    enum class ReadKind : std::uint8_t { DispatchElement = 0, GpuCopy, AddressBased, Indirect, StorageUpload, CopySource, Count };
+    static bool ReadTracking();
+    void NotePendingRead(std::uint64_t address, std::size_t bytes, ReadKind kind);
+    void NotePendingReads(std::span<const std::pair<std::uint64_t, std::uint64_t>> ranges, ReadKind kind);
+    // Whether an unexecuted batch (open, or in flight with its fence unsignaled) reads the range in
+    // place. A hit in an in-flight batch whose fence signaled meanwhile is a miss (one status
+    // query per hit). `ignoreSignaled` false: every in-flight batch counts, whatever its fence.
+    bool PendingReadOverlaps(std::uint64_t address, std::size_t bytes, bool ignoreSignaled = true) const;
+    // The newest batch whose noted read overlaps the range (for the [copyhle] trace and the verify
+    // switch): its serial (the open batch reports the serial it will get), recording queue, reader
+    // kind, and whether its fence has signaled. Nullopt: no overlap.
+    struct PendingReadInfo {
+        std::uint64_t serial;
+        std::uint32_t queue;
+        ReadKind kind;
+        bool open;
+        bool signaled;
+    };
+    std::optional<PendingReadInfo> DescribePendingRead(std::uint64_t address, std::size_t bytes) const;
+    // Counters of the read tracking (cumulative, for the [recorder] line): ranges noted, overlap
+    // queries, hits by reader kind, and hits ignored because the reader's fence had signaled.
+    struct ReadStatistics {
+        std::uint64_t noted, queries, staleIgnored;
+        std::array<std::uint64_t, static_cast<std::size_t>(ReadKind::Count)> hits;
+    };
+    static ReadStatistics ReadCounts();
+    // Whether the OPEN (unsubmitted) batch writes the range: a wait on such a range must submit it.
+    bool OpenWriteOverlaps(std::uint64_t address, std::size_t bytes) const;
+    // What SyncThrough(address, bytes) would wait for, for the [hooksync] attribution: the newest
+    // batch whose noted range overlaps the access (the open batch counts first, with the serial it
+    // will get), whether that batch's fence has already signaled (its GPU work is done, so the
+    // access's bytes are already final; never for the open batch), that noted range, and how many
+    // batches the wait finishes. Nullopt: no overlap.
+    struct PendingWriteInfo {
+        std::uint64_t serial;
+        bool open;
+        bool signaled;
+        std::uint64_t rangeBegin;
+        std::uint64_t rangeEnd;
+        std::size_t batchesToFinish;
+    };
+    std::optional<PendingWriteInfo> DescribePendingWrite(std::uint64_t address, std::size_t bytes) const;
+    // Ends and submits the open batch without waiting.
+    void Submit();
+    // Submits the open batch and returns the serial of the newest submitted batch (0 when nothing was
+    // ever submitted); WaitSerial(serial) then covers all recorded work.
+    std::uint64_t SubmitAndEpoch();
+    bool HasTimeline() const { return timeline != VK_NULL_HANDLE; }
+    // Waits until every batch up to `serial` completed on the GPU. Called WITHOUT GuestMemory::GpuMutex:
+    // it touches only the immutable timeline semaphore (the caller keeps the device alive). Requires
+    // HasTimeline(); the debug switch APS5_NO_TIMELINE makes the device create the recorder without one.
+    void WaitSerial(std::uint64_t serial);
+    // Under the mutex: finishes (completions, release) the in-flight batches up to `serial`, from the
+    // front only; tolerates batches another thread finished meanwhile. Later batches stay in flight.
+    void FinishUpTo(std::uint64_t serial);
+    // Submits and waits for every batch, running completions in order.
+    void Sync();
+    // Waits only for the batches up to the newest one that writes the range (submitting the open
+    // batch when it is that one); later batches stay in flight. Fences of one queue signal in
+    // submission order, so completions still run in order. Debug aid: APS5_NO_SYNC_THROUGH=1 syncs all.
+    // `waitUnlocked` (the flush hook): when the caller's GpuMutex acquisition is the outermost one
+    // on this thread and the device has a timeline, the GPU wait runs WITHOUT the mutex (released
+    // and retaken here; the caller must hold nothing else that orders after it) and only the
+    // completions run under it, so other queues do not queue behind a CPU read's wait. Inside a
+    // batch's completion action no wait is made at all: every batch still in flight was recorded
+    // after the completing one, and its store is that batch's in-order write (see Recorder.cpp).
+    void SyncThrough(std::uint64_t address, std::size_t bytes, bool waitUnlocked = false);
+    // Completes batches that already finished; returns whether nothing is in flight.
+    bool Reap();
+    std::uint64_t Submissions() const { return submissions; }
+    // Batches submitted and not yet finished (what a full Sync() waits for); under GuestMemory::GpuMutex.
+    std::size_t InFlightBatches() const { return inFlight.size(); }
+    // Reaps that retired at least one batch so far (the [recorder] line's 'with work' count).
+    static std::uint64_t ReapsWithWork();
+
+    // A store of `bytes` into a host import (a label, or a COPY_DATA/DMA_DATA/DUMP_CONST_RAM store
+    // of up to 64 KiB; `address` is its guest address) recorded into the open batch: ordered
+    // after everything recorded before it and visible to the host and to later work. The stores
+    // of a batch form one run recorded as the batch's last commands at Submit: one barrier
+    // (everything so far -> transfer and host: every result of the batch reaches host memory
+    // before any label of it lands, the order the title's pollers rely on), the
+    // vkCmdUpdateBuffers, one transfer -> host barrier. A later command of the batch that writes
+    // over a queued store's bytes or reads them in place must find the store recorded (the
+    // callers that know their ranges call FlushStoresOverlapping, or FlushStores when they do
+    // not, before recording): the run is then recorded in place, with a trailing barrier toward
+    // the later work, and the next store starts a new run. Within a run a store contiguous with
+    // the last queued one joins its vkCmdUpdateBuffer, one inside it replaces its bytes (the
+    // later store wins, as in program order), and one overlapping a store queued earlier gets a
+    // transfer-to-transfer barrier first (WAW). Debug aids: APS5_LABEL_RUNS_INLINE=1 records the
+    // stores at once and closes the run at the next command (Commands()) or at Submit, as
+    // before; APS5_NO_LABEL_RUNS=1 gives every store barriers of its own.
+    void RecordStore(VkBuffer buffer, VkDeviceSize offset, std::span<const std::byte> bytes, std::uint64_t address);
+    bool HasQueuedStores() const { return open != nullptr && !open->run.queued.empty(); }
+    bool QueuedStoreOverlaps(std::uint64_t address, std::size_t bytes) const;
+    // Whether `overlaps(begin, end)` holds for a queued store's guest range.
+    bool AnyQueuedStore(const std::function<bool(std::uint64_t, std::uint64_t)>& overlaps) const;
+    void FlushStores();
+    void FlushStoresOverlapping(std::uint64_t address, std::size_t bytes) { if (HasQueuedStores() && QueuedStoreOverlaps(address, bytes)) FlushStores(); }
+
+    // Pending-label table. A GPU label (RELEASE_MEM/WRITE_DATA recorded as a store into the batch)
+    // is noted per 4-byte dword with the value it will store and a record-order stamp (the driver's
+    // event serial, taken when the label is recorded). A WAIT_REG_MEM whose submission was received
+    // (stamped) BEFORE the label was recorded may take the value from the table instead of waiting
+    // for the GPU: any CPU store the game ordered before that submit call precedes the label in
+    // program order exactly as on hardware, and later recorded work follows the label in queue
+    // order. Entries recorded before the submission (the previous frame's label at the same address)
+    // are trusted only under the late rule below (APS5_LABEL_TRUST_LATE=0 never trusts them).
+    // Entries leave the table with their batch (a memory bound only). The note also enters the
+    // label's range as a pending write of the open batch (the flush hook syncs CPU reads), marked
+    // as the label's own so it does not count as an overwrite of the entry.
+    void NoteLabel(std::uint64_t address, std::span<const std::byte> bytes, std::uint64_t stamp, std::uint32_t queue);
+    // Late rule: an entry whose stamp is not newer than the wait's submission is still the value
+    // memory will hold, unless a CPU store touched the dword since it was recorded (the table
+    // cannot see CPU stores; the title recycles label slots by CPU writes) or a later recorded
+    // non-label GPU write covers it. The table refuses a queued entry, one flagged `overwritten`
+    // (noteWrite), and one whose record group has not closed (generation 0); the caller checks the
+    // CPU side with GuestMemory::UnchangedSinceCollected against the hit's generation. Labels are
+    // noted in groups (a worker's deferred labels recorded back to back, or one immediate label
+    // or store): CloseLabelGroup, called by the recording thread after the group's last
+    // MarkWritten, stamps the group's entries with the write tracker's generation at that point;
+    // per-entry capture would fail, as the next label of the group stamps the shared 64 KiB block
+    // newer. A generation of 0 leaves the entries unclosed (a failed record), as does a batch
+    // already submitted at the close (its value may have landed before the generation was taken).
+    static void CloseLabelGroup(std::uint64_t trackerGeneration);
+    static bool LateTrust();
+    struct LabelHit {
+        std::uint64_t value;
+        // The recording queue of the first dword, and its record stamp.
+        std::uint32_t queue;
+        std::uint64_t stamp;
+        // Taken under the late rule: `generation` is the oldest group generation of the dwords.
+        bool late;
+        std::uint64_t generation;
+    };
+    // Why a lookup refused a late candidate (an entry with stamp <= afterStamp), for the trace.
+    // BehindCompletion replaces the other reasons when the refused entry's value reaches memory
+    // only through a completion action of its batch (AfterCompletions: the GPU has no view of the
+    // range, or a write-back overwrote the GPU's store): a poller must reap that batch to converge.
+    enum class LabelRefusal : std::uint8_t { None = 0, TrustOff, Queued, Overwritten, Unclosed, BehindCompletion };
+    // Per calling thread (lookups run on the waiting worker): late candidates seen, in both modes,
+    // and the refusals by reason ([packets] line deltas).
+    struct LateStatistics {
+        std::uint64_t candidates, queued, overwritten, unclosed;
+    };
+    static LateStatistics LateCounts();
+    // The 4- or 8-byte value the table holds for `address` when every dword is present with a stamp
+    // newer than `afterStamp`, or trustable under the late rule.
+    std::optional<LabelHit> PendingLabel(std::uint64_t address, std::size_t bytes, std::uint64_t afterStamp, LabelRefusal* refusal = nullptr) const;
+    std::size_t PendingLabels() const;
+    // Whether any tracked label dword (recorded, or queued by a worker and not recorded yet) lies
+    // inside [address, address + bytes): a range query for large ranges (a fill of megabytes),
+    // where a per-dword lookup would not do.
+    bool PendingLabelIn(std::uint64_t address, std::size_t bytes) const;
+    // PendingLabel of the active recorder WITHOUT GuestMemory::GpuMutex: the table has a small mutex
+    // of its own (every mutation holds both), so a WAIT_REG_MEM consults it without queueing behind
+    // device work. Nothing is done under the table mutex but the lookup (it never takes the GPU mutex).
+    static std::optional<LabelHit> LookupLabel(std::uint64_t address, std::size_t bytes, std::uint64_t afterStamp, LabelRefusal* refusal = nullptr);
+    // The value alone, for callers asking whether any label is pending in a dword (afterStamp 0).
+    static std::optional<std::uint64_t> LookupLabelValue(std::uint64_t address, std::size_t bytes, std::uint64_t afterStamp);
+    // A label a queue worker decoded but has not recorded yet (Driver.cpp DeferredLabels): it
+    // enters the table with no batch, and only a lookup made on the noting thread (a WAIT_REG_MEM
+    // of the same queue) takes its value, since that queue's later work follows the label in queue
+    // order whenever the group is recorded; other queues and the CPU see it once it is recorded.
+    // The range is also kept per thread, so a CPU access of it by the same worker (the flush hook)
+    // has the group recorded first through the function SetQueuedLabelRecorder installed. Takes
+    // the table mutex only. The queued entries live apart from the recorded ones: a recorded
+    // label of the same dword (an older store still pending on the GPU) keeps serving the other
+    // queues' waits and the driver's pending-label checks until the queued one is recorded.
+    // Debug aid: APS5_NO_SEPARATE_QUEUED_LABELS=1 lets a queued entry replace the recorded one.
+    static void NoteQueuedLabel(std::uint64_t address, std::span<const std::byte> bytes, std::uint64_t stamp, std::uint32_t queue);
+    // After the calling worker recorded (or dropped) its queued labels: its ranges are cleared and
+    // the entries of its dwords still without a batch (stored by the CPU, or dropped) leave the table.
+    static void ForgetQueuedLabels();
+    // The function the flush hook calls, on the accessing thread (which may hold the GPU mutex),
+    // when an access overlaps one of that thread's queued labels; it records them. Set once.
+    static void SetQueuedLabelRecorder(void (*recorder)());
+    // Whether the lock-free pending-write snapshot (open, in-flight and finishing batches) overlaps
+    // the range: false means no recorded work writes it, so a wait on it has nothing to submit.
+    static bool SnapshotWriteOverlaps(std::uint64_t address, std::size_t bytes);
+    // The snapshot itself (the sorted, merged union of the pending ranges; null when none), for a
+    // reader that tests many ranges against one loaded snapshot: one atomic shared_ptr load per
+    // validation instead of one per run, and every test sees the same snapshot (design13 R1's p0).
+    using WriteRanges = std::vector<std::pair<std::uint64_t, std::uint64_t>>;
+    static std::shared_ptr<const WriteRanges> PendingWriteSnapshot();
+    static bool SnapshotOverlaps(const WriteRanges* snapshot, std::uint64_t address, std::size_t bytes);
+    // A label behind pending completions (CPU write-backs the label must not precede): stored to
+    // guest memory by a completion action appended to the open batch, or to the newest in-flight
+    // batch when none is open (no empty submission), so it runs after every earlier write-back in
+    // submission order. The range is noted as a pending write of that batch so CPU reads through the
+    // flush hook still sync, and the label enters the table like a GPU one.
+    // `storedOnGpu`: the same bytes were also recorded into the open batch (a host import), so the
+    // completion re-stores them only when a CPU write-back noted by NoteWrittenBack overlapped the
+    // range since; the plain store of a label the GPU has no view of always runs. An unconditional
+    // completion store would land on memory the game may have reused by then (a stale label value
+    // over a fresh command buffer). Debug aid: APS5_LABEL_STORE_ALWAYS=1 stores unconditionally.
+    void AfterCompletions(std::uint64_t address, std::span<const std::byte> bytes, std::uint64_t stamp, std::uint32_t queue, bool storedOnGpu);
+    // A CPU store of GPU results into guest memory (GuestBufferMemory::WriteBack): recorded so a
+    // completion label store can tell whether its bytes were overwritten. Under GuestMemory::
+    // GpuMutex (every write-back runs inside finish() or a synchronous draw's wait): the first
+    // write-back overlapping a completion label stored on the GPU is what makes that label's
+    // completion store necessary, so it is counted into PendingCompletionLabels here rather than
+    // at registration (APS5_COUNT_ALL_COMPLETION_LABELS=1 counts every one at registration, as
+    // before). A write-back over any tracked label dword is counted on the [recorder] line (must
+    // stay 0: it would mean a CPU store landed over a label recorded on the GPU).
+    static void NoteWrittenBack(std::uint64_t address, std::size_t bytes);
+
+    // Lock-free readers for the queue workers' packet loops and WAIT_REG_MEM polls (values of the
+    // active recorder; a torn-down recorder never publishes):
+    // when the open batch received its first label (nullopt: none pending), so the label's
+    // submission can be bounded (APS5_LABEL_FLUSH_US) without taking the mutex to look;
+    static std::optional<std::chrono::steady_clock::time_point> PendingLabelSince();
+    // bumped by every note of a pending write into the open batch, so a poller learns that the
+    // producer may have recorded the label it waits for;
+    static std::uint64_t WriteGeneration();
+    // bumped right AFTER every store of the pending-write snapshot (the generation above moves
+    // before the store), so a reader that loads it before and after a set of in-place reads knows
+    // the snapshot it consulted in between was the one in force for all of them;
+    static std::uint64_t PublishGeneration();
+    // labels waiting in completion actions (the CPU must reap their batch for them to land);
+    static std::uint64_t PendingCompletionLabels();
+    // write-back completions (OnComplete) of unfinished batches: a reaper retires them when every
+    // worker sleeps, as for the labels above;
+    static std::uint64_t PendingWriteBackCompletions();
+    // dispatches and draws recorded since the last submit (the driver counts them: CountRecordedWork),
+    // so a batch is submitted after a bounded amount of work (APS5_BATCH_CAP).
+    static std::uint64_t RecordedWorkSinceSubmit();
+    static void CountRecordedWork();
+
+    // The recorder of the current device, for code that only has guest addresses (the flush hook)
+    // and for helpers that must not recycle resources the recorded work still uses.
+    static Recorder* Active();
+    // Whether the calling thread runs inside a batch's completion action (finish()): a store made
+    // there must not have a publish recorded for it (UnitShadow), see SyncThrough.
+    static bool InCompletion();
+    void Activate();
+    // APS5_PROFILE_DRAW: syncs by source (0 device idle, 1 CPU access to pending writes, 2 CPU read
+    // after a recorded store, 3 address-based dispatch, 4 other). Contract: the call announces the
+    // source for the Sync/SyncThrough this thread makes NEXT, which consumes it (a sync without a
+    // preceding CountSync counts as "other"); call it right before the sync and never without one,
+    // or the announcement attributes an unrelated later wait. `site` names the call site for the
+    // [recorder] "top sync sites" table (a return address, printed as a module offset like the
+    // [guestmem] callers): a caller that syncs on behalf of others (VulkanDevice::WaitIdle) passes
+    // its own return address, so the table names the driver site, not the wrapper; without it the
+    // return address of the Sync/SyncThrough call itself is taken, so an unannounced sync still
+    // names its real caller.
+    static void CountSync(int source, const void* site = nullptr);
+    // Names only the site for the sync or drain this thread makes next (its source stays as
+    // announced, or as the drain counts it): for a wrapper (VulkanDevice::ReapRecorded) whose
+    // FinishUpTo counts the drain itself, so a CountSync there would count it twice.
+    static void AnnounceSyncSite(const void* site);
+    // APS5_PROFILE_DRAW: the calling thread's fence and timeline waits so far, in milliseconds (0
+    // when not profiling): a caller reads it around a span of its own work to learn how much of
+    // that span waited for the GPU (a resource build's nested flush-hook waits, a draw's).
+    static double ThreadWaitedMs();
+    // The calling thread's flush-hook syncs (SyncThrough) that waited for a target batch whose
+    // fence had not signaled when the wait began: a read whose bytes are compared around such a
+    // sync observed the GPU's work; one that found the fence signaled already read the GPU's bytes
+    // both times and observed nothing (Driver.cpp's dword evidence reads it before and after).
+    static std::uint64_t ThreadHookWaits();
+    // Cumulative counts for the [labels] line (APS5_PROFILE_DRAW): stores recorded (RecordStore),
+    // the runs they formed (one barrier pair each), stores joined to a pending contiguous one,
+    // stores that replaced pending bytes, WAW barriers inside runs, joins refused because the
+    // joined store would overlap one the run already recorded, queued labels noted (of which
+    // over a recorded entry of the same dword), lookups a queued label satisfied on its own
+    // queue, flush-hook accesses that had queued labels recorded first, and hook accesses made
+    // by a completion action whose queued-label record was skipped.
+    struct StoreStatistics {
+        std::uint64_t stores, runs, joined, replaced, wawBarriers, joinsRefused, queuedNoted, queuedOverRecorded, queuedHits, queuedHookRecords, queuedHookInCompletion;
+        // DCC key stores queued, the runs that recorded them (at Submit, or before a writer of a
+        // queued range), and the queued stores a duplicate range joined.
+        std::uint64_t keyStores, keyStoreRuns, keyStoreRunsForWriter, keyStoresJoined;
+        // Store runs recorded at Submit, and in place before a later writer or reader of a queued
+        // store's bytes (FlushStores).
+        std::uint64_t runsAtSubmit, runsForced;
+    };
+    static StoreStatistics StoreCounts();
+    // APS5_PROFILE_GPU=1: GPU time of recorded work by key (a guest program address), from timestamp
+    // queries around each timed range; the totals per key are reported every 10 s. Begin returns the
+    // range index to pass to End, or NoTiming when timing is off or the batch's queries are used up.
+    static constexpr std::uint32_t NoTiming = 0xffffffffu;
+    static bool GpuTimingEnabled();
+    std::uint32_t BeginGpuTiming(std::uint64_t key);
+    // `bytes`: what the range moved (a fill's, a copy's), summed per key on the [gputime] line.
+    void EndGpuTiming(std::uint32_t index, std::uint64_t bytes = 0);
+    // The key of the whole-batch range (first to last command; reported as "batch" on the [gputime]
+    // line, apart from the per-program totals).
+    static constexpr std::uint64_t BatchTimingKey = 0x3;
+    // The driver's own command classes: their [gputime] ranges take the reserved keys 0x10 + class
+    // (printed by name, with counts and bytes, per 10 s and per present) and their barriers are
+    // counted on the [barriers] line (CountBarriers, printed every 10 s under APS5_PROFILE_DRAW
+    // or APS5_PROFILE_GPU). A class range covers the command with its own barriers (a dispatch's
+    // barriers are classes of their own, its program range stays keyed by the program), so the
+    // union of every range of a batch and the batch span differ by what no class times ('untimed').
+    enum class CommandClass : std::uint8_t { DispatchLeading = 0, DispatchTrailing, IndirectArguments, LabelRun, Fill, FillClear, Copy, StagingIn, StagingOut, Draw, StorageUpload, StorageWriteBack, DccClear, DccKeyStore, PresentBlit, ShadowPublish, TemplateDataRefresh, Count };
+    static constexpr std::uint64_t ClassKey(CommandClass which) { return 0x10 + static_cast<std::uint64_t>(which); }
+    std::uint32_t BeginGpuTiming(CommandClass which) { return BeginGpuTiming(ClassKey(which)); }
+    static void CountBarriers(CommandClass which, std::uint32_t count = 1);
+    // A leading barrier a command left out because the previous trailing barrier covered its
+    // accesses (see Commands), on the [barriers] line as 'merged'. Debug aid: APS5_FULL_BARRIERS=1
+    // (or APS5_NO_BARRIER_ELISION=1) records every leading barrier, as before.
+    static bool MergeBarriers();
+    static void CountMerged(CommandClass which);
+    // Hazard tracker, counting mode (APS5_BARRIER_VALIDATE=1): every command tells the tracker
+    // what it reads and writes (guest ranges through host imports, images) with its stage before
+    // it is recorded; the tracker simulates barriers emitted only on a hazard against the accesses
+    // since its last simulated barrier (RAW, WAW, WAR, an image written; a command whose ranges are
+    // unknown, an address-based build or a BDA draw, is a hazard by itself) and counts, per class,
+    // the leading barriers a hazard-driven recorder would have emitted and skipped, on the
+    // [barriers] line; the real barriers are recorded as before. APS5_TRACE_BARRIERS=<n> prints
+    // the first n skipped decisions with their ranges. Costs nothing when off.
+    struct Access {
+        std::span<const std::pair<std::uint64_t, std::uint64_t>> reads;
+        std::span<const std::pair<std::uint64_t, std::uint64_t>> writes;
+        // (image, written)
+        std::span<const std::pair<VkImage, bool>> images;
+        VkPipelineStageFlags stages;
+        bool conservative = false;
+    };
+    static bool BarrierValidate();
+    void NoteAccess(CommandClass which, const Access& access);
+    // A range timed outside the recorder's batches (the presenter's blit, stamped into its own
+    // pool and read after its fence): enters the class totals like a recorded one.
+    static void AddGpuTiming(CommandClass which, double nanoseconds, std::uint64_t bytes);
+    // Presentations, so the [gputime] class totals can be given per present.
+    static void CountPresent();
+    // Presentations so far (never reset): a frame clock for policies that count per frame, such as
+    // StorageTexture's write-back hysteresis.
+    static std::uint64_t Presents();
+
+    // What the presenter learns about finished batches (Driver::Present's [present] line and the
+    // frame record): finish() writes an entry per submitted batch into a ring of the last 128,
+    // under GuestMemory::GpuMutex and the ring's own mutex; the readers below take only the ring
+    // mutex, so the presenter reads them WITHOUT the GpuMutex. gpuStartNs/gpuEndNs are the
+    // whole-batch stamps (0 without APS5_PROFILE_GPU); `reads` and `readGeneration` (the
+    // write-watch generation collected over the noted in-place reads at submit) are filled with
+    // APS5_FLIP_READ_CHECK=1 only.
+    struct Completed {
+        std::uint64_t serial = 0;
+        double gpuStartNs = 0;
+        double gpuEndNs = 0;
+        std::chrono::steady_clock::time_point submittedAt{};
+        std::chrono::steady_clock::time_point fenceSeenAt{};
+        std::uint64_t readGeneration = 0;
+        std::vector<std::pair<std::uint64_t, std::uint64_t>> reads;
+    };
+    // The entries with serial in (afterSerial, throughSerial], oldest first; `missing` counts the
+    // serials of that range not in the ring (not finished yet, or overwritten by newer batches).
+    // An entry is overwritten once the batch CompletedRingSize serials later finishes.
+    static constexpr std::size_t CompletedRingSize = 512;
+    std::vector<Completed> CompletedBatches(std::uint64_t afterSerial, std::uint64_t throughSerial, std::size_t& missing) const;
+    // The newest submitted serial and its vkQueueSubmit time (ring mutex only).
+    std::uint64_t NewestSubmitted(std::chrono::steady_clock::time_point* submittedAt = nullptr) const;
+    // In-flight batches whose fence has not signaled (one status query each); under the mutex.
+    std::size_t UnsignaledBatches() const;
+    static bool FlipReadCheck();
+
+private:
+    struct Batch {
+        VkCommandBuffer commands = VK_NULL_HANDLE;
+        VkFence fence = VK_NULL_HANDLE;
+        std::vector<std::shared_ptr<void>> kept;
+        std::vector<std::function<void()>> completions;
+        std::vector<std::pair<std::uint64_t, std::uint64_t>> writes;
+        // In-place reads (see NotePendingRead), dying with the batch: a finished batch's reads are done.
+        struct Read {
+            std::uint64_t begin;
+            std::uint64_t end;
+            ReadKind kind;
+        };
+        std::vector<Read> reads;
+        // Submission number (1-based): identifies a batch after its allocation may have been reused.
+        std::uint64_t serial = 0;
+        bool submitted = false;
+        // The GpuMutex queue tag of the thread that opened the batch, and when it was submitted
+        // (the [syncwait] line reports what a wait's target and the batches ahead of it were).
+        std::uint32_t queue = 0xffffffffu;
+        std::chrono::steady_clock::time_point submittedAt{};
+        VkQueryPool queries = VK_NULL_HANDLE;
+        std::vector<std::uint64_t> timedKeys;
+        std::vector<std::uint64_t> timedBytes;
+        // The whole-batch timed range (BatchTimingKey) and its stamps once read (see Completed).
+        std::uint32_t batchTiming = NoTiming;
+        double gpuStartNs = 0;
+        double gpuEndNs = 0;
+        std::uint64_t readGeneration = 0;
+        // Dword addresses this batch noted in the label table (removed when it finishes).
+        std::vector<std::uint64_t> labelDwords;
+        // Labels stored by this batch's completion actions (see AfterCompletions); subtracted from
+        // the lock-free pending count when the batch finishes, whether or not its completions ran.
+        std::uint32_t completionLabelCount = 0;
+        // Write-back completions (OnComplete), likewise.
+        std::uint32_t writeBackCompletionCount = 0;
+        // The ranges of the completion labels also stored on the GPU, and whether each entered the
+        // pending count (NoteWrittenBack counts one the first time a write-back overlaps it).
+        struct CompletionLabel {
+            std::uint64_t begin;
+            std::uint64_t end;
+            bool counted;
+        };
+        std::vector<CompletionLabel> completionLabelRanges;
+        // The store run in progress (RecordStore): whether its leading barrier was recorded and
+        // the trailing one is still owed, the store not yet recorded (joined by contiguous ones),
+        // and the [buffer, offset, end) of the stores already recorded (the WAW check).
+        struct StoreRun {
+            bool open = false;
+            VkBuffer buffer = VK_NULL_HANDLE;
+            VkDeviceSize offset = 0;
+            std::vector<std::byte> bytes;
+            std::vector<std::tuple<VkBuffer, VkDeviceSize, VkDeviceSize>> recorded;
+            // The run's [gputime] range (leading barrier to trailing barrier) and its stored bytes.
+            std::uint32_t timing = NoTiming;
+            std::uint64_t storedBytes = 0;
+            // The stores queued for the batch's run (per-batch runs; see RecordStore), in order.
+            struct Queued {
+                VkBuffer buffer;
+                VkDeviceSize offset;
+                std::vector<std::byte> bytes;
+                std::uint64_t address;
+            };
+            std::vector<Queued> queued;
+        } run;
+        // See Commands(): the accesses the last recorded command's trailing barrier covers.
+        VkAccessFlags coveredAccess = 0;
+        // The hazard tracker's accesses since its last simulated barrier (see NoteAccess).
+        struct Tracker {
+            struct Range {
+                std::uint64_t begin;
+                std::uint64_t end;
+                VkPipelineStageFlags stages;
+            };
+            std::vector<Range> reads;
+            std::vector<Range> writes;
+            struct Image {
+                VkImage image;
+                bool written;
+                VkPipelineStageFlags stages;
+            };
+            std::vector<Image> images;
+        } tracker;
+        // The render pass a recorded draw left open (see ContinuesRenderPass).
+        struct RenderPass {
+            bool open = false;
+            bool continuable = false;
+            std::uint64_t key = 0;
+            std::uint32_t timing = NoTiming;
+        } renderPass;
+        // A pass ended in this batch: Submit records the host-read barrier its draws left out.
+        bool hostReadOwed = false;
+        // Queued DCC key stores (see QueueKeyStore).
+        struct KeyStore {
+            VkBuffer buffer;
+            VkDeviceSize first;
+            VkDeviceSize last;
+            std::shared_ptr<void> seed;
+            std::uint64_t begin;
+            std::uint64_t end;
+        };
+        std::vector<KeyStore> keyStores;
+    };
+    struct LabelEntry {
+        std::uint32_t value;
+        std::uint32_t queue;
+        std::uint64_t stamp;
+        // Null for a queued label (NoteQueuedLabel): not recorded yet, so no batch removes it.
+        const Batch* batch;
+        // Write-tracker generation at the close of the entry's record group (CloseLabelGroup);
+        // 0 while the group is open. A recorded non-label GPU write over the dword noted after the
+        // entry sets `overwritten`: memory will not end with `value`. Both serve the late rule only.
+        std::uint64_t generation = 0;
+        bool overwritten = false;
+        // The value lands in memory only when the batch's completion actions run (LabelRefusal::BehindCompletion).
+        bool behindCompletion = false;
+    };
+    void readGpuTiming(Batch& batch);
+    // BeginGpuTiming on the open batch without Commands() (RecordStore times its own run, which
+    // Commands() would close).
+    std::uint32_t beginTiming(std::uint64_t key);
+    // Vulkan entry points resolved once (constructor): the loader's vkGetDeviceProcAddr is a name
+    // lookup per call, paid by every reap, store and submit otherwise. APS5_NO_PROC_TABLE=1 leaves
+    // them null and resolves per call as before; vkWaitSemaphoresKHR stays a lazy lookup (timeline
+    // semaphores are optional).
+    PFN_vkGetFenceStatus getFenceStatus = nullptr;
+    PFN_vkResetFences resetFences = nullptr;
+    PFN_vkWaitForFences waitForFences = nullptr;
+    PFN_vkCmdUpdateBuffer cmdUpdateBuffer = nullptr;
+    PFN_vkBeginCommandBuffer beginCommandBuffer = nullptr;
+    PFN_vkEndCommandBuffer endCommandBuffer = nullptr;
+    PFN_vkQueueSubmit queueSubmit = nullptr;
+    PFN_vkCmdPipelineBarrier cmdPipelineBarrier = nullptr;
+    template<typename TFunction>
+    TFunction function(TFunction resolved, const char* name) const {
+        return resolved != nullptr ? resolved : context.Function<TFunction>(name);
+    }
+    VkResult fenceStatus(VkFence fence) const { return function(getFenceStatus, "vkGetFenceStatus")(context.device, fence); }
+    void recordBarrier(VkCommandBuffer commands, VkPipelineStageFlags sourceStage, VkPipelineStageFlags destinationStage, VkAccessFlags sourceAccess, VkAccessFlags destinationAccess) const;
+    // Opens a batch when none is open, without closing a store run (Commands() does both).
+    void ensureOpen();
+    // Records the run's pending store (vkCmdUpdateBuffer), if any.
+    void flushPendingStore();
+    // Records the pending store and the run's trailing barrier; the next store starts a new run.
+    // Per-batch runs: records the queued stores with their barriers (`atSubmit`: the batch-end
+    // form, toward the host only); returns whether the batch's host-read barrier was included.
+    bool closeStoreRun(bool atSubmit = false);
+    // Ends the render pass a draw left open (vkCmdEndRenderPass, the pass's trailing barrier).
+    void endOpenRenderPass();
+    // Records the queued key stores as one run (`forWriter`: before a command writing, reading or
+    // labelling over one, not at Submit).
+    void recordKeyStores(bool forWriter);
+
+    // The unlocked wait of SyncThrough(waitUnlocked): the target batch (submitting the open one when
+    // it is the target), the timeline wait with the mutex released, then the completions up to it.
+    // Returns false when the locked path must run instead (no timeline, nested acquisition, off).
+    bool syncThroughUnlocked(std::uint64_t address, std::uint64_t end, int source, const void* site);
+    // `source` is the CountSync source the wait is attributed to.
+    void finish(std::unique_ptr<Batch> batch, bool wait, int source);
+    void release(Batch& batch) noexcept;
+    static bool overlaps(const Batch& batch, std::uint64_t address, std::uint64_t end);
+    // The first noted read of `batch` overlapping [address, end), or null.
+    static const Batch::Read* readOverlap(const Batch& batch, std::uint64_t address, std::uint64_t end);
+    bool signaled(const Batch& batch) const;
+    // Rebuilds the lock-free snapshot of pending writes from open, inFlight and finishing.
+    void publishPendingWrites() const;
+    // Appends one range to the open batch; returns whether the snapshot must be rebuilt for it.
+    // `ownLabel`: the range is a label's own store (NoteLabel, AfterCompletions), which does not
+    // overwrite the table entries it covers; any other range flags them (table mutex, briefly).
+    bool noteWrite(std::uint64_t address, std::size_t bytes, bool ownLabel = false);
+    // Appends one range to `batch` (open or in flight) and publishes the snapshot if needed.
+    void noteWriteOn(Batch& batch, std::uint64_t address, std::size_t bytes, bool ownLabel = false);
+    void noteLabelOn(Batch& batch, std::uint64_t address, std::span<const std::byte> bytes, std::uint64_t stamp, std::uint32_t queue, bool behindCompletion = false);
+    // Flags the recorded entries of `batch` over [begin, end) as stored by a completion action.
+    void markBehindCompletion(const Batch& batch, std::uint64_t begin, std::uint64_t end);
+    // Flags the recorded entries of the dwords overlapping [address, end) as overwritten.
+    void markOverwritten(std::uint64_t address, std::uint64_t end);
+    // Whether a write-back noted after `sequence` overlapped [begin, end); true when the ring no
+    // longer reaches back to `sequence` (conservative: the store runs as before).
+    bool writtenBackSince(std::uint64_t sequence, std::uint64_t begin, std::uint64_t end);
+    std::mutex writtenBackMutex;
+    std::deque<std::array<std::uint64_t, 3>> writtenBack;
+    std::uint64_t writtenBackSequence = 0;
+    // PendingLabel without the table mutex (the caller holds it, or the GPU mutex).
+    std::optional<LabelHit> lookupLabel(std::uint64_t address, std::size_t bytes, std::uint64_t afterStamp, LabelRefusal* refusal) const;
+    // Whether the in-flight batch with that serial has not signaled its fence (false when it is
+    // not in flight any more): what makes a wait for it a real GPU wait (ThreadHookWaits).
+    bool unsignaled(std::uint64_t serial) const;
+
+    Context context;
+    VkSemaphore timeline = VK_NULL_HANDLE;
+    // Identity for a thread that released the mutex around a wait (syncThroughUnlocked): the
+    // recorder may have been torn down meanwhile, so it is looked up by this id, not by pointer.
+    std::uint64_t id = 0;
+    // Every access holds the table mutex (Recorder.cpp): the recorded entries are mutated under
+    // GuestMemory::GpuMutex as well, the queued ones (NoteQueuedLabel) under the table mutex alone.
+    // Ordered, so a range (a noted write, PendingLabelIn) finds its dwords by lower_bound.
+    std::map<std::uint64_t, LabelEntry> labels;
+    // The queued entries (batch always null), by dword; a lookup on the noting queue's thread
+    // prefers them (its newest store in program order), every other lookup sees `labels` alone.
+    std::map<std::uint64_t, LabelEntry> queuedLabels;
+    // labels.size(), readable without the table mutex: a noted write skips the mutex while the
+    // table is empty (most of the time between label groups).
+    std::atomic<std::size_t> recordedLabels{0};
+    std::unique_ptr<Batch> open;
+    std::deque<std::unique_ptr<Batch>> inFlight;
+    // Batches popped from inFlight whose completions have not run yet: their writes stay in the
+    // snapshot (a rebuild from inside a completion must not drop them) until finish returns.
+    std::vector<const Batch*> finishing;
+    std::uint64_t submissions = 0;
+    // Command buffers and fences of completed batches, reused by later ones (hundreds of batches per
+    // frame would otherwise allocate and free their objects each time).
+    std::vector<std::pair<VkCommandBuffer, VkFence>> spare;
+    mutable std::mutex completedMutex;
+    std::array<Completed, CompletedRingSize> completed;
+    std::uint64_t newestSubmitted = 0;
+    std::chrono::steady_clock::time_point newestSubmittedAt{};
+};
+
+}
+
+#endif

core/libs/prx/libSceAgcDriver/Graphics/include/RenderCache.hpp

@@ -1,55 +0,0 @@
-#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_RENDERCACHE_HPP
-#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_RENDERCACHE_HPP
-
-#include "prx/libSceAgcDriver/Graphics/include/GpuColorTransfer.hpp"
-#include "prx/libc/include/GuestMemoryTracking.hpp"
-#include <map>
-
-namespace AgcDriver::Graphics {
-
-class ResidentColor {
-public:
-    ResidentColor(const Context& context, const ColorTarget& color);
-    ~ResidentColor();
-    void Begin(VkCommandBuffer commands);
-    void Download(VkCommandBuffer commands);
-    void Commit();
-    void Transition(VkCommandBuffer commands, VkImageLayout layout);
-    RenderTarget& Target() { return transfer.Target(color, false); }
-    const ColorTarget& Description() const { return color; }
-    bool Valid() const { return valid; }
-    bool Dirty() const { return dirty; }
-    std::uint64_t Generation() const { return generation; }
-    void Invalidate();
-    void ReleaseMemory();
-    bool SharesPages(const ColorTarget& other) const;
-
-private:
-    void resolveCpuAccess(GuestMemoryTracking::Access access);
-    Context context;
-    ColorTarget color;
-    GpuColorTransfer transfer;
-    VkImageLayout layout = VK_IMAGE_LAYOUT_UNDEFINED;
-    bool valid = false;
-    bool dirty = false;
-    std::uint64_t generation = 0;
-    std::unique_ptr<GuestMemoryTracking::Watch> memoryWatch;
-};
-
-class RenderCache {
-public:
-    explicit RenderCache(const Context& context) : context(context) {}
-    ~RenderCache();
-    std::shared_ptr<ResidentColor> Get(const ColorTarget& color, bool blending);
-    std::shared_ptr<ResidentColor> Find(std::uint64_t address) const;
-    void Resolve(std::uint64_t address, std::size_t bytes, bool writable);
-    void Flush();
-
-private:
-    Context context;
-    std::map<std::uint64_t, std::shared_ptr<ResidentColor>> entries;
-};
-
-}
-
-#endif

core/libs/prx/libSceAgcDriver/Graphics/include/Resources.hpp

@@ -14,7 +14,10 @@ public:
     Buffer& operator=(const Buffer&) = delete;
     VkBuffer Handle() const;
     VkDeviceAddress DeviceAddress() const;
+    // Host-visible buffers only: a device-local one (properties without HOST_VISIBLE, the staging
+    // shadows of GuestBufferMemory) has no mapping and its bytes move by GPU copies alone.
     std::span<std::byte> Bytes();
+    bool Mapped() const { return mapping != nullptr; }
     void Invalidate();
 
 private:
@@ -26,13 +29,50 @@ private:
     VkDeviceMemory memory = VK_NULL_HANDLE;
     void* mapping = nullptr;
     std::size_t size;
+    // The VkBuffer's own size, `size` rounded up to its pool class (see BufferPool::Capacity).
+    std::size_t capacity;
+    // Fully made (or taken from the pool), so release returns it to the pool instead of destroying
+    // the handles a failed construction left.
+    bool ready = false;
     VkDeviceSize allocationBytes = 0;
     VkBufferUsageFlags usage;
     VkMemoryPropertyFlags properties;
-    bool reusable = false;
     std::shared_ptr<BufferPool> cache;
 };
 
+// Device-local scratch memory for GPU-side layout conversion. The detiler reads and writes scattered
+// elements, which crawls across PCIe, so guest bytes move between host and device buffers with DMA
+// copies and are only swizzled in video memory.
+class DeviceBuffer {
+public:
+    DeviceBuffer(const Context& context, std::size_t size, VkBufferUsageFlags usage);
+    ~DeviceBuffer();
+    DeviceBuffer(const DeviceBuffer&) = delete;
+    DeviceBuffer& operator=(const DeviceBuffer&) = delete;
+    VkBuffer Handle() const;
+    std::size_t Size() const;
+
+private:
+    void release() noexcept;
+    Context context;
+    VkBuffer buffer = VK_NULL_HANDLE;
+    VkDeviceMemory memory = VK_NULL_HANDLE;
+    std::size_t size;
+    std::size_t capacity;
+    VkDeviceSize allocationBytes = 0;
+    VkBufferUsageFlags usage;
+    std::shared_ptr<BufferPool> cache;
+};
+
+// Records a whole-range buffer copy.
+void CopyBuffer(const Context& context, VkCommandBuffer commands, VkBuffer source, VkDeviceSize sourceOffset, VkBuffer destination, VkDeviceSize destinationOffset, VkDeviceSize bytes);
+
+// Records a global memory barrier.
+void RecordMemoryBarrier(const Context& context, VkCommandBuffer commands, VkPipelineStageFlags sourceStage, VkPipelineStageFlags destinationStage, VkAccessFlags sourceAccess, VkAccessFlags destinationAccess);
+
+// Resolves every DeviceFunctions entry point of the context's device (once, at device setup).
+void FillDeviceFunctions(const Context& context, DeviceFunctions& functions);
+
 class RenderTarget {
 public:
     RenderTarget(const Context& context, const ColorTarget& target, bool blending);
@@ -60,8 +100,6 @@ public:
     void SubmitAndWait();
     void Submit();
     void Wait();
-    bool IsComplete();
-    void Reset();
 
 private:
     void release() noexcept;

core/libs/prx/libSceAgcDriver/Graphics/include/Sampler.hpp

@@ -4,8 +4,11 @@
 #include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/GuestSamplerResource.hpp"
 #include <array>
+#include <cstdint>
 #include <map>
+#include <memory>
 #include <mutex>
+#include <span>
 
 namespace AgcDriver::Graphics {
 
@@ -21,18 +24,36 @@ public:
 private:
     void release() noexcept;
 
-    VkDevice device;
-    PFN_vkDestroySampler destroySampler;
+    Context context;
     VkSampler sampler = VK_NULL_HANDLE;
 };
 
+// One VkSampler per distinct S# (its 4 words plus the shader's depth-compare use): samplers are
+// immutable and DecodeSamplerResource is a pure function of the words, so equal keys mean equal
+// samplers and nothing ever invalidates an entry. Holders keep a shared_ptr, so an entry evicted from
+// the LRU-capped cache lives on while a descriptor set in flight references it. One cache per device.
+// APS5_NO_SAMPLER_CACHE=1 creates a sampler per binding as before.
 class SamplerCache {
 public:
-    std::shared_ptr<Sampler> Get(const Context& context, std::span<const std::uint32_t> words, const GuestSamplerResource& descriptor);
+    explicit SamplerCache(std::size_t capacity = 1024);
+    SamplerCache(const SamplerCache&) = delete;
+    SamplerCache& operator=(const SamplerCache&) = delete;
+    std::shared_ptr<Sampler> Get(const Context& context, std::span<const std::uint32_t> words, bool compareEnable);
+    // APS5_PROFILE_DRAW counters: lookups served by an existing sampler, and samplers created.
+    std::uint64_t Hits() const { return hits; }
+    std::uint64_t Misses() const { return misses; }
 
 private:
+    struct Entry {
+        std::shared_ptr<Sampler> sampler;
+        std::uint64_t lastUse;
+    };
     std::mutex mutex;
-    std::map<std::array<std::uint32_t, 5>, std::shared_ptr<Sampler>> entries;
+    std::map<std::array<std::uint32_t, 5>, Entry> entries;
+    std::uint64_t clock = 0;
+    std::size_t capacity;
+    std::uint64_t hits = 0;
+    std::uint64_t misses = 0;
 };
 
 }

core/libs/prx/libSceAgcDriver/Graphics/include/ShaderInputState.hpp

@@ -3,14 +3,22 @@
 
 #include "prx/libSceAgcDriver/Execution/include/QueueState.hpp"
 #include "Recompiler.hpp"
+#include <cstddef>
 #include <cstdint>
 #include <span>
+#include <vector>
 
 namespace AgcDriver::Graphics {
 
 ShaderRecompiler::ShaderPixelStageInfo DecodePixelStageInfo(const Registers& context, bool hasColorTarget, std::uint8_t colorComponentMapping);
 ShaderRecompiler::ShaderComputeStageInfo DecodeComputeStageInfo(const Registers& shader);
-ShaderRecompiler::ShaderVertexStageInfo DecodeVertexStageInfo(std::span<const std::byte> header, std::uint64_t headerAddress, std::span<const std::uint32_t> userData);
+// A guest range DecodeVertexStageInfo read (an attribute word, a vertex V#) with the bytes as read:
+// the draw cache validates them by value with the stage's capture (design_cpu_final rule RD).
+struct DecodeRead {
+    std::uint64_t address;
+    std::vector<std::byte> bytes;
+};
+ShaderRecompiler::ShaderVertexStageInfo DecodeVertexStageInfo(std::span<const std::byte> header, std::uint64_t headerAddress, std::span<const std::uint32_t> userData, std::vector<DecodeRead>* reads = nullptr);
 
 }
 

core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp

@@ -5,36 +5,224 @@
 #include "prx/libSceAgcDriver/Graphics/include/BdaResources.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/Texture.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/Sampler.hpp"
-#include "prx/libSceAgcDriver/Graphics/include/DescriptorCache.hpp"
 #include "Recompiler.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/Shaders.hpp"
 #include <array>
+#include <chrono>
+#include <list>
+#include <map>
 #include <memory>
+#include <mutex>
+#include <string>
+#include <unordered_map>
 #include <vector>
 
 namespace AgcDriver::Graphics {
 
+class Recorder;
+
+void FlushCachedTextures(VkDevice device);
+void ClearCachedTextures(VkDevice device);
+
+// The cached storage image of a surface (render targets use it as their resident image); brought up
+// to date with guest memory before it is returned.
+std::shared_ptr<StorageTexture> CachedStorageSurface(const Context& context, const GuestTextureResource& resource);
+// Whether `image` is still the storage cache's image of its surface (what CachedStorageSurface
+// would return); a true answer counts as a use for the cache's eviction order, as a lookup would.
+bool StorageImageCached(const Context& context, const StorageTexture* image);
+
+// Defined in Texture.cpp beside the pending-results registry, for the fast Revalidate below: whether
+// a storage image other than `except` has results pending in [address, address + bytes).
+bool PendingStorageOverlaps(std::uint64_t address, std::size_t bytes, const StorageTexture* except);
+
+// Per-device descriptor objects shared by ShaderResources builds: set layouts by their binding list
+// (immutable; kept until the device is torn down, which Vulkan allows even for pipeline layouts made
+// from them) and a chain of descriptor pools that sets are freed back to, so a build creates and
+// destroys no layout or pool of its own. A set lives exactly as long as its ShaderResources, which
+// its batch keeps until the GPU completed. The cache locks itself: stage A of a dispatch build
+// (ShaderResources::buildPrepare, from VulkanDevice::PrepareDispatch) takes layouts and sets from it
+// without GuestMemory::GpuMutex, while other threads free and update sets of the same pools under
+// that lock (Vulkan synchronizes the pool for allocate/free and the set for update, so that is
+// legal). APS5_NO_LAYOUT_CACHE=1 / APS5_NO_POOL_CACHE=1 restore the per-build objects.
+class DescriptorCache {
+public:
+    explicit DescriptorCache(const Context& context);
+    ~DescriptorCache();
+    DescriptorCache(const DescriptorCache&) = delete;
+    DescriptorCache& operator=(const DescriptorCache&) = delete;
+
+    // The layout for `key` (binding, type, count, stage flags per binding, as ShaderResources builds
+    // it from `bindings`), created on first use.
+    VkDescriptorSetLayout Layout(std::span<const std::uint32_t> key, std::span<const VkDescriptorSetLayoutBinding> bindings);
+    struct SetAllocation {
+        VkDescriptorSet set = VK_NULL_HANDLE;
+        VkDescriptorPool pool = VK_NULL_HANDLE;
+    };
+    // A set of `layout` needing `sizes` descriptors from the pool chain, opening a pool when no pool
+    // has room; a null set when the needs exceed what one chain pool holds (the caller then makes a
+    // dedicated pool, as before).
+    SetAllocation Allocate(VkDescriptorSetLayout layout, std::span<const VkDescriptorPoolSize> sizes);
+    void Free(const SetAllocation& allocation) noexcept;
+    // APS5_PROFILE_DRAW counters: layouts served from the map / created, sets allocated, pools opened.
+    struct Stats {
+        std::uint64_t layoutHits = 0;
+        std::uint64_t layoutMisses = 0;
+        std::uint64_t sets = 0;
+        std::uint64_t pools = 0;
+    };
+    Stats Counters() const;
+
+private:
+    Context context;
+    PFN_vkDestroyDescriptorSetLayout destroyLayout;
+    PFN_vkDestroyDescriptorPool destroyPool;
+    PFN_vkFreeDescriptorSets freeSets;
+    mutable std::mutex mutex;
+    std::map<std::vector<std::uint32_t>, VkDescriptorSetLayout> layouts;
+    std::vector<VkDescriptorPool> pools;
+    Stats stats;
+};
+
 class ShaderResources {
 public:
     ShaderResources(const Context& context, const ShaderRecompiler::RecompileResult& vertex, const ShaderRecompiler::RecompileResult& fragment, const ColorTarget& target, std::uint64_t indexAddress, std::size_t indexBytes);
     ShaderResources(const Context& context, std::span<const CompiledShader> shaders, const ColorTarget& target, std::uint64_t indexAddress, std::size_t indexBytes, std::span<const GuestMemorySnapshot> snapshots = {});
     ShaderResources(const Context& context, const CompiledShader& compute, std::span<const GuestMemorySnapshot> snapshots = {});
+    // Two-stage build for dispatches (see build): with `deferred` the constructor runs stage A only,
+    // which needs no device lock, and Complete() runs stage B under GuestMemory::GpuMutex; until
+    // then no other member may be used. `compute` and `snapshots` must outlive Complete(). An
+    // address-based shader (BDA tables) builds entirely in Complete(): its lease acquisition
+    // reconciles imports and refreshes mirrors, which needs the lock.
+    ShaderResources(const Context& context, const CompiledShader& compute, std::span<const GuestMemorySnapshot> snapshots, bool deferred);
+    void Complete();
+    bool Completed() const { return completed; }
     ~ShaderResources();
     ShaderResources(const ShaderResources&) = delete;
     ShaderResources& operator=(const ShaderResources&) = delete;
     VkDescriptorSetLayout Layout() const;
     void Bind(VkCommandBuffer commands, VkPipelineBindPoint bindPoint, VkPipelineLayout layout) const;
     void WriteBack();
+    // Deferred completion: MarkGpuWrites registers the results the recorded work leaves on the GPU
+    // (storage images stay there; buffer ranges are noted so CPU reads wait); WriteBackBuffers runs
+    // once the work completed.
+    void MarkGpuWrites(Recorder& recorder);
+    void WriteBackBuffers();
+    // Whether WriteBackBuffers has anything the CPU must see (copied written buffers, BDA faults).
+    bool NeedsCompletion() const { return bda != nullptr || guestMemory.HasCopiedWrites(); }
+    // Whether a written buffer was copied (its results reach guest memory by the CPU write-back).
+    bool HasCopiedWrites() const { return guestMemory.HasCopiedWrites(); }
+    bool HoldsLease() const { return guestMemory.HoldsLease(); }
     bool WritesOverlap(std::uint64_t address, std::size_t bytes) const { return guestMemory.WritesOverlap(address, bytes); }
+    // Whether a region the recorded work reads in place through a host import overlaps the range.
+    bool ReadsOverlap(std::uint64_t address, std::size_t bytes) const;
+    // The recorder's hazard tracker inputs (Recorder::NoteAccess): the written elements' guest
+    // ranges, the regions read in place, and the storage images with whether each is written.
+    const std::vector<std::pair<std::uint64_t, std::uint64_t>>& GpuWrites() const { return guestMemory.Writes(); }
+    std::vector<std::pair<std::uint64_t, std::uint64_t>> InPlaceReads() const { return guestMemory.InPlaceReads(); }
+    std::vector<std::pair<VkImage, bool>> StorageImages() const;
+    // Whether a use writes guest memory beyond a draw's attachments (storage images, written or
+    // copied buffers, an address-based build's unknown writes), or reads `image` (a view of it
+    // sampled, or the image itself bound): a recorded draw's render pass may only be continued by
+    // a draw for which neither holds (Draw.cpp).
+    bool WritesMemory() const;
+    bool ReadsImage(const StorageTexture* image) const;
     const std::vector<std::uint32_t>& LayoutKey() const { return layoutKey; }
+    // Debug aid: each bound guest resource with the fraction of sampled bytes that are nonzero.
+    std::string Describe() const;
+
+    // Reuse across dispatches of identical content (see the resource cache in VulkanDevice): the
+    // key is everything a compute build consumes from its compiled shader (variant, and per binding
+    // kind, role, slot, count, guest descriptor words, written images, depth-compare samplers); push
+    // constants are not part of it because every dispatch pushes its own. Reusable says whether this
+    // object can serve a later dispatch with the same key: nothing needs completion work and every
+    // guest buffer is read in place through a host import. Revalidate brings a reusable object up to
+    // date for another use, or says it cannot be reused any more (a buffer's import went away, or a
+    // texture or storage image the set references is no longer the one its memory maps to).
+    // Without `dataWords` the ShaderData and FlattenedSrt descriptor words stay out of the key
+    // (their count and size remain): a compute template then serves dispatches whose constants
+    // differ, and the hit refreshes its data buffers with the dispatch's words (RefreshData).
+    static std::vector<std::uint32_t> ContentKey(const CompiledShader& shader, bool dataWords = true);
+    // Records the shader's ShaderData and FlattenedSrt words into this object's data buffers
+    // (vkCmdUpdateBuffer, a transfer write the caller's pre-dispatch barrier makes visible; a
+    // buffer already holding the words is left alone). Returns whether anything was recorded. With
+    // `recorder` the updates are one [gputime] range of class TemplateDataRefresh.
+    bool RefreshData(VkCommandBuffer commands, const CompiledShader& shader, Recorder* recorder = nullptr);
+    // Walks the write watch over every image surface of a completed object (no device lock: the
+    // ranges are fixed by the build), so a Revalidate's collects on the same worker are memo hits.
+    void PrecollectSurfaces() const;
+    bool Reusable() const { return reusable; }
+    // `shaders` are the stages the object was built from, in build order (a recorded draw's vertex
+    // and fragment stages, or one compute stage): their bindings are walked like the build did.
+    // How a Revalidate proved (or refused) the object, for the [recipe] line: the proof path taken
+    // and, on a refusal, what failed.
+    enum class ProofPath : std::size_t { Fast, OwnRefreshed, Full, Count };
+    enum class ProofFailure : std::size_t { None, Imports, Evicted, Pending, Changed, Keys, Other, Count };
+    struct ProofReport {
+        ProofPath path = ProofPath::Fast;
+        ProofFailure failure = ProofFailure::None;
+    };
+    bool Revalidate(std::span<const CompiledShader> shaders, ProofReport* report = nullptr);
+    bool Revalidate(const CompiledShader& shader, ProofReport* report = nullptr) { return Revalidate(std::span<const CompiledShader>(&shader, 1), report); }
+    // The proof a recipe hit runs on its template (design_cpu_final M4): Revalidate by that name.
+    bool ProveCurrent(std::span<const CompiledShader> shaders, ProofReport* report = nullptr) { return Revalidate(shaders, report); }
+    bool ProveCurrent(const CompiledShader& shader, ProofReport* report = nullptr) { return Revalidate(shader, report); }
+    // FNV over the ShaderData/FlattenedSrt words, in binding order: of this object's data buffers
+    // as the recorded work will find them (maintained by addDataBuffer and RefreshData), and of a
+    // compiled shader's descriptors. A recipe hit refreshes the template iff the two differ.
+    std::uint64_t DataWordsHash() const { return dataWordsHash; }
+    static std::uint64_t DataWordsHash(const CompiledShader& shader);
+    // Whether RefreshData would record anything for `shader` (the per-word compare; verification).
+    bool DataWordsDiffer(const CompiledShader& shader) const;
+    // Why the fast proof of a Revalidate left the object to the full walk (the [rescache]
+    // revalidate line's reasons); Count: it did not.
+    enum class FastFail : std::size_t { NoRecord, Collect, Pending, Evicted, Changed, Keys, ClearedView, StorageKeys, Count };
+    // Why a Pending failure was left to the full walk instead of the own-object refresh (T1, see
+    // refreshOwnObjects); Count: it was not.
+    enum class OwnRefreshFallback : std::size_t { Disabled, Snapshot, Keys, ForeignView, SurfaceKey, NotImported, Uncached, Rerun, Count };
+    // APS5_PROFILE_DRAW: the build's sub-phases ([resources] totals) and their times in
+    // milliseconds (zero when not profiling), so the draw path can total them separately from
+    // dispatches. bindingsMs covers the binding plan (stage A) and the image lookups (stage B);
+    // prepareMs/completeMs are the two stages' totals.
+    enum class BuildPhase : std::size_t { Bindings, Precollect, Upload, Descriptors, StageA, Images, Bda, StageB, Count };
+    struct BuildTiming {
+        double bindingsMs = 0;
+        double uploadMs = 0;
+        double descriptorsMs = 0;
+        double prepareMs = 0;
+        double completeMs = 0;
+    };
+    const BuildTiming& Timing() const { return timing; }
+    // The image surfaces (address, bytes) whose stage-B lookup reads guest memory on the CPU and so
+    // would wait, under the device lock, for recorded work writing them: a surface without a host
+    // import, or a sampled one that stays a CPU snapshot (block compressed, or the storage-view path
+    // is off). VulkanDevice::PrepareDispatch waits for that work before the lock (the pre-sync).
+    // Usable right after the deferred constructor, for address-based builds too (it decodes the
+    // compiled shader's bindings itself when stage A has not run), and on a completed object from
+    // the resource cache (its Revalidate repeats the same lookups), without the device lock.
+    std::vector<std::pair<std::uint64_t, std::uint64_t>> PresyncSurfaces() const;
 
 private:
+    struct DescribedRange {
+        const char* kind;
+        std::uint64_t address;
+        std::uint64_t bytes;
+        std::uint32_t width = 0;
+        std::uint32_t height = 0;
+        std::uint32_t format = 0;
+        int tileMode = 0;
+        std::uint64_t dccAddress = 0;
+    };
+    std::vector<DescribedRange> describedRanges;
     struct Allocation {
         std::uint64_t address;
         std::size_t size;
         bool guest;
         std::unique_ptr<Buffer> buffer;
         ShaderRecompiler::DescriptorRole role = ShaderRecompiler::DescriptorRole::ShaderData;
+        // Guest buffers: whether the shader may store to the range (see addGuestBuffer).
+        bool written = true;
+        // Data buffers: the words the buffer holds once the recorded work ran (see RefreshData).
+        std::vector<std::uint32_t> dataWords;
     };
 
     struct Binding {
@@ -43,13 +231,125 @@ private:
         std::vector<std::size_t> imageAllocations;
     };
 
+    // A host-imported buffer region the set reads in place, with its import's identity at build time.
+    struct DirectRegion {
+        std::uint64_t begin;
+        std::uint64_t end;
+        std::uint64_t serial;
+    };
+
+    // The build is two stages. A (buildPrepare, no device lock): the binding plan with its layout
+    // entries, samplers, data buffers, the upload's prepare stage (imports already serving the
+    // regions, read-only copies), the set layout and the descriptor set. B (buildComplete, under
+    // GuestMemory::GpuMutex): the texture and storage image lookups (they refresh, upload and flush
+    // through the recorder), the rest of the upload, the BDA objects, the descriptor writes and the
+    // reusability record. build runs both.
     void build(std::span<const CompiledShader> shaders, const ColorTarget* target, std::uint64_t indexAddress, std::size_t indexBytes);
-    std::size_t addGuestBuffer(std::span<const std::uint32_t> words, const ColorTarget* target, std::uint64_t indexAddress, std::size_t indexBytes);
+    void buildPrepare(std::span<const CompiledShader> shaders, const ColorTarget* target, std::uint64_t indexAddress, std::size_t indexBytes);
+    void buildComplete();
+    // APS5_PROFILE_DRAW: closes the current sub-phase of the build into the [resources] totals.
+    double phase(BuildPhase which);
+    // `written` is the element's DescriptorBinding::bufferWritten: a read-only element binds the
+    // same way but is left out of the write set (no write-back, no pending-write note). `atomic`
+    // is its bufferAtomic (see GuestBufferMemory::AddWritable).
+    std::size_t addGuestBuffer(std::span<const std::uint32_t> words, const ColorTarget* target, std::uint64_t indexAddress, std::size_t indexBytes, bool written, bool atomic);
     std::size_t addDataBuffer(std::span<const std::uint32_t> words);
-    void addImageBinding(const ShaderRecompiler::DescriptorBinding& binding, VkShaderStageFlags flags, std::vector<Binding>& bindings);
+    // Stage A: the layout entry of an image binding (samplers are taken at once, the sampler cache
+    // locks itself); stage B looks the sampled textures and storage images up (resolveImageBinding).
+    void addImageBinding(const ShaderRecompiler::DescriptorBinding& binding, VkShaderStageFlags flags);
+    // Stage A: one record per planned image element (ImageRecord), in plan order: the decoded
+    // descriptor and its surface size (computed once for the build), the write watch walked so stage
+    // B's collects are memo hits (APS5_NO_PRECOLLECT=1 skips the pass entirely), and for a sampled
+    // element the cache entry it maps to, taken now so stage B needs only the fastRevalidate
+    // predicate under the lock (APS5_NO_STAGE_A_IMAGES=1 leaves every element to the full lookup).
+    // Whether it ran.
+    bool precollectImages();
+    struct ImageRecord {
+        bool sampled = false;
+        // The descriptor decoded and its surface described; false leaves the element to stage B's
+        // own decode (which reports the error).
+        bool decoded = false;
+        std::array<std::uint32_t, 8> words{};
+        GuestTextureResource resource{};
+        std::uint64_t guestBytes = 0;
+        VkComponentMapping components{};
+        // Sampled elements: the keys read in stage A and the collect made BEFORE stage B's checks
+        // (the generation the entry moves to when they pass).
+        DccKeys keys = DccKeys::Uncompressed;
+        std::uint64_t generation = 0;
+        // The cache entry's object, storage source, keys and generation as of stage A (texture null:
+        // no entry, the full lookup makes one).
+        std::shared_ptr<Texture> texture;
+        std::shared_ptr<StorageTexture> source;
+        DccKeys entryKeys = DccKeys::Uncompressed;
+        std::uint64_t entryGeneration = 0;
+    };
+    std::vector<ImageRecord> imageRecords;
+    // The next record resolveImageBinding consumes (records follow the deferredImages order).
+    std::size_t nextImageRecord = 0;
+    // Stage B: the record's texture when the fastRevalidate predicate proves it current under the
+    // lock and the cache still holds it; null sends the element to cachedTexture.
+    std::shared_ptr<Texture> fastTexture(const ImageRecord& record);
+    void resolveImageBinding(const ShaderRecompiler::DescriptorBinding& binding, Binding& item);
+    void forgetDeferredInputs();
     void release() noexcept;
     void prepareAddressBindings(std::span<const CompiledShader> shaders, std::span<const GuestMemorySnapshot> snapshots);
     VkDescriptorBufferInfo descriptor(const Allocation& allocation) const;
+    void noteReusable();
+    void reportDescriptorCaches() const;
+    // What a sampled texture was proved current against when the build (or the last full Revalidate)
+    // looked it up, so the next Revalidate can repeat the proof from write stamps and the DCC keys
+    // alone instead of repeating the lookup (see fastRevalidate). One record per textures[] entry; an
+    // element without a record always takes the full lookup. Storage images need no record: their
+    // proof comes from the image itself (its own surface and generation).
+    struct ValidatedSurface {
+        GuestTextureResource resource{};
+        std::uint64_t bytes = 0;
+        // Keys read at the lookup (a snapshot texture holds its clear texels while they are unchanged).
+        DccKeys keys = DccKeys::Uncompressed;
+        // Snapshot textures: write generation the content was known current at, always from a
+        // collect issued before the checks that proved it.
+        std::uint64_t generation = 0;
+        // The collect of the check in progress, moved into `generation` once every element passed.
+        std::uint64_t collected = 0;
+        // Storage-sourced textures: the image the view follows (kept alive by the texture).
+        const StorageTexture* source = nullptr;
+        bool valid = false;
+    };
+    void captureValidation();
+    // A surface of this object the fast proof found a foreign image pending over: the element
+    // (index into textures, or into storageTextures when `storage`) whose own object stands in for
+    // the full walk's lookup (refreshOwnObjects). `viewUncompressed`: a storage-sourced view under
+    // uncompressed keys, which the lookup serves by its own source whenever FindPending names it.
+    // `sourceEligible` (set by refreshOwnObjects on a resolved view): the lookup views the cached
+    // image of the surface's key, i.e. the view's source, when FindPending names nothing.
+    struct PendingOverlap {
+        std::size_t element;
+        bool storage;
+        bool viewUncompressed;
+        bool sourceEligible = false;
+    };
+    // `serialBefore` is StorageTexture::PendingSerial() loaded before any check of this Revalidate;
+    // `refreshed` the surfaces refreshOwnObjects resolved earlier in the same Revalidate (empty on
+    // the first run): a foreign image still pending over one of them (the own image's alias, an
+    // image whose pending units lie outside the surface) is no failure, since the lookup's answer
+    // for the surface is the own object its Refresh just brought current, whatever the registry
+    // holds; `reason` receives why the proof failed (Count: it passed), `overlapping` the surfaces
+    // of a Pending failure in element order (empty otherwise), `accepted` whether a foreign overlap
+    // was accepted by either rule (the verify switch then runs the full walk beside the proof).
+    bool fastRevalidate(std::uint64_t serialBefore, std::span<const PendingOverlap> refreshed, FastFail& reason, std::vector<PendingOverlap>& overlapping, bool& accepted);
+    // T1 (design_cpu_final M3): brings the overlapping surfaces' own objects up to date exactly as
+    // the full walk's lookups would, in the walk's binding order, without the lookups: a storage
+    // image's Refresh (which stores the foreign images over its surface first), a view's source's
+    // Refresh when the lookup would refresh the cached image of the view's surface key, nothing
+    // when FindPending names the source itself (the lookup hits on it). Count, or the fallback
+    // reason: a surface whose lookup could return another object (a snapshot texture, a view under
+    // fast-clear keys, a foreign pending image the lookup would view, a surface key mapping to
+    // another cached image, a surface outside the host imports, an object no longer cached), which
+    // leaves the whole object to the full walk. Under GuestMemory::GpuMutex, as Revalidate is.
+    OwnRefreshFallback refreshOwnObjects(std::span<const CompiledShader> shaders, std::span<PendingOverlap> overlapping);
+    // Today's per-element walk (APS5_NO_EPOCH_REVALIDATE=1).
+    bool fastRevalidateEach();
     Context context;
     std::vector<std::uint32_t> layoutKey;
     GuestBufferMemory guestMemory;
@@ -57,13 +357,114 @@ private:
     bool usesBda = false;
     bool usesFaultBuffer = false;
     VkDescriptorSetLayout _layout = VK_NULL_HANDLE;
+    // Whether the layout is this object's own (no cache) and destroyed with it.
+    bool ownsLayout = false;
     VkDescriptorSet _set = VK_NULL_HANDLE;
-    std::unique_ptr<DescriptorAllocation> descriptors;
+    // Dedicated pool of this object's set (no cache, or a set too large for a cache pool).
+    VkDescriptorPool pool = VK_NULL_HANDLE;
+    // The cache pool the set was allocated from, freed back to it on release.
+    VkDescriptorPool cachePool = VK_NULL_HANDLE;
     std::vector<Allocation> allocations;
+    // Guest buffer elements bound read-only: each use of this object skips that many pending-write
+    // notes (counted in MarkGpuWrites for the [buffers] line).
+    std::size_t readOnlyBuffers = 0;
     std::vector<std::shared_ptr<Texture>> textures;
+    std::vector<std::shared_ptr<StorageTexture>> storageTextures;
+    std::vector<std::uint32_t> storageMips;
+    std::vector<bool> storageWritten;
     std::vector<std::shared_ptr<Sampler>> samplers;
+    bool reusable = false;
+    std::vector<DirectRegion> directRegions;
+    std::vector<ValidatedSurface> validatedTextures;
+    // The pending registry's serial at the last Revalidate that proved this object, taken before
+    // its checks and kept only when unchanged after them (0: none): while it is still the serial,
+    // no image other than the object's own sources has results pending over its surfaces and
+    // regions, so those checks are skipped (see fastRevalidate).
+    std::uint64_t pendingSerialSeen = 0;
+    // The import table's identity when the direct regions' serials were last proved.
+    HostImportsProof importsProof;
+    // FNV-1a offset basis: the hash of no data buffers (DataWordsHash).
+    std::uint64_t dataWordsHash = 14695981039346656037ull;
+    void rehashDataWords();
+    BuildTiming timing;
+    // Build state carried from stage A to stage B: the bindings in plan order, the image bindings
+    // still to look up (index into `bindings`; the DescriptorBinding lives in the compiled shader),
+    // the descriptor counts the set was sized for, and the compute stage of a deferred build.
+    std::vector<Binding> bindings;
+    struct DeferredImages {
+        const ShaderRecompiler::DescriptorBinding* binding;
+        std::size_t index;
+    };
+    std::vector<DeferredImages> deferredImages;
+    std::uint64_t storageBuffers = 0;
+    std::uint32_t plannedSampledImages = 0;
+    std::uint32_t plannedStorageImages = 0;
+    // The compute constructor's shader and captured regions: the caller's objects, valid only until
+    // the build (Complete() for a deferred one) is done, and reset then (forgetDeferredInputs).
+    CompiledShader deferredCompute{};
+    std::span<const GuestMemorySnapshot> deferredSnapshots;
+    // The deferred build is address-based: everything runs in Complete().
+    bool lockedBuild = false;
+    // Stage A runs without the device lock (a deferred, non-address-based build).
+    bool unlockedPrepare = false;
+    bool completed = false;
+    std::chrono::steady_clock::time_point phaseStart;
 };
 
+// Built ShaderResources by the content the build consumed (see ShaderResources::ContentKey), so a
+// dispatch or recorded draw repeating an earlier one binds the earlier descriptor set instead of
+// building one (import lookups, sampler and descriptor objects, guest buffer uploads). Only reusable
+// objects (nothing to complete, every buffer read in place) are kept, and an entry serves a later
+// use only after ShaderResources::Revalidate brought it up to date. Entries hold their object alive;
+// the recorder keeps it as well while its batch runs. One LRU bound covers dispatch and draw entries
+// together (APS5_RESOURCE_CACHE_ENTRIES, default 1024); the map has its own mutex and locks itself:
+// VulkanDevice::PrepareDispatch probes it without GuestMemory::GpuMutex (a hit there is revalidated
+// under the lock by the dispatch; inserts and Revalidate stay under it). Compute keys start with the
+// stage, draw keys with a marker word, so the two never meet, and both name the device handle: the driver replaces
+// the headless device with the windowed one while workers may still hold the old one, so a replaced
+// device's entries (whose descriptor sets and pooled buffers belong to it) are unreachable by key
+// and dropped by that device's Clear(). APS5_NO_RESOURCE_CACHE=1 (dispatches) and
+// APS5_NO_DRAW_RESOURCE_CACHE=1 (draws) build every object as before; the device clears the cache at
+// teardown before its descriptor caches go.
+class ResourceCache {
+public:
+    using Key = std::vector<std::uint32_t>;
+    std::shared_ptr<ShaderResources> Find(const Key& key);
+    // `evicted`, when given, receives the objects the insert displaces (the entry replaced under the
+    // key, the ones over the bound) instead of their being destroyed here: a caller under the GPU
+    // mutex hands them to the recorder so the destruction runs off the lock (VulkanDevice::dispatch).
+    void Insert(const Key& key, std::shared_ptr<ShaderResources> resources, std::vector<std::shared_ptr<ShaderResources>>* evicted = nullptr);
+    // With `object`, erases the entry only while it still holds that object (a replacement made
+    // meanwhile by another worker stays).
+    void Remove(const Key& key, const ShaderResources* object = nullptr);
+    // Moves the entry to the front of the LRU (a recipe hit uses its template without Find, and a
+    // hot template must not age out under the recipes that depend on it); false when no entry.
+    bool Touch(const Key& key);
+    // Find and Touch calls so far (the [rescache] line: a recipe hit touches instead of finding).
+    static std::uint64_t Finds();
+    static std::uint64_t Touches();
+    void Clear();
+    std::size_t Size() const;
+
+private:
+    void erase(const Key& key);
+    void noteMiss(const Key& key);
+    struct KeyHash {
+        std::size_t operator()(const Key& key) const noexcept {
+            std::uint64_t hash = 14695981039346656037ull;
+            for (const auto word : key) hash = (hash ^ word) * 1099511628211ull;
+            return static_cast<std::size_t>(hash);
+        }
+    };
+    mutable std::mutex mutex;
+    std::list<std::pair<Key, std::shared_ptr<ShaderResources>>> entries;
+    std::unordered_map<Key, decltype(entries)::iterator, KeyHash> index;
+};
+
+// The process-wide cache the device and the draw path share (keys name the device; a device clears
+// it when it goes). Never destroyed: entries belong to the device, not to static teardown.
+ResourceCache& SharedResourceCache();
+
 }
 
 #endif

core/libs/prx/libSceAgcDriver/Graphics/include/State.hpp

@@ -4,9 +4,11 @@
 #include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/ColorTargetLayout.hpp"
 #include "prx/libSceAgcDriver/Execution/include/QueueState.hpp"
+#include <vector>
 #include <array>
 #include <cstddef>
 #include <cstdint>
+#include <string>
 #include "Recompiler.hpp"
 
 namespace AgcDriver::Graphics {
@@ -34,17 +36,25 @@ struct ColorTarget {
     std::size_t bytes;
     std::uint8_t componentMapping;
     ColorTileMode tileMode = ColorTileMode::Linear;
+    std::uint32_t elementBytes = 4;
+    // DCC metadata of a compressed target (CB_COLOR_INFO DCC_ENABLE), or 0 (see DccMetadata.hpp).
+    std::uint64_t dccAddress = 0;
+    bool dccAlphaOnMsb = false;
 };
 
 struct State {
     ShaderStages stages;
+    // MRT slot 0; `colors`/`blends` hold every written slot, attachment i being slot i.
     ColorTarget color;
+    std::vector<ColorTarget> colors;
+    std::vector<VkPipelineColorBlendAttachmentState> blends;
     bool hasColorTarget;
     bool rectList = false;
     VkExtent2D renderExtent;
     VkPrimitiveTopology topology;
     VkViewport viewport;
     bool negativeOneToOne;
+    bool depthClamp = false;
     VkRect2D scissor;
     VkCullModeFlags cullMode;
     VkFrontFace frontFace;
@@ -54,6 +64,64 @@ struct State {
 
 ShaderStages DecodeShaderStages(const QueueState& queue);
 State DecodeState(const QueueState& queue);
+// The message DecodeState (or the pixel stage decode after it) would throw for the register rules
+// this precheck covers, evaluated without exceptions before the draw is decoded; empty when they
+// pass (DecodeState still checks everything). A register a rule needs that is absent is no verdict.
+std::string DrawRejection(const QueueState& queue, bool indexed);
+
+// The recording facade of the draw decoders (design_cpu_final M8, step 8a): every register read
+// of DecodeShaderStages, DecodeState, DrawRejection, DecodePixelStageInfo and Driver::draw's
+// program prepare goes through NoteRegisterRead, which appends the bank and offset to the calling
+// thread's log while one is set (null in production; Driver::draw sets one on a miss under
+// APS5_VERIFY_DRAW_RECIPE=1 and checks the log against DrawKeyRegisters).
+enum class RegisterBank : std::uint8_t { Context, Shader, UserConfig, Count };
+struct RegisterRead {
+    RegisterBank bank;
+    std::uint32_t offset;
+};
+std::vector<RegisterRead>*& RegisterReadLog();
+inline void NoteRegisterRead(RegisterBank bank, std::uint32_t offset) {
+    if (auto* log = RegisterReadLog()) log->push_back({bank, offset});
+}
+inline const char* RegisterBankName(RegisterBank bank) {
+    return bank == RegisterBank::Context ? "context" : bank == RegisterBank::Shader ? "shader" : "user-config";
+}
+
+// The frozen table of the registers the draw decoders read, as [first, first + count) per bank in
+// bank and offset order: the draw key (Driver::draw) hashes every present register of these
+// ranges with its offset, so the decoded state, pixel and program inputs are a pure function of
+// the key. A range covers whole register groups (the eight CB_COLOR slots, the 32 user words of
+// each bank, the interpolator controls) whether or not a draw's slot count or user count reaches
+// them: a stricter key, never a wrong one. A register the decoders read that the table lacks would
+// be a wrong hit, which the facade's log finds (DrawKeyCovers) on every verified miss.
+struct DrawKeyRange {
+    RegisterBank bank;
+    std::uint32_t first;
+    std::uint32_t count;
+};
+inline constexpr std::array<DrawKeyRange, 38> DrawKeyRegisters{{
+    // PA_SC_SCREEN_SCISSOR, the window offset/scissor and clip rect, the edge rule, the hardware
+    // screen offset, CB_TARGET_MASK/CB_SHADER_MASK, the generic and viewport 0 scissors, the
+    // viewport 0 depth clamp, the blend constants, the viewport 0 transform.
+    {RegisterBank::Context, 0x00c, 2}, {RegisterBank::Context, 0x080, 4}, {RegisterBank::Context, 0x08c, 4}, {RegisterBank::Context, 0x090, 2}, {RegisterBank::Context, 0x094, 2}, {RegisterBank::Context, 0x0b4, 2}, {RegisterBank::Context, 0x105, 4}, {RegisterBank::Context, 0x10f, 6},
+    // SPI_PS_INPUT_CNTL_0..31, SPI_PS_INPUT_ENA/ADDR, SPI_PS_IN_CONTROL, SPI_SHADER_POS/Z/COL_FORMAT,
+    // CB_BLEND0..7_CONTROL, GE_MAX_OUTPUT_PER_SUBGROUP.
+    {RegisterBank::Context, 0x191, 32}, {RegisterBank::Context, 0x1b3, 2}, {RegisterBank::Context, 0x1b6, 1}, {RegisterBank::Context, 0x1c3, 3}, {RegisterBank::Context, 0x1e0, 8}, {RegisterBank::Context, 0x1ff, 1},
+    // DB_DEPTH_CONTROL .. PA_CL_VS_OUT_CNTL, PA_SC_MODE_CNTL_0/1, VGT_GS_MODE, VGT_GS_VERT_ITEMSIZE,
+    // VGT_SHADER_STAGES_EN/GS_ONCHIP, VGT_TF_PARAM/DB_ALPHA_TO_MASK, PA_SC_AA_CONFIG and
+    // PA_SU_VTX_CNTL, the sample masks, PA_SC_CONSERVATIVE_RASTERIZATION_CNTL.
+    {RegisterBank::Context, 0x200, 8}, {RegisterBank::Context, 0x292, 2}, {RegisterBank::Context, 0x29b, 1}, {RegisterBank::Context, 0x2ce, 1}, {RegisterBank::Context, 0x2d5, 2}, {RegisterBank::Context, 0x2db, 2}, {RegisterBank::Context, 0x2f8, 2}, {RegisterBank::Context, 0x30e, 2}, {RegisterBank::Context, 0x313, 1},
+    // CB_COLOR0..7_BASE .. DCC_BASE (15 words a slot), CB_COLOR0..7_BASE_EXT, DCC_BASE_EXT, ATTRIB2, ATTRIB3.
+    {RegisterBank::Context, 0x318, 0x78}, {RegisterBank::Context, 0x390, 8}, {RegisterBank::Context, 0x3a8, 0x18},
+    // The pixel program address, RSRC2 and user words; the geometry-back user pointer and program
+    // address; the vertex/geometry-front RSRC1/RSRC2 and user words; the vertex program address;
+    // the hull user pointer, program address, RSRC2 and user words; the local program address.
+    {RegisterBank::Shader, 0x008, 0x24}, {RegisterBank::Shader, 0x082, 2}, {RegisterBank::Shader, 0x088, 2}, {RegisterBank::Shader, 0x08a, 0x22}, {RegisterBank::Shader, 0x0c8, 2}, {RegisterBank::Shader, 0x102, 2}, {RegisterBank::Shader, 0x108, 2}, {RegisterBank::Shader, 0x10b, 0x21}, {RegisterBank::Shader, 0x148, 2},
+    // GE_PRIM_TYPE, GE_MULTI_PRIM_IB_RESET_EN, the geometry subgroup sizes.
+    {RegisterBank::UserConfig, 0x242, 1}, {RegisterBank::UserConfig, 0x24b, 1}, {RegisterBank::UserConfig, 0x25b, 1},
+}};
+// Whether DrawKeyRegisters holds the read.
+bool DrawKeyCovers(RegisterRead read);
 
 }
 

core/libs/prx/libSceAgcDriver/Graphics/include/Texture.hpp

@@ -2,37 +2,452 @@
 #define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_TEXTURE_HPP
 
 #include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/DccMetadata.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/GuestTextureResource.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/TextureDetiler.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/TextureTiling.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/UnitShadow.hpp"
+#include <array>
+#include <atomic>
+#include <chrono>
+#include <cstddef>
+#include <map>
+#include <memory>
+#include <span>
+#include <utility>
+#include <vector>
 
 namespace AgcDriver::Graphics {
 
 class ResidentColor;
 class CommandBatch;
+class StorageTexture;
+struct HostImport;
+
+// The Vulkan format storage images of a guest format use (sRGB formats store as their UNORM form).
+// Throws when the format has no storage form; the query is cached per format (a lookup cost a
+// vkGetPhysicalDeviceFormatProperties call before).
+VkFormat StorageFormatForGuest(const Context& context, std::uint32_t guestFormat);
+// Whether StorageFormatForGuest would succeed, without throwing (unknown guest formats included).
+bool StorageFormatAvailable(const Context& context, std::uint32_t guestFormat);
+// Whether a storage image of the guest format takes DCC clear `keys` as a GPU clear (see
+// StorageTexture::upload); false for integer formats and non-clear keys.
+bool StorageClearAvailable(const Context& context, std::uint32_t guestFormat, DccKeys keys);
+
+// A sampled texture's own VkImage with its memory, shared with the recorder while a recorded upload
+// still writes it (see the snapshot constructor), so the texture may go before the batch completes.
+struct OwnedImage {
+    OwnedImage(const Context& context, VkImage image, VkDeviceMemory memory) : context(context), image(image), memory(memory) {}
+    OwnedImage(const OwnedImage&) = delete;
+    OwnedImage& operator=(const OwnedImage&) = delete;
+    ~OwnedImage() {
+        if (image) context.Function<PFN_vkDestroyImage>("vkDestroyImage")(context.device, image, nullptr);
+        if (memory) context.Function<PFN_vkFreeMemory>("vkFreeMemory")(context.device, memory, nullptr);
+    }
+    Context context;
+    VkImage image;
+    VkDeviceMemory memory;
+};
 
 class Texture {
 public:
     Texture(const Context& context, TextureDetiler& detiler, const GuestTextureResource& descriptor, VkComponentMapping components, std::span<const std::byte> snapshot);
     Texture(const Context& context, const std::shared_ptr<ResidentColor>& source, const GuestTextureResource& descriptor, VkComponentMapping components);
+    // A view of a storage image's own VkImage: the sampled texture follows the image's content, so a
+    // compute pass writing it and the next pass sampling it share one image and copy nothing.
+    // CanCopyFrom says whether the two descriptors address the same surface compatibly.
+    Texture(const Context& context, const std::shared_ptr<StorageTexture>& source, const GuestTextureResource& descriptor, VkComponentMapping components);
+    static bool CanCopyFrom(const StorageTexture& source, const GuestTextureResource& descriptor);
     ~Texture();
     Texture(const Texture&) = delete;
     Texture& operator=(const Texture&) = delete;
 
     VkImageView View() const;
+    // The layout the image is kept in while sampled.
+    VkImageLayout Layout() const { return layout; }
     VkDeviceSize AllocationBytes() const { return allocationBytes; }
+    // Whether this texture is a view of a storage image (no snapshot of its own).
+    bool ViewsStorageImage() const { return storageSource != nullptr; }
+    const StorageTexture* StorageSource() const { return storageSource.get(); }
+    const std::shared_ptr<StorageTexture>& SharedStorageSource() const { return storageSource; }
+    // The last key scan of the sampled surface (ProvedClearKeys), for surfaces whose keys are not
+    // the source image's own (snapshots, views over other metadata); one per cache entry, so every
+    // object binding the texture shares it. Under GuestMemory::GpuMutex only.
+    DccKeyProof& KeyProof() const { return keyProof; }
 
 private:
     void release() noexcept;
 
+    // Held by value: cached textures outlive the Context of the draw that created them.
     Context context;
+    mutable DccKeyProof keyProof;
+    // The snapshot constructor's image (null for a view of a storage image); `owned` frees it.
+    VkImage image = VK_NULL_HANDLE;
+    std::shared_ptr<OwnedImage> owned;
+    VkImageView view = VK_NULL_HANDLE;
+    VkImageLayout layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
+    VkDeviceSize allocationBytes = 0;
+    std::shared_ptr<ResidentColor> source;
+    std::shared_ptr<StorageTexture> storageSource;
+    std::unique_ptr<CommandBatch> upload;
+};
+
+// A guest texture a shader writes through a storage image. It is uploaded like a sampled texture;
+// after the GPU work completes its results are stored to guest memory (retiled, changed bytes only),
+// either at once (WriteBack) or deferred: MarkDirty keeps them on the GPU until something reads that
+// memory (FlushPending, through the GuestMemory flush hook), the image is refreshed after a CPU write,
+// or it leaves the cache. Dispatches reusing the image meanwhile skip the round trip entirely.
+class StorageTexture : public std::enable_shared_from_this<StorageTexture> {
+public:
+    StorageTexture(const Context& context, TextureDetiler& detiler, const GuestTextureResource& descriptor, std::uint32_t mipLevel);
+    ~StorageTexture();
+    StorageTexture(const StorageTexture&) = delete;
+    StorageTexture& operator=(const StorageTexture&) = delete;
+
+    VkImageView View() const;
+    // The image holds every mip of the surface; one storage view per written mip is made on demand,
+    // so successive mip writes of a chain share one image and one write-back.
+    VkImageView View(std::uint32_t mip);
+    // Render targets live in the same images: draws attach mip 0 through a view of the color
+    // buffer's format and mark the image dirty like a storage write.
+    bool Attachable() const { return attachable; }
+    VkImageView AttachmentView(VkFormat format);
+    void WriteBack();
+    // Deferred write-back (APS5_EAGER_WRITEBACK=1 stores at once instead).
+    void MarkDirty();
+    void Flush();
+    // Stores every pending image overlapping the range, except `except`; returns whether any was.
+    // Stores into host-imported memory are recorded (not waited for): a CPU reader syncs afterwards.
+    // Then the unit shadows over the range are published into their imports as `scope` says (see
+    // UnitShadow.hpp; the reason string picks the publish reason), `published` receiving whether
+    // any unit was: a reader of the import must sync for a publish as for a store. Only images
+    // with a pending unit inside the range are stored (APS5_NO_FLUSH_PRETEST=1 lists every
+    // overlapping pending image, as before).
+    static bool FlushPending(std::uint64_t address, std::size_t bytes, const StorageTexture* except = nullptr, const char* reason = "memory access", PublishScope scope = PublishScope::Whole, bool* published = nullptr);
+    static void FlushAllPending(const char* reason);
+    // See PendingSerial: a change of a surface's source outside the registry (a unit shadow
+    // retile) moves it too.
+    static void BumpPendingSerial();
+    // Whether an image with results pending, or one a FlushPending is storing right now, overlaps
+    // any of the [begin, end) ranges: one acquisition of the registry's mutex.
+    static bool AnyPendingOverlaps(std::span<const std::pair<std::uint64_t, std::uint64_t>> ranges);
+    // Moves with every change of the pending registry (an image marked dirty, stored, flushed or
+    // dropped; after the store for a flush), bumped under the registry's mutex with the change, so
+    // a reader that loads it before and after its own reads knows the registry it consulted stayed
+    // as it was, and one that finds it unchanged since an earlier read knows the registry is what
+    // it was then.
+    static std::uint64_t PendingSerial();
+    // One registry scan for several ranges (the fast Revalidate): each query's `overlaps` receives
+    // whether an image other than `except` has results pending, or is being stored right now, over
+    // [begin, end), and `found` the image FindPending would return for the range (null: none);
+    // returns whether every query's `pending` image, when given, is still that image.
+    struct PendingQuery {
+        std::uint64_t begin;
+        std::uint64_t end;
+        const StorageTexture* except;
+        const StorageTexture* pending;
+        bool overlaps;
+        const StorageTexture* found = nullptr;
+    };
+    static bool ScanPending(std::span<PendingQuery> queries);
+    // APS5_PROFILE_DRAW, for the flush hook's recorded-store attribution ([hooksync]): the pending
+    // images overlapping a CPU access, whether a store of them would leave every 64 KiB block of the
+    // access to the CPU (the write-back keep rule: a block of a pending layer stamped since the
+    // layer's generation, after an uncached collect; a generation-0 layer is stored whole), and how
+    // many of them no consumer proved current for DeadImagePresents presents. No state changes.
+    struct AccessClassification {
+        std::size_t images = 0;
+        std::size_t dead = 0;
+        std::size_t live = 0;
+        bool allCpuWritten = false;
+        // Accesses over a megabyte are not scanned.
+        bool checked = false;
+    };
+    static constexpr std::uint64_t DeadImagePresents = 2;
+    static void ClassifyAccess(std::uint64_t address, std::size_t bytes, AccessClassification& out);
+    // The flush hook's skip (C4, Recorder.cpp FlushForAccess): whether every image FlushPending
+    // would store for the access keeps every 64 KiB block of it (the keep rule of writeBackLayers:
+    // stamped since the layer's generation, decided after an uncached collect over the access), so
+    // the access reads the same bytes with or without the store and the images stay pending, dirty
+    // and at their generation, except the layers a store for the access would leave empty-handed
+    // (evictStale; `evicted` receives the images that left the registry so). False when no image
+    // overlaps (`images` receives how many do), for an access over ClassifyLimit, or with a
+    // generation-0 layer (stored whole). The decision is taken under GuestMemory::GpuMutex (a
+    // short hold, no GPU wait), the collect before it.
+    static constexpr std::size_t ClassifyLimit = 1u << 20u;
+    static bool AccessKeptByCpu(std::uint64_t address, std::size_t bytes, std::size_t* images = nullptr, std::size_t* evicted = nullptr);
+    // FlushPending's publish of the unit shadows over the range without the image stores (what it
+    // does when it lists no image, and all an access AccessKeptByCpu approved needs): whether any
+    // unit was published.
+    static bool PublishShadowsOnly(std::uint64_t address, std::size_t bytes, PublishScope scope, const char* reason = "memory access");
+    // For the [hooksync] recorded-store line: images a FlushPending stored after a hook skip of
+    // theirs (AccessKeptByCpu), of which by the hook for the same read site as the last skip; 10 s deltas.
+    struct HookSkipCounts {
+        std::uint64_t flushedAfterSkip;
+        std::uint64_t flushedAfterSkipSameSite;
+    };
+    static HookSkipCounts TakeHookSkipCounts();
+    // A consumer found the image current and used it (Refresh, the fast revalidation).
+    void NoteProved() const;
+    // Whether the storage cache holds this image (set and cleared by the cache under its mutex, the
+    // clear before the eviction's flush): what a lookup of its surface would return.
+    bool Cached() const { return cached.load(std::memory_order_acquire); }
+    void SetCached(bool value) { cached.store(value, std::memory_order_release); }
+    // The pending image whose surface starts at `address` and covers at least `bytes` (a mip chain
+    // contains a descriptor of its first mips), if any; the caller decides whether the geometry fits
+    // (Texture::CanCopyFrom, ResidentPresentable).
+    static std::shared_ptr<StorageTexture> FindPending(std::uint64_t address, std::uint64_t bytes);
+    // How a buffer fill of [address, address + bytes) (Driver.cpp's fill HLE) meets the storage
+    // images alive: none of them; a whole surface (Exact) or one array layer of a surface (Layer),
+    // `image` and `layer` naming it, whatever other images lie over the range (`others`, of which
+    // `inside` wholly inside it: stale images of earlier uses of the memory, which the title's
+    // transient allocator hands out again); otherwise part of one image (the fill inside the
+    // surface, the surface inside the fill, or straddling), several, or none but exactly one
+    // surface's DCC metadata. Images the cache let go (see Flush) do not count.
+    enum class FillCover { None, Exact, Inside, Around, Straddle, Several, Keys, Layer };
+    struct FillCoverage {
+        FillCover cover = FillCover::None;
+        std::shared_ptr<StorageTexture> image;
+        std::uint32_t layer = 0;
+        std::size_t others = 0;
+        std::size_t inside = 0;
+    };
+    static FillCoverage ClassifyFill(std::uint64_t address, std::size_t bytes);
+    // Results pending in images lying wholly inside the range are dead (a fill overwrites every
+    // byte of them): they are dropped instead of stored. Returns how many images were.
+    static std::size_t DiscardPendingInside(std::uint64_t address, std::size_t bytes);
+    // A fill of the whole surface (`layer` WholeImage) or of one array layer with the 16-byte
+    // `pattern`, done as a GPU clear of the image instead of a store into guest memory: the
+    // pattern must be one texel of the storage format repeated, exactly representable as a clear
+    // value, the surface's DCC keys (if any) must read as uncompressed and not be written by
+    // recorded work, the memory must be host-imported so the write-back that eventually publishes
+    // the texels is GPU-direct, and for one layer the rest of the surface must be unchanged since
+    // the image matched it. Records into the active recorder under GuestMemory::GpuMutex and leaves
+    // the image dirty like a shader write: the guest bytes follow through the deferred write-back,
+    // the range is stamped written so every other reader of it (other images over the memory,
+    // captures, snapshots) takes the bytes through the flush hook, and this image alone is current.
+    // False, naming why in `refusal`, when the fill must be stored as before.
+    static constexpr std::uint32_t WholeImage = ~0u;
+    bool FillClear(std::span<const std::uint32_t, 4> pattern, std::uint32_t layer, const char*& refusal);
+    // The cached, live image whose surface is exactly [address, address + bytes), if any (the newest
+    // of several: see ClassifyFill).
+    static std::shared_ptr<StorageTexture> FindLive(std::uint64_t address, std::uint64_t bytes);
+    // Whether the two surfaces differ in nothing but their address (and DCC metadata).
+    bool SameSurfaceShape(const StorageTexture& other) const;
+    // The copy HLE's buffer copy of one whole surface into another (VulkanDevice::CopyBuffer,
+    // APS5_NO_COPY_ALIAS=1 disables): the source image's content is copied on the device into this
+    // image, which is then dirty like a shader-written one (the guest bytes follow through the
+    // deferred write-back; the range is stamped so every other reader takes them through the flush
+    // hook), and the source's pending results stay pending. The shapes must match, neither surface
+    // may carry DCC metadata, and this surface must be host-imported (the write-back is GPU-direct).
+    // False, naming why, when the copy must be a transfer.
+    bool CopyFrom(StorageTexture& source, const char*& refusal);
+    VkImage Image() const { return image; }
+    const GuestTextureResource& Descriptor() const { return descriptor; }
+    std::uint32_t ImageLayers() const { return geometry.imageLayers; }
+    std::uint32_t ImageDepth() const { return geometry.imageDepth; }
+    // Content version: advances when the image is re-uploaded or a shader wrote it. Together with
+    // Generation (the write generation guest memory was last known to match the content at) it
+    // validates textures copied from this image.
+    std::uint64_t Version() const { return version; }
+    std::uint64_t Generation() const { return generation; }
+    // The DCC keys the content was uploaded under and the last scan of the surface's keys
+    // (ProvedClearKeys): Refresh's own key rule for the fast revalidation. Under
+    // GuestMemory::GpuMutex only, as Refresh is; never from a build's stage A.
+    DccKeys UploadedKeys() const { return uploadedKeys; }
+    DccKeyProof& KeyProof() const { return keyProof; }
+    // Brings the image up to date with guest memory before another use; returns whether its content
+    // was still current (nothing uploaded).
+    // Keeps the image current with guest memory (see GuestMemory::CollectWrites).
+    bool Refresh();
+    std::uint64_t GuestBytes() const;
+
+private:
+    // The regions of every array layer, or of the tracked layers `layers` selects.
+    std::vector<VkBufferImageCopy> CopyRegions(const std::vector<bool>* layers = nullptr) const;
+    // Uploads the surface, or only the tracked layers `layers` selects (the direct path; the others
+    // upload everything).
+    void upload(const std::vector<bool>* layers = nullptr);
+    // Stores the pending tracked layers overlapping [address, address + bytes) to guest memory; in
+    // each, 64 KiB blocks the CPU wrote since the layer's generation keep the CPU's bytes. Block
+    // units asked for in pieces too often are all stored at once for a while (the hysteresis:
+    // APS5_BLOCK_WRITEBACK_PIECES within APS5_BLOCK_WRITEBACK_FRAMES presents, defaults 4 and 8;
+    // APS5_BLOCK_WRITEBACK_EACH=1 stores the touched units only).
+    void writeBack(std::uint64_t address, std::size_t bytes);
+    void writeBackLayers(const std::vector<bool>& layers);
+    // Tracked units as 64 KiB write-stamp blocks (`blockUnits`: a thin tiled surface at a 64 KiB
+    // aligned base; APS5_NO_BLOCK_TRACKING=1 tracks array layers as above instead): a fill of one
+    // layer, a CPU write or another image's store then costs the blocks it touched, moved through
+    // windows of their (layer, mip) slices (whole tile blocks in whole rows, see
+    // TextureDetiler::DetileWindow), not the surface. The last unit may be short.
+    std::uint64_t layerBytes(std::uint32_t layer) const { return std::min<std::uint64_t>(trackedLayerBytes, guestBytes - static_cast<std::uint64_t>(layer) * trackedLayerBytes); }
+    // Contiguous runs of the selected units, surface-relative [begin, end).
+    std::vector<std::pair<std::uint64_t, std::uint64_t>> unitRuns(const std::vector<bool>& units) const;
+    struct SliceWindow {
+        std::uint32_t layer;
+        std::uint32_t level;
+        // The window's tiled bytes, surface-relative, and the detiler window (mip-relative) that
+        // moves them out of a linear window of `linearBytes`.
+        std::uint64_t tiledBegin;
+        std::uint64_t tiledEnd;
+        DetileWindow window;
+        std::uint64_t linearBytes;
+        // Its tile blocks as buffer-image regions, bufferOffset relative to the linear window.
+        std::vector<VkBufferImageCopy> regions;
+    };
+    // The windows moving the surface-relative runs: whole tile blocks of thin tiled mips, a tail
+    // mip whole (its block holds every tail level), in slice order.
+    std::vector<SliceWindow> sliceWindows(std::span<const std::pair<std::uint64_t, std::uint64_t>> runs) const;
+    // The direct upload and GPU-direct write-back of the runs through their windows (block units);
+    // both return the bytes moved. The upload reads each unit from its unit shadow while fresh,
+    // else from the import (`discard`: the image has no layout yet, a whole-surface first upload).
+    // The write-back retiles into the import's unit shadow where a slab takes the piece
+    // (`shadowed` receives those guest ranges, `imported` the pieces written to the import).
+    std::uint64_t uploadWindows(const HostImport& import, std::span<const std::pair<std::uint64_t, std::uint64_t>> runs, bool discard = false);
+    std::uint64_t writeBackWindows(const HostImport& import, std::span<const std::pair<std::uint64_t, std::uint64_t>> keep, std::uint64_t firstStored, std::uint64_t lastStored, std::vector<ShadowedRange>& shadowed, std::vector<std::pair<std::uint64_t, std::uint64_t>>& imported);
+    // Per-layer validity (thin array surfaces whose layers are 64 KiB-aligned guest slices; every
+    // other surface is one tracked layer): a layer's generation is the write generation its guest
+    // bytes were last known to match the image at, and a pending layer holds results (a clear, a
+    // shader write, a device copy) guest memory has not received. `generation` stays the oldest
+    // layer generation (the whole-surface fast check), `dirty` whether any layer is pending.
+    std::uint64_t layerBegin(std::uint32_t layer) const { return descriptor.baseAddress + static_cast<std::uint64_t>(layer) * trackedLayerBytes; }
+    bool anyLayerPending() const;
+    void refreshGeneration();
+    // Marks `count` tracked layers from `first` pending and registers the image (MarkDirty's
+    // registration; APS5_EAGER_WRITEBACK=1 stores at once instead).
+    void markLayersPending(std::uint32_t first, std::uint32_t count);
+    // Registers or unregisters the image as its pending layers say (after a write-back).
+    void reconcilePending();
+    // Pending images whose surface shares one of the 64 KiB write-stamp blocks firstBlock..lastBlock
+    // (the span a write-back stamps) without overlapping this one: the tracked layer of theirs
+    // holding the shared block, when its memory is unchanged since its generation (see writeBack),
+    // with the generation seen at the check.
+    struct Adjacent {
+        std::shared_ptr<StorageTexture> texture;
+        std::uint32_t layer;
+        std::uint64_t generation;
+    };
+    std::vector<Adjacent> adjacentPendingUnchanged(std::uint64_t firstBlock, std::uint64_t lastBlock) const;
+    // Advances the entries' layers still at their seen generation to `now`, the generation this
+    // write-back's stamps of blocks firstBlock..lastBlock are below, unless the layer's other
+    // blocks were stamped since the check.
+    static void advanceAdjacent(const std::vector<Adjacent>& adjacent, std::uint64_t now, std::uint64_t firstBlock, std::uint64_t lastBlock);
+    // Per 64 KiB tracker block of the surface (block 0 holds the base address): the generation of
+    // the tracked layer it belongs to, for GuestMemory::ChangedBlocks.
+    void blockGenerations(std::vector<std::uint64_t>& generations) const;
+    // Another image of exactly this surface under another storage format with results pending
+    // (block units, uncompressed, still cached), if any: Refresh takes its pending units from its
+    // image on the device (borrowUnits) instead of having them stored and re-read, and the alias
+    // stays responsible for storing them until this image's own results supersede them
+    // (markLayersPending). APS5_NO_ALIAS_BORROW=1 stores and re-reads as before.
+    std::shared_ptr<StorageTexture> pendingAlias() const;
+    // The pending images FlushPending(address, bytes) would store (not the one a Refresh validates,
+    // a pending unit inside the range), kept alive across the caller's use.
+    static std::vector<std::shared_ptr<StorageTexture>> overlappingPending(std::uint64_t address, std::size_t bytes);
+    // Whether every 64 KiB block of [address, address + bytes) inside a pending layer of the images
+    // is stamped since that layer's generation (the caller collected the range first): the keep
+    // rule of writeBackLayers over the access. A generation-0 layer is stored whole: false.
+    static bool blocksKept(std::span<const std::shared_ptr<StorageTexture>> images, std::uint64_t address, std::size_t bytes);
+    // Under GuestMemory::GpuMutex, after blocksKept approved the access: the layers a write-back for
+    // it would select (see writeBack; every pending one of an image no consumer proved current for
+    // DeadImagePresents presents with APS5_STALE_EVICT=1, H3) are left as that write-back would
+    // leave them when every one of their blocks is stamped since its generation (it would store
+    // nothing: no results pending, the generation unchanged, `original` no longer vouching).
+    // Returns whether they were; `dropped` counts the image once it left the registry.
+    bool evictStale(std::uint64_t address, std::size_t bytes, std::size_t& dropped);
+    // Whether the results the last hook skip left pending over its access (AccessKeptByCpu) are
+    // still the pending ones and a write-back for the range would store them.
+    bool skippedResultsInside(std::uint64_t address, std::size_t bytes) const;
+    std::uint64_t borrowUnits(StorageTexture& source, const std::vector<bool>& units);
+    void forgetBorrowed(std::uint32_t first, std::uint32_t count);
+    bool overlaps(std::uint64_t address, std::size_t bytes) const;
+    bool pendingUnitInside(std::uint64_t address, std::size_t bytes) const;
+    VkImageView createView(std::uint32_t mip) const;
+    void release() noexcept;
+
+    Context context;
+    TextureDetiler& detiler;
+    GuestTextureResource descriptor;
+    std::vector<TileMipLayout> mips;
+    std::uint32_t arrayLayers = 1;
+    std::uint64_t guestBytes = 0;
+    std::uint64_t sliceLinearBytes = 0;
+    SurfaceGeometry geometry;
+    std::vector<std::byte> original;
+    // DCC keys the image content was uploaded under: a fast-cleared surface starts as its clear value.
+    DccKeys uploadedKeys = DccKeys::Uncompressed;
+    mutable DccKeyProof keyProof;
+    // Write generation `original` is known current at (the oldest of layerGeneration).
+    std::uint64_t generation = 0;
+    std::uint32_t trackedLayers = 1;
+    std::uint64_t trackedLayerBytes = 0;
+    std::vector<std::uint64_t> layerGeneration;
+    std::vector<bool> layerPending;
+    bool blockUnits = false;
+    // Write-back hysteresis (block units, see writeBack): partial stores in the current window of
+    // presents, its first present, and the present until which every pending unit is stored.
+    std::uint32_t partialStores = 0;
+    std::uint64_t partialWindowStart = 0;
+    std::uint64_t storeWholeUntil = 0;
+    // Units holding the alias's results (see pendingAlias), at the alias's version then.
+    std::weak_ptr<StorageTexture> borrowedFrom;
+    std::uint64_t borrowedVersion = 0;
+    std::vector<bool> borrowedUnits;
+    // Units of this image were borrowed since it was last written: the borrower's proofs
+    // (the resource cache's serial memo) hold only while this image's version does, so the next
+    // write here bumps the pending serial although the image was pending already.
+    bool lent = false;
     VkImage image = VK_NULL_HANDLE;
     VkDeviceMemory memory = VK_NULL_HANDLE;
     VkImageView view = VK_NULL_HANDLE;
-    VkDeviceSize allocationBytes = 0;
-    std::shared_ptr<ResidentColor> source;
-    std::unique_ptr<CommandBatch> upload;
+    std::uint32_t defaultMip = 0;
+    std::map<std::uint32_t, VkImageView> extraViews;
+    bool attachable = false;
+    std::map<VkFormat, VkImageView> attachmentViews;
+    VkFormat storageFormat = VK_FORMAT_UNDEFINED;
+    // Results are on the GPU only (guarded by the pending-write registry lock).
+    bool dirty = false;
+    std::uint64_t version = 0;
+    // `original` holds the guest bytes; false after a GPU-side clear, which never read them.
+    bool originalValid = true;
+    // The storage cache let the image go (Flush): a lookup makes a new image of the surface, so
+    // no fill may put results into this one (guarded by the live-image registry lock).
+    bool released = false;
+    std::atomic<bool> cached{false};
+    // The present a consumer last proved the image current at (NoteProved).
+    mutable std::atomic<std::uint64_t> provedPresent{0};
+    // Hook skips of this image since FlushPending last listed it (AccessKeptByCpu), the read site
+    // of the last one (a GuestMemory::ReadSite), and the first pending layer under its access with
+    // the version and layer generation seen: a listing counts a flush after a skip only while that
+    // layer is pending with those results (skippedResultsInside), for TakeHookSkipCounts.
+    std::atomic<std::uint32_t> hookSkips{0};
+    std::atomic<std::uint8_t> hookSkipSite{0};
+    std::uint32_t hookSkipLayer = 0;
+    std::uint64_t hookSkipVersion = 0;
+    std::uint64_t hookSkipGeneration = 0;
+    // The present evictStale last walked this image's layers beyond the access (once per present).
+    std::uint64_t staleCheckPresent = ~0ull;
+    // The generation at which a one-layer FillClear was refused as 'surface changed': the check
+    // fails again until `generation` moves, so ClassifyFill skips the layer match meanwhile.
+    std::uint64_t layerRefusedGeneration = ~0ull;
 };
 
+// APS5_PROFILE_DRAW: what the image lookups on this thread did since the last reset, by outcome
+// (count and time), so a caller running them under the GPU mutex can name what its hold spent on
+// (VulkanDevice's [indirect] line resets and reads it around stage B). The sampled rows are per
+// element (a fast miss is followed by a full lookup); the refresh, upload, DCC scan and pending
+// flush rows lie inside the storage and sampled rows.
+struct LookupOutcomes {
+    enum Kind : std::size_t { SampledFast, SampledFastMiss, SampledHitView, SampledHitClearedView, SampledHitSnapshot, SampledMadeView, SampledMadeSnapshot, StorageHit, StorageMade, RefreshUnchanged, RefreshCompared, UploadDirect, UploadCpu, UploadClear, DccScan, PendingFlush, Count };
+    static const char* Name(Kind kind);
+    static bool Profiled();
+    // Charges the time since `start` to `kind` on this thread and returns now.
+    static std::chrono::steady_clock::time_point Add(Kind kind, std::chrono::steady_clock::time_point start);
+    std::array<std::uint64_t, Count> counts{};
+    std::array<double, Count> ms{};
+};
+LookupOutcomes& ThreadLookupOutcomes();
+
 }
 
 #endif

core/libs/prx/libSceAgcDriver/Graphics/include/TextureDetiler.hpp

@@ -1,6 +1,7 @@
 #ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_TEXTUREDETILER_HPP
 #define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_TEXTUREDETILER_HPP
 
+#include <array>
 #include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/GuestTextureResource.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/TextureTiling.hpp"
@@ -10,6 +11,26 @@
 
 namespace AgcDriver::Graphics {
 
+    // The part of a mip one Dispatch moves: only elements whose tiled offset (relative to the mip's
+    // tiled base; the linear offset of a linear surface) lies in [rangeBegin, rangeEnd), with the
+    // tiled buffer holding the mip from tiledBase and the linear buffer holding `linearBytes` of
+    // it from linearBase (a window of whole rows: linearBase is a row's start, every element in
+    // the range lies inside the window; 0 for the rest of the mip). The element rectangle
+    // [columnBegin, columnEnd) x [rowBegin, rowEnd) (an end of 0: the mip's edge) bounds the
+    // elements the range can hold: only its workgroups are dispatched, the range check stays the
+    // guard. The default moves the whole mip out of whole buffers.
+    struct DetileWindow {
+        std::uint32_t rangeBegin = 0;
+        std::uint32_t rangeEnd = 0xffffffffu;
+        std::uint32_t tiledBase = 0;
+        std::uint32_t linearBase = 0;
+        std::uint64_t linearBytes = 0;
+        std::uint32_t columnBegin = 0;
+        std::uint32_t columnEnd = 0;
+        std::uint32_t rowBegin = 0;
+        std::uint32_t rowEnd = 0;
+    };
+
     class TextureDetiler {
     public:
         explicit TextureDetiler(const Context& context);
@@ -17,11 +38,16 @@ namespace AgcDriver::Graphics {
         TextureDetiler(const TextureDetiler&) = delete;
         TextureDetiler& operator=(const TextureDetiler&) = delete;
 
-        void Dispatch(VkCommandBuffer commands, TextureTileMode tileMode, std::uint32_t elementBytes, VkBuffer source, std::uint64_t sourceOffset, VkBuffer destination, std::uint64_t destinationOffset, const TileMipLayout& layout, std::uint32_t arrayLayer);
+        // Detiles `source` (tiled) into `destination` (linear), or with `retile` writes linear `source`
+        // into tiled `destination`; offsets always refer to the respective buffers, and `window`
+        // restricts the move to part of the mip (a partial upload or write-back, StorageTexture),
+        // leaving the other elements of both buffers alone.
+        void Dispatch(VkCommandBuffer commands, TextureTileMode tileMode, std::uint32_t elementBytes, VkBuffer source, std::uint64_t sourceOffset, VkBuffer destination, std::uint64_t destinationOffset, const TileMipLayout& layout, bool retile = false, std::uint32_t slice = 0, bool thick = false, const DetileWindow& window = {});
+        // Recycles the descriptor sets of the previous batch; call before recording a new command batch.
         void BeginBatch();
 
     private:
-        VkPipeline pipeline(TextureTileMode tileMode, std::uint32_t elementBytes);
+        VkPipeline pipeline(TextureTileMode tileMode, std::uint32_t elementBytes, bool retile, bool thick);
         void release() noexcept;
         VkDescriptorSet allocateSet();
 

core/libs/prx/libSceAgcDriver/Graphics/include/TextureFormat.hpp

@@ -6,10 +6,13 @@
 #endif
 #include <vulkan/vulkan.h>
 #include <cstdint>
+#include <optional>
 
 namespace AgcDriver::Graphics {
 
 VkFormat ResolveTextureFormat(std::uint32_t guestFormat);
+std::optional<std::uint32_t> FindGuestTextureFormat(VkFormat format, std::uint32_t elementBytes);
+std::optional<std::uint32_t> FindGuestColorTargetFormat(VkFormat format, std::uint32_t elementBytes);
 std::uint32_t BytesPerElement(std::uint32_t guestFormat);
 bool IsBlockCompressed(std::uint32_t guestFormat);
 std::uint32_t BlockWidth(std::uint32_t guestFormat);

core/libs/prx/libSceAgcDriver/Graphics/include/TextureSwizzleEquations.hpp

@@ -0,0 +1,63 @@
+#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_TEXTURESWIZZLEEQUATIONS_HPP
+#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_TEXTURESWIZZLEEQUATIONS_HPP
+
+#include <array>
+#include <cstdint>
+
+namespace AgcDriver::Graphics {
+
+// Address equations of the 64 KiB XOR swizzle modes (SW_64KB_Z_X, S_X, D_X, R_X) for the PS5 GPU, a
+// Navi1x-class part with 16 pipes. Generated from AMD addrlib's GFX10 non-RB+ swizzle pattern tables
+// (the 32 bpp R_X entry reproduces the scanout layout in ColorTargetLayout).
+// Each entry gives, for byte-address bit 0..15 inside a block, the element-coordinate bits XORed into it:
+// bits 0-11 select x bits, bits 12-23 select y bits and bits 24-31 select slice (or depth) bits.
+struct TextureSwizzleEquation {
+    std::uint32_t swizzleMode;
+    std::uint32_t elementBytes;
+    std::array<std::uint32_t, 16> bits;
+};
+
+inline constexpr TextureSwizzleEquation kTextureSwizzleEquations[] = {
+    {24u, 1u, {0x00000001u, 0x00001000u, 0x00000002u, 0x00002000u, 0x00000004u, 0x00004000u, 0x00000008u, 0x00010000u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00040000u, 0x00000040u, 0x00080000u, 0x00000080u}},
+    {24u, 2u, {0x00000000u, 0x00000001u, 0x00001000u, 0x00000002u, 0x00002000u, 0x00000004u, 0x00004000u, 0x00000008u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00010000u, 0x00000040u, 0x00040000u, 0x00000080u}},
+    {24u, 4u, {0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x00000002u, 0x00002000u, 0x00000004u, 0x00004000u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00008000u, 0x00000010u, 0x00040000u, 0x00000040u}},
+    {24u, 8u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x00000002u, 0x00002000u, 0x00000004u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00004000u, 0x00000008u, 0x00010000u, 0x00000040u}},
+    {24u, 16u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x00000002u, 0x00002000u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00004000u, 0x00000004u, 0x00008000u, 0x00000010u}},
+    {25u, 1u, {0x00000001u, 0x00000002u, 0x00000004u, 0x00000008u, 0x00001000u, 0x00002000u, 0x00004000u, 0x00008000u, 0x08010080u, 0x04080010u, 0x02020040u, 0x01040020u, 0x00040000u, 0x00000040u, 0x00080000u, 0x00000080u}},
+    {25u, 2u, {0x00000000u, 0x00000001u, 0x00000002u, 0x00000004u, 0x00001000u, 0x00002000u, 0x00004000u, 0x00000008u, 0x08008080u, 0x04040010u, 0x02010040u, 0x01020020u, 0x00020000u, 0x00000040u, 0x00040000u, 0x00000080u}},
+    {25u, 4u, {0x00000000u, 0x00000000u, 0x00000001u, 0x00000002u, 0x00001000u, 0x00002000u, 0x00004000u, 0x00000004u, 0x08008040u, 0x04040008u, 0x02010020u, 0x01020010u, 0x00020000u, 0x00000020u, 0x00040000u, 0x00000040u}},
+    {25u, 8u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x00002000u, 0x00000002u, 0x00000004u, 0x08004040u, 0x04020008u, 0x02008020u, 0x01010010u, 0x00010000u, 0x00000020u, 0x00020000u, 0x00000040u}},
+    {25u, 16u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00001000u, 0x00002000u, 0x00000001u, 0x00000002u, 0x08004020u, 0x04020004u, 0x02008010u, 0x01010008u, 0x00010000u, 0x00000010u, 0x00020000u, 0x00000020u}},
+    {26u, 1u, {0x00000001u, 0x00000002u, 0x00000004u, 0x00002000u, 0x00001000u, 0x00004000u, 0x00000008u, 0x00008000u, 0x08010080u, 0x04080010u, 0x02020040u, 0x01040020u, 0x00040000u, 0x00000040u, 0x00080000u, 0x00000080u}},
+    {26u, 2u, {0x00000000u, 0x00000001u, 0x00000002u, 0x00000004u, 0x00001000u, 0x00002000u, 0x00004000u, 0x00000008u, 0x08008080u, 0x04040010u, 0x02010040u, 0x01020020u, 0x00020000u, 0x00000040u, 0x00040000u, 0x00000080u}},
+    {26u, 4u, {0x00000000u, 0x00000000u, 0x00000001u, 0x00000002u, 0x00001000u, 0x00002000u, 0x00004000u, 0x00000004u, 0x08008040u, 0x04040008u, 0x02010020u, 0x01020010u, 0x00020000u, 0x00000020u, 0x00040000u, 0x00000040u}},
+    {26u, 8u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x00000002u, 0x00000004u, 0x00002000u, 0x08004040u, 0x04020008u, 0x02008020u, 0x01010010u, 0x00010000u, 0x00000020u, 0x00020000u, 0x00000040u}},
+    {26u, 16u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x00000002u, 0x00002000u, 0x08004020u, 0x04020004u, 0x02008010u, 0x01010008u, 0x00010000u, 0x00000010u, 0x00020000u, 0x00000020u}},
+    {27u, 1u, {0x00000001u, 0x00000002u, 0x00000004u, 0x00002000u, 0x00001000u, 0x00004000u, 0x00000008u, 0x00010000u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00040000u, 0x00000040u, 0x00080000u, 0x00000080u}},
+    {27u, 2u, {0x00000000u, 0x00000001u, 0x00000002u, 0x00000004u, 0x00001000u, 0x00002000u, 0x00004000u, 0x00000008u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00010000u, 0x00000040u, 0x00040000u, 0x00000080u}},
+    {27u, 4u, {0x00000000u, 0x00000000u, 0x00000001u, 0x00000002u, 0x00001000u, 0x00002000u, 0x00004000u, 0x00000004u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00008000u, 0x00000010u, 0x00040000u, 0x00000040u}},
+    {27u, 8u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x00000002u, 0x00000004u, 0x00002000u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00004000u, 0x00000008u, 0x00010000u, 0x00000040u}},
+    {27u, 16u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x00000002u, 0x00002000u, 0x08008008u, 0x04010010u, 0x02020040u, 0x01040020u, 0x00004000u, 0x00000004u, 0x00008000u, 0x00000010u}},
+    // Thick (3D) standard swizzles SW_4KB_S and SW_64KB_S use the S3 patterns, keyed 0x100 | SW_MODE.
+    {0x105u, 1u, {0x00000001u, 0x00000002u, 0x01000000u, 0x00001000u, 0x02000000u, 0x00002000u, 0x00000004u, 0x04000000u, 0x00004000u, 0x00000008u, 0x08000000u, 0x00008000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u}},
+    {0x105u, 2u, {0x00000000u, 0x00000001u, 0x01000000u, 0x00001000u, 0x02000000u, 0x00002000u, 0x00000002u, 0x04000000u, 0x00004000u, 0x00000004u, 0x08000000u, 0x00008000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u}},
+    {0x105u, 4u, {0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x01000000u, 0x00002000u, 0x00000002u, 0x02000000u, 0x00004000u, 0x00000004u, 0x04000000u, 0x00008000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u}},
+    {0x105u, 8u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x01000000u, 0x00001000u, 0x00000002u, 0x02000000u, 0x00002000u, 0x00000004u, 0x04000000u, 0x00004000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u}},
+    {0x105u, 16u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x01000000u, 0x00001000u, 0x00000001u, 0x02000000u, 0x00002000u, 0x00000002u, 0x04000000u, 0x00004000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u}},
+    {0x109u, 1u, {0x00000001u, 0x00000002u, 0x01000000u, 0x00001000u, 0x02000000u, 0x00002000u, 0x00000004u, 0x04000000u, 0x00004000u, 0x00000008u, 0x08000000u, 0x00008000u, 0x00000010u, 0x10000000u, 0x00010000u, 0x00000020u}},
+    {0x109u, 2u, {0x00000000u, 0x00000001u, 0x01000000u, 0x00001000u, 0x02000000u, 0x00002000u, 0x00000002u, 0x04000000u, 0x00004000u, 0x00000004u, 0x08000000u, 0x00008000u, 0x00000008u, 0x10000000u, 0x00010000u, 0x00000010u}},
+    {0x109u, 4u, {0x00000000u, 0x00000000u, 0x00000001u, 0x00001000u, 0x01000000u, 0x00002000u, 0x00000002u, 0x02000000u, 0x00004000u, 0x00000004u, 0x04000000u, 0x00008000u, 0x00000008u, 0x08000000u, 0x00010000u, 0x00000010u}},
+    {0x109u, 8u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000001u, 0x01000000u, 0x00001000u, 0x00000002u, 0x02000000u, 0x00002000u, 0x00000004u, 0x04000000u, 0x00004000u, 0x00000008u, 0x08000000u, 0x00008000u, 0x00000010u}},
+    {0x109u, 16u, {0x00000000u, 0x00000000u, 0x00000000u, 0x00000000u, 0x01000000u, 0x00001000u, 0x00000001u, 0x02000000u, 0x00002000u, 0x00000002u, 0x04000000u, 0x00004000u, 0x00000004u, 0x08000000u, 0x00008000u, 0x00000008u}},
+};
+
+inline const TextureSwizzleEquation* FindTextureSwizzleEquation(std::uint32_t swizzleMode, std::uint32_t elementBytes) {
+    for (const auto& equation : kTextureSwizzleEquations) {
+        if (equation.swizzleMode == swizzleMode && equation.elementBytes == elementBytes) return &equation;
+    }
+    return nullptr;
+}
+
+}
+
+#endif

core/libs/prx/libSceAgcDriver/Graphics/include/TextureTiling.hpp

@@ -2,6 +2,7 @@
 #define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_TEXTURETILING_HPP
 
 #include "prx/libSceAgcDriver/Graphics/include/GuestTextureResource.hpp"
+#include <array>
 #include <cstdint>
 #include <vector>
 
@@ -23,6 +24,47 @@ struct TileMipLayout {
 
 std::vector<TileMipLayout> ComputeMipLayout(TextureTileMode tileMode, std::uint32_t format, std::uint32_t width, std::uint32_t height, std::uint32_t mipCount);
 std::uint64_t ComputeSurfaceSize(const std::vector<TileMipLayout>& mips, std::uint32_t arrayLayers);
+// Mip chain of an uncompressed surface given its element size (color targets have no texture format).
+std::vector<TileMipLayout> ComputeElementMipLayout(TextureTileMode tileMode, std::uint32_t bytesPerElement, std::uint32_t width, std::uint32_t height, std::uint32_t mipCount);
+
+// Thick (3D) SW_4KB_S / SW_64KB_S blocks span several depth slices: {width, height, depth} in elements.
+std::array<std::uint32_t, 3> ThickBlockExtent(TextureTileMode tileMode, std::uint32_t bytesPerElement);
+// Thin (2D) tiled blocks: {block bytes, width, height} in elements for the tile mode and element size
+// (the detiler's blockExtent). A mip's tiled bytes are its blocks in row-major order, blocksPerRow per
+// row (TileMipLayout), except the tail mips, which share one block. Throws for linear tiling.
+std::array<std::uint32_t, 3> ThinBlockLayout(TextureTileMode tileMode, std::uint32_t bytesPerElement);
+
+// Single-mip 3D surface. Each depth slice z is detiled from slab z / blockDepth (slabBytes apart) with the
+// swizzle's slice input set to z, into linear slices sliceLinearBytes apart.
+struct ThickLayout {
+    TileMipLayout mip;
+    std::uint32_t depth;
+    std::uint32_t blockDepth;
+    std::uint64_t slabBytes;
+    std::uint64_t sliceLinearBytes;
+    std::uint64_t guestBytes;
+};
+ThickLayout ComputeThickLayout(TextureTileMode tileMode, std::uint32_t format, std::uint32_t width, std::uint32_t height, std::uint32_t depth);
+
+// Guest surface as the texture upload/write-back loops walk it: array layers of thin surfaces, or the
+// depth slices of a volume (which Vulkan holds as one layer with extent depth).
+struct SurfaceGeometry {
+    std::vector<TileMipLayout> mips;
+    std::uint32_t layers = 1;
+    std::uint32_t imageLayers = 1;
+    std::uint32_t imageDepth = 1;
+    std::uint64_t guestBytes = 0;
+    std::uint64_t sliceLinearBytes = 0;
+    bool thick = false;
+    std::uint32_t blockDepth = 1;
+    std::uint64_t layerBytes = 0;
+
+    std::uint64_t GuestLayerOffset(std::uint32_t layer) const { return thick ? static_cast<std::uint64_t>(layer / blockDepth) * layerBytes : static_cast<std::uint64_t>(layer) * layerBytes; }
+    std::uint64_t LinearLayerOffset(std::uint32_t layer) const { return static_cast<std::uint64_t>(layer) * sliceLinearBytes; }
+    std::uint32_t CopyLayer(std::uint32_t layer) const { return imageDepth > 1 ? 0u : layer; }
+    std::int32_t CopyDepth(std::uint32_t layer) const { return imageDepth > 1 ? static_cast<std::int32_t>(layer) : 0; }
+};
+SurfaceGeometry DescribeSurface(const GuestTextureResource& descriptor);
 
 }
 

core/libs/prx/libSceAgcDriver/Graphics/include/UnitShadow.hpp

@@ -0,0 +1,155 @@
+#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_UNITSHADOW_HPP
+#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_GRAPHICS_INCLUDE_UNITSHADOW_HPP
+
+#include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
+#include <atomic>
+#include <cstddef>
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <optional>
+#include <span>
+#include <string>
+#include <utility>
+#include <vector>
+
+namespace AgcDriver::Graphics {
+
+struct HostImport;
+
+// Device-local tiled shadows of host imports, indexed by 64 KiB unit (one unit = one write-tracker
+// block; unit index of guest address a inside a shadow = (a - base) / 65536 with base = the import's
+// base rounded down to 64 KiB). A storage image's GPU-direct write-back retiles its pending units into
+// the shadow instead of the import, and a later upload of any image over those units detiles from the
+// shadow: the title's aliased images then exchange their results in device memory, and only a host-
+// facing consumer of the bytes (the flush hook, a V# bound in place, indirect arguments, a copy, a
+// label or key store over the range, scanout, the import's retirement) publishes them into the
+// import with one vkCmdCopyBuffer per slab.
+//
+// Freshness is derived from the write tracker, never stored: a unit is shadowed ("fresh") while its
+// slab bytes were written at a tracker generation no block stamp exceeds (MarkShadowed takes the
+// generation the write-back settled at, after its own MarkWritten); every import writer already
+// stamps its blocks (labels, fills, copies, key stores, in-place V# writes, CPU stores), so a stale
+// unit reads from the import as today. A CPU store stamps only when a collect walks its page, so a
+// publish and a seed decision collect the units they judge first (as the write-back's keep decision
+// does): the CPU's bytes win the unit. A publish records the copy and marks the unit published (the
+// slab and the import agree until the next retile or stamp); a stale unit met by a publish is dropped
+// ("stale-dropped": its generation is cleared). A publish never stamps.
+//
+// Slabs are 8 MiB pieces of a shadow (APS5_UNIT_SHADOW_SLAB_MIB) with their own VkDeviceMemory,
+// made on the first retile into them under a budget (APS5_UNIT_SHADOW_MIB, default 1024, LRU
+// eviction with a publish of the evicted slab's fresh units). APS5_NO_UNIT_SHADOW=1 makes every
+// primitive inert. Every call but AnyShadowedOverlaps runs under GuestMemory::GpuMutex (they record
+// or mutate freshness); the registry's own mutex is a leaf (lock order GpuMutex -> Imports/Pending ->
+// Shadows -> tracker).
+bool UnitShadowEnabled();
+
+// None: a device-facing caller (a storage upload reads the slab itself); Whole: a reader of every
+// byte of the range; PartialUnits: a writer of the range (only units it covers partly are published,
+// so the neighbours' results in the rest of the unit reach the import before the write stamps it).
+enum class PublishScope : std::uint8_t { None, Whole, PartialUnits };
+enum class PublishReason : std::uint8_t { Hook, Region, Indirect, CopySource, CopyDestination, Fill, FillClear, Label, Keys, Scanout, Validate, Upload, TailMip, Evict, Retire, Teardown, Count };
+// The reason for a FlushPending reason string ("memory access" -> Hook, ...).
+PublishReason PublishReasonFor(const char* flushReason);
+
+struct ShadowSlab {
+    ShadowSlab(const Context& context, VkBuffer buffer, VkDeviceMemory memory, std::uint64_t firstUnit, std::uint32_t units);
+    ShadowSlab(const ShadowSlab&) = delete;
+    ShadowSlab& operator=(const ShadowSlab&) = delete;
+    // Destroys the handles: safe on the recorder's release thread (Keep's contract).
+    ~ShadowSlab();
+    Context context;
+    VkBuffer buffer;
+    VkDeviceMemory memory;
+    std::uint64_t firstUnit;
+    std::uint32_t units;
+    std::uint64_t lastUse = 0;
+    // Destinations handed out for a write-back in progress: such a slab is never evicted.
+    std::atomic<std::uint32_t> pins{0};
+};
+
+// Holds one pin of a slab (ShadowDestination::pin).
+struct ShadowSlabPin {
+    explicit ShadowSlabPin(std::shared_ptr<ShadowSlab> slab);
+    ShadowSlabPin(const ShadowSlabPin&) = delete;
+    ShadowSlabPin& operator=(const ShadowSlabPin&) = delete;
+    ~ShadowSlabPin();
+    std::shared_ptr<ShadowSlab> slab;
+};
+
+// A retile piece (guest range) and the slab its copies were recorded into.
+struct ShadowedRange {
+    std::uint64_t begin;
+    std::uint64_t end;
+    std::shared_ptr<ShadowSlab> slab;
+};
+
+// Records vkCmdCopyBuffer(slab -> import) for every fresh unpublished unit the scope selects over
+// [address, address + bytes) into the active recorder's open batch (a waited CommandBatch without
+// one), with the barriers, the pending-write note and the timing class of a fill; marks them
+// published, clears stale ones. Returns the units copied. Inside a completion action a store's
+// publish records nothing (the store's own stamp makes the unit stale; a recorded publish would land
+// over the store later).
+std::size_t PublishShadow(std::uint64_t address, std::size_t bytes, PublishScope scope, PublishReason reason);
+// No GpuMutex, no tracker: conservative (a unit with a generation and no publish counts), for
+// hook-free readers and a build's stage A.
+bool AnyShadowedOverlaps(std::uint64_t address, std::size_t bytes);
+bool AnyShadowedOverlaps(std::span<const std::pair<std::uint64_t, std::uint64_t>> ranges);
+
+// The sources of a storage upload's surface-relative runs: [begin, end) pieces of them, each read
+// from one buffer (a slab while the unit is fresh, else the import) at `offset` (byte x of the run
+// is at offset + (x - begin)), split at unit freshness and slab boundaries with one tracker query
+// over the span. A tail block (surface-relative, every unit of it) must have one source: a mixed
+// one is published (TailMip) and read from the import.
+struct ShadowRun {
+    std::uint64_t begin;
+    std::uint64_t end;
+    VkBuffer buffer;
+    VkDeviceSize offset;
+    bool shadow;
+    std::shared_ptr<ShadowSlab> slab;
+};
+std::vector<ShadowRun> ShadowSources(const Context& context, const HostImport& import, std::uint64_t surfaceBase, std::span<const std::pair<std::uint64_t, std::uint64_t>> runs, std::span<const std::pair<std::uint64_t, std::uint64_t>> tailBlocks);
+
+// The slab destination of a retile copy [begin, end) (guest addresses inside one slab: the caller
+// splits at SlabBoundary), making the slab under the budget; nullopt when refused (the caller writes
+// the import as before). `seedUnits` names the units the copy covers partly that are not fresh (as
+// [begin, end) clipped to the import, collected first): the caller records import -> slab for each
+// before its copy. The slab stays pinned against eviction while a copy of the destination lives:
+// the caller keeps one until MarkShadowed (an evicted slab would take the retile's copies with it).
+struct ShadowDestination {
+    VkBuffer buffer;
+    VkDeviceSize offset;
+    std::shared_ptr<ShadowSlab> slab;
+    std::vector<std::pair<std::uint64_t, std::uint64_t>> seedUnits;
+    std::shared_ptr<ShadowSlabPin> pin;
+};
+std::optional<ShadowDestination> ShadowDestinationFor(const Context& context, const HostImport& import, std::uint64_t begin, std::uint64_t end);
+// A seed copy (import -> slab of a partly covered unit) the caller recorded: counted and traced.
+void NoteShadowSeed(std::uint64_t begin, std::uint64_t end);
+// The first slab boundary above `address` (a guest address inside the import).
+std::uint64_t SlabBoundary(const HostImport& import, std::uint64_t address);
+// The slab offset of a guest address for a slab returned by ShadowDestinationFor or ShadowSources.
+VkDeviceSize SlabOffset(const HostImport& import, const ShadowSlab& slab, std::uint64_t address);
+// After a retile's copies were recorded: the units of `ranges` (guest) hold the newest bytes at
+// `generation`, unpublished. A unit whose slab is no longer the one the copies went into is not
+// marked (counted `slab lost`). Bumps StorageTexture's pending serial.
+void MarkShadowed(const HostImport& import, std::span<const ShadowedRange> ranges, std::uint64_t generation);
+// retireImport: publishes the fresh units whose memory `registered` still names as a readable
+// registered range into the (old) import buffer, hands the slabs to the batch and drops the shadow.
+// The import retires because its registration vanished or changed size, so the rest of its memory
+// belongs to the title again and is dropped (counted `dropped on retire`); `lost on retire` counts
+// the units nothing could record.
+void RetireShadow(const Context& context, const HostImport& import, const std::function<bool(std::uint64_t, std::uint64_t)>& registered);
+// Teardown: every shadow of the context's device is published; DestroyShadows then drops them.
+void PublishAllShadows(const Context& context, PublishReason reason);
+void DestroyShadows(VkDevice device);
+// The [storage] line's shadow segment (10 s deltas, live slab totals).
+std::string ShadowReport();
+// APS5_SHADOW_VERIFY=1: after a shadow-sourced upload, the units read must carry no stamp newer
+// than their generation (counted as mismatches otherwise).
+bool ShadowVerify();
+
+}
+
+#endif

core/libs/prx/libSceAgcDriver/Graphics/include/VertexInput.hpp

@@ -110,6 +110,19 @@ inline VertexInputLayout BuildVertexInputLayout(const Context& context, std::spa
     return result;
 }
 
+// The whole byte range a vertex buffer descriptor covers (records * stride, or records when the
+// stride is 0): what an indirect draw, whose counts only the GPU knows, copies for the fetch.
+inline std::size_t VertexBufferExtent(const ShaderRecompiler::VertexAttribute& attribute) {
+    const auto stride = (attribute.resource.fields[1] >> 16u) & 0x3fffu;
+    const auto records = attribute.resource.fields[2];
+    Require(attribute.fetchIndex <= 1, "unsupported vertex fetch index");
+    const auto bytes = stride == 0 ? static_cast<std::uint64_t>(records) : static_cast<std::uint64_t>(records) * stride;
+    Require(bytes != 0 && bytes <= std::numeric_limits<std::size_t>::max(), "empty or oversized vertex buffer descriptor");
+    const auto address = attribute.resource.fields[0] | (static_cast<std::uint64_t>(attribute.resource.fields[1] & 0xffffu) << 32u);
+    Require(address != 0 && bytes <= std::numeric_limits<std::uint64_t>::max() - address, "invalid vertex buffer address range");
+    return static_cast<std::size_t>(bytes);
+}
+
 inline std::size_t VertexBufferReadSize(const ShaderRecompiler::VertexAttribute& attribute, std::uint32_t maxIndex, std::uint32_t instances, std::uint32_t firstInstance = 0) {
     Require(instances != 0, "vertex input requires nonzero instance count");
     Require(firstInstance <= std::numeric_limits<std::uint32_t>::max() - (instances - 1u), "vertex input instance range overflow");

core/libs/prx/libSceAgcDriver/Graphics/shaders/ColorTransfer.comp

@@ -22,6 +22,8 @@ void main() {
         tiledIndex = ((y / 128u) * parameters.blocksPerRow + x / 128u) * 16384u + offset / 4u;
     }
     uint pixel = (parameters.mode & 1u) != 0u ? linear[linearIndex] : tiled[tiledIndex];
+    // Mode 8: A2B10G10R10 source (red in the low bits) presented as 8-bit RGBA.
+    if ((parameters.mode & 8u) != 0u) pixel = ((pixel >> 2u) & 0xffu) | (((pixel >> 12u) & 0xffu) << 8u) | (((pixel >> 22u) & 0xffu) << 16u) | 0xff000000u;
     if ((parameters.mode & 2u) != 0u) pixel = (pixel & 0xff00ff00u) | ((pixel & 0xffu) << 16u) | ((pixel >> 16u) & 0xffu);
     if ((parameters.mode & 1u) != 0u) tiled[tiledIndex] = pixel;
     else linear[linearIndex] = pixel;

core/libs/prx/libSceAgcDriver/Graphics/shaders/TextureDetile.comp

@@ -3,6 +3,29 @@
 layout(constant_id = 0) const uint ELEMENT_BYTES = 4u;
 layout(constant_id = 1) const uint BLOCK_BYTES = 256u;
 layout(constant_id = 2) const uint TILE_FAMILY = 0u;
+// 0: tiled input -> linear output (detile); 1: linear input -> tiled output (retile).
+layout(constant_id = 3) const uint RETILE = 0u;
+// TILE_FAMILY 2: XOR swizzle given per byte-address bit as masks of x (bits 0-11), y (12-23) and
+// slice (24-31) element-coordinate bits.
+layout(constant_id = 4) const uint EQ0 = 0u;
+layout(constant_id = 5) const uint EQ1 = 0u;
+layout(constant_id = 6) const uint EQ2 = 0u;
+layout(constant_id = 7) const uint EQ3 = 0u;
+layout(constant_id = 8) const uint EQ4 = 0u;
+layout(constant_id = 9) const uint EQ5 = 0u;
+layout(constant_id = 10) const uint EQ6 = 0u;
+layout(constant_id = 11) const uint EQ7 = 0u;
+layout(constant_id = 12) const uint EQ8 = 0u;
+layout(constant_id = 13) const uint EQ9 = 0u;
+layout(constant_id = 14) const uint EQ10 = 0u;
+layout(constant_id = 15) const uint EQ11 = 0u;
+layout(constant_id = 16) const uint EQ12 = 0u;
+layout(constant_id = 17) const uint EQ13 = 0u;
+layout(constant_id = 18) const uint EQ14 = 0u;
+layout(constant_id = 19) const uint EQ15 = 0u;
+// Nonzero: block extent in elements (thick 3D blocks are not the thin 2D shape derived below).
+layout(constant_id = 20) const uint BLOCK_WIDTH = 0u;
+layout(constant_id = 21) const uint BLOCK_HEIGHT = 0u;
 
 layout(local_size_x = 8, local_size_y = 8) in;
 
@@ -20,9 +43,34 @@ layout(push_constant) uniform Push {
     uint tailX;
     uint tailY;
     uint elementBytes;
-    uint arrayLayer;
+    uint slice;
+    // Only elements whose tiled offset (the linear offset for kLinear) lies in [rangeBegin,
+    // rangeEnd) are moved; tiledBase and linearBase are subtracted from the tiled and linear
+    // offsets of those elements (a window of the mip in each buffer; both 0 for the whole mip).
+    uint rangeBegin;
+    uint rangeEnd;
+    uint tiledBase;
+    uint linearBase;
+    // The first element column and row of the dispatched grid (a window's rectangle; 0 for the
+    // whole mip).
+    uint columnBegin;
+    uint rowBegin;
 } params;
 
+uint equationBit(uint mask, uvec2 p) {
+    uint selected = (p.x & (mask & 0xfffu)) ^ ((p.y << 12) & (mask & 0xfff000u)) ^ ((params.slice << 24) & (mask & 0xff000000u));
+    return bitCount(selected) & 1u;
+}
+
+uint equationOffset(uvec2 p) {
+    const uint eq[16] = uint[16](EQ0, EQ1, EQ2, EQ3, EQ4, EQ5, EQ6, EQ7, EQ8, EQ9, EQ10, EQ11, EQ12, EQ13, EQ14, EQ15);
+    uint offset = 0u;
+    for (uint bit = 0u; bit < 16u; ++bit) {
+        offset |= equationBit(eq[bit], p) << bit;
+    }
+    return offset;
+}
+
 uint standardOffset(uint x, uint y) {
     switch (ELEMENT_BYTES) {
         case 1u:
@@ -67,76 +115,9 @@ uint standard64Extra(uint x, uint y) {
 }
 
 
-uint renderTargetOffset(uint x, uint y) {
-    uint offset = 0u;
-    switch (ELEMENT_BYTES) {
-        case 1u:
-            offset ^= (y << 2u) & 0x0008u;
-            offset ^= (y << 4u) & 0x0010u;
-            offset ^= (y << 3u) & 0x00a0u;
-            offset ^= (y << 5u) & 0x0f00u;
-            offset ^= (y << 6u) & 0x1000u;
-            offset ^= (y << 7u) & 0x4000u;
-            offset ^= x & 0x0007u;
-            offset ^= (x << 3u) & 0x0040u;
-            offset ^= (x << 5u) & 0x0300u;
-            offset ^= (x << 4u) & 0x0400u;
-            offset ^= (x << 6u) & 0x0800u;
-            offset ^= (x << 7u) & 0x2000u;
-            offset ^= (x << 8u) & 0x8000u;
-            break;
-        case 2u:
-            offset ^= (y << 4u) & 0x0070u;
-            offset ^= (y << 5u) & 0x0f00u;
-            offset ^= (y << 8u) & 0x5000u;
-            offset ^= (x << 1u) & 0x000eu;
-            offset ^= (x << 4u) & 0x0480u;
-            offset ^= (x << 5u) & 0x0300u;
-            offset ^= (x << 6u) & 0x0800u;
-            offset ^= (x << 7u) & 0x2000u;
-            offset ^= (x << 8u) & 0x8000u;
-            break;
-        case 4u:
-            offset ^= (y << 4u) & 0x0070u;
-            offset ^= (y << 5u) & 0x0f00u;
-            offset ^= (y << 9u) & 0x1000u;
-            offset ^= (y << 8u) & 0x4000u;
-            offset ^= (x << 2u) & 0x000cu;
-            offset ^= (x << 5u) & 0x0380u;
-            offset ^= (x << 4u) & 0x0400u;
-            offset ^= (x << 6u) & 0x0800u;
-            offset ^= (x << 9u) & 0xa000u;
-            break;
-        case 8u:
-            offset ^= (y << 4u) & 0x0010u;
-            offset ^= (y << 6u) & 0x0080u;
-            offset ^= (y << 5u) & 0x0f00u;
-            offset ^= (y << 10u) & 0x5000u;
-            offset ^= (x << 3u) & 0x0008u;
-            offset ^= (x << 4u) & 0x0460u;
-            offset ^= (x << 5u) & 0x0300u;
-            offset ^= (x << 6u) & 0x0800u;
-            offset ^= (x << 10u) & 0x2000u;
-            offset ^= (x << 9u) & 0x8000u;
-            break;
-        case 16u:
-            offset ^= (x << 4u) & 0x0410u;
-            offset ^= (x << 5u) & 0x0340u;
-            offset ^= (x << 6u) & 0x0800u;
-            offset ^= (x << 11u) & 0xa000u;
-            offset ^= (y << 5u) & 0x0f20u;
-            offset ^= (y << 6u) & 0x0080u;
-            offset ^= (y << 10u) & 0x1000u;
-            offset ^= (y << 11u) & 0x4000u;
-            break;
-    }
-    uint z = params.arrayLayer;
-    return offset ^ ((z & 8u) << 5u) ^ ((z & 4u) << 7u) ^ ((z & 2u) << 9u) ^ ((z & 1u) << 11u);
-}
-
 uint blockOffset(uvec2 p) {
     if (TILE_FAMILY == 2u) {
-        return renderTargetOffset(p.x, p.y);
+        return equationOffset(p);
     }
     uint offset = standardOffset(p.x, p.y);
     if (BLOCK_BYTES > 4096u) {
@@ -146,6 +127,9 @@ uint blockOffset(uvec2 p) {
 }
 
 uvec2 blockExtent() {
+    if (BLOCK_WIDTH != 0u) {
+        return uvec2(BLOCK_WIDTH, BLOCK_HEIGHT);
+    }
     uint width;
     if (BLOCK_BYTES <= 256u) {
         width = ELEMENT_BYTES <= 2u ? 16u : (ELEMENT_BYTES <= 8u ? 8u : 4u);
@@ -172,7 +156,7 @@ void copyElement(uint src, uint dst) {
 }
 
 void main() {
-    uvec2 p = gl_GlobalInvocationID.xy;
+    uvec2 p = gl_GlobalInvocationID.xy + uvec2(params.columnBegin, params.rowBegin);
     if (p.x >= params.width || p.y >= params.height) {
         return;
     }
@@ -182,7 +166,14 @@ void main() {
     if (TILE_FAMILY == 0u) {
         // kLinear: guest and host layouts match; a straight copy at the
         // same pitch, no swizzle.
-        copyElement(params.srcBase + linearOffset, params.dstBase + linearOffset);
+        if (linearOffset < params.rangeBegin || linearOffset >= params.rangeEnd) {
+            return;
+        }
+        if (RETILE != 0u) {
+            copyElement(params.srcBase + linearOffset - params.linearBase, params.dstBase + linearOffset - params.tiledBase);
+        } else {
+            copyElement(params.srcBase + linearOffset - params.tiledBase, params.dstBase + linearOffset - params.linearBase);
+        }
         return;
     }
 
@@ -196,7 +187,16 @@ void main() {
 
     uint blockIndex = block.y * params.blocksPerRow + block.x;
     uint tiledOffset = blockIndex * BLOCK_BYTES + blockOffset(swizzle);
+    if (tiledOffset < params.rangeBegin || tiledOffset >= params.rangeEnd) {
+        return;
+    }
+    tiledOffset -= params.tiledBase;
+    linearOffset -= params.linearBase;
 
-    copyElement(params.srcBase + tiledOffset, params.dstBase + linearOffset);
+    if (RETILE != 0u) {
+        copyElement(params.srcBase + linearOffset, params.dstBase + tiledOffset);
+    } else {
+        copyElement(params.srcBase + tiledOffset, params.dstBase + linearOffset);
+    }
 }
 

core/libs/prx/libSceAgcDriver/Graphics/src/BdaResources.cpp

@@ -1,11 +1,123 @@
 #include "prx/libSceAgcDriver/Graphics/include/BdaResources.hpp"
+#include <algorithm>
 #include <bit>
+#include <cstdio>
+#include <cstdlib>
 #include <cstring>
 #include <limits>
+#include <list>
+#include <mutex>
 #include <sstream>
 
 namespace AgcDriver::Graphics {
 
+namespace {
+
+// The page tables built per device, reused by a build whose ranges (guest ranges, device
+// addresses, permissions) equal one of them: consecutive address-based builds map the same registered
+// set through the same imports and mirrors, so the ~1200-entry table repeats. The shader only reads
+// the table (its lookup keeps its state in function variables), and a batch in flight keeps its
+// BdaResources, so sharing the buffer is safe. The cache holds the tables weakly: they never outlive
+// the builds that own them (a static owner would destroy the buffer after its device, at exit or on
+// device replacement, and would fail the tests' leak check), and with deferred lease release the
+// previous builds are still in flight when the next one asks, which is what makes the hit rate.
+// One table was enough while every build repeated the last one exactly; now the misaligned
+// descriptor sub-ranges the GPU copies out of imports sit in pool staging buffers whose device
+// addresses cycle through the few buffers the batches in flight leave free, and the three
+// address-based programs of one queue interleave with the lease draws, so a table differs from the
+// previous one and comes back a few builds later (69% misses in the wave-8 run, 4% before the GPU
+// copies). Hence several recent tables, most recently used first, matched by a word hash before
+// the byte compare. APS5_BDA_TABLE_CACHE_ENTRIES sets how many (default 8; 1 is the previous
+// behaviour); APS5_NO_BDA_TABLE_CACHE=1 builds every table.
+// A table built for (or found equal to) the cached address space's ranges carries the space's
+// serial: a build served by that space alone binds it by the serial, with no hash and no compare.
+struct TableEntry {
+    std::uint64_t hash;
+    std::vector<ShaderRecompiler::BdaAbi::Range> ranges;
+    std::weak_ptr<Buffer> buffer;
+    std::uint64_t spaceSerial = 0;
+};
+
+struct TableCache {
+    std::mutex mutex;
+    VkDevice device = VK_NULL_HANDLE;
+    std::list<TableEntry> entries;
+    std::uint64_t hits = 0;
+    std::uint64_t misses = 0;
+    BdaResources::TableCacheStats classes;
+};
+
+bool sameRanges(const std::vector<ShaderRecompiler::BdaAbi::Range>& left, const std::vector<ShaderRecompiler::BdaAbi::Range>& right);
+
+// APS5_PROFILE_DRAW: why the most recently used entry did not serve this build (see
+// TableCacheStats). Under the cache mutex.
+void classifyFirstEntry(TableCache& cache, std::uint64_t hash, const std::vector<ShaderRecompiler::BdaAbi::Range>& ranges) {
+    static const bool profile = std::getenv("APS5_PROFILE_DRAW") != nullptr;
+    if (!profile) return;
+    auto& classes = cache.classes;
+    if (cache.entries.empty()) {
+        ++classes.firstEmpty;
+        return;
+    }
+    const auto& first = cache.entries.front();
+    if (first.hash == hash) {
+        if (first.buffer.expired()) ++classes.firstExpired;
+        else if (!sameRanges(first.ranges, ranges)) ++classes.firstSameHash;
+        return;
+    }
+    if (first.buffer.expired()) {
+        ++classes.firstExpired;
+        return;
+    }
+    const auto common = std::min(first.ranges.size(), ranges.size());
+    std::size_t at = 0;
+    while (at < common && std::memcmp(&first.ranges[at], &ranges[at], sizeof(ranges[at])) == 0) ++at;
+    const auto begin = at < ranges.size() ? ranges[at].begin : at < first.ranges.size() ? first.ranges[at].begin : 0;
+    ++(begin < 0x10000000ull ? classes.firstDiffersLow : classes.firstDiffersHeap);
+}
+
+TableCache& Tables() {
+    static TableCache cache;
+    return cache;
+}
+
+bool tableCacheEnabled() {
+    static const bool disabled = std::getenv("APS5_NO_BDA_TABLE_CACHE") != nullptr;
+    return !disabled;
+}
+
+std::size_t tableCacheEntries() {
+    static const std::size_t entries = [] {
+        const char* value = std::getenv("APS5_BDA_TABLE_CACHE_ENTRIES");
+        const auto parsed = value != nullptr ? std::strtoull(value, nullptr, 10) : 8ull;
+        return static_cast<std::size_t>(std::clamp<unsigned long long>(parsed, 1, 64));
+    }();
+    return entries;
+}
+
+bool sameRanges(const std::vector<ShaderRecompiler::BdaAbi::Range>& left, const std::vector<ShaderRecompiler::BdaAbi::Range>& right) {
+    return left.size() == right.size() && (left.empty() || std::memcmp(left.data(), right.data(), left.size() * sizeof(left.front())) == 0);
+}
+
+// A 64-bit mix of the table's words: a table is ~1200 ranges of four words, and comparing the bytes
+// against every held table would cost as much as building it.
+std::uint64_t hashRanges(const std::vector<ShaderRecompiler::BdaAbi::Range>& ranges) {
+    static_assert(sizeof(ShaderRecompiler::BdaAbi::Range) % sizeof(std::uint64_t) == 0);
+    std::uint64_t hash = 0x9e3779b97f4a7c15ull ^ ranges.size();
+    for (const auto& range : ranges) {
+        std::uint64_t words[sizeof(range) / sizeof(std::uint64_t)];
+        std::memcpy(words, &range, sizeof(range));
+        for (const auto word : words) {
+            hash ^= word;
+            hash *= 0xff51afd7ed558ccdull;
+            hash ^= hash >> 33u;
+        }
+    }
+    return hash;
+}
+
+}
+
 BdaResources::BdaResources(const Context& context) {
     static_assert(std::endian::native == std::endian::little);
     Require(sizeof(ShaderRecompiler::BdaAbi::Fault) <= context.limits.maxStorageBufferRange, "BDA fault buffer exceeds storage buffer range limit");
@@ -14,15 +126,94 @@ BdaResources::BdaResources(const Context& context) {
 }
 
 BdaResources::BdaResources(const Context& context, const GuestBufferMemory& memory) : BdaResources(context) {
-    const auto ranges = memory.AddressRanges();
+    const auto cached = memory.CachedAddressTable();
+    std::vector<ShaderRecompiler::BdaAbi::Range> built;
+    if (!cached.has_value()) built = memory.AddressRanges();
+    const auto& ranges = cached.has_value() ? *cached->ranges : built;
+    const auto serial = cached.has_value() ? cached->serial : 0;
     Require(ranges.size() <= std::numeric_limits<std::uint32_t>::max(), "BDA table range count overflow");
     Require(ranges.size() <= (std::numeric_limits<std::size_t>::max() - sizeof(ShaderRecompiler::BdaAbi::Header)) / sizeof(ShaderRecompiler::BdaAbi::Range), "BDA table size overflow");
     tableBytes = sizeof(ShaderRecompiler::BdaAbi::Header) + ranges.size() * sizeof(ShaderRecompiler::BdaAbi::Range);
     Require(tableBytes <= context.limits.maxStorageBufferRange && sizeof(ShaderRecompiler::BdaAbi::Fault) <= context.limits.maxStorageBufferRange, "BDA descriptors exceed storage buffer range limit");
-    table = std::make_unique<Buffer>(context, tableBytes, VK_BUFFER_USAGE_STORAGE_BUFFER_BIT);
+    auto& cache = Tables();
+    std::uint64_t hash = 0;
+    if (tableCacheEnabled()) {
+        std::lock_guard lock(cache.mutex);
+        // Another device's tables are not this one's (their buffers are dead or foreign).
+        if (cache.device != context.device) {
+            cache.entries.clear();
+            cache.device = context.device;
+        }
+        if (serial != 0) {
+            for (auto it = cache.entries.begin(); it != cache.entries.end(); ++it) {
+                if (it->spaceSerial != serial) continue;
+                if (auto shared = it->buffer.lock(); shared != nullptr) {
+                    ++cache.hits;
+                    ++cache.classes.spaceTables;
+                    table = std::move(shared);
+                    cache.entries.splice(cache.entries.begin(), cache.entries, it);
+                    return;
+                }
+                cache.entries.erase(it);
+                break;
+            }
+        }
+        hash = hashRanges(ranges);
+        classifyFirstEntry(cache, hash, ranges);
+        for (auto it = cache.entries.begin(); it != cache.entries.end();) {
+            // Compare the hash before taking a strong reference: locking a non-matching entry's
+            // buffer could make this thread its last owner (a concurrent reap releasing it) and
+            // run the pool release under the cache mutex; expired() takes no reference.
+            if (it->hash != hash) {
+                if (it->buffer.expired()) {
+                    it = cache.entries.erase(it);
+                } else {
+                    ++it;
+                }
+                continue;
+            }
+            auto shared = it->buffer.lock();
+            if (shared == nullptr) {
+                // Every build that owned the table is gone, and its buffer with it.
+                it = cache.entries.erase(it);
+                continue;
+            }
+            if (sameRanges(it->ranges, ranges)) {
+                ++cache.hits;
+                // An equal table of an earlier space (a rebuild that mapped the same ranges the
+                // same way): the next build of this space binds it by the serial.
+                if (serial != 0) it->spaceSerial = serial;
+                table = std::move(shared);
+                cache.entries.splice(cache.entries.begin(), cache.entries, it);
+                return;
+            }
+            ++it;
+        }
+    }
+    table = std::make_shared<Buffer>(context, tableBytes, VK_BUFFER_USAGE_STORAGE_BUFFER_BIT);
     const ShaderRecompiler::BdaAbi::Header header{ShaderRecompiler::BdaAbi::Version, static_cast<std::uint32_t>(ranges.size()), sizeof(ShaderRecompiler::BdaAbi::Range), 0};
     std::memcpy(table->Bytes().data(), &header, sizeof(header));
     if (!ranges.empty()) std::memcpy(table->Bytes().data() + sizeof(header), ranges.data(), ranges.size() * sizeof(ranges.front()));
+    if (tableCacheEnabled()) {
+        std::lock_guard lock(cache.mutex);
+        ++cache.misses;
+        if (cache.device != context.device) {
+            cache.entries.clear();
+            cache.device = context.device;
+        }
+        cache.entries.push_front({hash, cached.has_value() ? ranges : std::move(built), table, serial});
+        while (cache.entries.size() > tableCacheEntries()) cache.entries.pop_back();
+    }
+}
+
+BdaResources::TableCacheStats BdaResources::TableCacheCounters() {
+    auto& cache = Tables();
+    std::lock_guard lock(cache.mutex);
+    auto stats = cache.classes;
+    stats.hits = cache.hits;
+    stats.misses = cache.misses;
+    stats.held = cache.entries.size();
+    return stats;
 }
 
 VkDescriptorBufferInfo BdaResources::Table() const {
@@ -43,6 +234,12 @@ void BdaResources::CheckFault() const {
     }
     Require(report.state == ShaderRecompiler::BdaAbi::FaultState::Ready && report.reserved == 0, "incomplete or invalid BDA fault record");
     Require(report.reason != ShaderRecompiler::BdaAbi::FaultReason::InvalidRectangle, "rect-list requires finite nondegenerate axis-aligned positions with equal positive W");
+    if (report.reason == ShaderRecompiler::BdaAbi::FaultReason::LoopLimit) {
+        // APS5_LOOP_GUARD: the shader left a loop that ran past the guard; the dispatch result is kept.
+        std::fprintf(stderr, "[gpu] loop guard: the loop exit at pc 0x%x ran past %u evaluations\n", report.instruction, report.bytes);
+        std::memset(fault->Bytes().data(), 0, fault->Bytes().size());
+        return;
+    }
     std::ostringstream message;
     message << "BDA access failed: address=0x" << std::hex << report.address << " instruction=0x" << report.instruction << std::dec << " bytes=" << report.bytes << " stage=" << report.stage << " reason=" << static_cast<std::uint32_t>(report.reason);
     throw std::runtime_error(message.str());

core/libs/prx/libSceAgcDriver/Graphics/src/BufferPool.cpp

@@ -1,74 +1,129 @@
 #include "prx/libSceAgcDriver/Graphics/include/BufferPool.hpp"
+#include <algorithm>
+#include <bit>
+#include <chrono>
+#include <cstdio>
+#include <cstdlib>
 
 namespace AgcDriver::Graphics {
 
+namespace {
+
+// APS5_BUFFER_POOL_SHARED=1: one tier for every size, as before the split.
+bool sharedTiers() {
+    static const bool shared = std::getenv("APS5_BUFFER_POOL_SHARED") != nullptr;
+    return shared;
+}
+
+}
+
 BufferPool::BufferPool(const Context& context) : device(context.device), unmap(context.Function<PFN_vkUnmapMemory>("vkUnmapMemory")), destroyBuffer(context.Function<PFN_vkDestroyBuffer>("vkDestroyBuffer")), freeMemory(context.Function<PFN_vkFreeMemory>("vkFreeMemory")) {
-    buckets.fill(none);
-    for (std::size_t index = 0; index < capacity; ++index) freeSlots[index] = index;
+    smallTier.budget = smallBudget;
+    largeTier.budget = budget;
+    deviceTier.budget = DeviceBudget();
 }
 
 BufferPool::~BufferPool() {
-    for (const auto& slot : slots) {
-        if (slot.allocation) destroy(*slot.allocation);
-    }
+    for (const auto& slot : smallTier.free) destroy(slot.allocation);
+    for (const auto& slot : largeTier.free) destroy(slot.allocation);
+    for (const auto& slot : deviceTier.free) destroy(slot.allocation);
 }
 
-std::size_t BufferPool::bucket(std::size_t bytes, VkBufferUsageFlags usage, VkMemoryPropertyFlags properties) noexcept {
-    auto hash = bytes ^ (bytes >> 16u);
-    hash = hash * 16777619u ^ usage;
-    hash = hash * 16777619u ^ properties;
-    return (hash ^ (hash >> 8u) ^ (hash >> 16u)) % bucketCount;
-}
-
-BufferAllocation BufferPool::remove(std::size_t index) noexcept {
-    auto& slot = slots[index];
-    const auto allocation = *slot.allocation;
-    auto* link = &buckets[bucket(allocation.bytes, allocation.usage, allocation.properties)];
-    while (*link != index) link = &slots[*link].next;
-    *link = slot.next;
-    if (slot.older != none) slots[slot.older].newer = slot.newer;
-    else oldest = slot.newer;
-    if (slot.newer != none) slots[slot.newer].older = slot.older;
-    else newest = slot.older;
-    retainedBytes -= allocation.allocationBytes;
-    slot = Slot{};
-    freeSlots[freeCount++] = index;
-    return allocation;
+VkDeviceSize BufferPool::DeviceBudget() {
+    static const VkDeviceSize deviceBudget = [] {
+        const char* value = std::getenv("APS5_STAGING_POOL_MIB");
+        return (value != nullptr ? std::strtoull(value, nullptr, 10) : 512ull) << 20u;
+    }();
+    return deviceBudget;
 }
 
 void BufferPool::destroy(const BufferAllocation& allocation) noexcept {
+    // Device-local allocations (see DeviceBuffer) are never mapped.
     if (allocation.mapping != nullptr) unmap(device, allocation.memory);
     destroyBuffer(device, allocation.buffer, nullptr);
     freeMemory(device, allocation.memory, nullptr);
 }
 
+std::size_t BufferPool::Capacity(std::size_t bytes) {
+    static const bool exact = std::getenv("APS5_NO_BUFFER_CLASSES") != nullptr;
+    constexpr std::size_t smallest = 256;
+    if (exact || bytes >= classLimit) return bytes;
+    return std::max(smallest, std::bit_ceil(bytes));
+}
+
+BufferPool::Tier& BufferPool::tierFor(std::size_t capacity, VkMemoryPropertyFlags properties) {
+    if ((properties & VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) != 0 && (properties & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) == 0 && DeviceBudget() != 0) return deviceTier;
+    return !sharedTiers() && capacity < classLimit ? smallTier : largeTier;
+}
+
 std::optional<BufferAllocation> BufferPool::Take(std::size_t bytes, VkBufferUsageFlags usage, VkMemoryPropertyFlags properties) {
+    static const bool profile = std::getenv("APS5_PROFILE_DRAW") != nullptr;
+    const auto capacity = Capacity(bytes);
     std::lock_guard lock(mutex);
-    for (auto index = buckets[bucket(bytes, usage, properties)]; index != none; index = slots[index].next) {
-        const auto& allocation = *slots[index].allocation;
-        if (allocation.bytes == bytes && allocation.usage == usage && allocation.properties == properties) return remove(index);
+    ++clock;
+    if (profile) {
+        static auto lastReport = std::chrono::steady_clock::now();
+        const auto now = std::chrono::steady_clock::now();
+        if (now - lastReport > std::chrono::seconds(10)) {
+            lastReport = now;
+            std::fprintf(stderr, "[bufferpool] small: %llu hits, %llu misses, %llu evictions, %zu retained (%.0f MiB); large: %llu hits, %llu misses, %llu evictions, %zu retained (%.0f MiB)%s; device: %llu hits, %llu misses, %llu evictions, %zu retained (%.0f MiB of %llu)\n", static_cast<unsigned long long>(smallTier.hits), static_cast<unsigned long long>(smallTier.misses), static_cast<unsigned long long>(smallTier.evictions), smallTier.free.size(), smallTier.retainedBytes / 1048576.0, static_cast<unsigned long long>(largeTier.hits), static_cast<unsigned long long>(largeTier.misses), static_cast<unsigned long long>(largeTier.evictions), largeTier.free.size(), largeTier.retainedBytes / 1048576.0, sharedTiers() ? " (shared)" : "", static_cast<unsigned long long>(deviceTier.hits), static_cast<unsigned long long>(deviceTier.misses), static_cast<unsigned long long>(deviceTier.evictions), deviceTier.free.size(), deviceTier.retainedBytes / 1048576.0, static_cast<unsigned long long>(deviceTier.budget >> 20u));
+        }
     }
+    auto& tier = tierFor(capacity, properties);
+    for (auto it = tier.free.begin(); it != tier.free.end(); ++it) {
+        const auto& allocation = it->allocation;
+        if (allocation.bytes != capacity || allocation.usage != usage || allocation.properties != properties) continue;
+        auto result = allocation;
+        tier.retainedBytes -= allocation.allocationBytes;
+        *it = std::move(tier.free.back());
+        tier.free.pop_back();
+        ++tier.hits;
+        return result;
+    }
+    ++tier.misses;
     return std::nullopt;
 }
 
+void BufferPool::evictOldest(Tier& tier, std::vector<BufferAllocation>& evicted) {
+    const auto oldest = std::min_element(tier.free.begin(), tier.free.end(), [](const Slot& left, const Slot& right) { return left.lastUse < right.lastUse; });
+    // First: a throw here leaves the slot retained and counted.
+    evicted.push_back(oldest->allocation);
+    tier.retainedBytes -= oldest->allocation.allocationBytes;
+    *oldest = std::move(tier.free.back());
+    tier.free.pop_back();
+    ++tier.evictions;
+}
+
+std::size_t BufferPool::MaxSlots() {
+    // APS5_BUFFER_POOL_SLOTS=<n> bounds the retained allocations; 64 is the capacity the pool had
+    // before least-recently-used retention, for A/B runs.
+    static const std::size_t slots = [] {
+        const char* value = std::getenv("APS5_BUFFER_POOL_SLOTS");
+        const auto parsed = value != nullptr ? std::strtoull(value, nullptr, 10) : 0;
+        return parsed != 0 ? static_cast<std::size_t>(parsed) : defaultSlots;
+    }();
+    return slots;
+}
+
 void BufferPool::Put(const BufferAllocation& allocation) noexcept {
-    std::lock_guard lock(mutex);
-    if (allocation.allocationBytes > budget) {
+    // Evicted allocations are destroyed after the mutex is released (see evictOldest). The vector
+    // may throw on growth; a Put that cannot retain simply destroys, as noexcept requires.
+    std::vector<BufferAllocation> evicted;
+    try {
+        std::lock_guard lock(mutex);
+        auto& tier = tierFor(allocation.bytes, allocation.properties);
+        if (allocation.allocationBytes > tier.budget) {
+            evicted.push_back(allocation);
+        } else {
+            const auto maxSlots = MaxSlots();
+            while (!tier.free.empty() && (tier.retainedBytes + allocation.allocationBytes > tier.budget || tier.free.size() >= maxSlots)) evictOldest(tier, evicted);
+            tier.free.push_back({allocation, ++clock});
+            tier.retainedBytes += allocation.allocationBytes;
+        }
+    } catch (...) {
         destroy(allocation);
-        return;
     }
-    while (retainedBytes > budget - allocation.allocationBytes || freeCount == 0) destroy(remove(oldest));
-    const auto index = freeSlots[--freeCount];
-    auto& slot = slots[index];
-    auto& head = buckets[bucket(allocation.bytes, allocation.usage, allocation.properties)];
-    slot.allocation = allocation;
-    slot.next = head;
-    slot.older = newest;
-    if (newest != none) slots[newest].newer = index;
-    else oldest = index;
-    newest = index;
-    head = index;
-    retainedBytes += allocation.allocationBytes;
+    for (const auto& gone : evicted) destroy(gone);
 }
 
 std::shared_ptr<BufferPool> GetBufferPool(const Context& context) {

core/libs/prx/libSceAgcDriver/Graphics/src/ColorMemoryWriteback.cpp

@@ -1,28 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/GpuColorTransfer.hpp"
-#include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp"
-#include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
-#include "prx/libc/include/GuestMemoryBacking.hpp"
-
-namespace AgcDriver::Graphics {
-
-void GpuColorTransfer::WriteBack(std::uint64_t address) {
-    writeBack(address, false);
-}
-
-void GpuColorTransfer::WriteBackTracked(std::uint64_t address) {
-    writeBack(address, true);
-}
-
-void GpuColorTransfer::writeBack(std::uint64_t address, bool tracked) {
-    PerformanceTimer timing("ColorTransfer.WriteBack");
-    Require(tiled != nullptr && readback != nullptr, "color transfer is not prepared for writeback");
-    const ColorTargetLayout layout(width, height, mode);
-    Require(address != 0 && address % layout.Alignment() == 0, "misaligned color writeback address");
-    timing.Mark("validate");
-    readback->Invalidate();
-    if (tracked) GuestMemoryBacking::GuestMemoryBackingWrite_nid_postfix(address, readback->Bytes().data(), readback->Bytes().size());
-    else GuestMemory::Write(address, readback->Bytes(), layout.Alignment());
-    timing.Mark("guest_write", layout.Bytes());
-}
-
-}

core/libs/prx/libSceAgcDriver/Graphics/src/ColorTargetLayout.cpp

@@ -1,4 +1,6 @@
 #include "prx/libSceAgcDriver/Graphics/include/ColorTargetLayout.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/TextureSwizzleEquations.hpp"
+#include <bit>
 #include <cstring>
 #include <limits>
 #include <stdexcept>
@@ -10,9 +12,8 @@ void require(bool condition, const char* reason) {
     if (!condition) throw std::runtime_error(reason);
 }
 
-std::uint32_t blockOffset(std::uint32_t x, std::uint32_t y) {
-    return ((y << 4u) & 0x0070u) ^ ((y << 5u) & 0x0f00u) ^ ((y << 9u) & 0x1000u) ^ ((y << 8u) & 0x4000u)
-        ^ ((x << 2u) & 0x000cu) ^ ((x << 5u) & 0x0380u) ^ ((x << 4u) & 0x0400u) ^ ((x << 6u) & 0x0800u) ^ ((x << 9u) & 0xa000u);
+std::uint32_t parity(std::uint32_t value) {
+    return static_cast<std::uint32_t>(std::popcount(value)) & 1u;
 }
 
 }
@@ -26,28 +27,48 @@ ColorTileMode DecodeColorTileMode(std::uint32_t attrib3) {
     return static_cast<ColorTileMode>(mode);
 }
 
-ColorTargetLayout::ColorTargetLayout(std::uint32_t width, std::uint32_t height, ColorTileMode mode) : width(width), height(height), pitch(width), mode(mode), bytes(0) {
+ColorTargetLayout::ColorTargetLayout(std::uint32_t width, std::uint32_t height, ColorTileMode mode, std::uint32_t bytesPerElement) : width(width), height(height), pitch(width), mode(mode), bytes(0), elementBytes(bytesPerElement) {
     require(width != 0 && height != 0 && width <= 16384 && height <= 16384, "AGC graphics: invalid color surface extent");
+    require(std::has_single_bit(bytesPerElement) && bytesPerElement <= 16u, "AGC graphics: unsupported color element size");
     std::uint32_t paddedHeight = height;
     switch (mode) {
         case ColorTileMode::Linear:
-            require(width % 64u == 0, "AGC graphics: linear surface pitch requires a width aligned to 256 bytes");
+            require(width * bytesPerElement % 256u == 0, "AGC graphics: linear surface pitch requires a width aligned to 256 bytes");
             break;
-        case ColorTileMode::RenderTarget:
-            pitch = (width + 127u) & ~127u;
-            paddedHeight = (height + 127u) & ~127u;
+        case ColorTileMode::RenderTarget: {
+            // SW_64KB_R_X: 64 KiB blocks of 2^(16 - log2(bpe)) elements, wider than tall for odd powers.
+            const auto log2Elements = 16u - static_cast<std::uint32_t>(std::countr_zero(bytesPerElement));
+            blockWidth = 1u << ((log2Elements + 1u) / 2u);
+            blockHeight = 1u << (log2Elements / 2u);
+            pitch = (width + blockWidth - 1u) / blockWidth * blockWidth;
+            paddedHeight = (height + blockHeight - 1u) / blockHeight * blockHeight;
+            const auto* equation = FindTextureSwizzleEquation(27u, bytesPerElement);
+            require(equation != nullptr, "AGC graphics: no SW_64KB_R_X equation for the color element size");
+            xOffsets.resize(blockWidth);
+            yOffsets.resize(blockHeight);
+            for (std::uint32_t x = 0; x < blockWidth; ++x) {
+                std::uint32_t offset = 0;
+                for (std::uint32_t bit = 0; bit < 16u; ++bit) offset |= parity(x & equation->bits[bit] & 0xfffu) << bit;
+                xOffsets[x] = offset;
+            }
+            for (std::uint32_t y = 0; y < blockHeight; ++y) {
+                std::uint32_t offset = 0;
+                for (std::uint32_t bit = 0; bit < 16u; ++bit) offset |= parity((y << 12u) & equation->bits[bit] & 0xfff000u) << bit;
+                yOffsets[y] = offset;
+            }
             break;
+        }
         default: throw std::runtime_error("AGC graphics: unsupported color tile mode");
     }
-    const auto size = static_cast<std::uint64_t>(pitch) * paddedHeight * 4u;
+    const auto size = static_cast<std::uint64_t>(pitch) * paddedHeight * bytesPerElement;
     require(size <= std::numeric_limits<std::size_t>::max(), "AGC graphics: color surface size overflow");
     bytes = static_cast<std::size_t>(size);
 }
 
 std::size_t ColorTargetLayout::offset(std::uint32_t x, std::uint32_t y) const {
-    if (mode == ColorTileMode::Linear) return (static_cast<std::size_t>(y) * pitch + x) * 4u;
-    const auto block = static_cast<std::size_t>(y / 128u) * (pitch / 128u) + x / 128u;
-    return block * 65536u + blockOffset(x, y);
+    if (mode == ColorTileMode::Linear) return (static_cast<std::size_t>(y) * pitch + x) * elementBytes;
+    const auto block = static_cast<std::size_t>(y / blockHeight) * (pitch / blockWidth) + x / blockWidth;
+    return block * 65536u + (xOffsets[x % blockWidth] ^ yOffsets[y % blockHeight]);
 }
 
 std::size_t ColorTargetLayout::Offset(std::uint32_t x, std::uint32_t y) const {
@@ -58,18 +79,16 @@ std::size_t ColorTargetLayout::Offset(std::uint32_t x, std::uint32_t y) const {
 void ColorTargetLayout::Detile(std::span<const std::byte> source, std::span<std::byte> destination) const {
     require(source.size() == Bytes() && destination.size() == LinearBytes(), "AGC graphics: color detile buffer size mismatch");
     for (std::uint32_t y = 0; y < height; ++y) {
-        for (std::uint32_t x = 0; x < width; ++x) {
-            std::memcpy(destination.data() + (static_cast<std::size_t>(y) * width + x) * 4u, source.data() + offset(x, y), 4);
-        }
+        auto* row = destination.data() + static_cast<std::size_t>(y) * width * elementBytes;
+        for (std::uint32_t x = 0; x < width; ++x) std::memcpy(row + static_cast<std::size_t>(x) * elementBytes, source.data() + offset(x, y), elementBytes);
     }
 }
 
 void ColorTargetLayout::Tile(std::span<const std::byte> source, std::span<std::byte> destination) const {
     require(source.size() == LinearBytes() && destination.size() == Bytes(), "AGC graphics: color tile buffer size mismatch");
     for (std::uint32_t y = 0; y < height; ++y) {
-        for (std::uint32_t x = 0; x < width; ++x) {
-            std::memcpy(destination.data() + offset(x, y), source.data() + (static_cast<std::size_t>(y) * width + x) * 4u, 4);
-        }
+        const auto* row = source.data() + static_cast<std::size_t>(y) * width * elementBytes;
+        for (std::uint32_t x = 0; x < width; ++x) std::memcpy(destination.data() + offset(x, y), row + static_cast<std::size_t>(x) * elementBytes, elementBytes);
     }
 }
 

core/libs/prx/libSceAgcDriver/Graphics/src/ColorTargetTransfer.cpp

@@ -7,7 +7,7 @@ namespace AgcDriver::Graphics {
 
 void ReadColorTarget(const ColorTarget& target, std::span<std::byte> destination) {
     PerformanceTimer timing("ColorTarget.Read");
-    const ColorTargetLayout layout(target.extent.width, target.extent.height, target.tileMode);
+    const ColorTargetLayout layout(target.extent.width, target.extent.height, target.tileMode, target.elementBytes);
     Require(target.bytes == layout.Bytes() && destination.size() == layout.LinearBytes(), "color target transfer size mismatch");
     if (target.tileMode == ColorTileMode::Linear) {
         GuestMemory::Read(target.address, destination, layout.Alignment());
@@ -23,7 +23,7 @@ void ReadColorTarget(const ColorTarget& target, std::span<std::byte> destination
 
 void WriteColorTarget(const ColorTarget& target, std::span<const std::byte> source) {
     PerformanceTimer timing("ColorTarget.Write");
-    const ColorTargetLayout layout(target.extent.width, target.extent.height, target.tileMode);
+    const ColorTargetLayout layout(target.extent.width, target.extent.height, target.tileMode, target.elementBytes);
     Require(target.bytes == layout.Bytes() && source.size() == layout.LinearBytes(), "color target transfer size mismatch");
     if (target.tileMode == ColorTileMode::Linear) {
         GuestMemory::Write(target.address, source, layout.Alignment());

core/libs/prx/libSceAgcDriver/Graphics/src/CommandCompletion.cpp

@@ -1,15 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/Resources.hpp"
-
-namespace AgcDriver::Graphics {
-
-bool CommandBatch::IsComplete() {
-    Require(submitted, "cannot query an unsubmitted command batch");
-    if (!pending) return true;
-    const auto result = context.Function<PFN_vkGetFenceStatus>("vkGetFenceStatus")(context.device, fence);
-    if (result == VK_NOT_READY) return false;
-    Check(result, "vkGetFenceStatus graphics");
-    pending = false;
-    return true;
-}
-
-}

core/libs/prx/libSceAgcDriver/Graphics/src/DccMetadata.cpp

@@ -0,0 +1,532 @@
+#include "prx/libSceAgcDriver/Graphics/include/DccMetadata.hpp"
+#include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/GuestBufferMemory.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/Recorder.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/Resources.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/TextureFormat.hpp"
+#include "prx/libSceAgcDriver/Graphics/include/UnitShadow.hpp"
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <memory>
+#include <mutex>
+#include <optional>
+#include <set>
+#include <utility>
+#include <vector>
+
+namespace AgcDriver::Graphics {
+namespace {
+
+constexpr std::uint64_t KeyBytes = 256;
+
+// Whether every key in [keys, keys + count) equals `first`. Eight keys per compare: the scan runs
+// on every texture and storage validation, and an all-0xff (uncompressed) surface's keys are read
+// to the end each time (132 KiB for a 4K RGBA8 target). APS5_NO_DCC_WORD_SCAN=1 restores the byte loop.
+bool AllKeysEqual(const std::uint8_t* keys, std::size_t count, std::uint8_t first) {
+    static const bool byteScan = std::getenv("APS5_NO_DCC_WORD_SCAN") != nullptr;
+    if (byteScan) return std::all_of(keys, keys + count, [&](std::uint8_t key) { return key == first; });
+    const std::uint64_t pattern = 0x0101010101010101ull * first;
+    std::size_t i = 0;
+    for (; i + 8 <= count; i += 8) {
+        std::uint64_t word;
+        std::memcpy(&word, keys + i, sizeof(word));
+        if (word != pattern) return false;
+    }
+    for (; i < count; ++i) {
+        if (keys[i] != first) return false;
+    }
+    return true;
+}
+
+// APS5_PROFILE_DRAW: key scans, their bytes and time, printed as [dcc] every 10 s, with the hits of
+// the GPU-store memo below and the "uncompressed" key stores by where they were made. Flush syncs
+// stay 0 (no key read syncs the recorder); the field is kept so the line's shape does not change.
+struct ScanProfile {
+    std::atomic<std::uint64_t> scans{0};
+    std::atomic<std::uint64_t> bytes{0};
+    std::atomic<std::uint64_t> nanoseconds{0};
+    std::atomic<std::uint64_t> memoHits{0};
+    std::atomic<std::uint64_t> flushSyncs{0};
+    std::atomic<std::uint64_t> gpuStores{0};
+    // GPU stores whose range had a 4-byte unaligned head or tail (copied instead of filled).
+    std::atomic<std::uint64_t> gpuSplitStores{0};
+    std::atomic<std::uint64_t> cpuStores{0};
+    std::atomic<std::int64_t> lastReport{0};
+};
+
+ScanProfile& Scans() {
+    static ScanProfile profile;
+    return profile;
+}
+
+// ProvedClearKeys outcomes, counted with or without profiling (the tests read them).
+struct ProofCounters {
+    std::atomic<std::uint64_t> proved{0};
+    std::atomic<std::uint64_t> scanned{0};
+    std::atomic<std::uint64_t> unstable{0};
+};
+
+ProofCounters& Proofs() {
+    static ProofCounters counters;
+    return counters;
+}
+
+bool ScanProfileEnabled() {
+    static const bool profile = std::getenv("APS5_PROFILE_DRAW") != nullptr;
+    return profile;
+}
+
+void ReportScans(std::chrono::steady_clock::time_point now) {
+    auto& profile = Scans();
+    const auto nowMs = std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
+    auto last = profile.lastReport.load();
+    // The first event starts the period rather than reporting an empty one.
+    if (last == 0) {
+        profile.lastReport.compare_exchange_strong(last, nowMs);
+        return;
+    }
+    if (nowMs - last < 10000 || !profile.lastReport.compare_exchange_strong(last, nowMs)) return;
+    const auto& proofs = Proofs();
+    std::fprintf(stderr, "[dcc] %llu scans, %.1f MiB scanned, %llu memo hits, %llu flush syncs, %.1f ms; uncompressed keys stored: %llu on the GPU (%llu with an unaligned head or tail), %llu on the CPU; key proofs: %llu proved, %llu scanned, %llu unstable\n", static_cast<unsigned long long>(profile.scans.load()), profile.bytes.load() / 1048576.0, static_cast<unsigned long long>(profile.memoHits.load()), static_cast<unsigned long long>(profile.flushSyncs.load()), profile.nanoseconds.load() / 1e6, static_cast<unsigned long long>(profile.gpuStores.load()), static_cast<unsigned long long>(profile.gpuSplitStores.load()), static_cast<unsigned long long>(profile.cpuStores.load()), static_cast<unsigned long long>(proofs.proved.load()), static_cast<unsigned long long>(proofs.scanned.load()), static_cast<unsigned long long>(proofs.unstable.load()));
+}
+
+void CountScan(std::size_t bytes, std::chrono::steady_clock::time_point start) {
+    auto& profile = Scans();
+    const auto now = std::chrono::steady_clock::now();
+    profile.scans.fetch_add(1, std::memory_order_relaxed);
+    profile.bytes.fetch_add(bytes, std::memory_order_relaxed);
+    profile.nanoseconds.fetch_add(static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::nanoseconds>(now - start).count()), std::memory_order_relaxed);
+    ReportScans(now);
+}
+
+void CountStore(std::atomic<std::uint64_t>& counter) {
+    if (!ScanProfileEnabled()) return;
+    counter.fetch_add(1, std::memory_order_relaxed);
+    ReportScans(std::chrono::steady_clock::now());
+}
+
+// "Uncompressed" keys the driver stored on the GPU (MarkDccUncompressed's recorded fill) whose batch
+// may not have run yet: the bytes ReadDccKeys scans would still show the keys from before the store
+// (a fast-clear code), and a storage image whose write-back made the store would be re-uploaded as
+// cleared. While recorded work that writes the range is still running, the range reads as
+// uncompressed from here; the bytes are trusted again only once the newest recorded writer of the
+// range has signaled its fence (batches of the recorder's queue finish in order, so the fill and
+// every writer up to that one have landed and the bytes are final, whichever batch wrote last).
+// Dropping an entry on a later batch's serial alone would not do: until the fill's batch runs the
+// bytes PREDATE the fill (the previous clear code), and a later writer that is not a same-code clear
+// (a decompress kernel, a partial key write, a false overlap from an unrelated V#) would make the
+// next Refresh compare that stale code against the uploaded keys. The lookup runs with or without
+// GuestMemory::GpuMutex (a resource build's stage A reads keys unlocked, its stage B re-reads them
+// under the lock): unlocked it can only ask the lock-free snapshot whether the range is written at
+// all. Entries belong to one recorder: when the active recorder changes (device replacement; a new
+// recorder can reuse the old one's address) they are all dropped. Its mutex is a leaf (nothing is
+// taken under it).
+struct GpuKeyStore {
+    std::uint64_t begin;
+    std::uint64_t end;
+};
+
+struct KeyStoreMemo {
+    std::mutex mutex;
+    std::vector<GpuKeyStore> entries;
+    const Recorder* recorder = nullptr;
+};
+
+KeyStoreMemo& Memo() {
+    static KeyStoreMemo memo;
+    return memo;
+}
+
+// Under memo.mutex: the active recorder, with the entries of any other recorder dropped.
+Recorder* MemoRecorder(KeyStoreMemo& memo) {
+    auto* active = Recorder::Active();
+    if (active != memo.recorder) {
+        memo.entries.clear();
+        memo.recorder = active;
+    }
+    return active;
+}
+
+void ForgetStores(std::uint64_t begin, std::uint64_t end) {
+    auto& memo = Memo();
+    std::lock_guard lock(memo.mutex);
+    std::erase_if(memo.entries, [&](const GpuKeyStore& store) { return begin < store.end && store.begin < end; });
+}
+
+// Uncompressed when a GPU store still pending covers the whole range, else nullopt (the bytes decide).
+std::optional<DccKeys> MemoizedKeys(std::uint64_t begin, std::uint64_t end) {
+    auto& memo = Memo();
+    std::lock_guard lock(memo.mutex);
+    auto* recorder = MemoRecorder(memo);
+    for (auto it = memo.entries.begin(); it != memo.entries.end(); ++it) {
+        if (it->begin > begin || it->end < end) continue;
+        const auto bytes = static_cast<std::size_t>(it->end - it->begin);
+        bool pending = false;
+        if (recorder != nullptr) {
+            if (GuestMemory::GpuMutex().HeldByThisThread()) {
+                // The open batch is never signaled (also with APS5_NO_SYNC_THROUGH, where the info
+                // carries the open flag with the newest in-flight batch's fence state).
+                const auto info = recorder->DescribePendingWrite(it->begin, bytes);
+                pending = info.has_value() && !info->signaled;
+            } else {
+                pending = Recorder::SnapshotWriteOverlaps(it->begin, bytes);
+            }
+        }
+        if (!pending) {
+            memo.entries.erase(it);
+            return std::nullopt;
+        }
+        if (ScanProfileEnabled()) Scans().memoHits.fetch_add(1, std::memory_order_relaxed);
+        return DccKeys::Uncompressed;
+    }
+    return std::nullopt;
+}
+
+// Debug aid (APS5_TRACE_DCC_KEYS=1): every uncompressed key store, with the packet that made it.
+void TraceKeyStore(const char* path, std::uint64_t begin, std::size_t count) {
+    static const bool trace = std::getenv("APS5_TRACE_DCC_KEYS") != nullptr;
+    if (!trace) return;
+    const auto packet = GuestMemory::CurrentPacket();
+    std::fprintf(stderr, "[dcc-keys] uncompressed keys stored on the %s: 0x%llx+0x%zx (packet 0x%x queue 0x%x)\n", path, static_cast<unsigned long long>(begin), count, packet.opcode, packet.queue);
+}
+
+// The 0xff keys as a fill recorded into the active recorder's open batch, into the range's host
+// import (the WriteLabelOnGpu / FillBuffer pattern): ordered behind every earlier recorded read or
+// write of the range (the title's DCC clear or decompress kernel storing keys through a V#, which the
+// CPU store had to wait for) and visible to the work recorded after it and to the host. vkCmdFillBuffer
+// needs a 4-byte aligned offset and size; an unaligned head or tail is copied from a kept 4-byte
+// buffer of 0xff instead. Under GuestMemory::GpuMutex (it records). False when the keys cannot be
+// stored this way (memory the GPU has no view of): the caller stores them on the CPU.
+bool StoreUncompressedOnGpu(const Context& context, Recorder& recorder, std::uint64_t begin, std::size_t count) {
+    const auto* import = HostImportFor(context, begin, count);
+    if (import == nullptr) return false;
+    const VkDeviceSize first = begin - import->base;
+    const VkDeviceSize last = first + count;
+    const VkDeviceSize fillBegin = std::min((first + 3) & ~VkDeviceSize{3}, last);
+    const VkDeviceSize fillEnd = std::max(last & ~VkDeviceSize{3}, fillBegin);
+    // Everything that can throw (the seed's allocation, the batch's command buffer) comes before the
+    // first recorded command: a fill left in the batch without its note and memo entry would be an
+    // unordered write of the keys.
+    std::shared_ptr<Buffer> seed;
+    if (fillBegin > first || last > fillEnd) {
+        seed = std::make_shared<Buffer>(context, 4, VK_BUFFER_USAGE_TRANSFER_SRC_BIT);
+        std::memset(seed->Bytes().data(), 0xff, seed->Bytes().size());
+        CountStore(Scans().gpuSplitStores);
+    }
+    // A unit shadow's results in the key range's edge units reach the import before the fill
+    // lands over part of them (the stamp below then makes those units stale).
+    if (AnyShadowedOverlaps(begin, count)) PublishShadow(begin, count, PublishScope::PartialUnits, PublishReason::Keys);
+    // Queued on the open batch and recorded with the batch's other key stores as one run (or before
+    // a later command writing the keys; see Recorder::QueueKeyStore).
+    recorder.QueueKeyStore(import->buffer, first, last, std::move(seed), begin, begin + count);
+    recorder.NotePendingWrite(begin, count);
+    GuestMemory::MarkWritten(begin, count);
+    // After the note: an unlocked memo lookup drops an entry the snapshot does not cover yet. The
+    // entry is kept only for the active recorder (the one the flush hook and the lookups consult).
+    {
+        auto& memo = Memo();
+        std::lock_guard lock(memo.mutex);
+        const auto end = begin + count;
+        std::erase_if(memo.entries, [&](const GpuKeyStore& store) { return begin < store.end && store.begin < end; });
+        if (MemoRecorder(memo) == &recorder) memo.entries.push_back({begin, end});
+    }
+    CountStore(Scans().gpuStores);
+    TraceKeyStore("gpu", begin, count);
+    return true;
+}
+
+void StoreUncompressedOnCpu(std::uint64_t begin, std::size_t count) {
+    TraceKeyStore("cpu", begin, count);
+    // The write waits for recorded work over the range (the flush hook), so the bytes are current after.
+    ForgetStores(begin, begin + count);
+    const std::vector<std::byte> uncompressed(count, std::byte{0xff});
+    GuestMemory::Write(begin, uncompressed);
+    CountStore(Scans().cpuStores);
+}
+
+// Whether the keys already read as uncompressed: scans the same bytes as ReadDccKeys, so it counts
+// as a scan too.
+bool AlreadyUncompressed(std::uint64_t metaAddress, std::size_t count) {
+    const auto* keys = reinterpret_cast<const std::uint8_t*>(metaAddress);
+    const auto start = ScanProfileEnabled() ? std::chrono::steady_clock::now() : std::chrono::steady_clock::time_point{};
+    const bool uncompressed = AllKeysEqual(keys, count, 0xff);
+    if (ScanProfileEnabled()) CountScan(count, start);
+    return uncompressed;
+}
+
+bool CpuKeysOnly() {
+    static const bool cpuKeys = std::getenv("APS5_CPU_DCC_KEYS") != nullptr;
+    return cpuKeys;
+}
+
+enum class Kind { Unorm, Snorm, Uint, Sint, Float };
+
+struct Layout {
+    std::uint32_t channels;
+    std::uint32_t bits[4];
+    Kind kind;
+};
+
+bool LayoutFor(VkFormat format, Layout& layout) {
+    switch (format) {
+        case VK_FORMAT_R8_UNORM: case VK_FORMAT_R8_SRGB: layout = {1, {8}, Kind::Unorm}; return true;
+        case VK_FORMAT_R8_UINT: layout = {1, {8}, Kind::Uint}; return true;
+        case VK_FORMAT_R16_UNORM: layout = {1, {16}, Kind::Unorm}; return true;
+        case VK_FORMAT_R16_SNORM: layout = {1, {16}, Kind::Snorm}; return true;
+        case VK_FORMAT_R16_UINT: layout = {1, {16}, Kind::Uint}; return true;
+        case VK_FORMAT_R16_SINT: layout = {1, {16}, Kind::Sint}; return true;
+        case VK_FORMAT_R16_SFLOAT: layout = {1, {16}, Kind::Float}; return true;
+        case VK_FORMAT_R8G8_UNORM: case VK_FORMAT_R8G8_SRGB: layout = {2, {8, 8}, Kind::Unorm}; return true;
+        case VK_FORMAT_R8G8_SNORM: layout = {2, {8, 8}, Kind::Snorm}; return true;
+        case VK_FORMAT_R8G8_UINT: layout = {2, {8, 8}, Kind::Uint}; return true;
+        case VK_FORMAT_R8G8_SINT: layout = {2, {8, 8}, Kind::Sint}; return true;
+        case VK_FORMAT_R32_UINT: layout = {1, {32}, Kind::Uint}; return true;
+        case VK_FORMAT_R32_SINT: layout = {1, {32}, Kind::Sint}; return true;
+        case VK_FORMAT_R32_SFLOAT: layout = {1, {32}, Kind::Float}; return true;
+        case VK_FORMAT_R16G16_UNORM: layout = {2, {16, 16}, Kind::Unorm}; return true;
+        case VK_FORMAT_R16G16_SNORM: layout = {2, {16, 16}, Kind::Snorm}; return true;
+        case VK_FORMAT_R16G16_UINT: layout = {2, {16, 16}, Kind::Uint}; return true;
+        case VK_FORMAT_R16G16_SINT: layout = {2, {16, 16}, Kind::Sint}; return true;
+        case VK_FORMAT_R16G16_SFLOAT: layout = {2, {16, 16}, Kind::Float}; return true;
+        case VK_FORMAT_A2B10G10R10_UNORM_PACK32: layout = {4, {10, 10, 10, 2}, Kind::Unorm}; return true;
+        case VK_FORMAT_A2B10G10R10_UINT_PACK32: layout = {4, {10, 10, 10, 2}, Kind::Uint}; return true;
+        case VK_FORMAT_R8G8B8A8_UNORM: case VK_FORMAT_R8G8B8A8_SRGB: case VK_FORMAT_B8G8R8A8_UNORM: case VK_FORMAT_B8G8R8A8_SRGB: layout = {4, {8, 8, 8, 8}, Kind::Unorm}; return true;
+        case VK_FORMAT_R8G8B8A8_SNORM: layout = {4, {8, 8, 8, 8}, Kind::Snorm}; return true;
+        case VK_FORMAT_R8G8B8A8_UINT: layout = {4, {8, 8, 8, 8}, Kind::Uint}; return true;
+        case VK_FORMAT_R8G8B8A8_SINT: layout = {4, {8, 8, 8, 8}, Kind::Sint}; return true;
+        case VK_FORMAT_R32G32_UINT: layout = {2, {32, 32}, Kind::Uint}; return true;
+        case VK_FORMAT_R32G32_SINT: layout = {2, {32, 32}, Kind::Sint}; return true;
+        case VK_FORMAT_R32G32_SFLOAT: layout = {2, {32, 32}, Kind::Float}; return true;
+        case VK_FORMAT_R16G16B16A16_UNORM: layout = {4, {16, 16, 16, 16}, Kind::Unorm}; return true;
+        case VK_FORMAT_R16G16B16A16_SNORM: layout = {4, {16, 16, 16, 16}, Kind::Snorm}; return true;
+        case VK_FORMAT_R16G16B16A16_UINT: layout = {4, {16, 16, 16, 16}, Kind::Uint}; return true;
+        case VK_FORMAT_R16G16B16A16_SINT: layout = {4, {16, 16, 16, 16}, Kind::Sint}; return true;
+        case VK_FORMAT_R16G16B16A16_SFLOAT: layout = {4, {16, 16, 16, 16}, Kind::Float}; return true;
+        case VK_FORMAT_R32G32B32A32_UINT: layout = {4, {32, 32, 32, 32}, Kind::Uint}; return true;
+        case VK_FORMAT_R32G32B32A32_SINT: layout = {4, {32, 32, 32, 32}, Kind::Sint}; return true;
+        case VK_FORMAT_R32G32B32A32_SFLOAT: layout = {4, {32, 32, 32, 32}, Kind::Float}; return true;
+        default: return false;
+    }
+}
+
+// "1" in a channel of this kind and width.
+std::uint64_t One(Kind kind, std::uint32_t bits) {
+    switch (kind) {
+        case Kind::Unorm: case Kind::Uint: return bits >= 64 ? ~0ull : (1ull << bits) - 1u;
+        case Kind::Snorm: case Kind::Sint: return (1ull << (bits - 1u)) - 1u;
+        case Kind::Float: return bits == 16 ? 0x3c00u : 0x3f800000u;
+    }
+    return 0;
+}
+
+}
+
+const char* DccKeysName(DccKeys keys) {
+    switch (keys) {
+        case DccKeys::Uncompressed: return "uncompressed";
+        case DccKeys::Clear0000: return "0000";
+        case DccKeys::Clear0001: return "0001";
+        case DccKeys::Clear1110: return "1110";
+        case DccKeys::Clear1111: return "1111";
+        case DccKeys::ClearRegister: return "register";
+        case DccKeys::Mixed: return "mixed";
+        case DccKeys::Unreadable: return "unreadable";
+    }
+    return "?";
+}
+
+namespace {
+
+// ReadDccKeys, saying in `memoized` whether the answer came from the pending-store memo rather
+// than the bytes.
+DccKeys readDccKeys(std::uint64_t metaAddress, std::uint64_t surfaceBytes, bool& memoized) {
+    memoized = false;
+    const auto count = static_cast<std::size_t>(surfaceBytes / KeyBytes);
+    if (metaAddress == 0 || count == 0 || !GuestMemory::Accessible(reinterpret_cast<const void*>(metaAddress), count)) return DccKeys::Unreadable;
+    // A driver store still pending on the GPU is not in the bytes yet.
+    if (const auto memo = MemoizedKeys(metaAddress, metaAddress + count)) {
+        memoized = true;
+        return *memo;
+    }
+    // Keys under a unit shadow's fresh results (metadata aliased with a surface): under the
+    // device lock they are published and waited for; a build's unlocked stage answers "unstable"
+    // so its stage B reads them under the lock.
+    if (AnyShadowedOverlaps(metaAddress, count)) {
+        if (!GuestMemory::GpuMutex().HeldByThisThread()) {
+            memoized = true;
+            return DccKeys::Mixed;
+        }
+        PublishShadow(metaAddress, count, PublishScope::Whole, PublishReason::Keys);
+        if (auto* recorder = Recorder::Active()) {
+            Recorder::CountSync(2);
+            recorder->Sync();
+        }
+    }
+    const auto* keys = reinterpret_cast<const std::uint8_t*>(metaAddress);
+    const auto first = keys[0];
+    const auto start = ScanProfileEnabled() ? std::chrono::steady_clock::now() : std::chrono::steady_clock::time_point{};
+    const bool uniform = AllKeysEqual(keys + 1, count - 1, first);
+    if (ScanProfileEnabled()) CountScan(count, start);
+    if (!uniform) return DccKeys::Mixed;
+    switch (first) {
+        case 0x00: return DccKeys::Clear0000;
+        case 0x40: return DccKeys::Clear0001;
+        case 0x80: return DccKeys::Clear1110;
+        case 0xc0: return DccKeys::Clear1111;
+        case 0x20: return DccKeys::ClearRegister;
+        case 0xff: return DccKeys::Uncompressed;
+        default: return DccKeys::Mixed;
+    }
+}
+
+DccKeys textureClearKeys(const GuestTextureResource& resource, std::uint64_t guestBytes, bool& memoized) {
+    memoized = false;
+    if (resource.dccAddress == 0) return DccKeys::Uncompressed;
+    const auto keys = readDccKeys(resource.dccAddress, guestBytes, memoized);
+    if (keys == DccKeys::Uncompressed) return keys;
+    std::byte probe[16]{};
+    if (!IsDccClear(keys) || !FillDccClear(ResolveTextureFormat(resource.format), keys, resource.dccAlphaOnMsb, std::span(probe, std::min<std::size_t>(sizeof(probe), BytesPerElement(resource.format))))) {
+        static std::mutex reportedMutex;
+        static std::set<std::pair<std::uint64_t, int>> reported;
+        std::lock_guard lock(reportedMutex);
+        if (reported.size() < 32 && reported.insert({resource.baseAddress, static_cast<int>(keys)}).second) std::fprintf(stderr, "[gpu] texture 0x%llx (format %u) has %s DCC keys at 0x%llx; its texels are read as stored\n", static_cast<unsigned long long>(resource.baseAddress), resource.format, DccKeysName(keys), static_cast<unsigned long long>(resource.dccAddress));
+        return DccKeys::Uncompressed;
+    }
+    return keys;
+}
+
+}
+
+DccKeys ReadDccKeys(std::uint64_t metaAddress, std::uint64_t surfaceBytes) {
+    bool memoized = false;
+    return readDccKeys(metaAddress, surfaceBytes, memoized);
+}
+
+bool IsDccClear(DccKeys keys) {
+    return keys == DccKeys::Clear0000 || keys == DccKeys::Clear0001 || keys == DccKeys::Clear1110 || keys == DccKeys::Clear1111 || keys == DccKeys::ClearRegister;
+}
+
+void MarkDccUncompressed(std::uint64_t metaAddress, std::uint64_t surfaceBytes) {
+    const auto count = static_cast<std::size_t>(surfaceBytes / KeyBytes);
+    if (metaAddress == 0 || count == 0 || !GuestMemory::Accessible(reinterpret_cast<const void*>(metaAddress), count, true)) return;
+    // The bytes are trusted only when no recorded work writes them (lock-free, with or without
+    // GuestMemory::GpuMutex); otherwise the store runs and waits through the flush hook. A pending
+    // GPU store of the driver's own is no reason to skip: a title kernel recorded into the same
+    // batch after it may write clear keys, and the memo cannot order inside a batch.
+    if (!Recorder::SnapshotWriteOverlaps(metaAddress, count) && AlreadyUncompressed(metaAddress, count)) return;
+    StoreUncompressedOnCpu(metaAddress, count);
+}
+
+void MarkDccUncompressed(const Context& context, std::uint64_t metaAddress, std::uint64_t surfaceBytes) {
+    const auto count = static_cast<std::size_t>(surfaceBytes / KeyBytes);
+    if (metaAddress == 0 || count == 0 || !GuestMemory::Accessible(reinterpret_cast<const void*>(metaAddress), count, true)) return;
+    auto* recorder = CpuKeysOnly() || !GuestMemory::GpuMutex().HeldByThisThread() ? nullptr : Recorder::Active();
+    // The bytes are scanned only when no recorded work writes them: a pending clear kernel's keys are
+    // not in memory yet, so the bytes could read as uncompressed while the kernel will store a clear
+    // code, and the range is stored without looking (the fill lands after the kernel). That includes
+    // a pending GPU store of the driver's own (the memo): a title kernel recorded into the same batch
+    // after it may write clear keys, and nothing orders inside a batch, so a fresh fill is recorded
+    // (idempotent, no wait) rather than trusting the earlier one.
+    const bool pending = recorder != nullptr && recorder->PendingWriteOverlaps(metaAddress, count);
+    if (!pending && AlreadyUncompressed(metaAddress, count)) return;
+    if (recorder != nullptr && StoreUncompressedOnGpu(context, *recorder, metaAddress, count)) return;
+    StoreUncompressedOnCpu(metaAddress, count);
+}
+
+bool FillDccClear(VkFormat format, DccKeys keys, bool alphaOnMsb, std::span<std::byte> bytes) {
+    if (keys == DccKeys::Clear0000) {
+        std::fill(bytes.begin(), bytes.end(), std::byte{0});
+        return true;
+    }
+    Layout layout{};
+    if (!IsDccClear(keys) || keys == DccKeys::ClearRegister || !LayoutFor(format, layout)) return false;
+    const bool color = keys == DccKeys::Clear1110 || keys == DccKeys::Clear1111;
+    const bool alpha = keys == DccKeys::Clear0001 || keys == DccKeys::Clear1111;
+    const int alphaChannel = layout.channels == 3 ? -1 : alphaOnMsb ? static_cast<int>(layout.channels) - 1 : 0;
+    std::uint32_t elementBits = 0;
+    for (std::uint32_t channel = 0; channel < layout.channels; ++channel) elementBits += layout.bits[channel];
+    std::vector<std::byte> element(elementBits / 8u);
+    std::uint32_t bit = 0;
+    for (std::uint32_t channel = 0; channel < layout.channels; ++channel) {
+        const bool set = static_cast<int>(channel) == alphaChannel ? alpha : color;
+        const auto value = set ? One(layout.kind, layout.bits[channel]) : 0u;
+        for (std::uint32_t i = 0; i < layout.bits[channel]; ++i, ++bit) {
+            if (((value >> i) & 1u) != 0) element[bit / 8u] |= static_cast<std::byte>(1u << (bit % 8u));
+        }
+    }
+    for (std::size_t offset = 0; offset + element.size() <= bytes.size(); offset += element.size()) std::memcpy(bytes.data() + offset, element.data(), element.size());
+    return true;
+}
+
+bool DccAlphaOnMsb(VkFormat format, std::uint32_t componentSwap) {
+    Layout layout{};
+    const auto channels = LayoutFor(format, layout) ? layout.channels : 4u;
+    constexpr std::uint32_t standardReversed = 2;
+    constexpr std::uint32_t alternateReversed = 3;
+    if (channels == 1) return componentSwap == alternateReversed;
+    return componentSwap != standardReversed && componentSwap != alternateReversed;
+}
+
+DccKeys TextureClearKeys(const GuestTextureResource& resource, std::uint64_t guestBytes) {
+    bool memoized = false;
+    return textureClearKeys(resource, guestBytes, memoized);
+}
+
+bool KeyFastPath() {
+    static const bool enabled = std::getenv("APS5_NO_KEY_FAST_PATH") == nullptr;
+    return enabled;
+}
+
+DccKeys ProvedClearKeys(const GuestTextureResource& resource, std::uint64_t guestBytes, DccKeyProof& proof) {
+    if (resource.dccAddress == 0) return DccKeys::Uncompressed;
+    auto& counters = Proofs();
+    if (!KeyFastPath()) {
+        counters.scanned.fetch_add(1, std::memory_order_relaxed);
+        return TextureClearKeys(resource, guestBytes);
+    }
+    const auto count = static_cast<std::size_t>(guestBytes / KeyBytes);
+    // The collect precedes the scan: a write landing between them is stamped above `collected`, so
+    // the proof it would invalidate is never taken as current.
+    const auto collected = count == 0 ? 0 : GuestMemory::CollectWrites(resource.dccAddress, count);
+    if (proof.generation != 0 && collected != 0 && GuestMemory::UnchangedSince(resource.dccAddress, count, proof.generation)) {
+        counters.proved.fetch_add(1, std::memory_order_relaxed);
+        return proof.keys;
+    }
+    bool memoized = false;
+    const auto keys = textureClearKeys(resource, guestBytes, memoized);
+    bool stable = collected != 0 && !memoized;
+    if (stable) {
+        // After the scan: a write noted since the collect is stamped above it anyway; one noted
+        // before it may still land after the bytes were read.
+        if (GuestMemory::GpuMutex().HeldByThisThread()) {
+            if (auto* recorder = Recorder::Active(); recorder != nullptr) {
+                const auto info = recorder->DescribePendingWrite(resource.dccAddress, count);
+                stable = !info.has_value() || info->signaled;
+            }
+        } else {
+            stable = !Recorder::SnapshotWriteOverlaps(resource.dccAddress, count);
+        }
+    }
+    proof = stable ? DccKeyProof{keys, collected} : DccKeyProof{};
+    counters.scanned.fetch_add(1, std::memory_order_relaxed);
+    if (!stable) counters.unstable.fetch_add(1, std::memory_order_relaxed);
+    return keys;
+}
+
+DccKeyProofCounts KeyProofCounts() {
+    const auto& counters = Proofs();
+    return {counters.proved.load(std::memory_order_relaxed), counters.scanned.load(std::memory_order_relaxed), counters.unstable.load(std::memory_order_relaxed)};
+}
+
+void ReadTextureSurface(const GuestTextureResource& resource, DccKeys keys, std::span<std::byte> bytes) {
+    // Named for the [hooksync] attribution: the read goes through the flush hook.
+    const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::TextureRead);
+    if (keys == DccKeys::Uncompressed) GuestMemory::ReadCommitted(resource.baseAddress, bytes);
+    else FillDccClear(ResolveTextureFormat(resource.format), keys, resource.dccAlphaOnMsb, bytes);
+}
+
+}

core/libs/prx/libSceAgcDriver/Graphics/src/DescriptorCache.cpp

@@ -1,65 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/DescriptorCache.hpp"
-#include <algorithm>
-#include <utility>
-
-namespace AgcDriver::Graphics {
-
-DescriptorAllocation::DescriptorAllocation(const Context& context, std::span<const std::uint32_t> key, std::span<const VkDescriptorSetLayoutBinding> bindings, std::span<const VkDescriptorPoolSize> sizes) : device(context.device), destroyPool(context.Function<PFN_vkDestroyDescriptorPool>("vkDestroyDescriptorPool")), destroyLayout(context.Function<PFN_vkDestroyDescriptorSetLayout>("vkDestroyDescriptorSetLayout")), key(key.begin(), key.end()) {
-    try {
-        VkDescriptorSetLayoutCreateInfo info{VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO};
-        info.bindingCount = static_cast<std::uint32_t>(bindings.size());
-        info.pBindings = bindings.data();
-        Check(context.Function<PFN_vkCreateDescriptorSetLayout>("vkCreateDescriptorSetLayout")(device, &info, nullptr, &layout), "vkCreateDescriptorSetLayout");
-        if (bindings.empty()) return;
-        VkDescriptorPoolCreateInfo poolInfo{VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO};
-        poolInfo.maxSets = 1;
-        poolInfo.poolSizeCount = static_cast<std::uint32_t>(sizes.size());
-        poolInfo.pPoolSizes = sizes.data();
-        Check(context.Function<PFN_vkCreateDescriptorPool>("vkCreateDescriptorPool")(device, &poolInfo, nullptr, &pool), "vkCreateDescriptorPool");
-        VkDescriptorSetAllocateInfo allocation{VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO};
-        allocation.descriptorPool = pool;
-        allocation.descriptorSetCount = 1;
-        allocation.pSetLayouts = &layout;
-        Check(context.Function<PFN_vkAllocateDescriptorSets>("vkAllocateDescriptorSets")(device, &allocation, &set), "vkAllocateDescriptorSets");
-    } catch (...) {
-        release();
-        throw;
-    }
-}
-
-DescriptorAllocation::~DescriptorAllocation() {
-    release();
-}
-
-void DescriptorAllocation::release() noexcept {
-    if (pool) destroyPool(device, pool, nullptr);
-    if (layout) destroyLayout(device, layout, nullptr);
-}
-
-bool DescriptorAllocation::Matches(std::span<const std::uint32_t> candidate) const {
-    return std::equal(key.begin(), key.end(), candidate.begin(), candidate.end());
-}
-
-std::unique_ptr<DescriptorAllocation> DescriptorCache::Take(std::span<const std::uint32_t> key) {
-    std::lock_guard lock(mutex);
-    for (auto& allocation : available) {
-        if (allocation && allocation->Matches(key)) return std::move(allocation);
-    }
-    return nullptr;
-}
-
-void DescriptorCache::Put(std::unique_ptr<DescriptorAllocation> allocation) {
-    Require(allocation != nullptr, "cannot cache an empty descriptor allocation");
-    std::lock_guard lock(mutex);
-    for (std::size_t offset = 0; offset < available.size(); ++offset) {
-        const auto index = (cursor + offset) % available.size();
-        if (available[index]) continue;
-        available[index] = std::move(allocation);
-        cursor = (index + 1) % available.size();
-        return;
-    }
-    available[cursor] = std::move(allocation);
-    cursor = (cursor + 1) % available.size();
-}
-
-}

core/libs/prx/libSceAgcDriver/Graphics/src/DrawCompletion.cpp

@@ -1,56 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/DrawQueue.hpp"
-#include "prx/libSceAgcDriver/Execution/include/MemoryAccessScope.hpp"
-#include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
-
-namespace AgcDriver::Graphics {
-
-void DrawQueue::retire(Batch batch) {
-    PerformanceTimer timing("Graphics.DrawQueue.Retire");
-    const GuestMemory::MemoryAccessScope suspended(nullptr, nullptr);
-    Require(drawCount >= batch.entries.size(), "draw queue completion count underflow");
-    drawCount -= batch.entries.size();
-    for (auto& entry : batch.entries) entry.resources->WriteBack();
-    timing.Mark("resources_writeback");
-    batch.entries.clear();
-    available.push_back(std::move(batch.commands));
-    timing.Mark("resources_release");
-}
-
-void DrawQueue::Collect() {
-    while (!pending.empty() && pending.front().commands->IsComplete()) {
-        auto batch = std::move(pending.front());
-        pending.erase(pending.begin());
-        retire(std::move(batch));
-    }
-}
-
-void DrawQueue::WaitGpu() {
-    Flush();
-    for (auto& batch : pending) batch.commands->Wait();
-}
-
-void DrawQueue::Wait() {
-    if (pending.empty() && !recording.commands) return;
-    PerformanceTimer timing("Graphics.DrawQueue.Wait");
-    Flush();
-    timing.Mark("submit");
-    while (!pending.empty()) {
-        pending.front().commands->Wait();
-        timing.Mark("fence_wait");
-        auto batch = std::move(pending.front());
-        pending.erase(pending.begin());
-        retire(std::move(batch));
-        timing.Mark("retire");
-    }
-}
-
-void DrawQueue::RecordMemoryBarrier(const Context& context) {
-    const auto commands = Begin(context);
-    VkMemoryBarrier barrier{VK_STRUCTURE_TYPE_MEMORY_BARRIER};
-    barrier.srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT | VK_ACCESS_HOST_WRITE_BIT;
-    barrier.dstAccessMask = VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT;
-    context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier")(commands, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT | VK_PIPELINE_STAGE_HOST_BIT, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, 0, 1, &barrier, 0, nullptr, 0, nullptr);
-    recording.hasBarrier = true;
-}
-
-}

core/libs/prx/libSceAgcDriver/Graphics/src/DrawQueue.cpp

@@ -1,48 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/DrawQueue.hpp"
-#include "prx/libSceAgcDriver/Execution/include/MemoryAccessScope.hpp"
-#include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
-#include <algorithm>
-
-namespace AgcDriver::Graphics {
-
-DrawQueue::~DrawQueue() {
-    recording.commands.reset();
-    for (auto& batch : pending) batch.commands.reset();
-}
-
-VkCommandBuffer DrawQueue::Begin(const Context& context) {
-    Collect();
-    if (drawCount >= 64) Wait();
-    if (!recording.commands) {
-        if (available.empty()) recording.commands = std::make_unique<CommandBatch>(context);
-        else {
-            recording.commands = std::move(available.back());
-            available.pop_back();
-            recording.commands->Reset();
-        }
-    }
-    return recording.commands->Handle();
-}
-
-void DrawQueue::Enqueue(std::shared_ptr<ShaderResources> resources, std::shared_ptr<void> storage) {
-    Require(recording.commands != nullptr && resources != nullptr && storage != nullptr, "draw batch is incomplete");
-    recording.entries.push_back({std::move(storage), std::move(resources)});
-    ++drawCount;
-    if (recording.entries.size() >= 8) Flush();
-}
-
-void DrawQueue::Flush() {
-    if (!recording.commands) return;
-    Require(!recording.entries.empty() || recording.hasBarrier, "cannot submit an incomplete draw batch");
-    pending.push_back(std::move(recording));
-    recording = Batch{};
-    pending.back().commands->Submit();
-}
-
-void DrawQueue::Resolve(std::uint64_t address, std::size_t bytes) {
-    const auto overlaps = [&](const auto& entry) { return entry.resources->WritesOverlap(address, bytes); };
-    if (std::any_of(recording.entries.begin(), recording.entries.end(), overlaps) || std::any_of(pending.begin(), pending.end(), [&](const auto& batch) { return std::any_of(batch.entries.begin(), batch.entries.end(), overlaps); })) Wait();
-}
-
-
-}

core/libs/prx/libSceAgcDriver/Graphics/src/GpuColorTransfer.cpp

@@ -2,8 +2,6 @@
 #include "prx/libSceAgcDriver/Graphics/shaders/ColorTransfer_spv.h"
 #include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp"
 #include <array>
-#include <cstring>
-#include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
 
 namespace AgcDriver::Graphics {
 
@@ -72,11 +70,11 @@ void GpuColorTransfer::prepare(std::uint32_t newWidth, std::uint32_t newHeight,
     Require(layout.Bytes() <= context.limits.maxStorageBufferRange && layout.LinearBytes() <= context.limits.maxStorageBufferRange, "color transfer exceeds storage buffer limits");
     Require((newWidth + 7u) / 8u <= context.limits.maxComputeWorkGroupCount[0] && (newHeight + 7u) / 8u <= context.limits.maxComputeWorkGroupCount[1], "color transfer exceeds workgroup limits");
     if (tiled && width == newWidth && height == newHeight && mode == newMode) return;
-    auto newTiled = std::make_unique<Buffer>(context, layout.Bytes(), VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT);
-    auto newLinear = std::make_unique<Buffer>(context, layout.LinearBytes(), VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT);
-    auto newReadback = std::make_unique<Buffer>(context, layout.Bytes(), VK_BUFFER_USAGE_TRANSFER_DST_BIT, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_CACHED_BIT);
-    auto newUpload = std::make_unique<Buffer>(context, layout.Bytes(), VK_BUFFER_USAGE_TRANSFER_SRC_BIT);
-    const std::array<VkDescriptorBufferInfo, 2> buffers{{{newTiled->Handle(), 0, layout.Bytes()}, {newLinear->Handle(), 0, layout.LinearBytes()}}};
+    constexpr VkBufferUsageFlags copies = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT;
+    auto newTiled = std::make_unique<Buffer>(context, layout.Bytes(), copies);
+    auto newTiledDevice = std::make_unique<DeviceBuffer>(context, layout.Bytes(), copies | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT);
+    auto newLinear = std::make_unique<DeviceBuffer>(context, layout.LinearBytes(), copies | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT);
+    const std::array<VkDescriptorBufferInfo, 2> buffers{{{newTiledDevice->Handle(), 0, layout.Bytes()}, {newLinear->Handle(), 0, layout.LinearBytes()}}};
     std::array<VkWriteDescriptorSet, 2> writes{};
     for (std::uint32_t i = 0; i < writes.size(); ++i) {
         writes[i].sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
@@ -88,31 +86,30 @@ void GpuColorTransfer::prepare(std::uint32_t newWidth, std::uint32_t newHeight,
     }
     context.Function<PFN_vkUpdateDescriptorSets>("vkUpdateDescriptorSets")(context.device, static_cast<std::uint32_t>(writes.size()), writes.data(), 0, nullptr);
     tiled = std::move(newTiled);
+    tiledDevice = std::move(newTiledDevice);
     linear = std::move(newLinear);
-    readback = std::move(newReadback);
-    upload = std::move(newUpload);
     width = newWidth;
     height = newHeight;
     mode = newMode;
 }
 
 void GpuColorTransfer::Upload(std::uint64_t address, std::uint32_t newWidth, std::uint32_t newHeight, ColorTileMode newMode) {
-    PerformanceTimer timing("ColorTransfer.Upload");
     prepare(newWidth, newHeight, newMode);
-    timing.Mark("prepare");
     const ColorTargetLayout layout(width, height, mode);
-    GuestMemory::Read(address, upload->Bytes(), layout.Alignment());
-    timing.Mark("guest_read", layout.Bytes());
+    GuestMemory::Read(address, tiled->Bytes(), layout.Alignment());
 }
 
-void GpuColorTransfer::convert(VkCommandBuffer commands, bool toTiled, bool swapRedBlue) {
-    Require(commands != VK_NULL_HANDLE && tiled && linear, "color transfer is not prepared");
+void GpuColorTransfer::convert(VkCommandBuffer commands, bool toTiled, bool swapRedBlue, bool tenBit) {
+    Require(commands != VK_NULL_HANDLE && tiled && tiledDevice && linear, "color transfer is not prepared");
+    // The guest bytes move to video memory with one DMA copy; the shader's scattered accesses stay local.
+    RecordMemoryBarrier(context, commands, VK_PIPELINE_STAGE_HOST_BIT | VK_PIPELINE_STAGE_TRANSFER_BIT | VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_ACCESS_HOST_WRITE_BIT | VK_ACCESS_TRANSFER_WRITE_BIT | VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_SHADER_READ_BIT | VK_ACCESS_SHADER_WRITE_BIT, VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_TRANSFER_WRITE_BIT);
+    CopyBuffer(context, commands, tiled->Handle(), 0, tiledDevice->Handle(), 0, tiledDevice->Size());
     VkMemoryBarrier before{VK_STRUCTURE_TYPE_MEMORY_BARRIER};
-    before.srcAccessMask = VK_ACCESS_HOST_WRITE_BIT | VK_ACCESS_TRANSFER_WRITE_BIT | VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_SHADER_READ_BIT | VK_ACCESS_SHADER_WRITE_BIT;
+    before.srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT | VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_SHADER_READ_BIT | VK_ACCESS_SHADER_WRITE_BIT;
     before.dstAccessMask = VK_ACCESS_SHADER_READ_BIT | VK_ACCESS_SHADER_WRITE_BIT;
     const auto barrier = context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier");
-    barrier(commands, VK_PIPELINE_STAGE_HOST_BIT | VK_PIPELINE_STAGE_TRANSFER_BIT | VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, 0, 1, &before, 0, nullptr, 0, nullptr);
-    const std::array<std::uint32_t, 4> push{width, height, (width + 127u) / 128u, (toTiled ? 1u : 0u) | (swapRedBlue ? 2u : 0u) | (mode == ColorTileMode::RenderTarget ? 4u : 0u)};
+    barrier(commands, VK_PIPELINE_STAGE_TRANSFER_BIT | VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, 0, 1, &before, 0, nullptr, 0, nullptr);
+    const std::array<std::uint32_t, 4> push{width, height, (width + 127u) / 128u, (toTiled ? 1u : 0u) | (swapRedBlue ? 2u : 0u) | (mode == ColorTileMode::RenderTarget ? 4u : 0u) | (tenBit ? 8u : 0u)};
     context.Function<PFN_vkCmdBindPipeline>("vkCmdBindPipeline")(commands, VK_PIPELINE_BIND_POINT_COMPUTE, pipeline);
     context.Function<PFN_vkCmdBindDescriptorSets>("vkCmdBindDescriptorSets")(commands, VK_PIPELINE_BIND_POINT_COMPUTE, pipelineLayout, 0, 1, &descriptorSet, 0, nullptr);
     context.Function<PFN_vkCmdPushConstants>("vkCmdPushConstants")(commands, pipelineLayout, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof(push), push.data());
@@ -121,32 +118,24 @@ void GpuColorTransfer::convert(VkCommandBuffer commands, bool toTiled, bool swap
     after.srcAccessMask = VK_ACCESS_SHADER_WRITE_BIT;
     after.dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT;
     barrier(commands, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 1, &after, 0, nullptr, 0, nullptr);
+    if (toTiled) {
+        CopyBuffer(context, commands, tiledDevice->Handle(), 0, tiled->Handle(), 0, tiledDevice->Size());
+        RecordMemoryBarrier(context, commands, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_HOST_BIT, VK_ACCESS_TRANSFER_WRITE_BIT, VK_ACCESS_HOST_READ_BIT);
+    }
 }
 
-void GpuColorTransfer::Detile(VkCommandBuffer commands, bool swapRedBlue) {
-    Require(commands != VK_NULL_HANDLE && upload && tiled, "color upload is not prepared");
-    VkMemoryBarrier before{VK_STRUCTURE_TYPE_MEMORY_BARRIER};
-    before.srcAccessMask = VK_ACCESS_HOST_WRITE_BIT | VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT;
-    before.dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_TRANSFER_WRITE_BIT;
-    context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier")(commands, VK_PIPELINE_STAGE_HOST_BIT | VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 1, &before, 0, nullptr, 0, nullptr);
-    const VkBufferCopy copy{0, 0, upload->Bytes().size()};
-    context.Function<PFN_vkCmdCopyBuffer>("vkCmdCopyBuffer")(commands, upload->Handle(), tiled->Handle(), 1, &copy);
-    convert(commands, false, swapRedBlue);
+void GpuColorTransfer::Detile(VkCommandBuffer commands, bool swapRedBlue, bool tenBit) {
+    convert(commands, false, swapRedBlue, tenBit);
 }
 
 void GpuColorTransfer::Tile(VkCommandBuffer commands) {
     convert(commands, true, false);
-    VkMemoryBarrier before{VK_STRUCTURE_TYPE_MEMORY_BARRIER};
-    before.srcAccessMask = VK_ACCESS_SHADER_WRITE_BIT;
-    before.dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT;
-    const auto barrier = context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier");
-    barrier(commands, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 1, &before, 0, nullptr, 0, nullptr);
-    const VkBufferCopy copy{0, 0, readback->Bytes().size()};
-    context.Function<PFN_vkCmdCopyBuffer>("vkCmdCopyBuffer")(commands, tiled->Handle(), readback->Handle(), 1, &copy);
-    VkMemoryBarrier after{VK_STRUCTURE_TYPE_MEMORY_BARRIER};
-    after.srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT;
-    after.dstAccessMask = VK_ACCESS_HOST_READ_BIT;
-    barrier(commands, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_HOST_BIT, 0, 1, &after, 0, nullptr, 0, nullptr);
+}
+
+void GpuColorTransfer::WriteBack(std::uint64_t address) {
+    Require(tiled != nullptr, "color transfer is not prepared for writeback");
+    const ColorTargetLayout layout(width, height, mode);
+    GuestMemory::Write(address, tiled->Bytes(), layout.Alignment());
 }
 
 VkBuffer GpuColorTransfer::LinearBuffer() const {
@@ -154,13 +143,6 @@ VkBuffer GpuColorTransfer::LinearBuffer() const {
     return linear->Handle();
 }
 
-bool GpuColorTransfer::MatchesGuest(std::uint64_t address) {
-    Require(readback != nullptr, "color readback is unavailable");
-    const auto bytes = readback->Bytes();
-    GuestMemory::CheckRange(reinterpret_cast<const void*>(address), bytes.size(), 1);
-    return std::memcmp(reinterpret_cast<const void*>(address), bytes.data(), bytes.size()) == 0;
-}
-
 RenderTarget& GpuColorTransfer::Target(const ColorTarget& color, bool blending) {
     if (!target || targetExtent.width != color.extent.width || targetExtent.height != color.extent.height || targetFormat != color.format || targetBlending != blending) {
         auto replacement = std::make_unique<RenderTarget>(context, color, blending);

core/libs/prx/libSceAgcDriver/Graphics/src/GraphicsPipelineCache.cpp

@@ -1,138 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/GraphicsPipelineCache.hpp"
-#include "prx/libSceAgcDriver/Graphics/include/VertexInput.hpp"
-#include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
-#include <type_traits>
-#include <algorithm>
-#include <iterator>
-
-namespace AgcDriver::Graphics {
-namespace {
-
-template<typename TValue>
-void append(std::string& key, const TValue& value) {
-    static_assert(std::is_trivially_copyable_v<TValue>);
-    key.append(reinterpret_cast<const char*>(&value), sizeof(value));
-}
-
-std::string makeKey(const Context& context, const State& state, const std::shared_ptr<ResidentColor>& target, const ShaderResources& resources, std::span<const CompiledShader> shaders) {
-    PerformanceTimer timing("Graphics.PipelineKey");
-    std::string key;
-    key.reserve(512);
-    append(key, target ? target->Target().View() : VK_NULL_HANDLE);
-    append(key, state.hasColorTarget);
-    append(key, state.rectList);
-    append(key, state.renderExtent.width);
-    append(key, state.renderExtent.height);
-    append(key, state.topology);
-    append(key, state.viewport.x);
-    append(key, state.viewport.y);
-    append(key, state.viewport.width);
-    append(key, state.viewport.height);
-    append(key, state.viewport.minDepth);
-    append(key, state.viewport.maxDepth);
-    append(key, state.negativeOneToOne);
-    append(key, state.scissor.offset.x);
-    append(key, state.scissor.offset.y);
-    append(key, state.scissor.extent.width);
-    append(key, state.scissor.extent.height);
-    append(key, state.cullMode);
-    append(key, state.frontFace);
-    append(key, state.blend.blendEnable);
-    append(key, state.blend.srcColorBlendFactor);
-    append(key, state.blend.dstColorBlendFactor);
-    append(key, state.blend.colorBlendOp);
-    append(key, state.blend.srcAlphaBlendFactor);
-    append(key, state.blend.dstAlphaBlendFactor);
-    append(key, state.blend.alphaBlendOp);
-    append(key, state.blend.colorWriteMask);
-    for (const auto value : state.blendConstants) append(key, value);
-    append(key, state.stages.mesh.has_value());
-    append(key, state.stages.tessellation.has_value());
-    if (state.stages.mesh) {
-        const auto& mesh = *state.stages.mesh;
-        append(key, mesh.inputPrimitive);
-        append(key, mesh.primitivesPerGroup);
-        append(key, mesh.verticesPerGroup);
-        append(key, mesh.maxVertices);
-        append(key, mesh.maxPrimitives);
-        append(key, mesh.threadsPerGroup);
-        append(key, mesh.ldsSizeDwords);
-        append(key, mesh.provokingVertex);
-    }
-    if (state.stages.tessellation) {
-        const auto& tessellation = *state.stages.tessellation;
-        append(key, tessellation.inputControlPoints);
-        append(key, tessellation.outputControlPoints);
-        append(key, tessellation.domain);
-        append(key, tessellation.partitioning);
-        append(key, tessellation.outputTopology);
-    }
-    timing.Mark("state");
-    const auto input = BuildVertexInputLayout(context, shaders.front().program->vertexAttributes);
-    timing.Mark("vertex_layout");
-    append(key, input.bindings.size());
-    for (const auto& binding : input.bindings) {
-        append(key, binding.binding);
-        append(key, binding.stride);
-        append(key, binding.inputRate);
-    }
-    append(key, input.attributes.size());
-    for (const auto& attribute : input.attributes) {
-        append(key, attribute.location);
-        append(key, attribute.binding);
-        append(key, attribute.format);
-        append(key, attribute.offset);
-    }
-    append(key, resources.LayoutKey().size());
-    const auto layout = std::span(resources.LayoutKey());
-    if (!layout.empty()) key.append(reinterpret_cast<const char*>(layout.data()), layout.size_bytes());
-    append(key, PushConstantStages(shaders));
-    append(key, shaders.size());
-    timing.Mark("resources");
-    std::uint64_t shaderBytes = 0;
-    for (const auto& shader : shaders) {
-        append(key, shader.stage);
-        append(key, shader.program->spirv.size());
-        const auto bytes = std::as_bytes(std::span(shader.program->spirv));
-        if (!bytes.empty()) key.append(reinterpret_cast<const char*>(bytes.data()), bytes.size());
-        shaderBytes += bytes.size();
-    }
-    timing.Mark("shader_code", shaderBytes);
-    return key;
-}
-
-}
-
-std::shared_ptr<Pipeline> GraphicsPipelineCache::Get(const State& state, const std::shared_ptr<ResidentColor>& target, const ShaderResources& resources, std::span<const CompiledShader> shaders) {
-    PerformanceTimer timing("Graphics.PipelineCache");
-    auto key = makeKey(context, state, target, resources, shaders);
-    timing.Mark("key");
-    const auto found = lookup.find(key);
-    if (found != lookup.end()) {
-        const auto it = found->second;
-        auto pipeline = it->pipeline;
-        entries.splice(entries.end(), entries, it);
-        timing.Mark("hit");
-        return pipeline;
-    }
-    timing.Mark("miss");
-    auto pipeline = std::make_shared<Pipeline>(context, state, target ? &target->Target() : nullptr, resources, shaders);
-    timing.Mark("create");
-    entries.push_back({std::move(key), target, pipeline});
-    try {
-        const auto it = std::prev(entries.end());
-        Require(lookup.emplace(it->key, it).second, "duplicate graphics pipeline cache key");
-    } catch (...) {
-        entries.pop_back();
-        throw;
-    }
-    while (entries.size() > 128) {
-        const auto it = std::find_if(entries.begin(), entries.end(), [](const auto& entry) { return entry.pipeline.use_count() == 1; });
-        if (it == entries.end()) break;
-        lookup.erase(it->key);
-        entries.erase(it);
-    }
-    return pipeline;
-}
-
-}

core/libs/prx/libSceAgcDriver/Graphics/src/GuestSamplerResource.cpp

@@ -75,14 +75,17 @@ GuestSamplerResource DecodeSamplerResource(std::span<const std::uint32_t> words)
     const auto borderColorType = (words[3] >> 30u) & 0x3u;
 
     Require(!forceUnormCoords, "guest sampler descriptor uses unnormalized coordinates which are not implemented");
-    Require(anisoThreshold == 0, "guest sampler descriptor uses an anisotropy threshold override which is not implemented");
     Require(!forceSrgb, "guest sampler descriptor forces sRGB decoding which is not implemented");
-    Require(anisoBias == 0, "guest sampler descriptor uses an anisotropy bias which is not implemented");
-    Require(!truncCoord, "guest sampler descriptor uses coordinate truncation which is not implemented");
+    // TRUNC_COORD picks point-sampled texels by truncation instead of rounding, and the perf fields
+    // trade mip/depth precision for speed; Vulkan's nearest filtering already floors, so these only
+    // move texel selection by half a texel at most and are accepted as is. ANISO_THRESHOLD and
+    // ANISO_BIAS only tune how many anisotropic taps the hardware takes; the host filter picks its own.
+    static_cast<void>(truncCoord);
+    static_cast<void>(anisoThreshold);
+    static_cast<void>(anisoBias);
     Require(!disableCubeWrap, "guest sampler descriptor disables seamless cube filtering which is not implemented");
     Require(filterMode == 0, "guest sampler descriptor uses a reduction filter mode which is not implemented");
     Require(!disableDegamma, "guest sampler descriptor disables degamma which is not implemented");
-    Require(perfMip == 0 && perfZ == 0, "guest sampler descriptor uses performance counters which are not implemented");
     Require(lodBiasSec == 0, "guest sampler descriptor uses a secondary LOD bias which is not implemented");
     Require(!pointPreclamp, "guest sampler descriptor uses point preclamping which is not implemented");
     Require(!anisoOverride, "guest sampler descriptor uses an anisotropy override which is not implemented");

core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp

@@ -1,5 +1,7 @@
+#include <cstdio>
 #include "prx/libSceAgcDriver/Graphics/include/GuestTextureResource.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/Context.hpp"
+#include <algorithm>
 #include <stdexcept>
 #include <string>
 
@@ -17,7 +19,10 @@ TextureTileMode resolveTileMode(std::uint32_t raw) {
         case 0x01: return TextureTileMode::kStandard256B;
         case 0x05: return TextureTileMode::kStandard4KB;
         case 0x09: return TextureTileMode::kStandard64KB;
-        case 0x1b: return TextureTileMode::RenderTarget64KB;
+        case 0x18: return TextureTileMode::kZ64KBX;
+        case 0x19: return TextureTileMode::kS64KBX;
+        case 0x1a: return TextureTileMode::kD64KBX;
+        case 0x1b: return TextureTileMode::kR64KBX;
         default: throw std::runtime_error("AGC graphics: guest texture descriptor uses an unsupported tile mode " + std::to_string(raw));
     }
 }
@@ -26,6 +31,7 @@ TextureDimension resolveDimension(std::uint32_t raw) {
     switch (raw) {
         case 8: return TextureDimension::k1D;
         case 9: return TextureDimension::k2D;
+        case 10: return TextureDimension::k3D;
         case 11: return TextureDimension::kCube;
         case 13: return TextureDimension::k2DArray;
         default: throw std::runtime_error("AGC graphics: guest texture descriptor uses an unsupported image type " + std::to_string(raw));
@@ -51,7 +57,7 @@ GuestTextureResource DecodeTextureResource(std::span<const std::uint32_t> words)
     const auto dstSelZ = (words[3] >> 6u) & 0x7u;
     const auto dstSelW = (words[3] >> 9u) & 0x7u;
     const auto baseLevel = (words[3] >> 12u) & 0xfu;
-    const auto lastLevel = (words[3] >> 16u) & 0xfu;
+    auto lastLevel = (words[3] >> 16u) & 0xfu;
     const auto tileModeRaw = (words[3] >> 20u) & 0x1fu;
     const auto bcSwizzle = (words[3] >> 25u) & 0x7u;
     const auto typeRaw = (words[3] >> 28u) & 0xfu;
@@ -83,21 +89,54 @@ GuestTextureResource DecodeTextureResource(std::span<const std::uint32_t> words)
     requireValidDstSel(dstSelW);
 
     Require(minLod == 0, "guest texture descriptor uses a nonzero minimum LOD clamp which is not implemented");
-    Require(minLodWarn == 0, "guest texture descriptor uses a minimum LOD warning threshold which is not implemented");
-    Require(mipStatsCntId == 0 && !mipStatsCntEn, "guest texture descriptor uses mip statistics counters which are not implemented");
+    // The LOD warning threshold and mip statistics counters are texture-streaming feedback: the GPU
+    // reports which mips were wanted. Nothing is reported back here, which only affects what the title
+    // chooses to stream, not what this access returns.
+    if (minLodWarn != 0 || mipStatsCntId != 0 || mipStatsCntEn) {
+        static bool reported = false;
+        if (!reported) {
+            reported = true;
+            std::fprintf(stderr, "[gpu] texture streaming feedback (LOD warning / mip statistics) is not reported\n");
+        }
+    }
     Require(!cornerSample, "guest texture descriptor uses corner sampling which is not implemented");
     Require(!prtDefColor, "guest texture descriptor uses a partially resident default color which is not implemented");
     Require(arrayPitch == 0, "guest texture descriptor uses a nonzero array pitch which is not implemented");
     Require(!msaaDepth, "guest texture descriptor uses MSAA which is not implemented");
-    Require(maxUncompBlkSize == 0 && maxCompBlkSize == 0, "guest texture descriptor uses DCC block size overrides which are not implemented");
-    Require(!metaPipeAligned && !writeCompress && !metaCompress && !dccAlphaPos && !dccColorTransf && metaAddr == 0, "guest texture descriptor uses metadata compression which is not implemented");
+    // DCC block sizes only apply with metadata compression, which is rejected above.
+    static_cast<void>(maxUncompBlkSize);
+    static_cast<void>(maxCompBlkSize);
+    // Surfaces are always written uncompressed here (render targets and storage images bypass DCC), so
+    // only the fast-clear keys in DCC metadata change what a read returns (see DccMetadata.hpp).
+    static_cast<void>(metaPipeAligned);
+    static_cast<void>(writeCompress);
+    if (dccColorTransf) {
+        static bool reported = false;
+        if (!reported) {
+            reported = true;
+            std::fprintf(stderr, "[gpu] texture DCC color transform is ignored (word6=0x%08x word7=0x%08x)\n", words[6], words[7]);
+        }
+    }
     Require(bcSwizzle == 0, "guest texture descriptor uses a BC swizzle which is not implemented");
 
     Require(baseLevel <= lastLevel, "guest texture descriptor has a base mip level past its last mip level");
-    Require(lastLevel == maxMip, "guest texture descriptor must expose every mip level down to the last one");
 
-    const auto tileMode = resolveTileMode(tileModeRaw);
-    const auto dimension = resolveDimension(typeRaw);
+
+    const auto describe = [&] { return " (format " + std::to_string(format) + ", " + std::to_string(width) + "x" + std::to_string(height) + "x" + std::to_string(depth + 1) + ", type " + std::to_string(typeRaw) + ", tile " + std::to_string(tileModeRaw) + ", mips " + std::to_string(baseLevel) + ".." + std::to_string(lastLevel) + " of " + std::to_string(maxMip + 1) + ")"; };
+    TextureTileMode tileMode;
+    TextureDimension dimension;
+    try {
+        tileMode = resolveTileMode(tileModeRaw);
+        dimension = resolveDimension(typeRaw);
+    } catch (const std::exception& error) {
+        throw std::runtime_error(std::string(error.what()) + describe());
+    }
+    Require(baseLevel <= maxMip, "guest texture descriptor starts past the surface's last mip level" + describe());
+    // Views may name levels past MAX_MIP (a 512x512 view through 1x1 over a 9-level surface); the
+    // hardware never addresses them, so the view ends at the surface's last level.
+    lastLevel = std::min(lastLevel, maxMip);
+    // XOR swizzles fold a pipe/bank XOR into the low address bits; only unmodified 64 KiB bases are modeled.
+    Require(XorSwizzleMode(tileMode) == 0 || (baseAddress & 0xffffu) == 0, "guest texture descriptor combines an XOR swizzle with a pipe/bank XOR base which is not implemented");
 
     switch (dimension) {
         case TextureDimension::k1D:
@@ -106,6 +145,10 @@ GuestTextureResource DecodeTextureResource(std::span<const std::uint32_t> words)
         case TextureDimension::k2D:
             Require(depth == 0 && baseArray == 0, "guest 2D texture descriptor has a nonzero depth or base array");
             break;
+        case TextureDimension::k3D:
+            // word4 DEPTH holds the depth minus one; BASE_ARRAY has no meaning for volumes.
+            Require(baseArray == 0, "guest 3D texture descriptor has a nonzero base array");
+            break;
         case TextureDimension::k2DArray:
             Require(baseArray <= depth, "guest 2D array texture descriptor has a base array past its last array slice");
             break;
@@ -124,6 +167,7 @@ GuestTextureResource DecodeTextureResource(std::span<const std::uint32_t> words)
     result.baseArray = baseArray;
     result.mipCount = maxMip + 1u;
     result.baseLevel = baseLevel;
+    result.lastLevel = lastLevel;
     result.tileMode = tileMode;
     result.dimension = dimension;
     result.format = format;
@@ -131,6 +175,8 @@ GuestTextureResource DecodeTextureResource(std::span<const std::uint32_t> words)
     result.dstSelY = static_cast<std::uint8_t>(dstSelY);
     result.dstSelZ = static_cast<std::uint8_t>(dstSelZ);
     result.dstSelW = static_cast<std::uint8_t>(dstSelW);
+    result.dccAddress = metaCompress ? metaAddr << 8u : 0u;
+    result.dccAlphaOnMsb = dccAlphaPos;
     return result;
 }
 
@@ -138,9 +184,9 @@ bool MatchesGuestDimension(ShaderRecompiler::DescriptorImageShape shape, Texture
     switch (shape) {
         case ShaderRecompiler::DescriptorImageShape::Image1D: return dimension == TextureDimension::k1D;
         case ShaderRecompiler::DescriptorImageShape::Image2D: return dimension == TextureDimension::k2D;
-        case ShaderRecompiler::DescriptorImageShape::Image2DArray: return dimension == TextureDimension::k2DArray;
+        case ShaderRecompiler::DescriptorImageShape::Image2DArray: return dimension == TextureDimension::k2DArray || dimension == TextureDimension::kCube;
         case ShaderRecompiler::DescriptorImageShape::ImageCube: return dimension == TextureDimension::kCube;
-        case ShaderRecompiler::DescriptorImageShape::Image3D: return false;
+        case ShaderRecompiler::DescriptorImageShape::Image3D: return dimension == TextureDimension::k3D;
     }
     throw std::runtime_error("AGC graphics: MatchesGuestDimension encountered an unknown descriptor image shape");
 }

core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp

@@ -1,16 +1,53 @@
 #include "prx/libSceAgcDriver/Graphics/include/Pipeline.hpp"
 #include "prx/libSceAgcDriver/Graphics/include/VertexInput.hpp"
 #include <algorithm>
+#include <array>
+#include <chrono>
 #include <cmath>
+#include <cstdio>
+#include <cstdlib>
+#include <list>
+#include <mutex>
+#include <type_traits>
+#include <unordered_map>
 #include <vector>
 
 namespace AgcDriver::Graphics {
 
-Pipeline::Pipeline(const Context& context, const State& state, const RenderTarget* target, const ShaderResources& resources, std::span<const CompiledShader> shaders) : context(context), _modules(shaders.size()) {
-    Require((target != nullptr) == state.hasColorTarget, "render target does not match decoded color state");
-    Require(state.renderExtent.width != 0 && state.renderExtent.height != 0 && state.renderExtent.width <= context.limits.maxFramebufferWidth && state.renderExtent.height <= context.limits.maxFramebufferHeight, "framebuffer extent exceeds device limits");
+Framebuffer::Framebuffer(const Context& context, VkRenderPass renderPass, std::span<const VkImageView> targets, VkExtent2D extent) : context(context) {
+    // Cached objects outlive their device's teardown; they must not keep its buffer pool alive past it.
+    this->context.bufferPool.reset();
+    Require(extent.width != 0 && extent.height != 0 && extent.width <= context.limits.maxFramebufferWidth && extent.height <= context.limits.maxFramebufferHeight, "framebuffer extent exceeds device limits");
+    VkFramebufferCreateInfo framebufferInfo{VK_STRUCTURE_TYPE_FRAMEBUFFER_CREATE_INFO};
+    framebufferInfo.renderPass = renderPass;
+    framebufferInfo.attachmentCount = static_cast<std::uint32_t>(targets.size());
+    framebufferInfo.pAttachments = targets.empty() ? nullptr : targets.data();
+    framebufferInfo.width = extent.width;
+    framebufferInfo.height = extent.height;
+    framebufferInfo.layers = 1;
+    Check(context.Function<PFN_vkCreateFramebuffer>("vkCreateFramebuffer")(context.device, &framebufferInfo, nullptr, &framebuffer), "vkCreateFramebuffer");
+}
+
+Framebuffer::~Framebuffer() {
+    if (framebuffer) context.Function<PFN_vkDestroyFramebuffer>("vkDestroyFramebuffer")(context.device, framebuffer, nullptr);
+}
+
+void ValidateViewport(const Context& context, const VkViewport& viewport) {
+    Require(std::isfinite(viewport.minDepth) && std::isfinite(viewport.maxDepth), "non-finite viewport depth range");
+    Require(context.depthRangeUnrestricted || (viewport.minDepth >= 0 && viewport.minDepth <= 1 && viewport.maxDepth >= 0 && viewport.maxDepth <= 1), "viewport depth outside [0, 1] requires VK_EXT_depth_range_unrestricted");
+    Require(std::isfinite(viewport.x) && std::isfinite(viewport.y) && std::isfinite(viewport.width) && std::isfinite(viewport.height), "viewport arithmetic overflow");
+    Require(viewport.width <= context.limits.maxViewportDimensions[0] && std::abs(viewport.height) <= context.limits.maxViewportDimensions[1], "viewport dimensions exceed device limits");
+    Require(viewport.x >= context.limits.viewportBoundsRange[0] && viewport.x + viewport.width <= context.limits.viewportBoundsRange[1], "viewport X exceeds device bounds");
+    Require(std::min(viewport.y, viewport.y + viewport.height) >= context.limits.viewportBoundsRange[0] && std::max(viewport.y, viewport.y + viewport.height) <= context.limits.viewportBoundsRange[1], "viewport Y exceeds device bounds");
+}
+
+Pipeline::Pipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const ShaderResources& resources, std::span<const CompiledShader> shaders, VkImageLayout attachmentLayout) : context(context), _modules(shaders.size()), attachments(state.colors.size()) {
+    // A cached pipeline may outlive its device's teardown (see ClearCachedPipelines); it must not keep
+    // the buffer pool, which is reset with the device, alive past it.
+    this->context.bufferPool.reset();
+    Require(state.blends.size() == state.colors.size(), "blend states do not match decoded color state");
+    Require(state.colors.size() <= context.limits.maxColorAttachments, "color targets exceed device attachment limits");
     Require(state.hasColorTarget || (context.limits.framebufferNoAttachmentsSampleCounts & VK_SAMPLE_COUNT_1_BIT) != 0, "device does not support single-sample rendering without attachments");
-    ValidateShaders(shaders, state, context.subgroup, context.fragmentShaderBarycentric);
     Require(!state.negativeOneToOne || context.depthClipControl, "negative-one-to-one depth clipping requires VK_EXT_depth_clip_control with depthClipControl enabled");
     if (state.rectList) Require(context.tessellationShader && context.limits.maxTessellationPatchSize >= 4, "rect-list requires tessellation with four output control points");
     if (state.stages.tessellation) {
@@ -23,13 +60,6 @@ Pipeline::Pipeline(const Context& context, const State& state, const RenderTarge
         Require(mesh.threadsPerGroup <= context.meshLimits.maxMeshWorkGroupInvocations && mesh.threadsPerGroup <= context.meshLimits.maxMeshWorkGroupSize[0], "mesh workgroup exceeds device limits");
         Require(mesh.maxVertices <= context.meshLimits.maxMeshOutputVertices && mesh.maxPrimitives <= context.meshLimits.maxMeshOutputPrimitives && static_cast<std::uint64_t>(mesh.ldsSizeDwords) * 4 <= context.meshLimits.maxMeshSharedMemorySize, "mesh output or LDS exceeds device limits");
     }
-    const auto& viewport = state.viewport;
-    Require(std::isfinite(viewport.minDepth) && std::isfinite(viewport.maxDepth), "non-finite viewport depth range");
-    Require(context.depthRangeUnrestricted || (viewport.minDepth >= 0 && viewport.minDepth <= 1 && viewport.maxDepth >= 0 && viewport.maxDepth <= 1), "viewport depth outside [0, 1] requires VK_EXT_depth_range_unrestricted");
-    Require(std::isfinite(viewport.x) && std::isfinite(viewport.y) && std::isfinite(viewport.width) && std::isfinite(viewport.height), "viewport arithmetic overflow");
-    Require(viewport.width <= context.limits.maxViewportDimensions[0] && std::abs(viewport.height) <= context.limits.maxViewportDimensions[1], "viewport dimensions exceed device limits");
-    Require(viewport.x >= context.limits.viewportBoundsRange[0] && viewport.x + viewport.width <= context.limits.viewportBoundsRange[1], "viewport X exceeds device bounds");
-    Require(std::min(viewport.y, viewport.y + viewport.height) >= context.limits.viewportBoundsRange[0] && std::max(viewport.y, viewport.y + viewport.height) <= context.limits.viewportBoundsRange[1], "viewport Y exceeds device bounds");
     const auto pushStages = PushConstantStages(shaders);
     Require(pushStages == 0 || context.limits.maxPushConstantsSize >= PipelinePushConstantBytes, "graphics push constant range exceeds device limit");
     try {
@@ -46,6 +76,8 @@ Pipeline::Pipeline(const Context& context, const State& state, const RenderTarge
             stages[i].module = _modules[i];
             stages[i].pName = "main";
         }
+        // A descriptor set layout with the same bindings as this one is compatible with the pipeline
+        // layout, so later draws bind their own ShaderResources' set under it.
         const auto setLayout = resources.Layout();
         const VkPushConstantRange push{pushStages, 0, PipelinePushConstantBytes};
         VkPipelineLayoutCreateInfo layoutInfo{VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO};
@@ -54,61 +86,61 @@ Pipeline::Pipeline(const Context& context, const State& state, const RenderTarge
         layoutInfo.pushConstantRangeCount = pushStages != 0 ? 1 : 0;
         layoutInfo.pPushConstantRanges = pushStages != 0 ? &push : nullptr;
         Check(context.Function<PFN_vkCreatePipelineLayout>("vkCreatePipelineLayout")(context.device, &layoutInfo, nullptr, &layout), "vkCreatePipelineLayout graphics");
-        VkAttachmentDescription color{};
-        color.format = state.color.format;
-        color.samples = VK_SAMPLE_COUNT_1_BIT;
-        color.loadOp = VK_ATTACHMENT_LOAD_OP_LOAD;
-        color.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
-        color.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
-        color.stencilStoreOp = VK_ATTACHMENT_STORE_OP_DONT_CARE;
-        color.initialLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;
-        color.finalLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;
-        VkAttachmentReference reference{0, VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL};
+        std::vector<VkAttachmentDescription> colors;
+        std::vector<VkAttachmentReference> references;
+        for (std::uint32_t index = 0; index < state.colors.size(); ++index) {
+            VkAttachmentDescription color{};
+            color.format = state.colors[index].format;
+            color.samples = VK_SAMPLE_COUNT_1_BIT;
+            color.loadOp = VK_ATTACHMENT_LOAD_OP_LOAD;
+            color.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
+            color.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
+            color.stencilStoreOp = VK_ATTACHMENT_STORE_OP_DONT_CARE;
+            color.initialLayout = attachmentLayout;
+            color.finalLayout = attachmentLayout;
+            colors.push_back(color);
+            references.push_back({index, attachmentLayout});
+        }
         VkSubpassDescription subpass{};
         subpass.pipelineBindPoint = VK_PIPELINE_BIND_POINT_GRAPHICS;
-        subpass.colorAttachmentCount = state.hasColorTarget ? 1 : 0;
-        subpass.pColorAttachments = state.hasColorTarget ? &reference : nullptr;
+        subpass.colorAttachmentCount = static_cast<std::uint32_t>(references.size());
+        subpass.pColorAttachments = references.empty() ? nullptr : references.data();
         VkRenderPassCreateInfo passInfo{VK_STRUCTURE_TYPE_RENDER_PASS_CREATE_INFO};
-        passInfo.attachmentCount = state.hasColorTarget ? 1 : 0;
-        passInfo.pAttachments = state.hasColorTarget ? &color : nullptr;
+        passInfo.attachmentCount = static_cast<std::uint32_t>(colors.size());
+        passInfo.pAttachments = colors.empty() ? nullptr : colors.data();
         passInfo.subpassCount = 1;
         passInfo.pSubpasses = &subpass;
         Check(context.Function<PFN_vkCreateRenderPass>("vkCreateRenderPass")(context.device, &passInfo, nullptr, &renderPass), "vkCreateRenderPass");
-        const auto view = target ? target->View() : VK_NULL_HANDLE;
-        VkFramebufferCreateInfo framebufferInfo{VK_STRUCTURE_TYPE_FRAMEBUFFER_CREATE_INFO};
-        framebufferInfo.renderPass = renderPass;
-        framebufferInfo.attachmentCount = state.hasColorTarget ? 1 : 0;
-        framebufferInfo.pAttachments = state.hasColorTarget ? &view : nullptr;
-        framebufferInfo.width = state.renderExtent.width;
-        framebufferInfo.height = state.renderExtent.height;
-        framebufferInfo.layers = 1;
-        Check(context.Function<PFN_vkCreateFramebuffer>("vkCreateFramebuffer")(context.device, &framebufferInfo, nullptr, &framebuffer), "vkCreateFramebuffer");
         VkPipelineVertexInputStateCreateInfo input{VK_STRUCTURE_TYPE_PIPELINE_VERTEX_INPUT_STATE_CREATE_INFO};
-        const auto vertexInput = BuildVertexInputLayout(context, shaders.front().program->vertexAttributes);
         input.vertexBindingDescriptionCount = static_cast<std::uint32_t>(vertexInput.bindings.size());
         input.pVertexBindingDescriptions = vertexInput.bindings.data();
         input.vertexAttributeDescriptionCount = static_cast<std::uint32_t>(vertexInput.attributes.size());
         input.pVertexAttributeDescriptions = vertexInput.attributes.data();
         VkPipelineInputAssemblyStateCreateInfo assembly{VK_STRUCTURE_TYPE_PIPELINE_INPUT_ASSEMBLY_STATE_CREATE_INFO};
         assembly.topology = state.topology;
+        // Viewport and scissor are set per draw (Begin), so they do not multiply pipelines; the depth
+        // clip control stays baked in.
         VkPipelineViewportStateCreateInfo viewports{VK_STRUCTURE_TYPE_PIPELINE_VIEWPORT_STATE_CREATE_INFO};
         VkPipelineViewportDepthClipControlCreateInfoEXT depthClip{VK_STRUCTURE_TYPE_PIPELINE_VIEWPORT_DEPTH_CLIP_CONTROL_CREATE_INFO_EXT};
         depthClip.negativeOneToOne = state.negativeOneToOne;
         if (state.negativeOneToOne) viewports.pNext = &depthClip;
         viewports.viewportCount = 1;
-        viewports.pViewports = &state.viewport;
         viewports.scissorCount = 1;
-        viewports.pScissors = &state.scissor;
+        const std::array<VkDynamicState, 2> dynamicStates{VK_DYNAMIC_STATE_VIEWPORT, VK_DYNAMIC_STATE_SCISSOR};
+        VkPipelineDynamicStateCreateInfo dynamic{VK_STRUCTURE_TYPE_PIPELINE_DYNAMIC_STATE_CREATE_INFO};
+        dynamic.dynamicStateCount = static_cast<std::uint32_t>(dynamicStates.size());
+        dynamic.pDynamicStates = dynamicStates.data();
         VkPipelineRasterizationStateCreateInfo raster{VK_STRUCTURE_TYPE_PIPELINE_RASTERIZATION_STATE_CREATE_INFO};
         raster.polygonMode = VK_POLYGON_MODE_FILL;
+        raster.depthClampEnable = state.depthClamp && context.depthClamp ? VK_TRUE : VK_FALSE;
         raster.cullMode = state.cullMode;
         raster.frontFace = state.frontFace;
         raster.lineWidth = 1;
         VkPipelineMultisampleStateCreateInfo samples{VK_STRUCTURE_TYPE_PIPELINE_MULTISAMPLE_STATE_CREATE_INFO};
         samples.rasterizationSamples = VK_SAMPLE_COUNT_1_BIT;
         VkPipelineColorBlendStateCreateInfo blend{VK_STRUCTURE_TYPE_PIPELINE_COLOR_BLEND_STATE_CREATE_INFO};
-        blend.attachmentCount = state.hasColorTarget ? 1 : 0;
-        blend.pAttachments = state.hasColorTarget ? &state.blend : nullptr;
+        blend.attachmentCount = static_cast<std::uint32_t>(state.blends.size());
+        blend.pAttachments = state.blends.empty() ? nullptr : state.blends.data();
         std::copy(state.blendConstants.begin(), state.blendConstants.end(), blend.blendConstants);
         VkGraphicsPipelineCreateInfo pipelineInfo{VK_STRUCTURE_TYPE_GRAPHICS_PIPELINE_CREATE_INFO};
         pipelineInfo.stageCount = static_cast<std::uint32_t>(stages.size());
@@ -124,6 +156,7 @@ Pipeline::Pipeline(const Context& context, const State& state, const RenderTarge
         pipelineInfo.pRasterizationState = &raster;
         pipelineInfo.pMultisampleState = &samples;
         pipelineInfo.pColorBlendState = &blend;
+        pipelineInfo.pDynamicState = &dynamic;
         pipelineInfo.layout = layout;
         pipelineInfo.renderPass = renderPass;
         Check(context.Function<PFN_vkCreateGraphicsPipelines>("vkCreateGraphicsPipelines")(context.device, context.pipelineCache, 1, &pipelineInfo, nullptr, &pipeline), "vkCreateGraphicsPipelines");
@@ -138,33 +171,307 @@ Pipeline::~Pipeline() {
 }
 
 void Pipeline::release() noexcept {
+    framebuffers.clear();
     if (pipeline) context.Function<PFN_vkDestroyPipeline>("vkDestroyPipeline")(context.device, pipeline, nullptr);
-    if (framebuffer) context.Function<PFN_vkDestroyFramebuffer>("vkDestroyFramebuffer")(context.device, framebuffer, nullptr);
     if (renderPass) context.Function<PFN_vkDestroyRenderPass>("vkDestroyRenderPass")(context.device, renderPass, nullptr);
     if (layout) context.Function<PFN_vkDestroyPipelineLayout>("vkDestroyPipelineLayout")(context.device, layout, nullptr);
     for (auto module : _modules) {
         if (module) context.Function<PFN_vkDestroyShaderModule>("vkDestroyShaderModule")(context.device, module, nullptr);
     }
+    pipeline = VK_NULL_HANDLE;
+    renderPass = VK_NULL_HANDLE;
+    layout = VK_NULL_HANDLE;
+    _modules.clear();
+}
+
+void Pipeline::Abandon() noexcept {
+    for (auto& entry : framebuffers) entry.framebuffer->Abandon();
+    framebuffers.clear();
+    pipeline = VK_NULL_HANDLE;
+    renderPass = VK_NULL_HANDLE;
+    layout = VK_NULL_HANDLE;
+    _modules.clear();
+}
+
+void Framebuffer::Abandon() noexcept {
+    framebuffer = VK_NULL_HANDLE;
 }
 
 VkPipelineLayout Pipeline::Layout() const {
     return layout;
 }
 
-void Pipeline::Begin(VkCommandBuffer commands, VkExtent2D extent) const {
+std::shared_ptr<Framebuffer> Pipeline::AcquireFramebuffer(std::span<const VkImageView> targets, std::span<const std::shared_ptr<StorageTexture>> owners, VkExtent2D extent) {
+    Require(targets.size() == attachments && targets.size() == owners.size(), "render targets do not match the pipeline's color attachments");
+    const bool resident = std::all_of(owners.begin(), owners.end(), [](const auto& owner) { return owner != nullptr; });
+    if (resident) {
+        // Entries whose views are gone can never match again and go as soon as no recorded draw holds
+        // them (Kept keeps its framebuffer until the batch completes).
+        std::erase_if(framebuffers, [](const CachedFramebuffer& entry) {
+            return entry.framebuffer.use_count() == 1 && std::any_of(entry.owners.begin(), entry.owners.end(), [](const auto& owner) { return owner.expired(); });
+        });
+        for (auto it = framebuffers.begin(); it != framebuffers.end(); ++it) {
+            if (it->extent.width != extent.width || it->extent.height != extent.height || !std::equal(it->views.begin(), it->views.end(), targets.begin(), targets.end())) continue;
+            // View handles are recycled once a StorageTexture is destroyed, so the owners must be the
+            // very objects the views were made for.
+            bool same = true;
+            for (std::size_t i = 0; i < owners.size() && same; ++i) same = it->owners[i].lock().get() == owners[i].get();
+            if (!same) continue;
+            std::rotate(it, std::next(it), framebuffers.end());
+            return framebuffers.back().framebuffer;
+        }
+    }
+    auto framebuffer = std::make_shared<Framebuffer>(context, renderPass, targets, extent);
+    if (!resident) return framebuffer;
+    // Beyond the bound the least recently used unreferenced entry goes.
+    constexpr std::size_t bound = 8;
+    while (framebuffers.size() >= bound) {
+        const auto victim = std::find_if(framebuffers.begin(), framebuffers.end(), [](const CachedFramebuffer& entry) { return entry.framebuffer.use_count() == 1; });
+        if (victim == framebuffers.end()) break;
+        framebuffers.erase(victim);
+    }
+    CachedFramebuffer entry;
+    entry.views.assign(targets.begin(), targets.end());
+    entry.owners.assign(owners.begin(), owners.end());
+    entry.extent = extent;
+    entry.framebuffer = framebuffer;
+    framebuffers.push_back(std::move(entry));
+    return framebuffer;
+}
+
+void Pipeline::Begin(VkCommandBuffer commands, const Framebuffer& framebuffer, VkExtent2D extent, const VkViewport& viewport, const VkRect2D& scissor) const {
     VkRenderPassBeginInfo begin{VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO};
     begin.renderPass = renderPass;
-    begin.framebuffer = framebuffer;
+    begin.framebuffer = framebuffer.Handle();
     begin.renderArea = {{0, 0}, extent};
-    context.Function<PFN_vkCmdBeginRenderPass>("vkCmdBeginRenderPass")(commands, &begin, VK_SUBPASS_CONTENTS_INLINE);
-    context.Function<PFN_vkCmdBindPipeline>("vkCmdBindPipeline")(commands, VK_PIPELINE_BIND_POINT_GRAPHICS, pipeline);
+    context.Resolved(&DeviceFunctions::cmdBeginRenderPass, "vkCmdBeginRenderPass")(commands, &begin, VK_SUBPASS_CONTENTS_INLINE);
+    Continue(commands, viewport, scissor);
+}
+
+void Pipeline::Continue(VkCommandBuffer commands, const VkViewport& viewport, const VkRect2D& scissor) const {
+    context.Resolved(&DeviceFunctions::cmdBindPipeline, "vkCmdBindPipeline")(commands, VK_PIPELINE_BIND_POINT_GRAPHICS, pipeline);
+    context.Resolved(&DeviceFunctions::cmdSetViewport, "vkCmdSetViewport")(commands, 0, 1, &viewport);
+    context.Resolved(&DeviceFunctions::cmdSetScissor, "vkCmdSetScissor")(commands, 0, 1, &scissor);
 }
 
 void Pipeline::PushConstants(VkCommandBuffer commands, std::span<const CompiledShader> shaders) const {
     const auto stages = PushConstantStages(shaders);
     if (stages == 0) return;
     const auto bytes = AssemblePushConstants(shaders);
-    context.Function<PFN_vkCmdPushConstants>("vkCmdPushConstants")(commands, layout, stages, 0, PipelinePushConstantBytes, bytes.data());
+    context.Resolved(&DeviceFunctions::cmdPushConstants, "vkCmdPushConstants")(commands, layout, stages, 0, PipelinePushConstantBytes, bytes.data());
+}
+
+void Pipeline::PushConstants(VkCommandBuffer commands, VkShaderStageFlags stages, std::span<const std::byte, PipelinePushConstantBytes> bytes) const {
+    if (stages == 0) return;
+    context.Resolved(&DeviceFunctions::cmdPushConstants, "vkCmdPushConstants")(commands, layout, stages, 0, PipelinePushConstantBytes, bytes.data());
+}
+
+namespace {
+
+template<typename TValue>
+void append(std::vector<std::byte>& key, const TValue& value) {
+    static_assert(std::is_trivially_copyable_v<TValue>);
+    const auto bytes = std::as_bytes(std::span(&value, 1));
+    key.insert(key.end(), bytes.begin(), bytes.end());
+}
+
+// Everything the Pipeline objects are built from, or empty when a stage's result has no variant id
+// (the recompiler could not identify it, so nothing else may share its pipeline). The rect-list
+// control and evaluation stages are generated from the vertex and fragment results, which the key
+// already names, so they carry no id of their own.
+std::vector<std::byte> pipelineKey(const Context& context, const State& state, const VertexInputLayout& input, const ShaderResources& resources, std::span<const CompiledShader> shaders, VkImageLayout attachmentLayout) {
+    using Stage = ShaderRecompiler::ShaderStage;
+    std::vector<std::byte> key;
+    append(key, context.device);
+    append(key, attachmentLayout);
+    append(key, shaders.size());
+    for (const auto& shader : shaders) {
+        Require(shader.program != nullptr, "missing compiled shader");
+        const bool generated = state.rectList && (shader.stage == Stage::TessellationControl || shader.stage == Stage::TessellationEvaluation);
+        if (!generated && shader.program->variantId == 0) return {};
+        append(key, shader.stage);
+        append(key, generated ? std::uint64_t{0} : shader.program->variantId);
+        // Where the stage's push constants sit in the block (AssemblePushConstants).
+        append(key, shader.pushConstantOffset);
+    }
+    append(key, PushConstantStages(shaders));
+    append(key, input.bindings.size());
+    for (const auto& binding : input.bindings) {
+        append(key, binding.binding);
+        append(key, binding.stride);
+        append(key, binding.inputRate);
+    }
+    append(key, input.attributes.size());
+    for (const auto& attribute : input.attributes) {
+        append(key, attribute.location);
+        append(key, attribute.binding);
+        append(key, attribute.format);
+        append(key, attribute.offset);
+    }
+    append(key, resources.LayoutKey().size());
+    for (const auto word : resources.LayoutKey()) append(key, word);
+    append(key, state.hasColorTarget);
+    append(key, state.rectList);
+    append(key, state.topology);
+    append(key, state.cullMode);
+    append(key, state.frontFace);
+    append(key, state.negativeOneToOne);
+    append(key, state.depthClamp && context.depthClamp);
+    append(key, state.blends.size());
+    for (const auto& blend : state.blends) append(key, blend);
+    for (const auto value : state.blendConstants) append(key, value);
+    append(key, state.colors.size());
+    for (const auto& color : state.colors) append(key, color.format);
+    append(key, state.stages.mesh.has_value());
+    if (state.stages.mesh) {
+        const auto& mesh = *state.stages.mesh;
+        append(key, mesh.inputPrimitive);
+        append(key, mesh.primitivesPerGroup);
+        append(key, mesh.verticesPerGroup);
+        append(key, mesh.maxVertices);
+        append(key, mesh.maxPrimitives);
+        append(key, mesh.threadsPerGroup);
+        append(key, mesh.ldsSizeDwords);
+        append(key, mesh.provokingVertex);
+    }
+    append(key, state.stages.tessellation.has_value());
+    if (state.stages.tessellation) {
+        const auto& tessellation = *state.stages.tessellation;
+        append(key, tessellation.inputControlPoints);
+        append(key, tessellation.outputControlPoints);
+        append(key, tessellation.domain);
+        append(key, tessellation.partitioning);
+        append(key, tessellation.outputTopology);
+    }
+    return key;
+}
+
+std::uint64_t hashKey(const std::vector<std::byte>& key) {
+    std::uint64_t hash = 14695981039346656037ull;
+    for (const auto byte : key) {
+        hash ^= static_cast<std::uint8_t>(byte);
+        hash *= 1099511628211ull;
+    }
+    return hash;
+}
+
+struct PipelineStore {
+    struct Entry {
+        VkDevice device;
+        // The device's buffer pool at insertion: it is made and reset with the device, so it tells
+        // the device instance apart from a later one the loader gave the same handle value.
+        std::weak_ptr<BufferPool> pool;
+        std::uint64_t hash;
+        std::vector<std::byte> key;
+        std::shared_ptr<Pipeline> pipeline;
+    };
+    std::mutex mutex;
+    // Least recently used first.
+    std::list<Entry> entries;
+    std::unordered_map<std::uint64_t, std::list<Entry>::iterator> index;
+    std::uint64_t hits = 0;
+    std::uint64_t misses = 0;
+    std::uint64_t uncached = 0;
+    std::uint64_t evicted = 0;
+    std::chrono::steady_clock::time_point lastReport = std::chrono::steady_clock::now();
+};
+
+// Never destroyed: the pipelines belong to a device that may already be gone when statics die, and
+// the device's teardown (ClearCachedPipelines) is the place to destroy them.
+PipelineStore& Pipelines() {
+    static auto* store = new PipelineStore();
+    return *store;
+}
+
+// Whether the entry's objects belong to the device the context names. A context without a buffer
+// pool (tests) is identified by the handle alone.
+bool alive(const PipelineStore::Entry& entry, const Context& context) {
+    if (entry.device != context.device) return false;
+    return context.bufferPool == nullptr || entry.pool.lock() == context.bufferPool;
+}
+
+// Drops an entry of a device that is gone: its objects went with the device, so they are forgotten,
+// not destroyed.
+std::list<PipelineStore::Entry>::iterator abandon(PipelineStore& store, std::list<PipelineStore::Entry>::iterator it) {
+    it->pipeline->Abandon();
+    store.index.erase(it->hash);
+    return store.entries.erase(it);
+}
+
+void reportPipelines(PipelineStore& store) {
+    static const bool profile = std::getenv("APS5_PROFILE_DRAW") != nullptr;
+    if (!profile) return;
+    const auto now = std::chrono::steady_clock::now();
+    if (now - store.lastReport < std::chrono::seconds(10)) return;
+    store.lastReport = now;
+    const auto lookups = store.hits + store.misses + store.uncached;
+    std::fprintf(stderr, "[pipecache] %llu lookups over 10 s: %llu hits (%.0f%%), %llu misses, %llu private (no variant id), %llu evicted, %zu cached\n", static_cast<unsigned long long>(lookups), static_cast<unsigned long long>(store.hits), lookups != 0 ? 100.0 * static_cast<double>(store.hits) / static_cast<double>(lookups) : 0.0, static_cast<unsigned long long>(store.misses), static_cast<unsigned long long>(store.uncached), static_cast<unsigned long long>(store.evicted), store.entries.size());
+    store.hits = store.misses = store.uncached = store.evicted = 0;
+}
+
+}
+
+std::shared_ptr<Pipeline> CachedPipeline(const Context& context, const State& state, const VertexInputLayout& vertexInput, const ShaderResources& resources, std::span<const CompiledShader> shaders, VkImageLayout attachmentLayout) {
+    static const bool disabled = std::getenv("APS5_NO_PIPELINE_CACHE") != nullptr;
+    if (disabled) return std::make_shared<Pipeline>(context, state, vertexInput, resources, shaders, attachmentLayout);
+    auto& store = Pipelines();
+    std::lock_guard lock(store.mutex);
+    reportPipelines(store);
+    const auto key = pipelineKey(context, state, vertexInput, resources, shaders, attachmentLayout);
+    if (key.empty()) {
+        ++store.uncached;
+        return std::make_shared<Pipeline>(context, state, vertexInput, resources, shaders, attachmentLayout);
+    }
+    const auto hash = hashKey(key);
+    if (const auto found = store.index.find(hash); found != store.index.end()) {
+        const auto it = found->second;
+        if (it->key == key) {
+            // The key names the device handle, which the loader may reuse for a device created after
+            // this one was destroyed without ClearCachedPipelines: such an entry is a miss.
+            if (alive(*it, context)) {
+                ++store.hits;
+                store.entries.splice(store.entries.end(), store.entries, it);
+                return it->pipeline;
+            }
+            abandon(store, it);
+        } else {
+            // A different configuration with the same hash keeps the resident entry; this one stays private.
+            ++store.uncached;
+            return std::make_shared<Pipeline>(context, state, vertexInput, resources, shaders, attachmentLayout);
+        }
+    }
+    ++store.misses;
+    // Entries of another device belong to one the driver replaced (it does so under the GpuMutex
+    // before any draw reaches the new device), whose objects went with it: forget them.
+    for (auto it = store.entries.begin(); it != store.entries.end();) {
+        it = alive(*it, context) ? std::next(it) : abandon(store, it);
+    }
+    auto pipeline = std::make_shared<Pipeline>(context, state, vertexInput, resources, shaders, attachmentLayout);
+    store.entries.push_back({context.device, context.bufferPool, hash, key, pipeline});
+    store.index[hash] = std::prev(store.entries.end());
+    constexpr std::size_t bound = 256;
+    while (store.entries.size() > bound) {
+        // Only an entry no recorded draw still holds may go (Kept keeps its shared_ptr until the fence).
+        const auto victim = std::find_if(store.entries.begin(), store.entries.end(), [](const PipelineStore::Entry& entry) { return entry.pipeline.use_count() == 1; });
+        if (victim == store.entries.end()) break;
+        store.index.erase(victim->hash);
+        store.entries.erase(victim);
+        ++store.evicted;
+    }
+    return pipeline;
+}
+
+void ClearCachedPipelines(VkDevice device) {
+    auto& store = Pipelines();
+    std::lock_guard lock(store.mutex);
+    for (auto it = store.entries.begin(); it != store.entries.end();) {
+        if (it->device != device) {
+            ++it;
+            continue;
+        }
+        store.index.erase(it->hash);
+        it = store.entries.erase(it);
+    }
 }
 
 }

core/libs/prx/libSceAgcDriver/Graphics/src/RegisteredGpuMemory.cpp

@@ -1,21 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/GuestBufferMemory.hpp"
-#include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp"
-
-namespace AgcDriver::Graphics {
-
-void GuestBufferMemory::AcquireRegistered() {
-    Require(!uploaded && regions.empty() && lease.empty(), "guest allocation lease must precede resource registration");
-    lease = GuestAllocations::GuestAllocationsAcquire_nid_postfix();
-    regions.reserve(lease.size());
-    for (const auto& range : lease) {
-        validate(range->address, range->bytes);
-        std::vector<std::byte> snapshot;
-        if (!range->writable) {
-            snapshot.resize(range->bytes);
-            GuestMemory::Read(range->address, snapshot);
-        }
-        regions.push_back({range->address, range->address + range->bytes, range->writable, std::move(snapshot), nullptr});
-    }
-}
-
-}

core/libs/prx/libSceAgcDriver/Graphics/src/RenderCache.cpp

@@ -1,131 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/RenderCache.hpp"
-#include "prx/libSceAgcDriver/Graphics/include/DrawQueue.hpp"
-#include "prx/libSceAgcDriver/Execution/include/MemoryAccessScope.hpp"
-#include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
-#include <limits>
-
-namespace AgcDriver::Graphics {
-
-void ResidentColor::Transition(VkCommandBuffer commands, VkImageLayout next) {
-    VkImageMemoryBarrier barrier{VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER};
-    barrier.srcAccessMask = layout == VK_IMAGE_LAYOUT_UNDEFINED ? 0u : VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT;
-    barrier.dstAccessMask = VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT;
-    barrier.oldLayout = layout;
-    barrier.newLayout = next;
-    barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
-    barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
-    barrier.image = Target().Image();
-    barrier.subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1};
-    context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier")(commands, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, 0, 0, nullptr, 0, nullptr, 1, &barrier);
-    layout = next;
-}
-
-void ResidentColor::Begin(VkCommandBuffer commands) {
-    PerformanceTimer timing("Graphics.ResidentColor.Begin");
-    Require(generation != std::numeric_limits<std::uint64_t>::max(), "resident color generation overflow");
-    ++generation;
-    Require(memoryWatch != nullptr, "render target memory ownership was released");
-    if (!valid) {
-        memoryWatch->Protect(GuestMemoryTracking::Protection::Read);
-        const GuestMemory::MemoryAccessScope suspended(nullptr, nullptr);
-        transfer.Upload(color.address, color.extent.width, color.extent.height, color.tileMode);
-        transfer.Detile(commands);
-        Transition(commands, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL);
-        VkBufferImageCopy copy{};
-        copy.imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1};
-        copy.imageExtent = {color.extent.width, color.extent.height, 1};
-        context.Function<PFN_vkCmdCopyBufferToImage>("vkCmdCopyBufferToImage")(commands, transfer.LinearBuffer(), Target().Image(), VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &copy);
-        timing.Mark("upload", color.bytes);
-    } else {
-        timing.Mark("reuse");
-    }
-    Transition(commands, VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL);
-    valid = true;
-    dirty = true;
-    memoryWatch->Protect(GuestMemoryTracking::Protection::None);
-}
-
-void ResidentColor::Download(VkCommandBuffer commands) {
-    if (!dirty) return;
-    Transition(commands, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL);
-    VkMemoryBarrier reuse{VK_STRUCTURE_TYPE_MEMORY_BARRIER};
-    reuse.srcAccessMask = VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT;
-    reuse.dstAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT;
-    context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier")(commands, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 1, &reuse, 0, nullptr, 0, nullptr);
-    VkBufferImageCopy copy{};
-    copy.imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1};
-    copy.imageExtent = {color.extent.width, color.extent.height, 1};
-    context.Function<PFN_vkCmdCopyImageToBuffer>("vkCmdCopyImageToBuffer")(commands, Target().Image(), VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, transfer.LinearBuffer(), 1, &copy);
-    transfer.Tile(commands);
-}
-
-void ResidentColor::Commit() {
-    if (!dirty) return;
-    const GuestMemory::MemoryAccessScope suspended(nullptr, nullptr);
-    transfer.WriteBackTracked(color.address);
-    dirty = false;
-    memoryWatch->Protect(GuestMemoryTracking::Protection::Read);
-}
-
-std::shared_ptr<ResidentColor> RenderCache::Get(const ColorTarget& color, bool blending) {
-    Require(color.address != 0 && color.bytes != 0 && color.bytes <= std::numeric_limits<std::uint64_t>::max() - color.address, "invalid resident color range");
-    if (context.drawQueue) context.drawQueue->Resolve(color.address, color.bytes);
-    if (blending) {
-        VkFormatProperties properties{};
-        context.formatProperties(context.physical, color.format, &properties);
-        Require((properties.optimalTilingFeatures & VK_FORMAT_FEATURE_COLOR_ATTACHMENT_BLEND_BIT) != 0, "render target format does not support blending");
-    }
-    for (auto it = entries.begin(); it != entries.end();) {
-        const auto& previous = it->second->Description();
-        if (!it->second->SharesPages(color)) {
-            ++it;
-            continue;
-        }
-        if (previous.address == color.address && previous.bytes == color.bytes && previous.extent.width == color.extent.width && previous.extent.height == color.extent.height && previous.format == color.format && previous.tileMode == color.tileMode) return it->second;
-        Resolve(previous.address, previous.bytes, true);
-        it->second->ReleaseMemory();
-        it = entries.erase(it);
-    }
-    if (entries.size() >= 64) {
-        Flush();
-        for (const auto& [address, resident] : entries) resident->ReleaseMemory();
-        entries.clear();
-    }
-    auto entry = std::make_shared<ResidentColor>(context, color);
-    entries.emplace(color.address, entry);
-    return entry;
-}
-
-std::shared_ptr<ResidentColor> RenderCache::Find(std::uint64_t address) const {
-    const auto it = entries.find(address);
-    if (it != entries.end() && context.drawQueue) context.drawQueue->Resolve(address, it->second->Description().bytes);
-    return it != entries.end() && it->second->Valid() ? it->second : nullptr;
-}
-
-void RenderCache::Resolve(std::uint64_t address, std::size_t bytes, bool writable) {
-    std::vector<std::shared_ptr<ResidentColor>> affected;
-    for (const auto& [base, entry] : entries) {
-        const auto& color = entry->Description();
-        if (address >= base + color.bytes || base >= address + bytes) continue;
-        if (entry->Dirty()) affected.push_back(entry);
-        else if (writable) entry->Invalidate();
-    }
-    if (affected.empty()) return;
-    PerformanceTimer timing("Graphics.RenderCache.Resolve");
-    if (context.drawQueue) context.drawQueue->Wait();
-    CommandBatch batch(context);
-    for (const auto& entry : affected) entry->Download(batch.Handle());
-    batch.SubmitAndWait();
-    timing.Mark("download_wait");
-    for (const auto& entry : affected) {
-        entry->Commit();
-        if (writable) entry->Invalidate();
-    }
-    timing.Mark("guest_writeback");
-}
-
-void RenderCache::Flush() {
-    Resolve(0, std::numeric_limits<std::size_t>::max(), true);
-}
-
-}

core/libs/prx/libSceAgcDriver/Graphics/src/RenderMemoryOwnership.cpp

@@ -1,62 +0,0 @@
-#include "prx/libSceAgcDriver/Graphics/include/RenderCache.hpp"
-#include "prx/libSceAgcDriver/Graphics/include/DrawQueue.hpp"
-#include "prx/libSceAgcDriver/Execution/include/PerformanceTimer.hpp"
-#include <limits>
-
-namespace AgcDriver::Graphics {
-
-ResidentColor::ResidentColor(const Context& context, const ColorTarget& color) : context(context), color(color), transfer(context) {
-    memoryWatch = std::make_unique<GuestMemoryTracking::Watch>(color.address, color.bytes, this, [](void* owner, GuestMemoryTracking::Access access) {
-        static_cast<ResidentColor*>(owner)->resolveCpuAccess(access);
-    });
-}
-
-ResidentColor::~ResidentColor() = default;
-
-void ResidentColor::Invalidate() {
-    Require(!dirty, "cannot discard GPU-owned render target contents");
-    if (memoryWatch) memoryWatch->Protect(GuestMemoryTracking::Protection::ReadWrite);
-    valid = false;
-}
-
-void ResidentColor::ReleaseMemory() {
-    Require(!dirty, "cannot release GPU-owned render target memory");
-    Invalidate();
-    memoryWatch.reset();
-}
-
-bool ResidentColor::SharesPages(const ColorTarget& other) const {
-    const auto pageSize = GuestMemoryTracking::GuestMemoryTrackingPageSize_nid_postfix();
-    Require(color.bytes != 0 && other.bytes != 0, "empty render target page range");
-    Require(color.bytes <= std::numeric_limits<std::uint64_t>::max() - color.address && other.bytes <= std::numeric_limits<std::uint64_t>::max() - other.address, "render target page range overflow");
-    const auto first = color.address / pageSize;
-    const auto last = (color.address + color.bytes - 1) / pageSize;
-    const auto otherFirst = other.address / pageSize;
-    const auto otherLast = (other.address + other.bytes - 1) / pageSize;
-    return first <= otherLast && otherFirst <= last;
-}
-
-void ResidentColor::resolveCpuAccess(GuestMemoryTracking::Access access) {
-    PerformanceTimer timing("Graphics.RenderMemory.CpuAccess");
-    Require(memoryWatch != nullptr && context.drawQueue != nullptr, "render target memory resolver is unavailable");
-    if (dirty || access == GuestMemoryTracking::Access::Invalidate) {
-        context.drawQueue->WaitGpu();
-        timing.Mark("draw_wait");
-    }
-    if (dirty) {
-        CommandBatch batch(context);
-        Download(batch.Handle());
-        batch.SubmitAndWait();
-        timing.Mark("download_wait");
-        Commit();
-        timing.Mark("guest_writeback");
-    }
-    if (access != GuestMemoryTracking::Access::Read) Invalidate();
-    if (access == GuestMemoryTracking::Access::Invalidate) context.drawQueue->Wait();
-}
-
-RenderCache::~RenderCache() {
-    for (const auto& [address, entry] : entries) entry->ReleaseMemory();
-}
-
-}