fix(agc): keep CPU writes when a pending storage result is stored over untracked memory

On memory outside the write watch a write-back compares guest memory with
the bytes saved at the upload and leaves the 64 KiB blocks that differ to
the CPU. A store of some of an image's units marked the whole saved copy
invalid, and a fill turned into an image clear or an image copy never
saved the bytes under it, so the next store had nothing to compare and
wrote its layers whole over what the CPU had written since.

The saved copy is now valid per tracked unit. A store, a drop or a stale
eviction invalidates only its own units, a CPU-path store makes the units
it stored valid again, and the write-back compares each run of selected
units that still has saved bytes. FillClear and CopyFrom save the bytes
of the units they fill when the surface is not watched, as an upload does.

Results over a range that left the watch while they were pending still
have no saved bytes; storing them now throws instead of overwriting the
CPU's writes (technical debt).
This commit is contained in:
Dean Galvin committed 2026-10-13 21:46:12 +13:00
1 parent cac4e9688f
commit e5449936f4
4 files changed
+400 -34

No files matched your search

@@ -316,7 +316,7 @@ public:
// GuestMemory::GpuMutex only, as Refresh is; never from a build's stage A.
DccKeys UploadedKeys() const { return uploadedKeys; }
DccKeys FilledKeys() const { return filledKeys; }
bool GuestSnapshotValid() const { return originalValid; }
bool GuestSnapshotValid() const { return originalHolds(descriptor.baseAddress, static_cast<std::size_t>(guestBytes)); }
DccKeyProof& KeyProof() const { return keyProof; }
DccRangeProof& TargetKeyProof() const { return targetKeyProof; }
DccKeys ProvedKeys() const;
@@ -338,6 +338,7 @@ private:
// upload everything).
void upload(const std::vector<bool>* layers = nullptr);
void captureGuestBytes(const std::vector<bool>* layers);
bool originalHolds(std::uint64_t address, std::size_t bytes) const;
bool compareUntracked(std::uint64_t address, std::size_t bytes, std::span<std::uint8_t> changed, bool memoize = false) const;
// Stores the pending tracked layers overlapping [address, address + bytes) to guest memory; in
// each, 64 KiB blocks the CPU wrote since the layer's generation keep the CPU's bytes. Block
@@ -544,7 +545,7 @@ private:
bool dirty = false;
std::uint64_t version = 0;
// `original` holds the guest bytes; false after a GPU-side clear, which never read them.
bool originalValid = true;
std::vector<bool> layerOriginal;
// The storage cache let the image go (Flush): a lookup makes a new image of the surface, so
// no fill may put results into this one (guarded by the live-image registry lock).
bool released = false;
@@ -919,6 +919,7 @@ StorageTexture::StorageTexture(const Context& context, TextureDetiler& detiler,
trackedLayerBytes = blockUnits ? 65536 : guestBytes / trackedLayers;
layerGeneration.assign(trackedLayers, 0);
layerPending.assign(trackedLayers, false);
layerOriginal.assign(trackedLayers, true);
sliceLinearBytes = geometry.sliceLinearBytes;
const auto linearBytes = sliceLinearBytes * arrayLayers;
@@ -1655,7 +1656,7 @@ bool StorageTexture::Refresh() {
for (std::uint32_t layer = 0; layer < trackedLayers; ++layer) {
if (changed[layer] && layerPending[layer]) pendingChanged = true;
}
if (!unchanged && !keysChanged && !pendingChanged && originalValid && equalsOriginal()) {
if (!unchanged && !keysChanged && !pendingChanged && originalHolds(descriptor.baseAddress, static_cast<std::size_t>(guestBytes)) && equalsOriginal()) {
unchanged = true;
stamped = false;
}
@@ -1698,7 +1699,7 @@ bool StorageTexture::Refresh() {
static const bool traceUpload = std::getenv("APS5_TRACE_UPLOAD") != nullptr;
if (traceUpload) {
const auto keys = TextureClearKeys(descriptor, guestBytes);
std::fprintf(stderr, "[upload] 0x%llx+0x%llx %ux%u mips %u: %s (keys %s -> %s, originalValid %d, dirty %d)\n", static_cast<unsigned long long>(descriptor.baseAddress), static_cast<unsigned long long>(guestBytes), descriptor.width, descriptor.height, descriptor.mipCount, keys != uploadedKeys ? "DCC keys changed" : "guest memory changed", DccKeysName(uploadedKeys), DccKeysName(keys), originalValid ? 1 : 0, dirty ? 1 : 0);
std::fprintf(stderr, "[upload] 0x%llx+0x%llx %ux%u mips %u: %s (keys %s -> %s, originalValid %d, dirty %d)\n", static_cast<unsigned long long>(descriptor.baseAddress), static_cast<unsigned long long>(guestBytes), descriptor.width, descriptor.height, descriptor.mipCount, keys != uploadedKeys ? "DCC keys changed" : "guest memory changed", DccKeysName(uploadedKeys), DccKeysName(keys), originalHolds(descriptor.baseAddress, static_cast<std::size_t>(guestBytes)) ? 1 : 0, dirty ? 1 : 0);
}
// Changed layers with results pending: the CPU wrote this memory while GPU results were
// pending, so as with an immediate write-back followed by the CPU write, its blocks win and the
@@ -1884,7 +1885,6 @@ void traceKeyStore(const char* path, const GuestTextureResource& descriptor, std
void StorageTexture::captureGuestBytes(const std::vector<bool>* layers) {
comparedGuestBytes = {};
const bool complete = layers == nullptr || std::all_of(layers->begin(), layers->end(), [](bool selected) { return selected; });
std::vector<bool> selected(trackedLayers, true);
if (layers != nullptr) selected = *layers;
for (const auto& [begin, end] : unitRuns(selected)) GuestMemory::FlushGpuWrites(descriptor.baseAddress + begin, static_cast<std::size_t>(end - begin));
@@ -1892,12 +1892,19 @@ void StorageTexture::captureGuestBytes(const std::vector<bool>* layers) {
if (!selected[layer]) continue;
const auto offset = static_cast<std::size_t>(layerBegin(layer) - descriptor.baseAddress);
GuestMemory::ReadCommittedUnsynced(layerBegin(layer), std::span(original).subspan(offset, static_cast<std::size_t>(layerBytes(layer))));
layerOriginal[layer] = true;
}
originalValid = originalValid || complete;
}
bool StorageTexture::originalHolds(std::uint64_t address, std::size_t bytes) const {
for (auto layer = (address - descriptor.baseAddress) / trackedLayerBytes; layer < trackedLayers && layer * trackedLayerBytes < address - descriptor.baseAddress + bytes; ++layer) {
if (!layerOriginal[static_cast<std::size_t>(layer)]) return false;
}
return true;
}
bool StorageTexture::compareUntracked(std::uint64_t address, std::size_t bytes, std::span<std::uint8_t> changed, bool memoize) const {
if (!originalValid) return false;
if (!originalHolds(address, bytes)) return false;
struct Exempt {
const StorageTexture* previous;
~Exempt() { refreshing = previous; }
@@ -1906,7 +1913,7 @@ bool StorageTexture::compareUntracked(std::uint64_t address, std::size_t bytes,
Require(address >= descriptor.baseAddress && bytes <= guestBytes && address - descriptor.baseAddress <= guestBytes - bytes, "texture comparison exceeds its snapshot");
Require(original.size() == guestBytes, "texture comparison has no complete snapshot");
GuestMemory::FlushGpuWrites(address, bytes);
if (!originalValid) return false;
if (!originalHolds(address, bytes)) return false;
constexpr std::uint64_t blockBytes = 65536;
const auto end = address + bytes;
const auto first = address / blockBytes;
@@ -1972,7 +1979,7 @@ void StorageTexture::upload(const std::vector<bool>* layers) {
// batch of its own submitted the recorder's work first and waited for all of it, 25-40 ms
// under the GPU mutex at the movie stage. APS5_NO_RECORDED_CLEAR=1 waits as before.
static_cast<void>(HostImportFor(context, descriptor.baseAddress, static_cast<std::size_t>(guestBytes)));
if (GuestMemory::Watched(descriptor.baseAddress, static_cast<std::size_t>(guestBytes))) originalValid = false;
if (GuestMemory::Watched(descriptor.baseAddress, static_cast<std::size_t>(guestBytes))) layerOriginal.assign(trackedLayers, false);
else captureGuestBytes(nullptr);
forgetBorrowed(0, trackedLayers);
stampLayers(false);
@@ -2021,7 +2028,7 @@ void StorageTexture::upload(const std::vector<bool>* layers) {
return;
}
if (const auto* import = uploadedKeys == DccKeys::Uncompressed ? HostImportFor(context, descriptor.baseAddress, static_cast<std::size_t>(guestBytes)) : nullptr) {
if (GuestMemory::Watched(descriptor.baseAddress, static_cast<std::size_t>(guestBytes))) originalValid = false;
if (GuestMemory::Watched(descriptor.baseAddress, static_cast<std::size_t>(guestBytes))) layerOriginal.assign(trackedLayers, false);
else captureGuestBytes(layers);
stampLayers(true);
// A whole-surface upload of a block-unit image goes through the windows too when a unit
@@ -2143,7 +2150,7 @@ void StorageTexture::upload(const std::vector<bool>* layers) {
if (profile) LookupOutcomes::Add(LookupOutcomes::UploadDirect, start);
return;
}
originalValid = true;
layerOriginal.assign(trackedLayers, true);
forgetBorrowed(0, trackedLayers);
stampLayers(false);
GuestMemory::ReadCommitted(descriptor.baseAddress, original);
@@ -3223,9 +3230,10 @@ bool StorageTexture::evictStale(std::uint64_t address, std::size_t bytes, std::s
// As the empty write-back leaves the layers: no results pending, the generation unchanged (the
// stamps make their next Refresh re-upload), `original` no longer vouching for the bytes.
for (std::uint32_t layer = 0; layer < trackedLayers; ++layer) {
if (selected(layer)) layerPending[layer] = false;
if (!selected(layer)) continue;
layerPending[layer] = false;
layerOriginal[layer] = false;
}
originalValid = false;
if (!anyLayerPending()) {
++dropped;
staleEvicted.fetch_add(1, std::memory_order_relaxed);
@@ -3285,7 +3293,7 @@ std::size_t StorageTexture::DiscardPendingInside(std::uint64_t address, std::siz
// touched the bytes a CPU-path upload copied).
texture->dirty = false;
texture->layerPending.assign(texture->trackedLayers, false);
texture->originalValid = false;
texture->layerOriginal.assign(texture->trackedLayers, false);
it = pending.textures.erase(it);
++discarded;
} else if (texture != exempt && texture->blockUnits && texture->overlaps(address, bytes)) {
@@ -3297,13 +3305,11 @@ std::size_t StorageTexture::DiscardPendingInside(std::uint64_t address, std::siz
const auto unitBegin = texture->layerBegin(unit);
if (!texture->layerPending[unit] || unitBegin < address || unitBegin + texture->layerBytes(unit) > end) continue;
texture->layerPending[unit] = false;
texture->layerOriginal[unit] = false;
unitsDropped.fetch_add(1, std::memory_order_relaxed);
dropped = true;
}
if (dropped) {
texture->originalValid = false;
++discarded;
}
if (dropped) ++discarded;
if (dropped && !texture->anyLayerPending()) {
texture->dirty = false;
it = pending.textures.erase(it);
@@ -3501,7 +3507,7 @@ bool StorageTexture::clearByKeysFill(DccKeys keys, std::uint8_t key) {
toGeneral.newLayout = VK_IMAGE_LAYOUT_GENERAL;
context.Function<PFN_vkCmdPipelineBarrier>("vkCmdPipelineBarrier")(commands, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, 0, 0, nullptr, 0, nullptr, 1, &toGeneral);
recorder->EndGpuTiming(timing, guestBytes);
originalValid = false;
layerOriginal.assign(trackedLayers, false);
uploadedKeys = keys;
filledKeys = DccKeys::Uncompressed;
keyProof = {};
@@ -3594,7 +3600,7 @@ bool StorageTexture::FillClear(std::span<const std::uint32_t, 4> pattern, std::u
std::lock_guard lock(live.mutex);
for (auto* texture : live.textures) {
if (texture == this || texture->released || !texture->overlaps(begin, bytes)) continue;
texture->originalValid = false;
texture->layerOriginal.assign(texture->trackedLayers, false);
if (texture->descriptor.dccAddress != 0) aliasedKeys = true;
}
}
@@ -3643,13 +3649,19 @@ bool StorageTexture::FillClear(std::span<const std::uint32_t, 4> pattern, std::u
// memo hit of this submission's epoch would not consume pages the game dirtied since that
// walk, the write-back's own walk would stamp them above this generation and keep their
// blocks, and the fill's pattern would be lost in them.
originalValid = false;
const bool watched = GuestMemory::Watched(descriptor.baseAddress, static_cast<std::size_t>(guestBytes));
if (watched) layerOriginal.assign(trackedLayers, false);
uploadedKeys = DccKeys::Uncompressed;
if (layer != WholeImage && trackedLayers > 1) {
// The cleared layer's units alone: their stamps above lie below this generation, the other
// units keep theirs (their pages are consumed by their own collects).
const auto first = blockUnits ? static_cast<std::uint32_t>((begin - descriptor.baseAddress) / trackedLayerBytes) : layer;
const auto count = blockUnits ? static_cast<std::uint32_t>(bytes / trackedLayerBytes) : 1u;
if (!watched) {
std::vector<bool> filled(trackedLayers, false);
std::fill_n(filled.begin() + first, count, true);
captureGuestBytes(&filled);
}
const auto now = GuestMemory::CollectWritesUncached(begin, bytes);
for (auto unit = first; unit < first + count; ++unit) layerGeneration[unit] = now;
refreshGeneration();
@@ -3657,6 +3669,7 @@ bool StorageTexture::FillClear(std::span<const std::uint32_t, 4> pattern, std::u
markLayersPending(first, count);
return true;
}
if (!watched) captureGuestBytes(nullptr);
layerGeneration.assign(trackedLayers, GuestMemory::CollectWritesUncached(descriptor.baseAddress, static_cast<std::size_t>(guestBytes)));
refreshGeneration();
forgetBorrowed(0, trackedLayers);
@@ -3721,7 +3734,7 @@ bool StorageTexture::CopyFrom(StorageTexture& source, const char*& refusal) {
auto& live = Live();
std::lock_guard lock(live.mutex);
for (auto* texture : live.textures) {
if (texture != this && !texture->released && texture->overlaps(descriptor.baseAddress, static_cast<std::size_t>(guestBytes))) texture->originalValid = false;
if (texture != this && !texture->released && texture->overlaps(descriptor.baseAddress, static_cast<std::size_t>(guestBytes))) texture->layerOriginal.assign(texture->trackedLayers, false);
}
}
GuestMemory::MarkWritten(descriptor.baseAddress, static_cast<std::size_t>(guestBytes));
@@ -3772,7 +3785,8 @@ bool StorageTexture::CopyFrom(StorageTexture& source, const char*& refusal) {
// As after a clear: the image holds the surface, guest memory follows at the write-back (which
// keeps the 64 KiB blocks the CPU writes from now on; the collect is uncached for the reason
// given in FillClear).
originalValid = false;
if (GuestMemory::Watched(descriptor.baseAddress, static_cast<std::size_t>(guestBytes))) layerOriginal.assign(trackedLayers, false);
else captureGuestBytes(nullptr);
uploadedKeys = DccKeys::Uncompressed;
layerGeneration.assign(trackedLayers, GuestMemory::CollectWritesUncached(descriptor.baseAddress, static_cast<std::size_t>(guestBytes)));
refreshGeneration();
@@ -3999,7 +4013,7 @@ bool StorageTexture::guestBytesSettled() const {
}
bool StorageTexture::unchangedSinceBaseline(std::uint64_t from, std::uint64_t to) const {
const auto& baseline = originalValid ? original : generationBaseline;
const auto& baseline = originalHolds(from, static_cast<std::size_t>(to - from)) ? original : generationBaseline;
if (baseline.size() != guestBytes) return false;
const auto offset = static_cast<std::size_t>(from - descriptor.baseAddress);
return GuestMemory::EqualsCommittedUnsynced(from, std::span<const std::byte>(baseline).subspan(offset, static_cast<std::size_t>(to - from)));
@@ -4052,12 +4066,29 @@ void StorageTexture::writeBackLayers(const std::vector<bool>& layers) {
}
std::vector<std::uint8_t> changedBlocks(spanBlocks);
const bool tracked = GuestMemory::ChangedBlocks(firstStored, static_cast<std::size_t>(lastStored - firstStored), generations, changedBlocks);
const bool compared = !tracked && compareUntracked(firstStored, static_cast<std::size_t>(lastStored - firstStored), changedBlocks);
std::vector<bool> compared(trackedLayers, false);
if (!tracked) {
std::vector<bool> saved(trackedLayers, false);
for (std::uint32_t layer = 0; layer < trackedLayers; ++layer) saved[layer] = layers[layer] && layerOriginal[layer];
for (const auto& [begin, end] : unitRuns(saved)) {
const auto from = descriptor.baseAddress + begin;
const auto last = descriptor.baseAddress + end - 1;
if (!compareUntracked(from, static_cast<std::size_t>(end - begin), std::span(changedBlocks).subspan(static_cast<std::size_t>((from - spanBegin) / block), static_cast<std::size_t>(last / block - from / block + 1)))) continue;
for (auto layer = static_cast<std::uint32_t>(begin / trackedLayerBytes); layer < trackedLayers && layerBegin(layer) <= last; ++layer) compared[layer] = true;
}
}
const auto forgetStored = [&] {
for (std::uint32_t layer = 0; layer < trackedLayers; ++layer) {
if (layers[layer]) layerOriginal[layer] = false;
}
};
bool unsavedOutsideWatch = false;
for (std::uint32_t layer = 0; layer < trackedLayers; ++layer) {
if (!layers[layer]) continue;
const auto begin = layerBegin(layer);
const auto end = begin + layerBytes(layer);
if (!compared && (!tracked || layerGeneration[layer] == 0)) {
if (tracked ? layerGeneration[layer] == 0 : !compared[layer]) {
unsavedOutsideWatch = unsavedOutsideWatch || !tracked;
keep.emplace_back(begin, end);
continue;
}
@@ -4066,7 +4097,7 @@ void StorageTexture::writeBackLayers(const std::vector<bool>& layers) {
const auto to = std::min(at + block, end);
if (from >= to) continue;
const bool edge = from != at || to != at + block;
if (changedBlocks[static_cast<std::size_t>((at - spanBegin) / block)] == GuestMemory::BlockWritten && (compared || !edge || GuestMemory::StoredOver(from, static_cast<std::size_t>(to - from), layerGeneration[layer])) && !unchangedSinceBaseline(from, to)) {
if (changedBlocks[static_cast<std::size_t>((at - spanBegin) / block)] == GuestMemory::BlockWritten && (compared[layer] || !edge || GuestMemory::StoredOver(from, static_cast<std::size_t>(to - from), layerGeneration[layer])) && !unchangedSinceBaseline(from, to)) {
skippedAny = true;
skippedLayer[layer] = true;
++skipped;
@@ -4118,18 +4149,19 @@ void StorageTexture::writeBackLayers(const std::vector<bool>& layers) {
// store, the keys stay as they are, and the image is stale there (re-uploaded at its next
// use, never from a matching `original`).
emptyWriteBacks.fetch_add(1, std::memory_order_relaxed);
originalValid = false;
forgetStored();
settle(true);
return;
}
if (const auto* import = HostImportFor(context, descriptor.baseAddress, static_cast<std::size_t>(guestBytes))) {
Require(!unsavedOutsideWatch, "storing storage image results pending over memory outside the write watch without the guest bytes saved before them is not implemented");
if (blockUnits) {
// The kept blocks alone pass through the retiler (windows of their slices).
shadowImport = import;
const auto storedBytes = writeBackWindows(*import, keep, firstStored, lastStored, shadowed, imported);
countStorageWriteBack(storedBytes, true);
if (profile) Profile().storageGpu += timer.lap();
originalValid = false;
forgetStored();
traceKeyStore("block write-back", descriptor, guestBytes);
if (!IsDccClear(filledKeys) && uploadedKeys != DccKeys::Uncompressed) MarkDccUncompressed(context, descriptor.dccAddress, guestBytes, DccKeyCount(descriptor, guestBytes));
uploadedKeys = DccKeys::Uncompressed;
@@ -4234,7 +4266,7 @@ void StorageTexture::writeBackLayers(const std::vector<bool>& layers) {
countStorageWriteBack(storedBytes, true);
if (profile) Profile().storageGpu += timer.lap();
// The guest bytes now differ from `original`; other caches of the range see the write.
originalValid = false;
forgetStored();
// The keys go the same way as the texels: a fill recorded behind the retile when the
// metadata is host-imported (no CPU wait for the title's key-writing kernels), else a CPU
// store (APS5_CPU_DCC_KEYS=1 keeps the CPU store; see DccMetadata.hpp).
@@ -4258,7 +4290,7 @@ void StorageTexture::writeBackLayers(const std::vector<bool>& layers) {
static std::atomic<int> reports{0};
if (reports.fetch_add(1) < 4) std::fprintf(stderr, "[gpu] storage image 0x%llx+0x%llx: memory no longer registered; %zu pending ranges dropped\n", static_cast<unsigned long long>(descriptor.baseAddress), static_cast<unsigned long long>(guestBytes), keep.size());
unregisteredDropped.fetch_add(1, std::memory_order_relaxed);
originalValid = false;
forgetStored();
for (std::uint32_t layer = 0; layer < trackedLayers; ++layer) {
if (layers[layer]) layerPending[layer] = false;
}
@@ -4268,8 +4300,10 @@ void StorageTexture::writeBackLayers(const std::vector<bool>& layers) {
// them (a GPU clear, a dropped or kept unit). Reading them says nothing about the image: a
// stale unit (dropped, or kept for the CPU) still differs from them, so `original` vouches for
// the content again only where it did before, or once every unit is stored.
const bool wasValid = originalValid;
if (!wasValid) GuestMemory::ReadCommitted(descriptor.baseAddress, original);
Require(!unsavedOutsideWatch, "storing storage image results pending over memory outside the write watch without the guest bytes saved before them is not implemented");
std::vector<bool> unsaved = layerOriginal;
unsaved.flip();
for (const auto& [begin, end] : unitRuns(unsaved)) GuestMemory::ReadCommitted(descriptor.baseAddress + begin, std::span(original).subspan(static_cast<std::size_t>(begin), static_cast<std::size_t>(end - begin)));
DeviceBuffer linear(context, static_cast<std::size_t>(sliceLinearBytes * arrayLayers), VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT);
DeviceBuffer tiled(context, original.size(), VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT);
Buffer host(context, original.size(), VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT);
@@ -4353,7 +4387,9 @@ void StorageTexture::writeBackLayers(const std::vector<bool>& layers) {
uploadedKeys = DccKeys::Uncompressed;
// The store above is the only write to these pages, so `original` is current at a fresh
// generation, unless blocks were kept for the CPU: then the image is stale there.
originalValid = !skippedAny && (wasValid || std::all_of(layers.begin(), layers.end(), [](bool selected) { return selected; }));
for (std::uint32_t layer = 0; layer < trackedLayers; ++layer) {
if (layers[layer]) layerOriginal[layer] = !skippedLayer[layer];
}
// As in the GPU-direct path, but the store's memcpy dirtied this surface's pages in the write
// watch: a memoized collect would leave them for the next walk to stamp newer than this value
// (the shared edge block included, undoing the advance), so the walk is made here and
@@ -0,0 +1,328 @@
#include "prx/libSceAgcDriver/Execution/include/Driver.hpp"
#include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp"
#include "prx/libSceAgcDriver/Graphics/include/GuestTextureResource.hpp"
#include "prx/libSceAgcDriver/Graphics/include/Texture.hpp"
#include "prx/libSceAgcDriver/Graphics/include/TextureTiling.hpp"
#include "prx/libc/include/GuestAllocations.hpp"
#include "prx/libc/include/GuestArena.hpp"
#include "prx/libc/include/GuestWriteWatch.hpp"
#include "VulkanTestDevice.hpp"
#ifdef _WIN32
#ifndef NOMINMAX
#define NOMINMAX
#endif
#include <windows.h>
#else
#include <sys/mman.h>
#endif
#include <algorithm>
#include <array>
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <cstdio>
#include <iostream>
#include <mutex>
#include <optional>
#include <span>
#include <stdexcept>
#include <string_view>
#include <thread>
#include <vector>
namespace {
using AgcDriver::Graphics::Require;
using AgcDriver::Graphics::StorageTexture;
constexpr std::uint32_t Side = 256;
constexpr std::uint32_t Format32UInt = 20;
constexpr std::uint32_t TileR64KBX = 0x1b;
constexpr std::uint32_t Type2D = 9;
constexpr std::size_t Page = 0x1000;
constexpr std::size_t Block = 0x10000;
constexpr std::size_t SurfaceBytes = 4 * Block;
constexpr std::uint32_t Untouched = 0xdeadbeefu;
constexpr std::uint32_t Stored = 0x12345678u;
constexpr std::uint32_t Filled = 0x9abcdef0u;
alignas(256) constexpr std::array<std::uint32_t, 7> StoreCode{
0x7e040300, 0x7e060280, 0x7e0202ff, Stored, 0xf0200108, 0x00020102, 0xbf810000,
};
alignas(256) constexpr std::array<std::uint32_t, 9> FillCode{
0xd7460004, 0x04010c08, 0x7e000204, 0x7e020205, 0x7e040206, 0x7e060207, 0xe01c2000, 0x80000004, 0xbf810000,
};
std::uint64_t AddressOf(const void* data) {
return static_cast<std::uint64_t>(reinterpret_cast<std::uintptr_t>(data));
}
std::array<std::uint32_t, 8> TextureDescriptor(const std::uint32_t* texels) {
const auto address = AddressOf(texels);
return {
static_cast<std::uint32_t>(address >> 8u),
static_cast<std::uint32_t>((address >> 40u) & 0xffu) | (Format32UInt << 20u) | (((Side - 1u) & 3u) << 30u),
((Side - 1u) >> 2u) | ((Side - 1u) << 14u),
0xfacu | (TileR64KBX << 20u) | (Type2D << 28u),
0u,
0u,
0u,
0u,
};
}
void Dispatch(std::span<const std::uint32_t> code, std::uint32_t threads, std::uint32_t resource, std::span<const std::uint32_t> userData, std::uint32_t groups) {
std::vector<std::uint32_t> words;
const auto registers = [&](const std::uint32_t first, const std::span<const std::uint32_t> values) {
words.push_back(0xc0007600u | (static_cast<std::uint32_t>(values.size()) << 16u));
words.push_back(first);
words.insert(words.end(), values.begin(), values.end());
};
const std::array<std::uint32_t, 3> shape{threads, 1, 1};
registers(0x207, shape);
const auto address = AddressOf(code.data());
const std::array<std::uint32_t, 2> program{static_cast<std::uint32_t>(address >> 8u), static_cast<std::uint32_t>(address >> 40u)};
registers(0x20c, program);
const std::array<std::uint32_t, 1> layout{resource};
registers(0x213, layout);
registers(0x240, userData);
words.insert(words.end(), {0xc0031500u, groups, 1, 1, 0x41u});
Packet packet{words.data(), static_cast<std::uint32_t>(words.size()), 0, {}};
AgcDriver::Submit(&packet, 0);
}
void Store(const std::uint32_t* texels) {
std::array<std::uint32_t, 16> userData{};
const auto texture = TextureDescriptor(texels);
std::copy(texture.begin(), texture.end(), userData.begin() + 8);
Dispatch(StoreCode, 8, static_cast<std::uint32_t>(userData.size()) << 1u, userData, 1);
}
void Fill(const std::uint32_t* texels) {
const auto address = AddressOf(texels);
const std::array<std::uint32_t, 8> userData{
static_cast<std::uint32_t>(address),
static_cast<std::uint32_t>((address >> 32u) & 0xffffu) | (16u << 16u),
static_cast<std::uint32_t>(SurfaceBytes / 16u),
0x4bfacu,
Filled,
Filled,
Filled,
Filled,
};
Dispatch(FillCode, 64, (static_cast<std::uint32_t>(userData.size()) << 1u) | 0x80u, userData, static_cast<std::uint32_t>(SurfaceBytes / 1024u));
}
class Surface {
public:
explicit Surface(bool watched = false) : watched(watched) {
if (watched) {
#ifdef _WIN32
texels = static_cast<std::uint32_t*>(GuestArena::GuestArenaAllocate_nid_postfix(SurfaceBytes, Block));
if (texels != nullptr) GuestArena::GuestArenaCommit_nid_postfix(texels, SurfaceBytes, PAGE_READWRITE, SurfaceBytes);
#else
void* mapped = mmap(nullptr, SurfaceBytes + Block, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (mapped != MAP_FAILED) {
const auto begin = reinterpret_cast<std::uintptr_t>(mapped);
const auto aligned = (begin + Block - 1) & ~std::uintptr_t{Block - 1};
if (aligned != begin) munmap(mapped, aligned - begin);
if (aligned != begin + Block) munmap(reinterpret_cast<void*>(aligned + SurfaceBytes), begin + Block - aligned);
texels = reinterpret_cast<std::uint32_t*>(aligned);
GuestWriteWatch::GuestWriteWatchRegister_nid_postfix(texels, SurfaceBytes);
}
#endif
} else {
storage.resize((SurfaceBytes + Block) / 4u);
texels = reinterpret_cast<std::uint32_t*>((reinterpret_cast<std::uintptr_t>(storage.data()) + Block - 1) & ~std::uintptr_t{Block - 1});
}
Require(texels != nullptr, "store over untracked: cannot allocate the guest surface");
std::fill(texels, texels + SurfaceBytes / 4u, Untouched);
GuestAllocations::Mutation().Add(texels, SurfaceBytes, true, true, true);
}
~Surface() {
GuestAllocations::Mutation().Remove(texels);
if (!watched) return;
#ifdef _WIN32
GuestArena::GuestArenaReset_nid_postfix(texels, SurfaceBytes);
GuestArena::GuestArenaRelease_nid_postfix(texels, SurfaceBytes);
#else
munmap(texels, SurfaceBytes);
GuestWriteWatch::GuestWriteWatchUnregister_nid_postfix(texels, SurfaceBytes);
#endif
}
Surface(const Surface&) = delete;
Surface& operator=(const Surface&) = delete;
std::uint32_t* Texels() const { return texels; }
std::uint8_t* Bytes() const { return reinterpret_cast<std::uint8_t*>(texels); }
std::uint64_t Address() const { return AddressOf(texels); }
private:
bool watched;
std::vector<std::uint32_t> storage;
std::uint32_t* texels = nullptr;
};
std::uint8_t Pattern(std::size_t offset) {
return static_cast<std::uint8_t>(0x40u + offset % 0x3bu);
}
std::size_t LostBytes(const std::uint8_t* bytes) {
std::size_t lost = 0;
for (std::size_t offset = 0; offset < Page; ++offset) {
if (static_cast<const volatile std::uint8_t*>(bytes)[offset] != Pattern(offset)) ++lost;
}
return lost;
}
bool Holds(const std::uint32_t* texels, std::uint32_t value) {
const auto* words = static_cast<const volatile std::uint32_t*>(texels);
for (std::size_t index = 0; index < SurfaceBytes / 4u; ++index) {
if (words[index] == value) return true;
}
return false;
}
template <typename THold>
bool HoldPending(THold& hold, const Surface& surface, std::uint32_t result) {
const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10);
while (true) {
hold.lock();
if (StorageTexture::AnyPending()) return true;
hold.unlock();
if (Holds(surface.Texels(), result)) return false;
Require(std::chrono::steady_clock::now() < deadline, "store over untracked: the result neither became pending nor reached guest memory");
std::this_thread::yield();
}
}
template <typename THold>
void Settle(THold& hold, const Surface& surface, std::uint32_t result) {
if (!HoldPending(hold, surface, result)) return;
StorageTexture::FlushAllPending("store over untracked fixture");
hold.unlock();
}
enum class Kind { ReadAfter, ReadBefore, Filled, Copied, LeftWatch };
struct Case {
const char* name;
Kind kind;
std::size_t access;
};
enum class Outcome { Kept, Lost, Missed };
Outcome Run(const Case& test) {
std::optional<Surface> source;
if (test.kind == Kind::Copied) source.emplace();
Surface surface(test.kind == Kind::LeftWatch);
auto* bytes = surface.Bytes();
const auto address = surface.Address();
std::unique_lock hold(AgcDriver::GuestMemory::GpuMutex(), std::defer_lock);
bool unsaved = false;
if (test.kind == Kind::Copied) {
Store(source->Texels());
Settle(hold, *source, Stored);
Fill(source->Texels());
Settle(hold, *source, Filled);
}
Store(surface.Texels());
if (test.kind == Kind::Filled) {
Settle(hold, surface, Stored);
Fill(surface.Texels());
if (!HoldPending(hold, surface, Filled)) return Outcome::Missed;
} else if (test.kind == Kind::Copied) {
Settle(hold, surface, Stored);
hold.lock();
const auto from = StorageTexture::FindLive(source->Address(), SurfaceBytes);
const auto to = StorageTexture::FindLive(address, SurfaceBytes);
Require(from != nullptr && to != nullptr, "store over untracked: the surfaces have no storage images to copy between");
const char* refusal = nullptr;
if (!to->CopyFrom(*from, refusal)) {
Require(refusal != nullptr && std::string_view(refusal) == "not imported", "store over untracked: the image copy was refused");
std::fprintf(stderr, "[store over untracked] %s: not run, the surface has no host import\n", test.name);
return Outcome::Kept;
}
} else if (!HoldPending(hold, surface, Stored)) {
return Outcome::Missed;
}
if (test.kind == Kind::LeftWatch) {
if (!AgcDriver::GuestMemory::Watched(address, SurfaceBytes)) {
StorageTexture::FlushAllPending("store over untracked fixture");
std::fprintf(stderr, "[store over untracked] %s: not run, the surface is not write-watched\n", test.name);
return Outcome::Kept;
}
const auto image = StorageTexture::FindLive(address, SurfaceBytes);
Require(image != nullptr, "store over untracked: the surface has no storage image");
unsaved = !image->GuestSnapshotValid();
AgcDriver::GuestMemory::Unwatch(address, SurfaceBytes);
}
std::array<std::byte, Page> read{};
if (test.kind == Kind::ReadBefore) AgcDriver::GuestMemory::Read(address + Block + Page, read);
for (std::size_t offset = 0; offset < Page; ++offset) static_cast<volatile std::uint8_t*>(bytes)[offset] = Pattern(offset);
if (unsaved) {
bool refused = false;
try {
AgcDriver::GuestMemory::Read(address + test.access, read);
} catch (const std::runtime_error& error) {
if (std::string_view(error.what()).find("outside the write watch") == std::string_view::npos) throw;
refused = true;
}
static_cast<void>(StorageTexture::DiscardPendingInside(address, SurfaceBytes));
if (refused && LostBytes(bytes) == 0) return Outcome::Kept;
std::fprintf(stderr, "[store over untracked] %s: the result was stored with neither write stamps nor saved bytes to tell what the CPU wrote\n", test.name);
return Outcome::Lost;
}
AgcDriver::GuestMemory::Read(address + test.access, read);
const auto lostAtAccess = LostBytes(bytes);
StorageTexture::FlushAllPending("store over untracked fixture");
std::vector<std::byte> whole(SurfaceBytes);
AgcDriver::GuestMemory::Read(address, whole);
const auto lostAtEnd = LostBytes(bytes);
if (test.kind == Kind::Filled || test.kind == Kind::Copied) Require(static_cast<const volatile std::uint32_t*>(surface.Texels())[2 * Block / 4u] == Filled, "store over untracked: the result did not reach a block the CPU left alone");
if (lostAtAccess == 0 && lostAtEnd == 0) return Outcome::Kept;
std::fprintf(stderr, "[store over untracked] %s: of 0x%zx bytes the CPU wrote while the result was pending, %zu were lost after the access at +0x%zx and %zu after every result was stored\n", test.name, Page, lostAtAccess, test.access, lostAtEnd);
return Outcome::Lost;
}
}
int main() {
try {
{
const auto device = OpenVulkanTestDevice();
if (!device) return VulkanTestSkipped;
}
struct ShutdownGuard {
~ShutdownGuard() { try { AgcDriverShutdown_nid_postfix(); } catch (...) {} }
} shutdown;
{
Surface probe;
Require(AgcDriver::Graphics::DescribeSurface(AgcDriver::Graphics::DecodeTextureResource(TextureDescriptor(probe.Texels()))).guestBytes == SurfaceBytes, "store over untracked: unexpected surface size");
}
constexpr std::array<Case, 5> cases{{
{"another block read after the CPU write", Kind::ReadAfter, Block + Page},
{"another block read before the CPU write", Kind::ReadBefore, 0xe000},
{"a fill of the surface", Kind::Filled, 0xe000},
{"an image copy into the surface", Kind::Copied, 0xe000},
{"a surface that left the write watch", Kind::LeftWatch, 0xe000},
}};
bool kept = true;
for (const auto& test : cases) {
auto outcome = Outcome::Missed;
for (std::uint32_t attempt = 0; attempt < 256u && outcome == Outcome::Missed; ++attempt) outcome = Run(test);
Require(outcome != Outcome::Missed, "store over untracked: the result was never held pending");
kept = outcome == Outcome::Kept && kept;
}
Require(kept, "a pending storage result was stored over bytes the CPU wrote to untracked memory");
AgcDriverShutdown_nid_postfix();
std::puts("store over untracked tests passed");
return 0;
} catch (const std::exception& error) {
std::cerr << error.what() << '\n';
return 1;
}
}
+1
View File
@@ -41,6 +41,7 @@
- On a device without VK_EXT_robustness2's `nullDescriptor`, such as MoltenVK, a null image descriptor in a typed heap is a [placeholder image](../../../core/libs/prx/libSceAgcDriver/Graphics/src/PlaceholderImages.cpp) of that heap's type, 1×1 and cleared to zero, which every null descriptor of the heap shares. Reads give zeros, as `nullDescriptor` allows, but stores and atomics to it are kept, so a later read of a null storage image can return them, and a depth comparison compares against depth 0. A heap whose placeholder the device cannot create, for example 1D or 3D depth, 64-bit atomics without 64-bit image atomics, or multisampled storage without `shaderStorageImageMultisample`, throws when a null descriptor reaches it.
- [Draw input snapshots](../../../core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp) (libSceAgcDriver): `PrepareDrawBindings` caps read-only input buffer snapshots at 64 MiB (overridable via `APS5_DRAW_SNAPSHOT_MAX_MIB`). Larger inputs (streaming heaps, bindless pools) or draws exceeding the cumulative budget skip the per-draw snapshot and bind in place, avoiding multi-gigabyte CPU copy stalls (such as Ratchet & Clank: Rift Apart PPSA01474 stalling at frame ~380 copying 36 GB every 5 s). If the title rewrites those large ranges on the CPU before GPU completion, the in-place binding observes the modification, as verified by `drawSnapshotCapTests`.
- [Pending storage-image results](../../../core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp) (libSceAgcDriver) are stored when something reads their memory, not when the GPU work that wrote them completes. A title that waits for that work, frees the image and reuses its memory can have the new data overwritten by the late store (#1970). `APS5_STORE_AT_FLIP=1` stores every pending result at each flip; storing them at the title's GPU waits (labels, fences) is unimplemented.
- [Pending storage-image results over memory outside the write watch](../../../core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp) (libSceAgcDriver) leave to the CPU the 64 KiB blocks whose bytes differ from the ones saved at the upload, the fill or the image copy. Results over a range that left the watch after they became pending (a host import that does not stay watched, made while they were pending) have no saved bytes, and storing them throws; saving those bytes or storing the results when the range leaves the watch is unimplemented.
- Windows host-pointer imports of the guest mapping itself are compared instead of write-watched: a driver can mark imported pages dirty after the synchronous import window. Shared memory imported through a separate writable alias retains protection-based tracking on the guest views; driver access to the import alias does not disarm those views. [GuestMemory](../../../core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp) excludes every overlapping 64 KiB tracker block while its backing is retained. A full block replaced by fresh private backing regains tracking and gets a new generation; protection changes, partial replacements and shared remappings do not restore it. This preserves CPU edits through byte comparisons but disables snapshot reuse and unit shadows for excluded blocks; restoring tracking on import retirement without replacing the backing and measuring the comparison cost remain unimplemented.
- [Pages under an import window](../../../core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp) (libSceAgcDriver) whose write protection was resolved while a host import that leaves its range watched was made are compared with the bytes they held before the import and count as stored over only when those differ. A CPU store of the bytes already there within the window cannot be told apart from an import that resolves the write protection without storing, as the NVIDIA driver (610.57.04) does for every imported page, so such a store does not win over pending results. Where host-pointer and dma-buf imports differ in whether they stay watched, a failed host-pointer import and the dma-buf import after it are two windows, and the pages the failed one resolved count as stored over.
- [Unnormalized S# coordinates](../../../core/libs/prx/libSceAgcDriver/Graphics/src/GuestSamplerResource.cpp) (libSceAgcDriver) - `FORCE_UNNORMALIZED` uses a Vulkan `unnormalizedCoordinates` sampler, so the S# LOD range, LOD bias and mip filter are not applied, which is exact only on a one-level view.