Mirror-bot/AnyPS5
1
0 Fork 0
mirror of https://github.com/boykopovar/AnyPS5.git synced 2026-11-04 13:07:32 +00:00
Code Issues Projects Releases Activity
Files
bbf430e22daa824761df65014eec52f90e87e7fe
AnyPS5/core/libs/tests/GuestMemory.cpp
T
Dean Galvin d8c7c0ccdf feat(agc): track guest memory writes on Linux with async userfaultfd write protection 
GuestMemory::CollectWrites was built on the Windows arena's MEM_WRITE_WATCH
and answered 0 on Linux, so every cached copy of guest memory (textures,
resident render targets, buffers, DCC key proofs) read as changed at every
check: resident targets were stored and re-uploaded at every refresh and
unit shadows stayed off. Astro Bot's intro ran at ~3 fps.

libc gains GuestWriteWatch, the Linux counterpart of GetWriteWatch. Every
guest mapping libkernel makes (MapAligned: fixed, hinted and aligned) is
registered with one userfaultfd in asynchronous write-protect mode
(UFFD_FEATURE_WP_ASYNC: the kernel lifts a page's protection on its first
write without a handler thread, for user and kernel-mode writes alike), and
PAGEMAP_SCAN with PM_SCAN_WP_MATCHING reports the pages written since the
last scan of a range and protects them again in one call. Unmap forgets
the range. Direct memory is mapped from shared backings over its
reservation and a fixed remap replaces the mapping, so both register the
new mapping afresh. The watch is per mapping: a write through one mapping
of shared physical pages is not seen at another, so physical pages mapped
at more than one address leave the watch in every mapping of them.

A registered mapping reads as written until a collect covers it. The watch
keeps that itself instead of relying on how the kernel classifies pages
never faulted in: Linux 6.17 reports a range without page tables as not
written and leaves it unprotected, 7.2 reports it as written. The first
collect of a fresh range reports it written and write-protects it with
UFFDIO_WRITEPROTECT, which puts markers on its unpopulated pages
(UFFD_FEATURE_WP_UNPOPULATED), so from then on only real writes report.

Fallback: the userfaultfd is opened with UFFD_USER_MODE_ONLY where
vm.unprivileged_userfaultfd restricts it, and a probe at startup checks the
mechanism on scratch pages (a populated and an unpopulated page written
after protection report exactly, untouched ones do not). Without it (kernel
older than 6.7, no userfaultfd permission, headers without PAGEMAP_SCAN, or
APS5_NO_WRITE_WATCH=1) the watch reports unavailable and the tracker stays
unwatched exactly as before: CollectWrites answers 0 and callers compare
bytes. A range with a page outside the registered mappings also answers 0,
and a protection or scan that fails reports its remaining pages as written
and answers 0.

The driver's tracker keeps its contract and generations. On Linux its
block stamps live in 4 GiB leaves over the user address space, made on
first stamp, instead of the arena's flat arrays; the collect memo now works
on both platforms. GuestMemory::WriteWatched and Watched replace the arena
queries in UnitShadowEnabled and ShadowDestinationFor, so unit shadows run
on Linux over watched memory and stay off everywhere else. The Windows
arena path is unchanged apart from going through the same stamp helpers.

Host imports: a driver may lift the write protection of imported pages
itself. amdgpu validates the userptr BO behind a VK_EXT_external_memory_host
import with a writable HMM fault on every submission after the pages were
protected again, so PAGEMAP_SCAN reports them as written with no CPU store,
and a resident display image read as written at every present (found by
Pacuka on an RX 6650 XT with RADV: the presented image went black). NVIDIA
pins the pages once at the import and stays exact from then on. A probe at
device setup decides per device: it maps and watches scratch pages, collects
them clean, imports them, collects, submits a GPU copy that reads the import,
waits, and collects again. Pages written after the submission mean imports
resolve write protection: every import from then on takes its range off the
watch (GuestMemory::Unwatch) before it is made, so Watched is false for it,
CollectWrites answers 0 and callers compare bytes; GPU writes to watched
memory keep their MarkWritten stamps. The result is logged once:
"[write-watch] host imports resolve write protection: yes/no". A probe that
cannot run takes the unwatch side. APS5_WRITE_WATCH_IMPORTS=watch|unwatch
overrides the probe (default probe; anything else throws), and
APS5_NO_WRITE_WATCH=1 still turns the watch off. On the RTX 4090 the probe
reads 0 of 4 pages written after the GPU read (4 after the import itself),
so imports stay watched there.

The import itself lifts the protection once where imports stay watched:
on the RTX 4090 the pinning at vkAllocateMemory reports every imported page
as written (4 of 4 in the probe), with no CPU store. Such a report would
read as a CPU write over any GPU results pending in the range and drop
them. So every import of a watched range is made inside an import window
(GuestMemory::ImportWatched), under the tracker lock: the range is
collected right before the import, so earlier CPU writes are stamped as
usual, and again right after it, and what the window reports is stamped
"maybe written". Readers without pending results see it as a change
(UnchangedSince fails, ChangedBlocks reports it), so caches re-upload or
compare. Decisions about pending GPU results read it as untracked:
ChangedBlocks reports BlockMaybeWritten apart from BlockWritten, and the
drop rule, the write-back's keep blocks, the alias checks, the unit
shadow's stale drop and the whole-image stale checks (through the new
WrittenSince, which also answers false for a range off the watch) store
the results instead of dropping them. A real CPU store racing an import
also races the pending GPU writes to the same memory, which has no
defined order on hardware either, so the GPU results winning is
acceptable. From the second collect on the range is protected again and
watched normally. A range the watch does not cover whole, or a collect of
the window that fails, is taken off the watch instead.

An unwatched import stays off the watch for the rest of its mapping: a
remap registers the range afresh, the import kept over it takes it off
again at the next reconcile, and a re-import after an unmap follows the
decision. Unwatch invalidates the collect memo, so a collect memoized
before the import answers 0. A range that leaves the watch while a storage
image holds a generation for it no longer reads as CPU-written:
ChangedBlocks now tells whether the range is watched, and the drop rule,
the write-back's keep blocks and the alias borrow treat an unwatched range
as untracked (stored, never dropped), as a generation of 0 is.

Tests: guest_memory checks the watch through libkernel mappings (a CPU
store, a read, a kernel write through read(), another thread, a memcpy
across pages, an mprotect cycle, a partial munmap and a fixed remap) and
over a fresh 2 MiB-aligned mapping without page tables, checks direct
memory (watched, an alias takes the shared pages off the watch in both
mappings, a fixed direct remap over flexible memory is watched afresh), and
names the failing line and the reported pages. The recorder tests allocate watched
memory on both platforms, so the UnchangedSince, key proof, close race and
unit shadow tests run on Linux, and the storage refresh test gains a
watched variant: results kept on the GPU reach a CPU reader, and a plain
CPU store into one unit with newer results pending is found by the watch
alone. The recorder tests now also run the real import probe, check the
decision with the result injected (unwatch: an import leaves the watch, a
memoized collect, a partly imported range and the unimported neighbour
answer as they should, MarkWritten stamps only watched memory, a remap and
a re-import after an unmap follow the decision; watch: the import stays
watched and stamped), and check that results pending when their memory
leaves the watch are stored rather than dropped. A further test imports the
memory of a storage image with results pending in watch mode: the
results survive the import window and reach guest memory, a cache
generation taken before the import sees a change, the window reads as
maybe written and not as a CPU store, and a CPU store after it reads as
one again. Unit shadows and the
watched storage refresh report themselves as not run when imports are
compared.
2026-10-01 00:08:08 +13:00

469 lines
23 KiB
C++
Raw Blame History

#include "prx/libc/include/general/VabiMacros.hpp"
#include "prx/libc/include/GuestAllocations.hpp"
#include "prx/libc/include/GuestHeap.hpp"
#include "prx/libc/include/GuestArena.hpp"
#include "prx/libc/include/GuestWriteWatch.hpp"
#include <array>
#include "SceTypes.hpp"
#include <cstring>
#include <exception>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <limits>
#include <source_location>
#include <thread>
#include <utility>
#include <vector>
#if defined(__linux__)
#include <sys/mman.h>
#include <unistd.h>
#endif
extern "C" {
void* APS5_VABI mmap_nid_postfix(void*, std::size_t, int, int, int, std::int64_t) noexcept;
int APS5_VABI munmap_nid_postfix(void*, std::size_t) noexcept;
int* APS5_VABI __error_nid_postfix();
int APS5_VABI sceKernelMapNamedFlexibleMemory(void**, std::size_t, int, int, const char*);
int APS5_VABI sceKernelMapFlexibleMemory(void**, std::size_t, int, int);
int APS5_VABI sceKernelMunmap(void*, std::size_t);
int APS5_VABI sceKernelMprotect(const void*, std::size_t, int);
int APS5_VABI sceKernelVirtualQuery(const void*, int, VirtualQueryInfo*, std::uint64_t);
int APS5_VABI sceKernelSetVirtualRangeName(const void*, std::uint64_t, const char*);
int APS5_VABI sceKernelClearVirtualRangeName(const void*, std::uint64_t);
int APS5_VABI sceKernelAllocateDirectMemory(std::int64_t, std::int64_t, std::size_t, std::size_t, int, std::int64_t*);
int APS5_VABI sceKernelMapDirectMemory(void**, std::size_t, int, int, std::int64_t, std::size_t);
int APS5_VABI sceKernelReleaseDirectMemory(std::int64_t, std::size_t);
int APS5_VABI sceKernelReserveVirtualRange(void**, std::size_t, int, std::size_t);
}
static void Require(bool condition, std::source_location location = std::source_location::current()) {
if (!condition) {
std::fprintf(stderr, "Guest memory check failed at %s:%u\n", location.file_name(), static_cast<unsigned>(location.line()));
std::abort();
}
}
static const char* NameAt(const void* address) {
static VirtualQueryInfo info;
Require(sceKernelVirtualQuery(address, 0, &info, sizeof(info)) == 0);
return info.name;
}
static void CheckNamedAndHintedMappings() {
constexpr std::size_t length = 0x10000;
void* first = nullptr;
Require(sceKernelMapNamedFlexibleMemory(&first, length, 3, 0, "first mapping") == 0);
Require(std::strcmp(NameAt(first), "first mapping") == 0);
auto* middle = static_cast<unsigned char*>(first) + 0x4000;
Require(sceKernelSetVirtualRangeName(middle, 0x4000, "middle") == 0);
Require(std::strcmp(NameAt(first), "first mapping") == 0);
Require(std::strcmp(NameAt(middle), "middle") == 0);
Require(sceKernelClearVirtualRangeName(first, length) == 0);
Require(NameAt(middle)[0] == '\0');
Require(sceKernelSetVirtualRangeName(nullptr, length, "x") != 0);
#if defined(__linux__)
void* hinted = first;
Require(sceKernelMapFlexibleMemory(&hinted, length, 3, 0) == 0);
Require(hinted > first && (reinterpret_cast<std::uintptr_t>(hinted) & 0x3fff) == 0);
bool rejected = false;
void* overwrite = first;
try { sceKernelMapFlexibleMemory(&overwrite, 0x4000, 3, 0x90); } catch (const std::exception&) { rejected = true; }
Require(rejected && overwrite == first);
Require(sceKernelMunmap(hinted, length) == 0);
#endif
Require(sceKernelMunmap(first, length) == 0);
}
static void CheckDirectMemoryFollowsPhysicalPages() {
constexpr std::size_t page = 0x4000;
std::int64_t phys = 0;
Require(sceKernelAllocateDirectMemory(0, 0x7fffffffffll, page * 2, 0, 0, &phys) == 0);
void* first = nullptr;
Require(sceKernelMapDirectMemory(&first, page * 2, 3, 0, phys, 0) == 0);
static_cast<unsigned char*>(first)[0] = 11;
static_cast<unsigned char*>(first)[page + 5] = 22;
void* alias = nullptr;
Require(sceKernelMapDirectMemory(&alias, page, 3, 0, phys + page, 0) == 0);
Require(alias != first && static_cast<unsigned char*>(alias)[5] == 22);
static_cast<unsigned char*>(alias)[5] = 37;
Require(static_cast<unsigned char*>(first)[page + 5] == 37);
static_cast<unsigned char*>(first)[page + 6] = 48;
Require(static_cast<unsigned char*>(alias)[6] == 48);
Require(sceKernelMprotect(alias, page, 1) == 0);
static_cast<unsigned char*>(first)[page + 5] = 59;
Require(static_cast<unsigned char*>(alias)[5] == 59);
Require(sceKernelMprotect(alias, page, 3) == 0);
static_cast<unsigned char*>(alias)[5] = 22;
Require(static_cast<unsigned char*>(first)[page + 5] == 22);
Require(sceKernelMunmap(alias, page) == 0);
static_cast<unsigned char*>(first)[page + 5] = 22;
Require(sceKernelMunmap(first, page * 2) == 0);
void* filler = nullptr;
Require(sceKernelMapFlexibleMemory(&filler, page * 2, 3, 0) == 0);
void* second = nullptr;
Require(sceKernelMapDirectMemory(&second, page, 3, 0, phys + page, 0) == 0);
Require(second != first && static_cast<unsigned char*>(second)[5] == 22);
void* reserved = nullptr;
Require(sceKernelReserveVirtualRange(&reserved, page, 0, 0) == 0);
void* fixed = reserved;
Require(sceKernelMapDirectMemory(&fixed, page, 1, 0x10, phys, 0) == 0);
Require(fixed == reserved && static_cast<unsigned char*>(fixed)[0] == 11);
Require(sceKernelMunmap(fixed, page) == 0);
Require(sceKernelMunmap(second, page) == 0);
Require(sceKernelReleaseDirectMemory(phys, page * 2) == 0);
std::int64_t again = 0;
Require(sceKernelAllocateDirectMemory(0, 0x7fffffffffll, page * 2, 0, 0, &again) == 0 && again == phys);
void* fresh = nullptr;
Require(sceKernelMapDirectMemory(&fresh, page * 2, 3, 0, again, 0) == 0);
Require(static_cast<unsigned char*>(fresh)[0] == 0 && static_cast<unsigned char*>(fresh)[page + 5] == 0);
Require(sceKernelMunmap(fresh, page * 2) == 0);
Require(sceKernelMunmap(filler, page * 2) == 0);
Require(sceKernelReleaseDirectMemory(again, page * 2) == 0);
}
static void CheckSharedDirectMemoryLifecycle() {
constexpr std::size_t page = 0x4000;
std::int64_t phys = 0;
Require(sceKernelAllocateDirectMemory(0, 0x7fffffffffll, page * 3, 0, 0, &phys) == 0);
void* first = nullptr;
void* second = nullptr;
Require(sceKernelMapDirectMemory(&first, page * 3, 3, 0, phys, 0) == 0);
Require(sceKernelMapDirectMemory(&second, page * 3, 3, 0, phys, 0) == 0);
VirtualQueryInfo info{};
Require(sceKernelVirtualQuery(second, 0, &info, sizeof(info)) == 0);
Require(info.is_direct && !info.is_flexible && info.offset == static_cast<std::uint64_t>(phys));
auto* left = static_cast<unsigned char*>(first);
auto* right = static_cast<unsigned char*>(second);
left[0] = 31;
right[page] = 47;
left[page * 2] = 63;
Require(right[0] == 31 && left[page] == 47 && right[page * 2] == 63);
void* inaccessible = nullptr;
Require(sceKernelMapDirectMemory(&inaccessible, page, 0, 0, phys + page, 0) == 0);
left[page] = 48;
Require(sceKernelMprotect(inaccessible, page, 1) == 0);
Require(static_cast<const unsigned char*>(inaccessible)[0] == 48);
Require(sceKernelMunmap(inaccessible, page) == 0);
Require(sceKernelMunmap(left + page, page) == 0);
right[page] = 79;
Require(left[0] == 31 && left[page * 2] == 63);
void* middle = left + page;
Require(sceKernelMapDirectMemory(&middle, page, 3, 0x10, phys + page, 0) == 0);
Require(left[page] == 79);
Require(sceKernelVirtualQuery(middle, 0, &info, sizeof(info)) == 0);
Require(info.offset == static_cast<std::uint64_t>(phys) + page && info.start == reinterpret_cast<std::uintptr_t>(middle));
Require(sceKernelMprotect(second, page * 3, 0) == 0);
left[page] = 95;
Require(sceKernelMprotect(second, page * 3, 1) == 0);
Require(right[page] == 95);
Require(sceKernelMunmap(first, page) == 0);
Require(sceKernelMunmap(left + page * 2, page) == 0);
Require(sceKernelMunmap(middle, page) == 0);
Require(sceKernelMprotect(second, page * 3, 3) == 0);
right[page * 2] = 111;
void* reserved = nullptr;
Require(sceKernelReserveVirtualRange(&reserved, page * 3, 0, 0) == 0);
void* fixed = static_cast<unsigned char*>(reserved) + page;
Require(sceKernelMapDirectMemory(&fixed, page, 3, 0x10, phys + page * 2, 0) == 0);
Require(static_cast<unsigned char*>(fixed)[0] == 111);
static_cast<unsigned char*>(fixed)[0] = 127;
Require(right[page * 2] == 127);
Require(sceKernelMapFlexibleMemory(&fixed, page, 3, 0x10) == 0);
Require(static_cast<unsigned char*>(fixed)[0] == 0);
Require(sceKernelVirtualQuery(fixed, 0, &info, sizeof(info)) == 0);
Require(!info.is_direct && info.is_flexible && info.offset == 0);
static_cast<unsigned char*>(fixed)[0] = 143;
Require(right[page * 2] == 127);
Require(sceKernelMunmap(reserved, page * 3) == 0);
Require(sceKernelMunmap(second, page * 3) == 0);
Require(sceKernelReleaseDirectMemory(phys + page, page) == 0);
std::int64_t replacement = 0;
Require(sceKernelAllocateDirectMemory(phys + page, phys + page * 2, page, 0, 0, &replacement) == 0);
Require(replacement == phys + page);
void* mixed = nullptr;
Require(sceKernelMapDirectMemory(&mixed, page * 3, 3, 0, phys, 0) == 0);
const auto* data = static_cast<const unsigned char*>(mixed);
Require(data[0] == 31 && data[page] == 0 && data[page * 2] == 127);
Require(sceKernelMunmap(mixed, page * 3) == 0);
Require(sceKernelReleaseDirectMemory(phys, page * 3) == 0);
}
static void CheckHeapAfterMappingReuse() {
constexpr std::size_t bytes = 0x30000;
auto* pointer = static_cast<unsigned char*>(GuestHeap::GuestHeapAllocate_nid_postfix(bytes));
std::memset(pointer, 0x5a, bytes);
Require(pointer[0] == 0x5a && pointer[bytes - 1] == 0x5a);
GuestHeap::GuestHeapFree_nid_postfix(pointer);
pointer = static_cast<unsigned char*>(GuestHeap::GuestHeapAllocate_nid_postfix(bytes));
std::memset(pointer, 0xa5, bytes);
Require(pointer[0] == 0xa5 && pointer[bytes - 1] == 0xa5);
GuestHeap::GuestHeapFree_nid_postfix(pointer);
}
static void CheckSharedWriteTracking() {
#ifdef _WIN32
constexpr std::size_t page = 0x4000;
std::int64_t phys = 0;
Require(sceKernelAllocateDirectMemory(0, 0x7fffffffffll, page * 3, 0, 0, &phys) == 0);
void* first = nullptr;
void* second = nullptr;
Require(sceKernelMapDirectMemory(&first, page * 3, 3, 0, phys, 0) == 0);
Require(sceKernelMapDirectMemory(&second, page * 3, 3, 0, phys, 0) == 0);
const auto collect = [](void* address, std::size_t bytes, bool clear = true) {
std::array<void*, 32> pages{};
std::size_t count = pages.size();
Require(GuestArena::GuestArenaCollectWrites_nid_postfix(reinterpret_cast<std::uintptr_t>(address), bytes, pages.data(), &count, clear));
return count;
};
Require(collect(first, page * 3) == 12);
Require(collect(second, page * 3) == 12);
Require(collect(first, page * 3) == 0);
Require(collect(second, page * 3) == 0);
auto* left = static_cast<volatile unsigned char*>(first);
auto* right = static_cast<volatile unsigned char*>(second);
left[page + 5] = 21;
Require(right[page + 5] == 21);
Require(collect(first, page * 3, false) == 4);
Require(collect(first, page * 3, false) == 4);
Require(collect(first, page * 3) == 4);
Require(collect(second, page * 3) == 4);
Require(collect(first, page * 3) == 0);
right[page * 2] = 42;
Require(collect(first, page * 3) == 4);
Require(collect(second, page * 3) == 4);
Require(collect(second, page * 3) == 0);
Require(sceKernelMprotect(first, page * 3, 1) == 0);
collect(first, page * 3);
collect(second, page * 3);
right[0] = 63;
Require(left[0] == 63 && collect(first, page * 3) == 4);
Require(sceKernelMprotect(first, page * 3, 3) == 0);
collect(first, page * 3);
left[0] = 84;
Require(right[0] == 84 && collect(second, page * 3) != 0);
void* third = nullptr;
Require(sceKernelMapDirectMemory(&third, page, 3, 0, phys + page, 0) == 0);
collect(first, page * 3);
collect(second, page * 3);
static_cast<volatile unsigned char*>(third)[0] = 105;
Require(collect(first, page * 3) == 4 && collect(second, page * 3) == 4);
Require(sceKernelMunmap(third, page) == 0);
Require(sceKernelMunmap(first, page * 3) == 0);
Require(sceKernelMunmap(second, page * 3) == 0);
Require(sceKernelReleaseDirectMemory(phys, page * 3) == 0);
#endif
}
#if defined(__linux__)
using PageRuns = std::vector<std::pair<std::uintptr_t, std::uintptr_t>>;
static bool CollectRuns(const void* base, std::size_t offset, std::size_t bytes, PageRuns& runs) {
runs.clear();
const auto address = reinterpret_cast<std::uintptr_t>(base);
std::pair<std::uintptr_t, PageRuns*> context{address, &runs};
return GuestWriteWatch::GuestWriteWatchCollect_nid_postfix(address + offset, bytes, [](void* context, std::uintptr_t begin, std::uintptr_t end) {
auto& [origin, into] = *static_cast<std::pair<std::uintptr_t, PageRuns*>*>(context);
if (!into->empty() && into->back().second == (begin - origin) / 4096) into->back().second = (end - origin) / 4096;
else into->emplace_back((begin - origin) / 4096, (end - origin) / 4096);
}, &context);
}
static bool Written(const void* base, std::size_t bytes, PageRuns expected) {
PageRuns runs;
const bool complete = CollectRuns(base, 0, bytes, runs);
if (complete && runs == expected) return true;
std::fprintf(stderr, "write watch collect %s, written pages:", complete ? "complete" : "incomplete");
for (const auto& [first, last] : runs) std::fprintf(stderr, " [%zu, %zu)", static_cast<std::size_t>(first), static_cast<std::size_t>(last));
std::fputs("\n", stderr);
return false;
}
static void CheckWriteWatch() {
if (!GuestWriteWatch::GuestWriteWatchAvailable_nid_postfix()) {
std::puts("write watch unavailable: not tested");
return;
}
constexpr std::size_t length = 0x100000;
constexpr std::size_t small = 4096;
void* mapping = nullptr;
Require(sceKernelMapFlexibleMemory(&mapping, length, 3, 0) == 0);
auto* bytes = static_cast<volatile unsigned char*>(mapping);
const auto address = reinterpret_cast<std::uintptr_t>(mapping);
Require(GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address, length));
Require(GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address + small, small));
Require(!GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address, length + small));
Require(!GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(reinterpret_cast<std::uintptr_t>(&length), sizeof(length)));
Require(Written(mapping, length, {{0, length / small}}));
Require(Written(mapping, length, {}));
bytes[5 * small + 17] = 1;
Require(Written(mapping, length, {{5, 6}}));
Require(Written(mapping, length, {}));
static_cast<void>(bytes[10 * small]);
Require(Written(mapping, length, {}));
bytes[7 * small] = 1;
bytes[9 * small] = 1;
PageRuns runs;
Require(CollectRuns(mapping, 8 * small, small, runs) && runs.empty());
Require(CollectRuns(mapping, 7 * small + 100, 1, runs) && runs == PageRuns{{7, 8}});
Require(Written(mapping, length, {{9, 10}}));
int pipe[2];
Require(::pipe(pipe) == 0);
Require(::write(pipe[1], "kernel", 6) == 6);
Require(::read(pipe[0], const_cast<unsigned char*>(bytes + 20 * small + 8), 6) == 6);
::close(pipe[0]);
::close(pipe[1]);
Require(bytes[20 * small + 8] == 'k' && Written(mapping, length, {{20, 21}}));
std::thread([&] { bytes[30 * small + 5] = 3; }).join();
Require(Written(mapping, length, {{30, 31}}));
std::vector<unsigned char> source(2 * small, 0xab);
std::memcpy(const_cast<unsigned char*>(bytes + 40 * small + 2048), source.data(), source.size());
Require(Written(mapping, length, {{40, 43}}));
Require(sceKernelMprotect(mapping, length, 1) == 0);
Require(sceKernelMprotect(mapping, length, 3) == 0);
Require(Written(mapping, length, {}));
bytes[50 * small] = 1;
Require(Written(mapping, length, {{50, 51}}));
constexpr std::size_t guestPage = 0x4000;
auto* middle = const_cast<unsigned char*>(bytes + 4 * guestPage);
Require(sceKernelMunmap(middle, guestPage) == 0);
Require(!GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address, length));
Require(!CollectRuns(mapping, 0, length, runs) && runs.empty());
void* fixed = middle;
Require(sceKernelMapFlexibleMemory(&fixed, guestPage, 3, 0x10) == 0 && fixed == middle);
Require(GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address, length));
Require(Written(mapping, length, {{16, 20}}));
Require(Written(mapping, length, {}));
middle[1] = 1;
Require(Written(mapping, length, {{16, 17}}));
Require(sceKernelMunmap(mapping, length) == 0);
Require(!GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address, small));
constexpr std::size_t tableSpan = 0x200000;
constexpr std::size_t spanned = 2 * tableSpan;
void* raw = mmap(nullptr, spanned + tableSpan, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
Require(raw != MAP_FAILED);
const auto rawAddress = reinterpret_cast<std::uintptr_t>(raw);
void* region = reinterpret_cast<void*>((rawAddress + tableSpan - 1) & ~(tableSpan - 1));
GuestWriteWatch::GuestWriteWatchRegister_nid_postfix(region, spanned);
Require(GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(reinterpret_cast<std::uintptr_t>(region), spanned));
Require(Written(region, spanned, {{0, spanned / small}}));
Require(Written(region, spanned, {}));
static_cast<volatile unsigned char*>(region)[tableSpan + 3 * small] = 1;
static_cast<volatile unsigned char*>(region)[7 * small] = 1;
Require(Written(region, spanned, {{7, 8}, {tableSpan / small + 3, tableSpan / small + 4}}));
Require(Written(region, spanned, {}));
GuestWriteWatch::GuestWriteWatchUnregister_nid_postfix(region, spanned);
Require(munmap(raw, spanned + tableSpan) == 0);
}
static void CheckDirectMemoryWriteWatch() {
if (!GuestWriteWatch::GuestWriteWatchAvailable_nid_postfix()) {
std::puts("write watch unavailable: direct memory not tested");
return;
}
constexpr std::size_t page = 0x4000;
constexpr std::size_t small = 4096;
std::int64_t phys = 0;
Require(sceKernelAllocateDirectMemory(0, 0x7fffffffffll, page * 3, 0, 0, &phys) == 0);
void* first = nullptr;
Require(sceKernelMapDirectMemory(&first, page * 3, 3, 0, phys, 0) == 0);
const auto address = reinterpret_cast<std::uintptr_t>(first);
auto* bytes = static_cast<volatile unsigned char*>(first);
Require(GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address, page * 3));
Require(Written(first, page * 3, {{0, page * 3 / small}}));
Require(Written(first, page * 3, {}));
bytes[page + 5] = 1;
Require(Written(first, page * 3, {{page / small, page / small + 1}}));
void* alias = nullptr;
Require(sceKernelMapDirectMemory(&alias, page, 3, 0, phys + page, 0) == 0);
const auto aliasAddress = reinterpret_cast<std::uintptr_t>(alias);
Require(!GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(aliasAddress, page));
Require(!GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address + page, page));
Require(GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address, page) && GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address + page * 2, page));
PageRuns runs;
Require(!CollectRuns(first, 0, page * 3, runs));
bytes[0] = 2;
static_cast<volatile unsigned char*>(alias)[7] = 3;
Require(bytes[page + 7] == 3);
Require(CollectRuns(first, 0, page, runs) && runs == PageRuns{{0, 1}});
Require(CollectRuns(first, page * 2, page, runs) && runs.empty());
Require(sceKernelMunmap(alias, page) == 0);
Require(sceKernelMunmap(first, page * 3) == 0);
Require(!GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(address, page));
void* flexible = nullptr;
Require(sceKernelMapFlexibleMemory(&flexible, page * 3, 3, 0) == 0);
Require(Written(flexible, page * 3, {{0, page * 3 / small}}));
Require(Written(flexible, page * 3, {}));
void* fixed = static_cast<unsigned char*>(flexible) + page;
Require(sceKernelMapDirectMemory(&fixed, page, 3, 0x10, phys + page * 2, 0) == 0);
Require(fixed == static_cast<unsigned char*>(flexible) + page);
Require(GuestWriteWatch::GuestWriteWatchCovers_nid_postfix(reinterpret_cast<std::uintptr_t>(flexible), page * 3));
Require(Written(flexible, page * 3, {{page / small, page * 2 / small}}));
Require(Written(flexible, page * 3, {}));
static_cast<volatile unsigned char*>(fixed)[9] = 4;
Require(Written(flexible, page * 3, {{page / small, page / small + 1}}));
Require(sceKernelMunmap(flexible, page * 3) == 0);
Require(sceKernelReleaseDirectMemory(phys, page * 3) == 0);
}
#endif
int main() {
CheckNamedAndHintedMappings();
CheckDirectMemoryFollowsPhysicalPages();
CheckSharedDirectMemoryLifecycle();
CheckHeapAfterMappingReuse();
CheckSharedWriteTracking();
#if defined(__linux__)
CheckWriteWatch();
CheckDirectMemoryWriteWatch();
#endif
constexpr std::size_t page = 0x4000;
const auto failed = reinterpret_cast<void*>(static_cast<std::uintptr_t>(-1));
const auto reject = [&](std::size_t length, int protection, int flags, int fd,
std::int64_t offset, int error) {
*__error_nid_postfix() = 0;
Require(mmap_nid_postfix(nullptr, length, protection, flags, fd, offset) == failed);
Require(*__error_nid_postfix() == error);
};
reject(0, 3, 0x1002, -1, 0, 22);
reject(std::numeric_limits<std::size_t>::max(), 3, 0x1002, -1, 0, 22);
reject(page, 8, 0x1002, -1, 0, 22);
reject(page, 3, 0x1002, 0, 0, 22);
reject(page, 3, 0x1002, -1, 1, 22);
reject(page, 3, 0x1001, -1, 0, 45); // shared
reject(page, 3, 0x1012, -1, 0, 45); // fixed
reject(page, 3, 0x2, 0, 0, 45); // file-backed
reject(page, 3, 0x22, -1, 0, 45); // Linux MAP_ANON is not guest MAP_ANON
auto* memory = static_cast<unsigned char*>(mmap_nid_postfix(nullptr, page * 3 - 1, 3, 0x1002, -1, 0));
Require(memory != failed && (reinterpret_cast<std::uintptr_t>(memory) & (page - 1)) == 0);
for (std::size_t i = 0; i < page * 3; ++i) Require(memory[i] == 0);
memory[0] = 42;
memory[page * 2] = 73;
{
GuestAllocations::Mutation mutation;
const auto range = mutation.Find(memory);
Require(range.bytes == page * 3 && range.readable && range.writable);
}
Require(munmap_nid_postfix(memory + 1, page) == -1 && *__error_nid_postfix() == 22);
Require(munmap_nid_postfix(memory, 0) == -1 && *__error_nid_postfix() == 22);
Require(memory[0] == 42);
Require(munmap_nid_postfix(memory + page, 1) == 0); // round to one guest page
Require(memory[0] == 42 && memory[page * 2] == 73);
Require(munmap_nid_postfix(memory, page) == 0);
Require(memory[page * 2] == 73);
Require(munmap_nid_postfix(memory + page * 2, page) == 0);
Require(munmap_nid_postfix(memory, page) == -1);
for (int protection : {0, 1, 3, 5}) {
void* mapped = mmap_nid_postfix(memory, 1, protection, 0x1002, -1, 0);
Require(mapped != failed);
{
GuestAllocations::Mutation mutation;
const auto range = mutation.Find(mapped);
Require(range.readable == ((protection & 3) != 0));
Require(range.writable == ((protection & 2) != 0));
}
Require(munmap_nid_postfix(mapped, 1) == 0);
}
}
Reference in new issue View Git Blame Copy Permalink