A resident direct memory mapping created a SharedPage, a Physical, a
physical map node and an aliases buffer for every 16 KiB page, about
18 MiB of heap per GiB mapped, although a page mapped once holds no
state there. Resident pages now keep only their View, with pins and the
compare-when-shared mark in it; an index from (section, offset) to the
resident run finds a live resident mapping of a physical page, and the
shared state is created when a run is demoted, or when it is evicted
with data or a compare mark. ForgetSection drops the index entries of
the section's runs.
guest_memory on Windows 11 26200 (Ryzen 5 5600X): private memory per
GiB mapped at map time drops from 30.9 MiB to 13.4 MiB.
Co-Authored-By: Claude <[email protected]>
A page marked to be compared when shared is refused by every collect once a
second mapping demotes it, so the writes made to it since its last collect
were never reported. Each view of such a page now carries an unreported mask:
set when the view is demoted, when the page is first marked while the view is
already shared, and when a new view of a marked page is mapped. ComparedPages
lists the compared shared pages of a range, or takes the unreported ones.
A private guest commit on Windows (every anonymous kernel map commits
with a 16 KiB granule) ran reset() on each page before replacing it:
10 VirtualQuery calls, a placeholder split and the VirtualAlloc2, so
12 calls per page. At the arena's first-fit frontier the placeholder is
the arena's free tail, hundreds of GiB.
Commit now queries a placeholder once, cuts the range out of it (one
split where the range starts inside the placeholder, one where the
placeholder extends past the range), and carves the pages from that
bounded placeholder: one split and one VirtualAlloc2 per page, and no
split for the last page. Each page is still its own allocation with the
same protection, write watch and created ranges, so munmap, MAP_FIXED
remaps, views and collects see the same layout as before. A piece that
would cross the end of its placeholder (a whole-range granule over
several placeholders) still takes the coalescing path through reset(),
and committed memory in the range still only changes its protection. A
zero granule now throws std::invalid_argument before touching the
address space.
windows_write_watch counts the queries with a compile-time counter that
only that test defines (APS5_COUNT_MAPPING_QUERIES, copied from #9505)
and checks: 256 fresh pages one page into a 1 GiB placeholder take one
query (main's loop makes 2,561), each page is its own allocation and
the placeholders around them are intact; the release and recommit of
one page; a protection-only commit; the write watch of carved pages; a
whole-range granule inside one placeholder and across two; committed
pieces and adjacent placeholders inside the range; a partial release
and a fixed remap that keep their neighbours; a granule that would
cross the end of the placeholder being carved, which takes the
coalescing path; a short last piece; a commit that runs to the end of
its placeholder; an unwatched commit; and the zero-granule throw.
PPSA14363 boot on an RX 6700 XT, idle PC, 4 runs on main and 3 with
this change: the first 10 s after the first present go from 546-552
presents with a 578-704 ms frame to 588-594 presents with a 97-117 ms
frame.
Co-Authored-By: Claude <[email protected]>
#9159 marks memory that a rewatch or an import kept watched while it was
resident with SharedPage::comparedWhenShared, so its shared views stay
compared instead of armed. The mark lived only on the SharedPage, which
a physical slot holds weakly: once every view of that physical page was
unmapped, Map's pageOf made a fresh, unmarked SharedPage for the next
mapping, and the next clearing collect armed its aliases again.
The mark is now also kept on the physical slot. CompareWhenShared sets
it through each view's slot, and pageOf copies it into a SharedPage it
creates. ForgetSection still drops the slot and the mark with it, so a
reused section handle starts unmarked, and memory shared before it was
ever marked keeps its tracking. MapAlias and its runs create no
SharedPage, so they need no change.
The test maps one page resident, marks it, unmaps it, maps it into two
views, and requires a clearing collect over either view to refuse and
neither to be made read-only; after ForgetSection the same mapping is
collected and armed again.
Grand Theft Auto V (PPSA04264): tibzejoker's remaining resumed write
faults in zlib's inflate_fast with #9181 and #9159
(https://github.com/boykopovar/AnyPS5/pull/9159#issuecomment-6355849767)
may come from memory remapped this way. This change does not claim to
remove them.
A private collect read from the allocation map makes no query, so it
can't see memory that code bypassing WindowsMappings changed inside a
recorded allocation. A page decommitted in place there is the case it
can't catch: GetWriteWatch still succeeds over it, so the collect
reports no writes where a query would have refused the range.
With APS5_VERIFY_DIRECT_COLLECT set, each such collect also queries its
span's regions, and Collect throws, naming the span and the switch, when
GetWriteWatch succeeded over memory that is no longer committed
private. ENVIRONMENT.md lists the switch.
windows_write_watch_verify_direct (the same test binary with
--verify-direct, the switch set) collects a recorded private allocation,
checks its write is still reported, decommits one page in place and
requires the next collect not to succeed. Without the check the collect
succeeds.
Adapts fable-void's 2760ad808.
Collecting private guest writes from the allocation map trusts the
extent the mappings recorded and calls GetWriteWatch without a query.
When memory in that extent was released by code that bypasses
WindowsMappings, the call fails and Collect threw "collect
private guest writes". Before the map, the query found the memory no
longer committed private and Collect returned false, which its callers
already handle.
A failed GetWriteWatch now queries the region at the cursor. If it is
no longer committed private memory, Collect returns false as before;
otherwise the failure is still fatal.
windows_write_watch releases a collected private allocation with
VirtualFree behind the mappings, then collects it again: it must return
false without throwing. Without the query it throws.
Adapts fable-void's 2760ad808.
Mapping shared direct memory while shared maps are imported placed one
view per 16 KiB page: a placeholder split, a map and a protect for every
page. A 1 GiB map made about 200,000 system calls, held the mapping
thread for about 0.4 s and took and dropped the mappings mutex on every
page, so threads taking write faults queued behind it.
Map now maps each run of shared pages that need a view as one view: one
placeholder split, one map at read-write-execute (the cap later guest
protections need) and one protect per run, and builds the per-page
bookkeeping under the mutex while yielding to waiting faults. The views
move the run's page references instead of copying them, so releasing the
run does not hold the mutex. Protections and write tracking change
subpages inside the view as before. A view is split only where a release
or a remap cuts it: reset unmaps the view, maps the two pieces again over
the same section offsets and restores every 4 KiB protection, so armed
write tracking and guest protections survive the split. An alias of a
page elsewhere and a per-page protection leave the view whole.
Tested: windows_write_watch gains a case that maps 1 GiB of shared
memory (65536 views before, one after), then protects one page, aliases
one page, arms write tracking on one page, releases one page and remaps
another: the run splits only at the released and remapped pages (2 and
4 views), every other page keeps its bytes and protection, and the armed
page still faults and is collected. It fails before ("a shared gibibyte
was mapped as 65536 views instead of one") and passes after.
agc_driver_committed_imports: the 1 GiB shared map and import takes
190-196 ms on the mapping thread instead of 418-427 ms, and the longest
write fault on another thread stays at 46-97 us. Built; the write watch,
committed import, guest memory, write guard, host and window import,
residency and mapping tests pass (58 tests).
Mapping shared guest memory page by page queried the address space
three times for every 16 KiB page: twice to find the placeholder the
page splits from, and once more to see whether the section page still
had to be committed, which it then committed on its own. For a range of
fresh direct memory that is seven system calls a page, and with shared
imports attached every direct mapping takes this path on the guest
thread that maps it, under the mapping lock.
A mapping that goes page by page now commits its section range once
through one window, and splits each page off the placeholder the
previous page left, whose extent it already knows; only a page after a
resident run or a demotion looks the placeholder up again. A page now
costs a placeholder split, the view and its protection.
Tested: built; windows_write_watch gains a case that maps 256 shared
pages and counts the address-space queries the mapping makes, then
stores to and reads back every page. With the per-page lookups it fails
(775 queries for 256 pages) and with this change it passes.
agc_driver_committed_imports, agc_write_watch_coverage,
agc_private_collect, agc_write_guards, agc_driver_dcc_keys_first_write,
agc_driver_residency_dirty_mirrors, agc_driver_residency_driver,
guest_memory and the host import, window import and alias store tests
pass.
Since #8172, a host import that stays in the write watch keeps each of
its watched runs in the watch instead of taking the whole import out
when part of it was unwatched. Grand Theft Auto V (PPSA04264) imports
resident direct memory that way, and later maps the same memory at a
second address, which turns it into shared views. From then on the write
watch arms them, and the game's own stores fault and are resumed, the
red zone hazard of #361 (#9136, eboot 0x162b9e4 and 0x28f413a).
ImportWatched now marks the resident pages of each run it keeps in the
watch with GuestArenaCompareResidentWhenShared. They stay write-watched
while they are private, as #8172 needs for resident DCC targets. Once
they are shared, their collects fail and they are compared, as before
#8172, instead of being armed. Memory that is already shared when it is
imported is not marked and stays tracked as before.
agc_rewatched_direct_shared gains the imported case: resident direct
memory imported with the watch kept, then mapped a second time, must
collect as 0 through either mapping, stay writable and show a store
through both, while unimported shared direct memory is still collected.
windows_write_watch gains CheckResidentOnlyComparedWhenShared: a
resident-only mark leaves a shared view armed and collected, and still
keeps a resident page compared once a second mapping makes it shared.
WindowsMappings keys its physical pages by the section handle's value. A
handle value that an earlier case closed without ForgetSection can come
back from CreateFileMappingW with that case's stored pages, and then the
first mapping is shared from the start instead of resident, so
CheckComparedWhenShared would fail its residency check. It passes today
only because the case before it forgets its section. The case now
forgets its own handle value before it maps.
On Windows a fault that AnyPS5 resumes in guest code can overwrite the
guest's red zone, [rsp-0x80, rsp-0x10) (#361). WindowsMappings arms every
alias of a shared page read-only when a clearing collect settles it, so
each guest store into that page faults and is resumed.
CompareWhenShared marks the pages of a range, and disarms any alias of
them that is already armed. Once a marked page is shared, collectable()
and the fresh pre-check refuse a clearing collect over it before anything
is armed, so its aliases are never armed and the caller compares the
range instead. While the page is still a resident run it is collected
through GetWriteWatch as before, which takes no fault.
windows_write_watch gains CheckComparedWhenShared: a marked resident
range stays collected, then a second mapping makes it shared, and from
then on its collects fail, neither mapping is armed and stores into it
take no fault, while the unmarked half of the same mapping is still
collected.
WindowsMappings keys each physical page's state (moved into the section,
saved bytes of an unmapped resident page) by the section handle value.
sceKernelReleaseDirectMemory frees the backing before it unmaps the views,
so ForgetSection ran while those views still existed and kept their slots.
After the views went, the slots stayed under a handle value that Windows
hands out again, and the next direct allocation whose section got that
value started with the old page state: its first mapping went to the
section instead of private memory, or received the old allocation's bytes.
Views now share ownership of their slot, ForgetSection drops every slot
of the section at once, and JIT shared memory forgets its section before
closing it too.
Adrià