Merge pull request #10809 from frankea/fix/direct-memory-12gib

fix(libkernel): report 12 GiB of direct memory, as a PS5 does

core/libs/prx/libkernel/DirectMemory/DirectMemory.hpp

@@ -7,7 +7,7 @@
 
 #include "prx/libkernel/KernelErrors.hpp"
 
-static constexpr size_t DIRECT_MEMORY_SIZE = 13824ULL * 1024 * 1024;
+static constexpr size_t DIRECT_MEMORY_SIZE = 12288ULL * 1024 * 1024;
 static constexpr size_t PS5_PAGE_SIZE = 0x4000;
 
 #ifdef _WIN32

core/libs/tests/GuestAprRead.cpp

@@ -68,7 +68,7 @@ private:
 class DirectBuffer {
 public:
     DirectBuffer() {
-        Require(sceKernelAllocateDirectMemory(0, 13824ll << 20, MappingBytes, 0x4000, 0, &physical) == 0, "cannot allocate direct memory");
+        Require(sceKernelAllocateDirectMemory(0, 12288ll << 20, MappingBytes, 0x4000, 0, &physical) == 0, "cannot allocate direct memory");
         if (sceKernelMapDirectMemory(&pointer, MappingBytes, 0x33, 0, physical, 0x4000) != 0) {
             sceKernelReleaseDirectMemory(physical, MappingBytes);
 

core/libs/tests/GuestMemory.cpp

@@ -726,6 +726,10 @@ static void CheckReleaseDirectMemoryClearsMappings() {
     }
 }
 
+static void CheckDirectMemorySize() {
+    Require(sceKernelGetDirectMemorySize() == 0x300000000ull);
+}
+
 static void CheckReleaseDirectMemoryRejectsInvalidRanges() {
     constexpr std::size_t page = 0x4000;
     Require(sceKernelReleaseDirectMemory(-1, page) == SCE_KERNEL_ERROR_EINVAL);
@@ -3584,6 +3588,7 @@ int main(int argc, char** argv) {
     CheckSavedDirectMemorySurvivesCachePressure();
     CheckReusedDirectMemorySections();
     CheckReleaseDirectMemoryClearsMappings();
+    CheckDirectMemorySize();
     CheckReleaseDirectMemoryRejectsInvalidRanges();
     CheckFixedMappingReplacesPartialOverlap();
     CheckFixedMappingOverHoleInsideAllocation();

docs/dev/debt/libkernel.md

@@ -63,7 +63,7 @@
 - [sceKernelGetModuleList2 and sceKernelGetModuleInfo2](../../../core/libs/prx/libkernel/Module/src/ModuleInfo.cpp) (libkernel) - the title-module variants follow [shadPS4](https://github.com/shadps4-emu/shadPS4/blob/main/src/core/libraries/kernel/process.cpp): `sceKernelGetModuleList2` lists what `sceKernelGetModuleList` lists without system libraries, and `sceKernelGetModuleInfo2` answers `EPERM` for a system library after the `EFAULT`, `EINVAL` and `ESRCH` checks of `sceKernelGetModuleInfo`, as a payload process on PS5 firmware 13.60 got for libkernel's handle, a size of 0, a null info and a bad handle. A system library here is a loaded `.prx` image that is not a relinked title module (`.guest.prx`); the main program and the title's own modules are listed. Which modules the console lists is in the entry above. PPSA34063 imports both and fails to load without them
 - [sceKernelGetLibkernelTextLocation](../../../core/libs/prx/libkernel/Module/src/ModuleInfo.cpp) (libkernel) - returns 0 and the main program's first segment, as [fpPS4](https://github.com/red-prig/fpPS4/blob/04cefd43e6fddd1ab033e7980cd356d14c964905/kernel/ps4_libkernel.pas#L338-L348) does, where a payload process on PS5 firmware 13.60 got 0 with address 0 and size 0, twice, though libkernel's text was at 0x800000000; what a title gets is not measured. A null address or size faulted there and throws here. libkernel.sprx and libkernel_sys.sprx export the function at the same address
 - [sceKernelReleaseFlexibleMemory](../../../core/libs/prx/libkernel/DirectMemory/Export.cpp) (libkernel) - on a PS5 (firmware 13.60) the released range stays mapped, with its protection and contents (64 MiB checked after 0.5 s), and a fixed no-overwrite map over it fails with `ENOMEM`, as here. What it does to the flexible budget was not measured, because a payload process has none (`sceKernelAvailableFlexibleMemorySize` did not change on map or release); here the released bytes return to the budget while the pages stay mapped, and a later map over them charges them again. The host pages behind a released range stay committed until `sceKernelMunmap`, as `madvise` keeps them for every advice; freeing them would lose the contents the console still returns.
-- [sceKernelAvailableFlexibleMemorySize](../../../core/libs/prx/libkernel/DirectMemory/Export.cpp) (libkernel) - the segments of the title's executable and of its `.guest.prx` modules count against the configured flexible-memory budget (448 MiB by default) while loaded, for this size and for flexible maps. A title may declare a larger budget through `kernel.flexibleMemorySize` in `param.json`; this does not reduce the 13.5 GiB direct-memory size. Rejection of zero or a value not aligned to 16 KiB has not been measured on a console. This segment-accounting model follows [shadPS4](https://github.com/shadps4-emu/shadPS4/blob/3a9f57265d28975658f1ca941daa5c879c7a74bd/src/core/module.cpp#L165), which maps every segment of a module outside its system module directory as flexible memory on the PS4. [Counted](../../../core/libs/prx/libkernel/Module/src/ModuleInfo.cpp) are the segments the title's ELF gives permissions, each rounded to 16 KiB pages; not counted are segments without permissions, the relinker's own blocks (on Linux both are the read/write/execute `PT_LOAD`s, plus the header block at file offset 0; on Windows everything outside the `.elf` sections) and the host system libraries. The Linux image carries no marker for the relinker's blocks, so they are told apart by those flags and that offset only: a title segment that is writable and executable, or starts at file offset 0, is not counted. A module whose segments exceed what is left still loads, and the available size then reads 0; shadPS4 fails the segment map instead ([memory.cpp](https://github.com/shadps4-emu/shadPS4/blob/3a9f57265d28975658f1ca941daa5c879c7a74bd/src/core/memory.cpp#L519-L524) returns `EINVAL`, [module.cpp](https://github.com/shadps4-emu/shadPS4/blob/3a9f57265d28975658f1ca941daa5c879c7a74bd/src/core/module.cpp#L168-L169) asserts). Which segments the PS5 charges, its rounding, whether system modules use the title's budget and when an unload releases the charge are not measured. PPSA15508 subtracts the reported size and a 10 MiB module allowance from its own 448 MiB budget at start-up and stops when the result exceeds its 146 MiB code and data limit ([#2883](https://github.com/boykopovar/AnyPS5/issues/2883))
+- [sceKernelAvailableFlexibleMemorySize](../../../core/libs/prx/libkernel/DirectMemory/Export.cpp) (libkernel) - the segments of the title's executable and of its `.guest.prx` modules count against the configured flexible-memory budget (448 MiB by default) while loaded, for this size and for flexible maps. A title may declare a larger budget through `kernel.flexibleMemorySize` in `param.json`; this does not reduce the 12 GiB direct-memory size. Rejection of zero or a value not aligned to 16 KiB has not been measured on a console. This segment-accounting model follows [shadPS4](https://github.com/shadps4-emu/shadPS4/blob/3a9f57265d28975658f1ca941daa5c879c7a74bd/src/core/module.cpp#L165), which maps every segment of a module outside its system module directory as flexible memory on the PS4. [Counted](../../../core/libs/prx/libkernel/Module/src/ModuleInfo.cpp) are the segments the title's ELF gives permissions, each rounded to 16 KiB pages; not counted are segments without permissions, the relinker's own blocks (on Linux both are the read/write/execute `PT_LOAD`s, plus the header block at file offset 0; on Windows everything outside the `.elf` sections) and the host system libraries. The Linux image carries no marker for the relinker's blocks, so they are told apart by those flags and that offset only: a title segment that is writable and executable, or starts at file offset 0, is not counted. A module whose segments exceed what is left still loads, and the available size then reads 0; shadPS4 fails the segment map instead ([memory.cpp](https://github.com/shadps4-emu/shadPS4/blob/3a9f57265d28975658f1ca941daa5c879c7a74bd/src/core/memory.cpp#L519-L524) returns `EINVAL`, [module.cpp](https://github.com/shadps4-emu/shadPS4/blob/3a9f57265d28975658f1ca941daa5c879c7a74bd/src/core/module.cpp#L168-L169) asserts). Which segments the PS5 charges, its rounding, whether system modules use the title's budget and when an unload releases the charge are not measured. PPSA15508 subtracts the reported size and a 10 MiB module allowance from its own 448 MiB budget at start-up and stops when the result exceeds its 146 MiB code and data limit ([#2883](https://github.com/boykopovar/AnyPS5/issues/2883))
 - [sceKernelAvailableDirectMemorySize](../../../core/libs/prx/libkernel/DirectMemory/Export.cpp) (libkernel) - its arguments follow a payload process on a PS5 (firmware 13.60) ([probes](https://gist.github.com/orezraey/42a06a7ea0d9fb57ac9c06eaf9e6756c)). One answer is not modelled: in a process that had not yet called `sceKernelGetDirectMemorySize` or otherwise touched direct memory, the console returned `EBADF` (`0x80020009`); here the call always searches the pool. Titles open the direct-memory device during start-up, so they are not expected to see it
 - [sceKernelReleaseDirectMemory](../../../core/libs/prx/libkernel/DirectMemory/MemoryPool.cpp) (libkernel) - releases the allocated pages of the range and skips pages that are not allocated, returning 0, while `sceKernelCheckedReleaseDirectMemory` returns `ENOENT` for them, as [shadPS4](https://github.com/shadps4-emu/shadPS4/blob/c206f35a546230d9c9808b823c0f36f5850cd4af/src/core/memory.cpp#L295-L313) does; not measured on a console. Absolum (PPSA28311) releases such a range when it disposes of its video logos
 - [Memory protection bits 0x100 and 0x200](../../../core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp) (libkernel) - taken as audio coprocessor (ACP) read and write, as in the [KytyPS5](https://github.com/KytyPS5/KytyPS5) and [etaHEN](https://github.com/etaHEN/etaHEN) headers; the coprocessor work runs on the host CPU, so they grant host read and read/write access like the GPU bits

docs/user/USAGE.md

@@ -244,7 +244,7 @@ For bundled title modules, use the `.guest.prx` files the relinker writes into `
 
 On Windows, a self-built `libs/` also needs `libgcc_s_seh-1.dll`, `libstdc++-6.dll` and `libwinpthread-1.dll` from the `mingw64/bin` directory of the toolchain that built the libraries; the release archives already contain them. The libraries are loaded without searching `PATH`, so a copy elsewhere on the system is not used, and a library that needs one of them fails with error 126 (the specified module could not be found) although its `.prx` file is present. With `APS5_MIMALLOC` set, the title loads `mimalloc.dll` and `mimalloc-redirect.dll` from `libs/` and allocates through mimalloc. Copy both from `build/core/libs/libs`; the release archives contain them. Without them the title keeps the C runtime's heap and prints nothing.
 
-On Windows, direct memory (`sceKernelAllocateDirectMemory`, up to 13824 MiB per title) is committed when the title maps it, not when the title allocates it or first uses its pages. The system commit limit (installed memory plus page file size, the second value of Committed in Task Manager) must cover the mapped direct memory together with all other committed memory. Otherwise `sceKernelMapDirectMemory` throws `replace guest placeholder with resident direct memory` with the Windows error (`commit direct memory pages` when a page is mapped at a second address, `keep unmapped direct memory` when it is unmapped); enlarge the page file or close other applications.
+On Windows, direct memory (`sceKernelAllocateDirectMemory`, up to 12288 MiB per title) is committed when the title maps it, not when the title allocates it or first uses its pages. The system commit limit (installed memory plus page file size, the second value of Committed in Task Manager) must cover the mapped direct memory together with all other committed memory. Otherwise `sceKernelMapDirectMemory` throws `replace guest placeholder with resident direct memory` with the Windows error (`commit direct memory pages` when a page is mapped at a second address, `keep unmapped direct memory` when it is unmapped); enlarge the page file or close other applications.
 
 On Linux, when the Vulkan driver imports dma-buf memory (not the NVIDIA proprietary driver), shared direct memory is imported through `/dev/udmabuf`, and the user who runs the game needs read-write access to it. Many distributions create it as `root:kvm` with mode `0660`: add the user to the `kvm` group and log in again (an ACL such as `setfacl -m u:$USER:rw /dev/udmabuf` lasts until the next reboot). Without access, startup prints `[gpu] open /dev/udmabuf: Permission denied`, these ranges are copied instead of imported, and GPU stores to them through FLAT/GLOBAL addresses fail with `BDA access failed`. A range larger than 64 MiB is imported as one sparse buffer over 64 MiB udmabuf buffers when the device binds sparse memory (RADV, ANV). Without sparse binding, or when the device's sparse block size is larger than 64 KiB, ranges above udmabuf's `size_limit_mb` (64 MiB by default) are copied as well.