Merge pull request #11156 from orezraey/fix/libc-fdopen-measured-ps5

fix(libc): check the fdopen mode before the descriptor and skip the access check as measured on a PS5

docs/dev/debt/libc.md

@@ -11,7 +11,7 @@
 
 - [sceLibcBacktraceGetBufferSize, sceLibcBacktraceSelf](../../../core/libs/prx/libc/src/Backtrace.cpp) (libc) - signatures and frame record layout inferred from a title (next pointer, u32 module offset, module name at +0x10); the u32 at +0xC is unknown and written as 0. Module names are host file names without the `.guest.prx` suffix, so their case may differ from the PS5 module name. On Windows, unwinding uses `_Unwind_Backtrace` and stops at a frame without `.ehmeta` or `.ehfram` tables whose PE unwind data has an exception or termination handler, leaving the trace empty or cut short while still returning 0. Implemented only on Linux and Windows; other platforms throw
 - [ApplicationGuestHeapApi](../../../core/libs/prx/libc/src/ApplicationProcessParameters.cpp) (libc) - on Linux, HLE application allocation uses a loaded guest SDK's ten allocator exports when the title's replacement table has an allocator entry and the exports belong to one native module. With an empty table it keeps the default heap, since a bundled `libc.prx` may export the allocator NIDs while forwarding them to HLE. Its CRT owns application allocator initialization and finalization; HLE retains the module until process exit. Partial or mixed providers throw. This avoids reinitializing an allocator already used by SDK startup; console behavior and other host platforms remain unverified.
-- [fdopen](../../../core/libs/prx/libc/src/FileAndHeap.cpp) (libc) validates descriptor access modes as FreeBSD 11.4 [fdopen.c](https://github.com/freebsd/freebsd-src/blob/8c639945a44588f6cfdddfb057bb5aad88735da0/lib/libc/stdio/fdopen.c) does: a requested mode incompatible with a descriptor not opened for both reading and writing returns null with `EINVAL`, leaving the descriptor open. Windows reads the handle's granted access through the same host helper as `fcntl(F_GETFL)`; a closed descriptor returns null with `EBADF`. Console behavior is unverified.
+- [fdopen](../../../core/libs/prx/libc/src/FileAndHeap.cpp) (libc) - a closed or unknown descriptor returns null with `EBADF`, where the system `libSceLibcInternal` of PS5 firmware 13.60 returns a stream whose later flush and close fail with `EBADF`; a null mode returns null with `EINVAL`, where the console returns a stream; on Linux a mode the descriptor's access does not allow returns null with `EINVAL`, as the host `fdopen` checks it. Otherwise it follows the console: a mode whose first letter is not `r`, `w` or `a` fails with `EINVAL` before the descriptor is looked at, a negative descriptor then returns null without setting `errno`, and the descriptor's access mode is not checked, so the first read or write it does not allow fails instead
 - [scriptingGetMem, scriptingFreeMem](../../../core/libs/prx/libc/src/ApplicationHeap.cpp) (libc) - runtime allocator hooks for Unity titles. The allocation signature `(alignment, size)`, minimum alignment 16 and null result for a non-power-of-two alignment follow [KytyPS5](https://github.com/KytyPS5/KytyPS5/blob/df5b9b92be7dfd463b47756b707d6d356a9f0550/src/libs/libKernel.cpp); the free hook's one-pointer signature follows [Raeen's recorded contract](https://github.com/Whoisraeen/Raeen/blob/2db56923204b89719916b95e29c2f3c4e1d9f90b/docs/unity-dlsym-blocker.md). Both use libc's own guest heap, not the title's registered allocator. A title that ships its own `libc.prx` (Metal Gear Solid Master Collection, PPSA18194) has already initialized that allocator, and reaching it from here would run its initializer a second time. That title's IL2CPP runtime looks up only `scriptingGetMem`, by name from `Il2cppUserAssemblies.prx`, and asks it for 256 KiB blocks aligned to 4 KiB. Which console library exports the hooks is unknown, and there is no console measurement. Allocation failure and sizes that overflow return null.
 
 - [_Mtx_trylock](../../../core/libs/prx/libc/src/Threads.cpp) (libc, NID `k6pGNMwJB08`) - returns busy for a non-recursive mutex already owned by the caller. The `libc.prx` that titles ship (read from disassembly of PPSA13934's copy, 0x5cf0, not measured on a console) passes its argument to `scePthreadMutexTrylock` and returns 0 for success, `_Thrd_busy` (3) for `SCE_KERNEL_ERROR_EBUSY` and `_Thrd_error` (4) for any other error, so busy for an owned non-recursive mutex comes from the kernel mutex there; this libc keeps its own owner and count instead, and its result for a kernel error other than busy is not compared.