A chain of table bytes that decodes to a return after a padding run looked
like trailing gap code and stopped the mask inside a real table, which hands
the rest of it back to the decoder. Truncating now needs the block to contain
a syscall, or to end where the gap turns into padding, so a 2 KiB table with
00/cc/90 runs and return-like bytes stays masked.
A gap that holds a table, padding and then a syscall stub followed by
non-padding data classified the stub as invalid, so the mask ran to the end
of the gap and hid the syscall from the scan. Mark any instruction that ends
control flow as a terminator, which stops the mask before that block.
Covered by the helper test and by the SCE, SysV and guest-module conversion
tests, which convert before the change and refuse syscall 9 after it.
Every pending gap target walked the whole unwind function vector, so a
large executable cost targets x functions. Sort and merge the ranges once,
then use upper_bound for each target.
trailing[next] is Padding for an instruction that starts with 00 or CC
even when the rest of its bytes are code, so it is not the same as
padding[next]. The recurrence treated the two as interchangeable and
overwrote the trailing state of a syscall that precedes such an
instruction, which let the fallback extend the mask over it and made
both conversion routes accept the gap. Decide from the trailing state
whether the successor is padding.
The regression covers the table helper plus the executable and guest
module routes; without the correction the executable and module
conversions succeed instead of refusing syscall 9.
A 2 MiB text with 32 gaps of 64 KiB, the largest gap the analysis
follows, fails the fixture if the padding, trailing-code and syscall
queries go back to a quadratic walk, and asserts the bounded table end.
Read the unwind functions of an image that only has PT_GNU_EH_FRAME so
inter-FDE data tables are still found and masked, but leave the syscall
and call-site scan ranges empty without PT_SCE_DYNLIBDATA as main does.
The fixture now emits standard dynamic tags for its SysV block, and the
debt entry stays within the 1000 character limit.
In PS5 Prospero PRX binaries, compilers place embedded constant data tables (such as the OpenSSL Whirlpool lookup table in ubiservices.prx) in inter-FDE gaps after function code, referenced via RIP-relative LEA instructions.
- In RelativeJumpTables, detect constant data tables in inter-FDE gaps that begin with padding, contain no direct branch targets, and contain undecodable non-code bytes, distinguishing them from valid unannotated code routines and multi-hop gap branches.
- In GuestModuleBuilder and RelinkerPipeline, extract unwind functions from PT_GNU_EH_FRAME as well as PT_SCE_DYNLIBDATA, and mask embedded data tables with 0x90 in scan buffers before SyscallScanner.
- In CodeInstructionCollector, populate table boundaries whenever unwind functions are present to prevent addAddressTakenRoot from adding data table addresses to CFG roots on Windows and macOS.
- Update relinker technical debt documentation and add regression tests.
Fixes#7547
Unblocks: PPSA28155 #7547
Dean