In a leaf without a frame pointer, --frame-red-zone lowers RSP after
the prologue and adds the shift to every RSP operand in the body,
assuming RSP does not move between the prologue and the exits. It
refused explicit RSP arithmetic but not opcode-encoded pushes and pops,
so after "push rax" in the body the read of the pushed slot at [rsp]
became [rsp+0x10], a red-zone slot, and "pushfq; and qword [rsp], -2;
popfq" changed that slot instead of the saved flags. Such a leaf now
keeps its red zone and is reported as "push or pop inside the body":
push and pop of a register, push imm, push r/m, pop r/m, pushf, popf
and push or pop of fs or gs outside the exits. The pops of an exit
still frame, and leaves with a frame pointer, whose RSP operands are
already refused, still frame when they push and pop.
extensionBits and memoryUse in the red-zone framer took every 8F lead
byte without REX for an XOP prefix. For pop r/m (8F /0) that read the
ModRM byte as XOP's inverted R, X and B bits, so the base of
"pop qword [rsp-0x10]" looked like R12, and "pop rsp" through 8F was
not seen as a use of RSP. A leaf with a frame pointer that pops to an
RSP operand was framed with the operand left in place, and leaves that
pop into RSP through 8F were framed, while the REX.W forms were refused
as other use of rsp. Both now take 8F for XOP only when the map field
of the next byte is 8 or more, the rule the instruction decoder
applies.
The check that refuses leaves without pushes that push or pop also refused
`push rbp; mov rbp, rsp` leaves without other pushes, whose `pop rbp` it
counted, since rbp is not in their list of pushes. A unit test and an
execution test now cover such a leaf with a 4-byte prologue and a 2-byte
exit. The debt entry also notes that the check refuses shrink-wrapped leaves
whose later pushes and pops balance.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--frame-red-zone replaces a leaf's prologue and each exit with a 5-byte jump
to a stub, so it skipped leaves whose pushes or pop-and-ret exit were shorter
than that, and leaves with neither pushes nor an allocation. A short prologue
now takes in the straight-line instructions after it and a short exit those
before it, re-encoded in the stub with the moved displacement. Widening stops
at branches, rets, RIP-relative instructions, pushes and pops, and the leaf is
skipped when a branch targets the instructions taken in or the first
instruction of a widened prologue, or when a leaf without pushes pushes or
pops.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Branch target collection added the branch displacement to the next
instruction address as std::int64_t, which is undefined behaviour for
a segment mapped at or above 2^63 (found by fuzzing with UBSan, see
#10653). The addresses now add as unsigned values and only targets
that wrap past 2^64 are dropped, so the branch-target check also
applies to segments mapped across 2^63.
Fixes#10653
XSAVEOPT raises an illegal instruction under Rosetta on an M5 running
macOS 27.0.1, as XSAVEC does, while XSAVE and XRSTOR save and restore
x87, SSE and AVX state in the standard format. The PS5's Zen 2 has
XSAVEOPT, so code built for it can use the instruction without a CPUID
check.
XSAVEOPT differs from XSAVE only in the init and modified optimizations:
it may skip writing state that is in its initial configuration or
unchanged since the last XRSTOR from the same area. XRSTOR restores the
same state from the area XSAVE writes, so --to-rosetta rewrites
XSAVEOPT (NP 0F AE /6, memory operand) to XSAVE (/4) in place, keeping
the prefixes and the operand, as RDSEED becomes RDRAND. CLWB (66 0F AE
/6), MFENCE, UMONITOR, UMWAIT and the Intel target are left alone.
XSAVEC is not converted: it writes the compacted format.
The converter tests check the rewrite for the plain, REX.W,
RSP-relative, extended-base, RIP-relative and FS-relative forms, check
that the neighbouring 0F AE encodings are not taken, and execute the
rewritten XSAVEOPT64 followed by XRSTOR on YMM3 in the x86-64 build.
The --to-rosetta integration test now covers XSAVEOPT as well.
When the instruction after an rsp operand that cannot move in place is
a branch, a rip-relative instruction or a freeing add, the operand's stub
could stay shorter than a jump and the leaf was skipped. The stub now
also takes the contiguous sequential instructions before the operand,
moving their own rsp operands by the shift; the branch-target, site and
existing-stub checks run on the widened range, and the instructions it
covers leave the in-place moves. zlib's inflate_fast, whose only operand
left is a 4-byte cmp %rdi,(%rsp) between an add and a jne, frames this
way.
The backward extension is tibzejoker's, from their comment on
boykopovar/AnyPS5#5640: with it, 243 of Grand Theft Auto V's 991
red-zone leaves frame (240 before), inflate_fast among them, and one
leaf is left with no room for a stub.
red_zone_framer's leaf whose operand at rsp sits before a rip-relative
load now frames; the one before an exit stays refused, its forward
extension reaching the exit's pops.
Co-authored-by: tibzejoker <[email protected]>
A 4-byte 256-bit store at the start of a segment followed by an AMD-only
site that can only be emitted in place (RDSEED, CLWB, MOVNTSS, ...) had no
preceding instruction to move, so WindowPlacer gave up with "No room for a
jump to the stub of 256-bit store within its neighbours or a nearby island".
Try a last-resort window placement that allows EmitOutOfLine to use the
in-place lowering for the follower. Preferred preceding and following moves
are unchanged.
Function starts were direct call targets plus 16-byte-aligned code after
int3 padding. A function reached only through a pointer and not preceded
by int3 then ended up inside the previous region, and its pop rbp; ret
exit could get the epilogue lea without a matching prologue.
The bounds now come from the executable's .eh_frame FDEs (the existing
EhFrameReader), each function is decoded from its FDE start to its FDE
end, and code outside every FDE is not framed. Conversion stops with an
error when the executable has no FDE. Sites added in one pass also count
as taken for the next function.
A leaf whose rsp operand could not keep its encoding after the shift
was skipped: an 8-bit displacement that overflows, an operand at (%rsp)
without a displacement, or an indexed (%rsp,%idx,s) operand. Such an
instruction now moves to a stub, re-encoded with the moved displacement
in 8 bits when it fits and in 32 bits otherwise, followed by a jump
back.
The site covers the instruction and the sequential instructions after
it until it is 5 bytes long. It is refused when a branch lands inside
it, when it would have to cover a branch, ret or rip-relative
instruction (these are not relocated), or when it overlaps the
prologue, an exit, another operand site or an existing stub. Legacy,
REX and VEX forms are re-encoded; EVEX operands, whose 8-bit
displacement is scaled, are still refused. A leaf with no room for a
site is skipped as "rsp operand that cannot move and has no room for a
stub". Operands inside a site that could move in place are carried by
the stub instead, so the site's bytes stay as they were.
The report counts the operand stubs and the operands they carry.
LEA computes an effective address and the multi-byte NOP ignores its operand, so neither accesses memory and a GS prefix on them is ignored. The Windows TLS builder and macOS image writer treated every ModRM memory form as an access and replaced these instructions with UD2 and NOP padding.
The skip of syscall patterns outside the unwind functions let some real
syscalls through. The scan now merges nested and overlapping unwind
ranges with the instruction scanner's MergeCodeRanges, so a pattern
inside an outer function is refused. A constant syscall number is also
looked for in the bytes in front of the pattern, whatever alignment the
decoding took: a mov eax, mov rax or xor eax, eax that ends at the
pattern or before register moves and zeroings that leave rax alone. The
decoded lookback requires each instruction to end where the next one
starts. Guest modules without PT_SCE_DYNLIBDATA get no unwind ranges,
so their scan stays strict even when their unwind table is read.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
CLZERO (0F 01 FC) clears the cache line at the logical address in rAX and applies a GS override, so mark its register-form encoding as a memory operand. RDPRU (0F 01 FD) has no memory operand and stays unchanged.
UMONITOR (F3 0F AE /6) takes its effective address from the register operand and uses DS with segment overrides, so mark its register-form encoding as a memory operand. UMWAIT and TPAUSE use the same opcode with F2/66 and have no memory operand.
MASKMOVQ, MASKMOVDQU, VMASKMOVDQU, MONITOR and MONITORX use a register-form ModRM byte but still access memory at DS:rDI or DS:rAX. Mark them as memory operands and cover the encodings in the Windows and macOS GS trap tests.
Rosetta ends the process instead of raising SIGSEGV or SIGBUS when a
VMASKMOVPS, VMASKMOVPD, VPMASKMOVD or VPMASKMOVQ store faults on any
element other than an active first element ("rosetta error: unexpectedly
need to EmulateForward on a synchronous exception"). Measured on an M5
with macOS 27.0.1 for all eight 128- and 256-bit store forms. A masked
store into a page the macOS write watch keeps read-only therefore ends
the title.
--to-rosetta now moves each masked store to a stub that saves RAX, RCX
and RFLAGS below the red zone, computes the operand address with LEA,
reads the mask sign bits with VMOVMSKPS or VMOVMSKPD, and writes to the
first and last byte of every active element with LOCK OR of 0. Those
writes fault normally, so the write watch sees the page and unprotects
it, and they leave memory unchanged. The stub then restores the
registers and runs the original store. Segment and address-size
prefixes are kept on the probes, RSP-based operands skip the spill
frame, and RIP-relative operands are relocated. Masked loads, the
Intel target and a plain relink are unchanged.
masked_store_lowering decodes every form, register and extension,
checks the stub bytes, and on x86-64 Linux and macOS runs 1728 stubs
across all forms, six addressing modes, six offsets around a read-only
page and six mask patterns. It checks the stored bytes, that exactly one
fault reaches the handler when an active element touches the page,
that no register, flag or red-zone byte changes, and that RSP is
restored. Without the probes the x86-64 build aborts under Rosetta.
test_to_rosetta.py checks that only --to-rosetta lowers a VMASKMOVPS.
Dean