An island or relocation window could start in the int3 or nop padding after a jump or return and run into the next function. A function that only a pointer reaches, such as a virtual method, is not a known branch target, so its first bytes became the middle of a jump to a stub. An instruction that follows a jump, a return, a trap or padding after one of them can now only start a window. An int3 inside a window does not count as such an instruction, so a window that lies entirely in int3 padding is still used, as before; nop padding stays excluded, since a function built with -fpatchable-function-entry can begin with a nop.
Operand-size-prefixed (0x66) relative branches are classified as having
no branch target, so the AdjustedDelta check never saw them. The
truncated-branch guard only checked whether the branch and its target
move apart, so `66 EB 01 B8 11 22 33 44 C3` rewritten at offset 3 was
accepted and the branch landed in the middle of the new bytes.
Reject a truncated branch whose target is inside the cut, and run that
check for same-length rewrites too, matching the rel8/rel32 path.
In a leaf without a frame pointer, --frame-red-zone lowers RSP after
the prologue and adds the shift to every RSP operand in the body,
assuming RSP does not move between the prologue and the exits. It
refused explicit RSP arithmetic but not opcode-encoded pushes and pops,
so after "push rax" in the body the read of the pushed slot at [rsp]
became [rsp+0x10], a red-zone slot, and "pushfq; and qword [rsp], -2;
popfq" changed that slot instead of the saved flags. Such a leaf now
keeps its red zone and is reported as "push or pop inside the body":
push and pop of a register, push imm, push r/m, pop r/m, pushf, popf
and push or pop of fs or gs outside the exits. The pops of an exit
still frame, and leaves with a frame pointer, whose RSP operands are
already refused, still frame when they push and pop.
extensionBits and memoryUse in the red-zone framer took every 8F lead
byte without REX for an XOP prefix. For pop r/m (8F /0) that read the
ModRM byte as XOP's inverted R, X and B bits, so the base of
"pop qword [rsp-0x10]" looked like R12, and "pop rsp" through 8F was
not seen as a use of RSP. A leaf with a frame pointer that pops to an
RSP operand was framed with the operand left in place, and leaves that
pop into RSP through 8F were framed, while the REX.W forms were refused
as other use of rsp. Both now take 8F for XOP only when the map field
of the next byte is 8 or more, the rule the instruction decoder
applies.
The check that refuses leaves without pushes that push or pop also refused
`push rbp; mov rbp, rsp` leaves without other pushes, whose `pop rbp` it
counted, since rbp is not in their list of pushes. A unit test and an
execution test now cover such a leaf with a 4-byte prologue and a 2-byte
exit. The debt entry also notes that the check refuses shrink-wrapped leaves
whose later pushes and pops balance.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--frame-red-zone replaces a leaf's prologue and each exit with a 5-byte jump
to a stub, so it skipped leaves whose pushes or pop-and-ret exit were shorter
than that, and leaves with neither pushes nor an allocation. A short prologue
now takes in the straight-line instructions after it and a short exit those
before it, re-encoded in the stub with the moved displacement. Widening stops
at branches, rets, RIP-relative instructions, pushes and pops, and the leaf is
skipped when a branch targets the instructions taken in or the first
instruction of a widened prologue, or when a leaf without pushes pushes or
pops.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Branch target collection added the branch displacement to the next
instruction address as std::int64_t, which is undefined behaviour for
a segment mapped at or above 2^63 (found by fuzzing with UBSan, see
#10653). The addresses now add as unsigned values and only targets
that wrap past 2^64 are dropped, so the branch-target check also
applies to segments mapped across 2^63.
Fixes#10653
Dean