red_zone_framer frames two leaves whose 4-byte cmp %rdi,(%rsp) sits
before a jne: after an add (inflate_fast's shape) and after a load below
rsp that heads the loop and moves by the shift inside the stub, and no
operand in a stub is also moved in place. A cmp whose loop branches back
to it stays refused, the widened stub covering the branch target. The
execution test runs both framed leaves under writes below rsp.
When the instruction after an rsp operand that cannot move in place is
a branch, a rip-relative instruction or a freeing add, the operand's stub
could stay shorter than a jump and the leaf was skipped. The stub now
also takes the contiguous sequential instructions before the operand,
moving their own rsp operands by the shift; the branch-target, site and
existing-stub checks run on the widened range, and the instructions it
covers leave the in-place moves. zlib's inflate_fast, whose only operand
left is a 4-byte cmp %rdi,(%rsp) between an add and a jne, frames this
way.
The backward extension is tibzejoker's, from their comment on
boykopovar/AnyPS5#5640: with it, 243 of Grand Theft Auto V's 991
red-zone leaves frame (240 before), inflate_fast among them, and one
leaf is left with no room for a stub.
red_zone_framer's leaf whose operand at rsp sits before a rip-relative
load now frames; the one before an exit stays refused, its forward
extension reaching the exit's pops.
Co-authored-by: tibzejoker <[email protected]>
Function starts were direct call targets plus 16-byte-aligned code after
int3 padding. A function reached only through a pointer and not preceded
by int3 then ended up inside the previous region, and its pop rbp; ret
exit could get the epilogue lea without a matching prologue.
The bounds now come from the executable's .eh_frame FDEs (the existing
EhFrameReader), each function is decoded from its FDE start to its FDE
end, and code outside every FDE is not framed. Conversion stops with an
error when the executable has no FDE. Sites added in one pass also count
as taken for the next function.
A leaf whose rsp operand could not keep its encoding after the shift
was skipped: an 8-bit displacement that overflows, an operand at (%rsp)
without a displacement, or an indexed (%rsp,%idx,s) operand. Such an
instruction now moves to a stub, re-encoded with the moved displacement
in 8 bits when it fits and in 32 bits otherwise, followed by a jump
back.
The site covers the instruction and the sequential instructions after
it until it is 5 bytes long. It is refused when a branch lands inside
it, when it would have to cover a branch, ret or rip-relative
instruction (these are not relocated), or when it overlaps the
prologue, an exit, another operand site or an existing stub. Legacy,
REX and VEX forms are re-encoded; EVEX operands, whose 8-bit
displacement is scaled, are still refused. A leaf with no room for a
site is skipped as "rsp operand that cannot move and has no room for a
stub". Operands inside a site that could move in place are carried by
the stub instead, so the site's bytes stay as they were.
The report counts the operand stubs and the operands they carry.
A leaf's body ran from its start to the next known start or int3 padding,
so a function reached only through a pointer, placed right after the leaf
without int3 padding, was framed as part of the leaf: its rsp operands were
moved and its exits raised rsp although nothing had lowered it.
The body now also ends at 16-byte-aligned code after a ret or ud2, and at
the target of a jmp when only padding lies between them (a tail call to the
next function). Code past such a point is framed only from a real start.
A leaf whose code continues past it is skipped as branching out of the
function. Aligned code after any other jmp stays in the body: loop headers
are aligned there, and cutting after every jmp skipped 2,129 of 2,131
framed leaves in PPSA01556. With this cut 2,132 are framed, and none of
them has rewritten bytes in another function of the unwind table.
Windows writes an exception frame directly below RSP, so a fault resumed on
a guest thread overwrites values a System V leaf function keeps in its red
zone. With --frame-red-zone, a leaf function with a
push rbp; mov rbp, rsp; push ... prologue that reads or writes below its
pushes gets two kinds of trampoline: the prologue lowers RSP by 128 bytes
after its pushes, and each pop ...; pop rbp; ret exit raises it back first.
Sites a branch enters, sites shorter than a jump and sites over a --to-intel
stub are skipped; the conversion reports the framed and skipped counts.
Yunus Shaikh