red_zone_framer frames two leaves whose 4-byte cmp %rdi,(%rsp) sits
before a jne: after an add (inflate_fast's shape) and after a load below
rsp that heads the loop and moves by the shift inside the stub, and no
operand in a stub is also moved in place. A cmp whose loop branches back
to it stays refused, the widened stub covering the branch target. The
execution test runs both framed leaves under writes below rsp.
Function starts were direct call targets plus 16-byte-aligned code after
int3 padding. A function reached only through a pointer and not preceded
by int3 then ended up inside the previous region, and its pop rbp; ret
exit could get the epilogue lea without a matching prologue.
The bounds now come from the executable's .eh_frame FDEs (the existing
EhFrameReader), each function is decoded from its FDE start to its FDE
end, and code outside every FDE is not framed. Conversion stops with an
error when the executable has no FDE. Sites added in one pass also count
as taken for the next function.
A leaf whose rsp operand could not keep its encoding after the shift
was skipped: an 8-bit displacement that overflows, an operand at (%rsp)
without a displacement, or an indexed (%rsp,%idx,s) operand. Such an
instruction now moves to a stub, re-encoded with the moved displacement
in 8 bits when it fits and in 32 bits otherwise, followed by a jump
back.
The site covers the instruction and the sequential instructions after
it until it is 5 bytes long. It is refused when a branch lands inside
it, when it would have to cover a branch, ret or rip-relative
instruction (these are not relocated), or when it overlaps the
prologue, an exit, another operand site or an existing stub. Legacy,
REX and VEX forms are re-encoded; EVEX operands, whose 8-bit
displacement is scaled, are still refused. A leaf with no room for a
site is skipped as "rsp operand that cannot move and has no room for a
stub". Operands inside a site that could move in place are carried by
the stub instead, so the site's bytes stay as they were.
The report counts the operand stubs and the operands they carry.
The framer's tests check its sites, stub bytes and moved displacements,
but nothing yet showed that framed code survives a write below rsp.
red_zone_framer_execution frames synthetic leaves with
FrameRedZoneFunctions in a writable and executable buffer, writes the
moved displacements and the stubs as Conversion.cpp and the trampoline
builders do, and calls each leaf with the trap flag set. A SIGTRAP
handler fills [rsp-0x80, rsp) of the interrupted context at every
instruction, as Windows exception dispatch may.
Five leaves each loop over slots they spilled, so a clobbered slot
changes the result: #5640's frameless shape, a leaf reaching 0x28 below
rsp, a leaf that also reads its saved rbx at +0x10 (moved in place), a
leaf that takes the address of a slot below rsp (shifted by the whole red
zone), and the inflate_fast-shaped leaf that allocates with sub after six
pushes. Each must return the right value unframed without the clobber
and a wrong one with it, and the right value framed with it.
The test is built and registered on Linux x86-64 only; elsewhere its main
returns 77.
Yunus Shaikh