AnyPS5/docs/dev/HW_ORACLE.md

Hardware oracle

tools/hw-oracle/ runs a few lines of RDNA assembly on a local AMD GPU, one input row per lane, and returns what the hardware computed. Use it to settle instruction semantics that the ISA documentation, LLVM or Mesa leave open, then pin the measured rows as expected values in core/libs/prx/libSceAgcDriver/tests/execution/.

The PS5 GPU is gfx1013. Other RDNA2 GPUs (gfx103x) share the VALU, SALU, DS, MUBUF and MIMG encodings, apart from ops that exist only on gfx10.1 (v_mad_legacy_f32, v_mac_legacy_f32, v_mul_lo_i32, s_dcache_discard, ...). RDNA3 also runs the tool, but differs from gfx1013 in more places. Say which GPU you measured on in the PR.

Requirements

Linux

• Linux with the amdgpu driver and read/write access to /dev/kfd (usually the render group). Root is not needed.
• The ROCm HSA runtime with its headers: libhsa-runtime-dev (Debian, Ubuntu), rocm-runtime-devel (Fedora), hsa-rocr (Arch), or AMD's ROCm packages. A runtime under ROCM_PATH or /opt/rocm is used first. The rest of ROCm is not needed.
• A C compiler, and clang + ld.lld with the AMDGPU target (llc --version lists amdgcn). If they aren't in PATH, they are taken from $ROCM_PATH/llvm/bin.

The first run builds oracle.c into $XDG_CACHE_HOME/anyps5-hw-oracle/, or ~/.cache/anyps5-hw-oracle/ when XDG_CACHE_HOME is unset or empty (HW_ORACLE_CACHE overrides both). The target is read from the GPU (HW_ORACLE_TARGET overrides it).

Windows

• AMD's Windows driver, which installs the HIP runtime (amdhip64_7.dll, amdhip64_6.dll or amdhip64.dll) and the code object manager (amd_comgr_3.dll, amd_comgr_2.dll or amd_comgr.dll) in System32. ROCm, a C compiler, clang and ld.lld are not needed.
• Python 3 from python.org or the Microsoft Store.

On Windows oracle_hip.py replaces oracle.c: comgr assembles and links the kernel, and HIP runs it. The CLI, the Python interface and the results are the same as on Linux. HW_ORACLE_RUNTIME picks the runtime (hip on Windows, hsa elsewhere; vulkan is never the default, see Recompiler comparison). HW_ORACLE_HIP and HW_ORACLE_COMGR name other DLLs. ROCR_VISIBLE_DEVICES selects the GPU, through HIP_VISIBLE_DEVICES, unless that variable is already set.

Kernel

template.s wraps the body under test. Per lane:

• v4-v7 hold the row's 4 input dwords. v10-v25 start at 0 and are stored as the result (16 dwords).
• Leave v0 (lane id), v1 (input offset), v2 (output offset) and s[4:7] (input and output addresses) unchanged.
• 4 KiB of LDS, or the group segment size --lds (Python lds) gives in bytes, up to 64 KiB. Rows run in workgroups of up to 1024 lanes, padded to whole waves with zero rows.
• Bytes passed as extra follow the rows of each dispatch, at s[4:5] + 16 * rows: buffer contents, texels, etc. Build buffer and image descriptors in SGPRs from that address. A kernel that needs more than 4 input dwords per lane takes the rest from there too, for example v_lshlrev_b32 v40, 3, v0 / v_add_nc_u32 v40, 16 * rows, v40 / global_load_dwordx2 v[8:9], v40, s[4:5] for two more dwords per lane, with rows the padded count.

Every run must explicitly set all floating-point controls, including runs of integer instructions. The CLI flags use hyphens; Python keywords use underscores. Record these settings alongside the GPU and measured results.

Python keyword	Values
ieee	0 or 1: IEEE mode disabled or enabled
dx10_clamp	0 or 1: DX10 clamp disabled or enabled
denorm32, denorm16	0: flush input/output; 1: preserve input, flush output; 2: flush input, preserve output; 3: preserve both
round32, round16	0: nearest even; 1: toward +infinity; 2: toward -infinity; 3: toward zero
fp16_overflow	0: overflow to infinity; 1: clamp computed overflow to the largest finite value (an infinite source operand still produces infinity, as does v_div_fixup_f16 for a zero denominator or an infinite numerator, but not for an infinite quotient; v_rcp_f16, v_rsq_f16 and v_log_f16 of zero are clamped)

denorm16 and round16 control both f16 and f64. These values follow the LLVM AMDGPU kernel descriptor documentation. Float atomics need coarse: on fine-grained system memory they do nothing.

Use

cat > body.s <<'EOF'
  v_add_f16 v10, v4, v5
EOF
printf '0x3c00 0x3c00 0 0\n0x7e01 0x3c00 0 0\n' > rows.txt
python3 tools/hw-oracle/hw_oracle.py body.s rows.txt --outs 1 \
  --ieee 0 --dx10-clamp 1 --denorm32 0 --denorm16 3 \
  --round32 0 --round16 0 --fp16-overflow 0

prints one line per row with the output dwords in hex. From Python:

import sys
sys.path.insert(0, "tools/hw-oracle")
from hw_oracle import run

rows = [(a, b, 0, 0) for a in values for b in values]
results = run(body, rows, wave64=True, ieee=0, dx10_clamp=1,
              denorm32=0, denorm16=3, round32=0, round16=0, fp16_overflow=0)

Usual workflow: run the instruction on rows that separate the candidate models (edge values, NaN payloads, rounding ties), write a model that matches every row, implement it, and put measured rows in the execution test.

Recompiler comparison

The vulkan runtime runs the same assembled kernel through the shader recompiler on the Vulkan device instead of on AMD hardware, so one body and one set of rows can be measured on both and the results compared. Build the runner:

cmake --build build --target agc_oracle_runner

Variable	Meaning
HW_ORACLE_RUNTIME=vulkan	selects the runtime
HW_ORACLE_VULKAN_RUNNER	path of the built agc_oracle_runner executable
ANYPS5_GPU	case-insensitive substring of the Vulkan device name, as for every AnyPS5 tool

On Windows the runner needs build/core/libs/libs/unpatched in PATH (BUILD). Run the hardware with the default runtime and the recompiler with vulkan, with the same body, rows and settings, each with --json, then compare:

SETTINGS="--ieee 0 --dx10-clamp 1 --denorm32 0 --denorm16 3 --round32 0 --round16 0 --fp16-overflow 0"
python3 tools/hw-oracle/hw_oracle.py body.s rows.txt --json $SETTINGS > hardware.json
HW_ORACLE_RUNTIME=vulkan HW_ORACLE_VULKAN_RUNNER=<path to agc_oracle_runner> \
  python3 tools/hw-oracle/hw_oracle.py body.s rows.txt --json $SETTINGS > recompiler.json
python3 tools/hw-oracle/compare_results.py hardware.json recompiler.json

The report's runtime field says which side produced it.

• Compute only.
• Whole waves: Python pads the rows with zeros to the wave size, so cross-lane instructions see the padded lanes.
• At most 1024 lanes per dispatch.
• No coarse.
• The assembly target comes from the AMD driver on Windows or from HW_ORACLE_TARGET; the recompiler has no target of its own, and the encoding assembled is the RDNA2 one.

Limits

Only compute kernels. Pixel-shader ops that read their inputs from LDS also work: v_interp_* reads the attribute parameters from LDS at M0, so writing them there by hand is enough. Behaviour that depends on the graphics pipeline (parameter cache, exports, rasterization, SPI_* state) can't be measured with it.

Argument handling can be tested without a GPU:

python3 tools/hw-oracle/test_hw_oracle.py