diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 00000000..0c0c1c74 --- /dev/null +++ b/tools/README.md @@ -0,0 +1,20 @@ +# Tools + +The top-level scripts here are the repository's own tooling: setup and asset preparation (`setup.py`, +`prepare_assets.py`, `asset_pipeline/`, `owned_game/`), the updater, release and build helpers, and their +tests. They are documented where they are used (the main README and the scripts' own help). + +The folders below are standalone helper tools for development and research. Each has a `README.md` +with what it does, its inputs, usage, example output and requirements. They read your own copy of the +game; none of them contain game code or data. Default work folders are under `.local/` (gitignored). + +| Folder | What it is for | +|---|---| +| [`regression-checks/`](regression-checks/README.md) | Map validation and collision counts, baked spawns, customiser outputs against your own baselines; a muted crash smoke test per map. | +| [`setup-equivalence/`](setup-equivalence/README.md) | Prove a faster setup path gives identical bytes (district stream loader, native RefPack DLL). | +| [`collision-inspect/`](collision-inspect/README.md) | Extract a district's collision for analysis: triangles, surfaces at a point, surface ids and flags, surfaceless meshes. | +| [`world-stream-inspect/`](world-stream-inspect/README.md) | List / extract `.big` archives; survey the RW4 arenas of the district simulation streams; dump named trigger volumes. | +| [`vault-inspect/`](vault-inspect/README.md) | Look up fields in the attribute database (converted skater collections) and class layouts in the schema. | +| [`recomp-code-search/`](recomp-code-search/README.md) | **For use with the Skate 3 recomp's research hooks** ([`research-hooks` branch](https://github.com/Hailey-Ross/skate3recomp/tree/research-hooks)): search the recompiled sources and the memory image. Reference only; you build the recomp and set up the paths yourself. | + +`regression-checks/check_maps.py` uses the game's `--validate-maps` mode when the exe has it and falls back to `--check-assets` per map otherwise. diff --git a/tools/collision-inspect/README.md b/tools/collision-inspect/README.md new file mode 100644 index 00000000..f4f2a099 --- /dev/null +++ b/tools/collision-inspect/README.md @@ -0,0 +1,48 @@ +# Collision inspection + +Extract a district's collision from your own disc and look at it, without touching the installation. +Used to place map spawns and to find the invisible trigger volumes (see +`docs/hails-additions/04-map-spawns.md` and `05-collision-volumes.md`). + +| Script | What it does | +|---|---| +| `map_collision.py` | Extracts `world.big` into `//raw` (once), runs the map converter's collision pass and saves every triangle to `collision.npy` (N x 3 x 3 float64, Y-up metres). Prints the triangle count, bounds and the spawn the current spawn rule would choose. | +| `analyse_spawn.py` | Lists every collision surface at a point's X/Z (height, normal-up sign, area), plus the area-weighted centre of the flat surfaces. Nothing below a spawn = void; a much lower surface under it = roof or platform. Needs `map_collision.py` first. | +| `collision_attrs.py` | Saves the triangles with their surface id, unit flags (0x80 = has a surface id), mesh flags and mesh index to `collision_attrs.npy`. Needs the extracted district. | +| `scan_surfaceless.py` | For each district: counts collision meshes that are surfaced, mixed or surfaceless (no surface ids at all) and lists the non-surfaced ones with their size and bounds. | +| `unsigned_collision.py` | Imported by the others: makes the vendored collision decoder read 16-bit vertex deltas unsigned, as the game does. Without it, clusters spanning more than 32.8 m wrap by 65.536 m. | + +## Inputs + +- Your extracted disc: `--disc DIR` (the folder holding `data/`), default `$SKATE3_DISC` or `.local/skate3-disc`. +- District names without the `DIST_` prefix: `MegaPark`, `StartPark`, `BlackBoxPark`, `SkateSchool`, + `IndustrialSkatePark`, `DownTownSkatePark`, `MaloofMoneyCup`, `University`, `Industrial`, `DownTown`. +- Work folder: `--work DIR`, default `.local/collision` (gitignored). + +## Usage + +``` +py -3.13 tools/collision-inspect/map_collision.py MegaPark +py -3.13 tools/collision-inspect/analyse_spawn.py MegaPark 21.7 21.6 -19.7 +py -3.13 tools/collision-inspect/collision_attrs.py SkateSchool +py -3.13 tools/collision-inspect/scan_surfaceless.py SkateSchool StartPark +``` + +Parks take about a second; the city districts take minutes and a lot of memory. + +## Example output + +(made-up values) + +``` +triangles 12000 +bounds min [-120.0 -5.2 -130.0] max [140.0 40.1 125.0] +spawn (current rule) [10.5 0.1 6.0] + +MapA surfaces at (21.7, -19.7) : [(0.0, 1.0, 12.5), (6.3, 1.0, 3.1)] +``` + +## Requirements + +Python 3.13 with `numpy`; the repository's setup modules (`tools/asset_pipeline`) and the vendored +map extraction tools (`tools/vendor/university`, `tools/vendor/utt`). diff --git a/tools/collision-inspect/analyse_spawn.py b/tools/collision-inspect/analyse_spawn.py new file mode 100644 index 00000000..3c87f1db --- /dev/null +++ b/tools/collision-inspect/analyse_spawn.py @@ -0,0 +1,47 @@ +"""List every collision surface height at a spawn's XZ (orientation ignored). + +Usage (repo root, after map_collision.py): + py -3.13 tools/collision-inspect/analyse_spawn.py MegaPark 21.742 21.605 -19.683 [--work DIR] +Output: (height, normal-y sign, area m2) for each surface under/over the point, plus the +area-weighted centre of flat surfaces. Nothing below the spawn height = void; a much +lower surface under it = roof/platform. +""" +import argparse +from pathlib import Path +import numpy as np + +REPO = Path(__file__).resolve().parents[2] +parser = argparse.ArgumentParser(description="List every collision surface height at a point's XZ.") +parser.add_argument('district', help='district name without DIST_') +parser.add_argument('x', type=float) +parser.add_argument('y', type=float, help='ignored (every height at x, z is listed)') +parser.add_argument('z', type=float) +parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='work folder (default: %(default)s)') +args = parser.parse_args() +name, x, z = args.district, args.x, args.z +t = np.load(args.work / name / 'collision.npy') +a, b, c = t[:, 0], t[:, 1], t[:, 2] +ab, ac = b - a, c - a +cross = np.cross(ab, ac) +area = np.linalg.norm(cross, axis=1) / 2 +up = cross[:, 1] / np.maximum(2 * area, 1e-9) + +def surfaces(px, pz): + dx, dz = px - a[:, 0], pz - a[:, 2] + det = ab[:, 0] * ac[:, 2] - ac[:, 0] * ab[:, 2] + ok = np.abs(det) > 1e-12 + d = np.where(ok, det, 1.0) + u = (dx * ac[:, 2] - ac[:, 0] * dz) / d + v = (ab[:, 0] * dz - dx * ab[:, 2]) / d + inside = ok & (u >= -1e-5) & (v >= -1e-5) & (u + v <= 1.00001) + height = a[:, 1] + u * ab[:, 1] + v * ac[:, 1] + return sorted({(round(float(height[i]), 2), round(float(up[i]), 2), round(float(area[i]), 1)) + for i in np.nonzero(inside)[0]}) + +print(name, 'surfaces at', (x, z), ':', surfaces(x, z)) +centre = (a + b + c) / 3 +flat = (np.abs(up) >= .9) & (area >= 2) +w = area[flat] +cx = float((centre[flat, 0] * w).sum() / w.sum()) +cz = float((centre[flat, 2] * w).sum() / w.sum()) +print('flat-area centre', (round(cx, 1), round(cz, 1)), ':', surfaces(cx, cz)[:6]) diff --git a/tools/collision-inspect/collision_attrs.py b/tools/collision-inspect/collision_attrs.py new file mode 100644 index 00000000..42925405 --- /dev/null +++ b/tools/collision-inspect/collision_attrs.py @@ -0,0 +1,41 @@ +"""Capture a district's collision triangles with surface ids, unit flags and mesh flags. + +Usage (repo root, after map_collision.py extracted the district): + py -3.13 tools/collision-inspect/collision_attrs.py SkateSchool [--work DIR] +Writes //collision_attrs.npy (default work .local/collision) with rows: + 9 vertex floats, surface id, unit flags (0x80 = has surface id), mesh flags, mesh index. +""" +import argparse, sys +from pathlib import Path +import numpy as np + +REPO = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO)) +from tools.asset_pipeline import install +sys.path.insert(0, str(install.TOOLS / 'vendor/university/tools/vanilla_map_extraction/tools')) +from prepare_hawaiian_dream import prepare +from prepare_university import EXCLUDED_NORMAL_TEXTURE_IDS + +parser = argparse.ArgumentParser(description="Capture a district's collision triangles with surface ids and flags.") +parser.add_argument('district', nargs='?', default='SkateSchool', help='district name without DIST_ (default: %(default)s)') +parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='work folder (default: %(default)s)') +args = parser.parse_args() +name = args.district +district = 'DIST_' + name +work = args.work / name +stream = work / 'raw/data/content/world/stream' / district +rows = [] +mesh_index = [0] +def consume(meshes): + for mesh in meshes: + for t in mesh.triangles: + rows.append((*t.a, *t.b, *t.c, t.surface, t.unit_flags, mesh.mesh_flags, mesh_index[0])) + mesh_index[0] += 1 +prepare(stream_directory=stream, output_root=work / 'intermediate2', utt_root=install.TOOLS / 'vendor/utt', + district_name=district, map_name=name, package_name='analysis', cache_format='skate3-rust-map-v1', + texture_stream_names=('Tex',) if any(stream.glob('cTex_*.xsf')) else (), + excluded_normal_texture_ids=EXCLUDED_NORMAL_TEXTURE_IDS, raw_texture_cache=True, + collision_consumer=consume, write_render_sources=False) +a = np.asarray(rows, dtype=np.float64) +np.save(work / 'collision_attrs.npy', a) +print('triangles', len(a), 'meshes', mesh_index[0]) diff --git a/tools/collision-inspect/map_collision.py b/tools/collision-inspect/map_collision.py new file mode 100644 index 00000000..f3530e3e --- /dev/null +++ b/tools/collision-inspect/map_collision.py @@ -0,0 +1,56 @@ +"""Extract one district's collision triangles for analysis (does not touch the install). + +Usage (repo root): py -3.13 tools/collision-inspect/map_collision.py MegaPark [--disc DIR] [--work DIR] +Extracts world.big from the disc into //raw if needed, then writes +//collision.npy (N x 3 x 3 float64, Y-up metres) and +prints the bounds and the spawn the current map_writer rule would choose. +Parks take ~1 s; city districts take minutes and a lot of memory. +""" +import argparse +import os +import sys +from pathlib import Path +import numpy as np + +REPO = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO)) +from tools.asset_pipeline import install +from tools.asset_pipeline.map_writer import SpawnSelector +sys.path.insert(0, str(install.TOOLS / 'vendor/university/tools/vanilla_map_extraction/tools')) +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import unsigned_collision # noqa: F401 (unsigned vertex deltas, as the game reads them) +from prepare_hawaiian_dream import prepare +from prepare_university import EXCLUDED_NORMAL_TEXTURE_IDS + +parser = argparse.ArgumentParser(description="Extract one district's collision triangles for analysis.") +parser.add_argument('district', nargs='?', default='MegaPark', help='district name without DIST_ (default: %(default)s)') +parser.add_argument('--disc', type=Path, default=Path(os.environ.get('SKATE3_DISC', REPO / '.local/skate3-disc')), + help='extracted disc root, the folder holding data/ (default: %(default)s)') +parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='work folder (default: %(default)s)') +args = parser.parse_args() +name = args.district +district = 'DIST_' + name +archive = args.disc / 'data/content' / f'world{district}.big' +work = args.work / name +if not (work / 'raw').is_dir(): + install.extract(archive, work / 'raw') +stream = work / 'raw/data/content/world/stream' / district +spawn = SpawnSelector(district) +triangles = [] + +def consume(meshes): + spawn.consider(meshes) + for mesh in meshes: + triangles.extend((t.a, t.b, t.c) for t in mesh.triangles) + +prepare(stream_directory=stream, output_root=work / 'intermediate', utt_root=install.TOOLS / 'vendor/utt', + district_name=district, map_name=name, package_name='analysis', cache_format='skate3-rust-map-v1', + texture_stream_names=('Tex',) if any(stream.glob('cTex_*.xsf')) else (), + excluded_normal_texture_ids=EXCLUDED_NORMAL_TEXTURE_IDS, raw_texture_cache=True, + collision_consumer=consume, write_render_sources=False) +t = np.asarray(triangles, dtype=np.float64) +np.save(work / 'collision.npy', t) +points = t.reshape(-1, 3) +print('triangles', len(t)) +print('bounds min', points.min(0).round(1), 'max', points.max(0).round(1)) +print('spawn (current rule)', np.round(spawn.result(name), 3)) diff --git a/tools/collision-inspect/scan_surfaceless.py b/tools/collision-inspect/scan_surfaceless.py new file mode 100644 index 00000000..ab4d52ac --- /dev/null +++ b/tools/collision-inspect/scan_surfaceless.py @@ -0,0 +1,60 @@ +"""For every district: find collision meshes with no surface ids and describe them. + +Usage (repo root): py -3.13 tools/collision-inspect/scan_surfaceless.py [District ...] [--disc DIR] [--work DIR] +Extracts each district into // (default .local/collision) if needed (cities take minutes), +then prints per-mesh counts: surfaced / mixed (some units lack surface ids) / surfaceless. +""" +import argparse, collections, json, os, sys +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO)) +from tools.asset_pipeline import install +TOOLS = install.TOOLS / 'vendor/university/tools/vanilla_map_extraction/tools' +sys.path.insert(0, str(TOOLS)) +from prepare_hawaiian_dream import prepare +from prepare_university import EXCLUDED_NORMAL_TEXTURE_IDS +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import unsigned_collision # noqa: F401 (unsigned vertex deltas, as the game reads them) +from retail_collision_mesh import decode_rx2_clustered_meshes + +parser = argparse.ArgumentParser(description='Find collision meshes with no surface ids and describe them.') +parser.add_argument('districts', nargs='*', help='district names without DIST_ (default: all ten)') +parser.add_argument('--disc', type=Path, default=Path(os.environ.get('SKATE3_DISC', REPO / '.local/skate3-disc')), + help='extracted disc root, the folder holding data/ (default: %(default)s)') +parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='work folder (default: %(default)s)') +args = parser.parse_args() +names = args.districts or ['MegaPark', 'IndustrialSkatePark', 'StartPark', 'BlackBoxPark', 'DownTownSkatePark', + 'MaloofMoneyCup', 'SkateSchool', 'University', 'Industrial', 'DownTown'] +for name in names: + district = 'DIST_' + name + work = args.work / name + if not (work / 'raw').is_dir(): + install.extract(args.disc / f'data/content/world{district}.big', work / 'raw') + stream = work / 'raw/data/content/world/stream' / district + out = work / 'scan' + if not (out / 'manifest.json').is_file(): + prepare(stream_directory=stream, output_root=out, utt_root=install.TOOLS / 'vendor/utt', + district_name=district, map_name=name, package_name='analysis', cache_format='skate3-rust-map-v1', + texture_stream_names=('Tex',) if any(stream.glob('cTex_*.xsf')) else (), + excluded_normal_texture_ids=EXCLUDED_NORMAL_TEXTURE_IDS, raw_texture_cache=True, + collision_consumer=lambda meshes: None, write_render_sources=False) + manifest = json.loads((out / 'manifest.json').read_text()) + total = collections.Counter() + special = [] + for entry in manifest['simulation_assets']: + if not entry.get('collision_meshes'): + continue + for mesh in decode_rx2_clustered_meshes((out / entry['rx2']).read_bytes()): + flags = {bool(t.unit_flags & 0x80) for t in mesh.triangles} + kind = 'surfaced' if flags == {True} else 'surfaceless' if flags == {False} else 'mixed' + total[kind] += 1 + if kind != 'surfaced': + size = [round(mesh.bounds_max[k] - mesh.bounds_min[k], 1) for k in range(3)] + lo = [round(v, 2) for v in mesh.bounds_min] + hi = [round(v, 2) for v in mesh.bounds_max] + special.append(f"{kind} {entry['stream_file']} tris={len(mesh.triangles)} size={size} " + f"min={lo} max={hi}") + print(f'== {name}: {dict(total)}', flush=True) + for line in special: + print(' ', line, flush=True) diff --git a/tools/collision-inspect/unsigned_collision.py b/tools/collision-inspect/unsigned_collision.py new file mode 100644 index 00000000..e3ffdd65 --- /dev/null +++ b/tools/collision-inspect/unsigned_collision.py @@ -0,0 +1,31 @@ +"""Make the vendored collision decoder read 16-bit vertex deltas UNSIGNED, as the game does. + +The vendored `retail_collision_mesh._decode_vertices` unpacks compression-1 vertex deltas with `>3h` +(signed). The game's own decoder (crates/skate-data/src/retail_collision.rs, `be16` → u16) treats them as +unsigned, as the game does. With signed deltas, vertices of clusters spanning more than 32.8 m wrap by +65.536 m (granularity 0.001); see docs/hails-additions/05-collision-volumes.md. + +Analysis tools only: `import unsigned_collision` before using `decode_rx2_clustered_meshes`. The converter +(map_writer.spawn_point, prepare_hawaiian_dream bounds) still uses the vendored behaviour. +""" +import struct + +import retail_collision_mesh as _rcm + + +def _decode_vertices_unsigned(cluster, vertex_count, compression, granularity, vertex_data_end): + if compression != 1: + return _original(cluster, vertex_count, compression, granularity, vertex_data_end) + _rcm._require(vertex_data_end >= 28, "compressed cluster omits its base") + base = struct.unpack_from(">3i", cluster, _rcm.CLUSTER_HEADER_SIZE) + vertices = [] + for index in range(vertex_count): + offset = _rcm.CLUSTER_HEADER_SIZE + 12 + index * 6 + _rcm._require(offset + 6 <= vertex_data_end, "16-bit compressed vertex extends into the unit stream") + delta = struct.unpack_from(">3H", cluster, offset) + vertices.append(tuple(_rcm._f32((base[axis] + delta[axis]) * granularity) for axis in range(3))) + return vertices + + +_original = _rcm._decode_vertices +_rcm._decode_vertices = _decode_vertices_unsigned diff --git a/tools/recomp-code-search/README.md b/tools/recomp-code-search/README.md new file mode 100644 index 00000000..c8a9085b --- /dev/null +++ b/tools/recomp-code-search/README.md @@ -0,0 +1,55 @@ +# Recomp code search + +**For use with the Skate 3 recomp's research hooks:** +[Hailey-Ross/skate3recomp, branch `research-hooks`](https://github.com/Hailey-Ross/skate3recomp/tree/research-hooks) +(described in [`docs/hails-additions/13-recomp-research-hooks.md`](../../docs/hails-additions/13-recomp-research-hooks.md)). + +> **Reference only.** These scripts search the recompiled sources and the game's memory image that +> *you* generate from your own legally owned copy of the game. You must build the recomp yourself and +> set up the paths below for your machine; nothing from the game is included here. Use what you find +> as a reference for re-implementing behaviour, never copy game code into the engine. + +Finding the code behind a behaviour is the first step before hooking it (see the research-hooks +branch for the hooks themselves). + +| Script | What it does | +|---|---| +| `fn.sh ADDR` | Prints the disassembly comments of the recompiled function `sub_ADDR`. | +| `grepfn.sh 'ERE'` | Prints `sub_XXXXXXXX: ` for every disassembly line matching the pattern, with its function. | +| `callctx.sh ADDR [before] [after]` | Every call site of `sub_ADDR` (first 20), with the surrounding disassembly and the calling function. | +| `ppcxref.py` | Cross-references in a big-endian PowerPC memory image: code that builds an address (`lis` + `addi` / load / store), strings and their users, callers of a function, the function containing an address, and data words pointing at an address (vtables, tables). | + +## Inputs + +- `RECOMP_GENERATED`: your skate3recomp build's `generated/` folder (the code generator's output, + `skate3_recomp.*.cpp`), for the shell scripts. +- For `ppcxref.py`: `--image` (or `PPC_IMAGE`), a dump of the executable's loaded memory image from + your own copy, base `0x82000000` by default (`--base`); `--funcs` (or `PPC_FUNCS`), a list of function + starts built from your generated sources (the command is in the script's help); optional + `--code LO-HI` to scan only the code section. + +## Usage + +``` +export RECOMP_GENERATED=/path/to/skate3recomp/generated +sh tools/recomp-code-search/fn.sh 82XXXXXX +sh tools/recomp-code-search/grepfn.sh 'lfs .*,-?[0-9]+\(r31\)' +sh tools/recomp-code-search/callctx.sh 82XXXXXX 6 14 + +py -3.13 tools/recomp-code-search/ppcxref.py --image image.bin --funcs funcs.txt str some_string +py -3.13 tools/recomp-code-search/ppcxref.py --image image.bin --funcs funcs.txt calls 82XXXXXX +``` + +## Example output + +`ppcxref.py calls` (made-up addresses): + +``` +82123450 in sub_82123400 +82345670 in sub_82345600 +``` + +## Requirements + +Git Bash or any POSIX shell with `grep` / `awk` for the shell scripts; Python 3.13 (standard library) +for `ppcxref.py`. diff --git a/tools/recomp-code-search/callctx.sh b/tools/recomp-code-search/callctx.sh new file mode 100644 index 00000000..4141d16f --- /dev/null +++ b/tools/recomp-code-search/callctx.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# usage: RECOMP_GENERATED= callctx.sh ADDR [before=6] [after=14] -> every call site of sub_X (first 20) +# with the surrounding asm and the calling function +B=${2:-6}; A=${3:-14} +# Needs your own skate3recomp build: RECOMP_GENERATED = its generated/ folder (the code generator's output). +: "${RECOMP_GENERATED:?set RECOMP_GENERATED to your skate3recomp generated/ folder}" +cd "$RECOMP_GENERATED" || exit 1 +x=$(echo "$1" | tr 'A-F' 'a-f') +grep -n -E "DEFINE_REX_FUNC\(sub_|^ // bl 0x$x\$" skate3_recomp.*.cpp | awk -F: '/DEFINE_REX_FUNC/{f=$1; fn=$0; next} {print f":"$2}' | head -20 | while IFS=: read file line; do + fnl=$(awk -v L=$line 'NR<=L && /DEFINE_REX_FUNC/{l=$0} NR==L{print l; exit}' $file | grep -o 'sub_[0-9A-F]*') + echo "=== in $fnl" + awk -v L=$line -v B=$B -v A=$A 'NR>=L-B*4 && NR<=L+A*4 && (/^\t\/\/ /||/^loc_/){sub(/^\t\/\/ /," ");print}' $file +done diff --git a/tools/recomp-code-search/fn.sh b/tools/recomp-code-search/fn.sh new file mode 100644 index 00000000..fbd3802e --- /dev/null +++ b/tools/recomp-code-search/fn.sh @@ -0,0 +1,7 @@ +#!/bin/sh +# usage: RECOMP_GENERATED= fn.sh ADDR -> prints the asm comments of function sub_ADDR (8 hex digits, upper case) +# Needs your own skate3recomp build: RECOMP_GENERATED = its generated/ folder (the code generator's output). +: "${RECOMP_GENERATED:?set RECOMP_GENERATED to your skate3recomp generated/ folder}" +cd "$RECOMP_GENERATED" || exit 1 +f=$(grep -l "DEFINE_REX_FUNC(sub_$1)" skate3_recomp.*.cpp | head -1) +awk -v s="DEFINE_REX_FUNC(sub_$1)" 'index($0,s){p=1;print;next} p&&/^DEFINE_REX_FUNC/{exit} p&&(/^\t\/\/ /||/^loc_/){sub(/^\t\/\/ /," ");print}' "$f" diff --git a/tools/recomp-code-search/grepfn.sh b/tools/recomp-code-search/grepfn.sh new file mode 100644 index 00000000..f15615a3 --- /dev/null +++ b/tools/recomp-code-search/grepfn.sh @@ -0,0 +1,8 @@ +#!/bin/sh +# usage: RECOMP_GENERATED= grepfn.sh 'ERE' -> prints "sub_XXXX: asm line" for every asm line matching (grep -E syntax) +# Needs your own skate3recomp build: RECOMP_GENERATED = its generated/ folder (the code generator's output). +: "${RECOMP_GENERATED:?set RECOMP_GENERATED to your skate3recomp generated/ folder}" +cd "$RECOMP_GENERATED" || exit 1 +grep -n -E "DEFINE_REX_FUNC\(sub_|^ // .*($1)" skate3_recomp.*.cpp | awk ' +/DEFINE_REX_FUNC\(sub_/ { match($0,/sub_[0-9A-F]+/); f=substr($0,RSTART,RLENGTH); next } +{ sub(/^[^\t]*\t\/\/ /,""); print f": "$0 }' diff --git a/tools/recomp-code-search/ppcxref.py b/tools/recomp-code-search/ppcxref.py new file mode 100644 index 00000000..3a782065 --- /dev/null +++ b/tools/recomp-code-search/ppcxref.py @@ -0,0 +1,167 @@ +"""Cross-reference helper for a big-endian PowerPC memory image of the game (the Xbox 360 executable's +loaded image, which you dump yourself from your own copy; base address 0x82000000 by default). + +usage: + py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS addr ADDR [ADDR ...] # code that builds the address (lis + addi/load/store) + py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS str TEXT [TEXT ...] # find strings, then xref each hit + py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS calls ADDR [ADDR ...] # callers (bl) of a function + py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS func ADDR [ADDR ...] # containing function start + py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS ptr ADDR [ADDR ...] # data words equal to the address + # (vtables, binding tables) +Options: --base (image base, default 0x82000000), --code LO-HI (code range to scan, hex; default the whole image). +Environment: PPC_IMAGE, PPC_FUNCS stand in for --image / --funcs. + +FUNCS lists function starts, one hex address first on each line. Build it from your recomp's generated +sources: + cd /generated && grep -o "DEFINE_REX_FUNC(sub_[0-9A-F]*)" skate3_recomp.*.cpp \\ + | sed 's/:DEFINE_REX_FUNC(sub_/ /;s/)//' | awk '{print $2, $1}' | sort > funcs.txt +Heuristic: a lis value is valid for 48 instructions in the same function. +""" +import argparse +import bisect +import os +import re +import struct +import sys +from pathlib import Path + +IMG = b'' +BASE = 0x82000000 +CODE_LO = CODE_HI = 0 +FUNCS = [] + +# D-form opcodes whose rA+simm is an address: addi 14, lwz 32, lwzu 33, lbz 34, stw 36, stb 38, lhz 40, +# sth 44, lfs 48, lfd 50, stfs 52, stfd 54, ld/ldu 58, std 62 +DFORM = {14, 32, 33, 34, 35, 36, 37, 38, 39, 40, 42, 44, 48, 50, 52, 54, 58, 62} + + +def func_of(a): + i = bisect.bisect_right(FUNCS, a) - 1 + return FUNCS[i] if i >= 0 else 0 + + +_W = None + + +def W(): + global _W + if _W is None: + n = (CODE_HI - CODE_LO) // 4 + _W = struct.unpack_from('>%dI' % n, IMG, CODE_LO - BASE) + return _W + + +def xrefs(targets, slack=0): + """targets: addresses (or (lo, hi) ranges). Returns a list of (pc, ea, func).""" + w = W() + out = [] + ranges = [(t, t + slack) if isinstance(t, int) else t for t in targets] + lis = {} + fstarts = set(FUNCS) + for i, ins in enumerate(w): + pc = CODE_LO + 4 * i + if pc in fstarts: + lis.clear() + op = ins >> 26 + rd = (ins >> 21) & 31 + ra = (ins >> 16) & 31 + imm = ins & 0xFFFF + simm = imm - 0x10000 if imm & 0x8000 else imm + if op == 15 and ra == 0: # lis + lis[rd] = ((simm << 16) & 0xFFFFFFFF, i) + continue + if op in DFORM and ra in lis and i - lis[ra][1] < 48: + d = simm & ~3 if op in (58, 62) else simm + ea = (lis[ra][0] + d) & 0xFFFFFFFF + for lo, hi in ranges: + if lo <= ea <= hi: + out.append((pc, ea, func_of(pc))) + break + if op not in (14, 15) and rd in lis and op in (14, 32, 33, 34, 40, 58, 31): + lis.pop(rd, None) # destination overwritten + return out + + +def callers(target): + out = [] + for i, ins in enumerate(W()): + if ins >> 26 == 18 and ins & 1: # bl + li = ins & 0x03FFFFFC + if li & 0x02000000: + li -= 0x04000000 + pc = CODE_LO + 4 * i + t = li if ins & 2 else pc + li + if t & 0xFFFFFFFF == target: + out.append((pc, func_of(pc))) + return out + + +def find_str(s): + return [m.start() + BASE for m in re.finditer(re.escape(s.encode('latin1')), IMG)] + + +def cstr(a): + o = a - BASE + return IMG[o:IMG.index(b'\0', o)].decode('latin1') + + +def ptrs(a): + b = struct.pack('>I', a) + return [m.start() + BASE for m in re.finditer(re.escape(b), IMG) if m.start() % 4 == 0] + + +def main(): + global IMG, BASE, CODE_LO, CODE_HI, FUNCS + parser = argparse.ArgumentParser(description='Cross-reference helper for a big-endian PowerPC memory image.', + epilog=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument('cmd', choices=('addr', 'str', 'calls', 'func', 'ptr')) + parser.add_argument('args', nargs='+') + parser.add_argument('--image', default=os.environ.get('PPC_IMAGE'), help='memory image file') + parser.add_argument('--funcs', default=os.environ.get('PPC_FUNCS'), help='function start list') + parser.add_argument('--base', default='0x82000000', help='image base address (hex)') + parser.add_argument('--code', help='code range LO-HI (hex); default the whole image') + a = parser.parse_args() + if not a.image or not a.funcs: + parser.error('--image and --funcs (or PPC_IMAGE / PPC_FUNCS) are required') + IMG = Path(a.image).read_bytes() + BASE = int(a.base, 16) + if a.code: + lo, hi = a.code.split('-') + CODE_LO, CODE_HI = int(lo, 16), int(hi, 16) + else: + CODE_LO, CODE_HI = BASE, BASE + len(IMG) // 4 * 4 + FUNCS = sorted(int(l.split()[0], 16) for l in Path(a.funcs).read_text().splitlines() if l.strip()) + if a.cmd == 'addr': + for pc, ea, f in xrefs([int(x, 16) for x in a.args]): + print(f'{pc:08X} -> {ea:08X} in sub_{f:08X}') + elif a.cmd == 'str': + starts = set() + for s in a.args: + for h in find_str(s): + o = h - BASE + while o > 0 and 0x20 <= IMG[o - 1] < 0x7f: + o -= 1 + starts.add(o + BASE) + res = xrefs(sorted(starts)) + for st in sorted(starts): + print(f'{st:08X} "{cstr(st)[:80]}"') + for pc, ea, f in res: + if ea == st: + print(f' {pc:08X} in sub_{f:08X}') + elif a.cmd == 'calls': + for x in a.args: + for pc, f in callers(int(x, 16)): + print(f'{pc:08X} in sub_{f:08X}') + elif a.cmd == 'func': + for x in a.args: + print(f'sub_{func_of(int(x, 16)):08X}') + elif a.cmd == 'ptr': + for x in a.args: + addr = int(x, 16) + for p in ptrs(addr): + nb = struct.unpack_from('>4I', IMG, p - BASE - 4) + print(f'{addr:08X} @ {p:08X} [-4..+12] ' + ' '.join(f'{v:08X}' for v in nb)) + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/regression-checks/README.md b/tools/regression-checks/README.md new file mode 100644 index 00000000..7f9880a9 --- /dev/null +++ b/tools/regression-checks/README.md @@ -0,0 +1,64 @@ +# Regression checks + +Checks to run after changing the asset pipeline, the converter, input, native dependencies, build +scripts or game code, before calling a change done. Each one compares the current install or build +against a baseline you record yourself on a known-good state. + +| Script | What it checks | +|---|---| +| `check_maps.py` | Loads every installed map in one game process (`skate3rust --validate-maps`) and compares each map's loaded collision triangle count with the baseline. Fails on any map error or warning (spawn support, startup, invisible floor). Falls back to one `--check-assets` launch per map on builds without `--validate-maps`. | +| `check_spawns.py` | Reads the spawn baked into every installed `.skate` map and compares it with the baseline (0.01 m tolerance). | +| `check_customiser.py` | Snapshots the character customiser outputs (every stage receipt: path, size, SHA-256 of each file) and compares a later install with it. Ignores differences that are only the random set id embedded in JSON paths. | +| `smoke_maps.sh` | Muted crash smoke test: launches the game on each map for N seconds (windowed, `--mute`) and counts panics, `ERROR` lines, non-finite physics and render-ready lines in the log. | + +## Inputs + +- An installation set up by `tools/setup.py` (`data/installation.json`, `data/installations//`). +- `bin/skate3rust.exe` for `check_maps.py` (`--exe` for another build) and `smoke_maps.sh` (`SKATE_EXE`). +- Baselines live in `.local/regression/` (gitignored) unless you pass `--baseline`; the customiser + snapshot in `.local/customiser-baseline/` (`--snapshot`). + +## Usage + +From the repository root: + +``` +# once, on a known-good build / install +py -3.13 tools/regression-checks/check_maps.py --update +py -3.13 tools/regression-checks/check_spawns.py --update +py -3.13 tools/regression-checks/check_customiser.py --save + +# after a change +py -3.13 tools/regression-checks/check_maps.py +py -3.13 tools/regression-checks/check_spawns.py +py -3.13 tools/regression-checks/check_customiser.py + +# smoke test (Git Bash): 25 s per map, default maps StartPark University DownTown MegaPark +bash tools/regression-checks/smoke_maps.sh 25 StartPark Industrial +``` + +Every check exits 1 on a difference. The smoke test opens a game window per map; don't run it while +someone is playing on the same machine. + +## Example output + +`check_maps.py` (made-up numbers): + +``` +TEST_WORLD ... +MapA 12345 ok (0.41s) +MapB 67890 CHANGED (baseline 67880, delta 10) +``` + +`smoke_maps.sh` (from a real run; exit 124 = stopped by the timeout, as intended): + +``` +StartPark exit=124 panics=0 errors=0 nonfinite=0 render_ready=1 native_lines=7 +University exit=124 panics=0 errors=0 nonfinite=0 render_ready=1 native_lines=7 +logs: /.local/smoke/20261003_090413 +``` + +## Requirements + +- Windows, Python 3.13 (standard library only). +- `smoke_maps.sh`: Git Bash (`timeout`, `taskkill`). diff --git a/tools/regression-checks/check_customiser.py b/tools/regression-checks/check_customiser.py new file mode 100644 index 00000000..08032c0a --- /dev/null +++ b/tools/regression-checks/check_customiser.py @@ -0,0 +1,82 @@ +"""Snapshot / compare the character customiser outputs of the current installation. + +Usage (repo root): + py -3.13 tools/regression-checks/check_customiser.py --save # snapshot the current install + py -3.13 tools/regression-checks/check_customiser.py # compare current install to snapshot + +Snapshot (default .local/customiser-baseline/, --snapshot DIR): every stage receipt (*-complete.json: relative path -> +size + sha256 of each output file), every JSON file of the set, and the set id. JSON files may embed +the random set id in paths, so a JSON file whose hash differs is compared again with the set id +replaced. Exit 1 on any difference. Use after changing anything the customiser could touch +(stage scheduling, shared parsers, PIL/numpy versions, ...). +""" +import argparse +import json +import shutil +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +SNAPSHOT = REPO / '.local/customiser-baseline' + + +def current_set(): + marker = json.loads((REPO / 'data/installation.json').read_text()) + custom = REPO / 'data' / marker['directory'] / 'assets/private/customisation' + set_id = json.loads((custom / 'current.json').read_text())['set'] + return set_id, custom / 'sets' / set_id + + +def receipts(directory): + files = {} + for path in sorted(directory.glob('*-complete.json')): + files.update(json.loads(path.read_text())['files']) + return files + + +def save(): + set_id, directory = current_set() + if SNAPSHOT.exists(): + shutil.rmtree(SNAPSHOT) + (SNAPSHOT / 'json').mkdir(parents=True) + for path in directory.glob('*-complete.json'): + shutil.copy2(path, SNAPSHOT / path.name) + for path in directory.rglob('*.json'): + target = SNAPSHOT / 'json' / path.relative_to(directory) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(path, target) + (SNAPSHOT / 'SET_ID').write_text(set_id) + print(f'snapshot of set {set_id}: {len(receipts(directory))} receipted files') + return 0 + + +def compare(): + set_id, directory = current_set() + if not (SNAPSHOT / 'SET_ID').is_file(): + print(f'no snapshot in {SNAPSHOT}: run once with --save on a known-good install') + return 1 + old_id = (SNAPSHOT / 'SET_ID').read_text().strip() + old, new = receipts(SNAPSHOT), receipts(directory) + problems = [] + for name in sorted(set(old) | set(new)): + if name not in new or name not in old: + problems.append(f'{"missing" if name not in new else "new"}: {name}') + elif old[name]['sha256'] != new[name]['sha256']: + saved = SNAPSHOT / 'json' / name + if name.endswith('.json') and saved.is_file(): + if saved.read_text(encoding='utf-8').replace(old_id, set_id) == (directory / name).read_text(encoding='utf-8'): + continue # differs only by the embedded set id + problems.append(f'changed: {name}') + print(f'compared {len(old)} snapshot files with {len(new)} current files (set {old_id} -> {set_id})') + for problem in problems[:40]: + print(' ' + problem) + print('IDENTICAL' if not problems else f'{len(problems)} DIFFERENCES') + return 1 if problems else 0 + + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description='Snapshot / compare the character customiser outputs of the current installation.') + parser.add_argument('--save', action='store_true', help='snapshot the current install (else compare with the snapshot)') + parser.add_argument('--snapshot', type=Path, default=SNAPSHOT, help='snapshot folder (default: %(default)s)') + args = parser.parse_args() + SNAPSHOT = args.snapshot + raise SystemExit(save() if args.save else compare()) diff --git a/tools/regression-checks/check_maps.py b/tools/regression-checks/check_maps.py new file mode 100644 index 00000000..8ea0d1e7 --- /dev/null +++ b/tools/regression-checks/check_maps.py @@ -0,0 +1,92 @@ +"""Validate every installed map in ONE game process and compare against baselines. + +Usage (repo root, after building/staging bin\\skate3rust.exe): + py -3.13 tools/regression-checks/check_maps.py --update # first, on a known-good build: record the baseline + py -3.13 tools/regression-checks/check_maps.py # later: compare against it + +Uses `skate3rust --validate-maps` (crates/skate-game/src/map_validation.rs): TEST_WORLD plus +each map. Fails if a map is not ok, has warnings (spawn support / startup / invisible floor), +or its loaded collision triangle count differs from the baseline +(default .local/regression/collision_baseline.json; --baseline PATH, --exe PATH). +Falls back to one `--check-assets` launch per map if the exe has no --validate-maps. +""" +import argparse +import json +import re +import subprocess +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +BASELINE = REPO / '.local/regression/collision_baseline.json' + + +def validate(exe, maps): + process = subprocess.Popen([str(exe), '--assets', str(REPO / 'assets'), '--validate-maps'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, + text=True, cwd=REPO) + if not process.stdout.readline().startswith('SKATE_VALIDATOR_READY'): + process.kill() + return None + results = {} + for request in ['TEST_WORLD', *map(str, maps)]: + process.stdin.write(request + '\n'); process.stdin.flush() + line = process.stdout.readline() + if not line.startswith('SKATE_MAP_CHECK '): + raise RuntimeError(f'validator stopped at {request}') + result = json.loads(line[len('SKATE_MAP_CHECK '):]) + results['TEST_WORLD' if request == 'TEST_WORLD' else Path(request).stem] = result + process.stdin.close(); process.wait() + return results + + +def check_assets(exe, maps): + results = {} + for path in maps: + run = subprocess.run([str(exe), '--assets', str(REPO / 'assets'), '--map', str(path), '--check-assets'], + capture_output=True, text=True, errors='replace', cwd=REPO) + match = re.search(r'SKATE_RWCM_READY triangles=(\d+)', run.stdout + run.stderr) + results[path.stem] = {'ok': run.returncode == 0 and 'SKATE_ASSETS_READY' in run.stdout + run.stderr, + 'errors': [], 'warnings': [], 'collision_triangles': int(match.group(1)) if match else None} + return results + + +def main(): + parser = argparse.ArgumentParser(description='Validate every installed map and compare collision triangle counts ' + 'with a baseline.') + parser.add_argument('--update', action='store_true', help='write the current values as the new baseline') + parser.add_argument('--baseline', type=Path, default=BASELINE, help='baseline JSON (default: %(default)s)') + parser.add_argument('--exe', type=Path, default=REPO / 'bin/skate3rust.exe', help='game exe (default: %(default)s)') + args = parser.parse_args() + marker = json.loads((REPO / 'data/installation.json').read_text()) + maps = sorted((REPO / 'data' / marker['directory'] / 'maps').glob('*.skate')) + exe = args.exe + results = validate(exe, maps) + if results is None: + print('(no --validate-maps in this exe: using --check-assets per map, no spawn checks)') + results = check_assets(exe, maps) + current = {name: r['collision_triangles'] for name, r in results.items() if name != 'TEST_WORLD'} + failed = False + for name, r in results.items(): + if not r['ok'] or r['warnings']: + failed = True + print(f"{name:20} NOT OK errors={r['errors']} warnings={r['warnings']}") + if args.update: + args.baseline.parent.mkdir(parents=True, exist_ok=True) + args.baseline.write_text(json.dumps(current, indent=2) + '\n') + print(f'baseline updated with {len(current)} maps from {marker["directory"]}') + return 1 if failed else 0 + if not args.baseline.is_file(): + print(f'no baseline at {args.baseline}: run once with --update on a known-good build') + return 1 + baseline = json.loads(args.baseline.read_text()) + for name in sorted(set(baseline) | set(current)): + want, got = baseline.get(name), current.get(name) + status = 'ok' if want == got else f'CHANGED (baseline {want}, delta {None if None in (want, got) else got - want})' + failed |= want != got + seconds = results.get(name, {}).get('seconds') + print(f'{name:20} {got} {status}' + (f' ({seconds:.2f}s)' if seconds is not None else '')) + return 1 if failed else 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/tools/regression-checks/check_spawns.py b/tools/regression-checks/check_spawns.py new file mode 100644 index 00000000..f24fa1b8 --- /dev/null +++ b/tools/regression-checks/check_spawns.py @@ -0,0 +1,65 @@ +"""Compare every installed map's baked spawn with the known-good baseline. + +Usage (from the repo root): + py -3.13 tools/regression-checks/check_spawns.py --update # first, on a known-good install: record the baseline + py -3.13 tools/regression-checks/check_spawns.py # later: compare against it + +The installation is read from data/installation.json; the baseline defaults to +.local/regression/spawn_baseline.json (--baseline PATH). Exit code 1 if any map differs. +""" +import argparse +import json +import struct +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +BASELINE = REPO / '.local/regression/spawn_baseline.json' +TOLERANCE = 0.01 + + +def read_spawn(path): + # SKATE14 header: magic, u32, length-prefixed name, then 3 x f32 spawn. + with open(path, 'rb') as f: + magic = f.read(8) + if not magic.startswith(b'SKATE'): + raise ValueError(f'{path.name}: not a .skate file') + f.read(4) + (length,) = struct.unpack('//.log and one summary line per map (also in summary.txt). +# Runs from Git Bash on Windows (uses taskkill to make sure the game is gone between maps). +set -u +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +EXE="${SKATE_EXE:-$ROOT/bin/skate3rust.exe}" +SECS="${1:-25}"; shift || true +MAPS=("$@"); [ ${#MAPS[@]} -eq 0 ] && MAPS=(StartPark University DownTown MegaPark) +INST=$(ls -d "$ROOT"/data/installations/*/ 2>/dev/null | head -1) +if [ -z "$INST" ] || [ ! -x "$EXE" ]; then echo "need an installation under data/installations and $EXE"; exit 2; fi +OUT="${SMOKE_OUT:-$ROOT/.local/smoke}/$(date +%Y%m%d_%H%M%S)"; mkdir -p "$OUT" +for m in "${MAPS[@]}"; do + log="$OUT/$m.log" + SKATE_REPORT_CHILD=1 SKATE3_MODS="$ROOT/mods" timeout "$SECS" "$EXE" --assets "$ROOT/assets" --map "$INST/maps/$m.skate" --mute >"$log" 2>&1 + code=$? + taskkill //F //IM "$(basename "$EXE")" >/dev/null 2>&1 + clean=$(sed 's/\x1b\[[0-9;]*m//g' "$log") + panics=$(grep -ciE "panicked|panic" <<<"$clean"); errors=$(grep -c " ERROR " <<<"$clean") + nonfinite=$(grep -cE "NONFINITE|Non-finite|physics_failed:true" <<<"$clean") + ready=$(grep -c "SKATE_RENDER_READY" <<<"$clean"); native=$(grep -c "Game audio: native" <<<"$clean") + echo "$m exit=$code panics=$panics errors=$errors nonfinite=$nonfinite render_ready=$ready native_lines=$native" | tee -a "$OUT/summary.txt" +done +echo "logs: $OUT" diff --git a/tools/setup-equivalence/README.md b/tools/setup-equivalence/README.md new file mode 100644 index 00000000..d0610a8c --- /dev/null +++ b/tools/setup-equivalence/README.md @@ -0,0 +1,41 @@ +# Setup equivalence checks + +Prove that a faster setup path produces exactly the same bytes as the code it replaces. Both were +written to verify setup speed-ups (see `docs/hails-additions/08-setup-performance.md`). + +| Script | What it proves | +|---|---| +| `compare_streams.py` | A changed district stream loader (`tools/vendor/university/.../skate3_streams.py`) returns exactly what a reference copy returns: every asset's record, source path, offset, stored size and decoded bytes, per district and stream (Pres, Sim, Tex). Also times both. | +| `compare_refpack.py` | The native RefPack DLL (`target/native/refpack.dll`, built by `scripts\Build.ps1`) decodes exactly like the pure-Python decoder: every entry of `createacharacter.big` and `marquee.big`, and with `--streams` every extracted district stream asset. | + +## Inputs + +- Your extracted disc: `--disc DIR` (the folder holding `data/`), default `$SKATE3_DISC` or `.local/skate3-disc`. +- Extracted districts under `--work` (default `.local/collision`), made by + `tools/collision-inspect/map_collision.py` or `scan_surfaceless.py`. +- For `compare_streams.py`, a reference copy of the loader: + `git show :tools/vendor/university/tools/vanilla_map_extraction/tools/skate3_streams.py > .local/ref.py` + +## Usage + +``` +py -3.13 tools/setup-equivalence/compare_streams.py .local/ref.py MegaPark StartPark +py -3.13 tools/setup-equivalence/compare_refpack.py --streams +``` + +Both exit 1 on any difference. + +## Example output + +(made-up timings) + +``` +MegaPark Pres assets 812 old 1.90s new 0.85s IDENTICAL +MegaPark Sim assets 405 old 0.70s new 0.31s IDENTICAL +TOTAL old 2.6s new 1.2s +``` + +## Requirements + +Python 3.13 (standard library); the repository's setup modules (`tools/asset_pipeline`, +`tools/owned_game`, the vendored map tools). `compare_refpack.py` needs the DLL from `scripts\Build.ps1`. diff --git a/tools/setup-equivalence/compare_refpack.py b/tools/setup-equivalence/compare_refpack.py new file mode 100644 index 00000000..7fc7fe47 --- /dev/null +++ b/tools/setup-equivalence/compare_refpack.py @@ -0,0 +1,65 @@ +"""Prove the native RefPack DLL decodes exactly like the pure-Python decoder. + +Usage (repo root, after scripts\\Build.ps1 built target/native/refpack.dll): + py -3.13 tools/setup-equivalence/compare_refpack.py [--streams] [--disc DIR] [--work DIR] + +Decodes every entry of the character archives (createacharacter.big, marquee.big — what +the customiser reads) with the Python decoder (fast_refpack._library = None) and with the +DLL, and compares bytes. --streams also compares every district stream asset (map stage) of the +districts extracted under --work (default .local/collision, see tools/collision-inspect). +--disc is the extracted disc root (the folder holding data/; default $SKATE3_DISC or .local/skate3-disc). +Exit 1 on any difference. +""" +import argparse, os, sys, time +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO)) +sys.path.insert(0, str(REPO / 'tools/vendor/university/tools/vanilla_map_extraction/tools')) +from tools.asset_pipeline import fast_refpack +from tools.owned_game.big import BigArchive + +parser = argparse.ArgumentParser(description='Prove the native RefPack DLL decodes exactly like the Python decoder.') +parser.add_argument('--streams', action='store_true', help='also compare every extracted district stream asset') +parser.add_argument('--disc', type=Path, default=Path(os.environ.get('SKATE3_DISC', REPO / '.local/skate3-disc')), + help='extracted disc root (default: %(default)s)') +parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='extracted districts (default: %(default)s)') +args = parser.parse_args() +native = fast_refpack._library +if native is None: + sys.exit('target/native/refpack.dll not loaded: run scripts\\Build.ps1 first') +disc = args.disc / 'data/content' + + +def decode_all(read): + fast_refpack._library = None + start = time.perf_counter(); python = read(); t_python = time.perf_counter() - start + fast_refpack._library = native + start = time.perf_counter(); fast = read(); t_native = time.perf_counter() - start + return python, fast, t_python, t_native + + +failed = False +for name in ('createacharacter.big', 'marquee.big'): + archive = BigArchive(disc / name) + python, fast, t_python, t_native = decode_all(lambda: [archive.read(e) for e in archive.entries]) + compressed = sum(1 for e in archive.entries if e.compression) + same = python == fast + failed |= not same + print(f'{name:22} entries {len(archive.entries):5} (compressed {compressed:5}) python {t_python:6.1f}s native {t_native:5.1f}s {"IDENTICAL" if same else "DIFFERENT"}', flush=True) + +if args.streams: + import skate3_streams + for district in sorted(args.work.iterdir()): + stream = district / 'raw/data/content/world/stream' / f'DIST_{district.name}' + for kind in ('Pres', 'Sim', 'Tex'): + if not (stream / f'DIST_{district.name}_{kind}.xst').is_file(): + continue + python, fast, t_python, t_native = decode_all( + lambda: [a.data for a in skate3_streams.load_district_stream(stream, kind, f'DIST_{district.name}')]) + same = python == fast + failed |= not same + print(f'{district.name:20} {kind:4} assets {len(python):5} python {t_python:6.1f}s native {t_native:5.1f}s {"IDENTICAL" if same else "DIFFERENT"}', flush=True) + +print('ALL IDENTICAL' if not failed else 'DIFFERENCES FOUND') +sys.exit(1 if failed else 0) diff --git a/tools/setup-equivalence/compare_streams.py b/tools/setup-equivalence/compare_streams.py new file mode 100644 index 00000000..1c27c041 --- /dev/null +++ b/tools/setup-equivalence/compare_streams.py @@ -0,0 +1,47 @@ +"""Prove a changed stream loader returns exactly what a reference copy returns. + +Usage (repo root): + py -3.13 tools/setup-equivalence/compare_streams.py [District ...] [--work DIR] +Get a reference with: git show :tools/vendor/university/tools/vanilla_map_extraction/tools/skate3_streams.py > .local/ref.py +Needs //raw (default work .local/collision) from tools/collision-inspect/map_collision.py +or scan_surfaceless.py. + +For every district and stream (Pres, Sim, Tex): load with the reference module and the current one, and +compare every asset's record, source path, source offset, stored size and decoded bytes. Also times both. +Exit 1 on the first difference. +""" +import argparse, importlib.util, sys, time +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +TOOLS = REPO / 'tools/vendor/university/tools/vanilla_map_extraction/tools' +sys.path.insert(0, str(TOOLS)) +import skate3_streams as new + +parser = argparse.ArgumentParser(description='Prove a changed stream loader returns exactly what a reference copy returns.') +parser.add_argument('reference', type=Path, help='reference copy of skate3_streams.py') +parser.add_argument('districts', nargs='*', help='district names without DIST_ (default: all ten)') +parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='extracted districts (default: %(default)s)') +args = parser.parse_args() +spec = importlib.util.spec_from_file_location('skate3_streams_original', args.reference) +old = importlib.util.module_from_spec(spec); sys.modules[spec.name] = old; spec.loader.exec_module(old) +import dataclasses + +names = args.districts or ['MegaPark', 'StartPark', 'BlackBoxPark', 'IndustrialSkatePark', 'DownTownSkatePark', + 'MaloofMoneyCup', 'SkateSchool', 'Industrial', 'University', 'DownTown'] +total_old = total_new = 0.0 +for name in names: + stream = args.work / name / 'raw/data/content/world/stream' / f'DIST_{name}' + for kind in ('Pres', 'Sim', 'Tex'): + if not (stream / f'DIST_{name}_{kind}.xst').is_file(): + continue + t = time.perf_counter(); a = old.load_district_stream(stream, kind, f'DIST_{name}'); t_old = time.perf_counter() - t + t = time.perf_counter(); b = new.load_district_stream(stream, kind, f'DIST_{name}'); t_new = time.perf_counter() - t + total_old += t_old; total_new += t_new + same = len(a) == len(b) and all( + (dataclasses.astuple(x.record) == dataclasses.astuple(y.record) and x.source_path == y.source_path and x.source_offset == y.source_offset + and x.stored_size == y.stored_size and x.data == y.data) for x, y in zip(a, b)) + print(f'{name:20} {kind:4} assets {len(a):5} old {t_old:7.2f}s new {t_new:7.2f}s {"IDENTICAL" if same else "DIFFERENT"}', flush=True) + if not same: + sys.exit(1) +print(f'TOTAL old {total_old:.1f}s new {total_new:.1f}s') diff --git a/tools/vault-inspect/README.md b/tools/vault-inspect/README.md new file mode 100644 index 00000000..9ce0e3a7 --- /dev/null +++ b/tools/vault-inspect/README.md @@ -0,0 +1,42 @@ +# Vault inspection + +Look up values in the game's attribute database ("vault"): the skater collections setup converts to +`assets/private/stock/skater-collections.json`, and the class layouts in the disc's schema. +Classes, collections and fields are 64-bit name hashes (`Hash_XXXXXXXXXXXXXXXX`). + +| Script | What it does | +|---|---| +| `find_field.py` | Finds field hashes anywhere in the converted database: prints class, collection, field, type and the decoded value. | +| `vault_fields.py` | Prints one class's collections (or one collection), with parent, and every field (or the chosen ones), floats / ints decoded and arrays listed. | +| `vault_layout.py` | Prints a class's field layout from the schema: field hash, type, offset, count, maximum, flags, alignment. Fixed-layout fields (flags bit 2) first, by offset. | + +## Inputs + +- `--vault`: the converted database, default `assets/private/stock/skater-collections.json` (written by + `tools/setup.py`). +- `--schema` (for `vault_layout.py`): the schema path without extension; the folder must hold + `skaterschema.vlt` and `skaterschema.bin`. Extract them from your disc's `data/big/db.big`: + `py -3.13 tools/world-stream-inspect/big_list.py /data/big/db.big skaterschema --out .local/db` + +## Usage + +``` +py -3.13 tools/vault-inspect/find_field.py 0123456789ABCDEF +py -3.13 tools/vault-inspect/vault_fields.py Hash_0123456789ABCDEF default +py -3.13 tools/vault-inspect/vault_fields.py 0123456789ABCDEF default FEDCBA9876543210 +py -3.13 tools/vault-inspect/vault_layout.py 0123456789ABCDEF --schema .local/db/data/db/skaterschema +``` + +## Example output + +(made-up hashes and values) + +``` +== Hash_0123456789ABCDEF default parent= + Hash_1111111111111111 ('EA::Reflection::Float', 0.5) + Hash_2222222222222222 ('EA::Reflection::Int32', 3) +``` + +## Requirements + +Python 3.13 (standard library); `vault_layout.py` uses `tools/asset_pipeline/vlt.py`. diff --git a/tools/vault-inspect/find_field.py b/tools/vault-inspect/find_field.py new file mode 100644 index 00000000..3fd4ddd7 --- /dev/null +++ b/tools/vault-inspect/find_field.py @@ -0,0 +1,35 @@ +"""Find vault field hashes in the install's converted database (skater-collections.json): prints class, +collection, field, type and the decoded value (big-endian float or int for 4-byte values) of every hit. + +usage: py -3.13 tools/vault-inspect/find_field.py HASH [HASH ...] [--vault JSON] + HASH: 16 hex digits, with or without Hash_. + --vault defaults to assets/private/stock/skater-collections.json (written by setup). +""" +import argparse +import json +import struct +from pathlib import Path + +DEFAULT = Path(__file__).resolve().parents[2] / 'assets/private/stock/skater-collections.json' + + +def main(): + parser = argparse.ArgumentParser(description='Find vault field hashes in skater-collections.json.') + parser.add_argument('hashes', nargs='+', help='16 hex digits, with or without Hash_') + parser.add_argument('--vault', type=Path, default=DEFAULT, help='converted database (default: %(default)s)') + args = parser.parse_args() + want = {('Hash_' + h.replace('Hash_', '').upper().zfill(16)) for h in args.hashes} + for c in json.loads(args.vault.read_text())['collections']: + for w in want & set(c['fields']): + v = c['fields'][w] + h = ''.join(v.get('data', '').split()) + val = h + if 'Float' in v['type'] and len(h) == 8: + val = struct.unpack('>f', bytes.fromhex(h))[0] + elif len(h) == 8: + val = struct.unpack('>i', bytes.fromhex(h))[0] + print(c['class'], c['key'], w, v['type'], val) + + +if __name__ == '__main__': + main() diff --git a/tools/vault-inspect/vault_fields.py b/tools/vault-inspect/vault_fields.py new file mode 100644 index 00000000..3b989504 --- /dev/null +++ b/tools/vault-inspect/vault_fields.py @@ -0,0 +1,65 @@ +"""Print vault fields from the install's converted database (skater-collections.json) by class / +collection / field keys, with the collection's parent. Floats and ints are decoded; arrays list their items. + +usage: py -3.13 tools/vault-inspect/vault_fields.py CLASS [COLLECTION] [FIELD ...] [--vault JSON] + CLASS / FIELD: 16 hex digits (with or without Hash_); COLLECTION: a name ('default') or hex. + --vault defaults to assets/private/stock/skater-collections.json (written by setup). +""" +import argparse +import json +import struct +from pathlib import Path + +DEFAULT = Path(__file__).resolve().parents[2] / 'assets/private/stock/skater-collections.json' + + +def norm(k): + k = k.strip() + if k.lower() == 'default': + return 'default' + return k if k.startswith('Hash_') else 'Hash_' + k.upper().replace('0X', '').zfill(16) + + +def decode(v): + t = v.get('type', '') + + def one(h): + b = bytes.fromhex(h) + if 'Float' in t and len(b) == 4: + return round(struct.unpack('>f', b)[0], 6) + if len(b) == 4: + return struct.unpack('>i', b)[0] + if len(b) == 1: + return b[0] + return h + if 'array' in v: + return t, [one(x) for x in v['array']['items']] + return t, one(v['data']) + + +def main(): + parser = argparse.ArgumentParser(description='Print vault fields by class / collection / field.') + parser.add_argument('cls', metavar='CLASS') + parser.add_argument('collection', nargs='?', metavar='COLLECTION') + parser.add_argument('fields', nargs='*', metavar='FIELD') + parser.add_argument('--vault', type=Path, default=DEFAULT, help='converted database (default: %(default)s)') + args = parser.parse_args() + d = json.loads(args.vault.read_text()) + items = d['collections'] if 'collections' in d else d + items = list(items.values()) if isinstance(items, dict) else items + cls = norm(args.cls) + col = norm(args.collection) if args.collection else None + fields = [norm(f) for f in args.fields] + by = {c['key']: c for c in items if c['class'] == cls} + for key, c in by.items(): + if col and key != col: + continue + print(f'== {cls} {key} parent={c.get("parent")}') + for f, v in sorted(c['fields'].items()): + if fields and f not in fields: + continue + print(f' {f} {decode(v)}') + + +if __name__ == '__main__': + main() diff --git a/tools/vault-inspect/vault_layout.py b/tools/vault-inspect/vault_layout.py new file mode 100644 index 00000000..60159fc0 --- /dev/null +++ b/tools/vault-inspect/vault_layout.py @@ -0,0 +1,55 @@ +"""Print a vault class's field layout (field hash, type, offset, count, flags) from the disc schema. + +usage: py -3.13 tools/vault-inspect/vault_layout.py CLASS --schema STEM [--name NAME ...] + CLASS: 16 hex digits (with or without Hash_). + --schema: path of skaterschema without extension (the folder must hold skaterschema.vlt and .bin; + extract them from data/big/db.big with tools/world-stream-inspect/big_list.py db.big skaterschema --out DIR). + --name: type names to label (their 64-bit hash is shown by name when it matches); repeatable. +flags bit 2 = a fixed-layout field: the offset is where the record keeps it. +""" +import argparse +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT)) +from tools.asset_pipeline.vlt import vault, unpack, hash64 # noqa: E402 + +CLASS_EXPORT = 0x2A7895AC4A876152 +# Generic reflection type names, so the common field types print by name. +TYPE_NAMES = ('EA::Reflection::Int32', 'EA::Reflection::Bool', 'EA::Reflection::Float', 'Attrib::RefSpec') + + +def layout(cls: int, stem: Path): + sv, sb, se = vault(stem) + for _, kind, size, at in se: + if kind != CLASS_EXPORT: + continue + key, reserve, count, defs, static_size, static, lay = unpack('QIIIIII', sv, at) + if key != cls: + continue + rows = [] + for i in range(count): + fkey, typ, offset, n, maximum, flags, alignment = unpack('QQHHHBB', sb, defs + i * 24) + rows.append((fkey, typ, offset, n, maximum, flags, alignment)) + return rows + return None + + +def main(): + parser = argparse.ArgumentParser(description="Print a vault class's field layout from the disc schema.") + parser.add_argument('cls', metavar='CLASS', help='16 hex digits, with or without Hash_') + parser.add_argument('--schema', type=Path, required=True, help='skaterschema path without extension') + parser.add_argument('--name', action='append', default=[], help='extra type name to label; repeatable') + args = parser.parse_args() + cls = int(args.cls.replace('Hash_', ''), 16) + names = {hash64(n): n for n in (*TYPE_NAMES, *args.name)} + rows = layout(cls, args.schema) + if rows is None: + sys.exit('class not found') + for fkey, typ, offset, n, maximum, flags, alignment in sorted(rows, key=lambda r: (not r[5] & 2, r[2])): + print(f'Hash_{fkey:016X} {names.get(typ, hex(typ)):36} off={offset:4} n={n:3} max={maximum} flags={flags} align={alignment}') + + +if __name__ == '__main__': + main() diff --git a/tools/world-stream-inspect/README.md b/tools/world-stream-inspect/README.md new file mode 100644 index 00000000..cb89ebd5 --- /dev/null +++ b/tools/world-stream-inspect/README.md @@ -0,0 +1,42 @@ +# World stream inspection + +Read the world districts' simulation streams (`cSim_*.xsf`, RW4 arenas) straight from your own +disc's `.big` archives, without extracting anything to disk. + +| Script | What it does | +|---|---| +| `big_list.py` | Lists the entries of any `.big` archive (path and unpacked size), filtered by a regex, or extracts the matching entries with `--out`. | +| `sim_types.py` | Per stream file of a district: the arena dictionary's entry types and counts. `--find HEX64` searches every arena for a 64-bit value (both word orders) and names the entry that holds it; `--dump TYPE` hex-dumps the entries of one type. | +| `volumes_dump.py` | Dumps every named volume (arena type `0x00EB0019`, trigger / zone / teleport boxes): name, instance id, link id, bounds, base corners, plus each arena's collision meshes and other entry types. Writes JSON too. | +| `arenas.py` | Shared helper: walks the arenas of a district (or any `.big` with `*_Sim.xst` + `cSim_*.xsf`, such as `missions.big`) and parses the arena dictionary. | + +The volume layout is documented at the top of `volumes_dump.py`. + +## Inputs + +- Your extracted disc: `--disc DIR` (the folder holding `data/`), default `$SKATE3_DISC` or `.local/skate3-disc`. +- District ids with the prefix: `DIST_SkateSchool`, `DIST_MegaPark`, … (`volumes_dump.py` defaults to all ten). + +## Usage + +``` +py -3.13 tools/world-stream-inspect/big_list.py .local/skate3-disc/data/big/db.big skaterschema +py -3.13 tools/world-stream-inspect/big_list.py .local/skate3-disc/data/audio/audiofiles.big "\.bnk$" --out .local/bnk +py -3.13 tools/world-stream-inspect/sim_types.py DIST_SkateSchool +py -3.13 tools/world-stream-inspect/sim_types.py DIST_SkateSchool --dump 00eb0019 +py -3.13 tools/world-stream-inspect/volumes_dump.py DIST_SkateSchool > volumes.txt +``` + +## Example output + +`sim_types.py` (made-up ids and counts): + +``` +cSim_0_0_high.xsf 0123456789abcdef 00080001:1 00080006:3 00eb000a:1 +TOTAL 00080001:40 00080006:112 00eb000a:40 00eb0019:2 +``` + +## Requirements + +Python 3.13 (standard library); the repository's `tools/owned_game` and the vendored map extraction +tools (`tools/vendor/university/.../skate3_streams.py`, `retail_collision_mesh.py`). diff --git a/tools/world-stream-inspect/arenas.py b/tools/world-stream-inspect/arenas.py new file mode 100644 index 00000000..5b3278b5 --- /dev/null +++ b/tools/world-stream-inspect/arenas.py @@ -0,0 +1,89 @@ +"""Shared helpers: walk the RW4 arenas in a district's simulation streams (cSim_*.xsf) straight from a +world.big (or any .big holding *_Sim.xst + cSim_*.xsf, such as missions.big), nothing extracted. + + import arenas + for stream_file, asset_id, arena in arenas.district_arenas(disc, 'DIST_SkateSchool'): + for offset, _, size, _, _, type_id in arenas.entries(arena): + ... + +Uses the vendored stream reader (tools/vendor/university/.../skate3_streams.py) for the asset table and +the section decoder. +""" +import os +import struct +import sys +import tempfile +from collections import defaultdict +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO)) +sys.path.insert(0, str(REPO / 'tools/vendor/university/tools/vanilla_map_extraction/tools')) +from tools.owned_game.big import BigArchive # noqa: E402 +from skate3_streams import read_atoc, _decode_section # noqa: E402 + +ARENA_PROCESSOR = 0xAB329A6A +DISTRICTS = ['DIST_SkateSchool', 'DIST_MegaPark', 'DIST_IndustrialSkatePark', 'DIST_StartPark', 'DIST_BlackBoxPark', + 'DIST_DownTownSkatePark', 'DIST_MaloofMoneyCup', 'DIST_University', 'DIST_Industrial', 'DIST_DownTown'] + + +def default_disc() -> Path: + """Extracted disc root (the folder holding data/): $SKATE3_DISC, else .local/skate3-disc.""" + return Path(os.environ.get('SKATE3_DISC', REPO / '.local/skate3-disc')) + + +def entries(arena: bytes): + """The arena's dictionary: one (offset, ?, size, align?, ?, type id) tuple of BE u32 per entry.""" + n = struct.unpack_from('>I', arena, 0x20)[0] + at = struct.unpack_from('>I', arena, 0x30)[0] + return [struct.unpack_from('>6I', arena, at + 24 * i) for i in range(n)] + + +def big_arenas(bigpath, only_dirs=None): + """Yield (dir name, stream file, asset id, arena bytes) for every arena asset of every cSim_*.xsf in a + .big, using the *_Sim.xst in the same directory.""" + big = BigArchive(Path(bigpath)) + by = defaultdict(list) + for e in big.entries: + by[Path(e.path).parent.as_posix()].append(e) + with tempfile.TemporaryDirectory() as tmpdir: + tmp = Path(tmpdir) / '_tmp.xst' + for dname, ents in sorted(by.items()): + if only_dirs and Path(dname).name not in only_dirs: + continue + xst = [e for e in ents if e.path.endswith('_Sim.xst')] + if not xst: + continue + tmp.write_bytes(big.read(xst[0])) + recs = {r.asset_id: r for r in read_atoc(tmp)} + seen = set() + for e in ents: + if not (Path(e.path).name.startswith('cSim_') and e.path.endswith('.xsf')): + continue + s = big.read(e) + cur = struct.unpack_from('>I', s, 16)[0] + while cur + 208 <= len(s): + aid = struct.unpack_from('>Q', s, cur)[0] + if aid == 0: + break + st, hs, stride = struct.unpack_from('>III', s, cur + 8) + r = recs.get(aid) + if r and r.processor_id == ARENA_PROCESSOR and aid not in seen: + seen.add(aid) + if st == r.total_size and hs >= 128: + data = s[cur + hs:cur + hs + st] + else: + v = struct.unpack_from('>11I', s, cur + 128) + c, q = _decode_section(s, cur + 208, v[0], v[1], v[2]) + g, q = _decode_section(s, q, v[8], v[9], v[10]) + data = c + g + yield Path(dname).name, Path(e.path).name, aid, data + if stride == 0: + break + cur += stride + + +def district_arenas(disc: Path, district: str): + """Yield (stream file, asset id, arena bytes) for one world district.""" + for _, fname, aid, data in big_arenas(Path(disc) / 'data/content' / f'world{district}.big'): + yield fname, aid, data diff --git a/tools/world-stream-inspect/big_list.py b/tools/world-stream-inspect/big_list.py new file mode 100644 index 00000000..2fad65d8 --- /dev/null +++ b/tools/world-stream-inspect/big_list.py @@ -0,0 +1,37 @@ +"""List the entries of a .big archive, or extract the ones matching a regex. + +usage: py -3.13 tools/world-stream-inspect/big_list.py [regex] [--out DIR] +Without --out prints ` ` per entry; with --out writes the matching entries +(decompressed) under DIR, keeping their archive paths. +""" +import argparse +import re +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[2])) +from tools.owned_game.big import BigArchive # noqa: E402 + + +def main(): + parser = argparse.ArgumentParser(description='List or extract entries of a .big archive.') + parser.add_argument('big', type=Path) + parser.add_argument('regex', nargs='?', help='only entries whose path matches (case-insensitive)') + parser.add_argument('--out', type=Path, help='extract the matching entries here') + args = parser.parse_args() + a = BigArchive(args.big) + pat = re.compile(args.regex, re.I) if args.regex else None + for e in a.entries: + if pat is not None and not pat.search(e.path): + continue + if args.out: + p = args.out / BigArchive.safe_relative(e.path) + p.parent.mkdir(parents=True, exist_ok=True) + p.write_bytes(a.read(e)) + print('wrote', p) + else: + print(e.path, e.unpacked_size) + + +if __name__ == '__main__': + main() diff --git a/tools/world-stream-inspect/sim_types.py b/tools/world-stream-inspect/sim_types.py new file mode 100644 index 00000000..302d0c43 --- /dev/null +++ b/tools/world-stream-inspect/sim_types.py @@ -0,0 +1,52 @@ +"""Survey RW4 arena dictionary types in a district's simulation streams (cSim_*.xsf). + +usage: py -3.13 tools/world-stream-inspect/sim_types.py DIST_SkateSchool [--find HEX64 ...] [--dump TYPEHEX] [--disc DIR] +Reads straight from /data/content/world.big (nothing extracted to disk). +Dictionary entry = 6 BE u32: (offset, ?, size, align?, ?, type). Prints type counts per stream file; +--find searches every arena for a 64-bit value (both word orders) and names the entry holding it; +--dump hex-dumps the first 256 bytes of every entry of one type. +""" +import argparse +import sys +from collections import Counter +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import arenas # noqa: E402 + + +def main(): + parser = argparse.ArgumentParser(description='Survey RW4 arena dictionary types in a district.') + parser.add_argument('district', help='e.g. DIST_SkateSchool') + parser.add_argument('--find', action='append', default=[], metavar='HEX64', help='64-bit value to search for') + parser.add_argument('--dump', metavar='TYPEHEX', help='hex-dump the entries of this type id') + parser.add_argument('--disc', type=Path, default=arenas.default_disc(), help='extracted disc root (default: %(default)s)') + args = parser.parse_args() + finds = [int(v, 16) for v in args.find] + dump = int(args.dump, 16) if args.dump else None + total = Counter() + for fname, aid, d in arenas.district_arenas(args.disc, args.district): + ents = arenas.entries(d) + c = Counter(e[5] for e in ents) + total.update(c) + if not finds and dump is None: + print(fname, '%016x' % aid, ' '.join('%08x:%d' % kv for kv in sorted(c.items()))) + for v in finds: + for pat, tag in ((v.to_bytes(8, 'big'), 'BE'), (((v & 0xFFFFFFFF) << 32 | v >> 32).to_bytes(8, 'big'), 'LOWFIRST')): + i = d.find(pat) + while i >= 0: + owner = [k for k, e in enumerate(ents) if e[0] <= i < e[0] + e[2]] + print('FIND %016x %s %s @%x in entry %s type %s' % (v, tag, fname, i, owner, [hex(ents[k][5]) for k in owner])) + i = d.find(pat, i + 1) + if dump is not None: + for k, e in enumerate(ents): + if e[5] == dump: + print(fname, k, 'off %x size %x' % (e[0], e[2])) + blob = d[e[0]:e[0] + min(e[2], 256)] + for j in range(0, len(blob), 32): + print(' ', blob[j:j + 32].hex(' ', 4)) + print('TOTAL', ' '.join('%08x:%d' % kv for kv in sorted(total.items()))) + + +if __name__ == '__main__': + main() diff --git a/tools/world-stream-inspect/volumes_dump.py b/tools/world-stream-inspect/volumes_dump.py new file mode 100644 index 00000000..49dc756b --- /dev/null +++ b/tools/world-stream-inspect/volumes_dump.py @@ -0,0 +1,86 @@ +"""Dump every named volume (arena type 0x00EB0019, 'volume set') in the world districts: trigger boxes, +teleport and zone volumes, with their bounds, base corners and link ids. + +usage: py -3.13 tools/world-stream-inspect/volumes_dump.py [DIST_x ...] [--disc DIR] [--json OUT] > volumes.txt +Default: all ten districts; JSON to .local/world-stream-inspect/volumes.json. +Volume-set layout (big-endian): + header: +0 magic, +4 count, +8 count, +12 items offset, +16 strings offset + item (240 bytes): +0 mat4, +64 aabb min (vec4), +80 aabb max, +96 up axis (vec4), +112 4 base corners (vec4 x4), + +176 u64 link GUID (an import of type 0x00EB006B), +184 u64 instance id, +192 16 bytes 0xFF, +220 import handle, + +224 name offset (from record start). +Each arena line also lists its clustered collision meshes (triangles, surfaceless or not) and other entry types. +""" +import argparse +import json +import struct +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import arenas # noqa: E402 +from retail_collision_mesh import decode_clustered_mesh # noqa: E402 (vendored, on the path via arenas) + +VOLUME_SET = 0x00EB0019 +CLUSTERED_MESH = 0x00080006 + + +def cstr(b, o): + e = b.index(b'\0', o) + return b[o:e].decode('latin1') + + +def items(b): + magic, n, n2, ioff, soff = struct.unpack_from('>5I', b, 0) + for i in range(n): + o = ioff + 240 * i + mn = struct.unpack_from('>3f', b, o + 64) + mx = struct.unpack_from('>3f', b, o + 80) + up = struct.unpack_from('>3f', b, o + 96) + corners = [struct.unpack_from('>3f', b, o + 112 + 16 * k) for k in range(4)] + guid, iid = struct.unpack_from('>QQ', b, o + 176) + imp, noff = struct.unpack_from('>II', b, o + 220) + mat = struct.unpack_from('>16f', b, o) + yield dict(name=cstr(b, noff), id='%016x' % iid, link='%016x' % guid, import_handle=imp, + min=[round(v, 3) for v in mn], max=[round(v, 3) for v in mx], up=[round(v, 3) for v in up], + corners=[[round(v, 3) for v in c] for c in corners], + translation=[round(v, 3) for v in mat[12:15]], + identity_rot=mat[0] == 1.0 and mat[5] == 1.0 and mat[10] == 1.0) + + +def main(): + parser = argparse.ArgumentParser(description='Dump every named volume in the world districts.') + parser.add_argument('districts', nargs='*', help='e.g. DIST_SkateSchool (default: all ten)') + parser.add_argument('--disc', type=Path, default=arenas.default_disc(), help='extracted disc root (default: %(default)s)') + parser.add_argument('--json', type=Path, default=arenas.REPO / '.local/world-stream-inspect/volumes.json', + help='JSON output (default: %(default)s)') + args = parser.parse_args() + out = {} + for dist in args.districts or arenas.DISTRICTS: + res = [] + for fname, aid, d in arenas.district_arenas(args.disc, dist): + ents = arenas.entries(d) + vs = [e for e in ents if e[5] == VOLUME_SET] + if not vs: + continue + meshes = [] + for e in ents: + if e[5] == CLUSTERED_MESH: + m = decode_clustered_mesh(d[e[0]:e[0] + e[2]]) + sl = all(not (t.unit_flags & 0x80) for t in m.triangles) + meshes.append(dict(tris=len(m.triangles), surfaceless=sl, + min=[round(v, 2) for v in m.bounds_min], max=[round(v, 2) for v in m.bounds_max])) + other = sorted({'%08x' % e[5] for e in ents} - {'00eb0019', '00080006', '00080001', '00eb000a', '00eb0008', '00eb000b'}) + for e in vs: + for it in items(d[e[0]:e[0] + e[2]]): + it.update(stream=fname, arena='%016x' % aid) + res.append(it) + short = it['name'].split('_0x')[0] + print(dist, fname, it['id'], short, it['min'], it['max'], 'link', it['link']) + print(' arena %016x meshes %s other types %s' % (aid, [(m['tris'], m['surfaceless']) for m in meshes], other)) + out[dist] = res + args.json.parent.mkdir(parents=True, exist_ok=True) + args.json.write_text(json.dumps(out, indent=1)) + + +if __name__ == '__main__': + main()