diff --git a/docs/hails-additions/11-audio.md b/docs/hails-additions/11-audio.md index 6d53d04a..36aaeb9d 100644 --- a/docs/hails-additions/11-audio.md +++ b/docs/hails-additions/11-audio.md @@ -1005,7 +1005,7 @@ rolling rattle, `Class_rolling`, footsteps, cloth / body slide, flips / treatmen **Files.** New: `crates/skate-audio/src/splice/{mod,format,tests}.rs`, `crates/skate-audio/src/player/contacts.rs`, `crates/skate-game/src/game_audio/{e2e,state_log}.rs`, `tools/audio-e2e/{scenarios,compare}.py`, `tools/recomp-trace/{retail_windows,retail_voices}.py`, `tools/audio-file-inspect/splc_fields.py`, -`tools/vault-inspect/vault_fields.py`; local tools `grind_census.py`, `img.py`, `stage_splice.py`. Changed: `skate-audio` `lib.rs`, `mixer.rs`, `runtime.rs`, `player/{mod,state,tuning,inputs, +`tools/vault-inspect/vault_fields.py` (PR #37); local tools `grind_census.py`, `img.py`, `stage_splice.py`. Changed: `skate-audio` `lib.rs`, `mixer.rs`, `runtime.rs`, `player/{mod,state,tuning,inputs, components}.rs`; `game_audio/{mod,library,native,player_audio,grain_bed,skate_events}.rs`; `tools/asset_pipeline/audio_export.py` (+ tests). The PoC probe (`player_audio/e2e.rs` in the PoC worktree) is local. @@ -1136,7 +1136,7 @@ only): grind start voices Skate_Metal 194 + Skate_Collisions 619 on the first gr landings; wheel spin in the air and in the 6.5-s manual. **Files.** New: `crates/skate-audio/src/player/{collision,wheels}.rs`, `crates/skate-audio/examples/pan_fold_probe.rs`, -`tools/vault-inspect/{vault_layout,find_field}.py`, `tools/recomp-trace/grec_clean.py`, +`tools/vault-inspect/{vault_layout,find_field}.py` (PR #37), `tools/recomp-trace/grec_clean.py`, `tools/audio-e2e/voices_summary.py`. Changed: `skate-audio` `player/{mod,state,tuning,contacts}.rs`, `runtime.rs` (stream bank); `skate-core` `physics/board_ground.rs` (+ test); `skate-game` `game_audio/{player_audio,native,e2e,library, skate_events,grain_bed,state_log,voices}.rs`; `tools/asset_pipeline/audio_export.py` (`collision_tuning`, the image's diff --git a/docs/hails-additions/13-recomp-research-hooks.md b/docs/hails-additions/13-recomp-research-hooks.md index 42eb1414..bdba589d 100644 --- a/docs/hails-additions/13-recomp-research-hooks.md +++ b/docs/hails-additions/13-recomp-research-hooks.md @@ -52,7 +52,7 @@ The recompilation is **not** a perfect copy of the console game, and its traces doc 11, "The per-player recomp hooks logged an NPC skater too". The analysis scripts we used are published in `tools/recomp-trace/` (trace readers, per-bank level and voice -tools, rolling-bed and send analysis) and `tools/recomp-code-search/` (searching the recompiled sources and the +tools, rolling-bed and send analysis) and `tools/recomp-code-search/` (PR #37) (searching the recompiled sources and the memory image); see [14](14-published-tools.md). They are reference only: anyone who wants to use them needs to build the recomp and set up the paths themselves. diff --git a/docs/hails-additions/14-published-tools.md b/docs/hails-additions/14-published-tools.md index 8ad2d517..63e8f2f8 100644 --- a/docs/hails-additions/14-published-tools.md +++ b/docs/hails-additions/14-published-tools.md @@ -86,3 +86,7 @@ vendored map extraction tools in `tools/vendor/`. - `tools/recomp-trace/bail_impacts.py`: per-bail ragdoll impacts and sound posts (BAILSTEP / BAILREG). - `tools/recomp-trace/collision_posts.py`: collision-sound posts by poster and owner, with the local-rider flag. - `tools/recomp-trace/trace.py`: knows the newer line kinds (BAIL*, COLLPOST, BANDQ, LOCALTEST). + +## Split (2026-10-04) + +The non-audio tools (`regression-checks/`, `setup-equivalence/`, `collision-inspect/`, `world-stream-inspect/`, `vault-inspect/`, `recomp-code-search/`) moved to their own PR, #37 (branch `tooling/published-tools`). The audio tools (`audio-e2e/`, `audio-bench/`, `audio-file-inspect/`, `recomp-trace/`) stay with the audio work in #32. diff --git a/docs/hails-additions/PULL-REQUESTS.md b/docs/hails-additions/PULL-REQUESTS.md index 0923e0b9..b2c32735 100644 --- a/docs/hails-additions/PULL-REQUESTS.md +++ b/docs/hails-additions/PULL-REQUESTS.md @@ -19,8 +19,8 @@ verification, open questions). | J | Feature (audio) | Game audio: setup group `audio` extracts the disc's audio (vgmstream r2117) and the native data (AEMS banks, MixMap, grains, Splice banks, vault tuning); a native port of the retail audio runtime (`crates/skate-audio`, `crates/skate-audio-fma`) plays all world and player audio; volume settings and `--mute` | [11](11-audio.md), [15](15-world-audio.md) | `gameplay/audio` (latest pushed `870940e`); upstream PR branch `audio/retail-audio` mirrors it exactly (`843cca9`). History: batch commit `f438017` "New audio who dis" (merged as `ca7b4f3`, 2026-10-02), then follow-up commits | H (branched from `gameplay/water`, #30, uses its water detection) and **#35** (50 solver iterations; needed for retail-like pops — #32 carries the same commit until #35 merges). Overlaps upstream #4/#1 (not adopted; doc 11) | Draft upstream [#32](https://github.com/SK8-ENGINE/skate-3-rust-engine/pull/32), **Work in Progress**. State 2026-10-03: native runtime is the only path; every retail player component ported; user: "basically retail". Road to ready (local plan): code-review fixes (host clock = physics steps, stray files, data tests, fallback) → world / NPC audio built out in #32 with an engine- and mod-facing hook-in surface (P0–P2 done 2026-10-03, uncommitted: map-change fix, `world_audio` components + `sdk.world_audio`, dev mod `mods/world-audio-test`; doc 15; P3–P5 next) → publish the specs (docs done 2026-10-03: `audio-specs/`; code comments still to repoint), repoint local references → moddability pass → second optimisation pass → move the non-audio tools to L → full regression check. #32's description is kept current (standing rule). Commits only when the user asks. | | G | Performance (setup) | Skip decoding duplicate stream copies; run the customiser beside the maps; parallel clothing library and pro roster, plus a native RefPack DLL in `Build.ps1` | [08](08-setup-performance.md) | `8327e94` (+ overlap hunks of `install.py` in `18f41a3`) | — | Draft upstream [#29](https://github.com/SK8-ENGINE/skate-3-rust-engine/pull/29) | | K | Gameplay fix (physics) | Board/skater solver: 50 constraint iterations (retail live value) instead of `RWMaxIterations` = 25 | [12](12-solver-iterations.md) | `physics/solver-iterations` (`3f6e989`, cherry-picked from `gameplay/audio`'s `7a9e6dc` onto `main` `7ae67f2`) | — (one-file change in `physics/settings.rs`) | Open upstream: [#35](https://github.com/SK8-ENGINE/skate-3-rust-engine/pull/35) (2026-10-03). #32 depends on it for retail-like pops and carries the same commit until #35 merges. Open question in the PR: the asset-gated `customiser_equipment_reaches_ground_force_and_torque` test (doc 12). Tests: 309 pass; the 2 failures (`pipelines_accept_…`, `sky_shader_validates`) also fail on `main`. | -| L | Tooling | Development and research tools in `tools//` (regression checks, setup equivalence, collision / world-stream / audio-file / vault readers, audio e2e and bench; recomp trace and code search, reference only) | [14](14-published-tools.md) | uncommitted (on `gameplay/audio` working tree) | `audio-e2e` / `audio-bench` / `audio-file-inspect` need J's native audio and setup modules; `setup-equivalence/compare_refpack.py` needs G's RefPack DLL; `regression-checks/check_maps.py` uses F's `--validate-maps` (falls back to `--check-assets` without it) | Not opened; the generic folders could go first, the audio ones with or after J | -| M | Feature (audio modding) | Content overlay (mods replace / add retail audio by identity), map audio as data, retail posts / globals / MixMap view for mods, observe-only audio events, docs and example mods | [16](16-audio-modding.md) | `audio/moddability` (from `gameplay/audio`; design commit `8bf4653`) | J (#32) | Draft upstream [#36](https://github.com/SK8-ENGINE/skate-3-rust-engine/pull/36), design only (2026-10-03); ~9 working days, the smaller cut chosen by the user. | +| L | Tooling | Development and research tools in `tools//` (regression checks, setup equivalence, collision / world-stream / audio-file / vault readers, audio e2e and bench; recomp trace and code search, reference only) | [14](14-published-tools.md) | uncommitted (on `gameplay/audio` working tree) | `audio-e2e` / `audio-bench` / `audio-file-inspect` need J's native audio and setup modules; `setup-equivalence/compare_refpack.py` needs G's RefPack DLL; `regression-checks/check_maps.py` uses F's `--validate-maps` (falls back to `--check-assets` without it) | Upstream [#37](https://github.com/SK8-ENGINE/skate-3-rust-engine/pull/37) (2026-10-04): the six non-audio tool folders, branch `tooling/published-tools` (`92d078f`); the audio tools stay in #32. | +| M | Feature (audio modding) | Content overlay (mods replace / add retail audio by identity), map audio as data, retail posts / globals / MixMap view for mods, observe-only audio events, docs and example mods | [16](16-audio-modding.md) | `audio/moddability` (R1–R6 `439a66f`…`f7de667`, from `gameplay/audio` via merge `4146738`) | J (#32) | Upstream [#36](https://github.com/SK8-ENGINE/skate-3-rust-engine/pull/36), **ready for review, needs testing** (2026-10-04): all phases implemented and proven headless; in-game test of the two mods pending; merges after #32. | Opened 2026-10-01 as drafts from per-PR branches cut from upstream `60efdef` (code only; `docs/hails-additions` stays in the fork). #29 (G) is stacked on diff --git a/tools/README.md b/tools/README.md index 3b62bfd8..75f0ef0b 100644 --- a/tools/README.md +++ b/tools/README.md @@ -10,15 +10,11 @@ game; none of them contain game code or data. Default work folders are under `.l | Folder | What it is for | |---|---| -| [`regression-checks/`](regression-checks/README.md) | Map validation and collision counts, baked spawns, customiser outputs against your own baselines; a muted crash smoke test per map. | -| [`setup-equivalence/`](setup-equivalence/README.md) | Prove a faster setup path gives identical bytes (district stream loader, native RefPack DLL). | -| [`collision-inspect/`](collision-inspect/README.md) | Extract a district's collision for analysis: triangles, surfaces at a point, surface ids and flags, surfaceless meshes. | -| [`world-stream-inspect/`](world-stream-inspect/README.md) | List / extract `.big` archives; survey the RW4 arenas of the district simulation streams; dump named trigger volumes. | | [`audio-file-inspect/`](audio-file-inspect/README.md) | Readers for the audio formats: ABKC banks and MOIR projects, SPLC banks, `.ems` emitter files, `.grain` data; decode bank samples. | -| [`vault-inspect/`](vault-inspect/README.md) | Look up fields in the attribute database (converted skater collections) and class layouts in the schema. | | [`audio-e2e/`](audio-e2e/README.md) | Scripted scenarios for the headless audio render and analysis of the renders (diffs, levels, voices, bus share). | | [`audio-bench/`](audio-bench/README.md) | Audio performance: hashed e2e bench runs, timing summaries, emitter bank memory. | | [`recomp-trace/`](recomp-trace/README.md) | **For use with the Skate 3 recomp's research hooks** ([`research-hooks` branch](https://github.com/Hailey-Ross/skate3recomp/tree/research-hooks)): read and analyse trace sessions. Reference only; you build the recomp and set up the paths yourself. | -| [`recomp-code-search/`](recomp-code-search/README.md) | **For use with the Skate 3 recomp's research hooks** ([`research-hooks` branch](https://github.com/Hailey-Ross/skate3recomp/tree/research-hooks)): search the recompiled sources and the memory image. Reference only; you build the recomp and set up the paths yourself. | + +The general research and regression helpers (regression checks, setup equivalence, collision / world-stream / vault inspection, recomp code search) are in their own PR, #37. See `docs/hails-additions/14-published-tools.md` for the background. diff --git a/tools/collision-inspect/README.md b/tools/collision-inspect/README.md deleted file mode 100644 index f4f2a099..00000000 --- a/tools/collision-inspect/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Collision inspection - -Extract a district's collision from your own disc and look at it, without touching the installation. -Used to place map spawns and to find the invisible trigger volumes (see -`docs/hails-additions/04-map-spawns.md` and `05-collision-volumes.md`). - -| Script | What it does | -|---|---| -| `map_collision.py` | Extracts `world.big` into `//raw` (once), runs the map converter's collision pass and saves every triangle to `collision.npy` (N x 3 x 3 float64, Y-up metres). Prints the triangle count, bounds and the spawn the current spawn rule would choose. | -| `analyse_spawn.py` | Lists every collision surface at a point's X/Z (height, normal-up sign, area), plus the area-weighted centre of the flat surfaces. Nothing below a spawn = void; a much lower surface under it = roof or platform. Needs `map_collision.py` first. | -| `collision_attrs.py` | Saves the triangles with their surface id, unit flags (0x80 = has a surface id), mesh flags and mesh index to `collision_attrs.npy`. Needs the extracted district. | -| `scan_surfaceless.py` | For each district: counts collision meshes that are surfaced, mixed or surfaceless (no surface ids at all) and lists the non-surfaced ones with their size and bounds. | -| `unsigned_collision.py` | Imported by the others: makes the vendored collision decoder read 16-bit vertex deltas unsigned, as the game does. Without it, clusters spanning more than 32.8 m wrap by 65.536 m. | - -## Inputs - -- Your extracted disc: `--disc DIR` (the folder holding `data/`), default `$SKATE3_DISC` or `.local/skate3-disc`. -- District names without the `DIST_` prefix: `MegaPark`, `StartPark`, `BlackBoxPark`, `SkateSchool`, - `IndustrialSkatePark`, `DownTownSkatePark`, `MaloofMoneyCup`, `University`, `Industrial`, `DownTown`. -- Work folder: `--work DIR`, default `.local/collision` (gitignored). - -## Usage - -``` -py -3.13 tools/collision-inspect/map_collision.py MegaPark -py -3.13 tools/collision-inspect/analyse_spawn.py MegaPark 21.7 21.6 -19.7 -py -3.13 tools/collision-inspect/collision_attrs.py SkateSchool -py -3.13 tools/collision-inspect/scan_surfaceless.py SkateSchool StartPark -``` - -Parks take about a second; the city districts take minutes and a lot of memory. - -## Example output - -(made-up values) - -``` -triangles 12000 -bounds min [-120.0 -5.2 -130.0] max [140.0 40.1 125.0] -spawn (current rule) [10.5 0.1 6.0] - -MapA surfaces at (21.7, -19.7) : [(0.0, 1.0, 12.5), (6.3, 1.0, 3.1)] -``` - -## Requirements - -Python 3.13 with `numpy`; the repository's setup modules (`tools/asset_pipeline`) and the vendored -map extraction tools (`tools/vendor/university`, `tools/vendor/utt`). diff --git a/tools/collision-inspect/analyse_spawn.py b/tools/collision-inspect/analyse_spawn.py deleted file mode 100644 index 3c87f1db..00000000 --- a/tools/collision-inspect/analyse_spawn.py +++ /dev/null @@ -1,47 +0,0 @@ -"""List every collision surface height at a spawn's XZ (orientation ignored). - -Usage (repo root, after map_collision.py): - py -3.13 tools/collision-inspect/analyse_spawn.py MegaPark 21.742 21.605 -19.683 [--work DIR] -Output: (height, normal-y sign, area m2) for each surface under/over the point, plus the -area-weighted centre of flat surfaces. Nothing below the spawn height = void; a much -lower surface under it = roof/platform. -""" -import argparse -from pathlib import Path -import numpy as np - -REPO = Path(__file__).resolve().parents[2] -parser = argparse.ArgumentParser(description="List every collision surface height at a point's XZ.") -parser.add_argument('district', help='district name without DIST_') -parser.add_argument('x', type=float) -parser.add_argument('y', type=float, help='ignored (every height at x, z is listed)') -parser.add_argument('z', type=float) -parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='work folder (default: %(default)s)') -args = parser.parse_args() -name, x, z = args.district, args.x, args.z -t = np.load(args.work / name / 'collision.npy') -a, b, c = t[:, 0], t[:, 1], t[:, 2] -ab, ac = b - a, c - a -cross = np.cross(ab, ac) -area = np.linalg.norm(cross, axis=1) / 2 -up = cross[:, 1] / np.maximum(2 * area, 1e-9) - -def surfaces(px, pz): - dx, dz = px - a[:, 0], pz - a[:, 2] - det = ab[:, 0] * ac[:, 2] - ac[:, 0] * ab[:, 2] - ok = np.abs(det) > 1e-12 - d = np.where(ok, det, 1.0) - u = (dx * ac[:, 2] - ac[:, 0] * dz) / d - v = (ab[:, 0] * dz - dx * ab[:, 2]) / d - inside = ok & (u >= -1e-5) & (v >= -1e-5) & (u + v <= 1.00001) - height = a[:, 1] + u * ab[:, 1] + v * ac[:, 1] - return sorted({(round(float(height[i]), 2), round(float(up[i]), 2), round(float(area[i]), 1)) - for i in np.nonzero(inside)[0]}) - -print(name, 'surfaces at', (x, z), ':', surfaces(x, z)) -centre = (a + b + c) / 3 -flat = (np.abs(up) >= .9) & (area >= 2) -w = area[flat] -cx = float((centre[flat, 0] * w).sum() / w.sum()) -cz = float((centre[flat, 2] * w).sum() / w.sum()) -print('flat-area centre', (round(cx, 1), round(cz, 1)), ':', surfaces(cx, cz)[:6]) diff --git a/tools/collision-inspect/collision_attrs.py b/tools/collision-inspect/collision_attrs.py deleted file mode 100644 index 42925405..00000000 --- a/tools/collision-inspect/collision_attrs.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Capture a district's collision triangles with surface ids, unit flags and mesh flags. - -Usage (repo root, after map_collision.py extracted the district): - py -3.13 tools/collision-inspect/collision_attrs.py SkateSchool [--work DIR] -Writes //collision_attrs.npy (default work .local/collision) with rows: - 9 vertex floats, surface id, unit flags (0x80 = has surface id), mesh flags, mesh index. -""" -import argparse, sys -from pathlib import Path -import numpy as np - -REPO = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(REPO)) -from tools.asset_pipeline import install -sys.path.insert(0, str(install.TOOLS / 'vendor/university/tools/vanilla_map_extraction/tools')) -from prepare_hawaiian_dream import prepare -from prepare_university import EXCLUDED_NORMAL_TEXTURE_IDS - -parser = argparse.ArgumentParser(description="Capture a district's collision triangles with surface ids and flags.") -parser.add_argument('district', nargs='?', default='SkateSchool', help='district name without DIST_ (default: %(default)s)') -parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='work folder (default: %(default)s)') -args = parser.parse_args() -name = args.district -district = 'DIST_' + name -work = args.work / name -stream = work / 'raw/data/content/world/stream' / district -rows = [] -mesh_index = [0] -def consume(meshes): - for mesh in meshes: - for t in mesh.triangles: - rows.append((*t.a, *t.b, *t.c, t.surface, t.unit_flags, mesh.mesh_flags, mesh_index[0])) - mesh_index[0] += 1 -prepare(stream_directory=stream, output_root=work / 'intermediate2', utt_root=install.TOOLS / 'vendor/utt', - district_name=district, map_name=name, package_name='analysis', cache_format='skate3-rust-map-v1', - texture_stream_names=('Tex',) if any(stream.glob('cTex_*.xsf')) else (), - excluded_normal_texture_ids=EXCLUDED_NORMAL_TEXTURE_IDS, raw_texture_cache=True, - collision_consumer=consume, write_render_sources=False) -a = np.asarray(rows, dtype=np.float64) -np.save(work / 'collision_attrs.npy', a) -print('triangles', len(a), 'meshes', mesh_index[0]) diff --git a/tools/collision-inspect/map_collision.py b/tools/collision-inspect/map_collision.py deleted file mode 100644 index f3530e3e..00000000 --- a/tools/collision-inspect/map_collision.py +++ /dev/null @@ -1,56 +0,0 @@ -"""Extract one district's collision triangles for analysis (does not touch the install). - -Usage (repo root): py -3.13 tools/collision-inspect/map_collision.py MegaPark [--disc DIR] [--work DIR] -Extracts world.big from the disc into //raw if needed, then writes -//collision.npy (N x 3 x 3 float64, Y-up metres) and -prints the bounds and the spawn the current map_writer rule would choose. -Parks take ~1 s; city districts take minutes and a lot of memory. -""" -import argparse -import os -import sys -from pathlib import Path -import numpy as np - -REPO = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(REPO)) -from tools.asset_pipeline import install -from tools.asset_pipeline.map_writer import SpawnSelector -sys.path.insert(0, str(install.TOOLS / 'vendor/university/tools/vanilla_map_extraction/tools')) -sys.path.insert(0, str(Path(__file__).resolve().parent)) -import unsigned_collision # noqa: F401 (unsigned vertex deltas, as the game reads them) -from prepare_hawaiian_dream import prepare -from prepare_university import EXCLUDED_NORMAL_TEXTURE_IDS - -parser = argparse.ArgumentParser(description="Extract one district's collision triangles for analysis.") -parser.add_argument('district', nargs='?', default='MegaPark', help='district name without DIST_ (default: %(default)s)') -parser.add_argument('--disc', type=Path, default=Path(os.environ.get('SKATE3_DISC', REPO / '.local/skate3-disc')), - help='extracted disc root, the folder holding data/ (default: %(default)s)') -parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='work folder (default: %(default)s)') -args = parser.parse_args() -name = args.district -district = 'DIST_' + name -archive = args.disc / 'data/content' / f'world{district}.big' -work = args.work / name -if not (work / 'raw').is_dir(): - install.extract(archive, work / 'raw') -stream = work / 'raw/data/content/world/stream' / district -spawn = SpawnSelector(district) -triangles = [] - -def consume(meshes): - spawn.consider(meshes) - for mesh in meshes: - triangles.extend((t.a, t.b, t.c) for t in mesh.triangles) - -prepare(stream_directory=stream, output_root=work / 'intermediate', utt_root=install.TOOLS / 'vendor/utt', - district_name=district, map_name=name, package_name='analysis', cache_format='skate3-rust-map-v1', - texture_stream_names=('Tex',) if any(stream.glob('cTex_*.xsf')) else (), - excluded_normal_texture_ids=EXCLUDED_NORMAL_TEXTURE_IDS, raw_texture_cache=True, - collision_consumer=consume, write_render_sources=False) -t = np.asarray(triangles, dtype=np.float64) -np.save(work / 'collision.npy', t) -points = t.reshape(-1, 3) -print('triangles', len(t)) -print('bounds min', points.min(0).round(1), 'max', points.max(0).round(1)) -print('spawn (current rule)', np.round(spawn.result(name), 3)) diff --git a/tools/collision-inspect/scan_surfaceless.py b/tools/collision-inspect/scan_surfaceless.py deleted file mode 100644 index ab4d52ac..00000000 --- a/tools/collision-inspect/scan_surfaceless.py +++ /dev/null @@ -1,60 +0,0 @@ -"""For every district: find collision meshes with no surface ids and describe them. - -Usage (repo root): py -3.13 tools/collision-inspect/scan_surfaceless.py [District ...] [--disc DIR] [--work DIR] -Extracts each district into // (default .local/collision) if needed (cities take minutes), -then prints per-mesh counts: surfaced / mixed (some units lack surface ids) / surfaceless. -""" -import argparse, collections, json, os, sys -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(REPO)) -from tools.asset_pipeline import install -TOOLS = install.TOOLS / 'vendor/university/tools/vanilla_map_extraction/tools' -sys.path.insert(0, str(TOOLS)) -from prepare_hawaiian_dream import prepare -from prepare_university import EXCLUDED_NORMAL_TEXTURE_IDS -sys.path.insert(0, str(Path(__file__).resolve().parent)) -import unsigned_collision # noqa: F401 (unsigned vertex deltas, as the game reads them) -from retail_collision_mesh import decode_rx2_clustered_meshes - -parser = argparse.ArgumentParser(description='Find collision meshes with no surface ids and describe them.') -parser.add_argument('districts', nargs='*', help='district names without DIST_ (default: all ten)') -parser.add_argument('--disc', type=Path, default=Path(os.environ.get('SKATE3_DISC', REPO / '.local/skate3-disc')), - help='extracted disc root, the folder holding data/ (default: %(default)s)') -parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='work folder (default: %(default)s)') -args = parser.parse_args() -names = args.districts or ['MegaPark', 'IndustrialSkatePark', 'StartPark', 'BlackBoxPark', 'DownTownSkatePark', - 'MaloofMoneyCup', 'SkateSchool', 'University', 'Industrial', 'DownTown'] -for name in names: - district = 'DIST_' + name - work = args.work / name - if not (work / 'raw').is_dir(): - install.extract(args.disc / f'data/content/world{district}.big', work / 'raw') - stream = work / 'raw/data/content/world/stream' / district - out = work / 'scan' - if not (out / 'manifest.json').is_file(): - prepare(stream_directory=stream, output_root=out, utt_root=install.TOOLS / 'vendor/utt', - district_name=district, map_name=name, package_name='analysis', cache_format='skate3-rust-map-v1', - texture_stream_names=('Tex',) if any(stream.glob('cTex_*.xsf')) else (), - excluded_normal_texture_ids=EXCLUDED_NORMAL_TEXTURE_IDS, raw_texture_cache=True, - collision_consumer=lambda meshes: None, write_render_sources=False) - manifest = json.loads((out / 'manifest.json').read_text()) - total = collections.Counter() - special = [] - for entry in manifest['simulation_assets']: - if not entry.get('collision_meshes'): - continue - for mesh in decode_rx2_clustered_meshes((out / entry['rx2']).read_bytes()): - flags = {bool(t.unit_flags & 0x80) for t in mesh.triangles} - kind = 'surfaced' if flags == {True} else 'surfaceless' if flags == {False} else 'mixed' - total[kind] += 1 - if kind != 'surfaced': - size = [round(mesh.bounds_max[k] - mesh.bounds_min[k], 1) for k in range(3)] - lo = [round(v, 2) for v in mesh.bounds_min] - hi = [round(v, 2) for v in mesh.bounds_max] - special.append(f"{kind} {entry['stream_file']} tris={len(mesh.triangles)} size={size} " - f"min={lo} max={hi}") - print(f'== {name}: {dict(total)}', flush=True) - for line in special: - print(' ', line, flush=True) diff --git a/tools/collision-inspect/unsigned_collision.py b/tools/collision-inspect/unsigned_collision.py deleted file mode 100644 index e3ffdd65..00000000 --- a/tools/collision-inspect/unsigned_collision.py +++ /dev/null @@ -1,31 +0,0 @@ -"""Make the vendored collision decoder read 16-bit vertex deltas UNSIGNED, as the game does. - -The vendored `retail_collision_mesh._decode_vertices` unpacks compression-1 vertex deltas with `>3h` -(signed). The game's own decoder (crates/skate-data/src/retail_collision.rs, `be16` → u16) treats them as -unsigned, as the game does. With signed deltas, vertices of clusters spanning more than 32.8 m wrap by -65.536 m (granularity 0.001); see docs/hails-additions/05-collision-volumes.md. - -Analysis tools only: `import unsigned_collision` before using `decode_rx2_clustered_meshes`. The converter -(map_writer.spawn_point, prepare_hawaiian_dream bounds) still uses the vendored behaviour. -""" -import struct - -import retail_collision_mesh as _rcm - - -def _decode_vertices_unsigned(cluster, vertex_count, compression, granularity, vertex_data_end): - if compression != 1: - return _original(cluster, vertex_count, compression, granularity, vertex_data_end) - _rcm._require(vertex_data_end >= 28, "compressed cluster omits its base") - base = struct.unpack_from(">3i", cluster, _rcm.CLUSTER_HEADER_SIZE) - vertices = [] - for index in range(vertex_count): - offset = _rcm.CLUSTER_HEADER_SIZE + 12 + index * 6 - _rcm._require(offset + 6 <= vertex_data_end, "16-bit compressed vertex extends into the unit stream") - delta = struct.unpack_from(">3H", cluster, offset) - vertices.append(tuple(_rcm._f32((base[axis] + delta[axis]) * granularity) for axis in range(3))) - return vertices - - -_original = _rcm._decode_vertices -_rcm._decode_vertices = _decode_vertices_unsigned diff --git a/tools/recomp-code-search/README.md b/tools/recomp-code-search/README.md deleted file mode 100644 index c8a9085b..00000000 --- a/tools/recomp-code-search/README.md +++ /dev/null @@ -1,55 +0,0 @@ -# Recomp code search - -**For use with the Skate 3 recomp's research hooks:** -[Hailey-Ross/skate3recomp, branch `research-hooks`](https://github.com/Hailey-Ross/skate3recomp/tree/research-hooks) -(described in [`docs/hails-additions/13-recomp-research-hooks.md`](../../docs/hails-additions/13-recomp-research-hooks.md)). - -> **Reference only.** These scripts search the recompiled sources and the game's memory image that -> *you* generate from your own legally owned copy of the game. You must build the recomp yourself and -> set up the paths below for your machine; nothing from the game is included here. Use what you find -> as a reference for re-implementing behaviour, never copy game code into the engine. - -Finding the code behind a behaviour is the first step before hooking it (see the research-hooks -branch for the hooks themselves). - -| Script | What it does | -|---|---| -| `fn.sh ADDR` | Prints the disassembly comments of the recompiled function `sub_ADDR`. | -| `grepfn.sh 'ERE'` | Prints `sub_XXXXXXXX: ` for every disassembly line matching the pattern, with its function. | -| `callctx.sh ADDR [before] [after]` | Every call site of `sub_ADDR` (first 20), with the surrounding disassembly and the calling function. | -| `ppcxref.py` | Cross-references in a big-endian PowerPC memory image: code that builds an address (`lis` + `addi` / load / store), strings and their users, callers of a function, the function containing an address, and data words pointing at an address (vtables, tables). | - -## Inputs - -- `RECOMP_GENERATED`: your skate3recomp build's `generated/` folder (the code generator's output, - `skate3_recomp.*.cpp`), for the shell scripts. -- For `ppcxref.py`: `--image` (or `PPC_IMAGE`), a dump of the executable's loaded memory image from - your own copy, base `0x82000000` by default (`--base`); `--funcs` (or `PPC_FUNCS`), a list of function - starts built from your generated sources (the command is in the script's help); optional - `--code LO-HI` to scan only the code section. - -## Usage - -``` -export RECOMP_GENERATED=/path/to/skate3recomp/generated -sh tools/recomp-code-search/fn.sh 82XXXXXX -sh tools/recomp-code-search/grepfn.sh 'lfs .*,-?[0-9]+\(r31\)' -sh tools/recomp-code-search/callctx.sh 82XXXXXX 6 14 - -py -3.13 tools/recomp-code-search/ppcxref.py --image image.bin --funcs funcs.txt str some_string -py -3.13 tools/recomp-code-search/ppcxref.py --image image.bin --funcs funcs.txt calls 82XXXXXX -``` - -## Example output - -`ppcxref.py calls` (made-up addresses): - -``` -82123450 in sub_82123400 -82345670 in sub_82345600 -``` - -## Requirements - -Git Bash or any POSIX shell with `grep` / `awk` for the shell scripts; Python 3.13 (standard library) -for `ppcxref.py`. diff --git a/tools/recomp-code-search/callctx.sh b/tools/recomp-code-search/callctx.sh deleted file mode 100644 index 4141d16f..00000000 --- a/tools/recomp-code-search/callctx.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/sh -# usage: RECOMP_GENERATED= callctx.sh ADDR [before=6] [after=14] -> every call site of sub_X (first 20) -# with the surrounding asm and the calling function -B=${2:-6}; A=${3:-14} -# Needs your own skate3recomp build: RECOMP_GENERATED = its generated/ folder (the code generator's output). -: "${RECOMP_GENERATED:?set RECOMP_GENERATED to your skate3recomp generated/ folder}" -cd "$RECOMP_GENERATED" || exit 1 -x=$(echo "$1" | tr 'A-F' 'a-f') -grep -n -E "DEFINE_REX_FUNC\(sub_|^ // bl 0x$x\$" skate3_recomp.*.cpp | awk -F: '/DEFINE_REX_FUNC/{f=$1; fn=$0; next} {print f":"$2}' | head -20 | while IFS=: read file line; do - fnl=$(awk -v L=$line 'NR<=L && /DEFINE_REX_FUNC/{l=$0} NR==L{print l; exit}' $file | grep -o 'sub_[0-9A-F]*') - echo "=== in $fnl" - awk -v L=$line -v B=$B -v A=$A 'NR>=L-B*4 && NR<=L+A*4 && (/^\t\/\/ /||/^loc_/){sub(/^\t\/\/ /," ");print}' $file -done diff --git a/tools/recomp-code-search/fn.sh b/tools/recomp-code-search/fn.sh deleted file mode 100644 index fbd3802e..00000000 --- a/tools/recomp-code-search/fn.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/sh -# usage: RECOMP_GENERATED= fn.sh ADDR -> prints the asm comments of function sub_ADDR (8 hex digits, upper case) -# Needs your own skate3recomp build: RECOMP_GENERATED = its generated/ folder (the code generator's output). -: "${RECOMP_GENERATED:?set RECOMP_GENERATED to your skate3recomp generated/ folder}" -cd "$RECOMP_GENERATED" || exit 1 -f=$(grep -l "DEFINE_REX_FUNC(sub_$1)" skate3_recomp.*.cpp | head -1) -awk -v s="DEFINE_REX_FUNC(sub_$1)" 'index($0,s){p=1;print;next} p&&/^DEFINE_REX_FUNC/{exit} p&&(/^\t\/\/ /||/^loc_/){sub(/^\t\/\/ /," ");print}' "$f" diff --git a/tools/recomp-code-search/grepfn.sh b/tools/recomp-code-search/grepfn.sh deleted file mode 100644 index f15615a3..00000000 --- a/tools/recomp-code-search/grepfn.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/sh -# usage: RECOMP_GENERATED= grepfn.sh 'ERE' -> prints "sub_XXXX: asm line" for every asm line matching (grep -E syntax) -# Needs your own skate3recomp build: RECOMP_GENERATED = its generated/ folder (the code generator's output). -: "${RECOMP_GENERATED:?set RECOMP_GENERATED to your skate3recomp generated/ folder}" -cd "$RECOMP_GENERATED" || exit 1 -grep -n -E "DEFINE_REX_FUNC\(sub_|^ // .*($1)" skate3_recomp.*.cpp | awk ' -/DEFINE_REX_FUNC\(sub_/ { match($0,/sub_[0-9A-F]+/); f=substr($0,RSTART,RLENGTH); next } -{ sub(/^[^\t]*\t\/\/ /,""); print f": "$0 }' diff --git a/tools/recomp-code-search/ppcxref.py b/tools/recomp-code-search/ppcxref.py deleted file mode 100644 index 3a782065..00000000 --- a/tools/recomp-code-search/ppcxref.py +++ /dev/null @@ -1,167 +0,0 @@ -"""Cross-reference helper for a big-endian PowerPC memory image of the game (the Xbox 360 executable's -loaded image, which you dump yourself from your own copy; base address 0x82000000 by default). - -usage: - py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS addr ADDR [ADDR ...] # code that builds the address (lis + addi/load/store) - py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS str TEXT [TEXT ...] # find strings, then xref each hit - py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS calls ADDR [ADDR ...] # callers (bl) of a function - py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS func ADDR [ADDR ...] # containing function start - py -3.13 tools/recomp-code-search/ppcxref.py --image IMG --funcs FUNCS ptr ADDR [ADDR ...] # data words equal to the address - # (vtables, binding tables) -Options: --base (image base, default 0x82000000), --code LO-HI (code range to scan, hex; default the whole image). -Environment: PPC_IMAGE, PPC_FUNCS stand in for --image / --funcs. - -FUNCS lists function starts, one hex address first on each line. Build it from your recomp's generated -sources: - cd /generated && grep -o "DEFINE_REX_FUNC(sub_[0-9A-F]*)" skate3_recomp.*.cpp \\ - | sed 's/:DEFINE_REX_FUNC(sub_/ /;s/)//' | awk '{print $2, $1}' | sort > funcs.txt -Heuristic: a lis value is valid for 48 instructions in the same function. -""" -import argparse -import bisect -import os -import re -import struct -import sys -from pathlib import Path - -IMG = b'' -BASE = 0x82000000 -CODE_LO = CODE_HI = 0 -FUNCS = [] - -# D-form opcodes whose rA+simm is an address: addi 14, lwz 32, lwzu 33, lbz 34, stw 36, stb 38, lhz 40, -# sth 44, lfs 48, lfd 50, stfs 52, stfd 54, ld/ldu 58, std 62 -DFORM = {14, 32, 33, 34, 35, 36, 37, 38, 39, 40, 42, 44, 48, 50, 52, 54, 58, 62} - - -def func_of(a): - i = bisect.bisect_right(FUNCS, a) - 1 - return FUNCS[i] if i >= 0 else 0 - - -_W = None - - -def W(): - global _W - if _W is None: - n = (CODE_HI - CODE_LO) // 4 - _W = struct.unpack_from('>%dI' % n, IMG, CODE_LO - BASE) - return _W - - -def xrefs(targets, slack=0): - """targets: addresses (or (lo, hi) ranges). Returns a list of (pc, ea, func).""" - w = W() - out = [] - ranges = [(t, t + slack) if isinstance(t, int) else t for t in targets] - lis = {} - fstarts = set(FUNCS) - for i, ins in enumerate(w): - pc = CODE_LO + 4 * i - if pc in fstarts: - lis.clear() - op = ins >> 26 - rd = (ins >> 21) & 31 - ra = (ins >> 16) & 31 - imm = ins & 0xFFFF - simm = imm - 0x10000 if imm & 0x8000 else imm - if op == 15 and ra == 0: # lis - lis[rd] = ((simm << 16) & 0xFFFFFFFF, i) - continue - if op in DFORM and ra in lis and i - lis[ra][1] < 48: - d = simm & ~3 if op in (58, 62) else simm - ea = (lis[ra][0] + d) & 0xFFFFFFFF - for lo, hi in ranges: - if lo <= ea <= hi: - out.append((pc, ea, func_of(pc))) - break - if op not in (14, 15) and rd in lis and op in (14, 32, 33, 34, 40, 58, 31): - lis.pop(rd, None) # destination overwritten - return out - - -def callers(target): - out = [] - for i, ins in enumerate(W()): - if ins >> 26 == 18 and ins & 1: # bl - li = ins & 0x03FFFFFC - if li & 0x02000000: - li -= 0x04000000 - pc = CODE_LO + 4 * i - t = li if ins & 2 else pc + li - if t & 0xFFFFFFFF == target: - out.append((pc, func_of(pc))) - return out - - -def find_str(s): - return [m.start() + BASE for m in re.finditer(re.escape(s.encode('latin1')), IMG)] - - -def cstr(a): - o = a - BASE - return IMG[o:IMG.index(b'\0', o)].decode('latin1') - - -def ptrs(a): - b = struct.pack('>I', a) - return [m.start() + BASE for m in re.finditer(re.escape(b), IMG) if m.start() % 4 == 0] - - -def main(): - global IMG, BASE, CODE_LO, CODE_HI, FUNCS - parser = argparse.ArgumentParser(description='Cross-reference helper for a big-endian PowerPC memory image.', - epilog=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument('cmd', choices=('addr', 'str', 'calls', 'func', 'ptr')) - parser.add_argument('args', nargs='+') - parser.add_argument('--image', default=os.environ.get('PPC_IMAGE'), help='memory image file') - parser.add_argument('--funcs', default=os.environ.get('PPC_FUNCS'), help='function start list') - parser.add_argument('--base', default='0x82000000', help='image base address (hex)') - parser.add_argument('--code', help='code range LO-HI (hex); default the whole image') - a = parser.parse_args() - if not a.image or not a.funcs: - parser.error('--image and --funcs (or PPC_IMAGE / PPC_FUNCS) are required') - IMG = Path(a.image).read_bytes() - BASE = int(a.base, 16) - if a.code: - lo, hi = a.code.split('-') - CODE_LO, CODE_HI = int(lo, 16), int(hi, 16) - else: - CODE_LO, CODE_HI = BASE, BASE + len(IMG) // 4 * 4 - FUNCS = sorted(int(l.split()[0], 16) for l in Path(a.funcs).read_text().splitlines() if l.strip()) - if a.cmd == 'addr': - for pc, ea, f in xrefs([int(x, 16) for x in a.args]): - print(f'{pc:08X} -> {ea:08X} in sub_{f:08X}') - elif a.cmd == 'str': - starts = set() - for s in a.args: - for h in find_str(s): - o = h - BASE - while o > 0 and 0x20 <= IMG[o - 1] < 0x7f: - o -= 1 - starts.add(o + BASE) - res = xrefs(sorted(starts)) - for st in sorted(starts): - print(f'{st:08X} "{cstr(st)[:80]}"') - for pc, ea, f in res: - if ea == st: - print(f' {pc:08X} in sub_{f:08X}') - elif a.cmd == 'calls': - for x in a.args: - for pc, f in callers(int(x, 16)): - print(f'{pc:08X} in sub_{f:08X}') - elif a.cmd == 'func': - for x in a.args: - print(f'sub_{func_of(int(x, 16)):08X}') - elif a.cmd == 'ptr': - for x in a.args: - addr = int(x, 16) - for p in ptrs(addr): - nb = struct.unpack_from('>4I', IMG, p - BASE - 4) - print(f'{addr:08X} @ {p:08X} [-4..+12] ' + ' '.join(f'{v:08X}' for v in nb)) - - -if __name__ == '__main__': - sys.exit(main()) diff --git a/tools/regression-checks/README.md b/tools/regression-checks/README.md deleted file mode 100644 index 7f9880a9..00000000 --- a/tools/regression-checks/README.md +++ /dev/null @@ -1,64 +0,0 @@ -# Regression checks - -Checks to run after changing the asset pipeline, the converter, input, native dependencies, build -scripts or game code, before calling a change done. Each one compares the current install or build -against a baseline you record yourself on a known-good state. - -| Script | What it checks | -|---|---| -| `check_maps.py` | Loads every installed map in one game process (`skate3rust --validate-maps`) and compares each map's loaded collision triangle count with the baseline. Fails on any map error or warning (spawn support, startup, invisible floor). Falls back to one `--check-assets` launch per map on builds without `--validate-maps`. | -| `check_spawns.py` | Reads the spawn baked into every installed `.skate` map and compares it with the baseline (0.01 m tolerance). | -| `check_customiser.py` | Snapshots the character customiser outputs (every stage receipt: path, size, SHA-256 of each file) and compares a later install with it. Ignores differences that are only the random set id embedded in JSON paths. | -| `smoke_maps.sh` | Muted crash smoke test: launches the game on each map for N seconds (windowed, `--mute`) and counts panics, `ERROR` lines, non-finite physics and render-ready lines in the log. | - -## Inputs - -- An installation set up by `tools/setup.py` (`data/installation.json`, `data/installations//`). -- `bin/skate3rust.exe` for `check_maps.py` (`--exe` for another build) and `smoke_maps.sh` (`SKATE_EXE`). -- Baselines live in `.local/regression/` (gitignored) unless you pass `--baseline`; the customiser - snapshot in `.local/customiser-baseline/` (`--snapshot`). - -## Usage - -From the repository root: - -``` -# once, on a known-good build / install -py -3.13 tools/regression-checks/check_maps.py --update -py -3.13 tools/regression-checks/check_spawns.py --update -py -3.13 tools/regression-checks/check_customiser.py --save - -# after a change -py -3.13 tools/regression-checks/check_maps.py -py -3.13 tools/regression-checks/check_spawns.py -py -3.13 tools/regression-checks/check_customiser.py - -# smoke test (Git Bash): 25 s per map, default maps StartPark University DownTown MegaPark -bash tools/regression-checks/smoke_maps.sh 25 StartPark Industrial -``` - -Every check exits 1 on a difference. The smoke test opens a game window per map; don't run it while -someone is playing on the same machine. - -## Example output - -`check_maps.py` (made-up numbers): - -``` -TEST_WORLD ... -MapA 12345 ok (0.41s) -MapB 67890 CHANGED (baseline 67880, delta 10) -``` - -`smoke_maps.sh` (from a real run; exit 124 = stopped by the timeout, as intended): - -``` -StartPark exit=124 panics=0 errors=0 nonfinite=0 render_ready=1 native_lines=7 -University exit=124 panics=0 errors=0 nonfinite=0 render_ready=1 native_lines=7 -logs: /.local/smoke/20261003_090413 -``` - -## Requirements - -- Windows, Python 3.13 (standard library only). -- `smoke_maps.sh`: Git Bash (`timeout`, `taskkill`). diff --git a/tools/regression-checks/check_customiser.py b/tools/regression-checks/check_customiser.py deleted file mode 100644 index 08032c0a..00000000 --- a/tools/regression-checks/check_customiser.py +++ /dev/null @@ -1,82 +0,0 @@ -"""Snapshot / compare the character customiser outputs of the current installation. - -Usage (repo root): - py -3.13 tools/regression-checks/check_customiser.py --save # snapshot the current install - py -3.13 tools/regression-checks/check_customiser.py # compare current install to snapshot - -Snapshot (default .local/customiser-baseline/, --snapshot DIR): every stage receipt (*-complete.json: relative path -> -size + sha256 of each output file), every JSON file of the set, and the set id. JSON files may embed -the random set id in paths, so a JSON file whose hash differs is compared again with the set id -replaced. Exit 1 on any difference. Use after changing anything the customiser could touch -(stage scheduling, shared parsers, PIL/numpy versions, ...). -""" -import argparse -import json -import shutil -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -SNAPSHOT = REPO / '.local/customiser-baseline' - - -def current_set(): - marker = json.loads((REPO / 'data/installation.json').read_text()) - custom = REPO / 'data' / marker['directory'] / 'assets/private/customisation' - set_id = json.loads((custom / 'current.json').read_text())['set'] - return set_id, custom / 'sets' / set_id - - -def receipts(directory): - files = {} - for path in sorted(directory.glob('*-complete.json')): - files.update(json.loads(path.read_text())['files']) - return files - - -def save(): - set_id, directory = current_set() - if SNAPSHOT.exists(): - shutil.rmtree(SNAPSHOT) - (SNAPSHOT / 'json').mkdir(parents=True) - for path in directory.glob('*-complete.json'): - shutil.copy2(path, SNAPSHOT / path.name) - for path in directory.rglob('*.json'): - target = SNAPSHOT / 'json' / path.relative_to(directory) - target.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(path, target) - (SNAPSHOT / 'SET_ID').write_text(set_id) - print(f'snapshot of set {set_id}: {len(receipts(directory))} receipted files') - return 0 - - -def compare(): - set_id, directory = current_set() - if not (SNAPSHOT / 'SET_ID').is_file(): - print(f'no snapshot in {SNAPSHOT}: run once with --save on a known-good install') - return 1 - old_id = (SNAPSHOT / 'SET_ID').read_text().strip() - old, new = receipts(SNAPSHOT), receipts(directory) - problems = [] - for name in sorted(set(old) | set(new)): - if name not in new or name not in old: - problems.append(f'{"missing" if name not in new else "new"}: {name}') - elif old[name]['sha256'] != new[name]['sha256']: - saved = SNAPSHOT / 'json' / name - if name.endswith('.json') and saved.is_file(): - if saved.read_text(encoding='utf-8').replace(old_id, set_id) == (directory / name).read_text(encoding='utf-8'): - continue # differs only by the embedded set id - problems.append(f'changed: {name}') - print(f'compared {len(old)} snapshot files with {len(new)} current files (set {old_id} -> {set_id})') - for problem in problems[:40]: - print(' ' + problem) - print('IDENTICAL' if not problems else f'{len(problems)} DIFFERENCES') - return 1 if problems else 0 - - -if __name__ == '__main__': - parser = argparse.ArgumentParser(description='Snapshot / compare the character customiser outputs of the current installation.') - parser.add_argument('--save', action='store_true', help='snapshot the current install (else compare with the snapshot)') - parser.add_argument('--snapshot', type=Path, default=SNAPSHOT, help='snapshot folder (default: %(default)s)') - args = parser.parse_args() - SNAPSHOT = args.snapshot - raise SystemExit(save() if args.save else compare()) diff --git a/tools/regression-checks/check_maps.py b/tools/regression-checks/check_maps.py deleted file mode 100644 index 8ea0d1e7..00000000 --- a/tools/regression-checks/check_maps.py +++ /dev/null @@ -1,92 +0,0 @@ -"""Validate every installed map in ONE game process and compare against baselines. - -Usage (repo root, after building/staging bin\\skate3rust.exe): - py -3.13 tools/regression-checks/check_maps.py --update # first, on a known-good build: record the baseline - py -3.13 tools/regression-checks/check_maps.py # later: compare against it - -Uses `skate3rust --validate-maps` (crates/skate-game/src/map_validation.rs): TEST_WORLD plus -each map. Fails if a map is not ok, has warnings (spawn support / startup / invisible floor), -or its loaded collision triangle count differs from the baseline -(default .local/regression/collision_baseline.json; --baseline PATH, --exe PATH). -Falls back to one `--check-assets` launch per map if the exe has no --validate-maps. -""" -import argparse -import json -import re -import subprocess -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -BASELINE = REPO / '.local/regression/collision_baseline.json' - - -def validate(exe, maps): - process = subprocess.Popen([str(exe), '--assets', str(REPO / 'assets'), '--validate-maps'], - stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, - text=True, cwd=REPO) - if not process.stdout.readline().startswith('SKATE_VALIDATOR_READY'): - process.kill() - return None - results = {} - for request in ['TEST_WORLD', *map(str, maps)]: - process.stdin.write(request + '\n'); process.stdin.flush() - line = process.stdout.readline() - if not line.startswith('SKATE_MAP_CHECK '): - raise RuntimeError(f'validator stopped at {request}') - result = json.loads(line[len('SKATE_MAP_CHECK '):]) - results['TEST_WORLD' if request == 'TEST_WORLD' else Path(request).stem] = result - process.stdin.close(); process.wait() - return results - - -def check_assets(exe, maps): - results = {} - for path in maps: - run = subprocess.run([str(exe), '--assets', str(REPO / 'assets'), '--map', str(path), '--check-assets'], - capture_output=True, text=True, errors='replace', cwd=REPO) - match = re.search(r'SKATE_RWCM_READY triangles=(\d+)', run.stdout + run.stderr) - results[path.stem] = {'ok': run.returncode == 0 and 'SKATE_ASSETS_READY' in run.stdout + run.stderr, - 'errors': [], 'warnings': [], 'collision_triangles': int(match.group(1)) if match else None} - return results - - -def main(): - parser = argparse.ArgumentParser(description='Validate every installed map and compare collision triangle counts ' - 'with a baseline.') - parser.add_argument('--update', action='store_true', help='write the current values as the new baseline') - parser.add_argument('--baseline', type=Path, default=BASELINE, help='baseline JSON (default: %(default)s)') - parser.add_argument('--exe', type=Path, default=REPO / 'bin/skate3rust.exe', help='game exe (default: %(default)s)') - args = parser.parse_args() - marker = json.loads((REPO / 'data/installation.json').read_text()) - maps = sorted((REPO / 'data' / marker['directory'] / 'maps').glob('*.skate')) - exe = args.exe - results = validate(exe, maps) - if results is None: - print('(no --validate-maps in this exe: using --check-assets per map, no spawn checks)') - results = check_assets(exe, maps) - current = {name: r['collision_triangles'] for name, r in results.items() if name != 'TEST_WORLD'} - failed = False - for name, r in results.items(): - if not r['ok'] or r['warnings']: - failed = True - print(f"{name:20} NOT OK errors={r['errors']} warnings={r['warnings']}") - if args.update: - args.baseline.parent.mkdir(parents=True, exist_ok=True) - args.baseline.write_text(json.dumps(current, indent=2) + '\n') - print(f'baseline updated with {len(current)} maps from {marker["directory"]}') - return 1 if failed else 0 - if not args.baseline.is_file(): - print(f'no baseline at {args.baseline}: run once with --update on a known-good build') - return 1 - baseline = json.loads(args.baseline.read_text()) - for name in sorted(set(baseline) | set(current)): - want, got = baseline.get(name), current.get(name) - status = 'ok' if want == got else f'CHANGED (baseline {want}, delta {None if None in (want, got) else got - want})' - failed |= want != got - seconds = results.get(name, {}).get('seconds') - print(f'{name:20} {got} {status}' + (f' ({seconds:.2f}s)' if seconds is not None else '')) - return 1 if failed else 0 - - -if __name__ == '__main__': - raise SystemExit(main()) diff --git a/tools/regression-checks/check_spawns.py b/tools/regression-checks/check_spawns.py deleted file mode 100644 index f24fa1b8..00000000 --- a/tools/regression-checks/check_spawns.py +++ /dev/null @@ -1,65 +0,0 @@ -"""Compare every installed map's baked spawn with the known-good baseline. - -Usage (from the repo root): - py -3.13 tools/regression-checks/check_spawns.py --update # first, on a known-good install: record the baseline - py -3.13 tools/regression-checks/check_spawns.py # later: compare against it - -The installation is read from data/installation.json; the baseline defaults to -.local/regression/spawn_baseline.json (--baseline PATH). Exit code 1 if any map differs. -""" -import argparse -import json -import struct -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -BASELINE = REPO / '.local/regression/spawn_baseline.json' -TOLERANCE = 0.01 - - -def read_spawn(path): - # SKATE14 header: magic, u32, length-prefixed name, then 3 x f32 spawn. - with open(path, 'rb') as f: - magic = f.read(8) - if not magic.startswith(b'SKATE'): - raise ValueError(f'{path.name}: not a .skate file') - f.read(4) - (length,) = struct.unpack('//.log and one summary line per map (also in summary.txt). -# Runs from Git Bash on Windows (uses taskkill to make sure the game is gone between maps). -set -u -ROOT="$(cd "$(dirname "$0")/../.." && pwd)" -EXE="${SKATE_EXE:-$ROOT/bin/skate3rust.exe}" -SECS="${1:-25}"; shift || true -MAPS=("$@"); [ ${#MAPS[@]} -eq 0 ] && MAPS=(StartPark University DownTown MegaPark) -INST=$(ls -d "$ROOT"/data/installations/*/ 2>/dev/null | head -1) -if [ -z "$INST" ] || [ ! -x "$EXE" ]; then echo "need an installation under data/installations and $EXE"; exit 2; fi -OUT="${SMOKE_OUT:-$ROOT/.local/smoke}/$(date +%Y%m%d_%H%M%S)"; mkdir -p "$OUT" -for m in "${MAPS[@]}"; do - log="$OUT/$m.log" - SKATE_REPORT_CHILD=1 SKATE3_MODS="$ROOT/mods" timeout "$SECS" "$EXE" --assets "$ROOT/assets" --map "$INST/maps/$m.skate" --mute >"$log" 2>&1 - code=$? - taskkill //F //IM "$(basename "$EXE")" >/dev/null 2>&1 - clean=$(sed 's/\x1b\[[0-9;]*m//g' "$log") - panics=$(grep -ciE "panicked|panic" <<<"$clean"); errors=$(grep -c " ERROR " <<<"$clean") - nonfinite=$(grep -cE "NONFINITE|Non-finite|physics_failed:true" <<<"$clean") - ready=$(grep -c "SKATE_RENDER_READY" <<<"$clean"); native=$(grep -c "Game audio: native" <<<"$clean") - echo "$m exit=$code panics=$panics errors=$errors nonfinite=$nonfinite render_ready=$ready native_lines=$native" | tee -a "$OUT/summary.txt" -done -echo "logs: $OUT" diff --git a/tools/setup-equivalence/README.md b/tools/setup-equivalence/README.md deleted file mode 100644 index d0610a8c..00000000 --- a/tools/setup-equivalence/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# Setup equivalence checks - -Prove that a faster setup path produces exactly the same bytes as the code it replaces. Both were -written to verify setup speed-ups (see `docs/hails-additions/08-setup-performance.md`). - -| Script | What it proves | -|---|---| -| `compare_streams.py` | A changed district stream loader (`tools/vendor/university/.../skate3_streams.py`) returns exactly what a reference copy returns: every asset's record, source path, offset, stored size and decoded bytes, per district and stream (Pres, Sim, Tex). Also times both. | -| `compare_refpack.py` | The native RefPack DLL (`target/native/refpack.dll`, built by `scripts\Build.ps1`) decodes exactly like the pure-Python decoder: every entry of `createacharacter.big` and `marquee.big`, and with `--streams` every extracted district stream asset. | - -## Inputs - -- Your extracted disc: `--disc DIR` (the folder holding `data/`), default `$SKATE3_DISC` or `.local/skate3-disc`. -- Extracted districts under `--work` (default `.local/collision`), made by - `tools/collision-inspect/map_collision.py` or `scan_surfaceless.py`. -- For `compare_streams.py`, a reference copy of the loader: - `git show :tools/vendor/university/tools/vanilla_map_extraction/tools/skate3_streams.py > .local/ref.py` - -## Usage - -``` -py -3.13 tools/setup-equivalence/compare_streams.py .local/ref.py MegaPark StartPark -py -3.13 tools/setup-equivalence/compare_refpack.py --streams -``` - -Both exit 1 on any difference. - -## Example output - -(made-up timings) - -``` -MegaPark Pres assets 812 old 1.90s new 0.85s IDENTICAL -MegaPark Sim assets 405 old 0.70s new 0.31s IDENTICAL -TOTAL old 2.6s new 1.2s -``` - -## Requirements - -Python 3.13 (standard library); the repository's setup modules (`tools/asset_pipeline`, -`tools/owned_game`, the vendored map tools). `compare_refpack.py` needs the DLL from `scripts\Build.ps1`. diff --git a/tools/setup-equivalence/compare_refpack.py b/tools/setup-equivalence/compare_refpack.py deleted file mode 100644 index 7fc7fe47..00000000 --- a/tools/setup-equivalence/compare_refpack.py +++ /dev/null @@ -1,65 +0,0 @@ -"""Prove the native RefPack DLL decodes exactly like the pure-Python decoder. - -Usage (repo root, after scripts\\Build.ps1 built target/native/refpack.dll): - py -3.13 tools/setup-equivalence/compare_refpack.py [--streams] [--disc DIR] [--work DIR] - -Decodes every entry of the character archives (createacharacter.big, marquee.big — what -the customiser reads) with the Python decoder (fast_refpack._library = None) and with the -DLL, and compares bytes. --streams also compares every district stream asset (map stage) of the -districts extracted under --work (default .local/collision, see tools/collision-inspect). ---disc is the extracted disc root (the folder holding data/; default $SKATE3_DISC or .local/skate3-disc). -Exit 1 on any difference. -""" -import argparse, os, sys, time -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(REPO)) -sys.path.insert(0, str(REPO / 'tools/vendor/university/tools/vanilla_map_extraction/tools')) -from tools.asset_pipeline import fast_refpack -from tools.owned_game.big import BigArchive - -parser = argparse.ArgumentParser(description='Prove the native RefPack DLL decodes exactly like the Python decoder.') -parser.add_argument('--streams', action='store_true', help='also compare every extracted district stream asset') -parser.add_argument('--disc', type=Path, default=Path(os.environ.get('SKATE3_DISC', REPO / '.local/skate3-disc')), - help='extracted disc root (default: %(default)s)') -parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='extracted districts (default: %(default)s)') -args = parser.parse_args() -native = fast_refpack._library -if native is None: - sys.exit('target/native/refpack.dll not loaded: run scripts\\Build.ps1 first') -disc = args.disc / 'data/content' - - -def decode_all(read): - fast_refpack._library = None - start = time.perf_counter(); python = read(); t_python = time.perf_counter() - start - fast_refpack._library = native - start = time.perf_counter(); fast = read(); t_native = time.perf_counter() - start - return python, fast, t_python, t_native - - -failed = False -for name in ('createacharacter.big', 'marquee.big'): - archive = BigArchive(disc / name) - python, fast, t_python, t_native = decode_all(lambda: [archive.read(e) for e in archive.entries]) - compressed = sum(1 for e in archive.entries if e.compression) - same = python == fast - failed |= not same - print(f'{name:22} entries {len(archive.entries):5} (compressed {compressed:5}) python {t_python:6.1f}s native {t_native:5.1f}s {"IDENTICAL" if same else "DIFFERENT"}', flush=True) - -if args.streams: - import skate3_streams - for district in sorted(args.work.iterdir()): - stream = district / 'raw/data/content/world/stream' / f'DIST_{district.name}' - for kind in ('Pres', 'Sim', 'Tex'): - if not (stream / f'DIST_{district.name}_{kind}.xst').is_file(): - continue - python, fast, t_python, t_native = decode_all( - lambda: [a.data for a in skate3_streams.load_district_stream(stream, kind, f'DIST_{district.name}')]) - same = python == fast - failed |= not same - print(f'{district.name:20} {kind:4} assets {len(python):5} python {t_python:6.1f}s native {t_native:5.1f}s {"IDENTICAL" if same else "DIFFERENT"}', flush=True) - -print('ALL IDENTICAL' if not failed else 'DIFFERENCES FOUND') -sys.exit(1 if failed else 0) diff --git a/tools/setup-equivalence/compare_streams.py b/tools/setup-equivalence/compare_streams.py deleted file mode 100644 index 1c27c041..00000000 --- a/tools/setup-equivalence/compare_streams.py +++ /dev/null @@ -1,47 +0,0 @@ -"""Prove a changed stream loader returns exactly what a reference copy returns. - -Usage (repo root): - py -3.13 tools/setup-equivalence/compare_streams.py [District ...] [--work DIR] -Get a reference with: git show :tools/vendor/university/tools/vanilla_map_extraction/tools/skate3_streams.py > .local/ref.py -Needs //raw (default work .local/collision) from tools/collision-inspect/map_collision.py -or scan_surfaceless.py. - -For every district and stream (Pres, Sim, Tex): load with the reference module and the current one, and -compare every asset's record, source path, source offset, stored size and decoded bytes. Also times both. -Exit 1 on the first difference. -""" -import argparse, importlib.util, sys, time -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -TOOLS = REPO / 'tools/vendor/university/tools/vanilla_map_extraction/tools' -sys.path.insert(0, str(TOOLS)) -import skate3_streams as new - -parser = argparse.ArgumentParser(description='Prove a changed stream loader returns exactly what a reference copy returns.') -parser.add_argument('reference', type=Path, help='reference copy of skate3_streams.py') -parser.add_argument('districts', nargs='*', help='district names without DIST_ (default: all ten)') -parser.add_argument('--work', type=Path, default=REPO / '.local/collision', help='extracted districts (default: %(default)s)') -args = parser.parse_args() -spec = importlib.util.spec_from_file_location('skate3_streams_original', args.reference) -old = importlib.util.module_from_spec(spec); sys.modules[spec.name] = old; spec.loader.exec_module(old) -import dataclasses - -names = args.districts or ['MegaPark', 'StartPark', 'BlackBoxPark', 'IndustrialSkatePark', 'DownTownSkatePark', - 'MaloofMoneyCup', 'SkateSchool', 'Industrial', 'University', 'DownTown'] -total_old = total_new = 0.0 -for name in names: - stream = args.work / name / 'raw/data/content/world/stream' / f'DIST_{name}' - for kind in ('Pres', 'Sim', 'Tex'): - if not (stream / f'DIST_{name}_{kind}.xst').is_file(): - continue - t = time.perf_counter(); a = old.load_district_stream(stream, kind, f'DIST_{name}'); t_old = time.perf_counter() - t - t = time.perf_counter(); b = new.load_district_stream(stream, kind, f'DIST_{name}'); t_new = time.perf_counter() - t - total_old += t_old; total_new += t_new - same = len(a) == len(b) and all( - (dataclasses.astuple(x.record) == dataclasses.astuple(y.record) and x.source_path == y.source_path and x.source_offset == y.source_offset - and x.stored_size == y.stored_size and x.data == y.data) for x, y in zip(a, b)) - print(f'{name:20} {kind:4} assets {len(a):5} old {t_old:7.2f}s new {t_new:7.2f}s {"IDENTICAL" if same else "DIFFERENT"}', flush=True) - if not same: - sys.exit(1) -print(f'TOTAL old {total_old:.1f}s new {total_new:.1f}s') diff --git a/tools/vault-inspect/README.md b/tools/vault-inspect/README.md deleted file mode 100644 index 9ce0e3a7..00000000 --- a/tools/vault-inspect/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# Vault inspection - -Look up values in the game's attribute database ("vault"): the skater collections setup converts to -`assets/private/stock/skater-collections.json`, and the class layouts in the disc's schema. -Classes, collections and fields are 64-bit name hashes (`Hash_XXXXXXXXXXXXXXXX`). - -| Script | What it does | -|---|---| -| `find_field.py` | Finds field hashes anywhere in the converted database: prints class, collection, field, type and the decoded value. | -| `vault_fields.py` | Prints one class's collections (or one collection), with parent, and every field (or the chosen ones), floats / ints decoded and arrays listed. | -| `vault_layout.py` | Prints a class's field layout from the schema: field hash, type, offset, count, maximum, flags, alignment. Fixed-layout fields (flags bit 2) first, by offset. | - -## Inputs - -- `--vault`: the converted database, default `assets/private/stock/skater-collections.json` (written by - `tools/setup.py`). -- `--schema` (for `vault_layout.py`): the schema path without extension; the folder must hold - `skaterschema.vlt` and `skaterschema.bin`. Extract them from your disc's `data/big/db.big`: - `py -3.13 tools/world-stream-inspect/big_list.py /data/big/db.big skaterschema --out .local/db` - -## Usage - -``` -py -3.13 tools/vault-inspect/find_field.py 0123456789ABCDEF -py -3.13 tools/vault-inspect/vault_fields.py Hash_0123456789ABCDEF default -py -3.13 tools/vault-inspect/vault_fields.py 0123456789ABCDEF default FEDCBA9876543210 -py -3.13 tools/vault-inspect/vault_layout.py 0123456789ABCDEF --schema .local/db/data/db/skaterschema -``` - -## Example output - -(made-up hashes and values) - -``` -== Hash_0123456789ABCDEF default parent= - Hash_1111111111111111 ('EA::Reflection::Float', 0.5) - Hash_2222222222222222 ('EA::Reflection::Int32', 3) -``` - -## Requirements - -Python 3.13 (standard library); `vault_layout.py` uses `tools/asset_pipeline/vlt.py`. diff --git a/tools/vault-inspect/find_field.py b/tools/vault-inspect/find_field.py deleted file mode 100644 index 3fd4ddd7..00000000 --- a/tools/vault-inspect/find_field.py +++ /dev/null @@ -1,35 +0,0 @@ -"""Find vault field hashes in the install's converted database (skater-collections.json): prints class, -collection, field, type and the decoded value (big-endian float or int for 4-byte values) of every hit. - -usage: py -3.13 tools/vault-inspect/find_field.py HASH [HASH ...] [--vault JSON] - HASH: 16 hex digits, with or without Hash_. - --vault defaults to assets/private/stock/skater-collections.json (written by setup). -""" -import argparse -import json -import struct -from pathlib import Path - -DEFAULT = Path(__file__).resolve().parents[2] / 'assets/private/stock/skater-collections.json' - - -def main(): - parser = argparse.ArgumentParser(description='Find vault field hashes in skater-collections.json.') - parser.add_argument('hashes', nargs='+', help='16 hex digits, with or without Hash_') - parser.add_argument('--vault', type=Path, default=DEFAULT, help='converted database (default: %(default)s)') - args = parser.parse_args() - want = {('Hash_' + h.replace('Hash_', '').upper().zfill(16)) for h in args.hashes} - for c in json.loads(args.vault.read_text())['collections']: - for w in want & set(c['fields']): - v = c['fields'][w] - h = ''.join(v.get('data', '').split()) - val = h - if 'Float' in v['type'] and len(h) == 8: - val = struct.unpack('>f', bytes.fromhex(h))[0] - elif len(h) == 8: - val = struct.unpack('>i', bytes.fromhex(h))[0] - print(c['class'], c['key'], w, v['type'], val) - - -if __name__ == '__main__': - main() diff --git a/tools/vault-inspect/vault_fields.py b/tools/vault-inspect/vault_fields.py deleted file mode 100644 index 3b989504..00000000 --- a/tools/vault-inspect/vault_fields.py +++ /dev/null @@ -1,65 +0,0 @@ -"""Print vault fields from the install's converted database (skater-collections.json) by class / -collection / field keys, with the collection's parent. Floats and ints are decoded; arrays list their items. - -usage: py -3.13 tools/vault-inspect/vault_fields.py CLASS [COLLECTION] [FIELD ...] [--vault JSON] - CLASS / FIELD: 16 hex digits (with or without Hash_); COLLECTION: a name ('default') or hex. - --vault defaults to assets/private/stock/skater-collections.json (written by setup). -""" -import argparse -import json -import struct -from pathlib import Path - -DEFAULT = Path(__file__).resolve().parents[2] / 'assets/private/stock/skater-collections.json' - - -def norm(k): - k = k.strip() - if k.lower() == 'default': - return 'default' - return k if k.startswith('Hash_') else 'Hash_' + k.upper().replace('0X', '').zfill(16) - - -def decode(v): - t = v.get('type', '') - - def one(h): - b = bytes.fromhex(h) - if 'Float' in t and len(b) == 4: - return round(struct.unpack('>f', b)[0], 6) - if len(b) == 4: - return struct.unpack('>i', b)[0] - if len(b) == 1: - return b[0] - return h - if 'array' in v: - return t, [one(x) for x in v['array']['items']] - return t, one(v['data']) - - -def main(): - parser = argparse.ArgumentParser(description='Print vault fields by class / collection / field.') - parser.add_argument('cls', metavar='CLASS') - parser.add_argument('collection', nargs='?', metavar='COLLECTION') - parser.add_argument('fields', nargs='*', metavar='FIELD') - parser.add_argument('--vault', type=Path, default=DEFAULT, help='converted database (default: %(default)s)') - args = parser.parse_args() - d = json.loads(args.vault.read_text()) - items = d['collections'] if 'collections' in d else d - items = list(items.values()) if isinstance(items, dict) else items - cls = norm(args.cls) - col = norm(args.collection) if args.collection else None - fields = [norm(f) for f in args.fields] - by = {c['key']: c for c in items if c['class'] == cls} - for key, c in by.items(): - if col and key != col: - continue - print(f'== {cls} {key} parent={c.get("parent")}') - for f, v in sorted(c['fields'].items()): - if fields and f not in fields: - continue - print(f' {f} {decode(v)}') - - -if __name__ == '__main__': - main() diff --git a/tools/vault-inspect/vault_layout.py b/tools/vault-inspect/vault_layout.py deleted file mode 100644 index 60159fc0..00000000 --- a/tools/vault-inspect/vault_layout.py +++ /dev/null @@ -1,55 +0,0 @@ -"""Print a vault class's field layout (field hash, type, offset, count, flags) from the disc schema. - -usage: py -3.13 tools/vault-inspect/vault_layout.py CLASS --schema STEM [--name NAME ...] - CLASS: 16 hex digits (with or without Hash_). - --schema: path of skaterschema without extension (the folder must hold skaterschema.vlt and .bin; - extract them from data/big/db.big with tools/world-stream-inspect/big_list.py db.big skaterschema --out DIR). - --name: type names to label (their 64-bit hash is shown by name when it matches); repeatable. -flags bit 2 = a fixed-layout field: the offset is where the record keeps it. -""" -import argparse -import sys -from pathlib import Path - -ROOT = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(ROOT)) -from tools.asset_pipeline.vlt import vault, unpack, hash64 # noqa: E402 - -CLASS_EXPORT = 0x2A7895AC4A876152 -# Generic reflection type names, so the common field types print by name. -TYPE_NAMES = ('EA::Reflection::Int32', 'EA::Reflection::Bool', 'EA::Reflection::Float', 'Attrib::RefSpec') - - -def layout(cls: int, stem: Path): - sv, sb, se = vault(stem) - for _, kind, size, at in se: - if kind != CLASS_EXPORT: - continue - key, reserve, count, defs, static_size, static, lay = unpack('QIIIIII', sv, at) - if key != cls: - continue - rows = [] - for i in range(count): - fkey, typ, offset, n, maximum, flags, alignment = unpack('QQHHHBB', sb, defs + i * 24) - rows.append((fkey, typ, offset, n, maximum, flags, alignment)) - return rows - return None - - -def main(): - parser = argparse.ArgumentParser(description="Print a vault class's field layout from the disc schema.") - parser.add_argument('cls', metavar='CLASS', help='16 hex digits, with or without Hash_') - parser.add_argument('--schema', type=Path, required=True, help='skaterschema path without extension') - parser.add_argument('--name', action='append', default=[], help='extra type name to label; repeatable') - args = parser.parse_args() - cls = int(args.cls.replace('Hash_', ''), 16) - names = {hash64(n): n for n in (*TYPE_NAMES, *args.name)} - rows = layout(cls, args.schema) - if rows is None: - sys.exit('class not found') - for fkey, typ, offset, n, maximum, flags, alignment in sorted(rows, key=lambda r: (not r[5] & 2, r[2])): - print(f'Hash_{fkey:016X} {names.get(typ, hex(typ)):36} off={offset:4} n={n:3} max={maximum} flags={flags} align={alignment}') - - -if __name__ == '__main__': - main() diff --git a/tools/world-stream-inspect/README.md b/tools/world-stream-inspect/README.md deleted file mode 100644 index cb89ebd5..00000000 --- a/tools/world-stream-inspect/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# World stream inspection - -Read the world districts' simulation streams (`cSim_*.xsf`, RW4 arenas) straight from your own -disc's `.big` archives, without extracting anything to disk. - -| Script | What it does | -|---|---| -| `big_list.py` | Lists the entries of any `.big` archive (path and unpacked size), filtered by a regex, or extracts the matching entries with `--out`. | -| `sim_types.py` | Per stream file of a district: the arena dictionary's entry types and counts. `--find HEX64` searches every arena for a 64-bit value (both word orders) and names the entry that holds it; `--dump TYPE` hex-dumps the entries of one type. | -| `volumes_dump.py` | Dumps every named volume (arena type `0x00EB0019`, trigger / zone / teleport boxes): name, instance id, link id, bounds, base corners, plus each arena's collision meshes and other entry types. Writes JSON too. | -| `arenas.py` | Shared helper: walks the arenas of a district (or any `.big` with `*_Sim.xst` + `cSim_*.xsf`, such as `missions.big`) and parses the arena dictionary. | - -The volume layout is documented at the top of `volumes_dump.py`. - -## Inputs - -- Your extracted disc: `--disc DIR` (the folder holding `data/`), default `$SKATE3_DISC` or `.local/skate3-disc`. -- District ids with the prefix: `DIST_SkateSchool`, `DIST_MegaPark`, … (`volumes_dump.py` defaults to all ten). - -## Usage - -``` -py -3.13 tools/world-stream-inspect/big_list.py .local/skate3-disc/data/big/db.big skaterschema -py -3.13 tools/world-stream-inspect/big_list.py .local/skate3-disc/data/audio/audiofiles.big "\.bnk$" --out .local/bnk -py -3.13 tools/world-stream-inspect/sim_types.py DIST_SkateSchool -py -3.13 tools/world-stream-inspect/sim_types.py DIST_SkateSchool --dump 00eb0019 -py -3.13 tools/world-stream-inspect/volumes_dump.py DIST_SkateSchool > volumes.txt -``` - -## Example output - -`sim_types.py` (made-up ids and counts): - -``` -cSim_0_0_high.xsf 0123456789abcdef 00080001:1 00080006:3 00eb000a:1 -TOTAL 00080001:40 00080006:112 00eb000a:40 00eb0019:2 -``` - -## Requirements - -Python 3.13 (standard library); the repository's `tools/owned_game` and the vendored map extraction -tools (`tools/vendor/university/.../skate3_streams.py`, `retail_collision_mesh.py`). diff --git a/tools/world-stream-inspect/arenas.py b/tools/world-stream-inspect/arenas.py deleted file mode 100644 index 5b3278b5..00000000 --- a/tools/world-stream-inspect/arenas.py +++ /dev/null @@ -1,89 +0,0 @@ -"""Shared helpers: walk the RW4 arenas in a district's simulation streams (cSim_*.xsf) straight from a -world.big (or any .big holding *_Sim.xst + cSim_*.xsf, such as missions.big), nothing extracted. - - import arenas - for stream_file, asset_id, arena in arenas.district_arenas(disc, 'DIST_SkateSchool'): - for offset, _, size, _, _, type_id in arenas.entries(arena): - ... - -Uses the vendored stream reader (tools/vendor/university/.../skate3_streams.py) for the asset table and -the section decoder. -""" -import os -import struct -import sys -import tempfile -from collections import defaultdict -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(REPO)) -sys.path.insert(0, str(REPO / 'tools/vendor/university/tools/vanilla_map_extraction/tools')) -from tools.owned_game.big import BigArchive # noqa: E402 -from skate3_streams import read_atoc, _decode_section # noqa: E402 - -ARENA_PROCESSOR = 0xAB329A6A -DISTRICTS = ['DIST_SkateSchool', 'DIST_MegaPark', 'DIST_IndustrialSkatePark', 'DIST_StartPark', 'DIST_BlackBoxPark', - 'DIST_DownTownSkatePark', 'DIST_MaloofMoneyCup', 'DIST_University', 'DIST_Industrial', 'DIST_DownTown'] - - -def default_disc() -> Path: - """Extracted disc root (the folder holding data/): $SKATE3_DISC, else .local/skate3-disc.""" - return Path(os.environ.get('SKATE3_DISC', REPO / '.local/skate3-disc')) - - -def entries(arena: bytes): - """The arena's dictionary: one (offset, ?, size, align?, ?, type id) tuple of BE u32 per entry.""" - n = struct.unpack_from('>I', arena, 0x20)[0] - at = struct.unpack_from('>I', arena, 0x30)[0] - return [struct.unpack_from('>6I', arena, at + 24 * i) for i in range(n)] - - -def big_arenas(bigpath, only_dirs=None): - """Yield (dir name, stream file, asset id, arena bytes) for every arena asset of every cSim_*.xsf in a - .big, using the *_Sim.xst in the same directory.""" - big = BigArchive(Path(bigpath)) - by = defaultdict(list) - for e in big.entries: - by[Path(e.path).parent.as_posix()].append(e) - with tempfile.TemporaryDirectory() as tmpdir: - tmp = Path(tmpdir) / '_tmp.xst' - for dname, ents in sorted(by.items()): - if only_dirs and Path(dname).name not in only_dirs: - continue - xst = [e for e in ents if e.path.endswith('_Sim.xst')] - if not xst: - continue - tmp.write_bytes(big.read(xst[0])) - recs = {r.asset_id: r for r in read_atoc(tmp)} - seen = set() - for e in ents: - if not (Path(e.path).name.startswith('cSim_') and e.path.endswith('.xsf')): - continue - s = big.read(e) - cur = struct.unpack_from('>I', s, 16)[0] - while cur + 208 <= len(s): - aid = struct.unpack_from('>Q', s, cur)[0] - if aid == 0: - break - st, hs, stride = struct.unpack_from('>III', s, cur + 8) - r = recs.get(aid) - if r and r.processor_id == ARENA_PROCESSOR and aid not in seen: - seen.add(aid) - if st == r.total_size and hs >= 128: - data = s[cur + hs:cur + hs + st] - else: - v = struct.unpack_from('>11I', s, cur + 128) - c, q = _decode_section(s, cur + 208, v[0], v[1], v[2]) - g, q = _decode_section(s, q, v[8], v[9], v[10]) - data = c + g - yield Path(dname).name, Path(e.path).name, aid, data - if stride == 0: - break - cur += stride - - -def district_arenas(disc: Path, district: str): - """Yield (stream file, asset id, arena bytes) for one world district.""" - for _, fname, aid, data in big_arenas(Path(disc) / 'data/content' / f'world{district}.big'): - yield fname, aid, data diff --git a/tools/world-stream-inspect/big_list.py b/tools/world-stream-inspect/big_list.py deleted file mode 100644 index 2fad65d8..00000000 --- a/tools/world-stream-inspect/big_list.py +++ /dev/null @@ -1,37 +0,0 @@ -"""List the entries of a .big archive, or extract the ones matching a regex. - -usage: py -3.13 tools/world-stream-inspect/big_list.py [regex] [--out DIR] -Without --out prints ` ` per entry; with --out writes the matching entries -(decompressed) under DIR, keeping their archive paths. -""" -import argparse -import re -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parents[2])) -from tools.owned_game.big import BigArchive # noqa: E402 - - -def main(): - parser = argparse.ArgumentParser(description='List or extract entries of a .big archive.') - parser.add_argument('big', type=Path) - parser.add_argument('regex', nargs='?', help='only entries whose path matches (case-insensitive)') - parser.add_argument('--out', type=Path, help='extract the matching entries here') - args = parser.parse_args() - a = BigArchive(args.big) - pat = re.compile(args.regex, re.I) if args.regex else None - for e in a.entries: - if pat is not None and not pat.search(e.path): - continue - if args.out: - p = args.out / BigArchive.safe_relative(e.path) - p.parent.mkdir(parents=True, exist_ok=True) - p.write_bytes(a.read(e)) - print('wrote', p) - else: - print(e.path, e.unpacked_size) - - -if __name__ == '__main__': - main() diff --git a/tools/world-stream-inspect/sim_types.py b/tools/world-stream-inspect/sim_types.py deleted file mode 100644 index 302d0c43..00000000 --- a/tools/world-stream-inspect/sim_types.py +++ /dev/null @@ -1,52 +0,0 @@ -"""Survey RW4 arena dictionary types in a district's simulation streams (cSim_*.xsf). - -usage: py -3.13 tools/world-stream-inspect/sim_types.py DIST_SkateSchool [--find HEX64 ...] [--dump TYPEHEX] [--disc DIR] -Reads straight from /data/content/world.big (nothing extracted to disk). -Dictionary entry = 6 BE u32: (offset, ?, size, align?, ?, type). Prints type counts per stream file; ---find searches every arena for a 64-bit value (both word orders) and names the entry holding it; ---dump hex-dumps the first 256 bytes of every entry of one type. -""" -import argparse -import sys -from collections import Counter -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -import arenas # noqa: E402 - - -def main(): - parser = argparse.ArgumentParser(description='Survey RW4 arena dictionary types in a district.') - parser.add_argument('district', help='e.g. DIST_SkateSchool') - parser.add_argument('--find', action='append', default=[], metavar='HEX64', help='64-bit value to search for') - parser.add_argument('--dump', metavar='TYPEHEX', help='hex-dump the entries of this type id') - parser.add_argument('--disc', type=Path, default=arenas.default_disc(), help='extracted disc root (default: %(default)s)') - args = parser.parse_args() - finds = [int(v, 16) for v in args.find] - dump = int(args.dump, 16) if args.dump else None - total = Counter() - for fname, aid, d in arenas.district_arenas(args.disc, args.district): - ents = arenas.entries(d) - c = Counter(e[5] for e in ents) - total.update(c) - if not finds and dump is None: - print(fname, '%016x' % aid, ' '.join('%08x:%d' % kv for kv in sorted(c.items()))) - for v in finds: - for pat, tag in ((v.to_bytes(8, 'big'), 'BE'), (((v & 0xFFFFFFFF) << 32 | v >> 32).to_bytes(8, 'big'), 'LOWFIRST')): - i = d.find(pat) - while i >= 0: - owner = [k for k, e in enumerate(ents) if e[0] <= i < e[0] + e[2]] - print('FIND %016x %s %s @%x in entry %s type %s' % (v, tag, fname, i, owner, [hex(ents[k][5]) for k in owner])) - i = d.find(pat, i + 1) - if dump is not None: - for k, e in enumerate(ents): - if e[5] == dump: - print(fname, k, 'off %x size %x' % (e[0], e[2])) - blob = d[e[0]:e[0] + min(e[2], 256)] - for j in range(0, len(blob), 32): - print(' ', blob[j:j + 32].hex(' ', 4)) - print('TOTAL', ' '.join('%08x:%d' % kv for kv in sorted(total.items()))) - - -if __name__ == '__main__': - main() diff --git a/tools/world-stream-inspect/volumes_dump.py b/tools/world-stream-inspect/volumes_dump.py deleted file mode 100644 index 49dc756b..00000000 --- a/tools/world-stream-inspect/volumes_dump.py +++ /dev/null @@ -1,86 +0,0 @@ -"""Dump every named volume (arena type 0x00EB0019, 'volume set') in the world districts: trigger boxes, -teleport and zone volumes, with their bounds, base corners and link ids. - -usage: py -3.13 tools/world-stream-inspect/volumes_dump.py [DIST_x ...] [--disc DIR] [--json OUT] > volumes.txt -Default: all ten districts; JSON to .local/world-stream-inspect/volumes.json. -Volume-set layout (big-endian): - header: +0 magic, +4 count, +8 count, +12 items offset, +16 strings offset - item (240 bytes): +0 mat4, +64 aabb min (vec4), +80 aabb max, +96 up axis (vec4), +112 4 base corners (vec4 x4), - +176 u64 link GUID (an import of type 0x00EB006B), +184 u64 instance id, +192 16 bytes 0xFF, +220 import handle, - +224 name offset (from record start). -Each arena line also lists its clustered collision meshes (triangles, surfaceless or not) and other entry types. -""" -import argparse -import json -import struct -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -import arenas # noqa: E402 -from retail_collision_mesh import decode_clustered_mesh # noqa: E402 (vendored, on the path via arenas) - -VOLUME_SET = 0x00EB0019 -CLUSTERED_MESH = 0x00080006 - - -def cstr(b, o): - e = b.index(b'\0', o) - return b[o:e].decode('latin1') - - -def items(b): - magic, n, n2, ioff, soff = struct.unpack_from('>5I', b, 0) - for i in range(n): - o = ioff + 240 * i - mn = struct.unpack_from('>3f', b, o + 64) - mx = struct.unpack_from('>3f', b, o + 80) - up = struct.unpack_from('>3f', b, o + 96) - corners = [struct.unpack_from('>3f', b, o + 112 + 16 * k) for k in range(4)] - guid, iid = struct.unpack_from('>QQ', b, o + 176) - imp, noff = struct.unpack_from('>II', b, o + 220) - mat = struct.unpack_from('>16f', b, o) - yield dict(name=cstr(b, noff), id='%016x' % iid, link='%016x' % guid, import_handle=imp, - min=[round(v, 3) for v in mn], max=[round(v, 3) for v in mx], up=[round(v, 3) for v in up], - corners=[[round(v, 3) for v in c] for c in corners], - translation=[round(v, 3) for v in mat[12:15]], - identity_rot=mat[0] == 1.0 and mat[5] == 1.0 and mat[10] == 1.0) - - -def main(): - parser = argparse.ArgumentParser(description='Dump every named volume in the world districts.') - parser.add_argument('districts', nargs='*', help='e.g. DIST_SkateSchool (default: all ten)') - parser.add_argument('--disc', type=Path, default=arenas.default_disc(), help='extracted disc root (default: %(default)s)') - parser.add_argument('--json', type=Path, default=arenas.REPO / '.local/world-stream-inspect/volumes.json', - help='JSON output (default: %(default)s)') - args = parser.parse_args() - out = {} - for dist in args.districts or arenas.DISTRICTS: - res = [] - for fname, aid, d in arenas.district_arenas(args.disc, dist): - ents = arenas.entries(d) - vs = [e for e in ents if e[5] == VOLUME_SET] - if not vs: - continue - meshes = [] - for e in ents: - if e[5] == CLUSTERED_MESH: - m = decode_clustered_mesh(d[e[0]:e[0] + e[2]]) - sl = all(not (t.unit_flags & 0x80) for t in m.triangles) - meshes.append(dict(tris=len(m.triangles), surfaceless=sl, - min=[round(v, 2) for v in m.bounds_min], max=[round(v, 2) for v in m.bounds_max])) - other = sorted({'%08x' % e[5] for e in ents} - {'00eb0019', '00080006', '00080001', '00eb000a', '00eb0008', '00eb000b'}) - for e in vs: - for it in items(d[e[0]:e[0] + e[2]]): - it.update(stream=fname, arena='%016x' % aid) - res.append(it) - short = it['name'].split('_0x')[0] - print(dist, fname, it['id'], short, it['min'], it['max'], 'link', it['link']) - print(' arena %016x meshes %s other types %s' % (aid, [(m['tris'], m['surfaceless']) for m in meshes], other)) - out[dist] = res - args.json.parent.mkdir(parents=True, exist_ok=True) - args.json.write_text(json.dumps(out, indent=1)) - - -if __name__ == '__main__': - main()