mirror of
https://github.com/SK8-ENGINE/skate-3-rust-engine.git
synced 2026-10-26 19:47:44 +00:00
Hails
8327e94090
- Skip decoding identical streamed asset copies (DownTown streams 107.8 -> 24.8 s) - Run the character customiser beside the map stage (memory-gated) - Build.ps1 builds the native RefPack DLL for dev checkouts - Clothing library texture warm-up and pro roster run in worker processes (library 126.5 -> 51.2 s, roster 89.6 -> 15.9 s); shared caches are published first-writer-wins to avoid Windows rename races - pipeline-equivalence.json: combined pairs (committed -> PR F+G) Outputs proven identical (8,288 customiser files; all maps/spawns). One-time customiser rebuild. See docs/hails-additions/08-setup-performance.md (PR G).
206 lines
10 KiB
Markdown
206 lines
10 KiB
Markdown
# 8. Setup performance: duplicate stream copies and customiser overlap (PR G)
|
||
|
||
Measured on a full setup (Windows 11, i7-14700KF, 28 threads, 3 map workers):
|
||
466 s total before this change — stock data 20 s, maps 183 s (DownTown alone
|
||
182 s), character customiser 226 s (run only after all maps), publish 24 s —
|
||
with total CPU use around 10%.
|
||
|
||
## 1. Skip decoding identical copies of streamed assets
|
||
|
||
### Problem
|
||
|
||
City districts are stored as a grid of streaming cells (`cPres_*`, `cTex_*`,
|
||
`cSim_*` `.xsf` files). The same asset (texture, model) is copied into every
|
||
cell that uses it. `skate3_streams.load_district_stream` called `read_sfil` on
|
||
every cell file, which **fully decoded every copy** (RefPack), and only then
|
||
compared it with the first copy and discarded it.
|
||
|
||
| District | Copies / unique (Pres) | Copies / unique (Tex) | RefPack data decoded / unique |
|
||
|---|---|---|---|
|
||
| DownTown | 13,991 / 3,627 | 10,510 / 985 | 1,548 MB / 347 MB |
|
||
| University | 7,736 / 2,665 | 6,427 / 689 | 1,332 MB / 285 MB |
|
||
| Industrial | 6,915 / 2,163 | 5,088 / 578 | 806 MB / 188 MB |
|
||
|
||
In source checkouts the native RefPack DLL is not built (only the release
|
||
script builds `target/native/refpack.dll`), so decoding runs in pure Python.
|
||
|
||
### Change
|
||
|
||
`tools/vendor/university/tools/vanilla_map_extraction/tools/skate3_streams.py`:
|
||
`read_sfil` takes an optional `known_copies` (asset id → BLAKE2b digest of the
|
||
first stored copy: asset header + stored payload). In a district load, a later
|
||
copy whose stored bytes have the same digest is not decoded and not returned.
|
||
A copy whose stored bytes differ is decoded and returned as before, so the
|
||
existing conflict check (`conflicting SFIL copies`) still sees it. Record
|
||
lookup, in-file duplicate detection and stride validation still run for every
|
||
copy. Without `known_copies` (all other callers) behaviour is unchanged.
|
||
|
||
Why identical: identical stored bytes (same section table, sizes and
|
||
compressed payload) decode to identical data, and the loader already kept the
|
||
first copy; the returned list is still built in ATOC order from the first
|
||
copies.
|
||
|
||
### Verification
|
||
|
||
- All 10 districts × Pres/Sim/Tex loaded with the original and the new module:
|
||
every asset's record fields, source path, source offset, stored size and
|
||
decoded bytes are identical.
|
||
|
||
| District | Stream load before | After |
|
||
|---|---|---|
|
||
| DownTown | 107.8 s | 24.8 s |
|
||
| University | 83.4 s | 19.1 s |
|
||
| Industrial | 49.3 s | 12.9 s |
|
||
| all 10 | 247.9 s | 64.1 s |
|
||
|
||
- Unit tests (`test_streams.py`): identical copies from another cell are
|
||
skipped, differing copies are still decoded and returned, in-file duplicates
|
||
are still rejected. (The fixture uses the uncompressed storage path; the
|
||
compressed path is covered by the full-district comparison above.)
|
||
|
||
## 2. Run the character customiser beside the map stage
|
||
|
||
### Problem
|
||
|
||
`customiser_setup.install` passes the customiser to `install._install` as
|
||
`finalize`, which ran after the whole map stage. It reads only stock data
|
||
(core group), default-skater textures (character group) and the disc — all
|
||
prepared before maps — and writes only `assets/private/customisation`. It
|
||
launches no processes (so it cannot race `spawn()`'s process-wide DLL setting).
|
||
|
||
### Change
|
||
|
||
`tools/asset_pipeline/install.py`: when the map stage runs and memory allows
|
||
(`overlap_customiser`: available memory ≥ 2 GiB desktop reserve + 3 GiB per
|
||
map worker + one more 3 GiB slot), the customiser starts on a background thread
|
||
(`Background`) just before the map jobs and is joined where it used to run;
|
||
its exception is re-raised there, so failures behave as before. Otherwise it
|
||
runs at the end exactly as before. `map_workers` and the new gate share
|
||
`available_memory()`.
|
||
|
||
### Verification
|
||
|
||
- Unit tests: `Background.join` waits and re-raises; the memory gate needs one
|
||
slot more than the map workers.
|
||
- Full setup: see below.
|
||
|
||
## 3. Faster clothing library and pro roster
|
||
|
||
### Problem
|
||
|
||
After section 2 the customiser was the critical path. Its two big stages were
|
||
single-threaded loops: the clothing library (126.5 s alone) decodes ~2,900
|
||
textures one at a time, and the pro roster (89.6 s) prepares 41 characters
|
||
one at a time. In a dev checkout both also used the pure-Python RefPack
|
||
decoder, because only `Build-Release.ps1` built `target/native/refpack.dll`.
|
||
|
||
### Change
|
||
|
||
- `scripts/Build.ps1`: after the cargo build, compiles
|
||
`tools/asset_pipeline/refpack_native.rs` into `target/native/refpack.dll`
|
||
with the same rustc flags as the release script, when the DLL is missing or
|
||
older than its source. `fast_refpack.py` already prefers it.
|
||
- New `tools/asset_pipeline/customisation_workers.py`: worker entry
|
||
(`--warm-textures`, `--roster`) plus `run_parallel`, which runs workers through
|
||
`install.run(task(...))` so frozen builds re-enter correctly. The worker count
|
||
is `customiser_workers()`: a quarter of the logical CPUs, from 1 to 8. If a
|
||
worker fails, `run_parallel` prints the tail of that worker's log.
|
||
- Library (`customisation_library.py`): `requested_textures(catalog)` lists
|
||
exactly the textures `prepare()` will decode, using the same skip rules. Before
|
||
the serial loop, `warm_up()` decodes them into the `decoded/` cache, spread
|
||
across worker processes in disjoint chunks. The loop itself and its
|
||
first-writer-wins ordering are unchanged; it now finds the decoded files
|
||
already present. Decoded files that the loop doesn't use, and source files
|
||
the warm-up created for them, are deleted before `prune_unavailable`. That
|
||
leaves the cache exactly as a serial run would.
|
||
- Roster (`native_roster.py`): `prepare_parallel()` splits the characters
|
||
round-robin over worker processes (`prepare(only=..., report_path=...)`).
|
||
Each character writes to its own outputs (`library/entries/<id>`,
|
||
`work/<key>`). The reports are merged back into roster order and
|
||
`work/report.json` is written as before. It falls back to a serial
|
||
`prepare()` with one worker, or if any worker fails or a report is missing.
|
||
- Shared roster caches (`work/source`, `work/decoded`) are published by
|
||
`_publish()`: write a pid-unique temporary file, then rename it into place
|
||
without overwriting. On Windows, renaming over a file that another process
|
||
is opening fails on one side: WinError 5 for the renamer, or Errno 13 for the
|
||
reader's `open()`. The first version used `os.replace`, which overwrites. In
|
||
the first real run, 4 of 7 workers died this way and the stage quietly fell
|
||
back to serial. Now `os.rename` is used (it doesn't overwrite on Windows). An
|
||
existing destination, or `FileExistsError`, means another worker already
|
||
published identical bytes, so that file is kept. A `PermissionError` is
|
||
retried briefly.
|
||
|
||
### Verification
|
||
|
||
- RefPack: the native and Python decoders give identical bytes for every
|
||
compressed entry in `createacharacter.big` (4,229 entries; 34.0 s → 1.4 s)
|
||
and `marquee.big` (11.2 s → 0.5 s), and for every district stream
|
||
(DownTown 23.9 s → 1.8 s).
|
||
- Library warm-up, 7 workers: all 2,921 requested textures decoded in 25.6 s,
|
||
byte-identical to the installed `decoded/` cache.
|
||
- Roster, 7 workers, no fallback: 18.5 s. All 41 characters are ready and each
|
||
entry's `character.glb`, `recipe.json`, `manifest.json` and `preview.png` is
|
||
byte-identical to the installed set. The report equals the installed one and
|
||
no temporary files are left behind.
|
||
- Unit tests (`test_customisation_workers.py`):
|
||
- merge order;
|
||
- fallback to serial on a failed worker;
|
||
- workers never exceed the number of characters;
|
||
- `_publish` is atomic and keeps an existing file;
|
||
- rename contention is tolerated and retried;
|
||
- `requested_textures` follows the skip rules.
|
||
- Refresh setup (customiser only, rebuilt once):
|
||
- library 126.5 → **51.2 s**, roster 89.6 → **15.9 s**;
|
||
- customiser stage total 73 s, with no worker failures;
|
||
- `check_customiser.py`: IDENTICAL, all 8,288 receipted files;
|
||
- `check_maps.py` and `check_spawns.py`: all OK, unchanged.
|
||
|
||
### Cost
|
||
|
||
The customiser fingerprint covers `customis*.py` and `native_roster.py`, so an
|
||
existing install rebuilds its customiser once after this change (about 75 s on
|
||
this machine). The outputs are identical, so nothing the user can see changes.
|
||
There is no equivalence mechanism for the customiser fingerprint, and the
|
||
rebuild is cheap, so none was added.
|
||
|
||
## Fingerprints
|
||
|
||
The stream change touches a file in the shared parser set, changing the
|
||
`character`, `environment` and `maps` fingerprints without changing outputs
|
||
(proven above); old→new pairs are in `pipeline-equivalence.json`. The
|
||
customiser fingerprint and stage versions are unchanged. An installation built
|
||
from the committed code rebuilds nothing.
|
||
|
||
## Full setup result
|
||
|
||
Full fresh setup, same machine and source, before → after:
|
||
|
||
| Stage | Before (466 s total) | After (331 s total) |
|
||
|---|---|---|
|
||
| Stock data, HUD, skater, environment | 0–20 s | 0–19 s |
|
||
| Map stage (10 maps, 3 workers, validated) | 21–204 s (183 s) | 19–138 s (**119 s**) |
|
||
| Character customiser | 216–442 s (226 s, after maps) | 19–311 s (292 s, beside maps) |
|
||
| Publish | 442–466 s | 311–331 s |
|
||
|
||
**Total: 466 s → 331 s (−29%), 0 warnings.**
|
||
|
||
- Identical outputs: all 8,288 receipted customiser files match a snapshot of
|
||
the previous (sequential) installation, with only the random set id
|
||
differing in 3 JSON paths; all maps pass validation with unchanged
|
||
collision counts and no warnings; all baked spawns unchanged.
|
||
- Contention: overlapped, the customiser's library stage took 193.9 s instead
|
||
of 126.5 s alone (three map workers compete for disk and memory bandwidth,
|
||
boost clocks drop, and the customiser thread shares the GIL with the setup
|
||
main thread). It is now the critical path; the roster (89.6 s) and library
|
||
are single-threaded loops and the next candidates.
|
||
|
||
## Files
|
||
|
||
- `tools/vendor/university/tools/vanilla_map_extraction/tools/skate3_streams.py`
|
||
- `tools/asset_pipeline/install.py` (`available_memory`, `overlap_customiser`,
|
||
`Background`, map stage), `pipeline-equivalence.json`
|
||
- `tools/asset_pipeline/test_streams.py`, `test_map_validator.py`
|
||
- Section 3: `scripts/Build.ps1`, `tools/asset_pipeline/customisation_workers.py` (new),
|
||
`customisation_library.py`, `native_roster.py`, `customiser_setup.py`,
|
||
`test_customisation_workers.py` (new), `test_customiser_setup.py`
|