Convert retail skaters and maps directly without Blender

This commit is contained in:
chazmm committed 2026-09-08 04:27:56 +01:00
1 parent 64d321763c
commit ae107e2876
12 files changed
+846 -45

No files matched your search

+5 -1
View File
@@ -1,6 +1,5 @@
param()
$ErrorActionPreference = 'Stop'
throw 'Release packaging is paused until the Blender-free ISO converter is ready.'
Push-Location $PSScriptRoot
try {
$packagePython = Join-Path $PSScriptRoot 'target/package-venv/Scripts/python.exe'
@@ -23,6 +22,10 @@ try {
if ($source.FullName -match '[\\/]__pycache__[\\/]') { continue }
if ($source.Extension -notin '.py','.json','.txt','.md','.toml' -and $source.Name -ne 'LICENSE') { continue }
$relative = [IO.Path]::GetRelativePath($toolsRoot, $source.FullName)
$portableName = $relative.Replace('\','/')
if ($portableName -match '(^|/)blender[^/]*(/|$)' -or
$portableName -in @('asset_pipeline/build_map.py','asset_pipeline/finish_character.py',
'add_onboard_ik_targets.py','apply_default_skater_materials.py','export_bevy_glb.py')) { continue }
$destination = Join-Path "$sourceStage/tools" $relative
New-Item -ItemType Directory -Path (Split-Path -Parent $destination) -Force | Out-Null
Copy-Item -LiteralPath $source.FullName -Destination $destination
@@ -30,6 +33,7 @@ try {
& $packagePython -m PyInstaller --noconfirm --clean --onefile --name skate3setup `
--icon "$PSScriptRoot/docs/images/skating-crab.ico" --paths $PSScriptRoot `
--hidden-import numpy --hidden-import PIL.Image --hidden-import tkinter `
--exclude-module bpy --exclude-module mathutils `
--copy-metadata numpy --copy-metadata Pillow --copy-metadata PyInstaller `
--add-data "$sourceStage/tools;tools" --add-data "$PSScriptRoot/docs/images/skating-crab.ico;docs/images" `
--distpath "$stage/support" --workpath target/setup-build/work --specpath target/setup-build tools/setup.py
+7 -5
View File
@@ -10,9 +10,10 @@ Includes skating, tricks, grinds, offboard movement, difficulty settings and
## Play
ISO setup is under development. Release packaging is paused while the asset
converter is changed to work without Blender. Use a prepared local asset set
for now.
Extract the Windows release ZIP and run `skate3rust.exe`. Select your Skate 3
Xbox 360 ISO. Setup prepares the skater, animations and all disc maps, then
starts University. No Blender, Python or Rust installation is needed.
The first setup needs internet access and can take a while.
Use an XInput controller to play. Escape opens graphics, difficulty and map
settings. Loading another map restarts the game session.
@@ -28,8 +29,9 @@ Use `PLAY-MAP.bat` to select a map. An XInput controller is required for gamepla
Escape opens difficulty and graphics settings.
Development builds use a prepared asset set in `assets/private/` or the
installed asset directory. The Windows release workflow is present, but
packaging is disabled until direct ISO conversion is ready.
installed asset directory. `Build-Release.ps1` builds the portable Windows
package and requires Python 3.13. Published GitHub releases build and attach
the package automatically.
Custom animations and climbing support remain available, but no custom clips
are shipped. The included format-demo map is original procedural content.
+5 -4
View File
@@ -17,10 +17,11 @@ reference: https://github.com/NFSTools/VaultLib. Its lookup8 hash and record
reader preserve data from the user's BIN/VLT files; no settings payload is
bundled.
Release setup is paused pending direct asset conversion. Blender is not
downloaded by setup. The retained authoring scripts can use a separately
supplied Blender executable. XboxDev extract-xiso retains its own license
files. Game assets are never downloaded from this project.
Release setup uses direct Python GLB and SKATE writers adapted from those
source converters. Blender is not downloaded, bundled or invoked. The retained
editor authoring scripts are excluded from release packages. XboxDev
extract-xiso retains its own license files. Game assets are never downloaded
from this project.
The setup executable bundles Python, NumPy, Pillow, Tcl/Tk and PyInstaller's
bootloader. Their license information is retained by the packager alongside
+4 -2
View File
@@ -514,9 +514,8 @@ impl SkateMap {
if materials.is_empty()
|| geometry.vertices.is_empty()
|| geometry.indices.is_empty()
|| geometry.collision.is_empty()
{
return Err("SKATE requires materials, render geometry, and collision".into());
return Err("SKATE requires materials and render geometry".into());
}
r.check_count(counts[5], 12)?;
let mut rails = Vec::new();
@@ -638,6 +637,9 @@ impl SkateMap {
});
}
}
if geometry.collision.is_empty() && !extensions.iter().any(|e| e.tag == *b"RWCM" && e.schema == 1 && !e.payload.is_empty()) {
return Err("SKATE requires triangle collision or an embedded RWCM archive".into());
}
if r.at != data.len() {
return Err(format!(
"SKATE contains {} unexplained trailing bytes",
+16 -12
View File
@@ -1,32 +1,32 @@
# Windows installation
**Paused:** the Blender-based conversion approach below was rejected. The
public setup entry point and release packaging are disabled pending direct
conversion. The notes below describe the earlier implementation, not a ready
release. Setup no longer downloads Blender.
`Build-Release.ps1` produces `target/skate3rust-windows-x64.zip`. Keep its
`support` directory beside `skate3rust.exe`. The game links Bevy and the MSVC
runtime statically; the setup helper bundles Python, NumPy, Pillow and Tcl/Tk.
On first launch, setup asks for a local Xbox 360 Skate 3 ISO. It downloads
hash-pinned portable Blender and extract-xiso distributions from their
publishers. No game files are downloaded or included in the package.
the hash-pinned extract-xiso utility from XboxDev. Asset conversion runs in
the bundled setup helper. Blender is neither required nor downloaded.
No game files are downloaded or included in the package.
The conversion pipeline:
1. Extracts the disc into a new installation workspace.
2. Preserves native animation banks and state graphs from `miscload.big`.
2. Preserves native animation banks and state graphs from `miscload.big`,
and the controller mapping from `miscboot.big`.
3. Reads the big-endian 64-bit AttribSys schema and collections from `db.big`.
Settings are decoded from the disc, with numeric hashes retained for names
absent from debug strings. The JSON export serves the fields consumed by
this engine; it is not a general VLT editor or portable array serialization.
4. Builds the default modular skater from the CAC archive, including textures,
morph settings, the native rig and its four board-relative IK targets.
Gameplay animations are evaluated from ABIN by the Rust runtime.
5. Converts each `worldDIST_*.big` into a `.skate` map. The supplied map exporter
runs in v14 storage mode, with native collision archives and grind splines
embedded. University uses the supplied exporter's starting location. Other
Writes GLB directly with retail skin weights, inverse-bind matrices and
the bone-local basis used by the runtime. Gameplay animations are evaluated
from ABIN by the Rust runtime, without offline animation baking.
5. Converts each `worldDIST_*.big` directly into a v14 `.skate` map. Native
collision archives and grind splines are embedded; collision triangles are
decoded from that archive by the game rather than duplicated in the file.
University uses the existing starting location. Other
districts select a broad upward collision face near the district origin.
6. Runs the game's asset and map loaders before publishing `installation.json`.
This validates loading, not gameplay. Conversion intermediates are removed
@@ -37,6 +37,10 @@ to the completed installation. Conversion tools are cached separately, so
retrying does not download them again. `setup-error.log` reports setup errors;
each installation also has `setup.log` with converter output.
For an already extracted disc, developers can use `tools/prepare_assets.py`
with `--game-root`, `--output` and `--game-exe`. The release helper can run the
same entry point with `--task tools/prepare_assets.py`.
The Escape menu discovers installed `.skate` files. Loading a map starts a new
game process and closes the current session so physics and render resources
are recreated. A fresh launch defaults to University. Development checkouts
+126
View File
@@ -0,0 +1,126 @@
"""Write the retail modular skater directly as glTF 2.0 binary."""
from pathlib import Path
import io,json,struct
from PIL import Image
import numpy as np
from tools.asset_pipeline.retail_character import ABIN,RX2,AnimSource,SkeletonSet,quat_matrix,decode_dense_morphs,morph_weight
class Glb:
def __init__(self):
self.data=bytearray()
self.doc={'asset':{'version':'2.0','generator':'Skate 3 Rust Engine'},'bufferViews':[],
'accessors':[],'images':[],'textures':[],'materials':[]}
def view(self,data):
self.data.extend(b'\0'*(-len(self.data)%4));offset=len(self.data);self.data.extend(data)
self.doc['bufferViews'].append({'buffer':0,'byteOffset':offset,'byteLength':len(data)})
return len(self.doc['bufferViews'])-1
def accessor(self,values,kind,component=5126,bounds=False):
a=np.asarray(values,dtype={5126:'<f4',5123:'<u2',5125:'<u4'}[component])
if component==5126 and not np.isfinite(a).all():raise ValueError('Non-finite character data')
item={'bufferView':self.view(a.tobytes()),'componentType':component,'count':len(a),'type':kind}
if bounds:item.update(min=a.min(axis=0).tolist(),max=a.max(axis=0).tolist())
self.doc['accessors'].append(item);return len(self.doc['accessors'])-1
def texture(self,path):
data=path if isinstance(path,bytes) else path.read_bytes()
self.doc['images'].append({'bufferView':self.view(data),'mimeType':'image/png'})
self.doc['textures'].append({'source':len(self.doc['images'])-1})
return {'index':len(self.doc['textures'])-1}
def save(self,path):
self.doc['buffers']=[{'byteLength':len(self.data)}]
text=json.dumps(self.doc,separators=(',',':'),allow_nan=False).encode();text+=b' '*(-len(text)%4)
self.data.extend(b'\0'*(-len(self.data)%4))
path.write_bytes(struct.pack('<III',0x46546c67,2,28+len(text)+len(self.data))+
struct.pack('<I4s',len(text),b'JSON')+text+struct.pack('<I4s',len(self.data),b'BIN\0')+self.data)
def convert(models,private,recipe):
source=AnimSource(private/'stock/data/anim/OnBoard.abin')
skeleton=SkeletonSet(str(models))
if skeleton.errors:raise ValueError(str(skeleton.errors))
mapping=skeleton.index_map(ABIN.BONE_NAMES,source.parents)
worlds={mapping[n]:m for n,m in skeleton.bind.items() if n in mapping}
names=list(ABIN.BONE_NAMES)
reference={}
for index,parent in enumerate(source.parents):
sqt=source.ref_pose.get(index)
local=quat_matrix(sqt.quat,sqt.scale) if sqt else np.eye(4)
if sqt:local[3,:3]=sqt.trans
reference[index]=local@reference[parent] if parent>=0 else local
for index,name in {32:'RIGHTTOEBASE_REPARENTED',33:'LEFTTOEBASE_REPARENTED',34:'LEFTHAND_REPARENTED',35:'RIGHTHAND_REPARENTED'}.items():
worlds[index]=reference[index];names[index]=name
# Retain the established mesh bone-local basis consumed by render_basis()
# in the game. This algebraic change needs no editor or animation bake.
basis_transpose=np.array([[1.,0,0,0],[0,0,-1.,0],[0,1.,0,0],[0,0,0,1.]])
worlds={i:basis_transpose@m for i,m in worlds.items()}
indices=sorted(worlds);joint_index={bone:i for i,bone in enumerate(indices)}
glb=Glb();nodes=[];root_children=[]
# RX2 matrices use row vectors. Their row-major bytes are glTF's
# column-major representation of the equivalent column-vector matrix.
for bone in indices:
parent=source.parents[bone]
while parent>=0 and parent not in worlds:parent=source.parents[parent]
local=worlds[bone]@np.linalg.inv(worlds[parent]) if parent>=0 else worlds[bone]
nodes.append({'name':names[bone],'matrix':local.ravel().tolist()})
if parent<0:root_children.append(joint_index[bone])
for bone in indices:
parent=source.parents[bone]
while parent>=0 and parent not in worlds:parent=source.parents[parent]
if parent>=0:nodes[joint_index[parent]].setdefault('children',[]).append(joint_index[bone])
inverse=glb.accessor([np.linalg.inv(worlds[i]).ravel() for i in indices],'MAT4')
primitives=[];components={c['slot']:c for c in recipe['components']}
meshes=skeleton.renderable_meshes(ABIN.BONE_NAMES,len(source.parents),source.parents)
if {m['folder'] for m in meshes}!=set(components):raise ValueError('Incomplete character mesh set')
for mesh in meshes:
slot=mesh['folder'];component=components[slot];path=models/slot/mesh['name']
parsed=RX2.parse_rx2(str(path));raw=[m for m in parsed['meshes'] if m.get('positions') and m.get('indices')]
if len(raw)!=1:raise ValueError('Ambiguous character mesh '+slot)
raw=raw[0];positions=np.asarray(mesh['pos'],dtype=np.float64)
morphs=decode_dense_morphs(path,parsed,len(positions),RX2)
if [m['name'] for m in morphs]!=recipe['morph_assembly']['expected_targets'][slot]:raise ValueError('Unexpected morph set '+slot)
for morph in morphs:
weight=morph_weight(morph['name'],recipe)
if weight and morph['nonzero_normal_deltas']:raise ValueError('Unsupported morph normals '+slot)
positions+=np.asarray(morph['deltas'])*weight
normals=np.asarray(raw['normals'],dtype=np.float64);normals/=np.maximum(np.linalg.norm(normals,axis=1,keepdims=True),1e-20)
joints=np.zeros((len(positions),4),dtype=np.uint16);weights=np.zeros((len(positions),4),dtype=np.float32)
if not mesh['skin']:raise ValueError('Missing skin '+slot)
for i,influences in enumerate(mesh['skin']):
combined={}
for bone,w in influences:combined[bone]=combined.get(bone,0)+w
if len(combined)>4 or not combined:raise ValueError('Invalid skin influences '+slot)
for j,(bone,w) in enumerate(combined.items()):joints[i,j]=joint_index[bone];weights[i,j]=w
weights/=weights.sum(axis=1,keepdims=True)
folder=private/'default_skater/textures/materials'
pbr={'baseColorFactor':[*component['tint'],1.0],
'baseColorTexture':glb.texture(folder/(slot+'_base_color.png')),
'metallicFactor':0.65 if slot=='SkateTruck' else 0.0,
'roughnessFactor':0.48 if slot in {'SkateTruck','SkateWheel'} else 0.72}
rough=folder/(slot+'_roughness.png')
if rough.is_file():
channel=Image.open(rough).convert('RGB').getchannel('R')
white=Image.new('L',channel.size,255);packed=Image.merge('RGB',(white,channel,white));buffer=io.BytesIO();packed.save(buffer,format='PNG')
pbr['metallicRoughnessTexture']=glb.texture(buffer.getvalue());pbr['roughnessFactor']=1.0
material={'name':'Retail_'+slot,'pbrMetallicRoughness':pbr,'doubleSided':False,
'alphaMode':component.get('alpha_mode','OPAQUE')}
if material['alphaMode']=='MASK':material['alphaCutoff']=component.get('alpha_cutoff',0.5)
if component['textures'].get('normal'):material['normalTexture']=glb.texture(folder/(slot+'_normal.png'))
glb.doc['materials'].append(material)
attributes={'POSITION':glb.accessor(positions,'VEC3',bounds=True),'NORMAL':glb.accessor(normals,'VEC3'),
'TEXCOORD_0':glb.accessor(raw['uvs'],'VEC2'),'JOINTS_0':glb.accessor(joints,'VEC4',5123),
'WEIGHTS_0':glb.accessor(weights,'VEC4')}
primitives.append({'attributes':attributes,'indices':glb.accessor(np.asarray(mesh['tris']).ravel(),'SCALAR',5125),
'material':len(glb.doc['materials'])-1})
mesh_node=len(nodes);nodes.append({'name':'Skate3_SkaterAndBoard','mesh':0,'skin':0})
root=len(nodes);nodes.append({'name':'Skate3_RX2_Rig','children':root_children+[mesh_node]})
glb.doc.update(nodes=nodes,meshes=[{'primitives':primitives}],skins=[{'joints':list(range(len(indices))),
'inverseBindMatrices':inverse}],scenes=[{'nodes':[root]}],scene=0)
glb.save(private/'skater.glb')
if __name__=='__main__':
import argparse
parser=argparse.ArgumentParser(description=__doc__)
parser.add_argument('--models',type=Path,required=True)
parser.add_argument('--private',type=Path,required=True)
parser.add_argument('--recipe',type=Path,required=True)
args=parser.parse_args()
convert(args.models,args.private,json.loads(args.recipe.read_text()))
print('CHARACTER_GLB_READY')
+8 -20
View File
@@ -54,8 +54,8 @@ def run(args,log,report):
kwargs={'creationflags':subprocess.CREATE_NO_WINDOW} if os.name=='nt' else {}
external=os.name=='nt' and getattr(sys,'frozen',False) and Path(args[0]).resolve()!=Path(sys.executable).resolve()
if external:
# PyInstaller's DLL directory is inherited by children. Blender and
# the game must load their own libraries, not the setup bundle's.
# External tools and the game must load their own libraries, not
# the setup bundle's DLL directory inherited by child processes.
import ctypes
ctypes.windll.kernel32.SetDllDirectoryW(None)
env=os.environ.copy()
@@ -82,13 +82,13 @@ def extract(archive,destination,entries=None):
data.extract_entries(data.entries if entries is None else [e for e in data.entries if entries(e)],destination)
return data
def convert_map(archive,work,maps,stage,blender,game_exe,log,report):
def convert_map(archive,work,maps,stage,game_exe,log,report):
map_tools=TOOLS/'vendor/university/tools/vanilla_map_extraction/tools'
sys.path.insert(0,str(map_tools))
from prepare_hawaiian_dream import prepare
from prepare_university import EXCLUDED_NORMAL_TEXTURE_IDS
from build_retail_collision_archive import build_archive
from embed_retail_collision_archive import embed_archive
from .map_writer import write as write_map
district=archive.stem.removeprefix('world')
label=district.removeprefix('DIST_')
district_work=work/district
@@ -99,13 +99,10 @@ def convert_map(archive,work,maps,stage,blender,game_exe,log,report):
utt_root=TOOLS/'vendor/utt',district_name=district,map_name=label,
package_name='Skate 3 owned disc',cache_format='skate3-rust-map-v1',
texture_stream_names=('Tex',),excluded_normal_texture_ids=EXCLUDED_NORMAL_TEXTURE_IDS)
raw_package=district_work/(label+'.skate')
run([blender,'--background','--factory-startup','--python-exit-code','1','--python',
TOOLS/'asset_pipeline/build_map.py','--',TOOLS,manifest_path,district_work/'map.blend',raw_package,district],log,report)
collision=district_work/'collision.rwcmset'
build_archive(manifest_path,collision)
final=maps/(label+'.skate')
embed_archive(raw_package,collision,final)
write_map(manifest_path,final,collision,report)
report('Checking converted map: '+label)
run([game_exe,'--assets',stage/'assets','--map',final,'--check-assets'],log,report)
entry={'name':label,'path':'maps/'+final.name,'sha256':digest(final)}
@@ -114,10 +111,6 @@ def convert_map(archive,work,maps,stage,blender,game_exe,log,report):
def install(iso,base,game_exe,report,game_root=None):
raise RuntimeError('ISO setup is paused while direct asset conversion is implemented. No Blender will be downloaded or installed.')
def _authoring_install(iso,base,game_exe,report,blender,game_root=None):
base=base.resolve();base.mkdir(parents=True,exist_ok=True)
lock=base/'setup.lock'
try:fd=os.open(lock,os.O_CREAT|os.O_EXCL|os.O_WRONLY)
@@ -169,13 +162,8 @@ def _authoring_install(iso,base,game_exe,report,blender,game_root=None):
run(task(TOOLS/'extract_default_skater.py','--owned-data-root',stock,'--work-root',character,
'--private-root',private/'default_skater','--utt-root',TOOLS/'vendor/utt'),log,report)
report('Building the skater model and rig')
blend=character/'skater.blend'
run([blender,'--background','--factory-startup','--python-exit-code','1',
'--python',TOOLS/'vendor/skate3_anim/blender_rx2_abin_export.py','--',
'--abin',stock/'data/anim/OnBoard.abin','--rx2',character/'selected/models',
'--clip','R_IDLE_HCOM_000','--output',blend],log,report)
run([blender,'--background','--factory-startup','--python-exit-code','1',blend,'--python',
TOOLS/'asset_pipeline/finish_character.py','--',TOOLS,character,private],log,report)
from .character_glb import convert as write_character
write_character(character/'selected/models',private,manifest)
game_manifest={'version':1,'character_scene':'private/skater.glb','initial_animation':'R_IDLE_HCOM_000',
'action_graph':'private/stock/data/state/ActionGraph_OnBoard.stategraph',
'motion_graph':'private/stock/data/state/MotionGraph_OnBoard.stategraph'}
@@ -187,7 +175,7 @@ def _authoring_install(iso,base,game_exe,report,blender,game_root=None):
catalog=[]
for number,archive in enumerate(archives,1):
report(f'Converting map {number}/{len(archives)}: {archive.stem}')
catalog.append(convert_map(archive,work,maps,stage,blender,game_exe,log,report))
catalog.append(convert_map(archive,work,maps,stage,game_exe,log,report))
if not any(m['name']=='University' for m in catalog):raise RuntimeError('University was not converted')
report('Validating installed runtime inputs')
run([game_exe,'--assets',stage/'assets','--test-world','--check-assets'],log,report)
+133
View File
@@ -0,0 +1,133 @@
"""Direct retail cache -> SKATE14 writer. All geometry remains in Y-up metres."""
from pathlib import Path
import io,json,struct,sys,zlib
import numpy as np
from PIL import Image
from tools.asset_pipeline.retail_material import _retail_shader_family,_retail_render_flags
def u(f,*v):f.write(struct.pack('<'+'I'*len(v),*v))
def floats(f,*v):f.write(struct.pack('<'+'f'*len(v),*v))
def string(f,s):
b=str(s).encode();u(f,len(b));f.write(b)
def stored(f,data):
packed=zlib.compress(data,6)
method=1
if len(packed)>=len(data):method=0;packed=data
u(f,method,len(packed));f.write(packed)
def normalise(a):return a/np.maximum(np.linalg.norm(a,axis=1,keepdims=True),1e-20)
def spawn_point(manifest,root):
from retail_collision_mesh import decode_rx2_clustered_meshes
university=manifest['district_name']=='DIST_University'
best=None
for entry in manifest['simulation_assets']:
if not entry.get('collision_meshes'):continue
for mesh in decode_rx2_clustered_meshes((root/entry['rx2']).read_bytes()):
for tri in mesh.triangles:
a,b,c=np.asarray([tri.a,tri.b,tri.c]);cross=np.cross(b-a,c-a);length=np.linalg.norm(cross)
if length<4 or cross[1]/length<0.9:continue
if university:
x,z=330.,-710.;basis=np.array([[b[0]-a[0],c[0]-a[0]],[b[2]-a[2],c[2]-a[2]]])
if abs(np.linalg.det(basis))<1e-12:continue
weights=np.linalg.solve(basis,[x-a[0],z-a[2]])
if weights.min() < -1e-5 or weights.sum()>1.00001:continue
height=a[1]+weights[0]*(b[1]-a[1])+weights[1]*(c[1]-a[1]);score=abs(height-132.)
point=(x,height+1.,z)
else:
p=(a+b+c)/3;score=p[0]**2+p[2]**2;point=(p[0],p[1]+1.,p[2])
if best is None or score<best[0]:best=(score,point)
if best is None:raise ValueError('No supported spawn surface in '+manifest['map_name'])
return best[1]
def write(manifest_path,output,collision,report=lambda _:None):
root=manifest_path.parent;m=json.loads(manifest_path.read_text());textures=m['textures']
ids={name:i+1 for i,name in enumerate(sorted(textures))}
excluded=set(m['normal_texture_policy']['excluded_texture_ids'])
mats=io.BytesIO();vertices=io.BytesIO();indices=io.BytesIO();nv=ni=nm=0
dtype=np.dtype([('p','<f4',(3,)),('n','<f4',(3,)),('uv','<f4',(2,)),('lm','<f4',(2,)),
('mat','<u4'),('decal','<f4',(2,)),('frame','i1',(4,))])
for number,model in enumerate(m['models']):
if number%100==0:report(f"Writing {m['map_name']}: model {number+1}/{len(m['models'])}")
with np.load(root/model['npz'],allow_pickle=False) as arrays:
for mesh in model['meshes']:
i=mesh['index'];pos=arrays[f'vertices_{i}'];faces=arrays[f'faces_{i}'].astype('<u4')
if not len(pos) or not len(faces):continue
if faces.max()>=len(pos):raise ValueError('Map face is outside vertex array')
shader=mesh.get('shader_name') or '';roles=mesh.get('retail_texture_ids',{})
albedo=mesh.get('texture_id');light=roles.get('lightmap') if f'lightmap_uvs_{i}' in arrays and not shader.startswith(('water.','ocean.')) else None
normal=roles.get('normal');normal=None if normal in excluded else normal
def tex(name):
if not name:return 0
if name not in ids:raise ValueError('Missing retail map texture '+name)
return ids[name]
string(mats,f"{model['asset_id']}_{i}");u(mats,1);floats(mats,.82,0.,*( (0.,0.,0.) if albedo=='0x0000119903e3870a' else (.8,.8,.8)),.68,0.)
u(mats,tex(albedo),tex(light));floats(mats,.25 if light else 0.);u(mats,tex(normal),0,0,mesh.get('alpha_mode',0));floats(mats,mesh.get('alpha_cutoff',.5))
# The established exporter derives layers from the authored
# alpha mode for the generated MAT_<texture-id> names.
depth=3 if mesh.get('alpha_mode',0)==2 else 1 if mesh.get('alpha_mode',0)==1 else 0
u(mats,3,1,0,depth,int(bool(shader)))
if shader:
mats.write(struct.pack('<QIi',int(mesh.get('retail_material_guid','0'),0),int(mesh.get('retail_material_handle','0'),0),mesh.get('retail_material_group_index',-1)))
string(mats,shader);u(mats,_retail_shader_family(shader),_retail_render_flags(shader,mesh.get('alpha_mode',0)),len(roles))
for role,name in sorted(roles.items()):
uv=1 if role in {'lightmap','chromaticity','alpha'} else 2 if role=='decal' else 0
clamp=int(role in {'lightmap','chromaticity'} or (role=='decal' and not shader.startswith('environment.decal_tileable')))
string(mats,role);u(mats,tex(name),uv,clamp,clamp)
params=mesh.get('retail_parameters',{});u(mats,len(params))
for name,values in params.items():
string(mats,name);u(mats,len(values))
for value in values:string(mats,value)
string(mats,json.dumps({'asset_id':model['asset_id'],'mesh_index':i,'source_offsets':mesh['source_offsets']}))
record=np.zeros(len(pos),dtype=dtype);record['p']=pos;record['mat']=nm+1
normals=arrays.get(f'retail_normals_{i}',arrays.get(f'normals_{i}'))
if normals is None:
normals=np.zeros_like(pos);face_normals=np.cross(pos[faces[:,1]]-pos[faces[:,0]],pos[faces[:,2]]-pos[faces[:,0]])
for corner in range(3):np.add.at(normals,faces[:,corner],face_normals)
normals=normalise(normals);record['n']=normals
for target,key in [('uv',f'uvs_{i}'),('lm',f'lightmap_uvs_{i}'),('decal',f'decal_uvs_{i}')]:
uv=np.array(arrays.get(key,arrays.get(f'uvs_{i}',np.zeros((len(pos),2)))),copy=True)
if target=='lm' and key in arrays and shader!='ocean.default':uv=np.abs(uv)
uv[:,1]=1.-uv[:,1];record[target]=uv
if f'retail_tangents_{i}' in arrays:
tangent=np.array(arrays[f'retail_tangents_{i}'],copy=True);tangent-=normals*np.sum(tangent*normals,axis=1,keepdims=True)
sign=np.array(arrays[f'retail_tangent_handedness_{i}'],copy=True).reshape(-1);sign[np.linalg.norm(tangent,axis=1)<1e-12]=0
binormal=np.cross(normals,normalise(tangent))*sign[:,None]
record['frame'][:,:3]=np.rint(np.clip(binormal,-1,1)*127).astype('i1');record['frame'][:,3]=np.rint(sign*127).astype('i1')
if not np.isfinite(pos).all():raise ValueError('Non-finite map geometry')
vertices.write(record.tobytes());indices.write((faces+nv).astype('<u4').tobytes());nv+=len(pos);ni+=faces.size;nm+=1
report('Selecting starting position: '+m['map_name']);spawn=spawn_point(m,root)
# Match the supplied exporter's environment defaults. Native sky shaders
# remain a separate runtime feature; no geometry is synthesized here.
environment=[.10,.36,.75,.64,.82,1.,.18,.24,.30,0.,11.,0.,18.,0.,
.045,.10,.26,1.,.32,.10,.05,.035,.06,.007,.015,.045,.045,.085,.17,.008,.014,.032,
1.,.96,.86,.42,.56,.92,1.,.18,.34,.10,1.,1.,1.]
rails=m['grind_splines'];output.parent.mkdir(parents=True,exist_ok=True)
with output.open('wb') as f:
f.write(b'SKATE14\0');u(f,0x12345678);string(f,m['map_name']);floats(f,*spawn,0.,*environment)
u(f,nm,len(ids),nv,ni,0,len(rails),0,0,0);f.write(mats.getvalue())
for name in sorted(ids):
image=Image.open(root/textures[name]['png']).convert('RGBA').transpose(Image.Transpose.FLIP_TOP_BOTTOM)
string(f,name);u(f,image.width,image.height,1);stored(f,image.tobytes())
stored(f,vertices.getvalue());stored(f,indices.getvalue());stored(f,b'')
for rail in rails:
string(f,f"{rail['asset_id']}_{rail['section_index']}_{rail['rail_index']}")
u(f,int(rail['closed']),1);f.write(struct.pack('<QQ',int(rail['spline_id'],0),int(rail['type_signature'],0)))
u(f,rail['flags'],rail['trailing_word'],rail['segment_count'])
for segment in rail['native_segment_payloads']:
raw=bytes.fromhex(segment)
if len(raw)!=120:raise ValueError('Invalid native spline segment')
f.write(np.frombuffer(raw,dtype='>u4').astype('<u4').tobytes())
u(f,2)
for tag,data in [(b'RWCM',collision.read_bytes()),(b'WMET',json.dumps(m,separators=(',',':')).encode())]:
f.write(tag);u(f,1,len(data));stored(f,data)
report('Map written: '+m['map_name'])
if __name__=='__main__':
import argparse
parser=argparse.ArgumentParser(description=__doc__)
parser.add_argument('--manifest',type=Path,required=True)
parser.add_argument('--output',type=Path,required=True)
parser.add_argument('--collision',type=Path,required=True)
args=parser.parse_args()
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'vendor/university/tools/vanilla_map_extraction/tools'))
write(args.manifest,args.output,args.collision,lambda text:print(text,flush=True))
+464
View File
@@ -0,0 +1,464 @@
"""Pure RX2/ABIN rig and morph decoding extracted from the project converters.
See tools/vendor/skate3_anim/PROVENANCE.md and THIRD_PARTY_NOTICES.md.
No scene editor or rendering application is used.
"""
from __future__ import annotations
import glob,math,os,struct,sys
from pathlib import Path
import numpy as np
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'vendor/skate3_anim'))
import abin_importer as ABIN
import rx2_skeleton as RX2
TYPE_RAW_BUFFER=0x00010031
TYPE_VTX_DESC=0x000200E9
TYPE_VB_DESC=0x000200EA
TYPE_VTX_DECL=0x00020081
TYPE_MESH_DESC=0x00EB0023
TYPE_MORPH_DESC=0x00EB000E
MORPH_VERTEX_STRIDE=16
MORPH_POSITION_SCALE=1.0/16384.0
def quat_matrix(quaternion, scale=(1.0, 1.0, 1.0)):
"""Viewer-equivalent row-vector SQT rotation matrix."""
x, y, z, w = quaternion
norm = x * x + y * y + z * z + w * w
if norm < 1e-12:
x = y = z = 0.0
w = 1.0
else:
inverse_norm = 1.0 / math.sqrt(norm)
x, y, z, w = (
x * inverse_norm,
y * inverse_norm,
z * inverse_norm,
w * inverse_norm,
)
xx, yy, zz = x * x, y * y, z * z
xy, xz, yz = x * y, x * z, y * z
wx, wy, wz = w * x, w * y, w * z
matrix = np.identity(4, dtype=np.float64)
matrix[0, 0] = 1 - 2 * (yy + zz)
matrix[0, 1] = 2 * (xy + wz)
matrix[0, 2] = 2 * (xz - wy)
matrix[1, 0] = 2 * (xy - wz)
matrix[1, 1] = 1 - 2 * (xx + zz)
matrix[1, 2] = 2 * (yz + wx)
matrix[2, 0] = 2 * (xz + wy)
matrix[2, 1] = 2 * (yz - wx)
matrix[2, 2] = 1 - 2 * (xx + yy)
sx, sy, sz = scale
matrix[0, :3] *= sx if abs(sx) >= 1e-9 else 1.0
matrix[1, :3] *= sy if abs(sy) >= 1e-9 else 1.0
matrix[2, :3] *= sz if abs(sz) >= 1e-9 else 1.0
return matrix
def rigid_inverse(matrix):
result = np.identity(4, dtype=np.float64)
rotation = matrix[:3, :3]
result[:3, :3] = rotation.T
result[3, :3] = -(matrix[3, :3] @ rotation.T)
return result
class AnimSource:
def __init__(self, path):
self.path = path
self.data = Path(path).read_bytes()
self.abin = ABIN.AbinFile(self.data)
self.hier = self.abin.hierarchy
self.parents = list(self.hier.parents) if self.hier else []
self.clips = list(self.abin.clips)
self.poses = list(self.abin.poses)
self._decoders = {}
self._cache = {}
self._cache_key = None
self.ref_pose = self._load_ref_pose()
def _load_ref_pose(self):
for pose in self.poses:
if pose.header.name == "RIG_TPOSE":
return ABIN.decode_pose_frame(
self.data, pose, self.hier
)
return {}
def frame(self, clip_index, frame_index):
if self._cache_key != clip_index:
self._cache_key = clip_index
self._cache = {}
self._decoders = {}
if frame_index not in self._cache:
self._cache[frame_index] = ABIN.decode_clip_frame(
self.data,
self.clips[clip_index],
self._decoders,
frame_index,
self.hier,
)
return self._cache[frame_index]
class SkeletonSet:
"""Blender-safe copy of SkateAnimViewer's RX2-to-ABIN mapper."""
def __init__(self, folder):
self.models = []
self.bind = {}
self.ibp = {}
self.errors = []
files = sorted(
glob.glob(os.path.join(folder, "**", "*.rx2"), recursive=True)
)
for path in files:
try:
parsed = RX2.parse_rx2(path)
except Exception as error:
self.errors.append(f"{Path(path).name}: {error}")
continue
bones = parsed.get("bones") or []
meshes = parsed.get("meshes") or []
self.models.append(
(Path(path).parent.name, Path(path).name, bones, meshes)
)
for bone in bones:
name = (bone.get("name") or "").upper()
if not name:
continue
if name not in self.bind and bone.get("bind_matrix") is not None:
self.bind[name] = np.asarray(
bone["bind_matrix"], dtype=np.float64
).reshape(4, 4)
if name not in self.ibp and bone.get("ibp_matrix") is not None:
self.ibp[name] = np.asarray(
bone["ibp_matrix"], dtype=np.float64
).reshape(4, 4)
def _rx2_parent_names(self):
if getattr(self, "_parent_names", None) is not None:
return self._parent_names
parent_names = {}
for _, _, bones, _ in self.models:
by_index = {bone.get("index", -1): bone for bone in bones}
for bone in bones:
name = (bone.get("name") or "").upper()
if not name or name in parent_names:
continue
parent = by_index.get(bone.get("parent", -1))
parent_names[name] = (
(parent.get("name") or "").upper() if parent else None
)
self._parent_names = parent_names
return parent_names
def index_map(self, names, parents=None):
upper = [name.upper() for name in names]
result = {}
self.rejected = []
if parents is None:
for index, name in enumerate(upper):
result.setdefault(name, index)
return result
known = set(upper[: len(parents)])
rx2_parents = self._rx2_parent_names()
for index in range(min(len(parents), len(upper))):
name = upper[index]
if not name or name in result:
continue
rx2_parent = rx2_parents.get(name)
if rx2_parent is not None and rx2_parent in known:
parent = parents[index]
expected = upper[parent] if 0 <= parent < len(upper) else None
if expected != rx2_parent:
self.rejected.append(
(index, name, expected, rx2_parent)
)
continue
result[name] = index
return result
def local_bind_matrices(self, parents, names):
mapping = self.index_map(names, parents)
world = {
index: matrix
for name, matrix in self.bind.items()
if (index := mapping.get(name)) is not None
and index < len(parents)
}
return {
index: (
matrix @ rigid_inverse(world[parent])
if (parent := parents[index]) in world
else matrix.copy()
)
for index, matrix in world.items()
}
def renderable_meshes(self, names, bone_count, parents=None):
mapping = self.index_map(names, parents)
output = []
for folder, filename, bones, meshes in self.models:
bones_by_index = {
bone.get("index", -1): bone for bone in bones
}
resolved = {}
def resolve(bone_index):
if bone_index in resolved:
return resolved[bone_index]
current = bone_index
seen = set()
answer = None
while (
current is not None
and current >= 0
and current not in seen
):
seen.add(current)
bone = bones_by_index.get(current)
if bone is None:
break
candidate = mapping.get(
(bone.get("name") or "").upper()
)
if candidate is not None and candidate < bone_count:
answer = candidate
break
current = bone.get("parent", -1)
resolved[bone_index] = answer
return answer
local_to_abin = {
bone.get("index", -1): answer
for bone in bones
if (answer := resolve(bone.get("index", -1))) is not None
}
for mesh in meshes:
positions = mesh.get("positions")
indices = mesh.get("indices")
if not positions or not indices:
continue
triangles = [
tuple(indices[index : index + 3])
for index in range(0, len(indices) - 2, 3)
]
triangles = [
triangle
for triangle in triangles
if max(triangle) < len(positions)
]
raw_indices = mesh.get("skin_indices")
raw_weights = mesh.get("skin_weights")
used = mesh.get("used_bone_indices") or []
skin = None
if (
raw_indices
and raw_weights
and len(raw_indices) >= len(positions)
and len(raw_weights) >= len(positions)
):
skin = []
for vertex_index in range(len(positions)):
influences = []
for palette_index, weight in zip(
raw_indices[vertex_index],
raw_weights[vertex_index],
):
if weight <= 1e-4:
continue
palette_index = int(palette_index)
model_bone = (
used[palette_index]
if palette_index < len(used)
else palette_index
)
abin_bone = local_to_abin.get(int(model_bone))
if abin_bone is not None:
influences.append(
(abin_bone, float(weight))
)
skin.append(influences)
fallback = next(
(
local_to_abin[bone]
for bone in used
if bone in local_to_abin
),
None,
)
if fallback is None:
fallback = self._nearest_bone(
positions, names, bone_count, parents
)
if skin is not None and fallback is not None:
skin = [
influences or [(fallback, 1.0)]
for influences in skin
]
output.append(
{
"name": filename,
"folder": folder,
"pos": positions,
"tris": triangles,
"skin": skin,
}
)
best = {}
for mesh in output:
key = mesh["folder"]
if key not in best or len(mesh["pos"]) > len(best[key]["pos"]):
best[key] = mesh
return [best[key] for key in sorted(best)]
def _nearest_bone(self, positions, names, bone_count, parents=None):
mapping = self.index_map(names, parents)
candidates = [
(index, matrix[3, :3])
for name, matrix in self.bind.items()
if (index := mapping.get(name)) is not None
and index < bone_count
]
if not candidates:
return None
centroid = np.asarray(positions, dtype=np.float64).mean(axis=0)
return min(
candidates,
key=lambda item: float(np.linalg.norm(item[1] - centroid)),
)[0]
def be_u32(data: bytes, offset: int) -> int:
return struct.unpack_from(">I", data, offset)[0]
def morph_weight(name: str, manifest: dict) -> float:
body = manifest["preset"]["body_mods"]
if name == "fat" or name.startswith("fat_"):
return float(body["fatness"])
if name == "thin" or name.startswith("thin_"):
return float(body["skinniness"])
if name in manifest["morph_assembly"]["face_targets"]:
return float(body["face_fields"])
raise RuntimeError(f"Retail morph target has no recipe mapping: {name}")
def decode_dense_morphs(
path: Path, parsed: dict, vertex_count: int, rx2_skeleton
) -> list[dict]:
data = path.read_bytes()
sections = parsed["sections"]
mesh_descs = [
section
for section in sections
if section["type_code"] == TYPE_MESH_DESC
]
if len(mesh_descs) != 1:
raise RuntimeError(
f"{path.name} expected one mesh descriptor, found {len(mesh_descs)}"
)
base = mesh_descs[0]["file_offset"]
morph_count = be_u32(data, base + 0x50)
morph_table = be_u32(data, base + 0x54)
morphs = []
for morph_index in range(morph_count):
entry = base + morph_table + morph_index * 16
target_hash, descriptor_index, name_offset = struct.unpack_from(
">QII", data, entry
)
name_start = base + name_offset
name_end = data.find(b"\0", name_start)
if name_end < 0:
raise RuntimeError(
f"{path.name} morph {morph_index} has no terminated name"
)
name = data[name_start:name_end].decode("ascii")
if descriptor_index < 4 or descriptor_index >= len(sections):
raise RuntimeError(
f"{path.name} morph {name} has invalid descriptor index "
f"{descriptor_index}"
)
expected_types = (
TYPE_RAW_BUFFER,
TYPE_VB_DESC,
TYPE_VTX_DECL,
TYPE_VTX_DESC,
TYPE_MORPH_DESC,
)
stream_sections = sections[descriptor_index - 4 : descriptor_index + 1]
actual_types = tuple(
section["type_code"] for section in stream_sections
)
if actual_types != expected_types:
raise RuntimeError(
f"{path.name} morph {name} stream layout changed: "
f"{actual_types} != {expected_types}"
)
raw, vb_desc, _vtx_decl, vtx_desc, _morph_desc = stream_sections
byte_size = be_u32(data, vb_desc["file_offset"] + 0x20)
if byte_size != vertex_count * MORPH_VERTEX_STRIDE:
raise RuntimeError(
f"{path.name} morph {name} byte size changed: "
f"{byte_size} != {vertex_count * MORPH_VERTEX_STRIDE}"
)
if raw["size"] != byte_size:
raise RuntimeError(
f"{path.name} morph {name} raw-buffer size changed"
)
descriptor = rx2_skeleton._parse_vertex_descriptor(
data, vtx_desc["file_offset"], vtx_desc["size"]
)
expected_elements = [
("POSITION", 0, "SHORT4", 0),
("NORMAL", 0, "PACKED11_11_10N", 8),
("BLENDWEIGHT", 0, "UNKNOWN_0x002C83A4", 12),
]
actual_elements = [
(
element["usage_name"],
element["usage_index"],
element["format_name"],
element["offset"],
)
for element in descriptor["elements"]
]
if (
descriptor["stride"] != MORPH_VERTEX_STRIDE
or actual_elements != expected_elements
):
raise RuntimeError(
f"{path.name} morph {name} vertex declaration changed"
)
deltas = []
nonzero_vertices = 0
nonzero_normal_deltas = 0
max_delta = 0.0
for vertex_index in range(vertex_count):
vertex_offset = (
raw["file_offset"] + vertex_index * MORPH_VERTEX_STRIDE
)
x, y, z, _remap = struct.unpack_from(
">4h", data, vertex_offset
)
if be_u32(data, vertex_offset + 8) != 0:
nonzero_normal_deltas += 1
if be_u32(data, vertex_offset + 12) != 0x3F800000:
raise RuntimeError(
f"{path.name} morph {name} has an unhandled vertex factor"
)
delta = (
x * MORPH_POSITION_SCALE,
y * MORPH_POSITION_SCALE,
z * MORPH_POSITION_SCALE,
)
length = math.sqrt(sum(value * value for value in delta))
if length > 0.0:
nonzero_vertices += 1
max_delta = max(max_delta, length)
deltas.append(delta)
morphs.append(
{
"name": name,
"hash": f"{target_hash:016x}",
"descriptor_index": descriptor_index,
"deltas": deltas,
"nonzero_vertices": nonzero_vertices,
"nonzero_normal_deltas": nonzero_normal_deltas,
"max_source_delta": max_delta,
}
)
return morphs
+62
View File
@@ -0,0 +1,62 @@
"""Retail shader classification shared with the supplied map exporter."""
def _retail_shader_family(shader_name: str) -> int:
shader = shader_name.lower()
if shader.startswith("environment.reflective_simple"):
return 6
if shader.startswith("environment.reflective_trans"):
return 13
if shader.startswith("environment.reflective"):
return 5
if shader.startswith("environment.decal_tileable"):
return 4
if shader.startswith("environment.decal"):
return 3
if shader.startswith("environment.default"):
return 1
if shader.startswith("environmentsimple.alphatest"):
return 7
if shader.startswith("environmentsimple.diffuse"):
return 8
if shader.startswith("environmentsimple.default"):
return 2
if shader.startswith("tree.default"):
return 9
if shader.startswith("animated.tree"):
return 10
if shader.startswith("proxyworld."):
return 11
if shader.startswith("incandescent.backlituvscroll"):
return 14
if shader.startswith("incandescent.default"):
return 12
if shader.startswith("water.flowing"):
return 30
if shader.startswith("ocean.default"):
return 31
if shader.startswith("ocean.reflection"):
return 32
if shader.startswith("sky."):
return 40
return 0
def _retail_render_flags(shader_name: str, alpha_mode: int) -> int:
shader = shader_name.lower()
flags = 0
if alpha_mode == 1:
flags |= 1
elif alpha_mode == 2:
flags |= 2
if (
shader.startswith(("tree.", "animated.tree"))
or "alphatest" in shader
):
flags |= 1 | 4
if shader.startswith("sky."):
flags |= 8
if shader.startswith("environment.decal"):
flags |= 16
if shader.startswith("environment.decal_tileable"):
flags |= 32
if shader.startswith(("water.", "ocean.")):
flags |= 64
return flags
+15
View File
@@ -0,0 +1,15 @@
"""Prepare a local extracted Skate 3 disc without a scene editor."""
from pathlib import Path
import argparse,sys
sys.path.insert(0,str(Path(__file__).resolve().parents[1]))
from tools.asset_pipeline.install import install
def main():
parser=argparse.ArgumentParser(description=__doc__)
parser.add_argument('--game-root',type=Path,required=True)
parser.add_argument('--output',type=Path,required=True)
parser.add_argument('--game-exe',type=Path,required=True)
args=parser.parse_args()
install(None,args.output,args.game_exe.resolve(),lambda text:print(text,flush=True),args.game_root)
if __name__=='__main__':main()
+1 -1
View File
@@ -29,7 +29,7 @@ def main():
if icon.is_file():window.iconbitmap(str(icon))
frame=ttk.Frame(window,padding=24);frame.pack(fill='both',expand=True)
ttk.Label(frame,text='Set up Skate 3 Rust Engine',font=('Segoe UI',20)).pack(anchor='w',pady=(0,16))
ttk.Label(frame,text='Select your Xbox 360 Skate 3 ISO.\nYour game files stay on this PC. Setup downloads conversion tools,\nthen prepares the skater, animations and all disc districts.\n\nMap conversion can take a long time and needs substantial free disk space.',
ttk.Label(frame,text='Select your Xbox 360 Skate 3 ISO.\nSetup extracts your game and prepares the skater, animations\nand all disc maps automatically. No other apps need installing.\n\nThe first setup needs internet access and free disk space.\nLarge maps can take a while to convert.',
font=('Segoe UI',11),justify='left').pack(anchor='w')
status=tk.StringVar(value='Choose your ISO to begin.')
ttk.Label(frame,textvariable=status,wraplength=600).pack(anchor='w',pady=(18,8))