diff --git a/.github/workflows/source-check.yml b/.github/workflows/source-check.yml index 2de8eeb..1f68e08 100644 --- a/.github/workflows/source-check.yml +++ b/.github/workflows/source-check.yml @@ -44,3 +44,4 @@ jobs: - run: python -m unittest discover -s tests -p 'test_hide_league_client_in_game.py' - run: python -m unittest discover -s tests -p 'test_client_in_game_tricks.py' - run: python -m unittest discover -s tests -p 'test_setup_portability.py' + - run: python -m unittest discover -s tests -p 'test_preflight.py' diff --git a/README.md b/README.md index bcf03f7..4121222 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ Game files, Riot installers, account databases, tokens, local prefixes, captures ## Requirements - x86-64 Linux **5.14 or newer**, an X11 session or Xwayland, and a working hardware Vulkan/OpenGL driver. AMD RADV and the NVIDIA driver (proprietary and open kernel modules) have been exercised, see [NVIDIA](docs/TROUBLESHOOTING.md#nvidia); other drivers are not validated. Hybrid laptops usually render on the integrated GPU unless started with PRIME offload. See [hybrid graphics](docs/TROUBLESHOOTING.md#hybrid-graphics-laptops-amdintel-igpu--nvidia). -- Git + Git LFS, Bash, curl, unzip, CMake, Make, Clang/LLD, LLVM development files, a C/C++ compiler, Python 3.12+ and the Darling build dependencies. The patch includes LLVM 22/23 branch-opcode compatibility; the new PC used LLVM/Clang 23. Node.js is needed only for the offline JavaScript checks. +- Git + Git LFS (2.13 or newer), Bash, curl, unzip, CMake 3.20+, Make, Clang/LLD, LLVM development files (16 or newer), a C/C++ compiler, Python 3.10+ and the Darling build dependencies. `scripts/build.sh` and `scripts/install-runtime.sh` run `scripts/preflight.py` first and stop on a version that cannot build or run the result; `RCLOL_SKIP_PREFLIGHT=1` bypasses that gate. The patch includes LLVM 22/23 branch-opcode compatibility, and LLVM 22 or 23 is what the Metal overlays are tested with. Node.js is needed only for the offline JavaScript checks. - X11/XCB/SHM development files for the current runtime helpers. - Enough free disk for Darling sources/build, Git LFS assets and the game: budget at least 50 GB. Building is substantially larger than this source-only repository. diff --git a/docs/SETUP.md b/docs/SETUP.md index a1ad931..cee8d57 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -55,6 +55,8 @@ Linux 5.14 or newer is required for `MADV_POPULATE_WRITE`, used when Darwin's al The runtime installer requires native `pkg-config` entries for `x11`, `x11-xcb`, `xcb-shm` and `xext`, plus the base Darling SDK. +`scripts/build.sh` and `scripts/install-runtime.sh` start with `scripts/preflight.py`, which checks the host versions the build and the runtime need: Python, CMake, Git LFS, the kernel, LLVM and its command line tools, the Vulkan API level, and the X display. A required check that fails stops immediately with one actionable line; the display check only warns, because a build does not need one. Run it directly with `python3 scripts/preflight.py` (`--json` for a machine-readable report), and set `RCLOL_SKIP_PREFLIGHT=1` to bypass the gate for one run. + Run configuration, builds and the per-user runtime installation as your ordinary account. Use sudo only for the base CMake installation and the incremental server installer. Running the whole build or per-user install under sudo can create root-owned files and configure the wrong guest home. `scripts/install-runtime.sh` initializes the default user prefix and creates `~/.darling/private/tmp` with mode `1777`, so the installer can write guest temporary files. It then shuts down Darling before replacing the paired runtime: first exit the game and both clients manually. The normal launcher starts neither an Electron inspector nor client API/UI helpers. It clears inherited `DYLD_INSERT_LIBRARIES`, retains Darling runtime library/framework overrides, supplies an absolute log directory to the launcher, and selects the League product. The log-directory/Chromium argument shim, empty-autofill hook and optional native VPX bridge are no longer injected or built by runtime installation. Existing copies on disk are inactive and may remain. Child-process log-directory rewriting and Chromium feature overrides previously supplied by the shim are removed; client startup without those workarounds needs manual validation. The first-time installer uses `--skip-to-install` and omits the installed client's `--app-root`, avoiding an immediate exit or inaccessible intro screen. The project ships no inspector hooks, client API helpers, input automation or client binary patches. Keep account data, logs and captures private. diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index d5c0909..fe753de 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -48,6 +48,29 @@ never delete a populated Riot/game prefix just to repair the launcher. Use the [setup instructions](SETUP.md) from the repository root as your ordinary user. Exit the game and both clients manually before a full installation or Darling restart. Incremental server/memory installers can replace files without interrupting processes, but their changes need a later full restart to activate. +## Setup stops at preflight + +`scripts/build.sh` and `scripts/install-runtime.sh` now run `scripts/preflight.py` before the source fetch, the build and the per-user install. Host versions that cannot build or run the result fail in seconds with one actionable line instead of after the long steps ([#137](https://github.com/lucannez64/darling-lol/issues/137), [#138](https://github.com/lucannez64/darling-lol/issues/138), [#143](https://github.com/lucannez64/darling-lol/issues/143)): + +- **python** 3.10 or newer. The per-user installers hash files with a streaming helper instead of `hashlib.file_digest`, which only exists from Python 3.11, so the Ubuntu 22.04 default no longer fails at the end of the install. +- **cmake** 3.20 or newer. The bundled indium CMakeLists requires it. +- **git-lfs** 2.13 or newer. The pinned submodules ship libraries as LFS objects. +- **lfs-pointer**: when `build/darling-src` is already prepared, small `.dylib`, `.a`, `.so` and `.o` files are checked for Git LFS pointer text. A pointer file links but crashes the guest, so this is a required check rather than a warning. It passes before the first fetch, when there is nothing to inspect yet. +- **kernel** 5.14 or newer for `MADV_POPULATE_WRITE`. +- **llvm** 16 or newer; 22 or 23 is what the patches are tested with. Iridium calls `LLVMGetVersion`, which LLVM 15 does not export, so an LLVM 15 runtime aborts as soon as Riot creates a Metal device ([#139](https://github.com/lucannez64/darling-lol/issues/139)). LLVM 16-21 builds with a warning. +- **llvm-tools**: `llvm-objdump` and `llvm-nm`. They are now resolved as an unversioned name, as `llvm--`, or from `llvm-config --bindir`, so Ubuntu's `llvm-15-dev` no longer needs symlinks in `/usr/local/bin`. +- **vulkan** API 1.3 and timeline semaphores. Indium rejects older devices, so this stops the setup unless one device reports both the version and the semaphores, as `vulkaninfo` shows per GPU. An installed `vulkaninfo` that fails or times out is also required to be fixed, because the loader or the ICD files are then broken; without `vulkaninfo` at all it only warns, because the tool may be missing even when the driver is fine. +- **x11** only warns: `DISPLAY` is probed with `xdpyinfo`, and the GL/EGL vendor is reported when a probe is installed. An inaccessible display is what made `iokitd` restart in a loop before [#140](https://github.com/lucannez64/darling-lol/issues/140); export `XAUTHORITY` from servers and SSH sessions. A build without a display is still valid. + +Run the same checks by hand, from the checkout: + +```sh +python3 scripts/preflight.py +python3 scripts/preflight.py --json +``` + +If a required check is stricter than your setup, `RCLOL_SKIP_PREFLIGHT=1 ./scripts/build.sh` (and the same for `scripts/install-runtime.sh`) skips the gate for that run. Include the preflight output when reporting a setup failure, and keep `--json` output when a tool needs to parse it. + ## Memory installer: permission denied An earlier sudo server installation can create root-owned `build/backups`. The old memory installer tried to create `build/backups/memory-runtime` there and failed at `backup_folder.mkdir(...)`. The corrected default is a separate user-owned `build/memory-runtime-backups` directory: diff --git a/scripts/build.sh b/scripts/build.sh index 6bb9a13..97422be 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -2,6 +2,9 @@ set -eu repo=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd) cd "$repo" +# Refuse before the long fetch and build when the host cannot build or run the result. +# Runs before configure.py so an old interpreter gets this message, not a traceback. +[ "${RCLOL_SKIP_PREFLIGHT:-0}" = 1 ] || python3 scripts/preflight.py python3 scripts/configure.py if [ -d build/venv/bin ]; then PATH="$repo/build/venv/bin:$PATH"; export PATH; fi python3 scripts/prepare.py diff --git a/scripts/install-runtime.sh b/scripts/install-runtime.sh index 9628760..b9099b9 100755 --- a/scripts/install-runtime.sh +++ b/scripts/install-runtime.sh @@ -13,6 +13,8 @@ fi python3 scripts/configure.py . build/runtime/tools/runtime-paths.sh if [ -d build/venv/bin ]; then PATH="$repo/build/venv/bin:$PATH"; export PATH; fi +# Check versions before any runtime helper touches the prefix or the running session. +[ "${RCLOL_SKIP_PREFLIGHT:-0}" = 1 ] || python3 scripts/preflight.py # A server killed by the base install leaves a stale .init.pid that makes darling shell fail. python3 build/runtime/tools/ensure-darling-runtime.py # Initialize the normal user-owned prefix before installing overrides. diff --git a/scripts/preflight.py b/scripts/preflight.py new file mode 100644 index 0000000..bd9bc4b --- /dev/null +++ b/scripts/preflight.py @@ -0,0 +1,306 @@ +#!/usr/bin/env python3 +"""Check host versions the build and the installed runtime need, before the expensive steps. + +Required checks exit 1 with one actionable line each; optional checks only warn. +Set RCLOL_SKIP_PREFLIGHT=1 in the environment to bypass the gate. Nothing here +touches the prefix, the guest or the network. +""" +import argparse +import json +import os +import platform +import re +import shutil +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +import llvm_tools + +MIN_PYTHON = (3, 10) +MIN_LLVM = 16 # LLVMGetVersion, checked by Iridium at startup, appeared in LLVM 16. +TESTED_LLVM = 22 # The patch tracks LLVM 22/23. +MIN_CMAKE = (3, 20) # The bundled indium CMakeLists requires 3.20. +MIN_LFS = (2, 13) # Debian 11 ships 2.13, the oldest release the setup is tested with. +MIN_KERNEL = (5, 14) # MADV_POPULATE_WRITE. +MIN_VULKAN = (1, 3) # Indium rejects older devices and needs timeline semaphores. + +SOURCE = Path(os.environ.get('RCLOL_PREFLIGHT_SOURCE') or ROOT / 'build/darling-src') +POINTER = b'version https://git-lfs.github.com/spec/v1' +LIBRARY_SUFFIXES = ('.dylib', '.a', '.so', '.o') + + +def version(text): + """(major, minor) from the first x.y in text, or None.""" + match = re.search(r'(\d+)\.(\d+)', text or '') + return (int(match.group(1)), int(match.group(2))) if match else None + + +def dotted(value): + return '.'.join(str(part) for part in value) if value else 'unknown' + + +def output(command, timeout=30): + """stdout of a command, or None when it is missing, fails or times out.""" + try: + result = subprocess.run(command, capture_output=True, text=True, errors='replace', + timeout=timeout) + except (OSError, subprocess.SubprocessError): + return None + return result.stdout if result.returncode == 0 else None + + +def check_python(report): + found = platform.python_version() + report('python', sys.version_info[:2] >= MIN_PYTHON, found, + f'Python {dotted(MIN_PYTHON)} or newer is required; older releases lack standard ' + 'library calls the helpers use. Ubuntu 22.04 ships 3.10, which is enough.') + + +def check_cmake(report): + text = output(['cmake', '--version']) + found = version(text.splitlines()[0]) if text else None + report('cmake', found is not None and found >= MIN_CMAKE, dotted(found), + f'CMake {dotted(MIN_CMAKE)} or newer is required by the configured build.') + + +def check_git_lfs(report): + text = output(['git', 'lfs', 'version']) + found = version(text) + report('git-lfs', found is not None and found >= MIN_LFS, dotted(found), + f'Git LFS {dotted(MIN_LFS)} or newer is required: the pinned submodules ship ' + 'libraries as LFS objects and pointer text crashes the guest.') + + +def lfs_pointers(source): + """Small library-like files under source that are still Git LFS pointer text.""" + found = [] + for folder, directories, files in os.walk(source): + directories[:] = [name for name in directories if name not in ('.git', '__pycache__')] + if Path(folder) == source: + directories[:] = [name for name in directories if name != 'build'] + for name in files: + if not name.endswith(LIBRARY_SUFFIXES): + continue + path = Path(folder) / name + # A pointer is ~130 bytes. Skip everything else before opening it, so a + # prepared checkout adds about a second instead of opening every file. + try: + if path.is_symlink() or path.stat().st_size > 1024: + continue + with path.open('rb') as handle: + if handle.read(len(POINTER)) == POINTER: + found.append(path) + except OSError: + continue + return found + + +def check_lfs_pointers(report): + if not SOURCE.is_dir(): + # The first build runs this before prepare.py fetches the pinned sources. + report('lfs-pointer', True, 'no prepared checkout yet') + return + pointers = lfs_pointers(SOURCE) + shown = ', '.join(str(path.relative_to(SOURCE)) for path in pointers[:3]) + if len(pointers) > 3: + shown += f' (+{len(pointers) - 3} more)' + report('lfs-pointer', not pointers, + f'{len(pointers)} library file(s) still contain pointer text: {shown}' + if pointers else 'no pointer text in the prepared libraries', + 'Git LFS did not fetch these objects. Install git-lfs and re-run ' + './scripts/build.sh (or git lfs pull in that checkout); a pointer file links but ' + 'crashes the guest at run time.') + + +def check_kernel(report): + release = os.environ.get('RCLOL_PREFLIGHT_KERNEL_RELEASE') or platform.release() + found = version(release) + report('kernel', found is not None and found >= MIN_KERNEL, release, + f'Linux {dotted(MIN_KERNEL)} or newer is required for MADV_POPULATE_WRITE, which ' + "Darwin's allocator uses when it reuses pages.") + + +def check_llvm(report): + configuration = llvm_tools.find('llvm-config') + text = output([configuration, '--version']) if configuration else None + found = version(text.splitlines()[0]) if text else None + major = found[0] if found else None + report('llvm', major is not None and major >= MIN_LLVM, + f'{dotted(found)} ({configuration or "llvm-config not found"})', + f'LLVM {MIN_LLVM} or newer is required: Iridium calls LLVMGetVersion at startup and ' + 'LLVM 15 does not export it. LLVM 22 or 23 is what the patches are tested with. ' + 'Install the development package (for example llvm-dev or llvm23-dev).') + if major is not None and MIN_LLVM <= major < TESTED_LLVM: + report('llvm-tested', False, dotted(found), + f'LLVM {major} builds, but the patch tracks LLVM {TESTED_LLVM}/23 and older ' + 'majors are untested; use 22 or 23 to be safe.', required=False) + + +def check_llvm_tools(report): + tools = {name: llvm_tools.find(name) for name in ('llvm-objdump', 'llvm-nm')} + missing = [name for name, path in tools.items() if not path] + present = ', '.join(f'{name}={path}' for name, path in tools.items() if path) + report('llvm-tools', not missing, present or 'none found', + f'Install the LLVM tools package (for example llvm-dev or llvm23-tools): {", ".join(missing)} ' + 'not found as an unversioned name, as llvm-- or in `llvm-config --bindir`. ' + 'The runtime install validates Darwin binaries with them.') + + +def gpu_blocks(text): + """{index: [lines]} for each `GPUn:` block in vulkaninfo output.""" + blocks = {} + current = None + for line in (text or '').splitlines(): + header = re.match(r'GPU(\d+):\s*$', line) + if header: + current = blocks.setdefault(int(header.group(1)), []) + continue + if current is not None: + current.append(line) + return blocks + + +def first_match(pattern, lines): + for line in lines: + found = re.search(pattern, line) + if found: + return found.group(1).strip() + return None + + +def vulkan_devices(): + """One record per Vulkan device, or None when vulkaninfo does not run. + + A device is kept as a whole record so a host with an old fast GPU and a new + GPU without timeline semaphores cannot pass by combining the two. The full + dump carries the features; `--summary` carries the versions that every + version of the tool prints. + """ + summary = output(['vulkaninfo', '--summary']) + if summary is None: + return None + features = gpu_blocks(output(['vulkaninfo'], timeout=60)) + devices = [] + for index, lines in sorted(gpu_blocks(summary).items()): + found = features.get(index, []) + values = [value == 'true' + for value in re.findall(r'timelineSemaphore\s*=\s*(true|false)', + '\n'.join(found))] + devices.append({'name': first_match(r'deviceName\s*=\s*(.+)', lines), + 'api': version(first_match(r'apiVersion\s*=\s*(\d+\.\d+)', lines)), + 'timeline': any(values) if values else None}) + return devices + + +def describe_device(device): + text = f'{device["name"] or "unnamed device"} {dotted(device["api"])}' + if device['timeline'] is not None: + text += ', timelineSemaphore ' + ('true' if device['timeline'] else 'false') + return text + + +def check_vulkan(report): + devices = vulkan_devices() + if devices is None and shutil.which('vulkaninfo'): + report('vulkan', False, 'vulkaninfo did not finish', + 'vulkaninfo is installed but exited with an error or timed out. Run ' + '`vulkaninfo --summary` for the loader or ICD error, then fix or reinstall the ' + 'driver: the runtime needs Vulkan 1.3 and timeline semaphores.') + return + if devices is None: + report('vulkan', False, 'vulkaninfo not available', + 'Install vulkan-tools to check the device. The runtime needs Vulkan 1.3 and ' + 'timeline semaphores; without a suitable device the game cannot render.', + required=False) + return + if not devices: + report('vulkan', False, 'no devices reported', + '`vulkaninfo --summary` listed no Vulkan device. Check the loader and the ICD ' + 'files, or reinstall the driver: the game cannot render without a device.') + return + # A dump that carries no feature section still gets the version check, as before. + checked = any(device['timeline'] is not None for device in devices) + usable = [device for device in devices + if device['api'] is not None and device['api'] >= MIN_VULKAN + and (device['timeline'] or not checked)] + report('vulkan', bool(usable), '; '.join(describe_device(device) for device in devices), + f'Vulkan {dotted(MIN_VULKAN)} or newer and timeline semaphores on the same device are ' + 'required: Indium rejects older devices and uses the semaphores. Update Mesa or the ' + 'vendor driver, or check `vulkaninfo`.') + + +def check_display(report): + display = os.environ.get('DISPLAY') + if not display: + report('x11', False, 'DISPLAY unset', + 'Without X11/Xwayland the runtime boots but the game cannot present; expected on ' + 'a headless host that is only used to build.', required=False) + return + if not shutil.which('xdpyinfo'): + report('x11', False, f'DISPLAY={display}, xdpyinfo not installed', + 'Install x11-utils to check the display before a launch.', required=False) + return + if output(['xdpyinfo'], timeout=15) is None: + report('x11', False, f'DISPLAY={display}', + f'Cannot open {display}. On servers, SSH and sudo sessions export XAUTHORITY ' + 'with a cookie the user running Darling can read; otherwise iokitd runs without ' + 'X11/EGL and the game cannot render. Builds still work without it.', + required=False) + return + vendor = '' + for probe in (['glxinfo', '-B'], ['eglinfo']): + text = output(probe, timeout=20) + if text: + found = re.search(r'(?:OpenGL vendor string|EGL_VENDOR):\s*(.+)', text) + if found: + vendor = f', {probe[0]} vendor {found.group(1).strip()}' + break + report('x11', True, f'DISPLAY={display} (xdpyinfo){vendor}') + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--json', action='store_true', help='machine-readable output') + args = parser.parse_args() + + checks = [] + + def report(name, ok, detail, hint=None, required=True): + checks.append({'check': name, 'ok': bool(ok), 'required': required, + 'detail': detail, 'hint': None if ok else hint}) + + for check in (check_python, check_cmake, check_git_lfs, check_lfs_pointers, check_kernel, + check_llvm, check_llvm_tools, check_vulkan, check_display): + check(report) + + failed = [check for check in checks if check['required'] and not check['ok']] + warned = [check for check in checks if not check['required'] and not check['ok']] + passed = [check for check in checks if check['ok']] + if args.json: + print(json.dumps({'ok': not failed, 'checks': checks}, indent=2)) + else: + print('preflight: host checks the build and runtime need') + for check in checks: + state = 'ok ' if check['ok'] else ('FAIL' if check['required'] else 'warn') + print(f' {state} {check["check"]:<13} {check["detail"]}') + if check['hint'] and not check['ok']: + print(f' {check["hint"]}') + summary = f'preflight: {len(passed)} checks passed' + if warned: + summary += f', {len(warned)} warning' + ('s' if len(warned) != 1 else '') + if failed: + summary += (f', {len(failed)} required check' + + ('s' if len(failed) != 1 else '') + ' failed') + print(summary) + if failed: + if not args.json: + print('preflight: fix the required checks above, or set RCLOL_SKIP_PREFLIGHT=1 to bypass', + file=sys.stderr) + raise SystemExit(1) + + +if __name__ == '__main__': + main() diff --git a/tests/test_preflight.py b/tests/test_preflight.py new file mode 100644 index 0000000..b755e4e --- /dev/null +++ b/tests/test_preflight.py @@ -0,0 +1,377 @@ +"""Preflight and LLVM tool discovery, with fake host tools and no Darling runtime.""" +import hashlib +import importlib.util +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[1] +PREFLIGHT = ROOT / 'scripts/preflight.py' +KERNEL = '6.1.0-test' + + +def load(name, path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class PreflightFixture(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory(prefix='preflight fixture ', dir=os.environ.get('TMPDIR')) + self.addCleanup(self.tmp.cleanup) + self.bin = Path(self.tmp.name) / 'bin' + self.bin.mkdir() + self.llvm_bindir = Path(self.tmp.name) / 'llvm bin' + self.llvm_bindir.mkdir() + self.good() + + def good(self): + self.stub('llvm-config', '''case "$1" in + --version) echo 23.1.5;; + --bindir) printf '%s\\n' "LLVM_BINDIR";; +esac'''.replace('LLVM_BINDIR', str(self.llvm_bindir))) + self.stub('llvm-objdump', 'exit 0') + self.stub('llvm-nm', 'exit 0') + self.stub('cmake', 'echo "cmake version 3.31.6"') + self.stub('git', 'case "$*" in "lfs version") echo "git-lfs/3.7.0 (GitHub; linux amd64)";; esac') + self.vulkan([('1.3.289', 'Test GPU', True)]) + self.stub('xdpyinfo', 'exit 0') + + def vulkan(self, devices): + """Stub vulkaninfo with [(apiVersion, name, timeline)] per device.""" + summary = ['Devices:', '========'] + features = ['Device Properties and Extensions:', '================================='] + for index, (api, name, timeline) in enumerate(devices): + summary += [f'GPU{index}:', f'\tapiVersion = {api}', f'\tdeviceName = {name}'] + features += [f'GPU{index}:', 'VkPhysicalDeviceProperties:', + f'\tapiVersion = {api}', f'\tdeviceName = {name}'] + if timeline is not None: + features += ['VkPhysicalDeviceVulkan12Features:', + f'\ttimelineSemaphore = {"true" if timeline else "false"}'] + self.stub('vulkaninfo', 'case "$*" in\n' + " --summary) printf '%s\\n' " + ' '.join(f"'{line}'" for line in summary) + ';;\n' + " *) printf '%s\\n' " + ' '.join(f"'{line}'" for line in features) + ';;\nesac') + + def stub(self, name, body, folder=None): + path = (folder or self.bin) / name + path.write_text('#!/bin/sh\nset -eu\n' + body + '\n') + path.chmod(0o755) + return path + + def preflight(self, *arguments, kernel=KERNEL, display=':0'): + env = dict(os.environ, PATH=str(self.bin), + RCLOL_PREFLIGHT_KERNEL_RELEASE=kernel, + RCLOL_PREFLIGHT_SOURCE=str(Path(self.tmp.name) / 'not fetched')) + if display is None: + env.pop('DISPLAY', None) + else: + env['DISPLAY'] = display + return subprocess.run([sys.executable, str(PREFLIGHT), *arguments], env=env, + text=True, capture_output=True, timeout=120) + + def test_all_required_checks_pass(self): + result = self.preflight() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('preflight: ', result.stdout) + self.assertNotIn('FAIL', result.stdout) + self.assertIn('ok llvm-tools', result.stdout) + + + def test_llvm_15_stops_with_the_reason(self): + self.stub('llvm-config', 'echo 15.0.7') + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL llvm', result.stdout) + self.assertIn('LLVM 16 or newer is required', result.stdout) + self.assertIn('LLVMGetVersion', result.stdout) + self.assertIn('RCLOL_SKIP_PREFLIGHT=1', result.stderr) + + + def test_untested_llvm_major_warns_but_builds(self): + self.stub('llvm-config', 'case "$1" in --version) echo 17.0.6;; esac') + result = self.preflight() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('warn llvm-tested', result.stdout) + self.assertIn('1 warning', result.stdout) + + + def test_versioned_llvm_tools_are_found(self): + (self.bin / 'llvm-objdump').unlink() + (self.bin / 'llvm-nm').unlink() + self.stub('llvm-objdump-15', 'exit 0') + self.stub('llvm-nm-15', 'exit 0') + result = self.preflight() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('llvm-objdump-15', result.stdout) + self.assertIn('llvm-nm-15', result.stdout) + + + def test_llvm_tools_come_from_llvm_config_bindir(self): + (self.bin / 'llvm-objdump').unlink() + (self.bin / 'llvm-nm').unlink() + self.stub('llvm-objdump-23', 'exit 0', folder=self.llvm_bindir) + self.stub('llvm-nm-23', 'exit 0', folder=self.llvm_bindir) + result = self.preflight() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('llvm bin', result.stdout) + + + def test_missing_llvm_tool_names_what_to_install(self): + (self.bin / 'llvm-nm').unlink() + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL llvm-tools', result.stdout) + self.assertIn('llvm-nm not found', result.stdout) + self.assertIn('llvm23-tools', result.stdout) + + + def test_missing_llvm_config_names_the_package(self): + (self.bin / 'llvm-config').unlink() + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL llvm', result.stdout) + self.assertIn('llvm-config not found', result.stdout) + self.assertIn('llvm-dev', result.stdout) + + + def test_old_cmake_stops_the_setup(self): + self.stub('cmake', 'echo "cmake version 3.16.3"') + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL cmake', result.stdout) + self.assertIn('CMake 3.20', result.stdout) + + + def test_old_git_lfs_stops_the_setup(self): + self.stub('git', 'case "$*" in "lfs version") echo "git-lfs/2.9.2 (GitHub; linux amd64)";; esac') + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL git-lfs', result.stdout) + self.assertIn('Git LFS 2.13', result.stdout) + + + def test_old_kernel_is_required(self): + result = self.preflight(kernel='5.10.0-28-amd64') + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL kernel', result.stdout) + self.assertIn('Linux 5.14', result.stdout) + + + def test_old_vulkan_stops_the_setup(self): + self.vulkan([('1.2.170', 'Old GPU', True)]) + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL vulkan', result.stdout) + self.assertIn('Vulkan 1.3 or newer', result.stdout) + self.assertIn('Old GPU 1.2', result.stdout) + + + def test_missing_vulkaninfo_only_warns(self): + (self.bin / 'vulkaninfo').unlink() + result = self.preflight() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('warn vulkan', result.stdout) + self.assertIn('vulkaninfo not available', result.stdout) + self.assertIn('Install vulkan-tools', result.stdout) + + + def test_broken_vulkaninfo_is_required(self): + self.stub('vulkaninfo', 'exit 1') + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL vulkan', result.stdout) + self.assertIn('vulkaninfo did not finish', result.stdout) + self.assertIn('timed out', result.stdout) + + + def test_missing_timeline_semaphores_is_reported(self): + self.vulkan([('1.3.289', 'Test GPU', False)]) + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL vulkan', result.stdout) + self.assertIn('timelineSemaphore false', result.stdout) + self.assertIn('on the same device', result.stdout) + + + def test_a_second_device_can_satisfy_both_requirements(self): + self.vulkan([('1.4.309', 'New GPU without semaphores', False), + ('1.4.309', 'Second GPU', True)]) + result = self.preflight() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('ok vulkan', result.stdout) + + + def test_requirements_are_not_combined_across_devices(self): + self.vulkan([('1.2.170', 'Old GPU with semaphores', True), + ('1.4.309', 'New GPU without semaphores', False)]) + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL vulkan', result.stdout) + self.assertIn('Old GPU with semaphores 1.2', result.stdout) + self.assertIn('New GPU without semaphores 1.4', result.stdout) + + + def test_no_reported_device_is_required(self): + self.vulkan([]) + result = self.preflight() + self.assertEqual(result.returncode, 1) + self.assertIn('FAIL vulkan', result.stdout) + self.assertIn('no devices reported', result.stdout) + + + def test_unreachable_display_only_warns(self): + self.stub('xdpyinfo', 'exit 1') + result = self.preflight(display=':99') + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('warn x11', result.stdout) + self.assertIn('XAUTHORITY', result.stdout) + + + def test_headless_host_only_warns(self): + result = self.preflight(display=None) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('DISPLAY unset', result.stdout) + + + def test_json_report_names_every_check(self): + result = self.preflight('--json') + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + report = json.loads(result.stdout) + self.assertTrue(report['ok']) + names = {check['check'] for check in report['checks']} + self.assertTrue({'python', 'cmake', 'git-lfs', 'lfs-pointer', 'kernel', 'llvm', + 'llvm-tools', 'vulkan', 'x11'} <= names) + self.assertTrue(all('required' in check and 'detail' in check for check in report['checks'])) + + +class LfsPointerFixture(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory(prefix='lfs pointer ', dir=os.environ.get('TMPDIR')) + self.addCleanup(self.tmp.cleanup) + self.source = Path(self.tmp.name) / 'darling-src' + self.source.mkdir() + self.preflight = load('preflight_lfs', PREFLIGHT) + self.preflight.SOURCE = self.source + + def check(self): + reports = [] + self.preflight.check_lfs_pointers(lambda *arguments, **keywords: reports.append( + (arguments, keywords))) + self.assertEqual(len(reports), 1) + arguments, keywords = reports[0] + name, ok, detail = arguments[:3] + hint = arguments[3] if len(arguments) > 3 else keywords.get('hint') + return name, ok, detail, hint + + def write(self, name, content): + path = self.source / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content) + return path + + def test_pointer_text_in_a_library_is_reported(self): + self.write('lib/libswiftCore.dylib', + b'version https://git-lfs.github.com/spec/v1\noid sha256:abc\nsize 123\n') + self.write('lib/libreal.dylib', b'\xcf\xfa\xed\xfe' + b'\0' * 64) + name, ok, detail, hint = self.check() + self.assertEqual(name, 'lfs-pointer') + self.assertFalse(ok) + self.assertIn('lib/libswiftCore.dylib', detail) + self.assertNotIn('libreal', detail) + self.assertIn('git lfs pull', hint) + + def test_pointer_text_outside_a_library_is_ignored(self): + self.write('notes.dylib.txt', b'version https://git-lfs.github.com/spec/v1\n') + self.write('lib/libreal.dylib', b'not a pointer') + _, ok, detail, _ = self.check() + self.assertTrue(ok, detail) + + def test_missing_checkout_is_not_a_failure(self): + self.preflight.SOURCE = self.source / 'not fetched' + _, ok, detail, _ = self.check() + self.assertTrue(ok) + self.assertIn('no prepared checkout', detail) + + +class LlvmToolsFixture(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory(prefix='llvm tools ', dir=os.environ.get('TMPDIR')) + self.addCleanup(self.tmp.cleanup) + self.bin = Path(self.tmp.name) / 'bin' + self.bin.mkdir() + self.saved = os.environ.get('PATH') + os.environ['PATH'] = str(self.bin) + self.addCleanup(self.restore) + self.tools = load('llvm_tools', ROOT / 'tools/llvm_tools.py') + + def restore(self): + os.environ['PATH'] = self.saved or '' + + def stub(self, name, body='exit 0', folder=None): + path = (folder or self.bin) / name + path.write_text('#!/bin/sh\n' + body + '\n') + path.chmod(0o755) + return path + + def test_prefers_the_unversioned_name(self): + unversioned = self.stub('llvm-objdump') + self.stub('llvm-objdump-23') + self.assertEqual(self.tools.find('llvm-objdump'), str(unversioned)) + + + def test_prefers_the_highest_versioned_name(self): + self.stub('llvm-objdump-15') + newest = self.stub('llvm-objdump-23') + self.stub('llvm-objdump-9') + self.assertEqual(self.tools.find('llvm-objdump'), str(newest)) + + + def test_ignores_a_name_that_is_not_a_version(self): + self.stub('llvm-objdump-tools') + self.assertIsNone(self.tools.find('llvm-objdump')) + + + def test_falls_back_to_llvm_config_bindir(self): + libexec = Path(self.tmp.name) / 'libexec' + libexec.mkdir() + expected = self.stub('llvm-nm-22', folder=libexec) + self.stub('llvm-config', 'echo "%s"' % libexec) + self.assertEqual(self.tools.find('llvm-nm'), str(expected)) + + + def test_llvm_config_timeout_is_reported(self): + self.stub('llvm-config', 'exec /bin/sleep 5') + self.tools.TIMEOUT = 1 + with self.assertRaises(SystemExit) as raised: + self.tools.config('--version') + self.assertIn('did not finish within 1 seconds', str(raised.exception)) + self.assertIn('reinstall the LLVM development package', str(raised.exception)) + + + def test_llvm_config_timeout_does_not_hang_the_bindir_probe(self): + self.stub('llvm-config', 'exec /bin/sleep 5') + self.tools.TIMEOUT = 1 + self.assertIsNone(self.tools.find('llvm-objdump')) + + + def test_require_explains_what_to_install(self): + with self.assertRaises(SystemExit) as raised: + self.tools.require('llvm-objdump') + self.assertIn('llvm-objdump not found', str(raised.exception)) + self.assertIn('llvm-config --bindir', str(raised.exception)) + + +class FileDigestFixture(unittest.TestCase): + def test_sha256_matches_hashlib(self): + digest = load('file_digest', ROOT / 'tools/file_digest.py') + with tempfile.TemporaryDirectory(prefix='digest ', dir=os.environ.get('TMPDIR')) as folder: + path = Path(folder) / 'payload' + payload = b'darling' * 400_000 # More than one chunk. + path.write_bytes(payload) + self.assertEqual(digest.sha256(path), hashlib.sha256(payload).hexdigest()) diff --git a/tests/test_runtime_cleanup_scope.py b/tests/test_runtime_cleanup_scope.py index afc76f8..4f89351 100644 --- a/tests/test_runtime_cleanup_scope.py +++ b/tests/test_runtime_cleanup_scope.py @@ -13,6 +13,12 @@ SCRIPT = Path(__file__).resolve().parents[1] / 'tools/ensure-darling-runtime.py' class RuntimeCleanupScopeTests(unittest.TestCase): + def enterContext(self, manager): + """unittest.TestCase.enterContext() is Python 3.11+; Ubuntu 22.04 ships 3.10.""" + value = manager.__enter__() + self.addCleanup(manager.__exit__, None, None, None) + return value + def setUp(self): temporary = tempfile.TemporaryDirectory(prefix='cleanup-scope-') self.addCleanup(temporary.cleanup) diff --git a/tests/test_stop_darling_runtime.py b/tests/test_stop_darling_runtime.py index 78129b4..58b8f93 100644 --- a/tests/test_stop_darling_runtime.py +++ b/tests/test_stop_darling_runtime.py @@ -65,7 +65,11 @@ class StopRuntimeTests(unittest.TestCase): def __init__(self, *parts, **kwargs): if parts and str(parts[0]) == '/proc': parts = (proc, *parts[1:]) - super().__init__(*parts, **kwargs) + try: + super().__init__(*parts, **kwargs) + except TypeError: + # Python 3.10's PurePath.__init__ takes no arguments; 3.12 stores them. + super().__init__() def read_text(self, *args, **kwargs): if self.name == 'status' and owner.status_error is not None: diff --git a/tools/build-riot-air-check.py b/tools/build-riot-air-check.py index cd870f0..f897508 100644 --- a/tools/build-riot-air-check.py +++ b/tools/build-riot-air-check.py @@ -4,12 +4,15 @@ from pathlib import Path import shlex import subprocess import sys + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import llvm_tools root = Path(__file__).resolve().parent.parent if '--native' in sys.argv: indium = root / 'build/darling-src/src/external/metal/deps/indium' - llvm_include = subprocess.check_output(['llvm-config', '--includedir'], text=True).strip() - llvm_libdir = Path(subprocess.check_output(['llvm-config', '--libdir'], text=True).strip()) - llvm_names = shlex.split(subprocess.check_output(['llvm-config', '--link-shared', '--libnames'], text=True)) + llvm_include = llvm_tools.config('--includedir') + llvm_libdir = Path(llvm_tools.config('--libdir')) + llvm_names = shlex.split(llvm_tools.config('--link-shared', '--libnames')) if len(llvm_names) != 1: raise SystemExit('Iridium requires one shared LLVM library, got: ' + repr(llvm_names)) llvm_library = llvm_libdir / llvm_names[0] diff --git a/tools/build-riot-indium-runtime.py b/tools/build-riot-indium-runtime.py index a630cc5..6c5946d 100644 --- a/tools/build-riot-indium-runtime.py +++ b/tools/build-riot-indium-runtime.py @@ -5,6 +5,9 @@ import os import shlex import subprocess import sys + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import llvm_tools root = Path(__file__).resolve().parent.parent folder = root / 'build/darling-src/build/src/external/metal/deps/indium' # Keep the validated LLVM major locally when a system update replaces it. @@ -24,9 +27,9 @@ if (pinned_llvm / 'lib/libLLVM-22.so').exists() and os.environ.get('RIOT_IRIDIUM llvm_library = pinned_llvm / 'lib/libLLVM-22.so' else: # Do not combine cached CMake include paths with a newer llvm-config library. - llvm_include = Path(subprocess.check_output(['llvm-config', '--includedir'], text=True).strip()) - llvm_libdir = Path(subprocess.check_output(['llvm-config', '--libdir'], text=True).strip()) - llvm_names = shlex.split(subprocess.check_output(['llvm-config', '--link-shared', '--libnames'], text=True)) + llvm_include = Path(llvm_tools.config('--includedir')) + llvm_libdir = Path(llvm_tools.config('--libdir')) + llvm_names = shlex.split(llvm_tools.config('--link-shared', '--libnames')) if len(llvm_names) != 1: raise SystemExit('Iridium requires one shared LLVM library, got: ' + repr(llvm_names)) llvm_library = llvm_libdir / llvm_names[0] diff --git a/tools/check-darwin-tlv-calls.py b/tools/check-darwin-tlv-calls.py index 74cca28..68257bd 100644 --- a/tools/check-darwin-tlv-calls.py +++ b/tools/check-darwin-tlv-calls.py @@ -7,13 +7,17 @@ through it, leaving %rdi unrelated: the thunk then returns NULL and the first wr Usage: check-darwin-tlv-calls.py BINARY... ; exits 1 and lists every bad access it finds. """ import re +from pathlib import Path import subprocess import sys +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import llvm_tools + def thread_vars_ranges(path): """Address ranges of the __thread_vars sections, which hold the TLV descriptors.""" - output = subprocess.run(['llvm-objdump', '--macho', '--section-headers', path], + output = subprocess.run([llvm_tools.require('llvm-objdump'), '--macho', '--section-headers', path], capture_output=True, text=True, check=True).stdout ranges = [] for line in output.splitlines(): @@ -31,7 +35,7 @@ def bad_accesses(path): ranges = thread_vars_ranges(path) if not ranges: return [] - disassembly = subprocess.run(['llvm-objdump', '-d', '--no-show-raw-insn', path], + disassembly = subprocess.run([llvm_tools.require('llvm-objdump'), '-d', '--no-show-raw-insn', path], capture_output=True, text=True, check=True).stdout load = re.compile(r'(?:leaq|movq)\s+-?0x[0-9a-f]+\(%rip\), %(\w+)\s+#+ 0x([0-9a-f]+)') found, function, pending = [], '?', {} diff --git a/tools/file_digest.py b/tools/file_digest.py new file mode 100644 index 0000000..3fb2c10 --- /dev/null +++ b/tools/file_digest.py @@ -0,0 +1,15 @@ +"""SHA-256 of a file without hashlib.file_digest(). + +hashlib.file_digest() is Python 3.11+; Ubuntu 22.04 ships Python 3.10, where the +installers failed only after the build had finished (issue #137). This reads in +chunks, so the memory use does not depend on the file size. +""" +import hashlib + + +def sha256(path): + checksum = hashlib.sha256() + with open(path, 'rb') as source: + for chunk in iter(lambda: source.read(1 << 20), b''): + checksum.update(chunk) + return checksum.hexdigest() diff --git a/tools/install-riot-darling-server.py b/tools/install-riot-darling-server.py index acfa4f9..b20d9fc 100644 --- a/tools/install-riot-darling-server.py +++ b/tools/install-riot-darling-server.py @@ -1,7 +1,6 @@ #!/usr/bin/env python3 """Atomically install a built Darling server; never stop or start any process.""" import argparse -import hashlib import json import os from pathlib import Path @@ -9,12 +8,13 @@ import shutil import stat import tempfile +from file_digest import sha256 + ROOT = Path(__file__).resolve().parent.parent def digest(path): - with path.open("rb") as input: - return hashlib.file_digest(input, "sha256").hexdigest() + return sha256(path) def live_binaries(destination): diff --git a/tools/install-riot-foundation-runtime.py b/tools/install-riot-foundation-runtime.py index 2b5a2b1..49e7e27 100644 --- a/tools/install-riot-foundation-runtime.py +++ b/tools/install-riot-foundation-runtime.py @@ -9,7 +9,6 @@ tools/install-riot-memory-runtime.py (its default source is the rebuilt build/riot-libsystem_kernel.dylib); this installer does not touch it. """ import argparse -import hashlib import json import os from pathlib import Path @@ -18,14 +17,14 @@ import stat import tempfile from runtime_paths import runtime_paths +from file_digest import sha256 ROOT = Path(__file__).resolve().parents[1] DEFAULT_ARTIFACT = ROOT / "build/riot-Foundation" def digest(path): - with path.open("rb") as input: - return hashlib.file_digest(input, "sha256").hexdigest() + return sha256(path) def copy_framework(sdk_framework, framework): diff --git a/tools/install-riot-libs-runtime.py b/tools/install-riot-libs-runtime.py index 4a2a6b6..00f998e 100644 --- a/tools/install-riot-libs-runtime.py +++ b/tools/install-riot-libs-runtime.py @@ -15,7 +15,6 @@ and is never run against the real prefix by the build. Nothing is activated until the guest processes are restarted. """ import argparse -import hashlib import json import os from pathlib import Path @@ -24,6 +23,7 @@ import stat import tempfile from runtime_paths import validate_runtime_paths +from file_digest import sha256 ROOT = Path(__file__).resolve().parents[1] @@ -40,8 +40,7 @@ LIBRARIES = [ def digest(path): - with path.open("rb") as input: - return hashlib.file_digest(input, "sha256").hexdigest() + return sha256(path) def destinations(prefix, guest_home): diff --git a/tools/install-riot-memory-runtime.py b/tools/install-riot-memory-runtime.py index 55f3fb4..1cc6d4a 100644 --- a/tools/install-riot-memory-runtime.py +++ b/tools/install-riot-memory-runtime.py @@ -1,7 +1,6 @@ #!/usr/bin/env python3 """Install the tested reclamation library atomically, without restarting clients.""" import argparse -import hashlib import json import os from pathlib import Path @@ -11,6 +10,7 @@ import stat import tempfile from runtime_paths import runtime_paths +from file_digest import sha256 ROOT = Path(__file__).resolve().parents[1] # The optimized relink from tools/build-riot-kqueue-runtime.py is the library the @@ -19,8 +19,7 @@ DEFAULT_ARTIFACT = ROOT / "build/riot-libsystem_kernel.dylib" def digest(path): - with path.open("rb") as input: - return hashlib.file_digest(input, "sha256").hexdigest() + return sha256(path) def install(binary, destinations, backup_folder): diff --git a/tools/llvm_tools.py b/tools/llvm_tools.py new file mode 100644 index 0000000..2de0b0e --- /dev/null +++ b/tools/llvm_tools.py @@ -0,0 +1,105 @@ +"""Find LLVM command line tools, including versioned Debian/Ubuntu names. + +Ubuntu's llvm--dev installs llvm-objdump and llvm-nm in /usr/lib/llvm-/bin +without an unversioned name in PATH, so a bare ['llvm-objdump'] run fails on an +otherwise complete host (issue #138). Resolution order: an unversioned tool on +PATH, then the highest versioned name on PATH, then the directory llvm-config +reports. Nothing here runs a guest process. +""" +import os +import subprocess + +TIMEOUT = 30 # A probe that hangs must not stall the preflight or the build. + + +def _collect(directory, name, found): + """Add every `name` and `name-` executable in the directory to found.""" + try: + entries = os.listdir(directory) + except OSError: + return + for entry in entries: + suffix = entry[len(name) + 1:] + if entry == name: + version = 0 # An unversioned name always wins. + elif entry.startswith(name + '-') and suffix.isdigit(): + version = int(suffix) + else: + continue + path = os.path.join(directory or '.', entry) + if os.path.isfile(path) and os.access(path, os.X_OK): + found.setdefault(version, path) + + +def _executables(name): + """{version: path} for `name` and for every `name-` on PATH.""" + found = {} + for folder in os.environ.get('PATH', '').split(os.pathsep): + _collect(folder or '.', name, found) + return found + + +def _preferred(candidates): + return candidates[0 if 0 in candidates else max(candidates)] if candidates else None + + +def _run(command): + """Run a short probe; return (completed process, problem), one of them None.""" + try: + result = subprocess.run(command, capture_output=True, text=True, errors='replace', + timeout=TIMEOUT) + except subprocess.TimeoutExpired: + return None, f'did not finish within {TIMEOUT} seconds' + except OSError as error: + return None, f'could not be started: {error}' + return result, None + + +def _bindir(): + """`llvm-config --bindir` from the preferred llvm-config, or None.""" + configuration = _preferred(_executables('llvm-config')) + if not configuration: + return None + result, _ = _run([configuration, '--bindir']) + if result is None: + return None + directory = result.stdout.strip() + if result.returncode or not os.path.isdir(directory): + return None + return directory + + +def find(name): + """Path of an LLVM tool, or None when this host does not have it.""" + candidates = _executables(name) + if candidates: + return _preferred(candidates) + directory = _bindir() + if directory: + in_bindir = {} + _collect(directory, name, in_bindir) + if in_bindir: + return _preferred(in_bindir) + return None + + +def require(name): + """Path of an LLVM tool, or exit with what was tried and what to install.""" + path = find(name) + if path: + return path + raise SystemExit(f'{name} not found: looked for an unversioned {name}, a versioned ' + f'{name}- in PATH and `llvm-config --bindir`. Install the LLVM tools ' + f'package (for example llvm-dev, llvm15-tools or llvm23-tools).') + + +def config(*arguments): + """Run llvm-config with the arguments, or exit with an actionable message.""" + path = require('llvm-config') + result, problem = _run([path, *arguments]) + called = f'{path} {" ".join(arguments)}' + if problem: + raise SystemExit(f'{called} {problem}; reinstall the LLVM development package.') + if result.returncode: + raise SystemExit(f'{called} failed: {result.stderr.strip()}') + return result.stdout.strip() diff --git a/tools/riot-llvm-version-offline-check.py b/tools/riot-llvm-version-offline-check.py index 036670b..dbac414 100644 --- a/tools/riot-llvm-version-offline-check.py +++ b/tools/riot-llvm-version-offline-check.py @@ -3,11 +3,15 @@ from pathlib import Path import re import subprocess +import sys import tempfile +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import llvm_tools + root = Path(__file__).resolve().parent.parent indium = root / 'build/darling-src/src/external/metal/deps/indium' -include = Path(subprocess.check_output(['llvm-config', '--includedir'], text=True).strip()) +include = Path(llvm_tools.config('--includedir')) header = (include / 'llvm/Config/llvm-config.h').read_text() major = int(re.search(r'^#define LLVM_VERSION_MAJOR (\d+)', header, re.MULTILINE)[1]) with tempfile.TemporaryDirectory(prefix='iridium-llvm-version-') as directory: diff --git a/tools/riot_overlay_audit.py b/tools/riot_overlay_audit.py index 4761f0b..c518490 100644 --- a/tools/riot_overlay_audit.py +++ b/tools/riot_overlay_audit.py @@ -23,6 +23,9 @@ import subprocess import sys from pathlib import Path +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import llvm_tools + ROOT = Path(__file__).resolve().parent.parent GAME = (Path.home() / ".darling/Applications/League of Legends.app/Contents/LoL/Game" / "LeagueofLegends.app/Contents/MacOS/LeagueofLegends") @@ -44,7 +47,9 @@ MIN_FUNCTION_INSTRUCTIONS = 20 def tool(*arguments): - return subprocess.run(arguments, capture_output=True, text=True, errors="replace").stdout + """Run an LLVM tool, resolving versioned names from PATH or llvm-config.""" + return subprocess.run((llvm_tools.require(arguments[0]), *arguments[1:]), + capture_output=True, text=True, errors="replace").stdout def load_commands(path, requested): diff --git a/tools/riot_overlay_verify.py b/tools/riot_overlay_verify.py index a6fe45b..98dec51 100644 --- a/tools/riot_overlay_verify.py +++ b/tools/riot_overlay_verify.py @@ -16,6 +16,9 @@ import subprocess import sys from pathlib import Path +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import llvm_tools + ROOT = Path(__file__).resolve().parent.parent @@ -27,11 +30,16 @@ def load_audit(): return module -def headers(path): - return subprocess.run(["llvm-objdump", "--macho", "--private-headers", str(path)], +def run(*arguments): + """Run an LLVM tool, resolving versioned names from PATH or llvm-config.""" + return subprocess.run((llvm_tools.require(arguments[0]), *arguments[1:]), capture_output=True, text=True, errors="replace").stdout +def headers(path): + return run("llvm-objdump", "--macho", "--private-headers", str(path)) + + def identity(path): """Install name and the two versions from LC_ID_DYLIB.""" text = headers(path) @@ -43,8 +51,7 @@ def identity(path): def dependencies(path): - text = subprocess.run(["llvm-objdump", "--macho", "--dylibs-used", str(path)], - capture_output=True, text=True, errors="replace").stdout + text = run("llvm-objdump", "--macho", "--dylibs-used", str(path)) return sorted(re.findall(r"^\t(\S+) \(compatibility version", text, re.M)) @@ -70,8 +77,7 @@ def weak_definitions(path): """Symbols the library defines only weakly: the -O0 build emits every inline function out of line, an optimized build may drop the unused ones, and the ABI makes a weak definition optional.""" - text = subprocess.run(["llvm-nm", "-m", str(path)], capture_output=True, text=True, - errors="replace").stdout + text = run("llvm-nm", "-m", str(path)) weak = set() for line in text.splitlines(): if "weak external" in line: @@ -80,8 +86,7 @@ def weak_definitions(path): def symbols(path, flag): - text = subprocess.run(["llvm-nm", flag, str(path)], capture_output=True, text=True, - errors="replace").stdout + text = run("llvm-nm", flag, str(path)) found = set() for line in text.splitlines(): parts = line.split()