From c95b91d5d7055a08666c7c85036622297b8e9c0f Mon Sep 17 00:00:00 2001 From: fenleon Date: Tue, 13 Oct 2026 10:09:11 +0200 Subject: [PATCH] =?UTF-8?q?registration:=20derive=20the=20pear-AES=20IV=20?= =?UTF-8?q?from=20the=20blob=20slot=20(blobFieldMap^-1)=20=E2=80=94=20clos?= =?UTF-8?q?es=20the=205080;=20spec=20rev=2042?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/imessage-protocol-spec.md | 5 +-- .../imessage/registration/NacRequestBlob.kt | 12 +++++-- .../registration/NacRequestBlobTest.kt | 32 ++++++++++++++++--- 3 files changed, 40 insertions(+), 9 deletions(-) diff --git a/docs/imessage-protocol-spec.md b/docs/imessage-protocol-spec.md index ab6715a..372a9b4 100644 --- a/docs/imessage-protocol-spec.md +++ b/docs/imessage-protocol-spec.md @@ -1,6 +1,6 @@ # iMessage protocol spec (clean-room) -**Status:** draft — revision 41. The full revision history is at the bottom ([Revision history](#revision-history)); the sections below state the facts as of rev 31. +**Status:** draft — revision 42. The full revision history is at the bottom ([Revision history](#revision-history)); the sections below state the facts as of rev 31. **Firewall rule:** this document contains facts only — no code, no function/type names, no copied expression from any reference implementation. Wire-format field names are quoted as on-the-wire protocol data (observable in captures, capture-arbitrated), never as source expression. Facts derived from SSPL/AGPL sources are used facts-only; nothing from them is reproduced. Where the source-derived fact sets agree, the fact is stated once with both citations; where they disagree, the disagreement is stated and tagged `TODO(capture):` — running mitmproxy captures arbitrate. All `TODO(capture)` items are collected in §7. @@ -334,7 +334,8 @@ Ordered by value. IDs match the inline `TODO(capture):` markers above. - Rev 37 (2026-10-12, live-provisioning wire facts + the step-2 EM A/B): driving the Kotlin `ClearAnisetteProvider` against Apple's live endpoints for the first time (the session-26 EM A/B, NAC-NOTES session-26 addendum) surfaced two wire facts the offline scripted-HTTP tests could not catch: (1) the GSA lookup GET **404s without `X-Mme-Client-Info`**, and accepts only the angle-bracket form ` ` — a bare `com.apple.AuthKit/1 (com.apple.akd/1.0)` value 404s (the app's real value: ` `; imessage's `loginInfoFor`/`DEFAULT_LOGIN_INFO` already use the angle form); (2) the start/finish provisioning POST body keys are **case-sensitive**: `{header, request}` (lowercase — what the Kotlin sent) gets a **silent HTTP 200 with an EMPTY body**, `{Header, Request}` (as §1.5/C6 records) gets the real `Response.spim`/`ptm`/`tk` — fixed in `ClearAnisetteProvider` (Kotlin now sends the capitalized keys; 217 registration tests green; first live Kotlin provisioning run succeeded: metadata 400 B / mid 60 B). The vendored anisette client's fixed companion headers (rev-14 set) were confirmed against a live 2026-10-03 capture of the app's own lookup. Step-2 gate A/B (all via the identical transport): the app's genuine 10-08 blob replay → 5003/EM; our minted blob → 5003/EM; a FRESH blob minted by the app's own unmodified `ValidationCtx::new` the same hour (replay-state eliminated) → 5003/EM; with the full anisette header set on step 2 (± hardware headers, both blobs) → 5003/EM ×4 — **the anisette-on-step-2 hypothesis is falsified**. Residue: transport fingerprinting vs a server-side gate for everyone; the app's own-flow verdict is blocked on interactive IDS re-registration (user Apple-ID login) — removing `anisette_test/state.plist` re-fires provisioning but never the mint (no `ValidationCtx::new` hit in a fully-trapped run), and the vendored anisette client no longer traverses an intercepting proxy (baked DER root, SSL_CERT_FILE ignored). - Rev 38 (2026-10-12 late, the §1.1 step-2 5003/EM gate CLOSED — it is a TLS-stack fingerprint, not a blob/server-gate problem): after the user's 2FA re-login re-fired the app's own IDS registration and the full mint chain was observed ACCEPTED live (`vctx_new → step2_serialize → key_establishment → vctx_sign` ×3, gdb traps), fresh genuine blobs minted per-test (gdb `ValidationCtx::new`) settled the transport matrix: reqwest 0.11 + rustls 0.21 (+ the vendored client's baked Apple root DER, extracted from the .so at 0x24afa9 — webpki- roots/Mozilla does not include this Apple private root) POSTing the identical XML-plist envelope gets a real `session-info` response; curl (OpenSSL 3) over BOTH HTTP/2 and HTTP/1.1, and curl with UA/Accept headers removed (Accept-Language only), gets 5003/EM on the same fresh blobs. HTTP version and request headers are irrelevant: the VEN endpoint (Akamai-fronted identity.ess.apple.com) fingerprints the TLS ClientHello and rejects OpenSSL-shaped handshakes with the app-layer error 5003/EM. Earlier probes (JDK HttpURLConnection, python urllib, both cert-verif-off) were rejected for the same reason. Kotlin `NacRequestBlob` and the §1.1 framing are fully vindicated — the blob and envelope were never wrong. Consequence: the imessage tool's §1.1 HTTP must present a rustls-shaped handshake — JVM/Android default stacks (conscrypt/BoringSSL, OpenSSL-shaped) are expected to be rejected (unverified on-device); Conscrypt (Android's default stack, BoringSSL) is ALSO rejected — verified with a plain `HttpsURLConnection` probe on the lightos AVD posting a fresh genuine blob (EM-REJECTED) — so the accepted shape so far is rustls specifically, and the platform TLS stacks cannot do §1.1 step 2 at all. Implementation: a rust helper (rustls) behind UniFFI in `:server` (the calls/ pattern); a fingerprint-shaped JVM stack is not viable given the BoringSSL result. Evidence + probe sources: `registration/dev/clearadi/captures/c56j/tls-ab/` (NAC-NOTES session-26 EM-gate closure). - Rev 35 (2026-10-12, C56-a CLOSED — direct in-process replay of ValidationCtx::new, two byte-verified runs): the request blob's full recipe — header ‖ OAEP(pubkey, pearKey16 ‖ hmacKey20) ‖ RNG16 ‖ u32be(32) ‖ field32 ‖ HMAC-SHA1(hmacKey20, field32), with field32 = mapA[encrypt_cbc(mapB[02 ‖ u32be(1) ‖ innerKey16 ‖ zeros], iv, init_key(pearKey16))] and mapA/mapB the 0x2249f9/0x224af9 byte-substitution tables (table[byte], not xor). Kotlin `NacRequestBlob`; imessage unstubbed at C56-a; live-mint gate = C56-c only. -- Rev 41 (2026-10-13, live REGISTER attempt #1-2 — the §1.1 step-2 rejection is ISOLATED to the 16-B blob slot [266:282]): the real export-code flow ran end-to-end for the first time (export-code parse → the hardware-configuration serialization **byte-identical to the recorded capture** — the identifier grammar is verified with a real machine); the Apple-ID login chain completed live on-device including the second-factor exchange and the §1.5 re-login; then step-2 answered the error verdict **`EM status=5080`** through the rustls transport (distinct from the TLS-gate 5003 — the application layer is reached). Host A/B through the previously-accepted probe client: the genuine blobs remain accepted today, the freshly built blob is rejected, and a **field-by-field reconstruction of a genuine blob from its recorded randomness is byte-exact in every deterministic region and ACCEPTED with our own RSA-OAEP block** — exonerating the OAEP encoding, the whole derivation chain (map→white-box-AES→map, HMAC), and the transport. Isolation: replacing ANY one of the five random inputs with fresh bytes (including the `[266:282]` slot) while keeping the rest genuine → 5080. Conclusion: **the rev-35 reading of the 20-B `[266:286]` slot head as "a third fresh random 16" is falsified — the slot is a consistency value derived from the other four inputs** (the server checks it; its bytes are visible materializing between the white-box-AES step and the serializer in the session-25 replay dumps — the same rewritten-buffer window left untrapped in rev 34). C58/session250: CLOSED with real data; remaining: the slot derivation (trap the rewrite window 0x17945cc-0x1794651 / the `[rsp+0x260]` source at 0x179494c), then the register completes. Probe sources + inputs archived (gitignored) in `registration/dev/clearadi/captures/c56k/`. +- Rev 42 (2026-10-13, C56-a fully CLOSED — the 5080 root cause pinned and fixed: **the pear-AES IV is derived from the wire slot, never drawn**): the slot16 at `[266:282]` IS a fresh RNG16 draw (rev 35 was right about that) — but the `encrypt_cbc` IV is **`blobFieldMap⁻¹(slot16)`**, the inverse of the same 0x2249f9 substitution table applied to field32's ciphertext. Pinned from the c56j replay dumps without a gdb round: the IV reaches `encrypt_cbc` (r8 @0x17944f8) through a `SpecFromIterNested::from_iter` Vec built over the `[rsp+0x260]` sample (entry-arg rdx = the closure context), and on the genuine data `ecbc_arg_r8 == tableA⁻¹[blob[266:282]]` byte-exactly (tableA = the mapA at 0x2249f9; `xortable.bin` in the c56j dumps is its first 64 B). The server recomputes the IV from the blob's slot — which explains every observed verdict: the rev-41 draw-isolation 5080s were the Kotlin builder drawing an INDEPENDENT fourth random (iv16) instead of deriving it, so any fresh slot or fresh iv broke the slot↔iv pairing the server checks; the rev-41 "consistency value derived from the four draws" reading is corrected (the slot is an input, not a derived value). Fix: `NacRequestBlob` draws THREE values (hmacKey20, innerKey16, slot16) and computes `iv16 = blobFieldMap⁻¹(slot16)`; new c56j genuine-draw vector test (deterministic regions byte-exact incl. field32 + HMAC). **Live proof: a fully fresh Kotlin blob (fresh pear/hmac/inner/slot, derived IV, our own Java OAEP) through the accepted rustls probe → real `session-info` (HTTP 200, 1263 B)** — probe log `captures/c56k/ab_step2_fresh_iv_2026-10-13.txt`. `:registration:test` green. +- Rev 41 (2026-10-13, live REGISTER attempt #1-2 — the §1.1 step-2 rejectionis ISOLATED to the 16-B blob slot [266:282]): the real export-code flow ran end-to-end for the first time (export-code parse → the hardware-configuration serialization **byte-identical to the recorded capture** — the identifier grammar is verified with a real machine); the Apple-ID login chain completed live on-device including the second-factor exchange and the §1.5 re-login; then step-2 answered the error verdict **`EM status=5080`** through the rustls transport (distinct from the TLS-gate 5003 — the application layer is reached). Host A/B through the previously-accepted probe client: the genuine blobs remain accepted today, the freshly built blob is rejected, and a **field-by-field reconstruction of a genuine blob from its recorded randomness is byte-exact in every deterministic region and ACCEPTED with our own RSA-OAEP block** — exonerating the OAEP encoding, the whole derivation chain (map→white-box-AES→map, HMAC), and the transport. Isolation: replacing ANY one of the five random inputs with fresh bytes (including the `[266:282]` slot) while keeping the rest genuine → 5080. Conclusion: **the rev-35 reading of the 20-B `[266:286]` slot head as "a third fresh random 16" is falsified — the slot is a consistency value derived from the other four inputs** (the server checks it; its bytes are visible materializing between the white-box-AES step and the serializer in the session-25 replay dumps — the same rewritten-buffer window left untrapped in rev 34). C58/session250: CLOSED with real data; remaining: the slot derivation (trap the rewrite window 0x17945cc-0x1794651 / the `[rsp+0x260]` source at 0x179494c), then the register completes. Probe sources + inputs archived (gitignored) in `registration/dev/clearadi/captures/c56k/`. - Rev 40 (2026-10-13, the rev-39 "divergence" CLOSED as a debugger artifact; the real -29004 root cause fixed — the OTP `two` input wiring — and the host init A/B GREEN): a direct uninstrumented oracle over the vendored white-box (dlopen + call the local derivation at its known address; no app launch) proves **the derivation is breakpoint-sensitive**: identical inputs yield a DIFFERENT signature under debugger traps (software and hardware alike — the trap hits, not code patching), while the intermediate working state stays byte-identical. The rev-39 trace's recorded sig was therefore the breakpointed value, not the client's real one. Re-derived ground truth: the Kotlin transcription matches the binary exactly on real machine inputs (sig `786f83af…` both sides) and on the recorded synthetic vectors; the rev-39 item (6) "final mixing rounds divergence" dissolves, and the stage-B working-state block write (LE [h3,h2,h1,h0] at 0x110, [h4..h7] at 0x120) is confirmed byte-identical at the post-schedule boundary under hardware breakpoints. (2) **The real -29004 root cause: the header factory fed the 60-B mid as the cipher's `two`**; the rev-39 item (3) mapping (`two` = the 32-B client secret) was correct but the wiring still used the mid — mid-as-`two` yields a well-formed OTP with a wrong sig, i.e. exactly ladder rung two. Fixed in the Kotlin header factory (`two` = client secret; the mid rides only the MD-M header). (3) **Host init A/B GREEN:** fresh live provisioning from the host JVM (2.7 s; 400-B metadata / 60-B mid / 32-B client secret; Mac flavor byte 1 → the `gen_otp` path), then the GSA SRP init POST with a dummy account **passed the environment check** — init returned salt/B and SRP completed; the server answered `ec=-20101` (credential rejection for the dummy account), anisette accepted, no -29004. (4) TLS fact: **gsa.apple.com now serves an all-Apple chain** (leaf ← Apple Server Authentication CA ← Apple Root CA, observed 2026-10-13) — the §1.2 Apple-root pinning applies to the GSA endpoints too, not only IDS/ess; the JVM default store fails the handshake. Oracle + probe + A/B method archived (gitignored) in `registration/dev/clearadi/captures/c58live/` (NOTES-true-sig.md). - Rev 39 (2026-10-13, first live GSA login A/B of the Kotlin chain + gdb OTP capture — the anisette OTP input mapping is CORRECTED and a Kotlin transcription divergence is found): driving the imessage tool's real Apple-ID login on the lightos AVD (user-gated) produced a clean error ladder that the host JVM reproduces exactly (throwaway harness, deleted; method + data in `registration/dev/clearadi/captures/c58live/`): (1) GSA `GsService2` POSTs **404 without the device header set** — `GsaLoginChain.headerConfig`/`twoFactorHeaders` must be wired (the rev-37 lookup-404 signature applies to the SRP POSTs too); (2) the provisioning-time machine identity must MATCH the login-time GSA header slots (provisioned `Macmini9,1` + login `MacBookPro18,3` → `ec=-22421`); (3) **the OTP cipher's `two` input is the 32-byte `client_secret`, NOT the mid** — the rev-27 "32-byte mid" reading is another mid/secret swap (gdb entry-arg capture of the working client's `gen_otp` call: rdi = gen = unix_seconds/30, rsi = the 400-B provisioning **metadata** as `one`, rdx = `client_secret` as `two`; two=mid[0:32] → -22421, two=client_secret → past the OTP check); (4) the Mac machine dispatches **`gen_otp` (0x17fba60)** — the OTP flavor byte 1 — confirming `FLAVOR_MAC = 1` (rev-27's "0 (Mac) → gen_otp_ios" dispatch numbering is inverted on the Mac path too); (5) **error ladder: -22421 = structurally invalid OTP; -29004 "environment mismatch" = well-formed OTP whose sig payload is wrong** (client-info flavor, `X-Apple-I-MD-LU` form, and hardware headers all exonerated by A/B); (6) **the Kotlin `gen_otp` transcription diverges from the binary on real inputs**: live ctx checkpoints (`captures/c58live/otp-live-trace.log`) are byte-identical to Kotlin through gensource → gentable → stage A (the only delta was ctx[0x110..0x120] = LE [h3,h2,h1,h0] — the fitted "written but never read, order [h3,h1,h2,h0]" comment was wrong; the block write is added to `genOtp` stage B, synthetic suite still green) — so the divergence sits in the FINAL MIXING ROUNDS (tail/sieve/6×9-round loop/w-block, binary 0x17fbd70–0x17fc4c3); ground truth for the round-state hunt: the live trace (gen, sig16 per launch, ctx at every stage boundary) + the vendored app's provisioned machine (`obstate.json`) as oracle inputs. The vendored oracle: its own machine state through the Kotlin OTP does NOT reproduce its accepted logins — the transcription (not the machine/provisioning/transport) is the residue. Kotlin: `ClearAnisetteProvider` flavor flip + `ClearAdiOtp` h-block fix; 217+ registration tests green. diff --git a/registration/src/main/kotlin/dev/fenn/imessage/registration/NacRequestBlob.kt b/registration/src/main/kotlin/dev/fenn/imessage/registration/NacRequestBlob.kt index 39f0c40..087b806 100644 --- a/registration/src/main/kotlin/dev/fenn/imessage/registration/NacRequestBlob.kt +++ b/registration/src/main/kotlin/dev/fenn/imessage/registration/NacRequestBlob.kt @@ -19,6 +19,11 @@ import javax.crypto.spec.SecretKeySpec * ‖ u32be(32) — the next field's length marker (4 B) * ‖ field32 = blobFieldMap[PearAes.encryptCbc(plain32, iv16, PearAes.initKey(pearKey16))] * (32 B) + * + * `iv16 = blobFieldMap⁻¹(slot16)` — the pear-AES IV is never drawn or carried; the server + * recomputes it from the wire slot through the inverse of the same substitution table + * (pinned 2026-10-13 on the c56j replay dumps: `ecbc_arg_r8 == blobFieldMap⁻¹[blob[266:282]]` + * exactly; an independent IV draw is the 5080/EM residue — spec §1.1 rev 42). * ‖ HMAC-SHA1(hmacKey20, field32) (20 B) * ``` * @@ -62,7 +67,11 @@ object NacRequestBlob { val hmacKey20 = draw(20) val innerKey16 = draw(16) val slot16 = draw(16) - val iv16 = draw(16) + val fieldMap = PearTables.blobFieldMap + val fieldInv = ByteArray(256).also { inv -> + for (i in 0 until 256) inv[fieldMap[i].toInt() and 0xff] = i.toByte() + } + val iv16 = ByteArray(16) { fieldInv[slot16[it].toInt() and 0xff] } val paesKey = PearAes.initKey(pearKey16) val plain32 = ByteArray(32) @@ -72,7 +81,6 @@ object NacRequestBlob { val payloadMap = PearTables.blobPayloadMap val mappedPlain = ByteArray(32) { i -> payloadMap[plain32[i].toInt() and 0xff] } val enc32 = PearAes.encryptCbc(mappedPlain, iv16, paesKey) - val fieldMap = PearTables.blobFieldMap val field32 = ByteArray(32) { i -> fieldMap[enc32[i].toInt() and 0xff] } val oaep = Cipher.getInstance(OAEP_TRANSFORMATION).apply { diff --git a/registration/src/test/kotlin/dev/fenn/imessage/registration/NacRequestBlobTest.kt b/registration/src/test/kotlin/dev/fenn/imessage/registration/NacRequestBlobTest.kt index 286ff40..4318d45 100644 --- a/registration/src/test/kotlin/dev/fenn/imessage/registration/NacRequestBlobTest.kt +++ b/registration/src/test/kotlin/dev/fenn/imessage/registration/NacRequestBlobTest.kt @@ -10,7 +10,7 @@ import kotlin.test.assertEquals import kotlin.test.assertFailsWith import kotlin.test.assertTrue -/** Synthetic-vector tests for the §1.1 step-2 request blob (C56-a; spec §1.1 rev 35). */ +/** Synthetic-vector tests for the §1.1 step-2 request blob (C56-a; spec §1.1 rev 42). */ class NacRequestBlobTest { private class FixedDraw(values: List) { @@ -40,7 +40,6 @@ class NacRequestBlobTest { ByteArray(20) { it.toByte() }, // hmacKey20 ByteArray(16) { (it + 1).toByte() }, // innerKey16 ByteArray(16) { (it + 2).toByte() }, // slot16 - ByteArray(16) { (it + 3).toByte() }, // iv16 ), ) val (blob, pearKey16) = blob(pair.public, draw) @@ -67,10 +66,13 @@ class NacRequestBlobTest { val hmacKey20 = ByteArray(20) { (0x40 + it).toByte() } val innerKey16 = ByteArray(16) { (0x50 + it).toByte() } val slot16 = ByteArray(16) { (0x60 + it).toByte() } - val iv16 = ByteArray(16) { (0x70 + it).toByte() } - val draw = FixedDraw(listOf(hmacKey20, innerKey16, slot16, iv16)) + val draw = FixedDraw(listOf(hmacKey20, innerKey16, slot16)) val (blob, pearKey16) = blob(keyPair().public, draw) + val fieldInv = ByteArray(256).also { inv -> + for (i in 0 until 256) inv[PearTables.blobFieldMap[i].toInt() and 0xff] = i.toByte() + } + val iv16 = ByteArray(16) { fieldInv[slot16[it].toInt() and 0xff] } val plain32 = ByteArray(32) plain32[0] = 0x02 plain32[4] = 0x01 @@ -81,9 +83,29 @@ class NacRequestBlobTest { assertContentEquals(field32, blob.copyOfRange(286, 318)) } + /** Genuine c56j replay vector: the three real draws must reproduce the blob's every + * deterministic region (OAEP is randomized by construction and checked by length only). */ + @Test + fun c56jReplayVector() { + fun hx(s: String) = ByteArray(s.length / 2) { ((Character.digit(s[it * 2], 16) shl 4) + Character.digit(s[it * 2 + 1], 16)).toByte() } + val pearKey16 = hx("80422dce4d634e44da4996c8ec871c5f") + val hmacKey20 = hx("ccbe18c8d3bca949ec5f5be2060ca8c61a2cc3f4") + val innerKey16 = hx("0238210e70ba93d3b0188c21ccc64d17") + val slot16 = hx("018c1c81233a8b9d3fb8b9ff94228649") + val field32 = hx("16f0af0b2ac8ac222dcfea95d6a7c9ddad8182f3b395502517854d6913d91fc5") + val hmac = hx("0f117d2462b730a53797b324fe61c1bfff899348") + val draw = FixedDraw(listOf(hmacKey20, innerKey16, slot16)) + val blob = NacRequestBlob.build(keyPair().public, pearKey16) { n -> draw.next(n) } + assertEquals(NacRequestBlob.BLOB_BYTES, blob.size) + assertContentEquals(NacRequestBlob.header, blob.copyOfRange(0, 10)) + assertContentEquals(slot16, blob.copyOfRange(266, 282)) + assertContentEquals(field32, blob.copyOfRange(286, 318)) + assertContentEquals(hmac, blob.copyOfRange(318, 338)) + } + @Test fun rejectsWrongPearKeySize() { - val draw = FixedDraw(listOf(ByteArray(20), ByteArray(16), ByteArray(16), ByteArray(16))) + val draw = FixedDraw(listOf(ByteArray(20), ByteArray(16), ByteArray(16))) assertFailsWith { NacRequestBlob.build(keyPair().public, ByteArray(15)) { n -> draw.next(n) } }