registration: derive the pear-AES IV from the blob slot (blobFieldMap^-1) — closes the 5080; spec rev 42

docs/imessage-protocol-spec.md

@@ -1,6 +1,6 @@
 # iMessage protocol spec (clean-room)
 
-**Status:** draft — revision 41. The full revision history is at the bottom ([Revision history](#revision-history)); the sections below state the facts as of rev 31.
+**Status:** draft — revision 42. The full revision history is at the bottom ([Revision history](#revision-history)); the sections below state the facts as of rev 31.
 
 **Firewall rule:** this document contains facts only — no code, no function/type names, no copied expression from any reference implementation. Wire-format field names are quoted as on-the-wire protocol data (observable in captures, capture-arbitrated), never as source expression. Facts derived from SSPL/AGPL sources are used facts-only; nothing from them is reproduced. Where the source-derived fact sets agree, the fact is stated once with both citations; where they disagree, the disagreement is stated and tagged `TODO(capture):` — running mitmproxy captures arbitrate. All `TODO(capture)` items are collected in §7.
 
@@ -334,7 +334,8 @@ Ordered by value. IDs match the inline `TODO(capture):` markers above.
 - Rev 37 (2026-10-12, live-provisioning wire facts + the step-2 EM A/B): driving the Kotlin `ClearAnisetteProvider` against Apple's live endpoints for the first time (the session-26 EM A/B, NAC-NOTES session-26 addendum) surfaced two wire facts the offline scripted-HTTP tests could not catch: (1) the GSA lookup GET **404s without `X-Mme-Client-Info`**, and accepts only the angle-bracket form `<Model> <macOS;ver;build> <com.apple.AuthKit/1 (…)>` — a bare `com.apple.AuthKit/1 (com.apple.akd/1.0)` value 404s (the app's real value: `<MacBookPro18,3> <macOS;26.6.2;25G83> <com.apple.AuthKit/1 (com.apple.MobileSMS/1262.500.151.1.2)>`; imessage's `loginInfoFor`/`DEFAULT_LOGIN_INFO` already use the angle form); (2) the start/finish provisioning POST body keys are **case-sensitive**: `{header, request}` (lowercase — what the Kotlin sent) gets a **silent HTTP 200 with an EMPTY body**, `{Header, Request}` (as §1.5/C6 records) gets the real `Response.spim`/`ptm`/`tk` — fixed in `ClearAnisetteProvider` (Kotlin now sends the capitalized keys; 217 registration tests green; first live Kotlin provisioning run succeeded: metadata 400 B / mid 60 B). The vendored anisette client's fixed companion headers (rev-14 set) were confirmed against a live 2026-10-03 capture of the app's own lookup. Step-2 gate A/B (all via the identical transport): the app's genuine 10-08 blob replay → 5003/EM; our minted blob → 5003/EM; a FRESH blob minted by the app's own unmodified `ValidationCtx::new` the same hour (replay-state eliminated) → 5003/EM; with the full anisette header set on step 2 (± hardware headers, both blobs) → 5003/EM ×4 — **the anisette-on-step-2 hypothesis is falsified**. Residue: transport fingerprinting vs a server-side gate for everyone; the app's own-flow verdict is blocked on interactive IDS re-registration (user Apple-ID login) — removing `anisette_test/state.plist` re-fires provisioning but never the mint (no `ValidationCtx::new` hit in a fully-trapped run), and the vendored anisette client no longer traverses an intercepting proxy (baked DER root, SSL_CERT_FILE ignored).
 - Rev 38 (2026-10-12 late, the §1.1 step-2 5003/EM gate CLOSED — it is a TLS-stack fingerprint, not a blob/server-gate problem): after the user's 2FA re-login re-fired the app's own IDS registration and the full mint chain was observed ACCEPTED live (`vctx_new → step2_serialize → key_establishment → vctx_sign` ×3, gdb traps), fresh genuine blobs minted per-test (gdb `ValidationCtx::new`) settled the transport matrix: reqwest 0.11 + rustls 0.21 (+ the vendored client's baked Apple root DER, extracted from the .so at 0x24afa9 — webpki- roots/Mozilla does not include this Apple private root) POSTing the identical XML-plist envelope gets a real `session-info` response; curl (OpenSSL 3) over BOTH HTTP/2 and HTTP/1.1, and curl with UA/Accept headers removed (Accept-Language only), gets 5003/EM on the same fresh blobs. HTTP version and request headers are irrelevant: the VEN endpoint (Akamai-fronted identity.ess.apple.com) fingerprints the TLS ClientHello and rejects OpenSSL-shaped handshakes with the app-layer error 5003/EM. Earlier probes (JDK HttpURLConnection, python urllib, both cert-verif-off) were rejected for the same reason. Kotlin `NacRequestBlob` and the §1.1 framing are fully vindicated — the blob and envelope were never wrong. Consequence: the imessage tool's §1.1 HTTP must present a rustls-shaped handshake — JVM/Android default stacks (conscrypt/BoringSSL, OpenSSL-shaped) are expected to be rejected (unverified on-device); Conscrypt (Android's default stack, BoringSSL) is ALSO rejected — verified with a plain `HttpsURLConnection` probe on the lightos AVD posting a fresh genuine blob (EM-REJECTED) — so the accepted shape so far is rustls specifically, and the platform TLS stacks cannot do §1.1 step 2 at all. Implementation: a rust helper (rustls) behind UniFFI in `:server` (the calls/ pattern); a fingerprint-shaped JVM stack is not viable given the BoringSSL result. Evidence + probe sources: `registration/dev/clearadi/captures/c56j/tls-ab/` (NAC-NOTES session-26 EM-gate closure).
 - Rev 35 (2026-10-12, C56-a CLOSED — direct in-process replay of ValidationCtx::new, two byte-verified runs): the request blob's full recipe — header ‖ OAEP(pubkey, pearKey16 ‖ hmacKey20) ‖ RNG16 ‖ u32be(32) ‖ field32 ‖ HMAC-SHA1(hmacKey20, field32), with field32 = mapA[encrypt_cbc(mapB[02 ‖ u32be(1) ‖ innerKey16 ‖ zeros], iv, init_key(pearKey16))] and mapA/mapB the 0x2249f9/0x224af9 byte-substitution tables (table[byte], not xor). Kotlin `NacRequestBlob`; imessage unstubbed at C56-a; live-mint gate = C56-c only.
-- Rev 41 (2026-10-13, live REGISTER attempt #1-2 — the §1.1 step-2 rejection is ISOLATED to the 16-B blob slot [266:282]): the real export-code flow ran end-to-end for the first time (export-code parse → the hardware-configuration serialization **byte-identical to the recorded capture** — the identifier grammar is verified with a real machine); the Apple-ID login chain completed live on-device including the second-factor exchange and the §1.5 re-login; then step-2 answered the error verdict **`EM status=5080`** through the rustls transport (distinct from the TLS-gate 5003 — the application layer is reached). Host A/B through the previously-accepted probe client: the genuine blobs remain accepted today, the freshly built blob is rejected, and a **field-by-field reconstruction of a genuine blob from its recorded randomness is byte-exact in every deterministic region and ACCEPTED with our own RSA-OAEP block** — exonerating the OAEP encoding, the whole derivation chain (map→white-box-AES→map, HMAC), and the transport. Isolation: replacing ANY one of the five random inputs with fresh bytes (including the `[266:282]` slot) while keeping the rest genuine → 5080. Conclusion: **the rev-35 reading of the 20-B `[266:286]` slot head as "a third fresh random 16" is falsified — the slot is a consistency value derived from the other four inputs** (the server checks it; its bytes are visible materializing between the white-box-AES step and the serializer in the session-25 replay dumps — the same rewritten-buffer window left untrapped in rev 34). C58/session250: CLOSED with real data; remaining: the slot derivation (trap the rewrite window 0x17945cc-0x1794651 / the `[rsp+0x260]` source at 0x179494c), then the register completes. Probe sources + inputs archived (gitignored) in `registration/dev/clearadi/captures/c56k/`.
+- Rev 42 (2026-10-13, C56-a fully CLOSED — the 5080 root cause pinned and fixed: **the pear-AES IV is derived from the wire slot, never drawn**): the slot16 at `[266:282]` IS a fresh RNG16 draw (rev 35 was right about that) — but the `encrypt_cbc` IV is **`blobFieldMap⁻¹(slot16)`**, the inverse of the same 0x2249f9 substitution table applied to field32's ciphertext. Pinned from the c56j replay dumps without a gdb round: the IV reaches `encrypt_cbc` (r8 @0x17944f8) through a `SpecFromIterNested::from_iter` Vec built over the `[rsp+0x260]` sample (entry-arg rdx = the closure context), and on the genuine data `ecbc_arg_r8 == tableA⁻¹[blob[266:282]]` byte-exactly (tableA = the mapA at 0x2249f9; `xortable.bin` in the c56j dumps is its first 64 B). The server recomputes the IV from the blob's slot — which explains every observed verdict: the rev-41 draw-isolation 5080s were the Kotlin builder drawing an INDEPENDENT fourth random (iv16) instead of deriving it, so any fresh slot or fresh iv broke the slot↔iv pairing the server checks; the rev-41 "consistency value derived from the four draws" reading is corrected (the slot is an input, not a derived value). Fix: `NacRequestBlob` draws THREE values (hmacKey20, innerKey16, slot16) and computes `iv16 = blobFieldMap⁻¹(slot16)`; new c56j genuine-draw vector test (deterministic regions byte-exact incl. field32 + HMAC). **Live proof: a fully fresh Kotlin blob (fresh pear/hmac/inner/slot, derived IV, our own Java OAEP) through the accepted rustls probe → real `session-info` (HTTP 200, 1263 B)** — probe log `captures/c56k/ab_step2_fresh_iv_2026-10-13.txt`. `:registration:test` green.
+- Rev 41 (2026-10-13, live REGISTER attempt #1-2 — the §1.1 step-2 rejectionis ISOLATED to the 16-B blob slot [266:282]): the real export-code flow ran end-to-end for the first time (export-code parse → the hardware-configuration serialization **byte-identical to the recorded capture** — the identifier grammar is verified with a real machine); the Apple-ID login chain completed live on-device including the second-factor exchange and the §1.5 re-login; then step-2 answered the error verdict **`EM status=5080`** through the rustls transport (distinct from the TLS-gate 5003 — the application layer is reached). Host A/B through the previously-accepted probe client: the genuine blobs remain accepted today, the freshly built blob is rejected, and a **field-by-field reconstruction of a genuine blob from its recorded randomness is byte-exact in every deterministic region and ACCEPTED with our own RSA-OAEP block** — exonerating the OAEP encoding, the whole derivation chain (map→white-box-AES→map, HMAC), and the transport. Isolation: replacing ANY one of the five random inputs with fresh bytes (including the `[266:282]` slot) while keeping the rest genuine → 5080. Conclusion: **the rev-35 reading of the 20-B `[266:286]` slot head as "a third fresh random 16" is falsified — the slot is a consistency value derived from the other four inputs** (the server checks it; its bytes are visible materializing between the white-box-AES step and the serializer in the session-25 replay dumps — the same rewritten-buffer window left untrapped in rev 34). C58/session250: CLOSED with real data; remaining: the slot derivation (trap the rewrite window 0x17945cc-0x1794651 / the `[rsp+0x260]` source at 0x179494c), then the register completes. Probe sources + inputs archived (gitignored) in `registration/dev/clearadi/captures/c56k/`.
 - Rev 40 (2026-10-13, the rev-39 "divergence" CLOSED as a debugger artifact; the real -29004 root cause fixed — the OTP `two` input wiring — and the host init A/B GREEN): a direct uninstrumented oracle over the vendored white-box (dlopen + call the local derivation at its known address; no app launch) proves **the derivation is breakpoint-sensitive**: identical inputs yield a DIFFERENT signature under debugger traps (software and hardware alike — the trap hits, not code patching), while the intermediate working state stays byte-identical. The rev-39 trace's recorded sig was therefore the breakpointed value, not the client's real one. Re-derived ground truth: the Kotlin transcription matches the binary exactly on real machine inputs (sig `786f83af…` both sides) and on the recorded synthetic vectors; the rev-39 item (6) "final mixing rounds divergence" dissolves, and the stage-B working-state block write (LE [h3,h2,h1,h0] at 0x110, [h4..h7] at 0x120) is confirmed byte-identical at the post-schedule boundary under hardware breakpoints. (2) **The real -29004 root cause: the header factory fed the 60-B mid as the cipher's `two`**; the rev-39 item (3) mapping (`two` = the 32-B client secret) was correct but the wiring still used the mid — mid-as-`two` yields a well-formed OTP with a wrong sig, i.e. exactly ladder rung two. Fixed in the Kotlin header factory (`two` = client secret; the mid rides only the MD-M header). (3) **Host init A/B GREEN:** fresh live provisioning from the host JVM (2.7 s; 400-B metadata / 60-B mid / 32-B client secret; Mac flavor byte 1 → the `gen_otp` path), then the GSA SRP init POST with a dummy account **passed the environment check** — init returned salt/B and SRP completed; the server answered `ec=-20101` (credential rejection for the dummy account), anisette accepted, no -29004. (4) TLS fact: **gsa.apple.com now serves an all-Apple chain** (leaf ← Apple Server Authentication CA ← Apple Root CA, observed 2026-10-13) — the §1.2 Apple-root pinning applies to the GSA endpoints too, not only IDS/ess; the JVM default store fails the handshake. Oracle + probe + A/B method archived (gitignored) in `registration/dev/clearadi/captures/c58live/` (NOTES-true-sig.md).
 - Rev 39 (2026-10-13, first live GSA login A/B of the Kotlin chain + gdb OTP capture — the anisette OTP input mapping is CORRECTED and a Kotlin transcription divergence is found): driving the imessage tool's real Apple-ID login on the lightos AVD (user-gated) produced a clean error ladder that the host JVM reproduces exactly (throwaway harness, deleted; method + data in `registration/dev/clearadi/captures/c58live/`): (1) GSA `GsService2` POSTs **404 without the device header set** — `GsaLoginChain.headerConfig`/`twoFactorHeaders` must be wired (the rev-37 lookup-404 signature applies to the SRP POSTs too); (2) the provisioning-time machine identity must MATCH the login-time GSA header slots (provisioned `Macmini9,1` + login `MacBookPro18,3` → `ec=-22421`); (3) **the OTP cipher's `two` input is the 32-byte `client_secret`, NOT the mid** — the rev-27 "32-byte mid" reading is another mid/secret swap (gdb entry-arg capture of the working client's `gen_otp` call: rdi = gen = unix_seconds/30, rsi = the 400-B provisioning **metadata** as `one`, rdx = `client_secret` as `two`; two=mid[0:32] → -22421, two=client_secret → past the OTP check); (4) the Mac machine dispatches **`gen_otp` (0x17fba60)** — the OTP flavor byte 1 — confirming `FLAVOR_MAC = 1` (rev-27's "0 (Mac) → gen_otp_ios" dispatch numbering is inverted on the Mac path too); (5) **error ladder: -22421 = structurally invalid OTP; -29004 "environment mismatch" = well-formed OTP whose sig payload is wrong** (client-info flavor, `X-Apple-I-MD-LU` form, and hardware headers all exonerated by A/B); (6) **the Kotlin `gen_otp` transcription diverges from the binary on real inputs**: live ctx checkpoints (`captures/c58live/otp-live-trace.log`) are byte-identical to Kotlin through gensource → gentable → stage A (the only delta was ctx[0x110..0x120] = LE [h3,h2,h1,h0] — the fitted "written but never read, order [h3,h1,h2,h0]" comment was wrong; the block write is added to `genOtp` stage B, synthetic suite still green) — so the divergence sits in the FINAL MIXING ROUNDS (tail/sieve/6×9-round loop/w-block, binary 0x17fbd70–0x17fc4c3); ground truth for the round-state hunt: the live trace (gen, sig16 per launch, ctx at every stage boundary) + the vendored app's provisioned machine (`obstate.json`) as oracle inputs. The vendored oracle: its own machine state through the Kotlin OTP does NOT reproduce its accepted logins — the transcription (not the machine/provisioning/transport) is the residue. Kotlin: `ClearAnisetteProvider` flavor flip + `ClearAdiOtp` h-block fix; 217+ registration tests green.
 

registration/src/main/kotlin/dev/fenn/imessage/registration/NacRequestBlob.kt

@@ -19,6 +19,11 @@ import javax.crypto.spec.SecretKeySpec
  *   ‖ u32be(32) — the next field's length marker                                (4 B)
  *   ‖ field32 = blobFieldMap[PearAes.encryptCbc(plain32, iv16, PearAes.initKey(pearKey16))]
  *                                                                               (32 B)
+ *
+ * `iv16 = blobFieldMap⁻¹(slot16)` — the pear-AES IV is never drawn or carried; the server
+ * recomputes it from the wire slot through the inverse of the same substitution table
+ * (pinned 2026-10-13 on the c56j replay dumps: `ecbc_arg_r8 == blobFieldMap⁻¹[blob[266:282]]`
+ * exactly; an independent IV draw is the 5080/EM residue — spec §1.1 rev 42).
  *   ‖ HMAC-SHA1(hmacKey20, field32)                                             (20 B)
  * ```
  *
@@ -62,7 +67,11 @@ object NacRequestBlob {
         val hmacKey20 = draw(20)
         val innerKey16 = draw(16)
         val slot16 = draw(16)
-        val iv16 = draw(16)
+        val fieldMap = PearTables.blobFieldMap
+        val fieldInv = ByteArray(256).also { inv ->
+            for (i in 0 until 256) inv[fieldMap[i].toInt() and 0xff] = i.toByte()
+        }
+        val iv16 = ByteArray(16) { fieldInv[slot16[it].toInt() and 0xff] }
 
         val paesKey = PearAes.initKey(pearKey16)
         val plain32 = ByteArray(32)
@@ -72,7 +81,6 @@ object NacRequestBlob {
         val payloadMap = PearTables.blobPayloadMap
         val mappedPlain = ByteArray(32) { i -> payloadMap[plain32[i].toInt() and 0xff] }
         val enc32 = PearAes.encryptCbc(mappedPlain, iv16, paesKey)
-        val fieldMap = PearTables.blobFieldMap
         val field32 = ByteArray(32) { i -> fieldMap[enc32[i].toInt() and 0xff] }
 
         val oaep = Cipher.getInstance(OAEP_TRANSFORMATION).apply {

registration/src/test/kotlin/dev/fenn/imessage/registration/NacRequestBlobTest.kt

@@ -10,7 +10,7 @@ import kotlin.test.assertEquals
 import kotlin.test.assertFailsWith
 import kotlin.test.assertTrue
 
-/** Synthetic-vector tests for the §1.1 step-2 request blob (C56-a; spec §1.1 rev 35). */
+/** Synthetic-vector tests for the §1.1 step-2 request blob (C56-a; spec §1.1 rev 42). */
 class NacRequestBlobTest {
 
     private class FixedDraw(values: List<ByteArray>) {
@@ -40,7 +40,6 @@ class NacRequestBlobTest {
                 ByteArray(20) { it.toByte() },   // hmacKey20
                 ByteArray(16) { (it + 1).toByte() }, // innerKey16
                 ByteArray(16) { (it + 2).toByte() }, // slot16
-                ByteArray(16) { (it + 3).toByte() }, // iv16
             ),
         )
         val (blob, pearKey16) = blob(pair.public, draw)
@@ -67,10 +66,13 @@ class NacRequestBlobTest {
         val hmacKey20 = ByteArray(20) { (0x40 + it).toByte() }
         val innerKey16 = ByteArray(16) { (0x50 + it).toByte() }
         val slot16 = ByteArray(16) { (0x60 + it).toByte() }
-        val iv16 = ByteArray(16) { (0x70 + it).toByte() }
-        val draw = FixedDraw(listOf(hmacKey20, innerKey16, slot16, iv16))
+        val draw = FixedDraw(listOf(hmacKey20, innerKey16, slot16))
         val (blob, pearKey16) = blob(keyPair().public, draw)
 
+        val fieldInv = ByteArray(256).also { inv ->
+            for (i in 0 until 256) inv[PearTables.blobFieldMap[i].toInt() and 0xff] = i.toByte()
+        }
+        val iv16 = ByteArray(16) { fieldInv[slot16[it].toInt() and 0xff] }
         val plain32 = ByteArray(32)
         plain32[0] = 0x02
         plain32[4] = 0x01
@@ -81,9 +83,29 @@ class NacRequestBlobTest {
         assertContentEquals(field32, blob.copyOfRange(286, 318))
     }
 
+    /** Genuine c56j replay vector: the three real draws must reproduce the blob's every
+     *  deterministic region (OAEP is randomized by construction and checked by length only). */
+    @Test
+    fun c56jReplayVector() {
+        fun hx(s: String) = ByteArray(s.length / 2) { ((Character.digit(s[it * 2], 16) shl 4) + Character.digit(s[it * 2 + 1], 16)).toByte() }
+        val pearKey16 = hx("80422dce4d634e44da4996c8ec871c5f")
+        val hmacKey20 = hx("ccbe18c8d3bca949ec5f5be2060ca8c61a2cc3f4")
+        val innerKey16 = hx("0238210e70ba93d3b0188c21ccc64d17")
+        val slot16 = hx("018c1c81233a8b9d3fb8b9ff94228649")
+        val field32 = hx("16f0af0b2ac8ac222dcfea95d6a7c9ddad8182f3b395502517854d6913d91fc5")
+        val hmac = hx("0f117d2462b730a53797b324fe61c1bfff899348")
+        val draw = FixedDraw(listOf(hmacKey20, innerKey16, slot16))
+        val blob = NacRequestBlob.build(keyPair().public, pearKey16) { n -> draw.next(n) }
+        assertEquals(NacRequestBlob.BLOB_BYTES, blob.size)
+        assertContentEquals(NacRequestBlob.header, blob.copyOfRange(0, 10))
+        assertContentEquals(slot16, blob.copyOfRange(266, 282))
+        assertContentEquals(field32, blob.copyOfRange(286, 318))
+        assertContentEquals(hmac, blob.copyOfRange(318, 338))
+    }
+
     @Test
     fun rejectsWrongPearKeySize() {
-        val draw = FixedDraw(listOf(ByteArray(20), ByteArray(16), ByteArray(16), ByteArray(16)))
+        val draw = FixedDraw(listOf(ByteArray(20), ByteArray(16), ByteArray(16)))
         assertFailsWith<IllegalArgumentException> {
             NacRequestBlob.build(keyPair().public, ByteArray(15)) { n -> draw.next(n) }
         }