Prepare nested Metal layers before compositing

Traverse the layer tree before compositing so nested Metal layers consume queued drawables and recycle their pool. Add a nested presentation probe covering pool reuse and Vulkan-to-OpenGL pixel readback.

docs/TESTING.md

@@ -82,7 +82,7 @@ Useful probes (build tokenized Python helpers from `build/runtime/tools/`):
 - `riot-event-coalescing-check`: preserve key events and latest mouse motion.
 - `m327-cgl-resolver-check`: first calls through dynamically looked-up GL functions retain floating-point arguments; expected `first_sampler_linear=1 first_color_arguments=1 errors=0,0`. Build with `build-m58f327-opengl.py` and run with `run-riot-native-check.py m327-cgl-resolver-check`.
 - `m327-cgl-direct-check`: with `RCLOL_GPU_DIRECT_OTHER_WINDOW_TEST=1` and an owned 337x339 native target, interleave 338x339 fallback surfaces; expected `active_presenter_preserved=1 fallback_frames=357`. This needs the native presentation fixture; it is not a `run-riot-native-check.py` choice.
-- `riot-metal-presentation-check`: Vulkan clear reaches the GL texture, including late window/context attachment.
+- `riot-metal-presentation-check`: Vulkan clear reaches the GL texture, including late window/context attachment. Set `RIOT_PRESENTATION_NESTED=1` to place the Metal layer beneath two ordinary layers and verify that its drawables are still presented and recycled past the three-drawable pool limit.
 - `riot-metal-concurrency-check`: submission ordering, resource lifetime and concurrent recording.
 
 `RIOT_PRESENTATION_STRESS=1` makes the presentation probe run 120 frames. With the game and clients closed, force async encoding for this standalone probe using the guest flag `/tmp/metal-async-encode`, then remove it. The tested old path leaked 120 synchronization FDs over 120 frames; the fixed path leaked zero. Inspect the probe's host `/proc/PID/fd` for `anon_inode:syncobj_file` if reproducing that test.

docs/TROUBLESHOOTING.md

@@ -218,6 +218,8 @@ The first import of a drawable into the presenting OpenGL context, and the first
 
 A non-zero error or `FAILED` means the frames cannot reach the window. Before 2026-10-16 the import did not mark the memory as dedicated, which the extension requires; Mesa ignores it, and it is the suspected cause of the NVIDIA black screen (not yet confirmed). An NVIDIA log of 2026-10-16 showed `glTiling=optimal` with `setup_error=0x501 storage_error=0x501 FAILED`: NVIDIA's GL rejects `GL_LINEAR_TILING_EXT`, so the linear Vulkan image could not be imported. Since then the internal image and the GL texture use optimal tiling when `GL_VENDOR` contains NVIDIA (`vkTiling=optimal glTiling=optimal ok` is the expected line); everything else keeps linear. If the game shows a black screen on NVIDIA, report it with this file.
 
+An additional NVIDIA report on `68f816d` showed successful optimal-tiling import but repeated `Metal Device timed out giving us a Drawable to present` messages. Its [presentation trace](https://github.com/lucannez64/darling-lol/issues/3#issuecomment-6214732591) shows all three drawables queued and all three main-thread presentation callbacks running, followed by timeouts with three frames still queued. The compositor only prepared a Metal layer when it was the root layer, skipping nested Metal layers. It now prepares Metal layers throughout the tree. A standalone nested-layer reproduction exhausts the pool before this correction and passes ten presentations with pixel readback after it; the root-layer probe also passes. Confirmation on the reporter's NVIDIA system is still needed.
+
 ### Host-side checks
 
 These are host settings, not game settings. Run on AC power, and use a performance power profile (`powerprofilesctl set performance`, or `asusctl profile -P Performance` on ASUS laptops). On the tested desktop the game is limited mostly by its own main thread on the CPU, not by the GPU (see [status](STATUS.md#game-fps-2026-10-1314)), so laptop CPU power limits matter as much as the GPU choice.

patches/cocotron.patch

@@ -6871,7 +6871,7 @@ index 5dcac115..5de4f03f 100644
 +
  @end
 diff --git a/QuartzCore/CALayerContext.m b/QuartzCore/CALayerContext.m
-index 5dce8c56..e329b5f8 100644
+index 5dce8c56..dee2b80c 100644
 --- a/QuartzCore/CALayerContext.m
 +++ b/QuartzCore/CALayerContext.m
 @@ -1,47 +1,58 @@
@@ -7027,18 +7027,30 @@ index 5dce8c56..e329b5f8 100644
  }
  
  - (void) assignTextureIdsToLayerTree: (CALayer *) layer {
-@@ -93,7 +148,10 @@
+@@ -93,7 +148,22 @@
          [self assignTextureIdsToLayerTree: child];
  }
  
 +extern void darlingQCTiming(const char* what, double seconds);
++
++- (void) prepareMetalLayers: (CALayer *) layer {
++    // Metal layers may be nested beneath ordinary backing layers. Every queued
++    // drawable must be consumed before CARenderer samples the layer tree.
++    if ([[layer class] isSubclassOfClass: [CAMetalLayerInternal class]]) {
++        CFTimeInterval prepStart = CACurrentMediaTime();
++        [(CAMetalLayerInternal*)layer prepareRender];
++        darlingQCTiming("prepareRenderTotal", CACurrentMediaTime() - prepStart);
++    }
++    for (CALayer *child in layer.sublayers)
++        [self prepareMetalLayers: child];
++}
 +
  - (void) renderLayer: (CALayer *) layer {
 +    CFTimeInterval renderLayerStart = CACurrentMediaTime();
      CGLContextMakeCurrentAndAttachToWindow(_glContext, _cglWindow);
  
      glEnable(GL_DEPTH_TEST);
-@@ -118,27 +176,69 @@
+@@ -118,27 +188,63 @@
  
      [_deleteTextureIds removeAllObjects];
  
@@ -7047,13 +7059,12 @@ index 5dce8c56..e329b5f8 100644
      [self assignTextureIdsToLayerTree: layer];
 +    darlingQCTiming("assignTextureIds", CACurrentMediaTime() - assignStart);
  
-     // this is where the Metal layer renders to an internal texture for us to use
-     if ([[layer class] isSubclassOfClass: [CAMetalLayerInternal class]]) {
-         CAMetalLayerInternal* mtl = (CAMetalLayerInternal*)layer;
-+        CFTimeInterval prepStart = CACurrentMediaTime();
-         [mtl prepareRender];
-+        darlingQCTiming("prepareRenderTotal", CACurrentMediaTime() - prepStart);
-     }
+-    // this is where the Metal layer renders to an internal texture for us to use
+-    if ([[layer class] isSubclassOfClass: [CAMetalLayerInternal class]]) {
+-        CAMetalLayerInternal* mtl = (CAMetalLayerInternal*)layer;
+-        [mtl prepareRender];
+-    }
++    [self prepareMetalLayers: layer];
  
 +    CFTimeInterval rendererStart = CACurrentMediaTime();
      [_renderer render];
@@ -7108,7 +7119,7 @@ index 5dce8c56..e329b5f8 100644
  }
  
  - (void) startTimerIfNeeded {
-@@ -147,7 +247,7 @@
+@@ -147,7 +253,7 @@
                                                     target: self
                                                   selector: @selector(timer:)
                                                   userInfo: nil
@@ -7117,7 +7128,7 @@ index 5dce8c56..e329b5f8 100644
  }
  
  - (void) deleteTextureId: (NSNumber *) textureId {
-@@ -155,7 +255,10 @@
+@@ -155,7 +261,10 @@
  }
  
  - (void) flush {

tools/riot-metal-presentation-check.mm

@@ -40,7 +40,18 @@ int main() {
         layer.frame = CGRectMake(0,0,160,96);
         NSView* view = [[NSView alloc] initWithFrame:NSMakeRect(0,0,160,96)];
         view.wantsLayer = YES;
+        const bool nested = getenv("RIOT_PRESENTATION_NESTED") != nullptr;
+        if (nested) {
+            CALayer* root = [CALayer layer];
+            root.frame = layer.frame;
+            CALayer* container = [CALayer layer];
+            container.frame = layer.frame;
+            [container addSublayer:layer];
+            [root addSublayer:container];
+            view.layer = root;
+        } else {
             view.layer = layer;
+        }
         NSWindow* window = [[NSWindow alloc] initWithContentRect:NSMakeRect(50,50,160,96)
           styleMask:NSTitledWindowMask backing:NSBackingStoreBuffered defer:NO];
         window.title = @"Metal presentation check";
@@ -92,7 +103,8 @@ int main() {
             fprintf(stderr, "PASS: bounded presentation capture rearmed without restarting\n");
         }
         fprintf(stderr, "PASS: Vulkan-to-OpenGL presentation pixels\n");
-        fprintf(stderr, "PASS: late context attachment and %u Metal drawable presentations\n", frameCount);
+        fprintf(stderr, "PASS: %s layer, late context attachment and %u Metal drawable presentations\n",
+                nested ? "nested" : "root", frameCount);
         [window orderOut:nil];
         [queue release]; [view release]; [window release]; [device release];
     }