Merge pull request #10913 from thuanquang/fix/videodec2-discarded-frame

fix(libSceVideodec2): mark returned pictures as not discarded

core/libs/prx/libSceVideodec2/Export.cpp

@@ -38,6 +38,7 @@ constexpr int Videodec2ErrorAccessUnitPointer = static_cast<int>(0x811D010Eu);
 
 static_assert(sizeof(Vp9PictureInfo) == 88);
 static_assert(offsetof(Vp9PictureInfo, colorSpace) == 60);
+static_assert(sizeof(OutputInfo) == 0x38 && offsetof(OutputInfo, isDiscardedFrame) == 0xb && offsetof(OutputInfo, codecType) == 0xc && offsetof(OutputInfo, frameBuffer) == 0x20);
 
 class Vp9Bits {
 public:
@@ -458,6 +459,7 @@ void EmitDecoded(std::uint64_t handle, Decoder& decoder, FrameBuffer* frame, Out
     output->isValid = true;
     output->isErrorFrame = false;
     output->pictureCount = 1;
+    output->isDiscardedFrame = false;
     output->frameWidth = decoder.width;
     output->framePitch = pitch / decoder.sampleBytes;
     output->frameHeight = decoder.height;

core/libs/prx/libSceVideodec2/include/Api.hpp

@@ -101,6 +101,7 @@ struct OutputInfo {
     bool isValid;
     bool isErrorFrame;
     std::uint8_t pictureCount;
+    bool isDiscardedFrame;
     std::uint32_t codecType;
     std::uint32_t frameWidth;
     std::uint32_t framePitch;

core/libs/tests/GuestVideodec2.cpp

@@ -578,6 +578,53 @@ void testEmptyAccessUnit() {
     check(sceVideodec2DeleteDecoder_nid_postfix(handle) == 0, "delete failed");
 }
 
+void testDiscardedFrameFlag() {
+    static_assert(sizeof(OutputInfo) == 0x38 && offsetof(OutputInfo, pictureCount) == 0xa && offsetof(OutputInfo, codecType) == 0xc);
+    std::vector<std::uint8_t> buffer(frameBufferBytes());
+    const auto decodeAll = [&](std::uint64_t handle, const std::vector<std::vector<std::uint8_t>>& units, std::size_t expected, bool lastPicturesFromFlush) {
+        std::size_t pictures = 0;
+        std::size_t flushed = 0;
+        const auto call = [&](const InputData* input) {
+            FrameBuffer frame{sizeof(FrameBuffer), buffer.data(), buffer.size(), false};
+            OutputInfo output;
+            std::memset(&output, 0xff, sizeof(output));
+            output.thisSize = sizeof(OutputInfo);
+            const auto result = input ? sceVideodec2Decode_nid_postfix(handle, input, &frame, &output) : sceVideodec2Flush_nid_postfix(handle, &frame, &output);
+            check(result == 0, "decode or flush failed");
+            std::uint8_t discarded = 0;
+            std::memcpy(&discarded, reinterpret_cast<const std::uint8_t*>(&output) + 0xb, 1);
+            check(!output.isValid || discarded == 0, "a returned picture is marked as discarded");
+            check(output.isValid || discarded == 0xff, "a result without a picture wrote the discarded flag");
+            if (output.isValid) {
+                ++pictures;
+                if (!input) ++flushed;
+            }
+            return output.isValid;
+        };
+        {
+            FrameBuffer frame{sizeof(FrameBuffer), buffer.data(), buffer.size(), false};
+            OutputInfo output;
+            std::memset(&output, 0xff, sizeof(output));
+            output.thisSize = sizeof(OutputInfo);
+            const InputData empty{sizeof(InputData), units[0].data(), 0, 0, 0, 0};
+            check(sceVideodec2Decode_nid_postfix(handle, &empty, &frame, &output) == static_cast<int>(0x811D010Du), "an empty access unit was accepted");
+            std::uint8_t discarded = 0;
+            std::memcpy(&discarded, reinterpret_cast<const std::uint8_t*>(&output) + 0xb, 1);
+            check(discarded == 0xff, "a refused access unit wrote the discarded flag");
+        }
+        for (std::size_t unit = 0; unit < units.size(); ++unit) {
+            const InputData input{sizeof(InputData), units[unit].data(), units[unit].size(), 1000 + unit, unit, 0};
+            call(&input);
+        }
+        while (call(nullptr)) {}
+        check(pictures == expected, "a picture is missing");
+        check(!lastPicturesFromFlush || flushed > 0, "no picture came back from Flush");
+        check(sceVideodec2DeleteDecoder_nid_postfix(handle) == 0, "delete failed");
+    };
+    decodeAll(createDecoder(CodecVp9, 0), ivfFrames(), Vp9Hashes.size(), false);
+    decodeAll(createDecoder(1, 100), accessUnits(false), PictureHashes.size(), true);
+}
+
 void testFlushAfterEveryUnit() {
     std::vector<std::vector<std::uint8_t>> units;
     for (std::size_t index = 0; index + 4 < AvcIpppStream.size(); ++index) {
@@ -788,6 +835,7 @@ int main() {
         testTenBit();
         testDecodeAfterFlush();
         testEmptyAccessUnit();
+        testDiscardedFrameFlag();
         testFlushAfterEveryUnit();
         testPictureOwnership();
         testConcurrentOwnership();

docs/dev/debt/libSceVideodec2.md

@@ -17,3 +17,4 @@
 - [sceVideodec2Decode after sceVideodec2Flush](../../../core/libs/prx/libSceVideodec2/Export.cpp) (libSceVideodec2) without `sceVideodec2Reset` restarts FFmpeg's decoder (`avcodec_flush_buffers`) and decodes the unit, where it threw before. TEKKEN DARK RESURRECTION (PPSA31371), which runs in Sony's PSP emulator, decodes after a flush without a reset (#7118), so the console accepts the call. Pictures the flush decoded but has not returned yet are returned before the new ones. A flush with no picture pending (each decoded unit has returned its picture) leaves the decoder and its reference pictures as they are, so a stream that continues without a new IDR picture keeps decoding: Daxter (PPSA09695), also in the PSP emulator, flushes after every access unit of its H.264 movies, and restarting the decoder there lost every P picture until the next keyframe. What the console returns for pending pictures, and for a stream that continues across a flush that had pictures pending, is not measured.
 - [sceVideodec2Decode](../../../core/libs/prx/libSceVideodec2/Export.cpp) (libSceVideodec2) and `sceVideodec2Flush` output the coded picture, with the chroma plane after its luma rows, as the console does for 1080p H.264 and HEVC streams coded at 1920x1088. An HEVC or VP9 picture whose size is not a multiple of 16, such as HEVC coded at 1920x1080, is output at that size; whether the console pads it is unverified. `sceVideodec2GetPictureInfo` reports no frame cropping
 - [sceVideodec2Decode of an empty access unit](../../../core/libs/prx/libSceVideodec2/Export.cpp) (libSceVideodec2) returns `0x811D010E` for a null `auData` and `0x811D010D` for an `auSize` of 0, with no picture, where it threw before. The values are `ACCESS_UNIT_POINTER` and `ACCESS_UNIT_SIZE` in the error table of [shadPS4's videodec_error.h](https://github.com/shadps4-emu/shadPS4/blob/f3f0c14c1350a768558dfd1fd058d259ccecaca2/src/core/libraries/videodec/videodec_error.h), whose libSceVdecsw codes this repository already uses; they are not measured on a PS5, and neither is whether the console leaves the decoder's state alone on that call, as this does. Media Player Classic PS5 (PPSA99015) submits empty units: its own converter reads a 4-byte length prefix of `00 00 01 xx` as a start code and drops the rest as a non-base HEVC layer. It goes on after the error.
+- [sceVideodec2Decode and sceVideodec2Flush](../../../core/libs/prx/libSceVideodec2/Export.cpp) (libSceVideodec2): the decoder core's discarded flag is not modelled: a returned picture always has `OutputInfo.isDiscardedFrame` (byte 0xb) at 0, and an unshown VP9 frame returns no picture. Per a #10913 review reading firmware 13.42's module (not reproduced here), the byte is written only with a returned picture and is left as the caller set it otherwise, as here. Name and offset: [SharpProspero](https://github.com/SvenGDK/SharpProspero/blob/5dfdb4b0b166e83dc9ae275001afb695f12c7174/src/SharpProspero/Interop/Video/Videodec2.cs#L150-L192) and [KytyPS5](https://github.com/KytyPS5/KytyPS5/blob/83f8bfd3c9c35fcb135ef6c33dfd88bbfe066ee2/src/libs/libVideoDec2.cpp#L91-L105); shadPS4 leaves it as padding. PPSA05110 sets only `thisSize` before its VP9 Decode (0x13728dd) and Flush (0x1371f7e) and takes a picture only when the byte is 0