sceVdecCoreSetDecodeInput passed the 64-bit auSize to av_new_packet as an
int and to memcpy as a size_t, so a size past 32 bits allocated its low
bits and copied the whole length. The size is now checked once against
the largest packet av_new_packet accepts, INT_MAX less
AV_INPUT_BUFFER_PADDING_SIZE and one, and that one length is used for the
allocation and the copy. A larger unit is refused with the
invalid-argument error before it is read.
sceVdecCoreGetDecodeOutput took the caller's frameBufferSize as the
output's size even when the caller gave no address and the picture went
to a mapped slot. A size larger than the slot put the picture
information past the slot's end.
The size now follows the destination: the caller's size for the
caller's buffer, the frame buffer size for a mapped slot, and room for
the picture and its information in the fallback buffer. A picture
larger than its destination is still refused, and the picture
information is written only where it fits after the picture.
ResetDecoder, SetDecodeInput, SyncDecode, GetDecodeOutput,
FlushDecodeOutput, SyncDecodeWptr, TrySyncDecode and TrySyncDecodeWptr
follow shadPS4's vdeccore.cpp. CreateDecoder starts a decode thread of the
decoder's own that takes packet, flush and reset commands in order and
decodes with the FFmpeg context it opens, and DeleteDecoder stops and joins
it and frees what is still queued. GetDecodeOutput converts a picture to
NV12 and copies it, as shadPS4's CopyNV12Data does, to the caller's frame
buffer, else the next mapped slot, else a fallback buffer the decoder owns,
writes the hardware picture info at the end of the frame buffer and fills
the output info as shadPS4 does. The six exports that are stubs in shadPS4
stay unknown.
The tests decode the H.264 fixture into slots, a caller buffer and the
fallback, compare the visible picture with libSceVideodec2's decode of it,
and cover every argument error, the empty queue, an invalid pts, flush,
reset, try-sync, the write-pointer forwards and a delete with queued input.
Implement sceVdecCoreQueryInstanceSize, sceVdecCoreQueryFrameBufferInfo,
sceVdecCoreQueryComputeResourceInfo, sceVdecCoreCreateDecoder,
sceVdecCoreDeleteDecoder and sceVdecCoreMapMemoryBlock as shadPS4's
vdeccore.cpp does at 0614ce324b: its argument and codec-type checks, its
1920x1080 and 16-frame defaults, DPB counts from -1 to 16, its instance,
compute and frame-buffer sizes, and frame-buffer slots cut from mapped
blocks.
CreateDecoder opens an FFmpeg H.264 or HEVC context with the FFmpeg the
tree already links for libSceVideodec2 but starts no decode thread. The
library gets its own CMakeLists.txt and a guest_vdec_core test.
Yunus Shaikh