Merge pull request #11035 from orezraey/fix/libsceaudioout-repeated-init-measured-ps5

fix(libSceAudioOut): fail a repeated sceAudioOut2Initialize and sceAudioOutInit as measured on a PS5

core/libs/prx/libSceAudioOut/src/AudioOut.cpp

@@ -403,20 +403,28 @@ static int allocatePort(int type, std::uint32_t len, std::uint32_t freq, Format
 }
 
 static std::atomic<bool> g_audioOut2Initialized{false};
+static std::atomic<bool> g_audioOutInitialized{false};
+static constexpr int SCE_AUDIO_OUT2_ERROR_ALREADY_INITIALIZED = static_cast<int>(0x80268004);
+static constexpr int SCE_AUDIO_OUT_ERROR_ALREADY_INITIALIZED = static_cast<int>(0x8026000e);
 
 bool AudioOut2IsInitialized_nid_no_patch() {
     return g_audioOut2Initialized.load();
 }
 
+void AudioOut2MarkInitialized_nid_no_patch() {
+    g_audioOut2Initialized.store(true);
+}
+
 extern "C" {
 
 int APS5_VABI sceAudioOut2Initialize(void) {
-    g_audioOut2Initialized.store(true);
-    return 0;
+    return g_audioOut2Initialized.exchange(true) ? SCE_AUDIO_OUT2_ERROR_ALREADY_INITIALIZED : 0;
 }
 
 int APS5_VABI sceAudioOutInit() {
-    return sceAudioOut2Initialize();
+    if (g_audioOutInitialized.exchange(true)) return SCE_AUDIO_OUT_ERROR_ALREADY_INITIALIZED;
+    AudioOut2MarkInitialized_nid_no_patch();
+    return 0;
 }
 
 int APS5_VABI sceAudioOutOpen(int userId, int type, int index, std::uint32_t len,

core/libs/prx/libSceAudioOut/src/AudioOut2Internal.hpp

@@ -26,6 +26,7 @@ bool AudioOut2TraceEnabled();
 // Seconds since the first AudioOut2 call, for the trace.
 double AudioOut2TraceSeconds();
 bool AudioOut2IsInitialized_nid_no_patch();
+void AudioOut2MarkInitialized_nid_no_patch();
 
 #define AUDIOOUT2_TRACE(...) \
     do { \

core/libs/prx/libSceAudioOut/src/AudioOut2Low.cpp

@@ -3,11 +3,13 @@
 #include <cstdint>
 #include "SceTypes.hpp"
 #include "prx/libc/include/General.hpp"
+#include "AudioOut2Internal.hpp"
 
 extern "C" {
 
 int APS5_VABI sceAudioOut2LoInit(void) {
-    return sceAudioOut2Initialize();
+    AudioOut2MarkInitialized_nid_no_patch();
+    return 0;
 }
 
 int APS5_VABI sceAudioOut2LoTerminate(void) {

core/libs/prx/libSceAudioOut/src/AudioOut2System.cpp

@@ -15,7 +15,8 @@ extern "C" {
 
 int APS5_VABI sceAudioOut2InitializeEx(std::uint32_t param) {
     (void)param;
-    return sceAudioOut2Initialize();
+    AudioOut2MarkInitialized_nid_no_patch();
+    return 0;
 }
 
 int APS5_VABI sceAudioOut2Set3DLatency(int userId, std::uint32_t output) {

core/libs/prx/libSceAudioOut/tests/AudioOut2DeviceState.cpp

@@ -114,6 +114,12 @@ void SpeakerInfoAfterAudioOutInit() {
     Require(sceAudioOut2GetSpeakerInfo(&info, 0) == 0);
 }
 
+void RepeatedInitializationAfterAudioOutInit() {
+    Require(sceAudioOut2Initialize() == static_cast<int>(0x80268004));
+    Require(sceAudioOut2Initialize() == static_cast<int>(0x80268004));
+    Require(sceAudioOutInit() == static_cast<int>(0x8026000e));
+}
+
 void SpeakerInfoRejectsBadFlags() {
     AudioOut2SpeakerInfo info;
     std::memset(&info, 0xAB, sizeof(info));
@@ -145,6 +151,7 @@ int main() {
 
     SpeakerInfoRequiresInitialization();
     SpeakerInfoAfterAudioOutInit();
+    RepeatedInitializationAfterAudioOutInit();
 
     AudioOut2ContextParam params{};
     Require(sceAudioOut2ContextResetParam(&params) == 0);

core/libs/prx/libSceAudioOut/tests/AudioOut2Latency.cpp

@@ -228,6 +228,10 @@ void TestLowLatencyService() {
 
 int main() {
     Require(sceAudioOut2Initialize() == 0, "initialization must succeed");
+    Require(sceAudioOut2Initialize() == static_cast<int>(0x80268004), "a second AudioOut2 initialization fails");
+    Require(sceAudioOutInit() == 0, "the first legacy initialization succeeds after AudioOut2's");
+    Require(sceAudioOut2Initialize() == static_cast<int>(0x80268004), "AudioOut2 initialization fails after the legacy one");
+    Require(sceAudioOutInit() == static_cast<int>(0x8026000e), "a second legacy initialization fails");
     TestSet3DLatency();
     TestMasteringInit();
     TestMasteringTerm();

docs/dev/debt/libSceAudioOut.md

@@ -32,7 +32,7 @@
 - [sceAudioOut2PortGetState](../../../core/libs/prx/libSceAudioOut/src/AudioOut2Port.cpp) (libSceAudioOut) - the port state reports the channel count of the port's data format, output main, volume 127 and a zero reroute counter and flags. Taking the channel count from the data format follows [SharpEmu](https://github.com/sharpemu/sharpemu/blob/main/src/SharpEmu.Libs/Audio/AudioOut2Exports.cs), another emulator. A payload process on a PS5 (firmware 13.60), which is not the foreground application, read output 0, volume -1 and reroute counter 5 for its main ports, and 6 channels for an 8-channel port on a 5.1 system, so a title's values are unverified, including the output of a pad port, the volume after `sceAudioOut2PortSetAttributes` and the reroute counter on a device change.
 - [sceAudioOut2GetSpeakerInfo](../../../core/libs/prx/libSceAudioOut/src/AudioOut2SpeakerArray.cpp) (libSceAudioOut) - The speaker info reports type 0 with available bits `0x3`, the front left speaker at +30 degrees and the front right one at -30, and -360 for the azimuth and elevation of every other slot. On that console the type followed the HDMI device type and the available bits its channels: type 0 with the device set to TV, 1 to AV amplifier and 2 to sound bar, which [SharpProspero](https://github.com/SvenGDK/SharpProspero/blob/5dfdb4b0b166e83dc9ae275001afb695f12c7174/src/SharpProspero/Interop/Audio/AudioOut2.cs) names television, AV receiver and sound bar (3 is headphones there), and available bits `0x3` for 2 channels and `0x3F` for 5.1 (centre at 0, surround pair at +110 and -110, -360 for the LFE).
 - [sceAudioOut2GetSpeakerInfo](../../../core/libs/prx/libSceAudioOut/src/AudioOut2SpeakerArray.cpp) (libSceAudioOut) - The front pair and the empty slots read the same in every setting; 7.1 on the AV amplifier reported 2 channels on that console's HDMI chain. The host output is always reported as a stereo television here. The console also refuses flag 1 below a per-process libkernel value whose source is unidentified; BunnyFl measured the boundary between `0x14FFFFF` and `0x1500000`, and it is not modelled here. ASTRO BOT and ASTRO's PLAYROOM reportedly stop on an assertion that depends on these values (#4317), but which field matters has not been isolated
-- [sceAudioOutInit and sceAudioOut2Initialize repeats](../../../core/libs/prx/libSceAudioOut/src/AudioOut.cpp) (libSceAudioOut) - a repeated `sceAudioOutInit` returns 0 here, while the firmware returns `0x8026000e` on the second call; whether a repeated `sceAudioOut2Initialize` returns 0 is unmeasured. That `sceAudioOutInit` initializes AudioOut2, and that `sceAudioOut2GetSpeakerInfo` then succeeds, follow the firmware and are covered by tests.
+- [sceAudioOut2InitializeEx and sceAudioOut2LoInit repeats](../../../core/libs/prx/libSceAudioOut/src/AudioOut2System.cpp) (libSceAudioOut) - a repeated `sceAudioOut2InitializeEx` or `sceAudioOut2LoInit` returns 0, and either one after `sceAudioOut2Initialize` or `sceAudioOutInit` returns 0 too; the console was not measured for these two. On PS5 firmware 13.60 a repeated `sceAudioOut2Initialize` returns `0x80268004` and a repeated `sceAudioOutInit` `0x8026000e`, and `sceAudioOutInit` after `sceAudioOut2Initialize` still returns 0, which AnyPS5 follows.
 - [sceAudioOut2UserCreateEx](../../../core/libs/prx/libSceAudioOut/src/Unimplemented.cpp) (libSceAudioOut) - unknown signature
 - [sceAudioOutOpenEx](../../../core/libs/prx/libSceAudioOut/src/Unimplemented.cpp) (libSceAudioOut) - unknown signature
 - [sceAudioOutSetSystemDebugState](../../../core/libs/prx/libSceAudioOut/src/Unimplemented.cpp) (libSceAudioOut) - unknown signature