Rawaudio dev

[dingodoppelt]

Add a new "raw" audio quality setting

This PR adds uncompressed audio ("raw") to the quality settings so there is no Opus compression along the way
Discussion in #3654

This feature improves latency as well. I gained 2ms by using uncompressed audio while having a better audio quality.

CHANGELOG: Add uncompressed audio transmission - dedicated to the memory of Hans Petter Selasky (1982 - 2023)

Does this change need documentation? What needs to be documented and how?
Corresponding PR in jamulussoftware/jamuluswebsite #1133

Checklist

• I've verified that this Pull Request follows the general code principles
• I tested my code and it does what I want
• My code follows the style guide
• I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
• I've filled all the content above

[ann0see]

I'd prefer not to check for the Jamulus version number but rather based on capabilities - we don't have 4.0.0 out yet and it might break during the dev process.

[dingodoppelt]

I'd prefer not to check for the Jamulus version number but rather based on capabilities - we don't have 4.0.0 out yet and it might break during the dev process.

I wanted to reuse information already available as much as possible so I just added the code where there were version checks already implemented. (For sequence number and pan feature)
Capabilities would be nice but also would require more changes to client, channel, server and protocol which I don't really have an idea on how to make that backwards compatible. We should rather replace all version checks with some capabilities struct that client and server can agree upon so everything lands in one place. I just don't feel like the right person to take on that challenge and rather pursue my hacky approach, as long as it works for everybody.
The version check with 4.0.0 could be replaced by a point release 3.11.1 and would work right away.

[ann0see]

Tested it and yes, the noise would be unacceptable. What is our fallback if max is selected but the server doesn't support it?

[dingodoppelt]

Tested it and yes, the noise would be unacceptable. What is our fallback if max is selected but the server doesn't support it?

I just noticed that if you connect to a server with Max selected you get the noise unless you switch audio quality again while connected. The server code is fine and doesn't need changes, I misplaced the check for my introduced bRawAudioSupported in the client code. I'll have a closer look
Edit: Funny, the noise doesn't happen on legacy servers, only on rawaudio :D

[dingodoppelt]

I guess this is in a working state now and the only issue remaining is the infrequent unclean exits especially on macOS. As it might even be unrelated to this pull request and I don't have an idea what causes it, I'll set this PR to "ready for review" now.

[dingodoppelt]

As far as I understood these sentinel bytes come from the opus codec itself and are not deliberately set by Jamulus as a message ID of sorts. I'd have to overwrite actual audio bytes for that to work with my code. Or am I wrong here?

[rdica]

@dingodoppelt I've received reports of users on Mac and Windows that have small buffers enabled in their rawaudio clients being unable to hear anything or hear strange audio on rawaudio enabled servers that do not have small buffers enabled. They report when they disable that feature in the client they can hear audio fine. I don't have Mac or Windows to test, and I don't experience the issue in Linux.
From one Windows user I was able to get the client version as 3.11.1dev-4b8bf6a.
Server version is 3.11.1dev-09ae6dfc:1777232839 on Linux.

[softins]

I've received reports of users on Mac and Windows that have small buffers enabled in their rawaudio clients being unable to hear anything or hear strange audio on rawaudio enabled servers that do not have small buffers enabled. They report when they disable that feature in the client they can hear audio fine.

I've just checked this out locally, with a server on Raspberry Pi and a client on Mac, both built from the latest rawaudio-dev at 3f059b3

If I start the server with -F, then all buffer sizes in the client, with small network buffers either off or on, work fine.

If I start the server without -F, then 2.67ms (64) with Small Network Buffers enabled in the client results in very distorted audio. All other settings still work fine.

However, I have also just checked with the Opus quality settings: High, Normal and Low. In those cases, setting 2.67ms (64) with Small Buffers enabled, connected to a server running without -F, also gives the same strange audio. I haven't yet looked at what -F actually does in the code, but it suggests that this issue is not specifically related to Raw Audio.

From previous tests I've done in standard Jamulus, the "Small Network Buffers" checkbox does nothing when set to 5.33ms (128) or 10.67ms (256). If the checkbox is off, then 2.67ms (64) actually sends exactly the same as 5.33ms (128). It's only if the checkbox is on, that 2.67ms (64) sends smaller packets.

[ann0see]

Yes. Can confirm the distorted audio without -F.

Also if you bump both buffers to 1 there seems to be a cyclic distortion. Likely some rounding or timer issue. But it happens on other builds too.

[pljones]

It's definitely interesting that this lack of backwards compatibility without

  -F, --fastupdate        use 64 samples frame size mode

on the server when the client is using hasn't been spotted before -- I've certainly been using Small Network Buffers on every server I connect to and I can't say I've ever noticed anything. Here's my full settings:

Is this happening only on rawaudio servers not using -F?

[softins]

It's definitely interesting that this lack of backwards compatibility without

  -F, --fastupdate        use 64 samples frame size mode

on the server when the client is using Small Network Buffers hasn't been spotted before -- I've certainly been using Small Network Buffers on every server I connect to and I can't say I've ever noticed anything.
...
Is this happening only on rawaudio servers not using -F?

Yes, it seems so. I've just done some tests with Wireshark capturing the traffic between a client and a server running without -F:

• Server 3.11.0, client 3.11.0 - no problem.
• Server latest main, client recent main - no problem.
• Server latest rawaudio-dev 3f059b3, client recent main - 64 byte buffers with Small Buffers enabled gives the distorted audio, even though the client isn't a rawaudio build.

I have saved the Wireshark captures, but not yet examined them.

[pljones]

I asked VSCode CoPilot:

---

Likely cause identified: a packet-send regression in server mix/encode on this branch.

The server can require two sub-blocks per client frame when server runs without fastupdate (128-sample server frame) and a client is on small buffers (64-sample transport). That is set here: server.cpp:771.
In this branch, encode/copy still iterates over both sub-blocks, but packet send is done only once after the loop: server.cpp:1191, server.cpp:1205, server.cpp:1213.
Result: when two blocks are needed, only the last encoded/copied block is transmitted each cycle (the first is overwritten before send). That produces the garbled/alternating distortion pattern users report.
Why this matches your symptom:

It is triggered specifically by non-fastupdate server mode (128-frame server path) with 64-frame clients.
That is exactly the compatibility path that depends on two conversion blocks and is sensitive to send placement.

---

It supplied this patch

@@
-        if ( iCeltNumCodedBytes != static_cast<int> ( iClientFrameSizeSamples * vecNumAudioChannels[iChanCnt] * sizeof ( int16_t ) ) )
+        if ( iCeltNumCodedBytes != static_cast<int> ( iClientFrameSizeSamples * vecNumAudioChannels[iChanCnt] * sizeof ( int16_t ) ) )
         {
             // OPUS encoding
             if ( pCurOpusEncoder != nullptr )
             {
@@
                 for ( int iB = 0; iB < vecNumFrameSizeConvBlocks[iChanCnt]; iB++ )
                 {
                     const int iOffset = iB * SYSTEM_FRAME_SIZE_SAMPLES * vecNumAudioChannels[iChanCnt];
 
                     iUnused = opus_custom_encode ( pCurOpusEncoder,
                                                    &vecsSendData[iOffset],
                                                    iClientFrameSizeSamples,
                                                    &vecvecbyCodedData[iChanCnt][0],
                                                    iCeltNumCodedBytes );
+
+                    // send each encoded sub-block (important when vecNumFrameSizeConvBlocks == 2)
+                    vecChannels[iCurChanID].PrepAndSendPacket ( &Socket, vecvecbyCodedData[iChanCnt], iCeltNumCodedBytes );
                 }
             }
         }
         else
         {
             for ( int iB = 0; iB < vecNumFrameSizeConvBlocks[iChanCnt]; iB++ )
             {
                 const int iOffset = iB * SYSTEM_FRAME_SIZE_SAMPLES * vecNumAudioChannels[iChanCnt];
 
                 memcpy ( &vecvecbyCodedData[iChanCnt][0], &vecsSendData[iOffset], iCeltNumCodedBytes );
+
+                // send each raw sub-block (important when vecNumFrameSizeConvBlocks == 2)
+                vecChannels[iCurChanID].PrepAndSendPacket ( &Socket, vecvecbyCodedData[iChanCnt], iCeltNumCodedBytes );
             }
         }
-        // send separate mix to current clients
-        vecChannels[iCurChanID].PrepAndSendPacket ( &Socket, vecvecbyCodedData[iChanCnt], iCeltNumCodedBytes );

My fault for saying put the PrepAndSendPacket outside the if, I think...

[softins]

Wow, that's impressive! I haven't tried copilot yet.

Although I can't see what is different between the two versions of the first line?

I'll try the patch a little later and repeat my tests.

[dingodoppelt]

I can confirm the issue being fixed with the last commit in which I applied the suggested patch. (actually a revert)

[softins]

Yes, I can confirm the problem with small buffers enabled connected to a server without -F appears to be fixed.

Finally understood that the only difference between having -F or not is that with 64-byte packets, when -F is specified, the server sends them one at a time at 1.33ms intervals, and without -F, it sends them in pairs at 2.66ms intervals. The issue before the last commit was that only one of each pair was being sent.

[dingodoppelt]

Furthermore I can't reproduce the crash on closing anymore. On linux I was getting crashes very rarely in the first place but I could trigger it by connecting to a server with -c <IP> but this doesn't crash anymore when closing the client.
This should be tested on macos and windows, too since I've had reports (mostly from mac users) of the client crashing on close.