Audio stutters when resizing window

[unvermuthet]

The video and audio rendering should probably be moved to a thread. No clue how to do that with GDExtension :(

Also I feel like the audio is generally pretty delayed but I haven't found a good way to test this. Checking the timestamp and comparing with video might be a good idea.

[unvermuthet]

I'm testing with a sine wave. Sometimes the pitch seems to shift too. This might also be an issue with my planar to interleaved conversion.

[GoranKovac]

Tested with v0.0.4-alpha, so current issues:

1. Audio stutters if window is resized (known issue)
2. Audio and Video are out of sync. Audio comes earlier than video about in about 50-100ms range

[unvermuthet]

I'm using a Media Source in OBS to play this file

http://twitch-event-engineering-public.s3.amazonaws.com/sync-footage/Sync-Footage-V1-H264.mp4

When receiving that NDI Output in Godot the audio is about 50-100ms late. I'm not qualified to do this kind of testing properly though. To get a conclusive result I'd need to be able to eliminate low level factors like OS, display and audio interface latency. I'm using a 165 Hz display and a Saffire Pro 24 Audio Interface. But then there are also plenty of settings in Godot itself, like the "Driver Latency" in the Project Settings or the "Buffering Msec" property of the VideoStreamPlayer.

I should at least do my best to get the timecode of video and audio frames synced up, but if they are out of sync I'd need to delay one of them. Maybe I could sleep for a tiny amount between receiving the frame from the sdk and sending it to the engine. That delay would have to be based on the timecode difference.

[GoranKovac]

Was doing same test locally to have minimal latency as possible (eliminate any network issues):
On my setup the audio hits earlier than video (before center is hit). My setup is 144hz display but have another one 60hz (but that does not matter here) and M-Audio usb interface and RME Fireface 800. Fiddled a bit with various settings but didn't see any improvements.

EDIT:

Godot project setting:
Changing Sample rate 44.1k -> 48k -> 96k no impact
Changing buffering Msec (VideoStreamPlayer) -> no impact
Increasing output latency (100ms max) seems to nullify the delay (still a bit of but closer to target). This does not seem like a solution because it would impact the rest of the project

[GoranKovac]

Weird things happening, after restarting godot for like 27th time now the audio is late 50-100ms like on your setup.

EDIT: another 2-3 restarts and now its back to being earlier than video... (no settings changed, just close/open godot)

[GoranKovac]

Not sure how its related but this issue is also fixed with 0.0.6 alpha (audio stuttering on resize).

The only "issue" remaining is when running in tool mode, audio can get out of sync when changing scene tabs

Arch Linux
Godot 4.3 stable

EDIT2: False alarm. It seems on first initial run resizing window has no issue with audio. But on next consecutive runs it does introduce stutter

[unvermuthet]

I can confirm the desync after switching scenes in the editor

[GoranKovac]

Its probably the thread issue -> main gets stalled when scenes get switched (because audio gets cut off for a brief moment) and some weirdness is introduced then

[unvermuthet]

Tried out multithreading today. Worked but no improvement concerning stutter. Audio also stutters with the Theora implementation. Scraped that approach and focused on fixing the audio desync. The audio shifted with every stutter. Now it still stutters when resizing but will drop old frames instead of new ones fixing the desync - at least for me. See latest CI Build under Actions.

[GoranKovac]

Tested latest CI build:

Indeed desync is fixed, but in some scenarios when running in the editor it would still go out of sync if you switch tabs fast (hard to reproduce, but delay was very noticable like 200+ ms)

[unvermuthet]

You're right.. I see this too but don't know how to fix it. Maybe there is some way of properly pausing playback when the scene gets switched away from. I'm mostly out of ideas though. In my experimentation neither multi-threading nor the timecode helped, by the way.

How critical would you consider this bug? Do you have a particular use case in mind where this would be deal-breaking? I do see playback in the editor as very useful but not production critical, so I'm not super concerned with the delay. Does closing the scene and reopening it fix the desync? If so I'd close this or remove the "delayed" part of the issue title and mark as "Won't fix".

[GoranKovac]

Regarding production, not critical at all. In editor mode would just be to to set things up if needed or maybe just for quick preview while in pre-production (not to play the project million times just to preview something)

Regarding the desync it comes back in sync after few more swaps. But really I would not bother with this (i just keep testing it since issue was open). More critical thing was crashing when swapping which you fixed.

Regarding resize issue not sure if something can be done because godot also does weird things with graphics for example if global illumination is on and other little more intensive effects (everything flickers when window gets resized).

[unvermuthet]

Concerning the use case you mentioned. Changing the VideoStreamNDI Resource (so name or bandwidth) currently reinstantiates the VideoStreamPlaybackNDI but the VideoStreamPlayer doesn't start playing again, even with Autoplay enabled. This is a Godot thing. As a UX improvement I'll try to hijack the "Paused" property to also call stop and play when running in the editor. This way you can toggle that to start playback of the updated source without having to reopen the scene.

[GoranKovac]

After a quick search over godot git, seems that issue with resizing is its allocating new memory. Vulkan and GL suffer most from it but apparently on DX12 its smooth.

Anyway devs say its all by design

[unvermuthet]

I see. It would be cool if you could add a link to that discussion here. Thanks again for helping out!

[GoranKovac]

godotengine/godot#84565
godotengine/godot#94430

I think this is what we are experiencing (but maybe Im wrong):

godotengine/godot#88337 (comment)

So basicaly resize, gpu allocates new memory -> brief block -> video cant update -> audio stutter

[unvermuthet]

Good finds! Sadly DirectX doesn't make a difference for me. Tested on Windows.
Seems like there are a lot of eyes on it though. Guess we just hope this works one day.

[GoranKovac]

Went to test the plugin on Godot 4.4 beta1 and to my surprise audio stuttering when window is resized is gone. Works absolutely flawless both in embedded and windowed mode!

Regarding running the plugin in in editor (tool mode) desync can still happen when switching tabs + some audio dropouts/stutter like in 4.3 stable.

So in general resizing issue is non existent in godot 4.4beta1 ATM (we will see what happens with next dev cycles)

[GoranKovac]

Update regarding Godot 4.4beta above:
It seems that 4.4beta behaves differently with vsync, it always forces 60fps no matter what the refresh rate of the monitor (144hz here) is:

GODOT 4.4beta -> MAX FPS 0, VSYNC ON -> 60FPS
GODOT 4.3 -> MAX FPS 0, VSYNC ON -> 144FPS

Setting fps max to 60 in 4.3 makes audio stutter disappear when resizing window.
What actually happens is when window is resized fps dips alot when on high fps and makes a stutter, while on 60 it never dips (from my testings and can reliably reproduce it every time)