I cant really help, but I can confirm I have the same issue and I have had it with PI 3, 3+ and 4. I have tried many different parameters in the config file. It is not just you, you are not going mad!
Since I've got a new (usb) camera installed to replace my Pi Camera on my OctoPrint Pi I've been experiencing similar issues. Even a crash on rendering a time-lapse from a small print.
From what I've seen is that there was a frame-rate mismatch between the config in .octoprint/config.yaml and /boot/octopi.txt
I matched the frame-rates in both files, rebooted and it seems to be a bit better. Haven't tested with creating a time-lapse on prints yet but initially the ffmpeg stream and control tab feel smoother.
So you might want to check the frame-rates in your configuration files.
mates, thank you for your feedback. Indeed, after insane days of searching for a solution, I tried simply to connect the Rpi to an Ethernet adapter and then use a wire to connect it. The problem was definitely solved. The real thing that doesn't convince me is that with the Raspicam, wi-fi was sufficient and it worked without a single problem. Via USB there is something that Octoprint doesn't like, for sure. Thanks