My path was similar. I started by powering the Pi with the official PSU and only connecting TX, RX, 22 and 23. That didn't work. So then I experimented with everything in between.
One of my experiments was to unhook the official PSU and power the Pi4 exactly like the Pi Zero using the 5v header. That worked, but still was not connecting to the serial connection.
What I now know as I have it working is:
- I must have the GND connected or else I get garbage on the Prusa screen
- I have to manually define the serial connection
- The fan that came with my Pi4 sounds like a screaming child unless I undervolt it
- The case I printed for the Pi4 sucks
That said, I'm looking good right now using the beta code and the Pi4 8GB.