Can I take it that you're attempting to VNC into your Raspberry Pi which includes the OctoPi image?
The VNC server requires a GUI (Desktop, a.k.a. X windows) to be running for the endpoint. The vanilla OctoPi image is based upon the Raspbian Stretch Lite image itself so out-of-the-box, it doesn't include that.
I VNC into my Raspberry but one of the prerequisites is to remote into the Raspberry Pi using
putty and then run
sudo ~/scripts/install-desktop. The script will take a while but it eventually will reboot. If you have an HDMI monitor attached to the Raspberry Pi, you'd see the Raspbian PIXEL Desktop show up rather than the standard console. If you're like me—on a Mac—then it's necessary to reconfigure the authentication on the server side so that you can then VNC into the Raspberry.
The smaller window is a VNC session I've done to my Raspberry and you're seeing the Desktop of its TFT screen here.
If you don't want to install the GUI, then the command line
screen command can be used to remote into the console of the user who has logged in locally. It's not installed by default, btw.