Make sure the reverse proxy you are using to rewrite OctoPrint's URLs is setting the X-Script-Name header to your selected URL prefix so OctoPrint knows it is being accessed this way and knows to generate the correct URLs. Example: if you've set up your reverse proxy to serve OctoPrint on its address under http://<your server>/some/octoprint/installation you'll need to make it set the header X-Script-Name: /some/octoprint/installation in its requests to OctoPrint.
There are examples on the wiki on how to set everything up correctly. Alternatively, you can force OctoPrint to always assume a specific base URL by editing the configuration file.