OctoPi web page won't properly load anymore

What is the problem?
Recently, out of the blue it seems, my Octopi web page stopped working. I've had it working perfectly for about 6 months, but then suddenly I was in the middle of a small print and the page stopped loading. It was working fine when I started the print, but from then on, I've not been able to get it to work properly at all.

Sometimes it will show the haproxy "default" page (image 1), other times it will show a garbled half loaded mess of css/js (image 2). When I look at the javascript console on image 2, I see 404 errors on various files (image 3), but different files each time I hit refresh. But I can never get it to completely load anymore. Also, ssh access, and every other aspect of the system seems completely normal. The octoprint services appear to be and happily listening for requests.

Any help would be greatly appreciated!

Since I am only allowed to upload 1 image, I've combined three into one (stacked vertically):

What did you already try to solve it?
Restarted octoprint
Rebooted the pi
Restarted in safe mode
Upgraded to the latest release
Tried multiple browsers and computers
Cleared all browser caches
Took haproxy out of the mix and directly connected to the octoprint service, and I still got the 404s and errors

The behavior was unchanged after all of these steps.

Additional information about your setup (OctoPrint version, OctoPi version, printer, firmware, octoprint.log, serial.log or output on terminal tab, ...)
OctoPrint version: 1.3.10
Rpi RevB
Ender 3

The octoprint.log has nothing really interesting in it, for example:
2019-01-26 16:41:10,422 - tornado.access - WARNING - 409 GET /api/printer?history=false (::1) 20.14ms
2019-01-26 16:41:24,807 - octoprint.plugins.navbartemp - INFO - Checking SoC internal temperature
2019-01-26 16:41:54,855 - octoprint.plugins.navbartemp - INFO - Checking SoC internal temperature
2019-01-26 16:42:06,800 - tornado.access - WARNING - 409 GET /api/printer?history=false (::1) 19.96ms
2019-01-26 16:42:24,899 - octoprint.plugins.navbartemp - INFO - Checking SoC internal temperature

The serial.log is also boring:
2019-01-26 11:06:06,066 - Recv: T:15.39 /0.00 B:15.00 /0.00 @:0 B@:0
2019-01-26 11:06:08,065 - Recv: T:14.64 /0.00 B:15.27 /0.00 @:0 B@:0
2019-01-26 11:06:10,061 - Recv: T:14.69 /0.00 B:14.32 /0.00 @:0 B@:0
2019-01-26 11:06:12,060 - Recv: T:14.95 /0.00 B:14.69 /0.00 @:0 B@:0
2019-01-26 11:06:14,058 - Recv: T:16.21 /0.00 B:15.66 /0.00 @:0 B@:0

The first image doesn't look like haproxy to me; it looks like what it says it is: Apache2. If you installed the OctoPi image to a Raspberry Pi 3B, then there's no reason for it to include the Apache2 webserver. Assuming that Apache2 isn't running on this Pi, perhaps you have a DNS/arp/DHCP problem of some kind that is sending you to the wrong host.

Why would you do that? It was the part that was converting http://octopi.local:80/ to http://octopi.local:5000/ for you. Port 80 is the default port. OctoPrint itself runs on port 5000. You need that proxy to find the stylesheet, the graphics, JavaScript, downloadable content like gcode/timelapses/logs, etc.

Thanks that turned out to be the problem. I must have installed a package that included apache without my realizing it, and (probably after the next reboot) apache and haproxy were battling it out for control of port 80. Disabling apache solved the problem.

Although it really doesn't matter, I still can't explain some aspects of the symptoms, since it seemed like haproxy and apache were both answering requests on port 80, which I understand to be impossible. For example, how did I ever even launch that last job from the UI? Also I noticed that some scripts I write to call the API were getting intermittent 404 errors. Odd.

To answer your question on why I took haproxy out of the mix-- it was basic troubleshooting, and turned out to be the correct impulse in hindsight. I was just isolating components to try to learn which one was causing the problems.
Also, for clarity, it would appear that all assets including css, js, api calls, etc are actually handled by the octoprint daemon itself. Haproxy appears to only be in place to handle the /webcam/ redirect and the SSL certificate, at least on my default install of octopi.

Thanks again for the help!

If Apache2 owned port 80 traffic then that should be obvious: the default Apache webpage didn't have an /api endpoint so it 404'd, right?

Even if haproxy and Apache2 were fighting it out over port 80 then OctoPrint still would have been running on port 5000. It's difficult to say why you saw it at all unless you decorated your URL with ":5000".