I run several printers at my place of employment, and one of my main printers is a Creality CR10S Max that I have a Raspberry Pi 4b 4GB connected to running the latest version of Octoprint. This morning I woke up to check in on a 54 hour print that I started on Friday before I left work, and noticed via the webcam stream that the print was terribly blobbed - still printing, but the head was moving horribly slow around curves and the quality was just atrocious. So I jumped into another Chrome tab and tried to open the Octoprint interface and got the error page stating that the server wasn't available... So I SSH in, I verify everything that the error page asks me to through command line - the service is running with no issues from what I can see. I check the log file and see a whole lot of lines from Tornado.Access "GET" for about a 5 minute period of time this morning from 8:12AM through 8:17AM. I think it only stopped when the web server crashed.
A reboot of the service corrected the issue, I am not concerned about the PI as everything seems normal again. What I would like to know is - does this look like a DDOS attack of some kind? The items Tornado.Access appears to be requesting are odd... things like credit cards and passwd, and some gibberish that didn't make any sense to me. Based on this log file I am curious it looks as though anything might have been compromised?[octoprint (1).log|attachment]
Here is the log file I pulled, showing the lead up to and the post-restart conditions.
[LOG REMOVED] (1015.9 KB)
My setup is as follows:
Octoprint 1.4.0
Raspberry Pi 4B 4GB Ram
Creality CR10S Max 3D printer
Running in a corporate environment - static IP internally, has access to internet
Any help with this would be amazing! As I am running this box inside of a corporate network, I have to be very concerned about any kind of network threat that could compromise our site network security. If anyone can recommend any suggestions for further protecting my Octoprint instances here I would love to know about them! Thank you for your assistance in advance!
edit
I also want to mention that I do not have ports open to access this Pi remotely. I have to login to my corporate network through VPN and access through the browser from inside the network. My company has most of our ports blocked through hardware firewalls, Palo Alto hardware and such. Very difficult to get through unless you have access to the internal network. You can see in the attached picture here right where the "attack" started.
Michael Stockwell
I note several entries like:
2020-03-14 20:25:24,121 - backoff - INFO - Backing off __get_dev_settings__(...) for 140.7s HTTPError: 401 Client Error: Unauthorized for url: http://newstream.getanywhere.io/api/dev_settings)
My first guess would be that OctoPrint Anywhere was the cause of your problems.
And then someone starts running a script at this point, testing what they believe to be a variety of vulnerability points or maybe they're just trying to figure out what kind of host software you're running:
2020-03-15 08:11:09,882 - tornado.access - WARNING - 404 GET /CSCOSSLC/config-auth (::ffff:10.10.32.88) 4.09ms
Somewhere during this—probably related—it looks like tornado is confused by something they're doing (multiple instances of this happening):
2020-03-15 08:11:28,096 - tornado.application - ERROR - Uncaught exception GET /robots.txt (::ffff:10.10.32.88)
Traceback (most recent call last):
File "/home/pi/oprint/local/lib/python2.7/site-packages/tornado/web.py", line 1489, in _execute
result = self.prepare()
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/tornado.py", line 257, in prepare
self._fallback(self.request, b"")
File "/home/pi/oprint/local/lib/python2.7/site-packages/octoprint/server/util/tornado.py", line 621, in __call__
response.extend(app_response)
File "/home/pi/oprint/local/lib/python2.7/site-packages/werkzeug/wsgi.py", line 579, in __next__
data = self.file.read(self.buffer_size)
ValueError: I/O operation on closed file
Note especially the POST attempts.
If it were me, I would guess that the OctoPrint Anywhere newstream server could have something to do with this.
So now, I'm going to guess at this point that since you were able to remotely attempt to control this print job that you adjusted your firewall to allow some traffic through?
Looks like a vulnerability scan is running on the internal host with the IP address of 10.10.32.88. Your IT department may scan for vulnerabilities with a tool. You may need to ask them to exclude the address your OctoPrint instance has from being scanned.
The problem is that the internal/private IP address seems to be accessible from the Internet. Presumably the I.T. staff did some port forwarding to allow this to happen.
Or... this is a scripted attack sideways within the private network from a machine which has been hacked (usually less likely).
Not the case. OP says they use a VPN to access the OctoPrint instance. The IP address of accessing device causing the tornado log spam is an RFC1918 address so it is an internal device (or a VPN client.) The question to YOUR IT department should be what lives at the address of 10.10.32.88 and should it be looking for web server vulnerabilities.
1 Like
Thank you both for the great feedback! I did some more digging through that log file today and I think you are both correct! Raymondh, I do believe you are correct that it came from an internal security server, looks to be nothing more than a scan for vulnerabilities. Might explain why the attack lasted around 5 minutes exactly. I can't ask them to exclude it but I have hopes that after this happened they logged it as secure. Our corporate IT group has been working on testing raspberry pi devices recently for deployment at our sites anyway, so if they didn't like the scan report I am CERTAIN I will hear about it lol.
OG - I actually use our corporate VPN client to gain access to the internal network here, so there isn't any need to open ports - though I cannot be certain exactly what ports our network already has exposed. I know it requires a lot of documentation and deviations to be put in place in order to open ports on a new piece of equipment, so I don't feel like its too exposed. Running port scanner software only shows 8 ports open under 10000, so I think we are pretty locked down.
In any case, as I am not using Octoprint Anywhere anyway, I removed that plugin to see if that clears the logs up. You guys are great, I appreciate your time and detailed help!
Thank you and regards,
Michael Stockwell
Glad to hear that a VPN is being used (missed that in my earlier read); thanks. If you're really into security I would agree that dropping OctoPrint Anywhere is a good first step.
Having been an I.T. professional/manager for some time now, I think you ought to show yours what happened as a result of their vulnerability testing. They cost you time, effort and filament. These types of tests should be coordinated not randomly-scheduled, IMHO. If I had scheduled a vulnerability test against an internal machine and spoiled some aseptic pharmaceutical batch process, for example, I'm sure I would have been fired for doing so. "First, do no harm."
