Secure Infrastructure Setup for Small Business

Get Help
1

Hello everyone,

After a year of planning and fantasizing, I'm finally getting ready to convert my entire printing operation to utilize Octoprint. As of right now this is what how I'm planning on setting things up, feel free to chime in with advice anywhere you see fit, would really like to work out every last kink in my plan before I purchase all the hardware required to setup ~50 octoprint devices.

  • For OctoPrint - RasPi4 4GB powered via PoE Hat with network-booted OS

  • For Data Storage - RasPi4 - 8GB running a RAID array for storing OS's (my business sells quite a few different 3D printed products, we currently have about 8GB worth of GCODE that we keep on our "production" SD cards at all times. I really don't want to worry about SD cards burning out because I write them to nearly full, then wipe and replace with updated files way too often. Running a few 2TB hard drives in a RAID array actually ends up saving quite a bit of money on SD cards if you do things to the scale I do)

  • Approximately 4-6 GPU's for Spaghetti Detective (I used to mine Ethereum rather heavily some years back so I have a handful just laying around)

  • All of the Raspberry Pi's will be setup on an isolated network so as to not be directly forward-facing to the world. The isolated network will be bridged to regular networks through the computer that runs Spaghetti Detective.

  • My goal is to be able to control all my printers from a single dashboard, so I plan on utilizing the REST API rather than directly accessing the web dashboard for any Octoprint devices. My initial thought is to either use Apache/NGINX as a proxy server, but Kubernates could be a good option as well. Was really hoping someone might have some feedback about this. The dashboard I plan on building will also incorporate order data that has to be fetched from servers on the web, so it does have to be forward facing, I wouldn't be able to hide it on the isolated network.

  • Now the topic of securely connecting to Octoprint devices... obviously SSL is a must, but what would be the best way to issue proper SSL certificates for a private network? Since I plan on building a handful of custom integrations I don't want to have to worry about unexpected errors popping up due to SSL warnings. I have a fiber-optic connection with 5 static IP Addresses (I host all of my companies development and production servers for our websites using a RasPi cluster), so I'm totally setup to map actual domain names to devices on my network, however what would be the best way to properly issue and validate the certificates if they are on a private network? I've been toying around with the idea of registering a domain name example.io and then using CloudFlare to both issue the certificates and to restrict all remote access to the servers (via CloudFlare firewall settings), but then my question is can I reconfigure my proxy/Kubernates server to restrict access and will the SSL certificates still work? I've never actually ran Apache/NGINX as a proxy, although I have plenty of experience with them in general so any advice would be much appreciated.

(On the topic of using CloudFlare for this stated purpose - If any OctoPrint dev's see this, I almost feel using CloudFlare to protect any remote-access enabled Octoprint device should be a standard recommendation. I'll gladly write up an article and make some auto-configuration scripts if you're interested. Just the ability to immediately block out user agents with high threat scores and add geographic restrictions would really increase the overall security of the device by quite a bit. Anyone who's managed web servers knows how nuts bots are, you don't even need to have a domain name assigned to a server for it to receive 20-30 attempted connections from random countries per day. And CloudFlare offers a free plan that would work for everyone and they're a really cool company so I'm sure you could even talk to them and they might be down to help or even sponsor it, or donate back to Octoprint for the support.)

  • Removing the firmware updater from Octoprint? This is another idea I'm toying around with. Does the risk of ransomware outweigh the benefits from easier firmware updates? Is this even a concern I should have? My thought process is that if someone tries to ransomware me, they're going to try to do something that would lock up the firmware on my printers and I'm guessing potentially alter the bootloader as to prevent any attempts at a "factory reset"

Thank you to anyone who read through my endless paragraphs of text =) you are much appreciated

2

Sounds like a fun idea :slight_smile:

You sound like you have a solid understanding of how to run this stuff, with all the relevant security etc. etc.

I would recommend looking into OctoFarm, before you try and build your own dashboard. As the name implies, it is designed for multiple OctoPrint instances from one control panel. And if it doesn't work exactly as you want I'm sure you could get in touch and work with the guys developing it to work something out.

As for the wider OctoPrint security, we're currently strongly recommending using services such as OctoEverywhere rather than DIY, since we know that it will be secure (to the best of the developer's ability), rather than following a tutorial and it going wrong. Another simple way to configure this that is guaranteed security would be welcome.

Finally, for firmware updater - just don't install the plugin, and make sure that plugin install is not easy. The bundled pluginmanager can also be disabled as well, so it would require access to the Pi to re-enable, for maximum restriction. Firmware is exactly the example I use when people say they've port forwarded with little security... I use the example of installing malicious firmware on the printer and using it to cause damage.

3

Thank you very much for the OctoFarm reference, I hadn't seen that one yet. I may get in touch with the developer, it seems to have most the features I'm looking for and the rest are just fully custom to my needs, such as importing order data from eStores. Realistically though it's insanely easy to make a custom dashboard, really all you need is a SQL table that stores printer IP addresses and other connectivity settings and then it's just a matter of utilizing the REST API to send commands and read the status of printers.

OctoEverywhere was new to me as well, however upon looking into it I couldn't find any good documentation about it. I did a basic analysis of their domain and it appears to be a VPS from DigitalOcean that is utilizing their Cloud Firewall service. I didn't make an account so I didn't test any of the actual user endpoints but I'm willing to bet that's what's happening. If the developer of OctoEverywhere keeps the service free, then I'd say two-thumbs up. But if he ever tries to take it paid or just makes certain features unreasonably cost extra then it would be really easy to setup a service sponsored by Octoprint itself.

What the developer is doing is really the exact same as what I mentioned, just with DigitalOcean instead of CloudFlare. Both CloudFlare and DigitalOcean have API's that allow you to request SSL certificates so I'm betting that upon making an account and setting up a new connection to a printer, the plugin sends a request to a OctoEverywhere API that then requests a new public/private key along with an origin certificate using from the DigitalOcean API. The plugin then saves the keys and configures the Virtual Host/Server Block and Poof! SSL enabled. Origin certificate ensures that only the OctoEverywhere server is allowed to communicate with your Pi while public/private keys provide encryption.

I would say it'd be worth it for the Octoprint dev's to think about adding some of this in as standard features. Here's the real dilemma, currently its way too easy for a noob to accidently setup a publicly available Raspberry Pi that is at risk of being compromised. I could be wrong, but I imagine at the end of the day that it would be a simpler solution simply to have certain authentication features just built into the core of Octoprint. It would be fully possible to implement an asymmetrically encrypted token (or really just a standard Oauth2.0 protocol) that only Octoprint.com's servers know how to decode, similar to utilizing CSRF token's on websites. If the Octoprint device in question was to have SSH disabled then the built-in token would essentially eliminate the risk of any brute-forcing on a forward facing Octoprint device.

Do you know if the firmware updater verifies the hash of a firmware file before downloading and executing it? It may be a no-brainer but I could also see it not being a thing because the printer manufacturers would need to provide official hashes to check against.

I see you're a contributor to Octoprint, if you're intrigued by any of this at all let me know and I can come hang out in the discord and we can chat about it. The startup I run isn't in the position to donate to Octoprint, however I would gladly contribute to it's development.

4

I agree, there's a lot that could be added. It probably won't, since OctoPrint is run by a single developer (Gina), and what we have at the moment is pretty much at capacity, occasionally over. So it would be a prime candidate for someone else to run as a plugin etc. which saves resources for improving the existing core.

I don't think so, because while it can flash from a URL I suspect that most of the time users are uploading their custom built firmware. It doesn't 'check for updates' and download them from the web, it must be a manual 'hey there is a new firmware, let me download it and send it to OctoPrint'.

1 Like
5

When you say "at capacity" do you mean from a labor-contribution standpoint or is that referring to the RasPi's hardware being "at capacity"?

I'm guessing that maintaining backwards compatibility for older versions of RasPi probably hurts the case for adding more features. What a RasPi4 can handle and what a RasPi2/3/Zero can handle are very different.

6

The first, at capacity in terms of the number of hours in the day to get stuff done

7

What @Charlie_Powell is saying is that while I work on this fulltime I still have to sleep and need days off, and with that factored in we are in a 100-120% capacity situation as things are now :wink:

1 Like
8

As someone who's been running a startup for the last couple years, could you please explain this "sleep" thing to me? :joy:

If you are interested in implementing native Oauth 2.0 just let me know and I'll handle it. I've set up more Oauth integrations than I can even begin to count, I'd be glad to help out, mad respect for what you've done out here Gina