Got hacked :( (not much useful info available)

first need to give some intro..

my local network is split in 3 different independent networks connected to the router, one is wired network with all the servers, workstations etc.. second is wifi where the phones, tablets and stuff are and third is where IoT devices are (tasmota devices, air cleaner, vacuum cleaner, ip cameras and similar cr%$#^p) ... since the appartment is split into many rooms getting single wifi ap to cover it is not really possible so I have ethernet over powerline setup (1gbit/sec, actually works good) going from the router directly to few access points and since I don't really trust wifi to be a stable connection, no matter how many new protocols my ap's support and how highly reviewed they are, I link my octoprints directly on one of those ethernet over powerline devices via ethernet switch to all of them ... everything worked like that for approx 7 years (since I moved to this appt, not sure at what point octoprint(s) are added to the mix)

now, what happened... I was having troubles with the local network past few days, the wifi would stop working in the house, all phones, tablets would not be able to see internet and if you try to reconnect to wifi you could not get ip, the dhcpd was not sending you nothing... restarting bunch of stuff would get it back to working condition but would work for a day and then would stop again... sometimes 2am sometimes afternoon sometimes in the morning, no rule... then I decided to go about it bit more seriously and started rebooting one device at a time to figure out what's causing issue, and it took me few iteration to figure out the one of the octoprint servers was the culprit!!! the problem was that rebooting one of the switches or one of the ethoverpl would also solve it, any interruption of link to octoprint for a second would fix the issue for a while...

ssh to octoprint (rather old version of armbian - 2 years min, maybe older, running on orange pi one and latest 1.3 octoprint) did not show anything wrong, looks like the binaries were patched so that ps, netstat and similar tools will not show anything so I replaced the 1gbit switch with old 100mbps hub and attached my laptop and octoprint to it and waited for it to start killing network again and I seen "ton of traffic" (did not really bother to decrypt but looks like it was trying different exploits on my whole network + outside of my network, on wan) ... copied my config files out of the thing, put back fresh armbian and restored octoprint to get my printer back to speed ...

my octoprint was not exposed to internet (ever), I don't forward ports on my router, my router has pretty good firewall with snort and all the goodies pfsense can offer running on decent i7 machine with 16G ram so enough for everything .. so I doubt the intruder came from the WAN. the two points of entry are ethernet over power lines that's supposed to be encrypted, supposed to be inpenetrable since the devices "paired" etc etc .. and the second possible point of entry was WiFi (it's on that network) where all the AP's are either running latest openwrt or latest ubiquity firmware with unguessable password .. whatever the intruder did, it got somehow on the network, found the old armbian and hacked it ..

Not a very useful story, I know :frowning: but maybe someone finds it useful somehow..

EDIT: one additional info, can't remember if I changed default passwords or not, I always add the ssh key so I auto login to the machine, don't remember if I disabled password login / changed passwords

So, chances are Armbian got hacked, not OctoPrint.

2 Likes

well what got hacked is dedicated octoprint device + either my wifi or my ethernet over powerlines ... they got hacked trough something, wrt dedicated octoprint device it is def armbian, now was it hacked by me being stupid and not changing def. password or trough octoprint (no other service open there) no clue ...

the reason I wrote the post is just for ppl to pay attention, even behind firewall you can't be 100% safe so you have to pay attention... no harm was done in my case and I did not have time nor will to do a proper postmortem (store the sd card and do the proper analysis) so the details are scarce but the general info is still valid

It's also possible that you installed something on the server which came with a Trojan horse. So it reached out to "phone home" and nothing otherwise in your network configuration was to blame.

Another possibility although unlikely is that it's your neighbor in your apartment complex. The power circuits aren't 100% separated in old units which were formerly motels/hotels, for instance. If they share your power and they just happened to plug in the same brand of device that you're using it could come with a discovery/backdoor sort of method of introducing the device; the manufacturer might wrongly assume that this was a good idea.

I also had an experience in which a bored dump truck driver parked outside of my apartment in San Diego was attempting to hack Bluetooth on my macOS laptop. (I actually went out and confronted him.)

it's blank armbian with octoprint, nothing else installed there ever, I don't even use the streamer

it's a building, I tried to run these devices between appartments and they don't work, and I tried accessing network from the same device (I have multiple) without properly pairing it and it does not work ... so "by mistake" there's no way anyone could of done it ... now intentionally, the singnal can probbly be boosted, encryption broken...

What constitutes "ton of traffic" (did not really bother to decrypt but looks like it was trying different exploits on my whole network + outside of my network, on wan)"? You say encrypted, what kind of traffic was it?

iptraf shows tcp traffic, source the opi, target different ports on different ip's on the wan ... I did not read the content of the traffic at all, nor I bother setting up something to record it

I'm using the TPLink Deco9 Mesh and it solved alot of my problems.
Q: How do you know you've been hacked?
Do you have proof?
Why don't you just turn on a combination of devices and wire shark them or using the tools you mentioned..

I think I explained :smiley: but ...

  • there was unknown traffic sourced from a rpi targetng random computers on the lan and wan visible on iptraf on a computer attached to the same 100mbps hub as the rpi ... I recognised some of the connections (or better attempts of connection) as the old m$sql attack from previous decade, others I did not recognise ... there was both udb and tcp traffic but lot more tcp
  • none of that traffic was generated by any of the "ps ax" visible apps on the opi
  • disconnecting opi from the network solved all the network issues

not any more, I had to continue printing so I nuked the sd with latest armbian and restored octoprint config so that opi was back printing in few hours and I didn't care really to store the old sd card image as I have more than one archive of hacked computers waiting many years (some over 20) for postmortem that I never got time to do so I didn't want to bother and wait ages dumping 200G image that I will never look at :smiley:

don't get me wrong this is hugely embarrassing, I used to be security expert, dropped out of loop while back when kids came and other stuff preoccupied my attention ... I posted this as I'm sure

  • many probably don't forget to change the default password (still not sure if I did or not, I think I did as I normally disable password login completely on all my linux boxes but..)
  • many probably use old releases of armbian and raspbian and never upgrade them "as they work" (but are maybe not that safe?)
  • many use similar ethernet over powerline (mine are TENDA with PA3 protocol and 1gbit speed)

I will be re-testing the tenda ethernet over powerline more as that's serious problem if they are the problem as I'm currently using only few but have plans of many more when I move from appt to house this year ... If I manage to figure they are unsafe I will be updating this thread for sure

Humm you can try segment the network to afew vlan. And use firewall rules to guard outbound traffic per vlan to vlan and outbound traffic per vlan to wan :slight_smile:

I already have physically separated networks that have very strict rules between themselves hence nothing from the wifi nor "things" network can see any other, that's why my main lan was not affected at all ...

anyhow, looks like TENDA is unsafe as it seems the firmware automagically changed on this tenda ethernet + wifi + power "hub"

So your wifi network router is the cause of your woes? Talk about irony...

it is ethernet over powerline adapter with embedded wifi (wifi was disabled on it btw) .... no routing done there ... I hate all these network devices that are not open source but could not find ethernet over power lines with open source firmware :frowning: ... (I run pfsense router and openwrt access points so all open source, have also few of ubiquity unify access points but I doubt they are problematic)

Awe well... live long enough and you're going to get hacked at some point.

well .. not the first, nor the last time :smiley: :smiley: :smiley: ... (few years ago some script kiddie shut down all my blogs and wiki and .. darn wordpress was not updated on time and... some years before that someone hacked my PRC made IP cameras that were visible from internet .. now I have zoneminder collecting data from ip cameras and only zoneminder is visible from the world) ... but I really did not expect eth2plc to be vulnerable especially as it does not work trough the meter, I have 2 meters in the appt, they won't pair on the same phase but trough 2 meters, less then 20m of cable, there's something on the meter system that creates a problem (not sure what as modern meters just have a small coil around the wire with resistor to measure current) so pretty weird ...

Once upon a time I owned/operated a datacenter in NORCAL. And then one day Bush41 said or did something that annoyed China and suddenly the hacking attempts tripled overnight. These were the days of Microsoft IIS and I had to write an ISAPI filter a full two days before Microsoft itself offered a fix for their own vulnerability. <_<

In case you're wondering, a hacker could do a GET for something like...

http://domain.com/path/../../../../windows/cmd.exe?type+/path/file

...and the damned webserver would actually run that and return the response. :laugh: I almost forgot how I repaid their curiosity:

redirect to the chargen port

I remember those days, loved the IIS vulnerabilities as it was making our lives easier, I was a sysadmin in those days and running linux/irix/ultrix/solaris so it was great, we were parsing logs every 5 minutes and adding the IP's hitting different IIS vulnerabilities into ban list :smiley: ... I remember we had to add one linux in front of our boxes to route the traffic and push rules there as regular web servers had issues with large routing tables in parallel with serving web :smiley: ... but that was 25 years ago.. today it's all very very very different

install a radius server, and migrate all your wifi to use WPA2 enterprise with peap-mschapv2 authentication.

not sure how secure those Ethernet over power line devices are. i would rather lay cat5e/cat6 instead of using those.

I do have radius, wifi is as secure as it can be on it's own, eth-over-plc ... well, I have a whole bunch of cat7 around the appt but some need happened later on and new cabling is not easy to install... also, the printer is in the "movable tower rack" that only plugs into power wherever I put it so eth-over-plc was superb solution, while it worked :frowning: