OctoPrint plugin that works with The Spaghetti Detective

Showcase
1

I built an OctoPrint plugin to works with The Spaghetti Detective (http://thespaghettidetective.com).

The plugin can be installed using this link: https://github.com/TheSpaghettiDetective/OctoPrint-TheSpaghettiDetective/archive/master.zip

The Spaghetti Detective is a Deep Learning-based failure detection server using nothing but a webcam. Just as OctoPrint itself, The Spaghetti Detective is also open source at https://github.com/TheSpaghettiDetective/TheSpaghettiDetective

3 Likes
2

oooo, very interesting. I see you are listed on the team of TSD. I've been hoping something like this would come along.

I see the server can be run locally: https://github.com/TheSpaghettiDetective/TheSpaghettiDetective

Are you transcoding the webcam feed?

3

Yup I'm part of the team.

The plugin still has some rough edges and it's currently just sending jpeg. We briefly tested it out and it looks like RPi is not powerful enough to transcode raw images to video. Will be happy to hear if there is a way to send webcam feed more efficiently.

4

Can I see previous print attempts? Can I mark them as failed? (eg training)

5

That's neat. Similar to the other plugin Project Kronos Data Collector

@tedder wouldn't this be a good use case for gstreamer since you mentioned it was more server to server? I know you're one of the few that I've seen experimenting with other video options, like the uv4l webrtc stuff.

6

It is a good suggestion to use existing timelapses as training data. I think we can build this function in our next version!

7

Just checked out gstreamer and it looks pretty cool. We previously tested transcoding with ffmpeg which was too CPU-demanding. Will give gstreamer a shot when we get a chance. Any other CPU-efficient transcoding lib you are aware of?

We saw project kronos and will be happy to collaborate with the author.

8

Bonus points if you change the name of this to the FSM Detective.

3 Likes
9

except that was sketchy as hell for about 100 reasons.

Possibly, but occasional jpegs seems good for this, because it doesn't need ongoing streaming.

Remember the Pi has hardware mp4(?) encoding, which is helpful. It was a really bad fit for me because I couldn't shove it into a browser, but it might be a really good fit for y'all.

I always have that sticker on my motorized vehicles. I need a tattoo of it but the Octocat will be first. And maybe the golang gopher..

1 Like
10

Yeah both my RTMPStreamer plugin and YouTubeLive plugin are just using ffmpeg inside a docker container. I agree sending images in this use case is probably perfectly fine. I'm not even sure that my ffmpeg commands are the most efficient or using the GPU h264 encoding. I need to dig deeper into that maybe something like this...?

11

I didn't know Pi has a hardware mp4 encoder. Thank you so much for pointing this out.

Why couldn't you shove it into a browser?

12

I never knew that you can run a docker inside RPi. Will have to steal some code from your plugins. :wink: Thank you in advance.

13

HTML5 doesn't grok mp4 :confused:

14

You should look at a port of my YouTube plugin. Allows for a lot more customization of the ffmpeg command line and is a smaller docker image overall I think.Not sure if the ffmpeg is compiled with hardware encoding support, but best I can tell from reading online, it's the h264_omx option you would want to use probably.

I think you could get this to work with a standard video tag. My MultiCamView view plugin (very undeveloped) is able to stream mp4. The specific video tag stuff....https://github.com/jneilliii/OctoPrint-MultiCamView/blob/83547419973bb0fa41b2e1b816d34b3dfd1cfdf7/octoprint_multicamview/templates/multicamview_tab.jinja2#L10

1 Like
15

Perfect! I'll definitely take a look!

If you have an option to add an extra javascript, I have used https://videojs.com/ and it works pretty nicely with mp4.

2 Likes
16

running this trough few (successful) printes and I find it run awesome (I'm running a local server, and the weights pre-trained file does the job perfectly) ... I was adding some "crap" to the print bed during print to see how detecion works and I was blown away...

the thing I think is missing is "mask areas", on the client side before sending to server plugin could mark some areas as "ignore" (or just paint them "white" so server is ignoring them anyhow)

few samples

/me super happy with it :slight_smile:

1 Like
17

And thank you @arhi for helping us trouble-shooting tricky Docker problems, as well as contributing to the code already!

18

well it's selfish, I had to make it work for myself so :smiley: ..

but really, when someone mentioned (maybe you) while back that this type of plugin would be great I was, I think for the first time, super skeptical with "there's no chance in hell this will work universally" :smiley: .. I know ML is magic and I spent many many many hours playing with nnets some 25-30 years ago but what you guys did here, I'm super impressed ... this is pre-release beta and ... wow :slight_smile:

1 Like
19

"Told you so"...interesting results @arhi

20

what would be great is if @FormerLurker can merge this into octolapse so when octolapse fetches the image it sends it to the TSD server :slight_smile:

1 Like