Server Sent Events From SimpleApiPlugin

Development
1

Hello,
I am trying to write a plugin that will provide a streaming response conforming to the ServerSentEvents specification, I am struggling to understand how streaming responses in Flask are supposed to work. After hours of trying different options and searching the internet, I still don't understand how this event could work. What I have roughly is this:

def on_api_get(self, request):
  q = queue.Queue()
  self.queues.append(q)
  def make_generator(q):
    while True:
      yield q.get()
  q.put_nowait("data: {}\n\n".format({ "event": "initial-data", "data": None }))
  resp = Response(make_generator(q), mimetype="text/event-stream")
  res.automatically_set_content_length = False
  return res

and then hit that endpoint via

curl http://octipi.local/api/plugin/octosse -H "X-Api-Key: <api-key>"

The q variable will always get a valid SSE message when I want to trigger the event, and from the Flask documentation that to provide streaming responses can be done with generators and the python documentation seems to indicate that the queue module is the way to make this sort of async-stream work but the above will block the task that is trying to handle the response.

If I change the call to q.get to be get(False) or get(False, 60) or get(True, 60) I will only ever see the queue.Empty exception.

I've also tried to pull out the resp.stream property into a thread and call write on that but that never ends up triggering the response headers to make it to the client.

Any help here on where to look next would be greatly appreciated!

1 Like
2

This almost sounds like recreating the websocket communication that exists in OctoPrint already. Plugins can send custom messages across that websocket connection as well, so maybe backing up and explaining the use case would be helpful.

There are several examples of other web applications and mobile apps that utilize the websocket connection as well, so it's not just OctoPrint's web UI that can use it.

3

Thank you for the suggestion, however there are a couple of problems I am trying to avoid with that interface. First the Websocket API is expressly unstable according to this

The interface documented here is the status quo that might be changed while the interfaces are streamlined for a more general consumption. If you happen to want to develop against it, you should drop me an email to make sure I can give you a heads-up when something changes.

So I would prefer provide an interface that I can control the stability of.

Second, it is not currently an option to use the Application Keys plugin to authorize a client on behalf of a user, this means that a password will be required for any clients and that starts to get messy quickly.

My initial use case is to augment my SmartThings Edge Driver for OctoPrint which currently works by polling the /api/job endpoints but this could be done far more efficiently by having Server Sent Events stream to push updates on. Though I also have other ideas for how to use this in the future.

4

Your quote about websocket could probably be deleted from the docs, it's used by so many applications at this point, it should be considered stable. OctoApp, OctoDash, OctoScreen, OctoFarm (before it went away), FDM Monster (fork of OctoFarm) all utilize the websocket.

What do you mean? In Access Control, as an admin you have the option to create application keys for users.

image
image999×548 30.3 KB

Do you mean from an external interface where the user logins to the OctoPrint UI and clicks the allow button to generate, vs you as admin getting the prompt and being able to click allow?

It's also possible for a user to generate their own API key from the OctoPrint UI.

image
image580×585 21.9 KB

5

Do you mean from an external interface where the user logins to the OctoPrint UI and clicks the allow button to generate, vs you as admin getting the prompt and being able to click allow?

Yes, this is a much more secure option than requiring someone to copy and paste an api key or username & password. But last I checked the websocket connection does not support API keys and can only be used with a username and password in the auth message.

6

Totally possible to do this with the user logging in and clicking allow. Websocket push updates are allowed to login with API key.

The auth message expects the user id of the user to authenticate followed by : and a session key to be obtained from the successful payload of a (passive or active) login via the API.

There is a workflow basically for generating the application key based on user logging in to OctoPrint.

https://docs.octoprint.org/en/master/jsclientlib/socket.html#sec-jsclient-socket-authsample

7

Actually, this is better example of workflow.

https://docs.octoprint.org/en/master/bundledplugins/appkeys.html#workflow

8

Thank you very much for the pointers, I was able to get a session key from the POST /api/login?passive=true&api-key=<api-key> however I am only ever seeing the connected event followed by sock-js heartbeats. I am not sure if that indicates a failure to authenticate or if the fact that the socket doesn't get closed means the auth message was successfully received.

I am oddly not seeing much in the ~.octoprint/logs/octoprint.log, is there some magic to get the WS logs to show up there?

9

NVMD, I was able to figure out what was going wrong. Thanks again for your help!