Struggling to get reverse proxy working

What is the problem?

Accessing octoprint (on octopi) via an nginx reverse proxy (with auth basic) on a separate machine fails to connect and gives this message in the Chrome developer tools:

packed_libs.js?1fe9f29c:1607 WebSocket connection to 'wss://octopi.######/sockjs/799/r1jfw5fa/websocket' failed

What did you already try to solve it?

Lots and lots of different variants of nginx configuration, after following guides including Reverse proxy configuration examples and various other similar requests for help on the web.

Logs (syslog, dmesg, ... no logs, no support)

Interestingly, the octoprint log shows that the connection is appearing:

2022-02-02 18:09:39,144 - octoprint.server.util.flask - INFO - Passively logging in user myusername from 192.168.3.111
2022-02-02 18:09:39,145 - octoprint.access.users - INFO - Logged in user: myusername
2022-02-02 18:09:41,849 - octoprint.server.util.sockjs - INFO - New connection from client: 192.168.3.111
2022-02-02 18:10:41,978 - octoprint.server.util.sockjs - INFO - New connection from client: 192.168.3.111
2022-02-02 18:11:42,142 - octoprint.server.util.sockjs - INFO - New connection from client: 192.168.3.111
2022-02-02 18:12:42,217 - octoprint.server.util.sockjs - INFO - New connection from client: 192.168.3.111
2022-02-02 18:13:42,344 - octoprint.server.util.sockjs - INFO - New connection from client: 192.168.3.111
2022-02-02 18:14:42,451 - octoprint.server.util.sockjs - INFO - New connection from client: 192.168.3.111
2022-02-02 18:15:42,645 - octoprint.server.util.sockjs - INFO - New connection from client: 192.168.3.111
2022-02-02 18:16:09,050 - octoprint.server.util.sockjs - INFO - Client connection closed: ::ffff:192.168.3.111
2022-02-02 18:16:42,859 - octoprint.server.util.sockjs - INFO - New connection from client: 192.168.3.111
2022-02-02 18:17:30,844 - octoprint.server.util.sockjs - INFO - Client connection closed: ::ffff:192.168.3.111

However, no response is ever received by the browser.

Additional information about your network (Hardware you are trying to connect to, hardware you are trying to connect from, router, access point, used operating systems, ...)

Network consists of a server (Ubuntu 20.04 LTS) & the raspberry pi with octopi.

The server has various docker containers, including one running dnsmasq and several running nginx reverse proxies.

The current nginx configuration for the reverse proxy for octoprint looks like this (note that the commented out bits are other things I've tried along the way):

server {
    listen                  80;
    server_name             octopi.########;
    root                    /usr/share/nginx/html;
    location /static/ {
                proxy_pass http://octopi.lan/static/;
                proxy_set_header Host $http_host;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection “upgrade”;
    }
    location / {
                proxy_pass http://octopi.lan/;
                proxy_set_header Host $http_host;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_http_version 1.1;

                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Scheme $scheme;

#               proxy_set_header X-Scheme $scheme;
#               # Also tried: proxy_set_header X-Scheme https;
#               add_header       X-Served-By $host;
#               proxy_set_header Host $host;
#               proxy_set_header X-Forwarded-Scheme $scheme;
#               proxy_set_header X-Forwarded-For    $remote_addr;

                client_max_body_size 0;

                auth_basic "Restricted Content";
                auth_basic_user_file /etc/nginx/.htpasswd;
    }
    location /.well-known {
    }
}

All of this sits in a docker-compose set including letsencrypt-proxy-companion to make it all be https nicely. The /static location was added because I had errors in the javascript console about permission denied accessing manifest.json.

I currently have no reverseProxy section in config.yaml, but previously I have also tried adding this:

reverseProxy:
  prefixHeader: X-Script-Name
  schemeHeader: X-Scheme
  hostHeader: X-Forwarded-Host

For reference, this nginx configuration (for nextcloud) works perfectly, as do several others (some of which have basic auth):

server {
    listen                  80;
    server_name             nextcloud.###########;
    root                    /usr/share/nginx/html;
        client_max_body_size    20G;
    location / {
                proxy_max_temp_file_size 0;
                proxy_pass http://##########:8888/;
    }
    location /.well-known {
    }
}

Are there any more details that would be useful to help diagnose this problem?

Thanks for any help anyone can offer.

I'm mostly stumped as the examples you referred to as provided should Just Work(TM), but one thing I do notice is that your whole config refers to port 80 and thus likely http, whereas the (failing) websocket connection and its fallbacks are being attempted to be established via port 443 (protocol is wss, not ws). That /static sub path is also not needed.

1 Like

It looks like it's missing proxy_http_version 1.1.

It says this on the examples page:

:warning: Warning
Be extra sure to include proxy_http_version 1.1 or the web socket connection will fail.

Nope, that's in there.

I added that as there was a message in the javascript console saying manifest.json couldn't be accessed. It's all behind basic auth and the browser didn't seem to be using the credentials to access manifest.json. Given there's nothing secret in there, I thought it was easiest to just allow that without basic auth.

I'm guessing that might be something to do with the letsencrypt-proxy-companion thing. If I'm honest, it's not something I fully understand, but it "Just Works™" for all the other (about 10) services hosted on the PC. Octopi is the only one that doesn't work. There's a top-level one that listens on 443 (and 80, although it just responds to any http requests with "go away and use https" response) and checks the requested server name. It then picks the relevant service for that service name (all services are listening on 80 but not exposing that port publicly) to forward traffic to.

There are also two extra services, "nginx-gen" and "letsencrypt-nginx-proxy-companion". I don't really know what the former does, but the latter sorts out the https certificates. All of this was lifted from a docker hub thing called "jrcs/letsencrypt-nginx-proxy-companion".

Here is a subset of the docker-compose file:

nginx:
  image: nginx
  container_name: nginx
  ports:
    - "443:443"
    - "80:80"
  volumes:
    - "/zfs.sdd/website/transient/conf.d:/etc/nginx/conf.d"
    - "/zfs.sdd/website/transient/vhost.d:/etc/nginx/vhost.d"
    - "/zfs.mount/website:/usr/share/nginx/html"
    - "/zfs.sdd/website/certs:/etc/nginx/certs:ro"

nginx-gen:
  image: jwilder/docker-gen
  container_name: nginx-gen
  volumes:
    - "/var/run/docker.sock:/tmp/docker.sock:ro"
    - "./proxy/templates/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro"
  volumes_from:
    - nginx
  entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

letsencrypt-nginx-proxy-companion:
  image: jrcs/letsencrypt-nginx-proxy-companion
  container_name: letsencrypt-nginx-proxy-companion
  volumes_from:
    - nginx
  volumes:
    - "/var/run/docker.sock:/var/run/docker.sock:ro"
    - "/zfs.sdd/website/certs:/etc/nginx/certs:rw"
  environment:
    - NGINX_DOCKER_GEN_CONTAINER=nginx-gen
    - NGINX_PROXY_CONTAINER=nginx

int_web:
  image: nginx
  container_name: int_web
  volumes_from:
    - nginx
  volumes:
    - "./####.###/conf.d/:/etc/nginx/conf.d"
    - ./htpasswd:/etc/nginx/.htpasswd
  environment:
    - VIRTUAL_HOST=####.###
    - LETSENCRYPT_HOST=####.###
    - LETSENCRYPT_EMAIL=letsencrypt@####.###

rss:
  image: nginx
  container_name: rss
  volumes_from:
    - nginx
  volumes:
    - "./rss.####.###/conf.d/:/etc/nginx/conf.d"
    - ./htpasswd:/etc/nginx/.htpasswd
  environment:
    - VIRTUAL_HOST=rss.####.###
    - LETSENCRYPT_HOST=rss.####.###
    - LETSENCRYPT_EMAIL=letsencrypt@####.###

# [snip]

octopi:
  image: nginx
  container_name: octopi
  volumes_from:
    - nginx
  volumes:
    - "./octopi.####.###/conf.d/:/etc/nginx/conf.d"
    - ./htpasswd:/etc/nginx/.htpasswd
  environment:
    - VIRTUAL_HOST=octopi.####.###
    - LETSENCRYPT_HOST=octopi.####.###
    - LETSENCRYPT_EMAIL=letsencrypt@####.###

As another example, rss.####.###/conf.d/rss.####.###.conf looks like this:

server {
    listen                  80;
    server_name             rss.####.###;
    root                    /usr/share/nginx/html;
    location / {
                proxy_pass http://myserver.lan:8124/;
                proxy_set_header Host $http_host;
                auth_basic "Restricted Content";
                auth_basic_user_file /etc/nginx/.htpasswd;
    }
    location /.well-known {
    }
}

If I connect to https://rss.####.###/ with the web browser, it handles all the https stuff and forwards (over http) to a local service listening (http) on port 8124 - the traffic going to that service is not encrypted.

I suspect you're going to say I've made this all too complicated, but getting letsencrypt up and running took very little time using letsencrypt-proxy-companion and following the examples given. Every other service I've tried to set up has just worked. I suspect the issue is "auth basic", but I don't know what to do to fix it. I'd prefer to have the extra security step in place to make sure no-one can get near my octoprint instance.

I've so far not seen any kind of issues with basic auth setups and OctoPrint either, and in fact OctoPrint even supports logging you in through basic auth (if configured accordingly).

I would suggest to take it out of the equation as a quick test though while I agree that this is a good idea to have in general.

With hindsight, I guess I should have tested that earlier: good idea! Unfortunately it didn't make any difference.

To rule out any caching issues, I accessed octoprint via a new private browser window. It went to the octoprint log-in screen, I typed in my username and password and again got stuck at the "socket connection failed" message.

So it can't be anything to do with auth basic, but something else is stopping the wss stuff getting through.

I tried to check the top nginx instance (that looks at the host name and forwards the traffic accordingly); the config for that definitely includes proxy_http_version 1.1; so I don't think that's messing it up.

I'm stumped as to what to try next :slightly_frowning_face:

In case it helps, I dug out the configuration file for the nginx instance that differentiates between the different hostnames. It's generated from a template by one of the docker containers. I don't really understand much of what it's doing. Here's are the relevant bits:

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 80;
    access_log /var/log/nginx/access.log vhost;
    return 503;
}
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 443 ssl http2;
    access_log /var/log/nginx/access.log vhost;
    return 503;
    ssl_certificate /etc/nginx/certs/default.crt;
    ssl_certificate_key /etc/nginx/certs/default.key;
}
[snip]
upstream octopi.####.### {
            # a7d565d8cc08_octopi
            server 172.17.0.19:80;
}
server {
    server_name octopi.####.###;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    return 301 https://$host$request_uri;
}
server {
    server_name octopi.####.###;
    listen 443 ssl http2 ;
    access_log /var/log/nginx/access.log vhost;
        client_max_body_size 10G;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
    ssl_prefer_server_ciphers on;
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:50m;
    ssl_certificate /etc/nginx/certs/octopi.####.###.crt;
    ssl_certificate_key /etc/nginx/certs/octopi.####.###.key;
    ssl_dhparam /etc/nginx/certs/octopi.####.###.dhparam.pem;
    add_header Strict-Transport-Security "max-age=31536000";
    include /etc/nginx/vhost.d/default;
    location / {
        proxy_pass http://octopi.####.###;
    }
}
[snip]

If I'm understanding the set-up right, it seems like there's an nginx server that listens to the https traffic coming into the computer (this is the one with the configuration I've just posted). It sorts out all the SSL stuff and then forwards the request as http to a second nginx server (the one I posted the configuration for earlier), which listens on port 80 only and forwards the traffic onto octoprint.

The only thing I can think of is that one of these reverse proxies is blocking the wss connection (or possibly only the wss reply given that it was visible in the log on the octopi?). However, both seem to have proxy_http_version 1.1; in the configuration file.

Edit

In case it helps, this is the template (nginx.tmpl) that generates the above config file:

{{ define "upstream" }}
    {{ if .Address }}
        {{/* If we got the containers from swarm and this container's port is published to host, use host IP:PORT */}}
        {{ if and .Container.Node.ID .Address.HostPort }}
            # {{ .Container.Node.Name }}/{{ .Container.Name }}
            server {{ .Container.Node.Address.IP }}:{{ .Address.HostPort }};
        {{/* If there is no swarm node or the port is not published on host, use container's IP:PORT */}}
        {{ else }}
            # {{ .Container.Name }}
            server {{ .Address.IP }}:{{ .Address.Port }};
        {{ end }}
    {{ else }}
        # {{ .Container.Name }}
        server {{ .Container.IP }} down;
    {{ end }}
{{ end }}

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}

# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}

gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';

access_log off;

{{ if (exists "/etc/nginx/proxy.conf") }}
include /etc/nginx/proxy.conf;
{{ else }}
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
{{ end }}

server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 80;
    access_log /var/log/nginx/access.log vhost;
    return 503;
}

{{ if (and (exists "/etc/nginx/certs/default.crt") (exists "/etc/nginx/certs/default.key")) }}
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 443 ssl http2;
    access_log /var/log/nginx/access.log vhost;
    return 503;

    ssl_certificate /etc/nginx/certs/default.crt;
    ssl_certificate_key /etc/nginx/certs/default.key;
}
{{ end }}

{{ range $host, $containers := groupByMulti $ "Env.VIRTUAL_HOST" "," }}

upstream {{ $host }} {
{{ range $container := $containers }}
    {{ $addrLen := len $container.Addresses }}
    {{/* If only 1 port exposed, use that */}}
    {{ if eq $addrLen 1 }}
        {{ $address := index $container.Addresses 0 }}
        {{ template "upstream" (dict "Container" $container "Address" $address) }}
    {{/* If more than one port exposed, use the one matching VIRTUAL_PORT env var, falling back to standard web port 80 */}}
    {{ else }}
        {{ $port := coalesce $container.Env.VIRTUAL_PORT "80" }}
        {{ $address := where $container.Addresses "Port" $port | first }}
        {{ template "upstream" (dict "Container" $container "Address" $address) }}
    {{ end }}
{{ end }}
}

{{ $default_host := or ($.Env.DEFAULT_HOST) "" }}
{{ $default_server := index (dict $host "" $default_host "default_server") $host }}

{{/* Get the VIRTUAL_PROTO defined by containers w/ the same vhost, falling back to "http" */}}
{{ $proto := or (first (groupByKeys $containers "Env.VIRTUAL_PROTO")) "http" }}

{{/* Get the first cert name defined by containers w/ the same vhost */}}
{{ $certName := (first (groupByKeys $containers "Env.CERT_NAME")) }}

{{/* Get the best matching cert  by name for the vhost. */}}
{{ $vhostCert := (closest (dir "/etc/nginx/certs") (printf "%s.crt" $host))}}

{{/* vhostCert is actually a filename so remove any suffixes since they are added later */}}
{{ $vhostCert := replace $vhostCert ".crt" "" -1 }}
{{ $vhostCert := replace $vhostCert ".key" "" -1 }}

{{/* Use the cert specifid on the container or fallback to the best vhost match */}}
{{ $cert := (coalesce $certName $vhostCert) }}

{{ if (and (ne $cert "") (exists (printf "/etc/nginx/certs/%s.crt" $cert)) (exists (printf "/etc/nginx/certs/%s.key" $cert))) }}

server {
    server_name {{ $host }};
    listen 80 {{ $default_server }};
    access_log /var/log/nginx/access.log vhost;
    return 301 https://$host$request_uri;
}

server {
    server_name {{ $host }};
    listen 443 ssl http2 {{ $default_server }};
    access_log /var/log/nginx/access.log vhost;
	client_max_body_size 10G;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;

    ssl_prefer_server_ciphers on;
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:50m;

    ssl_certificate /etc/nginx/certs/{{ (printf "%s.crt" $cert) }};
    ssl_certificate_key /etc/nginx/certs/{{ (printf "%s.key" $cert) }};

    {{ if (exists (printf "/etc/nginx/certs/%s.dhparam.pem" $cert)) }}
    ssl_dhparam {{ printf "/etc/nginx/certs/%s.dhparam.pem" $cert }};
    {{ end }}

    add_header Strict-Transport-Security "max-age=31536000";

    {{ if (exists (printf "/etc/nginx/vhost.d/%s" $host)) }}
    include {{ printf "/etc/nginx/vhost.d/%s" $host }};
    {{ else if (exists "/etc/nginx/vhost.d/default") }}
    include /etc/nginx/vhost.d/default;
    {{ end }}

    location / {
        proxy_pass {{ trim $proto }}://{{ trim $host }};
        {{ if (exists (printf "/etc/nginx/htpasswd/%s" $host)) }}
        auth_basic  "Restricted {{ $host }}";
        auth_basic_user_file    {{ (printf "/etc/nginx/htpasswd/%s" $host) }};
        {{ end }}
                {{ if (exists (printf "/etc/nginx/vhost.d/%s_location" $host)) }}
                include {{ printf "/etc/nginx/vhost.d/%s_location" $host}};
                {{ else if (exists "/etc/nginx/vhost.d/default_location") }}
                include /etc/nginx/vhost.d/default_location;
                {{ end }}
    }
}
{{ else }}

server {
    server_name {{ $host }};
    listen 80 {{ $default_server }};
    access_log /var/log/nginx/access.log vhost;

    {{ if (exists (printf "/etc/nginx/vhost.d/%s" $host)) }}
    include {{ printf "/etc/nginx/vhost.d/%s" $host }};
    {{ else if (exists "/etc/nginx/vhost.d/default") }}
    include /etc/nginx/vhost.d/default;
    {{ end }}

    location / {
        proxy_pass {{ trim $proto }}://{{ trim $host }};
        {{ if (exists (printf "/etc/nginx/htpasswd/%s" $host)) }}
        auth_basic  "Restricted {{ $host }}";
        auth_basic_user_file    {{ (printf "/etc/nginx/htpasswd/%s" $host) }};
        {{ end }}
                {{ if (exists (printf "/etc/nginx/vhost.d/%s_location" $host)) }}
                include {{ printf "/etc/nginx/vhost.d/%s_location" $host}};
                {{ else if (exists "/etc/nginx/vhost.d/default_location") }}
                include /etc/nginx/vhost.d/default_location;
                {{ end }}
    }
}

{{ if (and (exists "/etc/nginx/certs/default.crt") (exists "/etc/nginx/certs/default.key")) }}
server {
    server_name {{ $host }};
    listen 443 ssl http2 {{ $default_server }};
    access_log /var/log/nginx/access.log vhost;
    return 503;

    ssl_certificate /etc/nginx/certs/default.crt;
    ssl_certificate_key /etc/nginx/certs/default.key;
}
{{ end }}

{{ end }}
{{ end }}

The plot thickens...

Testing the "inner" nginx

It occurred to me that it is possible to split the two nginxs out (the one that does all the https stuff and the one that passes everything over to octoprint). In the docker-compose configuration for the one that passes everything over to octoprint, I added:

ports:
  - "8641:80"

which means that the internal port 80 is available to other devices on the same network. I could then connect to http://myserver.lan:8641 and octoprint loaded correctly. That result on its own tells me that the "inner" nginx is correctly configured, implying that the problem is with the (autogenerated) outer one that does the letsencrypt stuff.

Testing the "outer" nginx

I then did another test, where I added some very nasty hacks into the template that generates the outer configuration file. The default server configuration looks like this:

{{ define "upstream" }}
    {{ if .Address }}
        {{/* If we got the containers from swarm and this container's port is published to host, use host IP:PORT */}}
        {{ if and .Container.Node.ID .Address.HostPort }}
            # {{ .Container.Node.Name }}/{{ .Container.Name }}
            server {{ .Container.Node.Address.IP }}:{{ .Address.HostPort }};
        {{/* If there is no swarm node or the port is not published on host, use container's IP:PORT */}}
        {{ else }}
            # {{ .Container.Name }}
            server {{ .Address.IP }}:{{ .Address.Port }};
        {{ end }}
    {{ else }}
        # {{ .Container.Name }}
        server {{ .Container.IP }} down;
    {{ end }}
{{ end }}

This figures out the address of the relevant container and forwards traffic to the right place. I added in an if/else block at the top, thus:

{{ if eq .Container.Name "octopi" }}
server octopi.lan;
{{ else }}

(with matching {{ end }} at the end).

That changes it so the outer nginx configuration forwards directly to the octopi rather than going through the inner nginx like all the other services.

Incredibly, it works (albeit without basic auth as that's set up on the inner nginx)! That result on its own tells me that the outer nginx is configured correctly, implying it's an issue with the inner nginx, but I've just shown in the first test that the inner one is configured correctly.

What does that all mean?!?!?

I can only assume that the problem is happening somewhere in between outer nginx and inner nginx. Since there's nothing to configure on the connection between the two, I'm at a bit of a loss as to what to do about it.

For now I guess I'll add a lot more hacky stuff into the template file so I can get it working, but I'd love to know how to sort this out properly.

Has anyone got any ideas?

I had an issue

enabling CORS under Settings > API made it work for me.

spend hours on this, hope it helps

2 Likes

I had an issue

enabling CORS under Settings > API made it work for me.

spend hours on this, hope it helps

I have been trying to solve this problem on and off for months. This finally was the solution I'd been looking for. Thank you!

Signed up to thank for this!