Problems with Fail2Ban because of missing HAProxy logs

First of all I want to thank you for Octoprint! For me, it's the best way to run a 3D printer. I have it running now for more than a year without a problem (that I couldn't solve on my own). However, now it seems that I need some help...

What is the problem?
I installed Fail2Ban to be safe from DDOS attacks. Out of the box, fail2ban works fine for failed attempts to connect by SSH (tested).

I want Fail2Ban to also target failed http authentication attempts. However, I cannot get HAProxy to log connection attempts in /var/log/haproxy.log so I also cannot configure Fail2Ban correctly.

So, in the first place I don't know how to configure HAProxy correctly to file those logs.

What did you already try to solve it?
I configured HAProxy to perform a http authentication (see my /etc/haproxy/haproxy.cfg below). I also changed the HAProxy log level from debug to info.

I made sure that rsyslog is installed on my OctoPi.

I restarted HAProxy as well as rsyslog with

$ sudo service haproxy restart
$ sudo service rsyslog restart

Then I opened my browser and connected to Octoprint - via LAN and also over the Internet.

I checked the haproxy log (see below) and expected my IP to be logged. However, the log only gives some information about HAProxy's status.

(Of course I also made some configurations in Fail2Ban, but I think this is not the main issue at the moment...)

Config Files
/etc/haproxy/haproxy.cfg

global
        maxconn 4096
        user haproxy
        group haproxy
#        log 127.0.0.1 local1 debug
        log 127.0.0.1 local1 info

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        option redispatch
        option http-server-close
        option forwardfor
        maxconn 2000
        timeout connect 5s
        timeout client  15min
        timeout server  15min

frontend public
        # Config for external access
        acl ValidOctoPrintUser http_auth(OctoPrintUsers)
        http-request auth realm octoprint if !ValidOctoPrintUser

        bind :::80 v4v6
        bind :::443 v4v6 ssl crt /etc/ssl/snakeoil.pem
        option forwardfor except 127.0.0.1
        use_backend webcam if { path_beg /webcam/ }
        default_backend octoprint

backend octoprint
        acl needs_scheme req.hdr_cnt(X-Scheme) eq 0

        reqrep ^([^\ :]*)\ /(.*) \1\ /\2
        reqadd X-Scheme:\ https if needs_scheme { ssl_fc }
        reqadd X-Scheme:\ http if needs_scheme !{ ssl_fc }
        option forwardfor
        server octoprint1 127.0.0.1:5000
        errorfile 503 /etc/haproxy/errors/503-no-octoprint.http

        # Config for external access
        # acl ValidOctoPrintUser http_auth(OctoPrintUsers)
        # http-request auth realm octoprint if !ValidOctoPrintUser

backend webcam
        reqrep ^([^\ :]*)\ /webcam/(.*)     \1\ /\2
        server webcam1  127.0.0.1:8080
        errorfile 503 /etc/haproxy/errors/503-no-webcam.http

        # Config for external access
        # acl ValidOctoPrintUser http_auth(OctoPrintUsers)
        # http-request auth realm octoprint if !ValidOctoPrintUser

# Config for external access
userlist OctoPrintUsers
        group G1
        user XXX password XXX

/etc/rsyslog.d/49-haproxy.conf

# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
if $programname startswith 'haproxy' then /var/log/haproxy.log
&~

Logs
cat /var/log/haproxy.log

Oct 18 01:00:47 octopi haproxy-systemd-wrapper[13725]: haproxy-systemd-wrapper: SIGTERM -> 13730.
Oct 18 01:00:47 octopi haproxy-systemd-wrapper[13725]: haproxy-systemd-wrapper: exit, haproxy RC=0
Oct 18 01:00:47 octopi haproxy-systemd-wrapper[13782]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Oct 18 01:00:47 octopi haproxy-systemd-wrapper[13782]: [WARNING] 290/010047 (13784) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Oct 18 01:21:53 octopi haproxy-systemd-wrapper[13782]: haproxy-systemd-wrapper: SIGTERM -> 13788.
Oct 18 01:21:53 octopi haproxy-systemd-wrapper[13782]: haproxy-systemd-wrapper: exit, haproxy RC=0
Oct 18 01:21:53 octopi haproxy-systemd-wrapper[13920]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Oct 18 01:21:53 octopi haproxy-systemd-wrapper[13920]: [WARNING] 290/012153 (13922) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

Additional information about your setup
OctoPrint version : 1.3.11
OctoPi version : 0.16.0

Please tell me, if I can provide some additional information!

And you read the haproxy documentation for logging, right?

1 Like

I did indeed. Multiple times.

However, it didn't seem to help...

Also I think there are some things configured differently in OctoPi - e.g. the default /etc/rsyslog.d/haproxy.conf (or /etc/rsyslog.d/49-haproxy.conf in OctoPi) looks different.

Specifically the Other Fields section includes information about host which is the IP address of the visitor.

1 Like

Thanks @OutsourcedGuru!

I finally could solve the problem.

Just for the records: (missing) http-request capture lines didn't cause the trouble.

Instead, the (default) content of /etc/rsyslog.d/49-haproxy.conf doesn't seem to be suitable for HAProxy to protocol connection attempts.

I created another file /etc/rsyslog.d/octoprint.conf with the following content (copied from the HAProxy documentation)

# Collect log with UDP
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

# Creating separate log files based on the severity
local2.info /var/log/haproxy.log

In combination with my /etc/haproxy/haproxy.cfg

global
        maxconn 4096
        user haproxy
        group haproxy
        log 127.0.0.1 local1 debug
        log 127.0.0.1:514 local2 info
[...]

and my /etc/fail2ban/jail.local

[haproxy-http-auth]
# # HAProxy by default doesn't log to file you'll need to set it up to forward
# # logs to a syslog server which would then write them to disk.
# # See "haproxy-http-auth" filter for a brief cautionary note when setting
# # maxretry and findtime.
enabled = true
port = 80,8080,443
filter = haproxy-http-auth
logpath  = /var/log/haproxy.log
maxretry = 10
findtime = 900
bantime = 600

and after restarting all services

$ sudo service rsyslog restart
$ sudo service haproxy restart
$ sudo service fail2ban restart

everything seems to work as expected.

Maybe it would be worth to create a wiki entry about this???

One thing is still unclear to me:

What's the purpose of the default /etc/rsyslog.d/49-haproxy.conf?

1 Like

Feel free to post a guide about how to do this in the guides section, that's what it's there for :slight_smile:

1 Like

@foosel yes, I will do that!

But as I am not an expert on this, I wanted the great people in this forum to validate my solution and clear up my question about /etc/rsyslog.d/49-haproxy.conf before I write a guide that maybe does more damage than good...

By the way, once more THANK YOU FOR OCTOPRINT, @foosel!

1 Like

When this gets written up, the link needs to be part of the "how to securely access OctoPrint remotely" threads. It's a nightmare out there outside of a firewall with the scripting attempts.

Thank you for your support so far.

I found out, that the default file /etc/rsyslog.d/49-haproxy.conf is part of the HAProxy package in Debian. So probably nothing that was added to OctoPi on purpose. Therefore I assume it is safe to replace the configuration in this file with the suggested configuration in the HAProxy documentation.

Still it puzzles me, why the package owner of HAProxy created this file...

As promised, I wrote a tutorial on how to setup Fail2Ban. Find it here:

By the way, provided that RSyslog is configured correctly (which I discuss in the link above) logging connections is done with log level info.

The host field in the header that @OutsourcedGuru referred would do something else.

2 Likes