TL;DR

FreeBSD: How to block port scanners from enumerating open ports on your server, by using fail2ban and an ASCII representation of pf logs.

Preface

I use fail2ban to keep away attackers and bots alike that attempt to scan my websites or brute force my mailboxes. Fail2ban works by scanning log files for specific patterns and keeping a count of matches per IP, and allows the systems administrator to define what to do when that count exceeds a defined threshold.

The patterns are indicative of malicious activity, such as attempting to guess a mailbox password, or attempt to scan a web site space for vulnerabilities.

The action to perform is most of the time to block the offending IP address via the machine firewall, but fail2ban supports any mechanism that you can conceive, as long as it can be enacted by a UNIX command.

PF and its logs

On my FreeBSD server I use the excellent pf packet filter to policy incoming traffic and to perform traffic normalization.

The PF logging mechanism is very UNIX-y, as it provides a virtual network interface (pflog0) onto which the initial bytes of packets blocked by a rule that has the log specifier are forwarded, so that real-time block logs can be inspected via a simple:

# tcpdump -eni pflog0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 262144 bytes
01:48:13.748353 rule 1/0(match): block in on vtnet0: 121.224.77.46.41854 > 46.38.233.77.6379: Flags [S], seq 1929621329, win 29200, options [mss 1460,sackOK,TS val 840989709 ecr 0,nop,wscale 7], length 0
01:48:15.726215 rule 1/0(match): block in on vtnet0: 192.241.235.20.37422 > 46.38.233.77.5632: UDP, length 3
01:48:17.993439 rule 1/0(match): block in on vtnet0: 145.239.244.34.54154 > 46.38.233.77.1024: Flags [S], seq 3365362952, win 1024, length 0
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

These logs can be saved by pflogd into a pcap format file in /var/log/pflog, that can be used for async troubleshooting and inspection, as well using tcpdump or anything that can parse pcap files (such as wireshark).

Limits of binary logs

I had already configured fail2ban to parse postfix, dovecot and nginx logs, so that if you try to brute an SMTP or IMAP passwd on my box or you try to run something like nikto against my web site you’ll soon be banned by fail2ban and your incoming connections will be dropped by pf.

However I could not ask fail2ban to read the binary pflog produced by pflogd, as fail2ban is regex-based and only understands text input.

Python to the rescue

I thought of a software that would:

• Start an async loop
• Run tcpdump and attach to its stdout and stderr
• Write the stdout and stderr to a file
• Trap a HUP and USR1 signal and re-open the file, to aid log rotation

Can I haz it?

Sure thing! Head over to github and check out pfasciilogd and the supporting fail2ban configuration.

I hope you find this useful.

Have fun!

Preface

In 2023, I still run my own mailserver. Yes, because I like to keep control of (at least part) my own digital life, and I enjoy having multiple domain names on which I have stuff on. However, I was paying 30€/month to AWS to get in exchange 2 cores, 2GiBs of RAM and 40G of disk, barely sufficient to run IMAP+SMTP+MySQL+Clamd, let alone any form of spam protection or full-text search on email bodies.

So, I was paying a lot of money to run a shitty service, and I even though about shutting everything off and move my mail and my web sites onto some form of fully hosted service.

I still want to do it

Say what, to host four domains with just some email redirects plus the web sites I run, I would have spent more I was paying to also cripple me to some service vendor and their politics.

So, I wanted to run FreeBSD and I started scouting on the ISPs page until I decided to review Hetzner and netcup, that both offer aggressive pricing and a old fashioned VPS and little more.

Settling on a vendor

Eventually, I settled on a netcup VPS 1000 that gives me, for 1/3 of the price I was paying to AWS, 4 times the resources: 6 cores, 8GiB of RAM, 160GiB of RAID10 SSD and an uncrippled, completely totally free FreeBSD installation.

However, the base image that Netcup provides has some limitations:

• It runs on UFS
• It is lacking a swap partition
• It has no encryption

Making a plan

As I was already into the configuration stage and I didn’t want to restart from scratch (this is an old-fashioned server, manually managed, no automation) I decided to:

• Spin up temporary servers on hetzner to experiment
• Peruse for the incantation required to have a full disk encryption bootable machine
• Copy over the / from the netcup server to hetzner and see whether it boots
• Rinse and repeat
• Once the incantation is stable:
  • Boot a hetzner target server to temporarily hold all the data
  • Reboot the netcup source server from a CD so to rsync over all the data to hetzner
  • Scratch the netcup server disk and recreate all the partitions and filesystems the way I like
  • Rsync all data back from hetzner to netcup and reboot

Executing it

Turns out, it actually works. I started using the FreeBSD installation CD, to then realise I didn’t need the installer at all, because I already had a live system I was migrating, so I ended up using mfsbsd to both spin up the target server, and as well to boot the source server when it was time to copy everything back and forth.

Starting from this freebsd forum thread and this wiki page for zfs boot I ended up cooking the following incantation:

Reboot from ramdisk and copy over the data to the temp server

This configures the network, updates rsync to the latest version, mounts the current filesystem in /mnt and rsyncs everything over to a temporary storage location

ifconfig vtnet0 inet6 2a03:4000:2:33c::42 prefixlen 64
route -6 add default fe80::1%vtnet0
echo 'nameserver 2a03:4000:0:1::e1e6' > /etc/resolv.conf

pkg install rsync
pkg upgrade libiconv

mount /dev/vtbd0p2 /mnt

cd /mnt
rsync --archive --recursive --times --executability --hard-links \
  --links --perms --compress --exclude .sujournal --exclude .swapfile \
  --exclude .snap --exclude 'dev/*' --exclude 'srv/www/*/dev/*' \
  . root@m17.openssl.it:/mnt

Create the partitions

Here we create a boot partition holding the gptboot executable, whose responsibility is to load and execute the freebsd loader from the clear text /boot partition.

Then we create a swap partition and eventually a zfs partition that will contain our ZFS pool.

gpart destroy -F vtbd0
gpart create -s GPT vtbd0

gpart add -s 472 -t freebsd-boot vtbd0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 vtbd0

gpart add -s 1G -t freebsd-ufs -l boot vtbd0
gpart set -a bootme -i 2 vtbd0

gpart add -s 2G -t freebsd-swap -l swap vtbd0
gpart add -t freebsd-zfs -l root vtbd0

Create /boot and the encrypted root device

Here we create a UFS filesystem for the unencrypted /boot partition that’ll hold the kernel and loader, and part of the encryption key used to encrypt the root. That key alone is not sufficient to gain access to the filesystem, as also an additional passphrase is needed.

newfs -O 2 -U -m 8 -o space /dev/vtbd0p2
mkdir /tmp/ufsboot
mount /dev/vtbd0p2 /tmp/ufsboot
mkdir -p /tmp/ufsboot/boot/geli
dd if=/dev/random of=/tmp/ufsboot/boot/geli/vtbd0p4.key bs=64 count=1

geli init -e AES-XTS -l 256 -s 4096 -bd -K /tmp/ufsboot/boot/geli/vtbd0p4.key /dev/vtbd0p4
cp /var/backups/vtbd0p4.eli /tmp/ufsboot/boot/geli
geli attach -k /tmp/ufsboot/boot/geli/vtbd0p4.key /dev/vtbd0p4

Create ZFS pool

This is my layout, that I mostly use to limit executability of paths that should not be executable, and also for ease of snapshotting separate parts of the filesystem that need different retention strategies

zpool create -R /mnt -O canmount=off -O mountpoint=none -O atime=off -O compression=lz4 tank /dev/vtbd0p4.eli
zfs create -o mountpoint=/ tank/ROOT
zfs create -o mountpoint=/tmp  -o exec=off     -o setuid=off  tank/tmp
zfs create -o canmount=off -o mountpoint=/usr                 tank/usr
zfs create                                     -o setuid=off  tank/usr/ports
zfs create -o canmount=off -o mountpoint=/var                 tank/var
zfs create                     -o exec=off     -o setuid=off  tank/var/log
zfs create -o atime=on         -o exec=off     -o setuid=off  tank/var/spool
zfs create                     -o exec=off     -o setuid=off  tank/var/tmp
zfs create -o canmount=off -o mountpoint=/srv                 tank/srv
zfs create                     -o exec=off     -o setuid=off  tank/srv/mail
zfs create                     -o exec=off     -o setuid=off  tank/srv/www

Eventually, mount the unencrypted UFS boot partition below the ZFS fs hierarchy,

umount /dev/vtbd0p2
mkdir /mnt/ufsboot
mount /dev/vtbd0p2 /mnt/ufsboot

Copy everything back!

Now it’s time to get back the stuff from the temporary location it was placed to, and write it onto the new shiny ZFS pool on the GELI-encrypted root:

rsync --archive --recursive --times --executability --hard-links \
  --links --perms --compress root@m17.openssl.it:/mnt/ /mnt

mv /mnt/boot/* /mnt/ufsboot/boot
rm -rf /mnt/boot
ln -s ufsboot/boot /mnt

We use a symlink to point /boot to /ufsboot/boot, so the system will behave as if /boot was a normal directory in /. It’s required to keep a /boot subdir in the boot partition because plenty of loader code depends on hardcoded /boot paths.

What’s left

/etc/fstab, with encrypted swap of course:

/dev/vtbd0p2 /ufsboot ufs rw 0 1
/dev/vtbd0p3.eli none swap sw,ealgo=AES-XTS,keylen=128,sectorsize=4096 0 0

/boot/loader.conf.d/geli.conf:

geom_eli_load="YES"
geli_vtbd0p4_keyfile0_load="YES"
geli_vtbd0p4_keyfile0_type="vtbd0p4:geli_keyfile0"
geli_vtbd0p4_keyfile0_name="/boot/geli/vtbd0p4.key"
zfs_load="YES"
vfs.root.mountfrom="zfs:tank/ROOT"

/etc/rc.conf:

zfs_enable="YES"

Did it blend?

Yes of course it did! And it’s happily working since :-)

 03:44:10 root@m42:/srv/www/sindro.me/staging
 # uname -a
FreeBSD m42.openssl.it 13.2-RELEASE-p2 FreeBSD 13.2-RELEASE-p2 GENERIC amd64

 03:44:13 root@m42:/srv/www/sindro.me/staging
 # df -hT
Filesystem      Type      Size    Used   Avail Capacity  Mounted on
tank/ROOT       zfs       140G    6.7G    134G     5%    /
devfs           devfs     1.0K    1.0K      0B   100%    /dev
/dev/vtbd0p2    ufs       992M    189M    723M    21%    /ufsboot
tank/var/spool  zfs       134G    1.1M    134G     0%    /var/spool
tank/tmp        zfs       134G    220K    134G     0%    /tmp
tank/srv/mail   zfs       138G    4.8G    134G     3%    /srv/mail
tank/srv/www    zfs       136G    2.1G    134G     2%    /srv/www
tank/var/log    zfs       134G     13M    134G     0%    /var/log
tank/var/tmp    zfs       134G    224K    134G     0%    /var/tmp
tank/usr/ports  zfs       136G    2.6G    134G     2%    /usr/ports
/dev            nullfs    1.0K    1.0K      0B   100%    /srv/www/admin.openssl.it/dev
/dev            nullfs    1.0K    1.0K      0B   100%    /srv/www/mail.openssl.it/dev
/dev            nullfs    1.0K    1.0K      0B   100%    /srv/www/nhaima.org/dev
/dev            nullfs    1.0K    1.0K      0B   100%    /srv/www/spadaspa.it/dev
tank/usr/src    zfs       134G    773M    134G     1%    /usr/src
tank/usr/obj    zfs       134G     96K    134G     0%    /usr/obj

Il vero sistemista e’ un po’ come il meccanico di una volta, quello che se gli portavi la macchina per rifare la convergenza e quando arrivavi sentiva che il minimo non andava bene, ti faceva la convergenza, e giustamente la pagavi, ma poi ti sistemava anche il minimo e non ti chiedeva nulla, lo faceva perche’ non sopportava di sentire una macchina che non era a punto come si deve.

Era quello che da ogni minimo e impercettibile rumore indovinava subito qualsiasi problema, anche quello di cui il cliente non si era ancora accorto.

Era quello che dopo cena a casa con la famiglia, tornava in officina, dove potevi vedere le luci accese fino a notte tarda, perche’ stava lavorando al “suo” gioiello, una qualche macchina semi d’epoca recuperata chissa’ dove che con passione piano piano sistemava fino a farla tornare nuova.

Ecco, il sistemista e’ come quel meccanico, e le sue auto sono i server.

Fonte: Veteran Unix Admins

English version

The real sysadmin is like the old-fashioned car mechanic, the one you brought your car to adjust the wheels’ convergence and when you got into his garage he heard also your engline while idling didn’t have the right RPM. He then fixed the wheels’ convergence and you paid him for it, but he also fixed the engine idling RPM without asking you nothing - he did it because he couldn’t stand a car that was not set up properly.

He is the one that from every tiny and imperceptible noise immediately guessed every car problem, even those the customer did not yet realize.

He is the one that after dinner with family, he went back to his garage, where you could see the lights on until late at night, because he was working at “his” jewel, some old vintage car found who knows where that he was slowly and passionately rejuvenating until it became like new.

The real sysadmin is like that mechanic, and his cars are servers.

In its own words:

Sources: 55179.13.c, 55471.c

Source code differences between two consecutive versions of the Security.framework, a MacOS/iOS component. The seemingly innocuous extra goto fail; is the cause of a severe security flaw in most Apple products.

Because I have been busy doing this: