Kicksecure on Matty Spangler

Gitea with Podman Quadlets on Kicksecure

Tue, 10 Mar 2026 00:00:00 +0000

This article is part of my Self-Hosting with Kicksecure series.

1. Introduction

After moving away from Kubernetes, I wanted something simpler but still declarative. Quadlets let me define containers as systemd units—a format I already know from managing other system services.

What I like about quadlets:

Systemd integration - Containers start on boot, restart on failure, and log to journald
GitOps friendly - Unit files are plain text, easy to track in version control
Rootless - I can run everything as a regular user with systemd user sessions
No docker-compose needed - Native systemd handles dependencies

2. Prerequisites

Before this, I had:

Rootless Podman configured (Podman Permissions on Kicksecure)
Encrypted storage mounted (Encrypted BTRFS Storage on Kicksecure)

3. Directory Structure

Quadlets live in ~/.config/containers/systemd/:

/home/user/.config/containers/systemd/
├── gitea.container
├── gitea-runner.container
└── gitea.network    # Optional: custom network definition

4. Prepare Storage

Created a subvolume for Gitea data:

sudo btrfs subvolume create /mnt/storage_name/gitea_data

Set ownership for rootless Podman:

podman unshare chown -R 1000:1000 /mnt/storage_name/gitea_data

This maps the container’s UID 1000 to my user’s permissions via the ID mapping I configured earlier.

5. The Gitea Quadlet

Created ~/.config/containers/systemd/gitea.container:

[Unit]
Description=Gitea
After=network-online.target

[Container]
Image=docker.io/gitea/gitea:latest
ContainerName=gitea

# Web interface - bind to localhost only
PublishPort=127.0.0.1:3000:3000

# SSH - binds to all interfaces for external git access
PublishPort=2222:22

Environment=USER_UID=1000
Environment=USER_GID=1000

Volume=/mnt/storage_name/gitea_data:/data:Z

[Service]
Restart=always

[Install]
WantedBy=default.target

5.1. Key Points

PublishPort 127.0.0.1:3000:3000 - Binds the web interface to localhost only. External access goes through my reverse proxy or VPN.

PublishPort 2222:22 - No IP specified means it binds to all interfaces. This allows git operations over SSH from external hosts.

Volume with :Z - The Z suffix tells Podman to relabel the SELinux context (even though Kicksecure uses AppArmor, Podman still recognizes this flag for private volumes).

6. Enable the Podman Socket

Quadlets require the Podman socket for user sessions:

systemctl --user enable --now podman.socket

This creates /run/user/1000/podman/podman.sock.

7. Starting Gitea

Reloaded systemd to recognize the quadlet, then started the service:

systemctl --user daemon-reload
systemctl --user enable --now gitea.service

Checked status:

systemctl --user status gitea.service

Viewed logs:

journalctl --user -u gitea.service

8. Gitea Runner for CI/CD

To run Gitea Actions, I deployed the act_runner container. Created ~/.config/containers/systemd/gitea-runner.container:

[Unit]
Description=Gitea Actions Runner
After=network-online.target gitea.service
Requires=podman.socket

[Container]
Image=docker.io/gitea/act_runner:latest
ContainerName=gitea-runner

# Host network to reach gitea at localhost
Network=host

# Pass the podman socket for Docker-compatible CI
Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:Z

Volume=/mnt/storage_name/app_cache/gitea_runner:/data:Z

Environment=GITEA_INSTANCE_URL=http://127.0.0.1:3000
Environment=GITEA_RUNNER_REGISTRATION_TOKEN=YOUR_REGISTRATION_TOKEN
Environment=DOCKER_HOST=unix:///var/run/docker.sock

[Service]
Restart=always

[Install]
WantedBy=default.target

Got the registration token from Gitea: Site Administration → Actions → Runners → Create new Runner.

8.1. Why Host Networking?

The runner uses Network=host so it can reach Gitea at localhost:3000 without network configuration. Simpler than creating a dedicated pod network.

8.2. Podman Socket Passthrough

The runner mounts the host’s Podman socket, allowing CI pipelines to build containers using the host’s Podman instance. This is powerful but has security implications—only use for trusted CI workloads.

9. Accessing Gitea

Web UI: http://localhost:3000 (or through reverse proxy)
SSH Git: git clone ssh://git@your-server:2222/username/repo.git

Configured my SSH client to use port 2222:

Host gitea
    HostName your-server
    Port 2222
    User git

10. Managing Containers

Common systemd commands I use:

# View status
systemctl --user status gitea.service
systemctl --user status gitea-runner.service

# Restart
systemctl --user restart gitea.service

# Stop
systemctl --user stop gitea.service

# View logs
journalctl --user -u gitea.service -f

# List containers
podman ps

11. Tracking Quadlets in Git

Since quadlets are just text files, I version control them:

cd ~/.config/containers/systemd
git init
git add *.container *.network
git commit -m "Initial container definitions"

This makes it easy to manage multiple services and push to a central repository.

12. Troubleshooting

Container Won’t Start

Check logs:

journalctl --user -u gitea.service -n 50

Common issues:

Permission denied on volume: Re-run podman unshare chown
Image pull failed: Check network access to docker.io
Port already in use: Something else is using 3000 or 2222

Quadlet Not Recognized

Ensure the file ends in .container and is in the correct directory:

ls -la ~/.config/containers/systemd/

Reload systemd:

systemctl --user daemon-reload

Firewall Issues

Locked Out

If I lock myself out:

Physical console access
Recovery mode
Reset UFW: ufw reset

Rules Not Applied

Reload the firewall:

ufw reload

Service Still Accessible

I check if the service is bound to the correct interface:

ss -tlnp

A service bound to 0.0.0.0 listens on all interfaces. One bound to 127.0.0.1 only listens on localhost.

13. UFW Firewall Configuration

I chose UFW (Uncomplicated Firewall) for its simple interface to iptables. While Kicksecure has some firewall defaults, I wanted explicit control over my network security—especially for my Podman container services.

13.1. Installation

apt install ufw

I also installed the GUI for easier visualization:

apt install gufw

13.2. Default Policy

I set a default-deny policy—block all incoming, allow all outgoing:

ufw default deny incoming
ufw default allow outgoing

This means only ports I explicitly open are accessible.

13.3. Enable the Firewall

ufw enable

Then enabled the systemd service:

systemctl enable ufw.service
systemctl start ufw.service

13.4. SSH Access

Since I manage this server remotely, I needed SSH access before enabling the firewall:

ufw allow ssh

Or specifying the port directly:

ufw allow 22/tcp

13.5. Gitea Ports

For Gitea, I opened port 2222 for SSH git operations:

ufw allow 2222/tcp

The web interface (port 3000) is bound to localhost in my quadlet, so it doesn’t need a firewall rule. External access goes through my reverse proxy or VPN.

13.6. Restricting Access by Source

For services that shouldn’t be publicly accessible, I restricted them to specific IPs or subnets.

VPN-Only Access

I use Netbird for VPN access, so I restricted sensitive services to the VPN subnet:

ufw allow from 100.64.0.0/10 to any port 3000 proto tcp

Specific IP Access

For trusted IPs only:

ufw allow from 192.168.1.100 to any port 3000 proto tcp

Removing Rules

Remove a rule by prefixing delete:

ufw delete allow 3000/tcp

Or by number:

ufw status numbered
ufw delete 3

13.7. Checking Status

View current rules:

ufw status verbose

Output looks like:

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)

New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
2222/tcp                   ALLOW IN    Anywhere

13.8. Why Default-Deny Matters

A default-deny firewall gives me:

Reduced attack surface - Unknown services aren’t exposed
Prevention of accidents - New services aren’t accidentally exposed
Limited reconnaissance - Attackers can’t discover services
Simple mental model - If it’s not allowed, it’s blocked

This is especially important on a server running multiple services.

13.9. Tracking Firewall Changes

As with all system configuration, I tracked this with etckeeper:

cd /etc
git add .
etckeeper commit "Configured UFW firewall rules"

14. What’s Next

This completes the core series. Future posts will cover ZFS RAID and Ansible automation with enroll.

Podman Permissions on Kicksecure

Mon, 09 Mar 2026 00:00:00 +0000

This article is part of my Self-Hosting with Kicksecure series.

1. Introduction

When I first tried running rootless Podman on Kicksecure, containers wouldn’t start. The errors looked like:

Error: command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH

Or:

zsh: permission denied: /usr/bin/newuidmap

Rootless Podman requires newuidmap and newgidmap to map a single user ID to the thousands of IDs needed inside containers. The issue? Kicksecure restricts access to these binaries by default as a security measure. This article documents how I solved this properly.

2. Understanding the Problem

I checked Kicksecure’s configuration:

dpkg-statoverride --list /usr/bin/newuidmap

Output:

root root 744 /usr/bin/newuidmap

Only root can execute these binaries. For most systems, this is good security. But rootless containers need access.

3. First Attempt: dpkg-statoverride

My first approach was to use dpkg-statoverride to change permissions:

addgroup podman-users
usermod -aG podman-users user
dpkg-statoverride --remove /usr/bin/newuidmap
dpkg-statoverride --remove /usr/bin/newgidmap
dpkg-statoverride --update --add root podman-users 4750 /usr/bin/newuidmap
dpkg-statoverride --update --add root podman-users 4750 /usr/bin/newgidmap

This worked temporarily. But then I ran an apt upgrade, and permission-hardener reverted my changes. My containers broke after the next update.

4. The Correct Solution

Kicksecure provides a proper way to whitelist binaries through permission-hardener.

4.1. Create Whitelist Configuration

mkdir /etc/permission-hardener.d

Created /etc/permission-hardener.d/50_podman.conf:

/usr/bin/newuidmap exactwhitelist
/usr/bin/newgidmap exactwhitelist

The exactwhitelist directive tells permission-hardener to leave these binaries alone while still applying restrictions to everything else.

4.2. Apply Changes

permission-hardener enable all

Verified the permissions changed:

ls -l /usr/bin/newuidmap

-rwsr-x--- 1 root podman-users 14976 Jan 10 12:00 /usr/bin/newuidmap

5. Configuring Podman

Even with correct permissions, Podman couldn’t find these binaries in its path. I created a containers configuration file.

As the user account, created ~/.config/containers/containers.conf:

[engine]
newuidmap_path = "/usr/bin/newuidmap"
newgidmap_path = "/usr/bin/newgidmap"

6. Verifying User ID Mappings

I confirmed my user had the necessary ID mappings:

grep $(whoami) /etc/subuid
grep $(whoami) /etc/subgid

Output:

user:100000:65536

This means user can map UIDs 100000-165535 inside containers.

7. Testing

Switched to the container user and tested:

su - user
podman system migrate

This succeeded without errors. Then I ran a test container:

podman run --rm hello-world

8. Why This Matters

Rootless containers are a significant security improvement:

Container escapes are contained to the user account
Kernel exploits have limited impact
File ownership is handled via ID mapping, not root privileges

Kicksecure’s default restrictions are correct for most use cases. Podman is an exception that requires explicit whitelisting.

9. Troubleshooting

Permission Denied

Check group membership:

groups user

Ensure podman-users appears in the list. A logout/login might be needed.

Path Not Found

Ensure the containers.conf file exists:

ls -la ~/.config/containers/containers.conf

Changes Reverted After Update

Verify the whitelist file:

cat /etc/permission-hardener.d/50_podman.conf

Then re-run:

permission-hardener enable all

10. Tracking the Change

This is exactly the kind of configuration change worth tracking:

cd /etc
git add .
etckeeper commit "Whitelisted newuidmap/newgidmap for rootless podman"

More on this in my Etckeeper on Kicksecure article.

10. What’s Next

Podman Quadlets on Kicksecure

Etckeeper on Kicksecure

Sun, 08 Mar 2026 00:00:00 +0000

This article is part of my Self-Hosting with Kicksecure series.

1. Introduction

Before making any system changes, I want to decide on a way to record every change I make so that:

I can easily reproduce my work if I ever have to reinstall or migrate to another machine
I can easily identify problematic configs and revert them
I don’t forget the reasoning behind a particular security decision or config change in 6 months
I can identify what changes I made as opposed to changes made by software or other people
I can compare the current state of my configs against a ‘known-good’ config

The /etc directory will contain most of my configs. Outside of /etc I will need to pay attention to my dotfiles which I like to manage with GNU Stow.

Etckeeper is a neat piece of software that turns /etc into a git repository. It automatically commits changes before and after package installations, and I can manually commit my own changes too.

2. Installation

Kicksecure protects itself by isolating system maintainence tasks from daily activity. This means administrative tasks like installing new software cannot be performed from the standard user account, even with sudo. You have to reboot and select an option to boot into a session with the sysmaint account, make your changes, and then reboot back into the standard user session to continue with daily usage.

Having booted with sysmaint, I install Etckeeper from the Debian repositories:

apt install etckeeper

3. Etckeeper Basics

Do the standard git stuff:

git config --global user.email "person@email.com"
git config --global user.name "I'm a person with a name!"


I start by initializing the repository:

```bash
cd /etc
etckeeper init

And making my initial commit:

etckeeper commit "Initial configuration snapshot"

Etckeeper hooks into apt. When I install or remove packages, it automatically commits them to the git repo. So I don’t need to worry about doing a commit myself! I only need to commit after I make manual changes in /etc.

For configuration changes I make outside of apt:

# within /etc
git add .
etckeeper commit "Descriptive message about what changed"

4. Viewing History

Standard git commands work:

cd /etc
git log --oneline

View a specific change:

git show COMMIT_HASH

Compare two states:

git diff COMMIT_HASH1 COMMIT_HASH2

5. The .gitignore File

Some files in /etc shouldn’t be tracked: temporary files, secrets, or auto-generated content.

On my server I ended up configuring a ZFS RAID. Here’s an example of excluding tracking of my ZFS cache with a few other things in /etc/.gitignore:

# Ignore auto-generated ZFS cache
/zfs/zfs-list.cache/

# Example: ignore temporary files
*.tmp
*.swp

After editing:

git add .gitignore
etckeeper commit "Updated gitignore to exclude my auto-generated ZFS cache"

6. Backups

Always back up your changes so you have access to these records if your system breaks! In my case, I back up to a local ‘bare repository’ on my BTRFS backup drive. I explain how I set up this drive on Kicksecure in a later post.

Additionally, because I’d like to be able to identify known-good states of my configs even if an attacker compromises my server or network, I sanatize + compress + back up my data (or just the hashes for big files) to an air-gapped system regularly using something similar to a data diode. The air gap setup is beyond the scope of this post, but I thought the suggestion might interest readers!

Here’s how I back up my /etc repo:

6.1. Create a Bare Repository

On my already mounted BTRFS storage:

mkdir -p /mnt/storage_name/host_vault/kicksecure-etckeeper.git
git init --bare /mnt/storage_name/host_vault/kicksecure-etckeeper.git

6.2. Add as Remote

cd /etc
git remote add localvault /mnt/storage_name/host_vault/kicksecure-etckeeper.git

6.3. Push Changes

git push -u localvault main

Future pushes:

git push localvault

6.4. Push After Every Commit

This became a habit:

etckeeper commit "Made a change" && git push localvault

7. What to Track

Here’s what I have tracked so far in /etc:

/etc/fstab - Mount point config
/etc/crypttab - Encrypted drive config
/etc/ufw/ - Firewall rules
/etc/ssh/ - SSH configuration
/etc/permission-hardener.d/ - Kicksecure permission whitelists
/etc/systemd/system/ - Custom systemd units
/etc/modprobe.d/ - Kernel module configuration

8. What’s Next

Encrypted BTRFS Storage on Kicksecure

Sat, 07 Mar 2026 00:00:00 +0000

This article is part of my Self-Hosting with Kicksecure series. It assumes you’ve already set up Etckeeper GitOps on Kicksecure so all configuration changes are tracked from the beginning.

1. Introduction

I needed encrypted storage for my container data. I decided to use a 4TB SSD with BTRFS for system backups and services that don’t require huge amounts of data. For services that require a lot like Nextcloud, I put them on a ZFS RAID 10 array with 4x 8TB HDDs.

My requirements for the BTRFS storage were:

Encryption at rest - If someone steals the drive, they can’t read it
Automount at boot - Using a keyfile so the server starts unattended
Subvolumes - Separate container data for easier snapshots
Compression - Save space where possible

2. Identifying the Drive

First, I identified my target drive by reviewing the size column from the lsblk command. In this case, my device was sdb.

Then got its device ID for reliable reference across reboots:

ls -l /dev/disk/by-id/ | grep sdb

I saw entries like:

ata-CT4000BX500SSD1_XXXXXXXXXXXX -> ../../sdb

I used the ata- prefixed ID—it’s more reliable than /dev/sdb since the latter mapping can be subject to change.

3. Creating LUKS Encryption

Create the LUKS container:

cryptsetup luksFormat --type luks2 /dev/sdb

It prompted me to create a passphrase.

Then I unlocked the encrypted container:

cryptsetup open /dev/sdb storage_crypt

This created /dev/mapper/storage_crypt, a path that allows for accessing the unlocked container.

4. Creating BTRFS Filesystem

I created a BTRFS filesystem on the encrypted container:

mkfs.btrfs -L "StorageName" /dev/mapper/storage_crypt

The output showed filesystem details including UUID and features like extref, skinny-metadata, no-holes, and free-space-tree.

5. Creating Subvolumes

I mounted temporarily to create subvolumes:

mkdir /mnt/storage_name
mount /dev/mapper/storage_crypt /mnt/storage_name

Created a subvolume for data:

btrfs subvolume create /mnt/storage_name/@gitea_data

BTRFS allows you to perform rollbacks on subvolumes by creating snapshots. When you snapshot a subvolume, BTRFS doesn’t copy all the data, it only records new changes. You can snapshot terabytes of data in seconds, and the snapshot only takes up space for the actual differences.

By organizing data into separate subvolumes like @jellyfin_data, @nextcloud_data, @homeassistant_data, and @gitea_data, I can make snapshots that act independently for my different podman services. This makes it easy to roll back a single container’s data without affecting others.

I unmounted when I was done with the mount:

umount /mnt/storage_name

6. Keyfile for Automount

I want to unlock the drive automatically at boot rather than always entering the passphrase. I start by creating a keyfile with random data in it:

dd if=/dev/urandom of=/root/storage.key bs=1 count=32
chmod 400 /root/storage.key

I added it to LUKS so that it can be used to unlock:

cryptsetup luksAddKey /dev/sdb /root/storage.key

I entered my original passphrase when prompted.

7. Configuring crypttab

Got the LUKS UUID:

cryptsetup luksUUID /dev/sdb

Backed up the existing crypttab:

cp /etc/crypttab /etc/crypttab.old

Added the encrypted drive to /etc/crypttab:

echo "storage_crypt UUID=YOUR_DISK_UUID /root/storage.key luks,nofail,discard" >> /etc/crypttab

The options explained:

luks - Use LUKS encryption
nofail - Boot continues if the drive fails to unlock
discard - Enable TRIM for SSDs

9. Configuring fstab

Added the BTRFS mount to /etc/fstab:

echo "/dev/mapper/storage_crypt /mnt/storage_name btrfs defaults,nofail,compress=zstd,noatime,subvol=@data 0 0" >> /etc/fstab

The options:

defaults - Standard mount options
nofail - Boot continues if mount fails
compress=zstd - Transparent compression (good balance of speed and ratio)
noatime - Don’t update access times (improves performance)
subvol=@data - Mount the @data subvolume I created

10. Testing

Reloaded systemd and tested the mount:

systemctl daemon-reload
mount -a

Verified it mounted:

lsblk

I saw my encrypted container mounted at /mnt/storage_name.

11. Permissions

Set ownership for the container user:

chown -R user:user /mnt/storage_name
chmod 770 /mnt/storage_name

12. Creating Additional Subvolumes

As I added more services, I created separate subvolumes:

mount /dev/mapper/storage_crypt /mnt/storage_name
btrfs subvolume create /mnt/storage_name/@gitea_data
btrfs subvolume create /mnt/storage_name/@cache
umount /mnt/storage_name

Then added additional fstab entries or adjusted container volume mounts.

13. BTRFS Snapshots

One advantage of BTRFS is easy snapshots. I create snapshots before upgrades:

btrfs subvolume snapshot /mnt/storage_name/@data /mnt/storage_name/@data-backup-$(date +%Y%m%d)

14. Troubleshooting

Drive Doesn’t Mount at Boot

Check if the keyfile path is correct in /etc/crypttab. It must be accessible during early boot.

Check journal logs:

journalctl -xe

Wrong Subvolume Mounted

If you’re not seeing your data, check that /etc/fstab includes subvol=@data.

Permission Denied for Containers

If containers can’t write, check ownership:

ls -la /mnt/storage_name

For Podman containers, I use podman unshare chown to set container-internal UIDs.

15. What’s Next

Podman Permissions on Kicksecure

Self-Hosting with Kicksecure

Fri, 06 Mar 2026 00:00:00 +0000

1. Intro

I formerly self-hosted my services on a Kubernetes cluster with FluxCD inside openSUSE MicroOS VMs hosted by Proxmox. While that was really flexible, it became a burden maintaining it in my freetime and demanded too much RAM and storage on my resource-constrained servers. I still want to keep my K8s cluster around for tinkering in my homelab, but for a personal setup that I need to “just work” for NAS, git hosting, and backups, the complexity didn’t justify the benefits.

Yet, I loved my former workflows: GitOps, infrastructure as code, and declarative container management. I wanted to manage my self-hosting from a git repository. That’s how I landed on etckeeper for managing my base system and Podman quadlets for managing my container services declaratively with systemd.

There was one more criteria I wanted to meet…

2. Why Kicksecure?

I also don’t have enough time in my day to exhaustively follow hardening guides. I’d prefer an opinionated, secure by default system. I evaluated several options before choosing Kicksecure.

Secureblue looks promising! But I’m more comfortable with Debian-based systems, and I wasn’t familiar with its development background and whether the project is reputable.

NixOS and Gentoo were considered. Gentoo could be a really secure option but would have been another maintenance burden. NixOS might have worked since I daily drive it on my gaming PC and I’ve already gone through some hardening guides so I could copy my existing configs. Unfortunately I ran into a known bug preventing NixOS/GuixSD from booting with Heads on my server. Heads is a security focused distribution based on the coreboot open source BIOS based with features to prevent boot tampering, and rather than toiling against this bug I opted to go with the next option:

Kicksecure is connected to the same team behind Whonix which I already use! Whonix is a well-established project and it currently fits my constraints.

Kicksecure does need extra configuration to work with podman, but it turned out to be much more manageable than my old stack.

3. Retrospective

By the end of this project, I had assembled:

An etckeeper managed Kicksecure server
An encrypted BTRFS storage with subvolumes for backing up extremely critical stuff
Rootless Podman containers managed with git tracked systemd quadlets
A running Gitea instance with SSH access on port 4022
Firewall rules restricting access to only necessary ports

3.1 Posts in this series

I’m documenting my journey through the setup process in these posts:

Step	Article	Description
1	Etckeeper GitOps on Kicksecure	Keep track of all my Kicksecure configuration changes from the start
2	Encrypted BTRFS Storage on Kicksecure	Configuring server storage on BTRFS subvolumes with LUKS encryption
3	Podman Permissions on Kicksecure	Allowing rootless Podman to use newuidmap/newgidmap via Kicksecure’s Permission Hardener
4	Podman Quadlets on Kicksecure	Deploying containers with Git-tracked Podman quadlets and UFW firewall

Additionally, I flashed and locked down Heads/coreboot, installed an open source zero trust mesh VPN for remote access, deployed host-based IDS, moved server management to Ansible roles, and set up a Nextcloud instance utilizing storage on a ZFS RAID 10 array with 4x 8TB HDD’s. In the future I hope to expand this series with posts detailing those efforts, as well as posts further exploring Kicksecure’s nifty security features.