Linux Server Automation with Native Tools: cron, systemd, SSH

Every fleet older than three years has a cron job graveyard: scripts that have run nightly since 2019, written by an engineer who left in 2021, cleaning up a directory that moved in 2023. Nobody dares delete them. That graveyard is what happens when server automation gets built without structure — and it's avoidable.

In part one of this series we catalogued the manual-SSH toil that eats ops time — log hunts, disk cleanups, service restarts, cert and package checks — and the safety fundamentals any server automation agent has to respect: key-based auth, restricted keys, trusted hosts, audit trails. This article is the DIY build. Everything below uses tools already installed on every mainstream distro: cron, systemd timers, OpenSSH itself, bash, logrotate, and unattended-upgrades. There's nothing to buy and nothing new to install; an afternoon covers the first host, a second one gets you to fleet-wide rollout.

The walkthrough runs tool by tool, starting with the scheduler and ending — honestly — with where this approach stops scaling.

Tool 1: cron and systemd timers — the scheduler

Recurring checks are the easiest wins: disk usage, service health, cert expiry. Classic cron still works fine for a single check:

# crontab -e
# cron mails any output to the crontab owner: print filesystems above 85%
*/30 * * * * df -P | awk 'NR>1 && $5+0 > 85 {print; exit 1}'

But for anything you care about, prefer systemd timers. You get output captured in the journal, Persistent=true to catch runs missed while a host was down, and RandomizedDelaySec so 200 hosts don't hammer your package mirror at the same second (systemd.timer docs).

A disk-check pair looks like this:

# /etc/systemd/system/disk-check.service
[Unit]
Description=Alert on filesystems above 85%

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/disk-check.sh

# /etc/systemd/system/disk-check.timer
[Unit]
Description=Run disk check every 30 minutes

[Timer]
OnCalendar=*:0/30
RandomizedDelaySec=120
Persistent=true

[Install]
WantedBy=timers.target

Enable and verify:

sudo systemctl daemon-reload
sudo systemctl enable --now disk-check.timer
systemctl list-timers disk-check.timer
journalctl -u disk-check.service --since today

That last command is the quiet superpower over cron: every run's output, timestamped, queryable, on-box. When a check misbehaves at 3 a.m., journalctl answers in seconds.

Tool 2: hardened SSH — the transport layer

Fleet automation means a key that can log into many machines, which makes that key your biggest liability. OpenSSH gives you three layers of containment; use all three.

Layer 1: key-based auth only, on trusted hosts only

On every managed host, in /etc/ssh/sshd_config (or a drop-in under /etc/ssh/sshd_config.d/):

PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
PermitRootLogin prohibit-password

Then sudo systemctl reload sshd. Keep an active session open while you test — locking yourself out of a fleet is a rite of passage you can skip. And maintain a curated known_hosts for your automation user rather than StrictHostKeyChecking no; the whole point of trusted hosts is that an unexpected key change stops the run.

Layer 2: restricted keys with command= and from=

The automation key should not be a general-purpose login. In the target host's ~/.ssh/authorized_keys, prefix the key with restrictions (sshd authorized_keys format):

command="/usr/local/sbin/healthcheck.sh",from="10.0.4.10",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAA... automation@bastion

Now this key can do exactly one thing (run your wrapper script), from exactly one source IP, with no interactive terminal and no forwarding. If it leaks, the blast radius is one read-only script. For automation that needs several verbs, make the command= target a dispatcher that reads $SSH_ORIGINAL_COMMAND and allowlists what it will run — one restricted key per privilege tier, not one god key.

Layer 3: client-side Match blocks and ProxyJump

On the control machine, structure ~/.ssh/config so automation traffic is explicit and routed through your bastion (ssh_config docs):

Host bastion
    HostName bastion.example.com
    User automation
    IdentityFile ~/.ssh/automation_ed25519
    IdentitiesOnly yes

Match host "prod-*" user automation
    ProxyJump bastion
    IdentityFile ~/.ssh/automation_ed25519
    IdentitiesOnly yes
    BatchMode yes
    ConnectTimeout 10

ProxyJump tunnels through the bastion without landing keys on it. BatchMode yes makes failures fail fast instead of hanging on a password prompt, and IdentitiesOnly stops ssh from offering every key in your agent to every host — which both leaks key fingerprints and trips MaxAuthTries.

Tool 3: fleet loops with ssh and bash

With hardened transport, a fleet-wide check is a loop over a hosts file:

#!/usr/bin/env bash
# fleet-check.sh — read-only inspection across the fleet
set -euo pipefail
while read -r host; do
  echo "== ${host} =="
  ssh -o BatchMode=yes -o ConnectTimeout=10 "${host}" \
    'df -P /var; systemctl is-active nginx postgresql; uptime' \
    2>&1 | tee -a "logs/${host}.$(date +%F).log"
done < hosts-prod.txt

Serial loops crawl past 20 hosts. Parallelize with xargs:

xargs -a hosts-prod.txt -P 10 -I {} \
  ssh -o BatchMode=yes -o ConnectTimeout=10 {} 'df -P / | tail -1' \
  2>&1 | tee "fleet-disk.$(date +%F).log"

Two rules keep this safe. First, separate read scripts from write scripts — different files, different restricted keys, so a typo in a report loop can never restart a service. Second, tee everything to dated logs; those files are your only audit trail, so treat them as append-only and rotate them (see the next tool). Anything mutating — restarts, cleanups, upgrades — gets a --dry-run-style echo mode and a human reading the output before the real pass.

Tool 4: logrotate — stop the disk-full pages at the source

A large share of the disk alerts from part one are unrotated application logs. Native fix, five minutes (logrotate man page):

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Validate without waiting a day:

sudo logrotate --debug /etc/logrotate.d/myapp
sudo logrotate --force /etc/logrotate.d/myapp

Prefer having the app reopen its log file on signal (a postrotate script with kill -USR1) over copytruncate where the app supports it — copytruncate can drop lines written during the truncate window. On modern distros logrotate is already driven by logrotate.timer, so there's nothing to schedule.

Tool 5: unattended-upgrades — patching on autopilot

The recurring "are we patched?" check from part one automates natively. Debian/Ubuntu (Debian wiki):

sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

Then scope it to security updates in /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Mail "ops@example.com";

RHEL-family hosts get the same behavior from dnf-automatic: install it, set apply_updates = yes and upgrade_type = security in /etc/dnf/automatic.conf, then systemctl enable --now dnf-automatic.timer. Either way, keep reboots manual and let your fleet loop report /var/run/reboot-required — patching is safe to automate; reboot timing usually isn't.

Where scripts stop scaling

Everything above is worth building. It will retire real toil this week. But run it for a year across 50+ hosts and the same failure pattern emerges, and it's worth naming precisely: scripts execute — they don't observe, decide, or explain.

They don't observe. A timer fires whether or not the thing it checks still matters. Nothing notices that the disk check has been failing silently since a hostname changed, or that a "temporary" cleanup job is now deleting a directory another team depends on. Cron mails root; nobody reads root's mail. This is exactly how the graveyard forms.

They don't decide. A script encodes one branch: threshold exceeded, run the cleanup. It can't weigh context — that the disk is filling because of a deploy in progress, that this host is mid-incident, that the right action tonight is different from last night's. Every judgment call still pages a human, usually the one who wrote the script.

They don't explain. When a fleet loop mutates 50 hosts, your audit trail is a pile of dated tee logs — what ran, but not why, requested by whom, or with what outcome. Reconstructing that after an incident is an evening's work, and after the author leaves it's archaeology.

The engineer-hours don't disappear either; they move. You stop typing df -h and start maintaining hosts files, rotating restricted keys, and debugging the automation itself. Useful — but it's a floor, not a ceiling.

What comes next

The gap the scripts leave — observing, deciding, explaining — is what part three of this series covers: CloudThinker agents working over the same hardened SSH you just built, with a dedicated key to trusted hosts only, read-only inspection by default, investigation before any action, and mutating commands gated behind your approval with a full audit trail of every command run. If you want to see it against your own fleet first: Try CloudThinker free — 100 premium credits, no card required.