jonas.splinter
Blog

Let monit tell you when an update is waiting


Let monit tell you when an update is waiting - featured image

An update can be available, installed, or actually running. How to make monit watch all three with small shell checks, and how to prove each check really fires.

I use monit to watch for pending updates on my servers, and in November 2026 I improved those checks. The main lesson: an update is not one thing to watch but three. It can be available and not yet installed. It can be installed but not yet active, because the old code is still in memory. Or it can be installed and actually running. Only the last state counts for security, and remembering to check by hand does not scale past a busy week. Below is a generic version of the approach, with config and scripts you can copy and adapt.

Sorting what is available

The first check answers one question: is anything waiting to be installed, and how urgent is it. A small script lists the pending updates and reports through its exit code. monit runs it and alerts on that code. I use three codes, because not every update deserves the same reaction:

  • 0: nothing pending, stay quiet.
  • 1: something urgent is pending, such as a security update. Alert now.
  • 2: only routine updates are pending. Alert only when they have waited too long.

On Ubuntu, apt list --upgradable gives the list, and security updates come from a suite whose name ends in -security. A minimal version:

#!/bin/sh
# /usr/local/bin/check-updates  exit 0 = none, 1 = urgent, 2 = routine
list=$(apt list --upgradable 2>/dev/null | grep -v '^Listing')
[ -z "$list" ] && exit 0
# urgent: anything from -security, plus packages you decide are always urgent
urgent=$(printf '%s\n' "$list" | grep -E -- '-security|^(nginx|openssl|libssl[^/]*)/')
if [ -n "$urgent" ]; then echo "$urgent"; exit 1; fi
echo "$list"; exit 2

The echo lines matter: monit includes the program's output in the alert, so the message tells you which packages are waiting.

The classification trap

The second half of that grep pattern is there for a reason. If you install software from its maker's own repository, for example nginx from nginx.org, your operating system's tools do not know which of its updates fix security problems. Those updates do not arrive through the -security suite. A check that only counts security updates would treat a critical web server fix as routine and wait days to tell you. I explain why I made that switch in my post about taking nginx from nginx.org.

So decide per package what counts as urgent. Anything that faces the internet, such as the web server and the TLS library, belongs on that list, wherever it comes from.

Catching what is installed but not running

Installing an update replaces files on disk. It does not replace what is already in memory. Three common cases each deserve their own check:

  • The kernel. The old kernel runs until you reboot. Ubuntu creates the file /var/run/reboot-required when an update needs a reboot, which makes this the easiest one to watch.
  • Libraries. After an OpenSSL update, long-running services such as PHP-FPM or MySQL keep using the old library until they restart.
  • Services themselves. After an nginx update, the old binary may keep serving until a restart. A reload is not enough, because the running master process stays in charge.

On Linux you can see the last two in /proc. When the package manager replaces a file that a process is using, the old file is removed but stays in memory. The link /proc/<pid>/exe then ends in (deleted), and /proc/<pid>/maps shows the old libraries marked (deleted). A script can look for both:

#!/bin/sh
# /usr/local/bin/check-stale-code  exit 1 if a process runs replaced code
found=0
for p in /proc/[0-9]*; do
  if readlink "$p/exe" 2>/dev/null | grep -q ' (deleted)$' ||
     grep -qE '\.so[^ ]* \(deleted\)$' "$p/maps" 2>/dev/null; then
    echo "$(cat "$p/comm" 2>/dev/null) ${p#/proc/}"; found=1
  fi
done
exit $found

It must run as root to read every process. The pattern only looks at shared libraries, because some programs deliberately keep deleted temporary files mapped, and you do not want an alert for those. A second approach, if you prefer to watch one service: compare the modification time of its binary with the start time of its main process. If the process is older than the file, the old version is still running.

Wiring it into monit

With the scripts in place, the monit side is short. check program runs a script, if status tests its exit code, and every decides how often it runs, so you do not ask the package manager for its list on every cycle:

check program updates_pending with path /usr/local/bin/check-updates
    every 30 cycles
    if status == 1 then alert
    if status == 2 for 14 cycles then alert

check program stale_code with path /usr/local/bin/check-stale-code
    every 30 cycles
    if status != 0 then alert

check file reboot_needed with path /var/run/reboot-required
    if exist then alert

The for 14 cycles clause is how routine updates "wait too long": monit only alerts when the routine status has been seen that many times in a row. How long that is in hours depends on your poll interval and the every line, so work it out for your own setup. If you want an alert each time the list changes, if changed status then alert is another option.

What I deliberately left out is any then exec or then restart to fix things. monit should tell you, not act. Applying updates, restarting a web server or restarting a database is a decision: you want to read what changed, pick a quiet moment and be there to watch it. For automation like this, the alert is the automated part, and the fix stays with a person.

Make every check fail once

A check that has never fired is a guess. Before you trust one, make it fail on purpose with a known-bad case, confirm the alert arrives, and then confirm it stays quiet on the healthy system. Some ways to do that:

  • Point the update check at a stub script that only does exit 1, and later exit 2, and watch each alert arrive.
  • Feed the classification grep a fake line such as nginx/noble 1.2.3 amd64 and confirm it counts as urgent.
  • Copy sleep into a temporary folder, keeping its name (on some systems it refuses to run under another name), start it, delete the file, and confirm the stale code check names that process.
  • Create the reboot file by hand on a test machine and watch the alert, then remove it again.

Then watch the timing. monit runs program checks in the background and collects the exit status on a later cycle. With every, a check keeps its last result until its next run. After you apply an update or restart a service, the status can stay red for a while, and the first verdict after the fix can still come from a run that started before it. Before you panic or celebrate, look at when the data was collected: monit status shows a "data collected" time for every check.

What I do not know yet

I run a few servers, not many. I do not know how well this approach scales to dozens or hundreds of machines, where a central patch management tool probably makes more sense. monit is not a patch manager: it reports what it is told to look for, and nothing more.

The scripts above cover the Ubuntu and Linux cases I described. Other distributions mark security updates differently, and the /proc method only sees processes, not, for example, code loaded inside a virtual machine or container image that has not been rebuilt.

What to do first

  1. Run apt list --upgradable and ls /var/run/reboot-required by hand to see where you stand today.
  2. Write down which packages you consider urgent, including any that come from a maker's own repository.
  3. Add the update check with separate exit codes for urgent and routine, and give monit the if status lines.
  4. Add the stale code check and the reboot check.
  5. Make each check fail once on purpose, then confirm it is quiet on the healthy system.
  6. Keep fixes manual: when an alert comes in, review, apply and restart at a moment you choose.

Frequently asked questions

Why not let monit apply the updates itself?

Because an update can need a restart of a web server or database, and that is a decision you want to make while you are watching. Let monit alert, and apply and restart by hand.

Is unattended-upgrades not enough?

It installs updates, but installed is not running. A new kernel waits for a reboot, and services keep old libraries until they restart. You still need something that tells you about those two states.

Why did my check stay red after I fixed the problem?

Program checks that run every few cycles keep their last result until the next run, and that result may come from before your fix. Check when the data was collected before you draw a conclusion.

If you want to go further, read the monit documentation (opens in a new tab), in particular the sections on program checks and the status test.

Related posts

More to read on the same topics

Related updates

Short notes on the same topics