Your application crashes at 3 AM. Without service management, it stays down until someone notices. With Systemd and Monit, it restarts in seconds — automatically, silently, with a log entry you can review in the morning.
This guide covers both tools in depth: Systemd for service lifecycle management, and Monit for resource-aware monitoring and recovery.
Systemd: The Service Manager
Systemd is the init system on virtually every modern Linux distribution — Ubuntu, Debian, Fedora, CentOS, Arch. It manages the entire lifecycle of your services: starting them at boot, restarting them on failure, and stopping them cleanly on shutdown.
Writing a Service Unit
A Systemd unit file defines how your application runs. Here's a production-ready example for a Node.js application:
[Unit]
Description=My Node.js API
After=network.target postgresql.service
Wants=postgresql.service
[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/node dist/server.js
Restart=on-failure
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60
# Environment
Environment=NODE_ENV=production
EnvironmentFile=/var/www/myapp/.env
# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/www/myapp/uploads /var/www/myapp/logs
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp
[Install]
WantedBy=multi-user.target
Save this to /etc/systemd/system/myapp.service, then enable and start it:
sudo systemctl daemon-reload
sudo systemctl enable myapp
sudo systemctl start myapp
Key Directives Explained
Restart=on-failure vs Restart=always: Use on-failure for most applications. always restarts even on clean exits (exit code 0), which can mask bugs where your app exits intentionally.
RestartSec=5: Wait 5 seconds between restart attempts. Without this, Systemd restarts immediately, which can hammer databases or APIs that aren't ready yet.
StartLimitBurst=5 and StartLimitIntervalSec=60: If the service fails 5 times within 60 seconds, Systemd stops trying. This prevents infinite restart loops from consuming resources. Check what went wrong with journalctl -u myapp --since "5 minutes ago".
After= and Wants=: After controls startup order (start my app after PostgreSQL). Wants creates a soft dependency (start PostgreSQL if it's not running, but don't fail if it can't).
Security Hardening
The ProtectSystem=strict and NoNewPrivileges=true directives are often overlooked but important:
# Prevent privilege escalation
NoNewPrivileges=true
# Mount the filesystem read-only except for specified paths
ProtectSystem=strict
ReadWritePaths=/var/www/myapp/uploads
# Hide /home, /root, and /run/user from the service
ProtectHome=true
# Private /tmp (isolated from other services)
PrivateTmp=true
These don't affect functionality but significantly reduce the blast radius if your application is compromised.
Debugging with journalctl
Systemd captures all stdout/stderr output through the journal. This is far more reliable than logging to files — journal entries are indexed, rotatable, and queryable:
# Follow live logs
journalctl -u myapp -f
# Logs from the last hour
journalctl -u myapp --since "1 hour ago"
# Logs from the last boot only
journalctl -u myapp -b
# Show only errors
journalctl -u myapp -p err
# Check why a service failed
systemctl status myapp
Socket Activation
For services that receive infrequent traffic, Systemd can listen on the socket and only start the application when a connection arrives:
# myapp.socket
[Socket]
ListenStream=3000
[Install]
WantedBy=sockets.target
This reduces memory usage on servers running many services, since idle applications consume zero resources until needed.
Monit: Resource-Aware Monitoring
Systemd handles start/stop/restart well, but it doesn't monitor resource usage. Your application might be running but consuming 95% of available memory, or stuck in an infinite loop burning CPU. That's where Monit comes in.
Monit watches processes, resources, files, and network services — and takes action based on thresholds you define.
Installation
# Ubuntu/Debian
sudo apt install monit
# CentOS/RHEL
sudo yum install monit
# Start and enable
sudo systemctl enable monit
sudo systemctl start monit
Monitoring a Service
Create a configuration file in /etc/monit/conf.d/myapp:
check process myapp matching "node dist/server.js"
start program = "/usr/bin/systemctl start myapp"
stop program = "/usr/bin/systemctl stop myapp"
# Restart if the HTTP health check fails
if failed
host 127.0.0.1
port 3000
protocol http
request "/health"
with timeout 10 seconds
then restart
# Resource thresholds
if cpu > 80% for 5 cycles then alert
if cpu > 95% for 3 cycles then restart
if memory > 500 MB for 5 cycles then alert
if memory > 800 MB for 3 cycles then restart
# Restart limits (prevent flapping)
if 5 restarts within 5 cycles then unmonitor
group application
The matching directive is more robust than pidfile — it finds the process by command name, so you don't need to manage PID files.
Health Check Monitoring
The HTTP health check is the most valuable Monit feature. Rather than just checking whether the process exists, it verifies the application is actually responding:
if failed
host 127.0.0.1
port 3000
protocol http
request "/health"
status = 200
content = "ok"
with timeout 10 seconds
then restart
Your /health endpoint should check downstream dependencies too — database connectivity, cache availability, disk space:
// Express health check endpoint
app.get('/health', async (req, res) => {
try {
await db.query('SELECT 1');
res.json({ status: 'ok', uptime: process.uptime() });
} catch (err) {
res.status(503).json({ status: 'error', message: err.message });
}
});
Email Alerts
Monit can send email notifications when thresholds are hit or services restart:
set mailserver smtp.gmail.com port 587
username "alerts@yourdomain.com"
password "app-specific-password"
using tls
set alert ops@yourdomain.com
# Per-check alert overrides
check process myapp matching "node dist/server.js"
alert ops@yourdomain.com only on { timeout, resource, nonexist }
Monit Web Dashboard
Monit includes a built-in web interface for checking service status:
set httpd port 2812
use address 127.0.0.1 # Only listen on localhost
allow admin:secretpassword
Access it at http://localhost:2812. In production, put it behind a reverse proxy with proper authentication rather than exposing it directly.
Using Both Together
The recommended setup: Systemd manages the service lifecycle, Monit monitors health and resources. Monit delegates start/stop commands to Systemd, so there's no conflict:
# Monit uses systemctl for start/stop
start program = "/usr/bin/systemctl start myapp"
stop program = "/usr/bin/systemctl stop myapp"
This gives you:
- Systemd: Boot-time startup, dependency ordering, security sandboxing, journal logging
- Monit: HTTP health checks, CPU/memory thresholds, email alerts, web dashboard
Monitoring Multiple Services
A typical production stack might have several services to monitor:
# /etc/monit/conf.d/stack
check process nginx with pidfile /run/nginx.pid
start program = "/usr/bin/systemctl start nginx"
stop program = "/usr/bin/systemctl stop nginx"
if failed port 80 protocol http then restart
check process postgresql with pidfile /var/run/postgresql/14-main.pid
start program = "/usr/bin/systemctl start postgresql"
stop program = "/usr/bin/systemctl stop postgresql"
if failed port 5432 protocol pgsql then restart
if cpu > 70% for 3 cycles then alert
check process redis with pidfile /run/redis/redis-server.pid
start program = "/usr/bin/systemctl start redis"
stop program = "/usr/bin/systemctl stop redis"
if failed port 6379 then restart
if memory > 1 GB then alert
Integrating with DeployHQ
When deploying through DeployHQ, include your service configuration files in the repository and use deployment scripts to apply them:
# SSH command (runs on the server after file transfer)
# Copy service file if changed
sudo cp config/myapp.service /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
# Graceful restart (finish current requests before stopping)
sudo systemctl reload-or-restart myapp
# Verify the service is healthy
sleep 3
if ! systemctl is-active --quiet myapp; then
echo "ERROR: Service failed to start after deployment"
sudo journalctl -u myapp --since "1 minute ago" --no-pager
exit 1
fi
# Reload Monit to pick up any config changes
sudo monit reload
The health check after restart is critical — without it, a deployment that breaks your app will report success. For more on this pattern, see our guide to zero-downtime deployments.
Systemd Timers (Replacing Cron)
Systemd timers are a modern replacement for cron jobs, with better logging and dependency management:
# /etc/systemd/system/myapp-cleanup.timer
[Unit]
Description=Clean up old uploads daily
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/myapp-cleanup.service
[Unit]
Description=Clean up old uploads
[Service]
Type=oneshot
User=deploy
ExecStart=/var/www/myapp/scripts/cleanup.sh
Enable with sudo systemctl enable --now myapp-cleanup.timer. Check scheduled timers with systemctl list-timers.
Troubleshooting
Service Won't Start
# Check the status and recent logs
systemctl status myapp
journalctl -u myapp -n 50 --no-pager
# Common causes:
# - Wrong ExecStart path (check with `which node`)
# - Permission issues (check User/Group directives)
# - Port already in use (check with `ss -tlnp | grep 3000`)
# - Missing environment variables (check EnvironmentFile path)
Monit Reports Does Not Exist
# Check if the process matching pattern is correct
pgrep -f "node dist/server.js"
# Test Monit configuration
sudo monit -t
# Check Monit's view of the service
sudo monit status myapp
Restart Loop
If Systemd keeps restarting a service that immediately crashes:
# Check the start limit
systemctl show myapp | grep -E "StartLimit|Result"
# Reset the failure counter
sudo systemctl reset-failed myapp
# Temporarily disable restart to debug
sudo systemctl edit myapp
# Add: [Service]
# Restart=no
# Then start manually and watch logs
For a comprehensive approach to deployment reliability, see our deployment checklist and deployment monitoring guide.
DeployHQ automates your entire deployment workflow — from Git push to production, including build scripts, file transfer, and SSH commands that restart your services. Start deploying for free.
Have questions? Reach out at support@deployhq.com or find us on X/Twitter.