Service Won't Start

Overview

A systemctl start fails, or a service that should auto-start on boot never comes up. The fix depends entirely on WHY it failed, which systemctl status and journalctl almost always reveal directly.

The Problem

systemctl status myapp shows failed or inactive (dead) when it should be running.

Investigation

systemctl status myapp             # current state + the last several log lines — start here, always
journalctl -u myapp -b               # full logs for this service, current boot
journalctl -u myapp -p err              # only error-level or worse
journalctl -u myapp --since "10 min ago"

systemctl show myapp | grep -i exec    # confirm the exact ExecStart command being run
systemctl cat myapp                       # view the FULL effective unit file, including any drop-in overrides

# Try running the exact ExecStart command manually, as the configured user, to see the RAW error
sudo -u <ServiceUser> /path/to/exact/exec/start/command

Running the command manually, outside of systemd, is often the fastest way to see an error message that journalctl only shows truncated or wrapped in systemd’s own formatting.

Root Cause Possibilities

• A configuration error — invalid syntax in the app’s own config file, caught only at startup.
• Port already in use — another process is bound to the port this service needs (ss -tulnp | grep :PORT).
• Missing dependency not yet started — the service starts before a database/network dependency is ready, and has no After=/Requires= relationship declared for it.
• Permission issues — the service’s configured User= can’t read a required file or write to a required directory.
• A stale PID file — for Type=forking services, a leftover PID file pointing at a dead process can confuse systemd’s tracking.
• Editing the unit file without daemon-reload — systemd is still running against the OLD in-memory definition.

Resolution

# Port conflict
sudo ss -tulnp | grep :8080          # find what's already using the port
sudo kill <conflicting_PID>             # or reconfigure one of the two services to use a different port

# Missing dependency ordering
sudo systemctl edit myapp
# [Unit]
# After=postgresql.service
# Requires=postgresql.service
sudo systemctl daemon-reload
sudo systemctl restart myapp

# Permission issue
sudo -u <ServiceUser> ls -la /path/to/required/file    # confirm the service user actually has access
sudo chown <ServiceUser>:<ServiceGroup> /path/to/required/file

# Stale PID file (Type=forking services)
sudo rm /var/run/myapp.pid
sudo systemctl start myapp

Prevention

• Always declare correct After=/Requires= relationships for services with real dependencies (databases, network targets) rather than relying on lucky timing.
• Validate configuration files as part of a deploy pipeline (most tools support a --check-config/-t style dry-run flag) before restarting the actual service.
• Remember systemctl daemon-reload after every unit file edit — make it a reflexive part of the edit workflow, not an afterthought.
• Prefer Type=simple/Type=notify over Type=forking where the application supports it — forking services are more prone to stale-PID-file issues.

Summary

systemctl status and journalctl -u <service> reveal the cause of a failed start in the vast majority of cases — configuration errors, port conflicts, missing dependencies, and permission issues account for nearly all real-world instances. Running the service’s exact ExecStart command manually, as its configured user, is the fastest way to see the raw, unfiltered error when systemd’s own logging is ambiguous.

Pro Tip

systemctl cat <service> shows the FULL effective configuration including any systemctl edit drop-in overrides layered on top of the original unit file — always check this before assuming you know what config is actually in effect, especially on a server you didn’t originally configure.