How to Troubleshoot a systemd Service That Fails to Start
When a systemd service won’t start, the cause is almost always a configuration error, a missing dependency, or a permissions problem. This guide walks you through a systematic process to find and fix the issue, from checking the service status to reading logs and testing the unit file. You’ll need a Linux system with systemd (most modern distributions) and sudo or root access to inspect and modify services.
What You’ll Need
- A Linux system running systemd (e.g., Ubuntu 16.04+, Debian 8+, CentOS/RHEL 7+, Fedora, Arch)
- Access to a terminal with sudo privileges
- Basic familiarity with the command line
- Optional: a text editor (like nano or vim) to edit unit files
Step-by-Step
- Identify the exact service name. Use
systemctl status service-nameto see the current state. If you don’t know the exact name, list all services withsystemctl list-units --type=service --allor search withsystemctl list-units | grep keyword. - Check the service status. Run
systemctl status service-name. This shows whether the service is active, failed, or inactive, and often includes the last few log lines. Note the exact error message if any. - Read the full journal logs. Use
journalctl -u service-nameto see all logs for the service. Add-n 50to show the last 50 lines, or-fto follow new entries. Look for lines containing “Failed”, “Error”, “Permission denied”, or “not found”. - Verify the unit file syntax. Run
systemd-analyze verify /etc/systemd/system/service-name.service(or the path where the unit file lives). This checks for syntax errors and missing directives. Fix any reported issues. - Check if the service is enabled. If the service is not enabled, it won’t start at boot, but you can still start it manually. Use
systemctl is-enabled service-name. To enable it:systemctl enable service-name. - Attempt to start the service manually. Run
sudo systemctl start service-nameand then immediately check the status again. If it fails, the error message may be more detailed than the previous attempt. - Review the ExecStart command. Look at the unit file with
systemctl cat service-name. Ensure the executable path exists, is executable, and that any arguments are correct. Test the command manually in a terminal (as the same user the service runs as) to see if it works standalone. - Check file permissions and ownership. The service may fail if it cannot read or write to required files or directories. Use
ls -lon those paths and compare with theUser=andGroup=directives in the unit file. Adjust permissions or ownership withchmodandchownif needed. - Inspect dependencies. If the service depends on another service (e.g., a database), ensure that dependency is running. Use
systemctl list-dependencies service-nameto see them. Start any missing dependencies first. - Check for port or resource conflicts. If the service binds to a network port, ensure no other process is using it. Use
ss -tulpn | grep portornetstat -tulpn. Also check for file locks or resource limits. - Edit the unit file if necessary. Use
sudo systemctl edit service-nameto create an override (this is safer than editing the original). Add or fix directives, then save and exit. Reload systemd withsudo systemctl daemon-reload. - After any change, reload and restart. Run
sudo systemctl daemon-reload(required after editing unit files), thensudo systemctl restart service-name. Check status again.
Common Mistakes
- Forgetting to run
daemon-reloadafter editing a unit file. systemd caches unit definitions; without reload, changes are ignored. - Using relative paths in
ExecStart. systemd requires absolute paths to executables. - Not quoting arguments with spaces. Use quotes or escape spaces properly in
ExecStart. - Assuming the service runs as root. If
User=is set, the service may lack permissions to access files or ports. - Checking logs only with
systemctl status. That output is truncated; always usejournalctlfor full logs. - Ignoring the exit code. The status output shows the main process exit code (e.g., exit code 1). That number often points to a specific error.
Tips
- Use
systemd-analyze blameto see how long services take to start, which can help identify slow dependencies. - Test the ExecStart command manually before assuming the unit file is correct. Run it as the same user the service uses.
- Use overrides instead of editing the original unit file.
systemctl editcreates a drop-in that survives package updates. - Check SELinux or AppArmor if the service fails with “Permission denied” even when permissions look correct. Use
ausearch -m avc(SELinux) oraa-status(AppArmor) to see denials. - Look for environment variable issues. If the service needs specific environment variables, set them with
Environment=orEnvironmentFile=in the unit file. - Use
systemctl reset-failedto clear the “failed” state after fixing the issue, otherwise the service may not start until you do.
Troubleshooting
If the service still fails after following the steps, try these additional checks:
- Check the system journal for earlier errors. Sometimes the failure is a symptom of a previous problem. Use
journalctl -p err -bto see all errors since boot. - Run the service in the foreground. Temporarily change
ExecStartto run the command without daemonizing (if the software supports it) to see the exact output. - Check for missing shared libraries. Use
ldd /path/to/executableto see if any dependencies are missing. - Verify the service type. If the unit file has
Type=forking, the process must fork and the parent must exit. If it doesn’t, systemd thinks it failed. UseType=simplefor long-running foreground processes. - Look for timeouts. If the service takes too long to start, it may hit
TimeoutStartSec. Increase it in an override. - Check for cyclic dependencies. If service A requires B and B requires A, neither will start. Use
systemd-analyze dotto visualize dependencies. - If all else fails, search the web for the exact error message. Include the service name and the error text. Often others have solved the same issue.
Frequently Asked Questions
Why does my service show “failed” but no error message?
Sometimes the error is in the logs but not in the status output. Run journalctl -u service-name -n 50 to see the full log. Also check the exit code in the status output; it may point to the issue.
How do I see the exact command that systemd runs?
Use systemctl cat service-name to view the unit file. The ExecStart line shows the command. You can also run systemd-analyze verify to check for issues.
What does “Unit not found” mean?
It means the service name is wrong or the unit file is not in the expected location. Check the spelling and ensure the file exists in /etc/systemd/system/ or /lib/systemd/system/.
Can I start a service without enabling it?
Yes. systemctl start service-name starts it immediately, but it won’t start at boot unless you run systemctl enable.
Why does my service stop immediately after starting?
This often happens with Type=simple services that fork or exit. Check the logs for an error. If the process is supposed to stay in the foreground, ensure the command doesn’t daemonize. Also check if the service is being killed by a timeout or a dependency failure.
How do I make a service restart automatically if it fails?
Add Restart=on-failure (or always) and optionally RestartSec=5 to the unit file. Then reload systemd and restart the service.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.