ThisCalculator

Everyday

How to Troubleshoot a systemd Service That Fails to Start

Updated · Standard formula, shown below

When a systemd service won’t start, the cause is almost always a configuration error, a missing dependency, or a permissions problem. This guide walks you through a systematic process to find and fix the issue, from checking the service status to reading logs and testing the unit file. You’ll need a Linux system with systemd (most modern distributions) and sudo or root access to inspect and modify services.

What You’ll Need

  • A Linux system running systemd (e.g., Ubuntu 16.04+, Debian 8+, CentOS/RHEL 7+, Fedora, Arch)
  • Access to a terminal with sudo privileges
  • Basic familiarity with the command line
  • Optional: a text editor (like nano or vim) to edit unit files

Step-by-Step

  1. Identify the exact service name. Use systemctl status service-name to see the current state. If you don’t know the exact name, list all services with systemctl list-units --type=service --all or search with systemctl list-units | grep keyword.
  2. Check the service status. Run systemctl status service-name. This shows whether the service is active, failed, or inactive, and often includes the last few log lines. Note the exact error message if any.
  3. Read the full journal logs. Use journalctl -u service-name to see all logs for the service. Add -n 50 to show the last 50 lines, or -f to follow new entries. Look for lines containing “Failed”, “Error”, “Permission denied”, or “not found”.
  4. Verify the unit file syntax. Run systemd-analyze verify /etc/systemd/system/service-name.service (or the path where the unit file lives). This checks for syntax errors and missing directives. Fix any reported issues.
  5. Check if the service is enabled. If the service is not enabled, it won’t start at boot, but you can still start it manually. Use systemctl is-enabled service-name. To enable it: systemctl enable service-name.
  6. Attempt to start the service manually. Run sudo systemctl start service-name and then immediately check the status again. If it fails, the error message may be more detailed than the previous attempt.
  7. Review the ExecStart command. Look at the unit file with systemctl cat service-name. Ensure the executable path exists, is executable, and that any arguments are correct. Test the command manually in a terminal (as the same user the service runs as) to see if it works standalone.
  8. Check file permissions and ownership. The service may fail if it cannot read or write to required files or directories. Use ls -l on those paths and compare with the User= and Group= directives in the unit file. Adjust permissions or ownership with chmod and chown if needed.
  9. Inspect dependencies. If the service depends on another service (e.g., a database), ensure that dependency is running. Use systemctl list-dependencies service-name to see them. Start any missing dependencies first.
  10. Check for port or resource conflicts. If the service binds to a network port, ensure no other process is using it. Use ss -tulpn | grep port or netstat -tulpn. Also check for file locks or resource limits.
  11. Edit the unit file if necessary. Use sudo systemctl edit service-name to create an override (this is safer than editing the original). Add or fix directives, then save and exit. Reload systemd with sudo systemctl daemon-reload.
  12. After any change, reload and restart. Run sudo systemctl daemon-reload (required after editing unit files), then sudo systemctl restart service-name. Check status again.

Common Mistakes

  • Forgetting to run daemon-reload after editing a unit file. systemd caches unit definitions; without reload, changes are ignored.
  • Using relative paths in ExecStart. systemd requires absolute paths to executables.
  • Not quoting arguments with spaces. Use quotes or escape spaces properly in ExecStart.
  • Assuming the service runs as root. If User= is set, the service may lack permissions to access files or ports.
  • Checking logs only with systemctl status. That output is truncated; always use journalctl for full logs.
  • Ignoring the exit code. The status output shows the main process exit code (e.g., exit code 1). That number often points to a specific error.

Tips

  • Use systemd-analyze blame to see how long services take to start, which can help identify slow dependencies.
  • Test the ExecStart command manually before assuming the unit file is correct. Run it as the same user the service uses.
  • Use overrides instead of editing the original unit file. systemctl edit creates a drop-in that survives package updates.
  • Check SELinux or AppArmor if the service fails with “Permission denied” even when permissions look correct. Use ausearch -m avc (SELinux) or aa-status (AppArmor) to see denials.
  • Look for environment variable issues. If the service needs specific environment variables, set them with Environment= or EnvironmentFile= in the unit file.
  • Use systemctl reset-failed to clear the “failed” state after fixing the issue, otherwise the service may not start until you do.

Troubleshooting

If the service still fails after following the steps, try these additional checks:

  • Check the system journal for earlier errors. Sometimes the failure is a symptom of a previous problem. Use journalctl -p err -b to see all errors since boot.
  • Run the service in the foreground. Temporarily change ExecStart to run the command without daemonizing (if the software supports it) to see the exact output.
  • Check for missing shared libraries. Use ldd /path/to/executable to see if any dependencies are missing.
  • Verify the service type. If the unit file has Type=forking, the process must fork and the parent must exit. If it doesn’t, systemd thinks it failed. Use Type=simple for long-running foreground processes.
  • Look for timeouts. If the service takes too long to start, it may hit TimeoutStartSec. Increase it in an override.
  • Check for cyclic dependencies. If service A requires B and B requires A, neither will start. Use systemd-analyze dot to visualize dependencies.
  • If all else fails, search the web for the exact error message. Include the service name and the error text. Often others have solved the same issue.

Frequently Asked Questions

Why does my service show “failed” but no error message?

Sometimes the error is in the logs but not in the status output. Run journalctl -u service-name -n 50 to see the full log. Also check the exit code in the status output; it may point to the issue.

How do I see the exact command that systemd runs?

Use systemctl cat service-name to view the unit file. The ExecStart line shows the command. You can also run systemd-analyze verify to check for issues.

What does “Unit not found” mean?

It means the service name is wrong or the unit file is not in the expected location. Check the spelling and ensure the file exists in /etc/systemd/system/ or /lib/systemd/system/.

Can I start a service without enabling it?

Yes. systemctl start service-name starts it immediately, but it won’t start at boot unless you run systemctl enable.

Why does my service stop immediately after starting?

This often happens with Type=simple services that fork or exit. Check the logs for an error. If the process is supposed to stay in the foreground, ensure the command doesn’t daemonize. Also check if the service is being killed by a timeout or a dependency failure.

How do I make a service restart automatically if it fails?

Add Restart=on-failure (or always) and optionally RestartSec=5 to the unit file. Then reload systemd and restart the service.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.