Safe Nomad Consul Vault Upgrades: Step-by-Step

Updated 14 November 2026 · Standard formula, shown below

This guide walks you through upgrading HashiCorp Nomad, Consul, and Vault in a rolling fashion, minimizing downtime and reducing the risk of cluster-wide failures. You’ll learn the correct order, version compatibility checks, and how to roll back if something goes wrong. Before starting, ensure you have SSH access to all servers, a recent backup of your cluster state, and a maintenance window if you expect brief service interruptions.

What You’ll Need

SSH access to all server and client nodes in your cluster.
Backups of Consul and Vault data (e.g., consul snapshot save and vault operator raft snapshot save).
Consul, Vault, and Nomad binaries for the target versions, downloaded to each node or accessible via a package manager.
A clear understanding of your current versions and the upgrade path (check official compatibility matrices).
Access to the Nomad, Consul, and Vault CLIs (or the API) to check cluster health.

Step-by-Step

Check current versions and cluster health. Run nomad version, consul version, and vault version on all nodes. Verify all servers are healthy: consul members should show all alive, vault status should show active and standby nodes, and nomad server members should list all servers.
Back up everything. Take a Consul snapshot (consul snapshot save backup.snap), a Vault snapshot (vault operator raft snapshot save backup.snap), and back up the Nomad job definitions and ACL tokens. Store these off-cluster.
Review release notes and upgrade guides. Read the official upgrade guides for each product. Note any breaking changes, required configuration changes, or new features that affect your setup.
Upgrade Consul first. Consul is the foundation for service discovery and KV storage for Nomad and Vault. Upgrade Consul servers one at a time: stop the Consul service, replace the binary, start the service, and check that it joins the cluster and becomes healthy (consul members and consul operator raft list-peers). Wait for the leader to be elected and all servers to be in the ‘alive’ state.
Upgrade Consul clients. After all servers are upgraded, upgrade the Consul agents on every client node (including Nomad clients) one at a time. This ensures the entire cluster speaks the same protocol.
Upgrade Vault servers. Vault should be upgraded after Consul. Upgrade Vault servers one at a time: stop Vault, replace binary, start it, and check vault status. Ensure the active node is healthy and standby nodes are in sync. Do not upgrade all at once; the cluster must maintain quorum.
Upgrade Nomad servers. Nomad servers depend on Consul. Upgrade them one at a time: stop Nomad, replace binary, start it, and check nomad server members. Wait for the leader to be re-elected and all servers to be healthy.
Upgrade Nomad clients. Finally, upgrade the Nomad agents on all client nodes. This is the least risky step, but do it gradually to avoid overwhelming the cluster with rescheduling. After each client upgrade, check that existing allocations are still running and new ones can be scheduled.
Verify the entire stack. After all upgrades, run a full health check: consul members, vault status, nomad server members, and nomad node status. Submit a test job to confirm scheduling works, and verify that Vault secrets are accessible.

Common Mistakes

Upgrading all servers at once, which can cause a loss of quorum and a cluster outage.
Upgrading Nomad before Consul, which can cause Nomad to fail because it expects a newer Consul API.
Ignoring version compatibility. For example, Nomad 1.5 may require Consul 1.14 or later. Check the official compatibility matrix.
Not backing up before starting. If an upgrade fails, you may need to restore from a snapshot.
Skipping the client upgrade. Leaving older clients can cause protocol mismatches and prevent scheduling.

Tips

Use a configuration management tool (Ansible, Terraform) to automate the binary swap and service restart, but still do it node-by-node.
Test the upgrade in a staging environment that mirrors your production setup.
Monitor the cluster during the upgrade using the built-in metrics endpoints (e.g., /v1/agent/metrics).
Have a rollback plan: keep the old binaries and configuration files on each node, and be ready to revert to a snapshot if needed.
Upgrade during low-traffic periods to minimize the impact of any brief service unavailability.

Troubleshooting

If a node fails to join the cluster after upgrade, check the service logs (journalctl -u consul, journalctl -u vault, journalctl -u nomad). Common issues include:

Consul server fails to start: Check that the data directory is writable and that the configuration file is valid. If the version is too old, the binary may refuse to start with an error.
Vault shows sealed after upgrade: This is normal; unseal it using your unseal keys or auto-unseal mechanism.
Nomad server cannot connect to Consul: Verify the Consul address in Nomad’s config and that Consul is healthy. Check that the ACL tokens are valid.
Loss of quorum: If you accidentally upgraded too many servers and lost quorum, you may need to restore from a snapshot and revert to the previous version on the affected nodes.
Allocations stuck in pending: Check Nomad client logs and ensure the client can connect to the servers. Also verify that the client’s Consul agent is healthy.

Frequently Asked Questions

Can I upgrade Consul and Vault in parallel?

It’s not recommended. Upgrade Consul first, then Vault, because Vault depends on Consul for storage and service discovery. Parallel upgrades can lead to unexpected interactions.

Do I need to upgrade all clients if I only use Nomad for batch jobs?

Yes, all clients should be upgraded to match the server version. Older clients may not be able to communicate with newer servers, and the scheduler may not be able to place jobs on them.

What if I skip a minor version? Can I jump from 1.2 to 1.4?

You can, but you must check the upgrade path. Some versions may require intermediate upgrades. Always read the release notes for any version you skip.

How long does a typical upgrade take?

It depends on the size of your cluster. For a small cluster (3 servers, 5 clients), a full upgrade can take 30–60 minutes if you are careful and monitor each step.

Can I roll back after a failed upgrade?

Yes, if you have backups and the old binaries. Stop the service, replace the binary, restore the snapshot if necessary, and start the service. However, rolling back may cause data loss if you’ve written new data since the backup.

Do I need to update the Nomad job files after upgrading?

Not necessarily, but new features may allow you to simplify or improve your jobs. Review the release notes for any breaking changes to job specifications.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.