Upgrade Nomad Consul Vault Safely in Production
Upgrading HashiCorp Nomad, Consul, and Vault in a production cluster is a high-stakes operation. This guide walks you through a safe, ordered upgrade that minimizes downtime and rollback risk. You will learn the correct upgrade sequence, how to back up state, how to verify each component, and what to do if something breaks.
What You’ll Need
- SSH access to all servers in the cluster (or a configuration management tool like Ansible, Terraform, or Chef).
- Sufficient disk space for backups and new binaries (check with
df -h). - An understanding of your current versions: run
nomad version,consul version, andvault versionon each node. - Access to the HashiCorp release page or your internal artifact repository.
- A rollback plan: keep the old binaries and configuration files in a known location.
Step-by-Step
- Review upgrade notes and compatibility. Before touching anything, read the official upgrade guides for the versions you are moving to. Check for breaking changes, new required configuration fields, and whether the new version supports rolling upgrades from your current version. For example, Consul 1.15+ requires a
gossip_encryptionkey if encryption is enabled. - Back up all state. For Vault, take a snapshot:
vault operator raft snapshot save backup.snap(if using integrated storage) or use your backend’s backup method. For Consul, snapshot the KV store and ACLs:consul snapshot save backup.snap. For Nomad, back up the job definitions and any state in thedata_dir(usually/opt/nomad). Store these backups off-cluster. - Upgrade Consul first. Consul is the foundation for service discovery and the backend for Vault and Nomad. Upgrade Consul servers one at a time, starting with the leader (you can find the leader with
consul operator raft list-peers). On each server: stop the Consul service, replace the binary, verify the configuration, start the service, and wait for it to join the cluster. Checkconsul membersto confirm all nodes are alive. - Verify Consul health. Run
consul operator raft list-peersto ensure all servers are in theleaderorfollowerstate. Runconsul infoto check for errors. Test a service discovery query:curl http://localhost:8500/v1/catalog/services. - Upgrade Vault. Vault should be upgraded after Consul is stable. Upgrade Vault servers one by one. On each: stop Vault, replace the binary, start Vault, and unseal it. Wait for the node to become active (check
vault status). For HA clusters, the active node will step down when you stop it; the next node becomes active. Upgrade the new active node last. - Verify Vault. After all Vault nodes are upgraded, run
vault statuson each to confirm they are sealed/unsealed correctly and that the HA cluster is healthy. Test a read/write to a test secret:vault kv put secret/test foo=barandvault kv get secret/test. - Upgrade Nomad. Nomad depends on Consul and Vault, so it goes last. Upgrade Nomad servers (the
servernodes) one at a time. Stop the service, replace the binary, start it, and wait for it to join the cluster. Then upgrade the client nodes. For each client, drain it first:nomad node drain -enable -self(run on the client), wait for allocations to migrate, then upgrade and re-enable:nomad node drain -disable -self. - Verify Nomad. Run
nomad server membersto see all servers. Runnomad node statusto see clients. Check that jobs are running:nomad job status. If any allocations are failed, investigate. - Update ACL tokens and policies. If you use ACLs, the upgrade may have changed token formats. For Vault, you may need to re-issue tokens. For Consul, ensure the ACL system is still functional. For Nomad, verify that the ACL tokens used by jobs still have the correct policies.
- Monitor for a day. After the upgrade, watch logs and metrics for errors. Check that Consul, Vault, and Nomad are all reporting healthy. Look for any unexpected restarts or leader elections.
Common Mistakes
- Upgrading in the wrong order. Upgrading Nomad before Consul or Vault can break service discovery and secret access.
- Skipping backups. Without a backup, a failed upgrade can mean data loss.
- Upgrading all servers at once. This defeats the purpose of a rolling upgrade and can cause a total outage if the new version has a bug.
- Not reading the upgrade notes. Breaking changes (like Consul’s ACL changes in 1.4+) can cause unexpected failures.
- Forgetting to unseal Vault after restart. If you restart Vault, it will be sealed. You must unseal it manually or via an unseal script.
- Not draining Nomad clients. Draining ensures that workloads are gracefully migrated to other nodes, avoiding downtime for applications.
Tips
- Use a staging environment that mirrors production to test the upgrade process first.
- Automate the upgrade with a configuration management tool, but still do it node-by-node.
- Keep a copy of the old binaries and configuration files on each server for quick rollback.
- Use a load balancer in front of Vault and Consul to avoid client connection issues during upgrades.
- Schedule the upgrade during a maintenance window, even if you expect zero downtime.
- After upgrading, update your documentation and runbooks to reflect the new versions and any configuration changes.
Troubleshooting
Consul servers won’t form a quorum after upgrade. Check that the raft_protocol version is compatible. If you are upgrading across a major version, you may need to set raft_protocol = 3 temporarily. Also verify that the gossip encryption key is the same on all nodes.
Vault is sealed after restart and won’t unseal. Check the unseal keys. If you use auto-unseal (e.g., with a KMS), verify that the KMS is reachable. If you use Shamir, you need at least the threshold number of keys.
Nomad clients fail to register after upgrade. Check the client configuration for any new required fields. Look at the Nomad logs (journalctl -u nomad) for errors. Ensure the client can reach the servers.
ACL tokens stop working. After a major upgrade, token formats may change. Re-create tokens using the new CLI commands. For Consul, check the ACL replication status.
Service discovery returns no results. Check that Consul is healthy and that services are registered. Run consul catalog services to see what’s registered.
Rollback needed. If the upgrade fails, stop the service, replace the binary with the old one, restore the configuration file, and start the service. For Vault, you may need to restore from the snapshot if data was corrupted.
Frequently Asked Questions
Q: Can I upgrade Nomad, Consul, and Vault at the same time?
A: Technically yes, but it is risky. If something breaks, you won’t know which component caused it. The safe approach is to upgrade them sequentially, with verification after each.
Q: Do I need to upgrade Consul before Vault?
A: Yes, if Vault uses Consul as its storage backend. Even if Vault uses integrated storage, Consul provides service discovery that Vault may rely on for HA. Upgrading Consul first ensures the foundation is stable.
Q: How long should I wait between upgrading each node?
A: Wait until the node is healthy and the cluster is stable. This can be a few seconds to a few minutes. Check logs and run health commands before moving to the next node.
Q: What is the best way to back up Vault?
A: Use vault operator raft snapshot save for integrated storage. For Consul-backed storage, use consul snapshot save. Always store backups off-site.
Q: Can I skip a version and upgrade directly to the latest?
A: Check the upgrade path. Some versions require an intermediate upgrade. For example, Consul 1.0 to 1.15 may require going through 1.5 first. Always check the official upgrade guides.
Q: What should I do if a node fails to join the cluster after upgrade?
A: Check the logs for errors. Verify that the node’s configuration is correct and that it can reach the other nodes on the required ports. If it’s a Consul server, check the Raft protocol version.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.