Leap Second Server Prep: Step-by-Step Guide
This guide walks you through preparing your servers and databases for the end of leap seconds, a change proposed by the International Bureau of Weights and Measures (BIPM) that could take effect as soon as 2035. You’ll learn how to assess your infrastructure, update time synchronization, patch databases, and test your systems to avoid timestamp-related errors. While the exact timeline is not finalized, proactive preparation ensures your services remain accurate and resilient.
What You’ll Need
- Administrative access to all servers (SSH or console)
- List of all servers and databases in your environment
- Access to your time synchronization configuration (e.g., NTP, chrony, systemd-timesyncd)
- Database credentials for patching and testing (read-only where possible)
- A test environment that mirrors production
- Monitoring and alerting tools to verify time sync
- Backup of critical databases before any changes
Step-by-Step
- Inventory your systems. Create a complete list of all servers, virtual machines, and database instances. Note the operating system, database version, and whether they rely on leap second handling (e.g., POSIX systems, Oracle, PostgreSQL, MySQL).
- Understand the proposed change. The current plan is to stop inserting leap seconds by 2035, allowing the difference between UTC and UT1 to grow. Familiarize yourself with how your OS and database handle leap seconds today (most use the “smeared” approach via NTP).
- Audit current time synchronization. Check your NTP configuration. On Linux, run
timedatectlor inspect/etc/ntp.confor/etc/chrony.conf. Ensure all servers use a consistent time source and that leap second handling is enabled (e.g.,leapsecmodein chrony). - Update time synchronization software. Upgrade NTP, chrony, or systemd-timesyncd to the latest versions that support the new leap second policy. For example, chrony versions after 4.0 allow you to set
leapsecmodetosystemorsmear. Test these settings in your test environment. - Patch databases. Apply the latest patches for your database systems. Oracle, PostgreSQL, and MySQL have all released updates that handle leap second removal. For Oracle, ensure you are on a version that supports the
LEAP_SECOND_ENDparameter. For PostgreSQL, check theleapsecondGUC. For MySQL, the time zone tables may need updating. - Adjust application code. Review any code that assumes a fixed number of seconds per day or uses
datefunctions that might be affected by leap seconds. Look for hard-coded offsets or manual leap second calculations. - Test in a staging environment. Simulate the end of leap seconds by setting your test servers to the new policy. Use the
datecommand or a script to verify that timestamps behave as expected. For databases, run queries that involve time arithmetic to ensure no off-by-one errors. - Monitor and verify. After patching, monitor your servers for any time sync anomalies. Use
ntpq -porchronyc trackingto confirm that the system clock is stable. Set up alerts for any jump or drift. - Update documentation. Record the changes you made, including software versions and configuration settings. This helps future administrators understand your setup.
- Plan for the transition day. When the change officially happens, have a rollback plan. Keep old binaries and configs ready, and ensure backups are current.
Common Mistakes
- Ignoring the change until it happens. Waiting until 2035 is risky; you’ll be scrambling to patch and test under pressure.
- Assuming all systems handle it the same. Different OSes and databases have different mechanisms. What works for Linux may not work for Windows or mainframe systems.
- Forgetting about embedded systems. IoT devices, network equipment, and other firmware-based devices may not be patchable easily. They might need to be replaced.
- Not testing with actual leap second scenarios. Simply upgrading software without simulating the end of leap seconds can miss subtle bugs.
- Overlooking time zone databases. The IANA time zone database (tzdata) may need updates to reflect the new policy. Update it on all servers.
Tips
- Use the “smearing” approach in NTP to make the transition seamless. This spreads the leap second across a 24-hour window, avoiding a 61-second minute.
- Keep your time synchronization software up to date regularly, not just for this change.
- Document your time source hierarchy (stratum levels) to ensure redundancy.
- Consider using a centralized time server with GPS or atomic clock reference for accuracy.
- Automate the detection of servers that are out of sync using monitoring tools like Prometheus or Nagios.
Troubleshooting
Time jumps after patching: If your server clock jumps forward or backward after applying updates, check the NTP configuration. Ensure that leapsecmode is set correctly and that the time source is reachable. Restart the NTP service and verify with chronyc tracking.
Database timestamp errors: If queries return unexpected results, check the database’s time zone settings and ensure the tzdata tables are updated. For PostgreSQL, run SELECT * FROM pg_timezone_names; to see if the time zones are correct. For Oracle, check the LEAP_SECOND_END parameter.
Application failures: If an application crashes or misbehaves, review its logs for time-related errors. Look for code that uses gettimeofday() or clock_gettime() and consider updating it to use the new APIs.
NTP synchronization fails: If NTP cannot sync, check firewall rules and network connectivity to your time servers. Verify that the server’s clock is not too far off (more than 1000 seconds) which can cause NTP to refuse to sync.
Unexpected behavior in logs: If log timestamps appear off, check the system time zone and ensure that the leap second handling is consistent across all components.
Frequently Asked Questions
Will the end of leap seconds affect my servers immediately?
No, the change is not expected until 2035, but it’s wise to prepare early. The actual transition may be gradual, and your systems should be ready to handle the new time scale.
What is the difference between UTC and UT1?
UTC is the coordinated universal time, which is based on atomic clocks and currently includes leap seconds. UT1 is the mean solar time, which is based on the Earth’s rotation. The difference between them is currently kept within 0.9 seconds by leap seconds.
How do I know if my database supports the new leap second policy?
Check the release notes for your database version. Most major databases have announced support for the end of leap seconds. For example, Oracle 21c and later, PostgreSQL 16 and later, and MySQL 8.0.30 and later have implemented changes.
What is “smearing” in the context of leap seconds?
Smearing is a technique used by NTP servers to distribute the leap second across a longer period, such as 24 hours, so that the clock never shows a 61-second minute. This avoids sudden jumps and is used by Google and other large providers.
Do I need to update my hardware RTC (real-time clock)?
Yes, you should ensure that the BIOS/EFI clock is set to UTC and that the operating system is configured to interpret it correctly. Most modern systems handle this automatically, but it’s good to verify.
What happens to existing leap second data in databases?
Existing data with timestamps that include leap seconds will remain valid. The change affects future timekeeping. You may need to adjust algorithms that calculate durations across the transition point, but historical data is unaffected.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.