Shift AI Computing to Off-Peak Hours

Updated 5 November 2026 · Standard formula, shown below

This guide explains how to use workload scheduling to move AI training, inference, and data processing jobs to periods when grid electricity demand is lower. This cuts your energy costs, reduces strain on the power grid, and can lower the carbon footprint of your AI work. You’ll learn the key steps to set up scheduling, what to watch out for, and how to fix common issues.

What You’ll Need

Access to the AI system’s job scheduler (e.g., SLURM, PBS, Kubernetes, or a cloud provider’s scheduler like AWS Batch or Google Cloud Scheduler).
Knowledge of your electricity grid’s peak and off-peak hours. These are often published by your utility or grid operator, and can vary by season.
For carbon-aware scheduling, access to a real-time grid carbon intensity API (e.g., from Electricity Maps or WattTime).
Basic familiarity with your scheduler’s job submission syntax (e.g., sbatch for SLURM, kubectl for Kubernetes).

Step-by-Step

Identify off-peak windows. Check your utility’s time-of-use rates or grid operator’s demand forecasts. Typically, off-peak hours are late night (e.g., 10 PM to 6 AM) and weekends. For carbon-aware scheduling, use an API to find times when the grid’s carbon intensity is lowest.
Choose your scheduling method. Decide between simple time-based scheduling (e.g., start at 11 PM) or dynamic carbon-aware scheduling that picks the best start time based on real-time data.
For time-based scheduling: Use your scheduler’s built-in delay feature. In SLURM, add --begin=23:00 to your sbatch command. In Kubernetes, use a CronJob with the schedule set to your off-peak time. On AWS, use a CloudWatch Events rule to trigger a Batch job at a specific time.
For carbon-aware scheduling: Write a small script that queries a carbon intensity API, finds the next window (e.g., next 6 hours) where the intensity is below a threshold, and then submits your job with that start time. For SLURM, you’d use --begin=2025-01-15T23:30.
Break large jobs into smaller chunks. If your AI workload is a long training run, split it into smaller jobs that can be scheduled independently. This gives you more flexibility to fill off-peak gaps and reduces the risk of a single long job spanning multiple peak periods.
Set up job dependencies. If you have a pipeline (e.g., data preprocessing â†’ training â†’ evaluation), use your scheduler’s dependency features (e.g., SLURM’s --dependency=afterok:jobid) so that each step runs in sequence, but the entire pipeline starts at the scheduled time.
Test with a small job. Before scheduling a large production job, run a small test job to verify that the scheduling works as expected. Check that the job starts at the right time and completes successfully.
Monitor and adjust. After your first scheduled jobs run, review their start times and energy usage. Adjust your off-peak window or carbon threshold based on actual grid conditions and job performance.

Common Mistakes

Assuming off-peak hours are the same everywhere. Off-peak times vary by region, season, and even day of the week. Always check your specific grid’s data.
Ignoring job duration. If your job takes 12 hours and you start it at 11 PM, it will run through the morning peak. Choose a start time that ensures the job finishes before peak hours begin, or split the job.
Not accounting for time zones. If your scheduler and your grid are in different time zones, make sure you’re using the correct local time for the grid.
Overlooking weekends and holidays. Off-peak periods often extend through weekends and holidays. Take advantage of these longer windows for big jobs.
Using carbon-aware scheduling without a fallback. If the API is down or no low-carbon window is available, your job might never start. Always set a maximum delay (e.g., start no later than 6 AM).

Tips

Combine time-based and carbon-aware approaches. For example, you can set a hard start time (e.g., 10 PM) but also check the carbon intensity and delay further if a cleaner window is coming.
Use preemptible/spot instances. On cloud platforms, spot instances are cheaper and often available during off-peak times. They can be interrupted, so design your workloads to be fault-tolerant (e.g., checkpointing).
Leverage checkpointing. Save model checkpoints frequently so that if a job is interrupted or delayed, you can resume from the last checkpoint rather than starting over.
Automate the process. Write a wrapper script that calculates the optimal start time and submits the job automatically. This avoids manual errors and saves time.
Consider energy storage. If you have on-site battery storage, you can charge it during off-peak and run your AI workloads from the battery during peak times, effectively shifting your grid demand.

Troubleshooting

Job doesn’t start at the scheduled time. Check the scheduler’s logs for errors. Verify that the time format is correct (e.g., 24-hour vs 12-hour). Ensure the scheduler’s timezone matches your intended timezone.
Job starts but fails immediately. This could be due to a dependency issue or a missing file. Check your job script for absolute paths and ensure all required data is available at the scheduled time.
Carbon-aware script returns no start time. The API might be unreachable or the threshold too strict. Add a fallback that uses a fixed off-peak time if the API fails. Also, consider relaxing your threshold slightly.
Job is delayed longer than expected. If using a queue, your job might be waiting for resources. Check the queue priority and consider using a lower-priority queue that is less congested during off-peak.
Energy savings are not as expected. Review your actual start time and job duration. You might be starting too late or running too long. Use monitoring tools to track your job’s energy consumption.

Frequently Asked Questions

Is workload scheduling only for large data centers?

No, even a single GPU workstation can benefit from scheduling jobs to run at night. Most schedulers (like SLURM or even cron jobs) work on a single machine.

Does shifting AI computing to off-peak hours really reduce carbon emissions?

Yes, because off-peak periods often have a higher share of renewable energy (e.g., wind at night). By shifting, you reduce the demand on fossil-fuel peaker plants that are used during peak times.

How do I find my grid’s off-peak hours?

Check your utility’s website for time-of-use rates or demand response programs. Grid operators also publish daily demand forecasts. For real-time carbon data, use an API like Electricity Maps.

Can I schedule jobs that need to run interactively?

Interactive jobs (like Jupyter notebooks) are harder to schedule. You can use a tool like screen or tmux to start a session at a scheduled time, but it’s less straightforward. Consider converting to batch jobs.

What if my AI job takes longer than the off-peak window?

Break the job into smaller chunks and schedule them sequentially. Use checkpointing to save progress between chunks. Alternatively, accept that part of the job will run during peak, but you still save on the majority.

Does this work with cloud AI services like AWS SageMaker?

Yes, most cloud providers offer scheduling features. For SageMaker, you can use AWS Step Functions or EventBridge to trigger training jobs at specific times. For managed services, check the documentation for scheduled scaling or batch transforms.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.