Backbone and Head Learning Rates: Should They Differ?

Yes, you should generally use different learning rates for your model’s backbone and head. The backbone (or feature extractor) is typically pre-trained and needs a smaller learning rate to avoid destroying its learned representations, while the head (the task-specific layers) is randomly initialized and benefits from a larger learning rate to converge quickly.

Formula

The core idea is to set two separate learning rates: one for the backbone (LR_backbone) and one for the head (LR_head). A common practice is to use a ratio, often 10:1, where the head has the larger rate. The effective learning rate at any step s during a linear warmup is:

LR_eff(s) = LR_max * (s / warmup_steps) for s < warmup_steps, then LR_max after warmup.

Here LR_max is the peak learning rate for that part (backbone or head), s is the current step, and warmup_steps is the number of steps over which the rate linearly increases from a small fraction (e.g., 0.1×) to the full value. The same formula applies independently to both parts.

Worked Example

Suppose you set backbone_lr = 0.0001 and head_lr = 0.001 (a 10× difference). With a warmup of 1000 steps, at step 500 the effective backbone learning rate is:

1. Compute the warmup fraction: 500 / 1000 = 0.5.
2. Multiply by the max backbone rate: 0.5 × 0.0001 = 0.00005.
3. For the head: 0.5 × 0.001 = 0.0005.

So at step 500, the head is learning at 10× the backbone rate, matching the ratio you set. After step 1000, the rates become the full 0.0001 and 0.001.

Why Different Rates Matter

Most modern vision and NLP models use a backbone pre-trained on large datasets (e.g., ImageNet, Wikipedia). That backbone already contains general features—edges, textures, shapes, or syntactic patterns. If you update it with a high learning rate, you risk “catastrophic forgetting,� where the model overwrites those useful features and degrades performance, especially when your target dataset is small.

The head, in contrast, is usually randomly initialized. It must learn the specific mapping from features to your task (e.g., classifying 10 custom classes). A larger learning rate helps it converge faster and escape poor local minima. Using the same rate for both forces a compromise: too high for the backbone, too low for the head.

How to Choose the Right Ratio

There is no universal best ratio, but these guidelines are widely used:

• Start with a 10:1 ratio (head 10× backbone). This works for many fine-tuning tasks (e.g., Faster R-CNN, BERT fine-tuning).
• For very small datasets, use a larger ratio (e.g., 20:1) or freeze the backbone entirely (ratio = âˆ�).
• For large datasets, a smaller ratio (e.g., 2:1 or even 1:1) can be better because the backbone has enough data to adapt without forgetting.
• When the backbone is randomly initialized (not pre-trained), use the same learning rate for both, since there is no pre-trained knowledge to protect.

Always monitor validation loss. If the backbone is overfitting (training loss drops but val loss rises), reduce its learning rate or freeze it. If the head is underfitting (both losses high), increase its learning rate.

Common Implementation Patterns

In practice, you don’t manually set two optimizers—you give the optimizer a list of parameter groups with different learning rates. For example, in PyTorch:

optimizer = torch.optim.SGD([
    {'params': backbone.parameters(), 'lr': 1e-4},
    {'params': head.parameters(), 'lr': 1e-3}
])

In TensorFlow/Keras, you can use separate optimizers for the backbone and head, or use a custom schedule that applies different decay rates. Many libraries (e.g., Hugging Face Transformers) automatically set a lower learning rate for the backbone when you pass a learning_rate and layerwise_decay parameter.

Another common pattern is layer-wise learning rate decay, where you apply a decay factor (e.g., 0.9) to each layer’s learning rate as you go deeper into the backbone. This is more granular than a single backbone/head split and can improve performance on tasks like object detection.

Common Mistakes and Pitfalls

• Forgetting to set the backbone to train mode when you intend to fine-tune it. If you leave it frozen, the different rates have no effect.
• Using the same learning rate schedule for both, but with different initial rates—this is fine, but ensure the schedule (e.g., decay) is applied to both, not just one.
• Setting the backbone rate too high (e.g., same as head). This often causes the model to diverge or forget pre-trained features, especially with small data.
• Ignoring batch size. If you change batch size, the effective learning rate changes. Scale the learning rate proportionally (e.g., double batch size → double LR) to maintain similar dynamics.
• Not using warmup. A short warmup (5-10% of steps) prevents large updates at the start, which is especially important when the head is randomly initialized and the backbone is pre-trained.

When Not to Use Different Rates

There are cases where a single learning rate is simpler and works just as well:

• Training from scratch: If your backbone is not pre-trained, there is no reason to treat it differently. Use one rate for all layers.
• Very large datasets: When you have millions of examples, the backbone can safely adapt with a higher rate because the data provides strong gradients that prevent forgetting.
• Small fine-tuning adjustments: If you only need to adapt to a very similar task (e.g., fine-tuning a sentiment model on a new product review dataset), a low single rate may suffice.
• When the backbone is tiny: If the backbone is only a few layers, the risk of forgetting is low, and a single rate is simpler.

Still, even in these cases, a different-rate setup rarely hurts—it just adds a hyperparameter to tune.

Frequently Asked Questions

What is a typical learning rate for the backbone vs head?

For a pre-trained backbone (e.g., ResNet on ImageNet), a common backbone learning rate is 1e-4 to 1e-5, while the head uses 1e-3 to 1e-2. The exact values depend on the optimizer (Adam often uses lower rates than SGD with momentum) and the batch size.

Can I use different optimizers for the backbone and head?

Yes, you can, but it’s usually unnecessary. Using the same optimizer with different learning rates is simpler and works well. If you do use different optimizers (e.g., Adam for the head, SGD for the backbone), be prepared to tune both sets of hyperparameters, which can be time-consuming.

Should I freeze the backbone entirely?

Freezing the backbone (setting its learning rate to 0) is a valid strategy when your dataset is very small (e.g., a few hundred images) or when the pre-trained features are already well-suited to your task. It saves computation and prevents overfitting. However, for larger datasets, fine-tuning usually yields better accuracy.

Does this apply to transformers (e.g., BERT, ViT)?

Yes, the same principle applies. For transformer models, many implementations use a lower learning rate for the pre-trained encoder and a higher one for the task-specific classification head. Some also use a layer-wise decay across the encoder’s layers, which is a more fine-grained version of the same idea.

How do I implement different learning rates in Keras?

In TensorFlow/Keras, you can either use two separate optimizers (one for the backbone layers, one for the head) and combine their updates manually, or use a custom learning rate schedule that returns different rates for different layers. The simplest is to train the backbone with a low LR and the head with a high LR by passing a list of (layer, optimizer) pairs to the model.compile method—though this is not natively supported, so many use the PyTorch approach instead.

What if my model has more than one head (multi-task)?

You can assign each head its own learning rate, or use a shared rate for all heads. The rule of thumb is the same: heads are task-specific and can use a higher rate, while the shared backbone uses a lower one. If one head is more important, you can give it a higher rate.

Related

• Nights in the Forest Diamonds: Are They Free?
• Budget Gaming Monitors: Is Now a Good Time to Buy?
• Arc Circumference Calculator: Find the Arc Length

in t | Total training steps | steps | 10000 | 1
in b | Backbone learning rate | - | 0.0001 | 0
in h | Head learning rate | - | 0.001 | 0
out ratio | Head/Backbone LR ratio | - | h / b | 2
out b_lr | Backbone LR (linear warmup) | - | b * 0.1 | 6
out h_lr | Head LR (linear warmup) | - | h * 0.1 | 6