ZFS ARC Memory: What Happens When Apps Need RAM?

When an application needs memory on a system running ZFS, the ZFS Adaptive Replacement Cache (ARC) automatically shrinks by evicting cached data, returning that memory to the operating system for the application to use. This is a core design feature of ZFS, not a bug or a failure, and it happens dynamically and transparently in most configurations.

Background: What the ARC Is and Why It Exists

The ARC is ZFS’s primary read cache, sitting in RAM to speed up access to frequently used data. Unlike a traditional filesystem cache that simply stores recently read blocks, the ARC uses an adaptive algorithm that tracks both recency and frequency of access. It maintains two lists for cached data—one for data accessed once (recency) and one for data accessed multiple times (frequency)—and dynamically adjusts how much memory each list gets based on workload patterns.

ZFS uses the ARC for more than just file data. It also caches metadata (directory entries, file attributes, and the indirect blocks that map file offsets to disk locations), which is often the most performance-critical part of a filesystem. The ARC also holds the ZIL (ZFS Intent Log) for synchronous writes, and the dnode cache for object lookups. All of these compete for the same memory pool, and the ARC’s job is to balance them efficiently.

How the ARC Gives Memory Back

The ARC does not hold memory permanently. It is a reclaimable cache, meaning the kernel can ask it to release pages when other subsystems—like application processes, page cache, or kernel structures—need them. On Linux, ZFS registers with the kernel’s shrinker API, which is the same mechanism used by the page cache and other caches. When memory pressure is detected, the kernel calls the shrinker, and the ARC responds by evicting data.

The eviction process is not random. The ARC tracks each cached item’s age and usage, and it evicts the least valuable data first: clean data that hasn’t been accessed recently, then clean data that has been accessed, and finally dirty data (which must be written back to disk before the memory is freed). Metadata is generally kept longer than data because losing it would require more disk I/O to reconstruct.

There is a subtlety: the ARC does not shrink all the way to zero. ZFS maintains a minimum ARC size (default is 64 MB on most systems, or a small fraction of total RAM), below which it will not go, even under heavy pressure. This ensures the filesystem can still function and respond to I/O requests, albeit with reduced caching benefit.

How to Choose the Right ARC Size

By default, ZFS on Linux sets the ARC maximum to 50% of physical RAM, while on FreeBSD it can be up to 75% (subject to other kernel limits). This is a deliberate conservative choice to avoid starving applications. However, you can tune this with the zfs_arc_max parameter (Linux) or vfs.zfs.arc_max (FreeBSD).

When deciding on a custom ARC size, consider:

• Workload type: A database server with a large working set benefits from a bigger ARC; a desktop with many interactive applications may need to keep more RAM free.
• Total RAM: If you have 64 GB, leaving 32 GB for the ARC is often fine; if you have 8 GB, 4 GB might be too much for a busy desktop.
• Other caches: The page cache (for regular files) and the ARC compete. On a NAS serving files over NFS/SMB, the ARC is more valuable; on a general-purpose server, the page cache might be equally important.
• Dedup and compression: If you use deduplication, the dedup table (DDT) lives in the ARC and can consume huge amounts of memory—often more than the data itself. In that case, you may need to reduce ARC size to avoid OOM.

A common strategy is to start with the default, monitor arcstat or arc_summary to see cache hit rates and memory pressure, and then adjust. If you see high cache misses and no memory pressure, increase zfs_arc_max. If you see swapping or OOM kills, decrease it.

Common Problems and Misconceptions

One frequent misconception is that the ARC is “stealing” memory that applications need. In reality, the ARC only uses memory that is otherwise idle, and it gives it back on demand. However, there are real issues that can arise:

• OOM Kills: On Linux, if memory pressure is sudden and extreme, the kernel may invoke the OOM killer before the ARC can shrink fast enough. This is more likely with a very large ARC and a workload that allocates memory in bursts. The fix is to lower zfs_arc_max or set zfs_arc_min to a lower value.
• Slow Eviction: The ARC’s eviction can be slow if there is a lot of dirty data that must be written to disk first. This can cause temporary latency spikes under pressure. Tuning zfs_dirty_data_max can help.
• ARC Size Not Shrinking: On some systems, the ARC may not shrink as fast as expected due to the way the shrinker is implemented. This is a known area of ongoing development in OpenZFS.

Comparison with Other Filesystem Caches

Most filesystems (ext4, XFS, NTFS) use the kernel page cache, which is a simple LRU (Least Recently Used) cache. The page cache is also reclaimable, but it doesn’t have the adaptive frequency awareness of the ARC. The ARC’s advantage is that it can keep frequently used data even if it wasn’t accessed recently, which is common in databases and file servers.

Another difference is that the ARC can be used for metadata and data simultaneously, while the page cache is mostly for data. This makes ZFS more efficient for metadata-heavy workloads like directories with millions of files. However, the ARC is more complex, and its memory usage is more visible to administrators, which is why you see more questions about it.

Costs and Trade-offs

The primary cost of a large ARC is that it reduces the memory available for applications and the page cache. If you set zfs_arc_max too high, you risk OOM kills or swapping. If you set it too low, you get more disk I/O and slower performance for ZFS workloads.

There is also a CPU cost: the ARC’s adaptive algorithm and the LRU lists require some overhead, but this is usually negligible compared to the I/O savings. On very high-throughput systems, the ARC’s lock contention can become a bottleneck, but that is rare and usually only with many CPUs and extremely high I/O rates.

Safety and Best Practices

To avoid memory-related issues with ZFS:

• Always leave some RAM for the OS and applications. A good rule of thumb is to set zfs_arc_max to no more than 70-80% of total RAM on a dedicated storage server, and less on a general-purpose system.
• Monitor memory pressure with tools like free, vmstat, and arcstat. Watch for swapping and OOM events.
• If you use deduplication, be very careful—the DDT can consume more RAM than the actual data. Test with a small dataset first.
• On Linux, consider setting zfs_arc_min to a lower value (e.g., 64 MB) to ensure the ARC can shrink fully if needed.
• Use the primarycache property on datasets to control what gets cached (all, data, metadata, or none) for specific workloads.

Frequently Asked Questions

Q: Does the ARC ever give back memory to applications automatically?
Yes, it does. The ARC is a reclaimable cache, and the kernel’s memory management can shrink it on demand. This is automatic and does not require any action from the administrator.

Q: Why is my ZFS ARC using 50% of my RAM even when I have plenty of free memory?
Because the default maximum is 50% of RAM. The ARC will grow to that limit as it caches data, even if there is no memory pressure. This is normal and improves performance. You can lower zfs_arc_max if you prefer to keep more RAM free.

Q: Can I disable the ARC entirely?
Yes, you can set zfs_arc_max to 0, but this is not recommended because it will drastically reduce ZFS performance. A better approach is to set it to a small value like 64 MB to keep minimal caching.

Q: What happens to cached data when the ARC shrinks?
Clean data is simply discarded—it can be re-read from disk if needed. Dirty data (for synchronous writes) is written to the ZIL or to disk before the memory is freed. No data is lost.

Q: Is it better to use the ARC or the page cache for a file server?
For ZFS datasets, the ARC is generally more effective because it handles both data and metadata adaptively. The page cache is used for non-ZFS filesystems and for ZFS data that is not in the ARC (e.g., if you set primarycache=metadata).

Q: How do I check the current ARC size and hit rate?
Use arcstat (Linux) or arc_summary (Linux/FreeBSD) to see the ARC size, hit rate, and other statistics. These tools are part of the ZFS utilities.

Related

• Crimson Desert Rings List & Locations
• Online Board Games to Play with Friends & Family
• Mirror Tablet Screen to PC: 4 Ways