Remove Duplicate Files Linux: Tools & Methods
To remove duplicate files on Linux, you can use command-line tools like fdupes or rdfind, or graphical applications like dupeGuru or czkawka. These tools scan directories, identify files with identical content (not just names), and let you delete or link the duplicates safely.
Why Duplicates Accumulate
Duplicate files appear for many reasons: repeated downloads, backup restores, photo imports, or copying folders without cleaning up. They waste disk space and make searching slower. On a system with limited storage, duplicates can be the difference between having room for updates or not. Even on large drives, duplicate media libraries (photos, videos, music) can consume gigabytes unnecessarily.
How Duplicate Detection Works
Most tools first compare file sizes—files with different sizes can’t be identical. Then they compute a hash (like MD5, SHA-1, or SHA-256) of the file contents. If two files have the same hash, they are almost certainly identical. Some tools also compare file names or metadata first to speed up the scan, but hash comparison is the gold standard for content-based deduplication.
Be aware that hash collisions are theoretically possible but extremely rare. For practical purposes, a SHA-256 match means the files are identical. Tools like fdupes let you choose the hash algorithm.
Command-Line Tools: fdupes and rdfind
fdupes is a classic. Install it with your package manager (e.g., sudo apt install fdupes on Debian/Ubuntu). Basic usage:
fdupes -r /path/to/folderThis lists duplicate files. To delete them interactively, add -d:
fdupes -r -d /path/to/folderIt will ask which file to keep. Use -N to delete all but the first without prompting (be careful!).
rdfind is another powerful tool. It can replace duplicates with hard links or symlinks, saving space without deleting files. Install with sudo apt install rdfind. Example:
rdfind -makehardlinks true /path/to/folderThis replaces duplicate files with hard links to the first occurrence. Hard links work only on the same filesystem and for regular files.
Graphical Tools: dupeGuru and czkawka
If you prefer a GUI, dupeGuru is cross-platform and supports music, pictures, and video specific matching. It lets you preview files and select which to delete. czkawka is a modern, fast Rust-based tool with a clean interface. Both are available in many distros’ repositories or as AppImages.
These tools often provide a preview pane, so you can visually confirm before deleting. They also allow you to exclude certain folders or file types.
Choosing the Right Tool
- For quick, scriptable scans: use
fdupesorrdfind. - For large media libraries: use
dupeGuru(it has special audio fingerprinting). - For a modern GUI with advanced filters: try
czkawka. - For system-wide deduplication: consider filesystem-level tools like
rdfindwith hard links, or evenrmlint(another CLI tool).
Common Pitfalls and Safety
Before deleting, always review the list. A common mistake is deleting files that are hard links already—they share the same inode and don’t waste space. Use ls -i to check inode numbers.
Another pitfall: symlinks are not duplicates of the target file. Tools usually ignore symlinks; if you want to include them, check the options.
Always back up important data before mass deletion. Even with careful review, you might delete a file you need. Use the “dry run� option if available (e.g., fdupes -n lists without prompting, rdfind -dryrun).
Alternatives: Using find and cmp
For a quick, no-install approach, you can use find and cmp to compare files. For example, to find files with the same size and then compare them:
find . -type f -exec md5sum {} + | sort | uniq -w32 -dThis prints MD5 hashes and shows groups with identical hashes. Then you can manually decide. This is less efficient but works anywhere.
Frequently Asked Questions
Will fdupes delete files automatically?
No, by default fdupes only lists duplicates. With -d it prompts you for each group. With -N it deletes without prompting, but only after you confirm the command.
Can I keep one copy in each subfolder?
Yes, but most tools treat all files equally. You can use fdupes -r and then manually choose which to keep. Some tools like czkawka allow you to set a “keep newest� or “keep in first folder� rule.
Are hard links safe?
Hard links are safe as long as you don’t edit one file expecting the other to stay unchanged—they share the same data. If you delete one link, the data remains until all links are removed. They don’t work across filesystems.
What about duplicate file names but different content?
Those are not duplicates by content. Tools like fdupes only consider content, so they won’t list them. If you want to find files with same name, use find.
How do I avoid duplicates in the future?
Use a deduplicating backup tool like restic or borg that stores only unique data chunks. Also, be careful with download managers that create multiple copies.
Is it safe to delete duplicates in system folders?
Generally, avoid deleting files in system directories unless you know what you’re doing. Duplicate libraries in /usr may be intentional (e.g., different versions). Stick to your home directory.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.