Finding duplicate files with same filename AND exact same size
You really do not need external utilities for this. You can go quite some distance by chaining a set of standard commands, that … Read more
deduplication
Is there a block-level storage file system?
The S3QL filesystem has block level deduplication. It’s promoted as an S3 cloud storage solution but it also works very nicely on local … Read more
Ways to deduplicate files
Create an md5sum of each file, duplicates md5sums suggest (but doesn’t guarantee) duplicate files.
How to remove duplicate rows based on some columns
In Excel 2013 Select all Data Select Data Tab -> Remove duplicates Check ‘My columns have headers’ if your data has column headers. … Read more
What does a rmlint’s “clone” for btrfs do?
What does a rmlint’s “clone” for btrfs do?
Is there a way to enable reflink on an existing XFS filesystem?
Is there a way to enable reflink on an existing XFS filesystem?
Make tar (or other) archive, with data block-aligned like in original files for better block-level deduplication?
Make tar (or other) archive, with data block-aligned like in original files for better block-level deduplication?
Deduplication on partition level
Your question is a tiny bit confusing due to the term “blocks”, which is a very overloaded word when it comes to disks … Read more
How to find data copies of a given file in Btrfs filesystem?
I have just released a program called fienode (← link) which computes a SHA1 hash of the physical extents of a file. Identical … Read more
What is the best method to remove duplicate image files from your computer?
Don’t Rely on MD5 sums. MD5 sums are not a reliable way to check for duplicates, they are only a way to check … Read more
