| rdfind(1) | rdfind | rdfind(1) |
rdfind - finds duplicate files
rdfind [ options ] directory1 | file1 [ directory2 | file2 ] ...
rdfind finds duplicate files across and/or within several directories. It calculates checksum only if necessary. rdfind runs in O(Nlog(N)) time with N being the number of files.
If two (or more) equal files are found, the program decides which of them is the original and the rest are considered duplicates. This is done by ranking the files to each other and deciding which has the highest rank. See section RANKING for details.
By default, no action is taken besides creating a file with the detected files and showing the possible amount of saved space.
If you need better control over the ranking than given, you can use some preprocessor which sorts the file names in desired order and then run the program using xargs. See examples below for how to use find and xargs in conjunction with rdfind.
To include files or directories that have names starting with -, use rdfind ./- to not confuse them with options.
Given two or more equal files, the one with the highest rank is selected to be the original and the rest are duplicates. The rules of ranking are given below, where the rules are executed from start until an original has been found. Given two files A and B which have equal size and content, the ranking is as follows:
If A was found while scanning an input argument earlier than B, A is higher ranked.
If A was found at a directory depth lower than B, A is higher ranked (A is closer to the root).
if A and B are found during scanning of the same input argument and share the same directory depth, the one that ranks highest depends on if deterministic operation is enabled. This is on by default, see option -deterministic). If enabled, which one ranks highest is unspecified but deterministic. If disabled, the one that was reported first from the file system is highest ranked.
Searching options etc:
Action options:
General options:
results.txt (the default name is results.txt and can be changed with option outputname, see above) The results file results.txt will contain one row per duplicate file found, along with a header row explaining the columns. A text describes why the file is considered a duplicate:
DUPTYPE_UNKNOWN some internal error
DUPTYPE_FIRST_OCCURRENCE the file that is considered to be the original.
DUPTYPE_WITHIN_SAME_TREE files in the same tree (found when processing the directory in the same input argument as the original)
DUPTYPE_OUTSIDE_TREE the file is found during processing another input argument than the original.
0 on success, nonzero otherwise.
When specifying the same directory twice, it keeps the first encountered as the most important (original), and the rest as duplicates. This might not be what you want.
The symlink creates absolute links. This might not be what you want. To create relative links instead, you may use the symlink (2) command, which is able to convert absolute links to relative links.
Older versions unfortunately contained a misspelling on the word occurrence. This is now corrected (since 1.3), which might affect user scripts parsing the output file written by rdfind.
Avoid manipulating the directories while rdfind is reading. rdfind is quite brittle in that case. Especially, when deleting or making links, rdfind can be subject to a symlink attack. Use with care!
Paul Dreik 2006-2018, reachable at rdfind@pauldreik.se Rdfind can be found at https://rdfind.pauldreik.se/
Do you find rdfind useful? Drop me a line! It is always fun to hear from people who actually use it and what data collections they run it on.
Several persons have helped with suggestions and improvements: Niels Möller, Carl Payne and Salvatore Ansani. Thanks also to you who tested the program and sent me feedback.
1.5.0 (release date 2021-08-13)
This program is distributed under GPLv2 or later, at your option.
| 1.5.0 | Aug 2021 |