maybe there are thousand utilities like this out there,
this one is different only because i probably will code it

so, it is a spark of idea, because file management in 2012 is a tedious job.

we store lots of pictures, songs, documents, binary files, videos, movies and maybe whole lot stuffs inside our hard disks.

we do lots of backup, we are so scare to loose them, it is better to have more copies of same file.

the worse and chaos place is inside your computer, your hard disk,

and i actually believe, you can understand a person behaviour through the way he stores things inside his pc.

so, what i want to achieve here is,

1. a tool to loop through defined drive, store file name, file size in sqlite database. (unicode is a must), i don't think i need file checksum in the beginning.

2. once we got the data in sqlite db, we let user to go through order by most duplicated file, a click to open all folders in one time, or delete all and leave only 1 copy, or etc feature to manage those duplication.

3. just thinking about able to thumbnail picture files, store thumbnail into another sqlite db.

Sleepsleep, when you are awake, maybe you might consider expand your filecompare utility a bit:

What is needed right now is to have a bit-by-bit compare utility comparing an ISO-file (essentially a DVD image file) to the image that has written into a DVD-R (or DVD-RW)

Disk burning utilities such as Nero has this function built in. Unfortunately, there is no standalone utility that can compare an ISO file with the DVD (or CD) disk itself.

If you have a database of your drives contents then you only need to store the file's hash and associate that with the name and path. After that, finding duplicates is easy with a simple function to sort the hashes and expose the duplicates.

You could also extend this to backup drives to ensure that that backups have a copy of each file by comparing the hashes and exposing singletons that exist in only one place.

I don't know how you can make the hash table be synchronised with the file system updates. This may the the really difficult part: to make sure that the current hashes are up-to-date each time a file is changed, deleted or added.

Perhaps if you use an FS that supports alternate streams then the hash can be put into a new stream with a date/time field to show when the hash was last computed? But that will require you to periodically scan the drive to find outdated hashes. However this suffers from the problem that the normal file attribute of the last modified date/time is writeable by applications and might be a false value. I know that truecrypt can be set to do this and a simple date/time last modified search would not show any new updates.

revolution, maybe this (and this for Linux)?

(I don't have experience with neither of both)

Comparing hash codes does NOT tell you that two files are identical, but it CAN, however, tell you they are not identical (when the hash values do not match). When the hash values do match, however, it is still possible the files are not the same, and so they should be compared bit by bit to determine whether they are, in fact, duplicates. Comparing the file sizes helps, too (if they are different, of course the files are different). But even when the file size and hash match, they can still be different, so a byte comparison is needed.

As revolution pointed out, you need some fool-proof way of making sure that whatever heuristics you use are based upon correct, up-to-date information. Otherwise, you're subject to accidentally destroying data.

Indeed this is theoretically true. But for practical purposes this won't be an issue when using "good" hashes like SHA256, Whirlpool, etc.

sleepsleep,

Beware of hard/symlinks and mount points (in general, reparse points for NTFS).

Use git for that. Fast and reliable.

as for example the one used in forensic tools,
cluster,sectors etc, a general glossary http://www.cnwrecovery.com/html/ntfs_forensic.html
Cheers,

Hashes have traditionally been used to determine whether a file has been changed at all, and the more hash output bits, the greater the confidence (assuming you have a good way of determining the hash algorithm is good). But it is still possible for two totally different files to have the same hash...

You are probably right, I don't know the real probabilities here, but I'm not so sure if this "won't be an issue" or if it "shouldn't be an issue". To me, it still seems safer to perform a bit comparison before deleting a file that a hash comparison says is an exact duplicate. Or at least, in a commercial product, let the end user determine which method to use to determine duplicates, therefore shifting the burden onto the user.

i still think i need such tool, after 5 years,