Dupinator

License: Freeware
Date Added: 22 January, 2013
Category: Scripts / File Manipulation
Author: activestate.com

Point this script at a folder or several folders and it will find and delete all duplicate files within the folders, leaving behind the first file found of any set of duplicates.


Advertisements



 

Point this script at a folder or several folders and it will find and delete all duplicate files within the folders, leaving behind the first file found of any set of duplicates. It is designed to handle hundreds of thousands of files of any size at a time and to do so quickly.It was written to eliminate duplicates across several photo libraries that had been shared between users. As the script was a one-off to solve a very particular problem, there are no options nor is it refactoring into any kind of modules or reusable functions. The script uses a multipass approach to finding duplicate files. First, it walks all of the directories pass in and groups all files by size. In the next pass, the script walks each set of files of the same size and checksums the first 1024 bytes. Finally, the script walks each set of files that are the same size with the same hash of the first 1024 bytes and checksums each file in its entirety.The very last step is to walk each set of files of the same length/hash and delete all but the first file in the set.It ran against a 3.5 gigabyte set of files composed of about 120,000 files, of which there were about 50,000 duplicates, most of which were over 1 megabyte. The total run took about 2 minutes on a 1.33ghz G4 powerbook. Fast enough for me and fast enough without actually optimizing anything beyond the obvious.


Operating Systems:  Python, Windows, Linux, BSD, Solaris, Mac OS


Dupinator Related Downloads
 
Download DP WIPER 1.1

DP WIPER 1.1  Support the following method: Single overwrite with Zeros and pseudoram, RCMP.DoD wipe: US Department of Defense of Defense 5220.22 compliant wiping) and Gutmann's Maximum Security 35 pass wiping.

Download PCMate Free Duplicates Delete 6.6.3

PCMate Free Duplicates Delete 6.6.3  PCMate Free Duplicates Delete is a feature-rich duplicate file detector which is able to easily find & remove the duplicates to help free up your disk space and keep chaos of duplicate files under control.

Download PCTuneUp Free Duplicate File Finder 4.2.5

PCTuneUp Free Duplicate File Finder 4.2.5  PCTuneUp Free Duplicate File Finder is necessary to help you scan the whole computer to locate duplicates because they are difficult to find in Windows Explorer. It can tremendously reduce the search and backup time.

Download PCBooster Free Duplicates Remover 7.3.4

PCBooster Free Duplicates Remover 7.3.4  PCBooster Free Duplicates Remover is able to locate identical files by verifying name, attribute & date of creation and remove them. It also uses file content analysis technology to verify the internal content of each file to ensure the similarity.

Download SmartBuster v.0.97

SmartBuster v.0.97  Uninstaller,Junk buster,Trace eliminator,Duplicate Buster,Task manager,File splitter,Folder size viewer, Fastest floppy disk formatter, File shredder, StartUP manager, Autoshutdown.All in a small and powerfull ,user friendly application.

Download Auslogics Duplicate File Finder 2.3.0

Auslogics Duplicate File Finder 2.3.0  Auslogics Duplicate File Finder is a nifty free tool that with help you get rid of duplicate files. Duplicate files can take up gigabytes of disk space. By using this free utility you can reclaim up to 50% of hard drive space.

Download Weeny Free Duplicate Finder 1.2

Weeny Free Duplicate Finder 1.2  Weeny Free Duplicate Finder is a free duplicate file finder and cleaner software to instantly scan your computer for duplicate files and list them for you to delete.

Download Auslogics Duplicate File Finder 5.0.0

Auslogics Duplicate File Finder 5.0.0  Auslogics Duplicate File Finder is a nifty free tool that with help you get rid of duplicate files. Duplicate files can take up gigabytes of disk space. By using this free utility you can reclaim up to 50% of hard drive space.

Download Duplicate File Finder Mini 3.1

Duplicate File Finder Mini 3.1  Duplicate File Finder Mini (100% FREE) is used to find and delete the large (1 MB+) duplicate files on your hard disks and removable storage devices, so as to recover the valuable disk space. It works very fast (about 1 minute per 300 GB files).

File Manipulation Popular Downloads
 
Download Free File Splitter Joiner 5.5.5

Free File Splitter Joiner 5.5.5  Free file splitter joiner software to split any file into smaller pieces for easier transfer, storage and file management, and then rejoin them later for use.

Download Parsing a full file specification 1.1

Parsing a full file specification 1.1  This function parses a full file specification into tuple of: a) list of drive and folders b) file name c) (last) file extension (including dot)A full file specification consists of drive letter, folders, file name, and extension.

Download php Download Manager 1.1

php Download Manager 1.1  php Download Manager is a script package written in php with a MySQL back-end. The script allows site owners to offer downloads by category. The program generates code to link to categories or individual downloads.

Download basE91 0.6.0

basE91 0.6.0  basE91 is an advanced method for encoding binary data as ASCII characters. It is similar to UUencode or base64, but is more efficient.The overhead produced by basE91 depends on the input data.

Download aria2 0.11.4

aria2 0.11.4  aria2 is a download utility with resuming and segmented downloading. Supported protocols are HTTP/HTTPS/FTP/BitTorrent. It also supports Metalink version 3.0. As of 0.10.

Download Progress bar class 1.0

Progress bar class 1.0  Here is a little class that lets you present percent complete information in the form of a progress bar using the '#' character to represent completed portions, space to represent incomplete portions, and the actual percent done (rounded to integer)

Download PKTorrents 0.1b 1.0

PKTorrents 0.1b 1.0  PKTorrents has the functionality to crawl the top torrent sites, Meganova, Mininova, Piratebay, Snarf, Torrentportal, Torrentspy and adding 1000's of torrents to the db as it gos using the well know ibitzy torrent spider/crawler!!

Download MyDMS 1.7.0

MyDMS 1.7.0  MyDMS is an open-source, web-based document management system written in PHP and supported by an SQL database. Originally coded by Markus Westphal, MyDMS provides document meta-data, version control, security and easy access to your documents.

Download PKZip library for PHP 0.4.1

PKZip library for PHP 0.4.1  PKZip library for PHP contains a set of functions to create PKZip (Winzip) files in PHP.

Submit Reviews for Dupinator
- required fields
     

Related Freeware