The Flexomizer

Dave Whitney's Random Thoughts

NAVIGATION - SEARCH

Free Disk Defragmenter

OK, so I've never been very happy with the disk defragmenter that comes with Windows 2000 and XP. It's a "lite" version of Executive Software's Diskeeper. I haven't tried any recent versions of Diskeeper, but the v4 release didn't do a very good job if the disk was heavily fragmented and nearly full. Its basic algorithm is "for each file, find a free slot big enough to hold that file and move the file there. Repeat until bored." Of course, if you haven't got a free slot that's big enough, you're hosed.

I tried Norton's defragger once and it corrupted my NTFS volume. I haven't even dared try a newer version. Things may have improved, so I have no warnings or recommendations there.

Then I found detailed documentation on how the defraggers are implemented here. The site also provided a cheap defragger and source code. Unfortunately, it suffered the same basic design flaw as Diskeeper - if there wasn't enough free space to hold the whole file, you weren't going to get any better.

So, I got to work. Here is the result. My defragger employs the tried and true defragmenting algorithm of "evacuate and place". It first evacuates a large portion of contiguous free space, then starts placing files in the new empty hole. Repeat until defragged. However, in order to do this, it needs to repeatedly iterate over every file on your disk. This makes the overall process quite slow if you have many files, but the algorithm never fails to completely defrag your disk, no matter how full it is (well, you need some free space). My machine at work has a disk full of about 112,000 files and folders taking up about 15GB and it takes less than 8 hours to do a total disk defrag.

It works with any OS-supported file system* on any OS-supported writable media. It will only work on Windows XP and later versions of Windows. It's been tried on XP, Server 2003, Vista, and some prerelease versions of Server 2008 (Longhorn). This program will not work on any other line of Windows (2000, 9x, Win 3.1, etc). There is no 64-bit version at this time. I highly recommend against trying to run this on any 64-bit version of Windows.

Features:

  • Can be run by any user (service needs to be installed by an administrator).
  • Complete whole-disk defragging, including private operating system files.
  • Multiple instances can be run simultaneously (if you have multiple disks).
  • Individual file defragging, with wildcards and recursion. This mode is an opportunistic defragger, marginally better than Diskeeper in that it will at least defrag the file as best it can rather than not at all.
  • Defrags NTFS, FAT** and FAT32** filesystems on any media (HD, Flash, ZipTM/JazTM, Floppy, etc).
  • A "shotgun" mode, where it severely fragments files. Useful for testing or really annoying someone.
  • Individual file fragmentation analysis. Find out where on disk a file resides, and how many fragments it's in.
  • 100% safe and interruptable. This program doesn't actually move the files - the OS does it. Even if you suffer a power failure while it's working, you run no higher risk of file system corruption than you would if you had happened to be saving a file from any application.
  • Boot-time defragmentation.

Future Features:

  • Functionality on 64-bit Windows
  • Defragmentation of NTFS metadata files

Given that Windows "7" has substanitally improved built-in defragmentation, I won't be doing any work to ensure full compatability with that version of Windows. I expect the current version will work as-is on Windows "7", and I don't plan on fixing any issues that are found.

*At this time, the OS only supports defragging on FAT, FAT32, and NTFS volumes. I don't know how far-reaching HPFS volumes are, or if other file system drivers even exist. For this program to work with those filesystems, the filesystem driver needs to support the defragging APIs of the OS.

** The operating system doesn't provide for defragmenting directories on FAT and FAT32 drives. While all your files will be as defragmented as possible, the immobility of directories totally throws a wrench in the process. Unless you really need to be using either of these file systems, I suggest you don't. NTFS is substantially more robust (you rarely need to run chkdsk) and it supports a huge variety of extra features, like volume striping, journaling, transactioning and so on.

Download Defrag.zip (82.04 KB, x86, 23 May 2004)

Comments (70) -

Trackback from John Howard

I'll zip that web server in 420MB - Small Virtual Machine Backups

Reply

Trackback from Dugie's Pensieve

VHD zip backups with VHDmount and 'Whitney' Defrag

Reply

Trackback from Eric Lee - A Humble Blog

Presentation/Demo Tips

Reply

Trackback from Jose Aguilar's Blog

Spring Cleaning or How to Optimize the Performance of Your PC Using a Free Tool

Reply

Trackback from Scott Hanselman's Computer Zen

VM Performance Checklist - Before you Complain that your Virtual Machine is Slow

Reply

Trackback from Analbrei.de

Defragmentierung bei Vista

Reply

Great tool Dave thanks!  Would just like to ask you if you could in the future add a GUI to it? Would make life a bit easier.

Thanks

Garth

Reply

Dave Whitney
Dave Whitney

One thing I'll do first is expose a proper API into the service, thereby making a GUI a possibility. As far as *actual* GUI goes, well, we'll see Smile

Reply

Mr. Whitney,

Thank you for the nice tool.  I have observed two things.

Unless I'm missing something, I cannot get Win2003 to defrag at boot time like I can with WinXP.

The second, using a 16gb disk where about 80% is full and using 4 1.4GB files that are badly fragmented, the process actually didn't defragment everything.  Everything was nicely compacted, but not completely defragmented.  After running your process, I just copied two of the file that were still fragmented and removed the orginals.  Because of the free space, that address the issue.  However, when I run your process again in the hopes of compacting the data, the exact same two files were fragmented again.  I have this in a MS virtual hard drive, but because of its size, I don't believe it would be practical to get it to you.

Anyway, I just wanted to let you know what I've run into.

Otherwise, it is a very nice product.

Kevin

Reply

Richard Johannik
Richard Johannik

It wiil not run at boot time, even though the cmd prompt says it will.

I've run it twice now & both times it failed--error 0xcooooo22

And that was on a drive with ALOT of room

It won't run on my second drive at all-fails immediately.

Reply

Excellent tool! Many thanks indeed.

One question, would it be possible to flag a particular filename so that it always defrags it to the end of the consolidated space? Very useful for defragging a drive daily that has one particular file that always grows numerous times per day, better to have it at the end than the beginning!

Reply

Dave Whitney
Dave Whitney

Adam - If you run the defragger in "file" mode, it'll relocate the recently grown and therefore now slightly fragmented file off to the end (assuming, of course, there's room for it there). Another run and the free space opened up will be filled by other files. If you run the "file" mode on a regular basis, it'll keep the large file at the end of the disk. Note that this trick will break down once the amount of contiguous free space drops below the size of your file.

Reply

Hi Dave

I also get an error with the -s switch which means I can't really even try this (and it came recommended)

The error is

DEFRAGENGINE::FindFile failed with 0xc000000f

Please let me know how to fix this. My windows install is acting quite broken these days anyway so I may need to reinstall at some point soon :/

Reply

Dave Whitney
Dave Whitney

That error code is STATUS_NO_SUCH_FILE, which is really spooky. You should run chkdsk /f and make sure you haven't got filesystem corruptions. I suggest you take a backup first.

Reply


Many thanks Dave, that worked! I run a full pass, wait 20 minutes for the file to grow and then defrag just the file, and it moves to free space at the end.

Is there a possibility you could add a "mirror" option, to do the defrag but place files from the end of the disk backwards instead of the start of the disk forwards? This would allow some fine tuning, by first clearing all the space from the front of the disk, then some manual processing deleting and recreating the pagefile so it is first on the disk, then defrag the windows install and subdirs next to the start, and so on, then a regular defrag for the rest. What do you think, is this feasible?

Thanks again for a great tool Smile

Reply

Dave Whitney
Dave Whitney

Adam - such a feature would be relatively easy to write. However, I've got bigger fish to fry (in the app, and in life in general). Such a feature will have to wait.

Reply

Dave,

I've used your defrag tool on numerous machines and it works great. However, on one machine I get these two errors a number of times:

DEFRAGENGINE::FindFile failed with 0xc000000f
DEFRAGENGINE::ForEachFile failed with 0xc000000f

I did a chkdsk /f as you recommended but still get these errors. Any ideas what could be causing this? I turned off my antivirus and paused the Windows Desktop Search indexer but still get these errors.

--Craig

Reply

Dave Whitney
Dave Whitney

Craig - If you could retry the defrag with the logging level set to something higher (try -l3) it may shed some light on this.

Reply

Dave,

Per your suggestion I ran with -l3 set. I'm not sure I understand any more but would be happy to email you the file (it's around 2.2MB zipped.) Here are two instances where I saw the error:

Counting files...DEFRAGENGINE::FindFile failed with 0xc000000f
DEFRAGENGINE::FindFile failed with 0xc000000f
DEFRAGENGINE::ForEachFile failed with 0xc000000f
done.
Files: 113910
Directories: 12783

...

Recursing on My Private Folder
Searching for (null)
Searching for (null)
DEFRAGENGINE::FindFile failed with 0xc000000f
Searching for (null)
Searching for (null)
DEFRAGENGINE::FindFile failed with 0xc000000f
DEFRAGENGINE::ForEachFile failed with 0xc000000f
Searching for (null)
Searching for (null)
Recursing on NetHood
Searching for (null)

In an attempt to get around this, I did run a boot-time defrag and that worked with no problems.

Reply

Dave Whitney
Dave Whitney

Craig, please send me the log.

Reply

Dave,
I've been using this tool for a year. I have no complaints. Thank you!
My question is: will this work on Windows Vista?

Reply

Dave Whitney
Dave Whitney

Vista: I haven't tried it myself, so I don't know for sure. I've had one person tell me it works fine and antoher tell it doesn't do anything.

Reply

instructions
instructions

just wondering if you might provide some better instructions as to how to operate this program to its fullest. i see that you can install it as a service but it gives no explanation of the advantage of this or why one would want to do it. also something like a boot time defrag isnt explained or afforded proper instruction.

i am about to try the program out, but i will be using trial and error to accomplish my goals when id rather just be able to read it and do it.

thank you though for sharing it with the community Smile

Reply

Dave Whitney
Dave Whitney

The service does the actual defragmentation. The service runs in the security context of the operating system itself which allows it to defragment all the files on NTFS disks. Strictly speaking, you don't *need* to install the service, as it'll make its best effort otherwise. However, you will need to be a machine administrator to defrag at all. If you're defragging an NTFS volume, there are several files (all under the System Volume Information folder) which you won't be able to defragment. Only the service can do that.

Reply

instructions
instructions

okay so is this correct:

i install the service
i go into windows services and start the service
the service is running (idly i suppose?)
when i run the program again to defrag something... the SERVICE is what actually wakes up and takes on the task of defragging at the system level rather than the user level of me running the program?

Reply

Dave Whitney
Dave Whitney

Correct. However, the service will start itself as needed. You do not need to start it yourself, nor do you need to set it to "Automatic" in the Services control panel.

Reply

Fidel Perez
Fidel Perez

Hello Dave,

Thank you very much for you tool, but: I was having some trouble running this on dynamic disks and partitions, your tool support those? I tried you tool since I'm writing a GetUsedSectors bitmap function,  must return the disk or volume bitmap, whit basic disks everything is fine, dynamic, caos. can you show me some light?

Thanks again.

Reply

Fantastic App

I'm kinda holding out for the 64bit version, any ideas when it might fall out of the pipe works ;)

Cheers
Ad

Reply

Hi Dave,

Firstly, thanks for a great tool  - used it quite and bit and it always works great - and does a much better job than Windows Defrag.

Just a quick question - is your code all self-contained? As in, does it use any windows components while defragging, or does it contain everything it needs?

Just curious....

Thanks

Dave, UK

Reply

Dave Whitney
Dave Whitney

The 64-bit version will have to wait until I buy myself 64-bit hardware. Which will be.... sooner or later. Shortly after "sometime."

As for the implementation, the OS provides 3 basic APIs: Get a freespace bitmap for the disk, get the disk sector usage of a file, move a fragment of a file to a specific disk sector. The APIs are "safe" in that any attempt to move file fragments to areas of the disk that are in-use, or even pending being claimed by some other file operation will fail. In other words, using these APIs will not allow disk corruption. I wrote the rest of the algorithms around these 3 APIs.

Reply

Thanks for the swift response Dave....

Reply

Dave,
Great application. Looks like it does a better job than Windows itself.

Thanks

Reply

I've successfully installed but am not using it successfully.

Here's what happens when I run it:

C:\Documents and Settings\Michael>defrag -ds c:

Disk Defragmenter
Copyright (C) 2002-2003 Dave Whitney
http://www.whitneyfamily.org
Press <ESC> to stop
(app built May 23 2004 19:27:58)
(eng built May 23 2004 19:27:54)
DEFRAGENGINE::OpenVolume failed with 0xc0000034
DEFRAGENGINE:LaughingefragDisk failed with 0xc0000034
Counting files...done.
Files: 8
Directories: 11
Free clusters: 15718624
Used clusters: 3818416
Evacuation chunk: 3818416
Evacuating disk clusters 0 thru 3818415
Defragging files into the space just evacuated
100% complete...
No used space on volume. Nothing to do.

Is this what I'm supposed to be seeing? Is my command line not constructed correctly?

Reply

Dave Whitney
Dave Whitney

Mike:

Try running

defrag -d c:

Note the lack of the s. The code is attempting to defrag drive s. You probably haven't got a local fixed drive S:

Reply

Hi Dave,

First - thanks a lot for your wonderful tool. I've been using it for several months and it always works like a charm Smile
Today I've run it on my laptop, which has shown suicidal tendencies lately - once in a while it completely freezes. It started doing so once every few days, and now I find myself rebooting my machine several times a day, sometimes several times in a row... So obviously, we're talking about a really sick machine here. Anyway, when I ran the defrag utility (to defrag my C drive on startup) I got the following message:

DEFRAGENGINE::ForEachFileStream failed with 0xc0000010 while working on G_Backup
DEFRAGENGINE::ForEachFile failed with 0xc0000010
done.

And then it continues to run, so far with no other interruptions.
Any ideas what could be the problem?
Thanks in advance,

Ilan

Reply

Seth Everson
Seth Everson

Hey, thanks for the great software! I love this thing- it works very well.

I would ask for one favor- the possibility of added a 10second cancel delay before the boot-time defrag starts?

I've run into a situation where the boot time defragger seemed to lock on a small partition. I couldn't get rid of the boot-time op, despite it constantly locking.

I don't want anyone else to go through the same situation- any thoughts?

Thanks again,
-Seth

Reply

Dave Whitney
Dave Whitney

I haven't figured out how to detect keypresses in the boot time environment. So, the ability to cancel is pretty limited...

Reply

Is there any chance you would consider publishing the source code?  I see the System Internals source is no longer available.  This is a wonderful tool.  Maybe one of us would recognize a way to make it work for the meta-data files or be willing to contribute in the effort of getting it to work on 64 bit versions.  Possibly even worth a sourceforge.net project if you can find a license you like.

Reply

If you publish the source maybe somebody ports it to *nix. Dual booting is common today but unix/linux do not have good defragger (because of ext-filesystem I know).

Reply

Dave Whitney
Dave Whitney

Trust me, the source code for my defragger is completely useless outside of the Windows environment. Really. Windows provides a huge infrastructure for writing a defragger. Unless *nix provides a similar API (which I highly doubt, given your assertion that there are no defraggers available), my source code will do you no good at all.

Reply

hi i hv been trying to do it for a while now...but i'm not exactly sure what the 1st line in the readme file means
"Copy defrag.exe and defragps.dll to a directory on your path which appears before your windows system directory"

so where should i copy them to?
(1) C:/windows/system32 or (2) C:/windows

and when i do (1)...the system is trying to replace the new defrag.exe with the old one...so i have to enter the "defrag -i" real quick to get the line "The Disk Defragmenter service was successfully installed." but even i got that...1 second later the new file got replaced automatically again with the old one...

please give directions.....thanks!!!

Reply

also, the only file i can see available for download is the one date 23/may/2004...is there a newer version???

many thanks

Reply

Dave Whitney
Dave Whitney

Zeck- you can't copy these files into Windows or Windows/System32 (as you've discovered, the OS undoes your changes). Copy them anywhere else - it really doesn't matter. All you need to remember is to change to that directory before you try to run it. So, say you've got a directory c:\bin. Copy the files into C:\bin. Then when you want to run the app:

c:
cd \bin
defrag blah

etc. You will need to uninstall and then reinstall since the copy you tried to put into c:\windows\system32 isn't there any more and things will get very confused.

Reply

Could this have any adverse effects on games and game files (E.g. Save game files and expansions)?

Reply

Dave,

Thanks for the awesome tool. I'm having a bit of trouble getting it to work in boot mode on Windows 2003 R2 SP2 (32-bit). The command line I'm using is: defrag -d C: -B

Basically on boot nothing seems to happen. The system just boots up like normal, and analyzing the disk with the Windows tool shows it is still fragmented. I've tried it without the -B, and that works fine but leaves some fragmentation as expected. I've also tried the same command (with the -B) on an XP SP2 box and that works just fine.

I've checked the event log and besides the messages indicating that the service started there's nothing else there. I tried adding the -l3 switch but that didn't help. Smile I also verified that the service is set to run as Local System.

Is there some way to force a log file or some other diagnostics during a boot-time defrag? I'm just flummoxed as to why it doesn't work.

Reply

when running "defrag -d c:" i keep getting DEFRAGENGINE::ForEachFileStream failed with 0xc0000103 and DEFRAGENGINE::ForEachFile failed with 0xc0000103 errors and when the defrag is complete, the files are badly fragmented (but free space is contiguous) i have 14% free space so its not a space issue??
any ideas?? what does the error number mean?

Reply

Fraser Simpson
Fraser Simpson

Hi there,

Your defrag tool is awesome, has worked sooooo well on XP PC's, but I can't get the reboot scan to work on Vista Home Premium, it says it's scheduled, but when you restart it just ignores the scan and doesn't do anything....... Frown

Seems to work fine when I start a manual scan whilst in Vista...... any ideas ??

Reply

Hello Dave,

Thank you for your great program, it performs faster and really much better (at a point that comparison becomes quite a non-sens) than the "thing" provided by Windows!!! In fact, it looks like to me now that the Defrag program provided by Windows is no more than a toy... it's only left reason of existence is, with its GUI, to show how your program is good! Smile

Yes, you do not provide GUI, but in a sens that's not that bad, I could concentrate on my work while defragging was quietly and efficiently performed.

THANK YOU!

System used is XP.

Reply

Dave,
I was using your defragger on a computer with SafeBoot.  After defragging SafeBoot complained about missing files and I had to use the recovery process to decrypt the disk.  After talking with support they were guess your program was not honoring the DoNotMove bit that is set on their files.  Something for a future update.
Thanks
Chris

Reply

I just wanted to say thank you for this great little application. I wanted to have a defrag that does the job, and not just do it halfheartedly.

The only thing I wish is some kind of visual feedback while defragging (even character-based in console) but it is not important, of course since the program is good.

Reply

Your defragger has been a life saver on my SageTV PVR. I started to experience some bad stuttering with my shows being spread across the drive, and the built in windows utility wouldn't touch half of them because of space issues. Your handly little utility came to the rescue and my PVR is behaving again. Cheers!

Reply

Ashraf Sabry
Egypt Ashraf Sabry

Greetings,
My system drive is highly fragmented. I want to make a complete defragmentation process, so, boot defragmentation was my choice, but it doesn't work on Windows Vista.
Is there a method to make it work?

Reply

Love the simplicity of the tool and the algorithm.  Any word on a x64 release?  I'd love to use the 64 bit version to help compact my vhds.

-Maurice

Reply


Dave,

Have to agree! Something helpful for VHDs! And an x64 version!

Let me just describe what I ran into: I had 80GB of files, many of which were ISOs, on a 160GB partition. Windows Server Backup (or CompletePC) is much faster at backups, so I used it. Biiig mistake. As you can see, when I restored, I had to restore to a partition at least 160GB large, since it's a VHD image. I tried "shrinking" the partition, but Disk Manager would only give me 4GB back! Ouch! I tried defragging. That did nothing. It would be super, duper helpful, given what behemoths VHDs can be, if you gave this the functionality to "move" files into a more compact space. Windows, as I've discovered, tends to leave them strewn wherever it damn well feels like it.

Reply

An x64 version please!

Reply

Andy Willis
United States Andy Willis

Hello,
I really appreciate this program... it is the best defragger I've found so far.  I only have one issue, how do I remove a boot time defrag?  I issued defrag -f * -s -B
which without the -B runs fine but at boot time I get:
DEFRAGENGINE::OpenVolume failed with 0xc0000034
DEFRAGENGINE::CommonAllFiles failed with 0xc0000034
I have run defrag -d -B and defrag -f c:\* -s -B successfully but each time I first get the above error and subsequent boots still give the error.  I had hoped the 2nd attempts would clear the first but did not.
Andy

Reply

Andy Willis
United States Andy Willis

I forgot... I also tried defrag -B
Andy

Reply

Andy Willis
United States Andy Willis

Finally found the registry entry and removed it.
Andy

Reply

Can you move folders on fat32?

Reply

Dave Whitney
United States Dave Whitney

No. See the small asterisk comment at the bottom of the original post.

Reply

Dave, I'm begging you, 64bit?  I'll offer my time (as if that might help Wink

Reply

paintless repaul
United States paintless repaul

I had use the degfracmenter tool and I get this

DEFRAGENGINE::FindFile failed with 0xc000000f
DEFRAGENGINE::ForEachFile failed with 0xc000000f

is it wrong??





http://www.nodents.com/

Reply

The 0xc000000f error code means it can't find a file. You might try running chkdsk on your disk in case there's some latent corruption and then try the defrag again.

Reply

Any chances on an updated 64-bit version for Server 2003 x64 use?  This would be a huge boon for some of us still running on older Server architecture.

Thank you.

Reply

Jargon Gas
United States Jargon Gas

Just wondering, why exactly does the whole disk defragmentaion go to 100% and then back to 0% bunch of times?

Is that when its evacuating space to put more files in?
-----------------
Do you think you can make a gui for this?
Even a ascii version of one would be nice.

I'll will try to make one for my self, but it probably wont conform to your standards.

Reply

@Jason: When the disk is over half full, the defragger has to do its work in several passes (it can only evacuate chunks that are no bigger than the available free space on the disk). The progress meter shows the progress on the evacuation, and then re-placement of the current chunk. There's a second indicator that shows what percentage of the disk has completed.

As for further features, development has stopped (a long time ago). As I noted at the end of the post, Windows 7 has vastly improved things and now has a preferable built-in defragging system.

Reply

Jargon Gas
United States Jargon Gas

I had actually needed this program because I had a partition with a fragmentation level of 49%. I know, ... sad.

Windows Defragger did completly nothing and some other third party defraggers also did nothing.

This program was able to take down that level of 49% to 2%
-----------------
I have a temporary gui working, It's a bit basic looking but I tested it on a 10% fragmented drive and it worked.

Reply

@Dave -- Thanks for the awesome tool. I've used it thousands of times over the years. It's one of those great tools that does one thing, and does it very well.

I understand your need to stop development. Have you considered releasing it as an open source project, possibly on github?

Reply

Dave Whitney
United States Dave Whitney

Keith: I have not. For starters, it's very tightly coupled to Windows. There's absolutely zero chance of it being useful on any other OS. Secondly, the source is presently in a disaster state - it doesn't compile. I sort of left off mid-thought and haven't given it any thought in years.

Reply

Kirk Barrett
United States Kirk Barrett

Dear Dave, I just wanted you to know that it may be 2013, but I still use your defrag on x86 Windows 2003 machines.  It is superb for VHDs and Servers alike.  This utility defrags and packs better than anything else i've used.  I really appreciate that you wrote it, and put so much brain power into doing the algorithms and code right.  Nice work  THANK YOU.
Kirk

Reply

Add comment

  Country flag

biuquote
  • Comment
  • Preview
Loading