I wonder if anyone else experiences this problem: Every time I save a new mpeg file to my video HDD, the file becomes fragmented, no matter how much open space is on the HDD (unless the drive is empty). I have had to do some projects 3 times because of video or audio glitches in the files caused, I think, by fragmentation. (I even burn at 4x, so burn speed is not the problem.)
Today, for instance, here is what happened:
I copied a program from VHS to the HDD of my standalone DVD recorder, then copied from that HDD to a transfer DVD.
Then I imported from the TS directory of the transfer DVD, using MF4. Before the import, I checked the space on the target drive. It was a 50gig partition on a 100 gig drive. There were only 2 files on it, totalling 8 gig. 42 gig free, with contiguous files at the beginning of the desk, lots of white space, then another group of contiguous files near the end of the disc. (According the graphics of the defragmentation analyzer; these were actually the two separate files that the disc contained). Plenty of room all around.
After the import, I checked the drive again with the defrag utility, and the file had been split into 7 segments. I defragged the drive again before editing the file in VideoReDo. Checked the drive when VRD was finished, and its output file had been split into 6 fragments. Had to defrag again before authoring the DVD.
There is simply no excuse for this. Each defrag takes an hour, even with only 8-12 gig of files on the drive. For some reason the XP system and the NTFS file system are stupid when it comes to large files. They start the file in an area of the disc where there may not be quite enough room, then have to break the file up, even though the copying/editing programs are reporting the file size ahead of tiime. The system should have the brains to put a 4 gig file in an area of the disc that has more than 4 gig open space, but it doesn't.
I have a third-party utility that does a better job of defragging than the built-in utility, but the problem still occurs. What I need is a utility that also allows the user to force Windows to locate large files close to each other, in sequence, near the beginning of the drive, so they aren't split randomly.
Is my only option to erase everything on the HDD, so it is completely empty, before I start any new project?
Anyone have any ideas, or know of utilities that do a better job of defragging? Mad Drummer? Steve? H.T.?
R
Defragging on NTFS volumes
-
maddrummer3301
- Posts: 2507
- Joined: Fri Dec 10, 2004 10:24 pm
- Location: US
-
maddrummer3301
- Posts: 2507
- Joined: Fri Dec 10, 2004 10:24 pm
- Location: US
I've never actually noticed that particular problem...
A little fragmentation is no big deal. So what if the drive's head has to reposition itself a dozen times during a several-gigabyte read or write?
But, fragmentation bugs me too! And it bugs me that the Windows defrag utility doesn't 100% defrage the drive! It disturbs my "sense of order" or something. But, that's just an emotional response...
...Trying.... to be... logical...
P.S. You would never believe the"sense of order" shtuff if you saw my home office!
A little fragmentation is no big deal. So what if the drive's head has to reposition itself a dozen times during a several-gigabyte read or write?
But, fragmentation bugs me too! And it bugs me that the Windows defrag utility doesn't 100% defrage the drive! It disturbs my "sense of order" or something. But, that's just an emotional response...
...Trying.... to be... logical...I believe Diskeeper is the gold standard. I think they were selling defrag programs before Windows came with one... Norton/Symantec System Works has one too. They've been at it a long time too... Defrag was one ot the original Norton Utitilies for DOS!Anyone have any ideas, or know of utilities that do a better job of defragging?
P.S. You would never believe the"sense of order" shtuff if you saw my home office!
[size=92][i]Head over heels,
No time to think.
It's like the whole world's
Out of... sync.[/i]
- Head Over Heels, The Go-Gos.[/size]
No time to think.
It's like the whole world's
Out of... sync.[/i]
- Head Over Heels, The Go-Gos.[/size]
-
maddrummer3301
- Posts: 2507
- Joined: Fri Dec 10, 2004 10:24 pm
- Location: US
-
nyco_ork
Guys, thanks for jumping in. Here are more details.
I have two phsyical HDDs I use during authoring, both internal. Both drives spin at 7200 rpm.
Disc 0 is partitioned into three parts: Drive C for the system files, Drive D for programs, and drive E for normal data, such as word processing files. The phyical drive is 60 gig, and each partition is about 20 gig.
Disc 1 is partitioned into 2 drives, tagged as F and H (drive G is the DVD burner). The physical drive has 100 gig, and each partition is 50 gig. I use both these partitions for video files.
I have 1024 gig of memory, and a 2048 gig pagefile. I have MF4 write its temp files to a directory on C: called \DVD\Temp. In order to leave enough space for the temp files, so they won't cause head jumps, I've moved the pagefile to the E partition, to get it out of the way. The system was parking the pagefile right in the middle of my clear space on the C drive (over 9 gig, plenty for burns; the pagefile reduced that to 7+ gig.) I keep the system drive defragged, so the temp files have a clear space.
I also have an outboard drive, a 250 gig Maxtor, partitioned into 3 partitions, that I use for storing archives of the video projects. It is not used during authoring or burning, and is not even turned on. It is connected by Firewire (which I find handles these big files better than USB 2.0)
The only drives being monitored by System Restore are C and D. I have it turned off for all other drives.
Maybe, as one of you says, the fragmentation is no big deal, but I'm trying to reduce every possibilty of the head jumping during a burn. Even a momentary jump seems to cause dropped frames.
For instance, the project I redid yesterday had about 15 bad spots, some lasting a good two or three seconds, in the 2nd version. When I reauthored it and reburned it at 4x, after defrragmenting, it was perfect.
My big book on Windows XP suggests, for burning CDs (they don't mention DVDs), creating a separate partition for the CD image and temp files, and always cleaning that drive of files before doing a burn, in order to avoid fragmentation that might compromise the burn. I guess they would give the same advice for this situation. Sure is a waste of a big drive.
Today I imported the 2nd half of the same project (it spans two discs), and the same thing happened. According to the defrag analysis before the import, I had two good clear sections of the drive, both large enough to handle a 4 gig file with ease. But the import was fragmented into six parts. The VideoReDo edit was also fragmented into six parts. I think Windows is fragmenting the files regardless of drive space; maybe the system simply can't handle a file that large, and creates fragments as it breaks the file into sections that the system can handle. I'm beginning to think that's what's really going on here.
I'll try one of the other utilities to see if it allows the user to dictate the placement and the order of the files. I seem to recall that tha old Norton Utilities for DOS defrag utility did have that functionality, but maybe that only worked on FAT discs.
For now, I guess I defrag before each burn, or save each project to a clean disc, which means archiving projects more frequently. (A time-consuming process in itself.)
R
I have two phsyical HDDs I use during authoring, both internal. Both drives spin at 7200 rpm.
Disc 0 is partitioned into three parts: Drive C for the system files, Drive D for programs, and drive E for normal data, such as word processing files. The phyical drive is 60 gig, and each partition is about 20 gig.
Disc 1 is partitioned into 2 drives, tagged as F and H (drive G is the DVD burner). The physical drive has 100 gig, and each partition is 50 gig. I use both these partitions for video files.
I have 1024 gig of memory, and a 2048 gig pagefile. I have MF4 write its temp files to a directory on C: called \DVD\Temp. In order to leave enough space for the temp files, so they won't cause head jumps, I've moved the pagefile to the E partition, to get it out of the way. The system was parking the pagefile right in the middle of my clear space on the C drive (over 9 gig, plenty for burns; the pagefile reduced that to 7+ gig.) I keep the system drive defragged, so the temp files have a clear space.
I also have an outboard drive, a 250 gig Maxtor, partitioned into 3 partitions, that I use for storing archives of the video projects. It is not used during authoring or burning, and is not even turned on. It is connected by Firewire (which I find handles these big files better than USB 2.0)
The only drives being monitored by System Restore are C and D. I have it turned off for all other drives.
Maybe, as one of you says, the fragmentation is no big deal, but I'm trying to reduce every possibilty of the head jumping during a burn. Even a momentary jump seems to cause dropped frames.
For instance, the project I redid yesterday had about 15 bad spots, some lasting a good two or three seconds, in the 2nd version. When I reauthored it and reburned it at 4x, after defrragmenting, it was perfect.
My big book on Windows XP suggests, for burning CDs (they don't mention DVDs), creating a separate partition for the CD image and temp files, and always cleaning that drive of files before doing a burn, in order to avoid fragmentation that might compromise the burn. I guess they would give the same advice for this situation. Sure is a waste of a big drive.
Today I imported the 2nd half of the same project (it spans two discs), and the same thing happened. According to the defrag analysis before the import, I had two good clear sections of the drive, both large enough to handle a 4 gig file with ease. But the import was fragmented into six parts. The VideoReDo edit was also fragmented into six parts. I think Windows is fragmenting the files regardless of drive space; maybe the system simply can't handle a file that large, and creates fragments as it breaks the file into sections that the system can handle. I'm beginning to think that's what's really going on here.
I'll try one of the other utilities to see if it allows the user to dictate the placement and the order of the files. I seem to recall that tha old Norton Utilities for DOS defrag utility did have that functionality, but maybe that only worked on FAT discs.
For now, I guess I defrag before each burn, or save each project to a clean disc, which means archiving projects more frequently. (A time-consuming process in itself.)
R
Hi,
Think about it... if your burning fails just because the head of a disk has to reposition, then there is something seriously wrong with your hardware configuration, or your system is extremely busy elsewhere. Today's disks will provide enough throughput even when heavily fragmented to keep up with a lousy 4x transfer to a DVD drive.
I think you should also forget about trying to make NTFS work the way you want it, instead of the way MS made it.
What to check? Start with stuff like this...
You need to check if DMA is enabled on your disks (BIOS, device manager, proper drivers for your chipset) and that the proper settings (master/slave) have been done, including the DVD drive of course. I assume the DVD is not on the same cable as the disks, but maybe your notebook is different.
Turn off your virus scanner (and any other tasks) and check again, in case you have wrong virus scanner settings causing the video file to be scanned!
You don't need any of that and you're wasting your time with all the unnecessary defragging.What I need is a utility that also allows the user to force Windows to locate large files close to each other, in sequence, near the beginning of the drive, so they aren't split randomly.
Is my only option to erase everything on the HDD, so it is completely empty, before I start any new project?
Think about it... if your burning fails just because the head of a disk has to reposition, then there is something seriously wrong with your hardware configuration, or your system is extremely busy elsewhere. Today's disks will provide enough throughput even when heavily fragmented to keep up with a lousy 4x transfer to a DVD drive.
I think you should also forget about trying to make NTFS work the way you want it, instead of the way MS made it.
What to check? Start with stuff like this...
You need to check if DMA is enabled on your disks (BIOS, device manager, proper drivers for your chipset) and that the proper settings (master/slave) have been done, including the DVD drive of course. I assume the DVD is not on the same cable as the disks, but maybe your notebook is different.
Turn off your virus scanner (and any other tasks) and check again, in case you have wrong virus scanner settings causing the video file to be scanned!
Henry
-
maddrummer3301
- Posts: 2507
- Joined: Fri Dec 10, 2004 10:24 pm
- Location: US
That should only be a potential problem during real-time capture. If your computer can't keep-up with the real-time video stream, you're in trouble. (You could get dropped frames or other corruption.)Even a momentary jump seems to cause dropped frames.
Once you've got a file on your hard disk, it's just data. If the data-transfer or processing pauses, you don't loose data, it just picks-up where it left-off. (Early CD burners couldn't deal with buffer-underrun, but those days are gone, thankfully.) If your computer was generating data errors, you'd be getting errors in your documents, spreadsheets, email, etc., as well as your video files. And, it would probably crash long before you had a chance to save the bad data!
You mentioned VideoReDo, which means you are working with MPEG files. I've had lots of trouble with MPEG file corruption. But, this was caused by software, not hardware!
Hmmmm... I wonder if it's the other way around.... something corrupt in the file that's causing the head to "jump". I don't know enough about MPEG encoding, but maybe a frame header has bad data and it "thinks" there are several gigabytes of data in a single video frame, or something like that...
[size=92][i]Head over heels,
No time to think.
It's like the whole world's
Out of... sync.[/i]
- Head Over Heels, The Go-Gos.[/size]
No time to think.
It's like the whole world's
Out of... sync.[/i]
- Head Over Heels, The Go-Gos.[/size]
-
heinz-oz
Re: Defragging on NTFS volumes
Whatever is causing you the problems you describe, drive fragmentation ain't it.nyco_ork wrote:I wonder if anyone else experiences this problem: Every time I save a new mpeg file to my video HDD, the file becomes fragmented, no matter how much open space is on the HDD (unless the drive is empty). I have had to do some projects 3 times because of video or audio glitches in the files caused, I think, by fragmentation. (I even burn at 4x, so burn speed is not the problem.).....
R
There is something else in your system causing the glitches.
Do a thorough check of your disks and memory.
-
heinz-oz
In your earlier post you say this
Something doesn't add up.
And, if you are using a laptop, I would throw "heat build up" into the equasion now.
but your system details show a laptop. Do you have two 7,200 rpm disk drives in a laptop???I have two phsyical HDDs I use during authoring, both internal. Both drives spin at 7200 rpm.
Something doesn't add up.
And, if you are using a laptop, I would throw "heat build up" into the equasion now.
-
nyco_ork
Well, a lot to respond to here.
To Heinz:
Yes, I have two 7200 RPM physical drives on a laptop. It's an IBM A31p, and it has two swappable drive bays that can hold anything from HDDs, to DVD burners, to numeric keypads. On the computer in question, one drive bay holds the 2nd HDD, and the 2nd drive bay holds the DVD burner.
I don't think heat is an issue because I don't leave this computer on unless I'm actually working on a project. I don't let the heat build up.
=======================================
DVDoug: I've had some mpeg file corruption, and in the project in question, there were two spots in which the file was corrupted. I don't know if that happened during capture on the standalone, or during the transfer from the standalone HDD to its DVD burner; I had long since deleted the HDD file.
But the other 13 instances were burn problems, and I know the data disc was fairly full when I did that burn, which is what led me to blame fragmentation. I also had not yet moved the pagefile when I did the problem burn, so perhaps the problem was created on the C drive when MF4 was wrfting temp files.
All I know is that these dropped frames have happened often enough to make me crazy. I followed the advice of others here (I think you are one of them) to burn at 4x because I found that most of the time (not all) that helped. I think the slower burn speed helps data throughput.
============================
Mad Drummer: I'm on the other computer now, and will check the settings you suggest the next time I'm on the computer I use to author DVDs. But I'm quite sure the settings you mention are kosher. I'll let you know if they aren't.
I Have monitored CPU usage, and have monitored pagefile usage as well, during burns, and see that everything is perking along in the efficient zone. Since the video authoring computer is rarely used for the Internet, there are no internet files slowing things down. That computer runs much faster than this one, which is an identical model, but has all the internet junk on it. I don't think the hardware is getting in the way, not in the ways you suggest.
But some of your suggested settings/configuration, such as 40-pin cables, don't apply to my laptop setup. Both the main and 2nd HDD use plugin modules that can't be modified in any way.
=============================
Snoops: quite sure the DMA settings are right, but I'll check them again. Regarding antivirus scanners, I NEVER turn that on. I only scan for viruses manually. And, since the computer in question is hardly ever used for the Internet, I especially don't run AV software in the background.
When I am authoring DVDs, I have no background programs of any kind running. I turn the computer completely over to the authoring process. I also turn off any power saving settings that might interfere.
==================================
You guys have all been very helpful. Thanks for taking the time to post on this issue. I'll keep tweaking and experimenting, and let you know if I find any answers.
R
To Heinz:
Yes, I have two 7200 RPM physical drives on a laptop. It's an IBM A31p, and it has two swappable drive bays that can hold anything from HDDs, to DVD burners, to numeric keypads. On the computer in question, one drive bay holds the 2nd HDD, and the 2nd drive bay holds the DVD burner.
I don't think heat is an issue because I don't leave this computer on unless I'm actually working on a project. I don't let the heat build up.
=======================================
DVDoug: I've had some mpeg file corruption, and in the project in question, there were two spots in which the file was corrupted. I don't know if that happened during capture on the standalone, or during the transfer from the standalone HDD to its DVD burner; I had long since deleted the HDD file.
But the other 13 instances were burn problems, and I know the data disc was fairly full when I did that burn, which is what led me to blame fragmentation. I also had not yet moved the pagefile when I did the problem burn, so perhaps the problem was created on the C drive when MF4 was wrfting temp files.
All I know is that these dropped frames have happened often enough to make me crazy. I followed the advice of others here (I think you are one of them) to burn at 4x because I found that most of the time (not all) that helped. I think the slower burn speed helps data throughput.
============================
Mad Drummer: I'm on the other computer now, and will check the settings you suggest the next time I'm on the computer I use to author DVDs. But I'm quite sure the settings you mention are kosher. I'll let you know if they aren't.
I Have monitored CPU usage, and have monitored pagefile usage as well, during burns, and see that everything is perking along in the efficient zone. Since the video authoring computer is rarely used for the Internet, there are no internet files slowing things down. That computer runs much faster than this one, which is an identical model, but has all the internet junk on it. I don't think the hardware is getting in the way, not in the ways you suggest.
But some of your suggested settings/configuration, such as 40-pin cables, don't apply to my laptop setup. Both the main and 2nd HDD use plugin modules that can't be modified in any way.
=============================
Snoops: quite sure the DMA settings are right, but I'll check them again. Regarding antivirus scanners, I NEVER turn that on. I only scan for viruses manually. And, since the computer in question is hardly ever used for the Internet, I especially don't run AV software in the background.
When I am authoring DVDs, I have no background programs of any kind running. I turn the computer completely over to the authoring process. I also turn off any power saving settings that might interfere.
==================================
You guys have all been very helpful. Thanks for taking the time to post on this issue. I'll keep tweaking and experimenting, and let you know if I find any answers.
R
Hi again,
If the bays are swapped, you might find new drive letters to adjust.
Use the device manager and view devices "by connection" to see which bay is on which controller. Maybe swapping the bays (if possible) will cause a better master/slave relationship, or bring the disks on one controller and the DVD by itself on a secondary controller. I'd check the BIOS settings again.one drive bay holds the 2nd HDD, and the 2nd drive bay holds the DVD burner.
If the bays are swapped, you might find new drive letters to adjust.
Henry
-
nyco_ork
Snoops,
The two HDDs are on the primary IDE channel. The DVD burner is on the secondary IDE channel, as it should be.
There is one swappable drive on each side of the computer. The one on the left side is always the one that is logged first (with two optical drives, the left will be F and the right will be G). The first partition of the 2nd HDD is F, the DVD burner is G, and the 2nd partition of the HDD is H.
When I did the burn that had the problem, the data files were on F. The burn I just did, which was flawless, had the data files on H. So regardless of which partition is being used, the HDD is ahead of the DVD burner in the pecking order.
R
The two HDDs are on the primary IDE channel. The DVD burner is on the secondary IDE channel, as it should be.
There is one swappable drive on each side of the computer. The one on the left side is always the one that is logged first (with two optical drives, the left will be F and the right will be G). The first partition of the 2nd HDD is F, the DVD burner is G, and the 2nd partition of the HDD is H.
When I did the burn that had the problem, the data files were on F. The burn I just did, which was flawless, had the data files on H. So regardless of which partition is being used, the HDD is ahead of the DVD burner in the pecking order.
R