Bit rate to preserve quality

[patflan]

I have a lot of home videos going back many years which I edited and transferred from camcorder dv tapes (sd quality) to dvd using videostudio.
I also saved the original files as uncompressed avi to harddrives and dvd-r.
When transferring to dvd I set the bit rate to a constant value of 7500k to ensure reasonable quality and playability on dvd players.
But now thinking about the future of dvd I am now no longer inclined to make dvd copies but save just to 2 hard drives in mpeg2 format.

So this is my question since I am no longer need to restrict the bit rate to 7500k for dvd playback should I just convert the avi files to mpeg2 with a much higher bit rate and if I do what is the limit? That is ,if I am not too worried about file size but want to preserve max quality is there a upper limit where there is no improvement.My rough guess to preserve quality is that as the uncompressed avi file size is 12G per hour and 7500k mpeg2 is about 4g then to get near the same quality and file size of 12g after converting should I just set the bit rate to 3 times which is about 20,000k bit rate?

The reason I ask is that even though the original video is all sd quality I do see a subtle difference in quality between the avi uncompressed files and the 7500k mpeg2 when playing back on my pc.
Also ideally the answer is not to convert at all but the avi files are not recognised on any of the 3 different tv media boxes I have.
Or should I convert to another format rather than mpeg2 to get better quality but remembering I am not trying to reduce file size (well up to 12g per hour).
Also whatever format I choose needs to be compatable to playing an most devices.

Any thoughts? - thanks.

[asik1]

So you trans-code your 124 hours home movies of original SD DV from uncompresed gezilyon MB/S and holds 32.6TB to just 24M/s and will hold maybe just 10TB, but still be 124 hours. Now what? who will ever watch it and will they screen it over the Jumbotron on the 2020 superball.

I always recommend just a proper WMV9 WMV at ~3.5MB at most.
The main problem of SD is not the datarate but the 70" screens.

[Ken Berry]

First off you need to correct your thought that the DV/AVI you filmed in is uncompressed AVI. DV/AVI is in fact compressed, though as you say it is around 12 GB per hour of video (we normally say 13 GB). True uncompressed AVI is in fact around 64 GB per hour. But DV/AVI is similar in the sense that it is in effect lossless, apart from being among the best quality standard definition video around. Unfortunately, though, as you have found DV will not be recognised as is by modern HDTVs... Neither will HDV which is also filmed on DV tapes...

As for an optimum bitrate for SD mpeg-2, how long is a piece of string? IMHO, though, there is probably little point in going beyond the maximum bitrate under the international DVD standard which is around 9600 kbps. I'm just not sure the human eye is going to detect any better quality in SD video with a bitrate higher than that. However, only you -- the observer -- will be the only one who can tell if you can or not. So try 10,000 kbps, and then with the same video, try something higher (12000 or 15000 kbps) and see if you can detect the difference.

DivX/XVid will also give you high quality at a more compressed/smaller size... You might also want to try AVCHD at around 15,000 or 16,000 kbps -- though I hasten to add that you should not be expecting a silk purse (high def) from the sow's ear of your original standard def video. But going from DV/AVI to AVCHD at the higher bitrate should IMHO preserve the original quality of the DV. But again you need to try it out to see for yourself what might be best for you.

DivX/XVid and AVCHD will also both be recognised by your HDTV...

[patflan]

Asik1
Thanks for pointing out that about sd on bigger screens (and the future thrend for screens to keep getting a bit bigger).
I had not fully appreciated that but I had experimented a little bit to see if I could noticed any difference between a 30 inch and 55 inch but really hard to tell without going nuts! But as my videos are now becoming older (10 -20 yrs+ ) I am starting to appreciate their value more with time and want to make sure I do not drop the quality any further in the transcode.And as Ken ponts out I mistakenly did not mean to try presrve fully uncompressed raw video but as near to the original dv compression as possible.

Ken-many thanks also.
Your first paragraph is an interesting point about the original tapes.You are saying we are (were) recording to a fairly good quality whether sdv or hdv tapes but that higher quality format is lost now because tape playback is defunct.But we can live with the reduction because it is not noticable.

But you have steered me in the right direction as I am about to also convert more dv tapes (and hope they are still ok!) and now that have moved away from dvd for storing I want to settle on a bit rate for all my remaing sd videos.The problem I had was ,with making Dvds I was forced to stick to an upper bit rate limit for playback but then without this limit for non dvd format I did not know where to start and at what point there was no further quality gain when viewing.As I say I can see a subtle difference between the avi and the mpeg2 -maybe for moving scenes but it is only very slight and as you say it depends on the viewer and "how long is a piece of string"
So I will take some short video clips and try your recommended bit rates and see if I can notice any difference worth while and then once I have settled on a bit rate or format I can get on with transcoding the remainder and preserve those videos until the next technology change.

[Curious]

Bit rate to preserve quality

So this is my question since I am no longer need to restrict the bit rate to 7500k for dvd playback should I just convert the avi files to mpeg2 with a much higher bit rate and if I do what is the limit?

Your are near the dvd video bitrate limit for mpeg2. So if you are not planning to create dvd with these files, this makes no sense to generate much higher mpeg2 files to preserve quality cause anyway these files could not be used for anything than dvd (or streaming, but there are much more space efficient codecs like avchd).

What i suggest is to keep the original source in DV format on an external hard disk or 2 (stored safely away from the computer), this is what i am doing (the price of HD is so cheap these days. Just need to refresh the hard disk(s) content every x years to make the content safe). Because of technology evolution, what codec is good today could be much better tomorrow, so you won't reconvert your new source files (already converted from DV) every couple of years to follow the evolution (tedious process and risk of loosing some sort of quality at some point if not done carefully, example your content is DV interlace, and you convert it to non interlace format for pc/tv viewing by mistake, you lost some irrecoverable information. You would also need to inspect every new sources to make sure no conversion bug could have make them not desirable as new sources).

If only the new files you want are for tv streaming or viewing, just batch generates viewable files at a reasonable quality you like for viewing using an efficient codec your tv could use. This depends on your tv. avchd (h264 encoding) is mostly supported now and is much more efficent in file size for the same quality than mpeg2. These files won't be your new original so it doesn't matter a lot (and you won't need to carefully inspect all these new files, hours job, but only when you will need to watch them, because you are keeping the originals). If your next tv won't support this codec in few years, you just have to generate viewable files in a new tv supported codec at that time (your originals always stays DV video files), so probably understand my point...

To batch covert your DV files to avchd h264, you could use this free tool (there are predefined conversion format, but you could easily create new ones to your taste and to match your tv requierements):
http://vidcoder.net/


It is always better to keep the original source material, no matter in what format it was created for archival purpose

[Curious]

But DV/AVI is similar in the sense that it is in effect lossless

While consumer DV is a very good codec and it is compressing video frame by frame, it is not a lossless video codec and could suffer from quality degradation in multiple generation conversion (when recompress to DV from generation to generation). But you could need a lot of generation depending of what DV codec is used and what kind of video content to see the degradation

From wikipedia:

"DV uses lossy compression of video while audio is stored uncompressed"

See this page to see different old computer DV codecs and what kind of degradation you could get after each compression generation (yes i know this is old stuff but shows the point):

http://www.codecpage.com/DV.html

Look into "DV Codec Test" section

[patflan]

Curious,
Referring to your previous post,I will give that converter a try and see how the dv files work out when converted to avchd-thanks.

My method is similar to yours in that I save the dv files to hard drive, unedited, as master copies

I then like to hedge my bets by also converting them to a 2nd unedited copy in another format (mpeg2),with a slight quality loss but which is more usable for tv viewing and compatible with any device.
This is my attempt to have the digital equivalent of the old shoebox for storing photographs.While the dv files are my master backups for any future editing or conversion, the mpeg2 files are nearer to being the shoebox storage since they are in a universal simple format always ready for viewing at any time without requiring further code conversion.(not just by myself but other family members ).I originally chose mpeg2 since I could make DVDs as well as being a universal format that could play on anything - and in those sd days avchd was not available or still very new.
However,thinking about your suggestion,avchd is probably more universal now than mpeg2 for playing across all devices ? (Especially for mobile devices). And as you point out, it is more efficient, so I may move over to avchd to be my shoebox universal format -with the dv/av files equivalent to the film negatives for any future conversions.

[Curious]

My method is similar to yours in that I save the dv files to hard drive, unedited, as master copies

Additionnal comments on what i have already explained before:

Actually, i am using 2 external hard drives used for archival purpose. The reason i am doing this is that HD are mechanical devices and at some point in future will fail to work, i don't want to loose all my precious photo/video archives at that moment. Having 2 units with identical content lower the risk of original lost to merely 0% chance of lost (keeping one of them in a different physical location is also better in case of robery, house burning, etc).These 2 hard disks used as archive medium are never connected to my computer, except when doing video editing, to avoid any virus or potentially external intruder on my computer to destroy these (avoid also any computer defect, power supply problem, or lightning to destroy these). My photo archives are also on these drives.

Warning: As an additionnal comment, i suggest if your are using hard disk for achival purpose (power off on the shelves) to refresh the hard disk content (read and rewrites all files every x months, say every year maybe), cause magnetic on the device can fade over the time. See this article:

https://larryjordan.com/articles/hard-disk-warning/


i am also sha1 hashing all these source files to keep control over time which one could be corrupted (without needing to view all of them manually), this way i could simply recover the originals from the 2nd hard disk if needed. I am using this free tool for quickly hashing a complete directoy full of source video files:

http://corz.org/windows/software/checksum/

Hashing is specially helpfull if you refresh the files on a hard disk, you can validate the content still ok without needing the 2nd hard disk (in case of a single hard disk failure), but with 2 hard disk, you can also compare the files content on both hard disk to see if everything matches