Motion search parameters:
Once the bitrate, CBR and quality are set next click on the Advanced menu button in the Compression tab. Once the Advanced menu opens you'll quickly realize that there are a ton of MPEG settings you never knew were there. For now ignore 98% of them and just deal with the following settings that deal with motion.
MPEG deals with motion by using what are called "motion vectors" to calculate how much an object has moved between frames. As long as an objects motion is within the default search parameter set by the encoder all is well, but let the motion exceed this value and artifacting is a likely result. Unfortunately many, if not most, encoders define the motion search area as small a box a few pixels square. OK for talking heads or birthday parties, but not so good when you're dealing with waterfalls or race cars.
The fix: increase the area and accuracy of the motion search.
Start by browsing to the "Advanced Video Settings tab" as shown below;
Once there check both the "Do half-pel search" box and the "Enable" box for "Auto set motion vectors". Half-pel (half pixel element) will enable subpixel searches, which are more accurate. Taking manual control over the horiz and vert motion vectors will help with pixelization due to fast object motion by searching for these changes over a larger area than the defaults would permit.
Now set a value of 16 or more in both the "Horz pel movement" and "Vert pel movement" spin boxes. Values can go much higher than this, so there's a lot of room for adjustment if the need arises. You can even define the search area as a rectangle to more closely tailor it to the situation at hand.
An example would be to set a wide rectangular motion search, say 32 horizontally by 8 vertically, when encoding car races. This would restrict the highest quality motion search to where it's most likely to be found; in the direction of the race cars motion. Turn this same rectangular search area vertical (8 horiz x 32 vert) and it's perfect for waterfalls. Keep the search pattern square for those situations where the motion is unpredictable (baseball, football, soccer, breaking waves etc.)
You might also want to increase the "Noise sensitivity" setting from its default value of 5 (3-7 is typical for DV clips) to a higher value more often used for analog captures (5-14 is typical for analog). I'd probably start with 10 or so. This could help with any pixelizations already in the camera footage due to the lighting, motion etc. Camera codecs are very good, but not perfect.
GOP adjustments:
MPEG GOP's (group of pictures) consist of mainly calculated frames (P and B frames) between widely spaced real bitmap data (I frames). More calculated frames, which are stored as motion vectors and not bitmaps, mean smaller file sizes.
MPEG's encoded for NTSC DVD generally use a GOP size of 15 while those for PAL DVD use a GOP size of 12, meaning each GOP equals about 1/2 second of video. In either case all but one of the pictures in each GOP will be calculated B and P frames and just one will be a real bitmap.
Because of this reducing the number of P & B frames and increasing the number of I frames can up the quality of high motion video by lessening the encoders dependence on motion vector calculations, but this comes at the cost of increased file size.
Yup....another tradeoff situation
If you want to try this technique browse to the MPEG Advanced/Video Settings menu and reduce the I frame value to 6 as shown in this image;
As indicated in the above image (NTSC shown) this will reduce the number of P and B frames per I frame and therefore increase the number of I frames (real bitmaps) per second. Generally speaking more I frames per second = higher quality.
These settings will make for slower rendering, but should reduce pixelization due to objects moving rapidly between frames. If the pixelization is reduced but not gone up the motion vector settings by a few numbers and see what happens.
Needless to say I'd start by rendering a short preview range of a troublesome portion of the project until it looks good and only then render the whole project.