You don't say which version of Video Studio you are using, though I see you have given us your system settings -- for which, many thanks. But before VS11, Video Studio 10 had extreme difficulty with the H264 codec, and before VS10, VS could not handle it at all. In fact, there are many who could not get VS10 to handle it at all either. In fact, it was a source of angst for many users who had QT movie format, precisely because they were encoded with H264. So that is potentially strike one against you in your workflow.
You have told us some details of your workflow, but not enough, I am afraid. OK, we know you started off with high quality DV format video from AfterEffects. But then you converted it to a highly compressed H264 QT .mov file. Why was this conversion necessary? And I suppose you realise that while H264 gives high quality, it is highly compressed, so in the conversion, it is inevitable that some quality is lost, even if it is not really noticeable to the naked eye while still in that format...
Then you opened that H264 file in Video Studio to burn it to DVD. But I am deducing that you did not first attempt to convert the H264 file to a DVD-compliant mpeg-2 file, but left the conversion to be part of the burning process when you made your DVD. Quite apart from the problematic nature of the H264 codec with earlier versions of VS, leaving any conversion to the burning process will often cause problems in its own right because it is loading an extra step -- and in this case an inherently difficult one with the H264 factor -- onto what is already a complex series of steps that has to be followed by the authoring/burning process.
In addition, going from one highly compressed format to another less compressed, and lossy format like mpeg-2, will entail a further loss of quality. Strike two.
It would possibly have been better if you had first converted the H264 video into mpeg-2 as a separate step. You could do this with VS (Share > Create Video File > DVD), and then when this was ready, you could insert it in the burning module for burning (Share > Create Disc). Or you could use a third party program to convert it. There is a freeware program you might also consider which can handle the H264 codec, and do a quick conversion too. You can download it at
http://www.erightsoft.com/SUPER.html
But I come back here to my earlier question as to why you converted the DV video to H264 in the first place. DV format is one of the highest qualities you can get in standard definition video -- though the file size is admittedly large (approx. 13 GB for one hour). Normally, we recommend people as far as possible use DV if they have it. In other words, you should have inserted this DV video directly into Video Studio, which can handle it with ease. And the conversion to DVD-compatible mpeg-2 (Share > Create Video File > DVD) is normally problem-free, particularly in more recent versions of VS. It will also be clear and high quality if you choose high quality settings such as those in the pre-set template for 'NTSC DVD'. (I am further assuming here that your project was probably much less than one hour in length, so would easily fit with the highest quality settings on a single layer DVD.)
Once you have your mpeg-2, save your project, then open a new one. Don't worry about a name for it -- the objective is just to clear the timeline.
Then you go to Share > Create Disc, insert your new mpeg-2 into the burning timeline, make your menu if you have one (probably not if it is a looping video) and burn. Also, make sure 'do not convert compliant mpeg files' is ticked in the middle of the three icons in the bottom left of the burning screen. You should thus not have to adjust any of the properties in the burn module.
