Chroma Key degrades video

[E-edit]

I've been struggling with this issue for some time and can't find any solutions anywhere.

I import a high quality mp4 file with greenscreen background into Video Studio Ultimate X7 and get the following:

Chroma key works well (usually set on 70) to remove the green screen. No real color challenges or issues, it's clean.

Problem though - Subsequent video is significantly reduced in quality when in viewer and more importantly after render (high qual render settings, producing an even bigger file). It's so bad, I'm limited to the backgrounds I can choose (only dark). I get pixely videos, jagged edges. It's weird - source video hd and very high quality! Pulling hair out.

Ever seen this?