For whatever reason xy-VSFilter does, though, try to do a X->YUY2 conversion.
I'm unable to reproduce this, but all my computers support YV12 Overlay. Was there any way to artificially reproduce this on Win7 with GraphStudioNext?
What you describe is usually a sign that Video Renderer and/or Video Decoder does not support re-connections with a different (non-default) color format. A number of poorly designed decoders like FFDShow, CoreAVC, XVID, and others all are unable to re-initialize themselves properly when a different color format is requested on re-connection and output a green screen to xy-VSFilter. I wouldn't be surprised if a similar bug existed in MPC-HC's Overlay Mixer implementation.
I'll have the xy-VSFilter developer take a look, but I somewhat doubt this is an xy-VSFilter bug. My understanding of how xy-VSFilter functions, is that the code you point to should be a non-issue because xy-VSFilter's first connection attempt with the video decoders default format (i.e. NV12 in your case) is always a dummy connection to query the Video Renderer's supported formats. xy-VSFilter then always disconnects -> reconnects with the with the highest preferred format of the Video Decoder which the Video Renderer supports (i.e. YUY2 in your case). The way we've implemented our transform filter re-connections and colorspace changes follow Microsoft's specifications for performing these operations very closely.
In any case, if this really does turn out to be an xy-VSFilter bug, we'd prefer that you allow us to fix it properly ourselves. The colorspace and re-connection logic is a very sensitive and specifically designed portion of xy-VSFilter code, which I wouldn't recommend randomly hacking around. At the very least, please let us review any proposed patch before releasing a 'fixed' build to the public.
. A new release candidate has been available for a bit now
-- and you have double the cores. With MadVR you wouldn't be getting the power savings anyways 