Okay, I retract my earlier statement, sadly that green line on the border with 10-bit video is still happening to me. No idea why it seemed to me it is gone, I must have missed something (like, another decoder being used before ffdshow or something /edit: explanation below, I must have had the osd filter enabled at the time/).
Hmm, I also got the green line on pictures captured with grab, if grab is moved below the resize & aspect filter. If I move grab above that in the chain, the green line is gone from the grab, but remains on the displayed video... (I'm using VMR7 window btw). The green line on the grab picture is also different, and happens in the valid part of the picture (as the grab picture isn't output as padded, it gets spit out as 714x480). On the resulting rgb image, the line seems to be 1-pixel wide only (crop to 713x480 removes it).
Here is how it looks:
screenshot made with mpc's "save image"
http://img411.imageshack.us/img411/4743/mpcscreenshot.pngthis is using grab placed below resize & aspect:
http://img171.imageshack.us/img171/5607/grabsy.pngthis is using grab placed above R&A:
http://img12.imageshack.us/img12/5273/grabmovedbeforeaspectsi.pngupdate:If I enable the OSD filter during playback, the green line on display (which was outside the actual 714x480 area of image) is gone, replaced by black padding. If I disable the OSD filter, it returns. This is without relaunching player or reloading file. OSD filter isn'T directly needed, the message displayed after a hotkey activation is enough (like "toogle OSD off" or "toggle sharpen off"). Hmm, actually it seems that for example the sharpen filter also disables the green line, so what seems to be happening is the filters are accidentaly "fixing" the pipeline in some way.
However, the filters seem unable to remove the green line from the grab pictures; if placed above grab & enabled, they cause the green line to appear in grabs (even if resize&border is below grab), even if at that time, the green line on display is gone!
As you say, Herp Derp.
This FFDShow thing needs to be deprecated and got rid of ASAP.
(There was also this issue of filters enabled during mpc file-opening causing changes in output colorspace selection that I ranted about on irc...)
Update2:Okay...
It occured to me that I shoudl try to reset the settings for the resize&aspect filter. In the main tab, the reset button was grayed-out, but in the sub-tabs, the buttons were active, so I used them.
What do you know, I got rid of the green lines, both in grab pictures and on display.
If I figure out which setting there can cause them to reappear, I'll post it here...
(I should have expected this, because I knew already that changing options there can affect ffdshow in unexpected ways, even if the filter itself isn't activated).
Edit: it is the method option in the resizer settings: if you change it to "none" (default seems to be bicubic), you get a green line (just to make it clear, this green line business seems to only happen with 10-bit h264, at least for me). IIRC I changed it to none to workaround some other non-mod4 bug elsewhere, earlier. I can't remember now.

....
Also, uhm -- that's either an avisynth 2.5.x-related limitation or a limitation of VirtualDub (IIRC it was more strict than Avisynth itself in some ways resolution-wise).
That was with Avisynth 2.5.8, playing back an avs script in MPC-HC with Directshowsource and DSS2.
IIRC this was supposed to be a limitation of vfw interface. Vdub uses it, thus it can't load an avs script with mod2 width yv12. However avisynth itself isn't limited this way, as you can see when you use avs2yuv, or x264 from the last two years or so, for that matter.
....
P.S.
As for the mod2 test sample:
WinXP SP3 32bit Home, Radeon x700 128MB, A64 3200, 8.593.100.0 | 10th Feb 2010 -> Works Fine (VMR7 W/R, VMR9 W/R, EVR, EVR CP, haali (not yv12 I guess), Overlay Mixer)
I didn't update directx despite mpc shouting at me to do so though.