Thomth, I have some experience with modding the iMac G5 and may be able to help. For the most part I have converted them to external monitors (using parts from a 20" Cinema Display), however I briefly thought about putting an ECX (3.5" core 2 duo motherboard in the G5). In terms of the depth of the iMac G5, it varies, the iMac G5's design evolved considerably over time. The first generation iMac G5 is about 2" deep and the LCD attaches to the front panel of the case. This will not fit a standard mini itx, but actually has a decent amount of room. Just to let you know the ECX board (Keex-4020) did fit in the area below the LCD, however, I do not remember if it would have fit behind the LCD itself. Here is a couple pics I took when I was working with the first generation case. The first picture shows the case open with the ECX beside it. The second picture is the completed mod from the side.
IMG_0010.JPG
IMG_0011.JPG
By the time you get to the iSight the case design is much slimmer, with the LCD now attached to the rear panel. Almost nothing fits here, even the nearly flat cinema display inverter and control board are a tight squeeze. Here is the iSight iMac G5 with the case open and the LCD in place. (The iSight and mic were converted to USB and that is the cable running across the screen):
IMG_0012.JPG
You mentioned that the iMac G5's LCD panels were LVDS, however, this is not the whole story. They are in fact TMDS panels (though they may have LVDS passthrough - I can not myself verify this). The LG LM201W01(A5)(K1,2,or3) are the pre-iSight models and are TMDS compatible. The LG LM201W01(ST)(B2) found in the iSight is also TMDS compatible. These are interchangeable with the 20" Aluminum Apple Cinema Displays LCD and the iMac G4's 20" ID Tech. However, the white intel iMacs use LM201W01(SL)(A3), which is LVDS only. This is an advantage as TMDS can be converted directly to DVI and the pinout is the same as my iMac G4 mod (I do not know how different the native inverter is though).
If you are interested in doing this let me know and I will open up my 1st generation mod and take measurements for you. However, I can fairly certainly say that a full size atx, matx, or mini itx will not fit within the depth of the case. As most iMacs are, the G5's components are closer to laptop than desktop components. A laptop mobo with an HDMI or DVI video out port would probably fit the best. Good luck and hope this information is helpful.