I've done a bit of Windows overclocking, and it seems like those systems always went a lot smoother than this. This is my first time OC'ing the new Intel chips and it seems like a new ball game.
I have your same MB, but with an i3-560, and that exact problem occurred. Any overclock at all drops everything down by 50-100% in Xbench and Cinebench. Xbench went from 117 to 235 just by resetting stock speeds in the BIOS. All I'd done at that point was increase BCH to reach 3.72GHz.
I got a CPU score of 1.12 with an overclock to 3.72 (stock is 3.33), and with my Geil 1600 memory running at 1450. I tried 1333 on the memory, changing multipliers around to keep the memory below 1600, and the CPU under 3.7, with the same results in OSX. The stock 3.33/1333 settings (i3-560 runs memory at 1333) brought the CPU score up to 2.8 or so.
The system is rock solid (stability-wise) with the OC, and I've been using it for about three weeks, but any real system load at all causes stutters. For instance, in Garage Band, playback of a track causes high latency. Once I set everything back to Auto in the BIOS, the performance was smooth as silk. OpenGL scores went up by 50%, to an unimpressive 16fps (9500GT, so I wasn't expecting much anyway.)
I installed this using the latest stuff from this site, and I've considered a few systematic adjustments in the BIOS just to see if there's an effect.
The thing that gives it away is the stuttering under load, but I'm not savvy enough to have figured out exactly where the bottleneck is occurring in the BIOS settings. Right now what seems logical is some basic relationship between memory and BCH frequencies, so that's where I'm going to start.
I'd recommend one setting change right at the start, set the Memory Performance Enhance (in the Advanced Memory Settings) to Standard, not Turbo. That's a standard overclocking move, in previous systems anyway. Just doing that with a mild overclock to BCH of 140 and memory speed of 1400 (With STD brought my scores up to 2.9, so there's part of the problem, I'd bet. Going to 150/1500/3.75 dropped the CPU score back to 1.18. With a 160/1600/3.52 (dropped multiplier down), I got a 1.23.
The BCH is important to stay back from a bit. If you can creep it up and just get your BCH to 150-160, holding the other frequencies close to what they are, you might get something better than 3.0. Your performance drop could have come from BCH increase without adjusting back down the other freqs.
I also disabled the Auto Load Line calibration feature, but I haven't made any voltage adjustments at this point. I just set NORMAL for all the voltages, instead of Auto.
I'm reading through this guide (for a different MB, but just for the basic concepts), and I'll see if it has anything to offer:
http://www.masterslair.com/199/how-to-overclock-intel-core-i7-i5-i3-cpu-overclocking-guide/
My final thought is that you can't disregard the possibility that there's something you could change about the way your CPU is identified by OSX. I am not sure at all how that works, but there are posts throughout the forums about CPU tweaking. That's the point at which I give up. If it didn't get automatically recognized with Chameleon or whatever, I've got no clue how to change it, and it doesn't really matter for a 15% overclock anyway.
Pretty crazy how 1 change can make a big difference.. The crazy thing is though I didn't really notice a change in performance when using it. I was using FCP7 earlier and also had a lynda.com tutorial video playing and it did suffer a little when playing back some HD videos.. I'm gonna try it again now and see if it will still do the same thing..