First of all, thanks for the solution posted in this thread. Although it did not work for me I still appreciate the help. As an alternative, I wanted to report on getting 5.1 surround specifically via SPDIF.
I know this is kinda old and I apologize in advance if this is the wrong thread.. just wanna help out. I figured there are some people who are still looking for answers like me a few minutes ago.
I have a GA--P55M-UD2 F11 (ALC887/ALC888b) running 10.6.8. My hackintosh is connected to a Denon AVR1312 via HDMI (for video - no HDMI audio with my 9600GT card) and SPDIF (Digital Optical Output). As I said, the Audio MIDI Setup did not work for me so I had to find another way. I followed the instructions on this link I found on another thread:
http://www.cod3r.com/2008/02/the-correc ... quicktime/ and it worked!
Essentially, you have to install Perian (a quicktime extension) and then run a terminal command. After following the simple instructions I was able to pass AC3 and DTS to my receiver (at least within Plex - VLC and Quicktime is still stereo I think). I tried to play music on Itunes and get stereo sound.
As for controlling the volume using the keyboard, unfortunately this does not work using the Digital Optical Out. I *think* this may be normal behavior for SPDIF whether working with 2 channel or 5.1 audio. You control the volume via the receiver's controls. So far, it works for me so I am not worried about it.
Hope this helped somebody.
