I use an EVGA 4GB GTX 770 in 10.9.3 and it is natively supported. I might look into whether it is supported in 10.8.5?
Another idea would be to boot in verbose mode so that you can see where the boot process is hanging. Add -v to your kernel flags line, or, if booting with Chimaera, hit space...