I have also tried installing OS X (10.10.5) on a Gen8 Microserver, with the same installation issues. I had the same garbled text on the screenshot you uploaded, i believe it is the 'Waiting for root device' error.
I would point out that the two USB 3.0 ports on the rear of the machine are disabled during installation/boot by the Bios, this is the same for a Windows or Linux installation and I could not find any way to alter this state. This is confirmed in the Microserver manual.
The four USB 2.0 ports are not using an Intel chipset, I believe this is the cause of the 'waiting for root device' error. OS X does not recognise the usb ports during installation and therefore can't find the installation media and stalls!
I tried a number of different USB 'fixes' that I found on this and a couple of other Hackintosh forums but could not move past the same point you are at. But as I hadn't installed OS X on my SSD I had a few different USB related issues visible in the text, before it becomes garbled and illegible.
The Microserver has a 1155 CPU/socket, a Celeron, which may also be raising issues with the installation of OS X. I believe Apple stopped supporting these lower level CPU's with Lion or Mountain Lion
I spent a week or more trying different ways to get this machine to boot OS X, even dropping back to Mavericks and Mountain Lion but with no more joy than with Yosemite. I believe this was due to the repeated USB issues and I eventually gave up and installed a Windows OS.
For clarity my Microserver had 8GB RAM (2 x 4GB), a 240GB SanDisk Extreme SSD, an AMD HD6450 2GB low profile graphics card, no other disks or accessories attached, the screen was attached by a new HDMI cable. My Bluetooth usb dongle was sitting ready and waiting for OS X to boot. I tried installing OS X with a couple of Corsair 16GB USB pen drives, using the latest Clover and a new downloaded copy of Yosemite/Mavericks and ML. I initially chose to install Yosemite over El Capitan because I thought there would be less chance of any USB issues arising. I was using the legacy Clover install, as the BIOS for the Gen8 Microserver is not UEFI.
Not sure if this information will help you in any way or make you more despondent, but I thought I would share my trials so you know you are not the only one who has tried and struggled with creating a Hackintosh with a Gen8 Microserver. Best of luck with your endeavours.