I just installed High Sierra on a GA X77X-UP4 TH with an SSD and a Radeon HD 6850 (not the system in my profile, an older system). Unibeast and Multibeast worked flawlessly.
My advice is to disable the 1080 for now, and use the IGPU (make sure it's enabled in the BIOS). I believe 7 series boards support UEFI boot so you should build your USB using Unibeast in UEFI mode (not legacy mode).
Follow the recommended BIOS settings, build the USB stick, boot from it, and at the clover boot screen select 'boot install macOs installer from USB' (or whatever you named your USB). You should reach the installer UI. Click install. You'll get a progress bar for a few minutes, then it'll tell you it needs to reboot.
Reboot, boot to USB stick, at clover screen select 'boot install macOS from SSD' (or whatever your SSD is called). If you convert to APFS (the default on SSDs), it'll reboot again, use the same process.
It'll reboot again, boot from the USB, at clover select 'boot macOS from SSD', and you should reach the desktop.
Let us know where you get hung up following this guide. To be clear, we will get sound, your 1080, and other stuff sorted later. We're just trying for now to get to the High Sierra desktop.