What ist your OS X Version? - since Mojave RX 560 should work.
To avoid Blackscreen (& some other things like most ports working) you must use Whatevergreen ( together with LILU).
disable any GPU related injects (Inject ATI / Radeoninit) and renames GFX0/PEGP ... (in ACPI part) of Clover.
It can work with enabled IGPU together, but better use SMBIOS iMacPro1,1 & disable iGPU in the BIOS - then HW Enc can work on the (faster) AMD than your old iGPU (whoch cant handle HW 265).