It so strange, but now I just put my RX580 Nitro in and it just works with DisplayPorts, no additional setup I did after GTX 760 card!I'm having the same issues. Only HDMI working, Displayports stay black but are recognised by the system. I've got a displayport to HDMI cable and when I switch the HDMI on the TV to HDMI modus 1 I get a picture but it has noise in it and sometimes goes to black aswell. Seems like the RX580 can't drive both screens in 4k however it wasn't an issue in High Sierra.
I have the same issue. It has come about after either 10.13.6 or switching to a recent WhateverGreen.kext as it was fine on my initial build using RX580.
The easy resolution is to simply unplug the DisplayPort cable and plug it back in. Irritating, but I can live with it for now
17,1 will work fine, you just need to use the correct SSDT injections. Upload your original DSDT and I can make a SSDT for you. Clover boot menu hit F4 to get a copy of original DSDT, it will be located in EFI>Clover>ACPI>origin