DVI and HDMI are the problem for Boot to black screen with monitors connected to these ports.
With Sapphire 580 cards at least, the Display ports are initialised first when osx loads and for some reason the HDMI and DVI are not working properly, so you need to patch the framebuffer to change that order.
I recently replaced my GTX 780 with a Pulse RX 580 8GB and had the same issues. I have a 1440p DVI monitor and an HDMI 1080p and none of them worked out of the box due to the issue described above.
I replaced my card after I had installed Mojave though, so my case is different than yours.
I solved my problem by patching the framebuffer so that osx initialises the DVI and HDMI ports first.
An excellent guide for patching framebuffers on ATI cards can be found here.
So if you recently installed High Sierra and have an AMD graphics card and were greeted at the end of the installation with black screens and Lilu.kext and WhateverGreen.kext wasn't able to resolve it, then try patching the framebuffer that corresponds to your graphics card to try resolving it...
www.tonymacx86.com
Since you have a Nitro, your port config should be identical to mine. Again the following will work if you have installed Mojave already using a different card first and then replace with the RX 580 or install using the display ports of the 580 (which should be working out of the box).
So for Nitro, using Clover Configurator, open your config.plist and
under Graphics select:
FB Name -> Orinoco
Video Ports -> 5
Inject ATI -> select
Under Kernel and Kext Patches, create a new entry and input the following:
Name -> AMD9500Controller
Find (HEX) ->
00040000 04030000 00010101 00000000 12040601 00000000 00040000 04030000 00010201 00000000 22050403 00000000 00080000 04020000 00010300 00000000 11020102 00000000 00080000 04020000 00010400 00000000 21030504 00000000 04000000 04000000 00010500 00000000 00000306 00000000
Replace (HEX) ->
00040000 04030000 00010501 00000000 12040601 00000000 00040000 04030000 00010301 00000000 22050403 00000000 00080000 04020000 00010400 00000000 11020102 00000000 00080000 04020000 00010200 00000000 21030504 00000000 04000000 04020000 00010100 00000000 00000306 00000000
Save your config and reboot.
Once in osx (still with display ports or with the different card) open a terminal window and type.
sudo chown -R 0:0 /System/Library/Extensions/
sudo chmod -R 755 /System/Library/Extensions/
sudo touch /System/Library/Extensions/
sudo kextcache -U/
Wait for the above to finish, shut down your machine, replace the existing card with your Nitro 580 and your DVI and HDMI ports should now work.
The problem with this method is that at every osx update, you have to go back and refresh the kextcache in SLE (maybe some future update will even break this patch and we'll have to correct it or patch it differently).
What I did for example with yesterday's patch update to 10.14.3, is use TeamViewer which I set up for unattended access before I start the update, so when the update finishes, I restart with no monitor connected, and refresh SLE with another computer via Teamviewer.
