Before I managed to get macOS working with the RX 560 installed, I tried just about everything including flashing an alternate vBIOS. I tried the vBIOS from a Lenovo because the clocks and thermal settings were the same. I reverted back to the original vBIOS because it made no difference.
It was really, really strange... Using anything other than Macmini8,1 system definition resulted in black screen from the RX 560 in macOS whether I had WhateverGreen enabled or not. I suspect that it might be because with Macmini8,1, it's treated as a Thunderbolt attached GPU.
Anyway, I'm just glad that I got it working. It spends most of the time in Batocera and I only really boot in to macOS when I need to do any file management for the Batocera folders.
I briefly had Windows installed... On a 100% fresh install with only 7zip, Rufus, and RPCS3 on the system, I tried launching RPCS3 and it already complained about missing DLLs. I was like, "WTF???" I'm not going to deal with this and deleted Windows immediately.