Correct me if I misunderstand the situation. Were you able to successfully boot the Mojave installer just once on the first attempt but not again or has it not worked the entire time?
Depends what you mean here by "boot." I plug in the installer USB, in the BIOS choose it and disable both the SSDs (both are blank, but the SSDs are still listed in the BIOS). Then Save and Exit the BIOS. I get the AMI display, then the OpenCore display which shows the "Install macOS Mojave" icon (plus "NVRAM" and "EFI" icons). Then I click on the installer icon and an Apple appears, but no progress bar. Then the Apple disappears, and I am left with nothing but the cursor in the upper left corner of a black display. This happens with an EFI folder in the installer flash drive that is identical to the one I always used in this computer when it had Big Sur installed. I have not seen another behavior since first attempting this installation. I am beginning to wonder if the installer does not contain a required driver for the RX 580 graphics card, and the installer app is in fact opening, but not sending a picture to the monitor. However my monitor usually indicates when it is not receiving a video signal, and I'm not seeing that message.
At one point I removed the RX 580, changed the DisplayPort output to the motherboard's built-in connector and changed the "Initial Graphics" from "PCie" to "IGFX." No change.
I will add "-v" boot argument to the NVRAM area and enable that quirk you mentioned above. (I've never had much luck with "-v" because it all goes by so fast and then leaves so quickly. But I'll try it.)
[Edit: No joy, same thing. Apple for about 5 seconds, then it disappears and black screen. I tried this with "IGFX" and internal graphics enabled, but I still don't get any text on screen or anything other than black. My next trick is to change Device Properties/Add/AAPL,ig-platform-id argument from "0000913E" (means internal graphics is used only for computing and does not drive a display) to "00093BE" which seems to be one solution to a black screen problem. Also added the frame buffer patch for Whatever Green.]
[Edit 2: My next trick made no change. Just for fun, I booted the installer on my High Sierra (also Coffeelake computer, "Mini-ITX 3" below), and got the error message "This version of Mac OS X is not compatible with this platform." Which is odd, I thought, because I could easily update to Mojave from my High Sierra computer using Software Update. Right?]