I seem to have missed something.
You are using a G41 motherboard and a Q9400 CPU right? If so, what you experienced in Mojave and Catalina is normal. The problem is, your CPU lacks the necessary SSE4.2 instructions which is required for the AMD graphics drivers in Mojave and Catalina. You are not going to be able to use the RX 580 in either Mojave or Catalina. I am not sure if this restriction also applies to Sierra 10.12.6 and High Sierra.
If you want to run MacOS on your (old) system, I would suggest you try to find a (second hand) old Nvidia card like the GTX 760, which should allow you to run MacOS on your system (including Mojave and maybe Catalina). You will need to remove the RX 580 graphics card from your system.
The RX 580 can be used with a more recent compatible system to run MacOS Sierra 10.12.6 / High Sierra / Mojave / Catalina.