After a day of struggle, I got it working.
Asus p8z77-m pro + i7-3770K +
MSI Radeon RX 5700 8GB.
To update Mojave to Catalina (first time, right to 10.15.2 and only because of 5700 which I got a week ago), I had to apply only one Acpi/DSDT patch, and remove Lilo & WEG from system folders.
Video card worked perfectly from the box, test scores looked good (they were always good if video card worked after booting up independently on configurations listed below):
View attachment 441691View attachment 441692
In a day, I found that it uses Soft decoder/encoder when playing 4K videos in Quick Time Player, and the same for Compressor:
View attachment 441690
I expected better performance from 5700 on Hackintosh in Final Cut Pro/Compressor than my new 16" with 5500M 8Gb, even though processor is much slower, but it just worked as on my iGPU before.
I tried everything, different configurations, and there's list of results below (only important). When I googled this thread, I quickly found a way to detect if encoding/decoding works - VideoProc App.
1. Almost native one for my i7-3770K: SMBIOS MacMini 6,2. iGPU as default in BIOS. Only in this configuration, both cards, iGPU and 5700, are detected and work together with and without Lilu/WEG/agdpmod=pikera. I would leave this configuration, but:
View attachment 441693
2. SMBIOS iMacPro1.1, iGPU enabled as default in BIOS. No Lilu/WEG/agdpmod=pikera:
View attachment 441694
with PCIE enabled as default in BIOS, and without Lilu/WEG/agdpmod=pikera,
both H264 and HEVC were disabled - the same as of all other configurations I tried.
3.
SMBIOS iMacPro1.1, PCIE enabled as default in BIOS. Lilu/WEG/agdpmod=pikera - only working configuration for me:
View attachment 441695
Only with such configuration, when I play 4K video, or use Compressor/Final Cut Pro, I don't see Soft Encoders/Decoders in Activity Monitor, but only something like:
View attachment 441696
When someone shows the scores in Metal/CL for hackintosh, my first question now is about CPU usage when playing 4K.