@macnb which settings are you using nowadays?
Being able to encode/decode h264 / hevc in Mojave?
Very interested, as I have the same mobo...
I am on MacOS 10.14.4.
I enabled the IGPU connector-less (needed enable H.264 encode/decode WITHOUT green artefacts)
I installed Dummy141.kext in EFI/Others (needed to enable HEVC encode/decode in RX580)
I use SMBIOS iMac13,2
I have DSDT.aml and SSDT.aml (for CPU power management)
For H.264 encode, IGPU is used and it's pretty good output (up to to 59.94fps). If I disable the IGPU, the H.264 is done by RX580 and it's very quick too BUT there are green glitches/artefacts in the output after about one second of play.
For HEVC 8-bit 29.97fps encode, the RX580 is utilised. Attempting to encode at 59.94fps HANGS the system.
For HEVC 10-bit encode, the CPU ONLY is used (i.e. software only encode). Output is pretty good BUT takes a LONG time.
For HEVC 8-bit decode (playback) at 29.97fps, the RX580 is utilised.
For HEVC 10-bit decode (playback) at 59.94fps, the CPU is utilised instead and frames are dropped (stutters since the i7 3770K is not powerful enough).
WITHOUT Dummy kext, it's NOT possible to utilise the RX580 for HEVC (or H.264) encode/decode.
Above steps done with FCPX.
I tried iMacPro1,1 + dummy141.kext + IGPU disabled.
Same results as above, except CPU power management is erratic (e.g. does not idle down to 1.6GHz)