No there's no way to export H.264 through RX 580.
Only way to get H.264 encode is by the CPU+IGPU.
So on the Ivybridge, e.g. you have to enable the RX 580 as primary GPU and enable IGPU in the BIOS.
Then, set the ig-platform-id to 0x01620007 (connector-less).
You will then be able to export H.264 in FCPX and VideoProc Apps.
But, with the IGPU enabled, you will NOT be able to export HEVC (H.265) at all.
If you only use HEVC, then you have to disable the IGPU in the BIOS and then RX 580 will be utilised and it's quite fast.
Quality is not too bad.
It's a case of one or the other (H.264 or HEVC) but not both.
This is the case even if have the latest CPU with IGPU (e.g. i7-8700k) !
In fact with the latest i7 and i9's, WITHOUT the cost of a dGPU, will give the ability to do BOTH codecs (with encode AND decode) without any hassle.
Currently 10.14.1 is even worst as the HEVC is disabled for RX 580 !
In effect, the RX 580 is useless for the current mainstream codec.