Your graphics card has two DisplayPort outputs and two HDMI outputs and one DVI output?
You are able to get a 5K virtual resolution with one DisplayPort cable connected to the right DisplayPort output?
Note that you can use SwitchResX to see the actual real output resolution by double clicking on the current display resolution to see the timing information.
View attachment 278081
Does the left DisplayPort output also allow a 5K virtual resolution with one DisplayPort cable? If it does not, then that means there's a problem with the AMD frame buffer setting. Read about it at:
https://www.tonymacx86.com/threads/radeon-compatibility-guide-ati-amd-graphics-cards.171291/
That guide says the reference port layout is 3x DP, HDMI, DVI. That differs from your RX580. I don't know if that's the cause of the problem. The guide also points to a post
#998 which might have a solution (since the 5K display is a MST display).
Are you using Lilu.kext and WhateverGreen.kext?
http://www.insanelymac.com/forum/topic/312582-amd-polaris-ids-on-sierra-high-sierra/
Maybe you need to wait for macOS High Sierra 10.13 for better RX580 support?