Disabling Whatsevergreen.kext and radeint to be unchecked did same results. Also tried to replace the Vega 64 with a RX580 board working on another z370 based system, then the result was kernel panic and reboot on 10.13.3. Moreover installing the RX580 to z170 system worked all fine. So it is not the motherboard issue.
I think you are right, but weird thing is happening, I can reach to 4k@1080p desktop screen but all UIs are so laggy, the launchpad opens like slow motion, chrome is keep rerendering, iTerm is ok, so my guess is that the Metal related functionality is not loaded properly, not sure at all though
Anyway I will try the MetalFeaturesInspector later today.
My feeling is that the problem is peculiar to the graphics card itself, I am thinking to sell it and get another new one...