I'm pretty sure you can achieve the 5K resolution with your GTX980Ti if you use a Dell... what GPU card model do you have? And which motherboard model?
Anyway if I were you, I wouldn't sell your 980Ti, which is really a blazing fast top gpu, for an RX480!
Don't get me wrong here, RX480 is really a nice card and probably in the mid range of performance is the best one you can get at the moment. It's very silent, it consumes 150W of power only and it gets nice performances. The only drawback actually is that you cannot have visibility of the boot process and it starts only at login screen. And I suspect the culprit is in Clover right now, so I hope in the future we can fix it.
But if we talk of pure performances 980Ti is way better and I would change it now. I would wait if there will be a Pascal support from Nvidia in a near future but I wouldn't go to Amd...