Used 1080Ti or new 5700XT for Mac Pro 5,1?

May 3, 2020
A1289-EMC2629, 48 Gb RAM
3,33GHz hexacore “Westmere”
nVidia GTX980 (Gigabyte)
iMac, MacBook Pro, Mac Pro
Mobile Phone
Hi guys,

I will keep this short as I’m not even sure if anyone is interested in my question, or in case I posted it in the wrong forum thread. It’s my first post, sorry for being such a noob.

I am running against the boundaries of what I can squeeze out of my nVidia GTX980 on macOS 10.13 “High Sierra” / Windows 10 x64Pro. I am driving a modest 4K display through the fastest DisplayPort cable I could find, and am struggling to keep the framerates above the 30fps in some games. My Mac Pro 5,1 is doing otherwise just fine with its 48 Gb of RAM.
High Sierra works fine for me, and there‘s still a little bit of headroom to upgrade my GPU while sticking to nVidia web drivers. I have my other systems upgrade to Mojave & Catalina, and feel no need to upgrade the Mac Pro’s OS.

Question: I want a GPU upgrade for, let’s say, $750 max. Would you choose:

a. A new (or used) AMD Radeon 5700XT (and upgrade to Mojave or Catalina)
b. A used nVidia 1080Ti and stick to High Sierra
c. do neither and wait for a new future high-end AMD card with native macOS support
d. do nothing or something else?

I’m very curious to hear from you. It’s just that I’m not very impressed with AMD’s current line-up of ’high-end’ GPUs, even though they’re miles faster than my old GTX980 (non-Ti). And yes, I know that my RAM is slow, my motherboard cache is small, my SSD has too few PCIe lanes and my processor is old, but all I’m asking is a few extra frames in current games and future-proofing for upcoming titles.
Thanks in advance and keep up the good work, TonyMac!