I'm not really sure what's happening. Your CPU benchmark results sound reasonable. I'm currently getting 23202 from Cinebench R23, which is 1.33x times your result. I have 1.28 more cores plus an overclock, though not a huge overclock at the moment - I tuned down my overclock temporarily and also have the max temperature set to 90C which is thermal throttling me on my AIO.
So your Cinebench score sounds reasonable relative to mine. My current Geekbench 5.4 CPU score is 17456 which is 1.6x times your score. I imagine here my overclock is making a bigger difference, as Geekbench doesn't run hard/long enough to hit any thermal throttling. But your score still sounds reasonable for a 14-core CPU without an overclock. (My highest Geekbench score in past testing was 19063, but that included a higher CPU overclock plus a RAM OC with tightened timings.)
You have a 48-lane CPU, meaning an NVMe in slot M2M will reduce PCIEX8_1 (slot 2; second down from RAM slots) to x4 speed. So if you have a GPU in PCIEX8_1 then yes it might perform badly - it'd be like running it as an eGPU .
But surely you have your GPU in PCIEX16_1 (slot 1) or PCIEX16_2 (slot 3)? There's no resource sharing with the x16 slots, so you can run 2 x full x16 GPUs at once on this board, regardless of NVMe usage.
However it seems like this CLO3D software doesn't have any GPU acceleration on macOS, it's CPU only? (I Googled and their website says it only supports NVidia CUDA acceleration.) In which case I have no idea why it would perform badly given your CPU benchmark results seem fine.
In case there are other apps you're using which do use GPU, what are your Geekbench Compute GPU scores in Metal and OpenCL modes? My Vega 64 gets around 64000 in the Metal benchmark. I'd assume your 5700XT should be a bit higher than that.
Based on what you said I can't see any obvious problems. As long as your GPU is in one of the x16 slots you should get full performance out of it, and your CPU benchmark results sound OK.
Is it just CLO3D that performs badly? What about other software? Maybe there's some weird issue specific to that software.
Do you also have a Windows installation? If so, I'd do some testing in that also, to get a comparison. Install some of the software you use in Windows, and install some Windows monitoring software like
HWInfo and check your core temperatures and other CPU/motherboard stats while running some high loads; maybe you're thermal throttling due to insufficient cooling?
You can also check temps in macOS using
HWMonitorSMC2 - and also install
Intel Power Gadget so that HWMonitorSMC2 can read all values from the CPU.