Another interesting bit: When I run Sierra and fire up Octane, I have >4GB of free VRAM on my 980Tis.
On High Sierra, free VRAM drops to less than 2GB. That makes rendering large scenes impossible.
Wondering, if Metal 2 screws up everything here...
It's not Metal that's the issue - it's that there's a huge VRAM memory leak issue with nVidia drivers and macOS. This is the reason why I dumped nVidia and moved to AMD.
Anyone who says nVidia drivers are fine to this day is losing their mind. They are either using their computer for simple things like browsing or social media.
Also remember that macOS Sierra 10.12.6 worked fine with nVidia. Apple made many changes to High Sierra and nVidia doesn't care to update.
I was using Octane as well along with Premiere Pro and Resolve with CUDA, but the memory leaks made it impossible to work.
The memory just fills up and macOS becomes unusable unless you restart, and then it will just fill up again.
Apple moved to Metal for the WindowServer in High Sierra and made major under the hood graphical changes by adding eGPU support and other things. nVidia just didn't keep up or optimize their drivers for High Sierra. This is why nVidia calls them "unofficial drivers."
I suggest you dump nVidia and move to AMD like I did. I don't have to worry about waiting for drivers every update with the Vega FE.
If you really want to use Octane with your 980Ti, either boot into Windows and use it on the same machine, or build a 2nd machine with nVidia GPU only and keep your Hackintosh with AMD.
Also FYI Otoy is working on a Metal version of Octane so AMD will be supported under macOS in a few months (most likely late Q4 of 2018).