The GTX 660 Ti is known to suffer from graphics artifact problems when running MacOS High Sierra (and probably later) :
Hi There, Anyone experienced a graphic glitch issue when opening any video? this problem appeared after updated to high sierra 10.13, however there was no problem before in sierra 10.12.6, and my graphic card supposed to be native and no need for Nvidia web driver, So, any solution for this or...
www.tonymacx86.com
Asus Z270-F? Is this your motherboard?
ROG Strix Z270 Gaming ATX motherboards deliver top performance, audio, and style with Aura Sync, Intel LAN and Dual M.2 (Successor to Z170 Pro Gaming).
rog.asus.com
If so, and if you want to run MacOS on your computer, I suggest you should remove the 660 Ti and (1) replace it with a compatible AMD card such as RX 580, or (2) just use the CPU integrated graphics of the i7 7700K. Your system will then be compatible with MacOS High Sierra / Mojave / Catalina / Big Sur.
You can keep the overclock if you want.