I'm also getting these issues with my GTX 660 and High Sierra.
At the moment, the pain is just about bearable. The glitch will appear maybe once every two or three hours. Some apps seem to set it off more than others. The GenyMotion Android emulator seems to trigger it most. Sometimes the glitches spread to other apps and the OS itself, whereas other times they're localised only to that one particular app. Restarting the app will often fix it. Changing the Display to another resolution, then back again, fixes it sometimes but not others.
Looking at alternative cards available in my neck of the woods, I can get a 2GB GTX 1050 for $160 (Australian). That would double that amount of video RAM over my GTX 660, at least. I don't know if it would add much else to performance though. The rest of my machine is quite old, so I don't want to spend huge amounts on upgrades.
My Mac graphics needs aren't that demanding, but I do a lot of Steam gaming under a Windows partition on the same machine - e.g. Batman Arkham Knight at the moment.