I had High Sierra with 17,1 and HD 6870 without problems, downgraded to 14,2 'cause someone noted that might work, but no luck!
Will try 13,1 later, and try 14,2 Board ID (Mac-27ADBB7B4CEE8E61) with your iMac18,3 as P1LGRIM mentioned, though have only DVI and HDMI, no DP adapter available at the...