Ok, I spent some time this evening looking at this particular kext, and why somehow it made my 5K screen works, and solved all my other screen display problem
Analysing the source code, and after a bit of test&trials, I found that the most important part to make it work was this code:
https://github.com/aerror2/NGFixLG5K/blob/master/NvidiaGraphicsFixup/kern_ngfx.cpp#L149
It loads AppleGraphicsDevicePolicy.kext and rename "board-id" into "board-ix" (my guess is that any replacements would do, so long as it's no longer "board-id")
So rather than making a custom kernel extension, I attempted to do the same using clover kernel patch:
Code:
Name* Find*[Hex] Replace* [Hex] Comment
AppleGraphicsDevicePolicy 626F6172 642D6964 626F6172 642D6978 AppleGraphicsDevicePolicy board-id -> board-ix
Reboot, and it all works....
All 3 DP ports working on my AMD Vega, and the 5K works at native resolution.
Hardware decoding *and* encoding works...
Funny on how it's always the most trivial solution that works the best..
@kgp may want to include this to your guide ...
@NWSnowBum can you try this on your side (after removing Lilu and the Nvidia kext) and see if things work better for you?