Overall I'd say this guide would work. I'd probably use a bigger chassis, these mini chassis don't have space for lots of cards.
-32gb ram
No problem. I'm using 32GB Ram
-support for noctua nf-f12's or similar ultra low noise cooling options
Cooling options shouldn't be an issue. These are largely BIOS controlled, so if the motherboard supports it, it should work.
-PCIe slot for HDX or HD Native card (incl. one power connection on motherboard), Thunderbolt-port for supported PCIe-chassis or CoreAudio-supported audio device
Shouldn't be an issue if it's a PCIe card and OS X has drivers.
-Supports 64-bit AAX plug-ins in Pro Tools
Don't know enough about that.
-supports FireWire 800
Shouldn't be an issue with a supported card.
-I basically want to drive this computer like a imac pro and use it for recording and post production.
With a bit of digging you can work out which brand of cards Apple uses and buy the same (or very similar) one. In these cases OsX has all the drivers already and it just works (possibly with a bit of config with Clover). For example Apple uses Broadcom wifi/BT cards. I got the same one and it just worked, but a lot of the supported options have already been found and are in the recommended hardware guide on this site. I would try and stick to the supported hardware in the tony Mac hardware guide as far as possible. Saves a LOT of digging!
If you can't find something supported for a specific feature you need, search ****** hackintosh thread and tonymac for the feature you want and see if you can find anyone who's tried a build with the piece before you buy it.
Finally- look at my Sierra guide. It's actually easier than this one and more up to date. (High Sierra will also offer native support for a lot of the new NVIDIA cards which can be a pain in ElCap and Sierra).