There are no performance penalties with either mATX or ATX. They will perform pretty much exactly the same. The only major difference would be the different number of PCI-e slots.
Just look for a motherboard that has the features you like/need. For example, Thunderbolt is not available on the Maximus VIII Gene, so if that's something you need, you should look for another motherboard.
If you are looking for a motherboard that overclocks well, you would have read reviews of the individual motherboards. All Z170 motherboards allow for overclocking. How much difference there is between the motherboards when it comes to overclocking, I don't know. But I suspect that whatever difference there is, it would be very minimal.
The motherboard should not affect the performance of your GTX 1080 Ti. All Z170 motherboards support up to 64GB of RAM as long as they have four DIMM slots. The number of hard drives a motherboard supports just comes down to how many SATA ports are included. The Z170 chipset supports up to six SATA ports.