Thinking of going Z170 mITX for my new build because I've always found the small form factor pretty nifty. Does anybody have any experience using a GTX 1080 with the hybrid cooler in a mITX case? Would it be too much to expect to use a hybrid 1080 and a Corsair H100 in a mITX case? 

Depends on the case. What case do you have in mind?

It would definitely be a tight fit, any cases you have in mind? And have you thought about maybe just a 120mm or 140mm rad AIO cooler?

Thinking of going with the NZXT Manta

I have already built a mini-ITX system using the M1 NCASE and the EVGA GTX1080 FTW Hybrid fits nicely.  I had to order special 90-degree PCI-E power connectors in order to be able to attach the side panel back onto the case.  Here are the specs:
 
Intel Core i5 6600K @ 4.5Ghz
Corsair H55 AiO CPU cooler
Gigabyte Z170N Gaming 5 mini-ITX motherboard
16GB Corsair Vengeance 3200Mhz DDR4
EVGA GTX1080 FTW Hybrid
Samsung Pro 950 PCI-E NVMe M.2 512GB (Windows 10 Pro 64-bit)
Samsung EVO 850 1TB SSD (storage)
Panasonic Blu ray writer slim drive
Corsair SFX 600W power supply
 
And some pics:
 

Iluv2raceit
I have already built a mini-ITX system using the M1 NCASE and the EVGA GTX1080 FTW Hybrid fits nicely.  I had to order special 90-degree PCI-E power connectors in order to be able to attach the side panel back onto the case.  Here are the specs:
 
Intel Core i5 6600K @ 4.5Ghz
Corsair H55 AiO CPU cooler
Gigabyte Z170N Gaming 5 mini-ITX motherboard
16GB Corsair Vengeance 3200Mhz DDR4
EVGA GTX1080 FTW Hybrid
Samsung Pro 950 PCI-E NVMe M.2 512GB (Windows 10 Pro 64-bit)
Samsung EVO 850 1TB SSD (storage)
Panasonic Blu ray writer slim drive
Corsair SFX 600W power supply
 
And some pics:
 

Iluv2raceit, I have the NCASE M1 as well. Currently in the middle of my build. My thought was to order the FTW Hybrid 1080 and mount the radiator to the upper back part of the case on the outside. Is this possible at all? I wanted to use the side panel to mount a 240mm AIO for the CPU. But I want to make sure the FTW Hybrid radiator will be able to mount to upper back of the NCASE M1 on the outside.

I have my FTW Hybrid in the Corsair 250D, fits nicely :)
 
 

I will put one in a Phanteks Enthoo Evolv mITX. Curious about how it will actually fit. Was a bit concerned about the tubing not being flexible enough but the pics shown in here suggest they are pretty flexible.