EVGA

Helpful ReplyStraight forward 4 GPU folding system - Thoughts?

Page: 12 > Showing page 1 of 2
Author
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
2016/10/04 20:51:28 (permalink)
Well Chris21010's rig has been inspirational, but I'm not quite the artist / craftsman he is, so I'm trying figure out a more straight-forward multi-gpu machine I could build.  Simple, reliable, high bang for the buck and minimal use of a dremel are the initial design criteria.
 
Four GPU's on air would generate a lot of heat, so hydro seems the way to go.  Given the simple design goal plus the fact that I have zero liquid cooling experience, I'm thinking all AIO cooling.  The CPU must have 40 PCIe lanes and the motherboard must accommodate 4 double width graphics cards without modifying the cards or motherboard.  I'm not sure how to pick a case that would handle 5 AIO radiators attached to it somehow.
 
Here are the basics:
GPU - EVGA GeForce GTX 1070 FTW HYBRID GAMING x 4 - $489 each
MB - EVGA X99 FTW K - $250
CPU - Xeon E5-2620 V4 2.1(3.0) GHz 85W – 8 – Core – 40PCIe -  85W - $450 - 10 cores adds $300 - E5-2630 V4
SSD - SAMSUNG 850 EVO M.2 500GB - no power or data cables is the motivation behind going M.2 - $160
PSU - EVGA SuperNOVA 1200 P2 80+ PLATINUM 1200W (a 1000W might do, but headroom is never bad) - $230
CPU Cooler - CORSAIR Hydro Series H60 - $60
RAM - 32GB of something compatible, and won't interfere with anything in the case - $200
Case - ??
 
Clearly open to overall suggestions, but here are the open items I'm thinking about:
CPU - Want to crunch with it, so total compute power matters.  When comparing processors, can all cores go Turbo at the same time?  The i7-6800K only supports 28 PCIe lanes, so I think it's out of the running.
MB - Can save ~$80 by going with a GigaByte GA-X99-UD3P, but I don't want the MB to be the weak link.  Is the EVGA X99 FTW K the right MB?
Case - The highest end case I've ever bought is $100 Antec Sonata back in the day.  So I'm starting from scratch on this one.  Any ideas?
 
Figured this just might be something fun to toss around.


#1
ipkha
CLASSIFIED Member
  • Total Posts : 2308
  • Reward points : 0
  • Joined: 2015/02/10 21:01:40
  • Location: Indiana
  • Status: offline
  • Ribbons : 6
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 05:25:00 (permalink)
Looks good to me. All cores can boost if heat is under control. If possible get a larger aio for the CPU to allow for more headroom. My 4 core boosts all with a Noctua DH-15 but a 8 core hyperthreaded CPU would put out more heat. 10 core isn't worth it given the frequency drop.there are others in the forum that have more experience with the Xeons.


#2
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 06:41:38 (permalink)
you can not run 4GPU and m.2 on that mobo. when you put something in that m.2 slot the bottom pci-e is disabled.


#3
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 09:10:58 (permalink)
Chris21010
you can not run 4GPU and m.2 on that mobo. when you put something in that m.2 slot the bottom pci-e is disabled.



Didn't think of this in terms of what features impact pci-e.  Like the idea of no cables, maybe I'll research M.2 add-in cards.  Otherwise it looks like a standard SSD is the way to go.


#4
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 09:13:54 (permalink)
ya m.2 either shares or replaces pci-e lanes. so no matter the motherboard if you are going to try for a 4 GPU setup you'll want to look into a standard SSD.


#5
bcavnaugh
The Crunchinator
  • Total Posts : 38977
  • Reward points : 0
  • Joined: 2012/09/18 17:31:18
  • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
  • Status: offline
  • Ribbons : 282
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 09:33:41 (permalink)
You would be much better off going with Full Waterblocks and a custom Water Loop with 4 Cards.
Finding a Place to set 4 HYBRID Radiators that will sit above the Graphics Cards my be a problem. 
Non-FTW HYBRIDs would work well with 2 Cards.
Remember that the VRAM is still Air Cooled on the HYBRID Graphics Cards Non-FTW HYBRID Models Only.
And having 4 Cards next to each other with no slot space between you cards and Folding 7x24 would run very hot.
post edited by bcavnaugh - 2016/10/05 10:04:13

Associate Code: 9E88QK5L7811G3H


 
#6
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 09:53:32 (permalink)
hybrids will be fine for a 4 card setup as the GPU is the primary source of heat. not to mention that EVGA hybrids also cover the VRAM so they are not strictly air cooled.


#7
bcavnaugh
The Crunchinator
  • Total Posts : 38977
  • Reward points : 0
  • Joined: 2012/09/18 17:31:18
  • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
  • Status: offline
  • Ribbons : 282
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 09:59:30 (permalink)
Chris21010
hybrids will be fine for a 4 card setup as the GPU is the primary source of heat. not to mention that EVGA hybrids also cover the VRAM so they are not strictly air cooled.


Yes the FTW Model has an added Copper Plate for the VRAM, but not the Non-FTW Models.
Corrected my Post Above.
post edited by bcavnaugh - 2016/10/05 10:01:51

Associate Code: 9E88QK5L7811G3H


 
#8
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 10:02:45 (permalink)
bcavnaugh
Chris21010
hybrids will be fine for a 4 card setup as the GPU is the primary source of heat. not to mention that EVGA hybrids also cover the VRAM so they are not strictly air cooled.

Yes the FTW Model has an added Copper Plate for the VRAM, but not the Non-FTW Models.

Great to know.  I didn't know there was a cooling difference.
 
Where did you read that the FTW Hybrid has an added copper plate?  I've been trying to figure out the difference between the FTW and regular 1070 hybrid, and the eVGA spec sheets are identical.


#9
bcavnaugh
The Crunchinator
  • Total Posts : 38977
  • Reward points : 0
  • Joined: 2012/09/18 17:31:18
  • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
  • Status: offline
  • Ribbons : 282
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 10:06:24 (permalink)
mektacular
bcavnaugh
Chris21010
hybrids will be fine for a 4 card setup as the GPU is the primary source of heat. not to mention that EVGA hybrids also cover the VRAM so they are not strictly air cooled.

Yes the FTW Model has an added Copper Plate for the VRAM, but not the Non-FTW Models.

Great to know.  I didn't know there was a cooling difference.
 
Where did you read that the FTW Hybrid has an added copper plate?  I've been trying to figure out the difference between the FTW and regular 1070 hybrid, and the eVGA spec sheets are identical.


http://www.evga.com/articles/01054/evga-geforce-gtx-1080-1070-ftw-hybrid/
 
        

Dedicated Memory Plate

A dedicated memory plate that makes DIRECT contact with the water cooling block. Giving the lowest memory temperatures possible.
post edited by bcavnaugh - 2016/10/05 10:07:53

Associate Code: 9E88QK5L7811G3H


 
#10
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 10:22:58 (permalink)
Will a hybrid 1070 even fit in the bottom slot of the EVGA X99 FTW K?  It looks like there are numerous headers, etc. that would be covered up.
 


#11
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/05 10:34:39 (permalink)
Yes it fits just make sure you plug in you i/o and fans first.


#12
sethleigh
SSC Member
  • Total Posts : 796
  • Reward points : 0
  • Joined: 2015/08/12 11:27:56
  • Status: offline
  • Ribbons : 4
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 02:22:45 (permalink)
I think going with five AIO coolers (4 gpu plus 1 cpu) would be redonkulous. That's now five weak links, making it much, much more likely that you will see a failure of some sort in the future. Water cooling isn't that hard, and there are plenty of people here who can help make this very simple for you. I'd suggest that you just go full-on water cooling and keep your points of failure to a minimum. One good, high-quality pump will be much more reliable than five of whatever crapco pumps they build into the AIO coolers.
 
AIO tubing is typically quite short. You're going to struggle to mount five AIO coolers close enough to all those cards and your cpu. Not to mention it's going to be a veritable rat's nest in there, a jungle of ugly black tubing making your machine look like crap and impeding internal airflow.
 
Seriously, start asking questions about water cooling. Start reading about it. It's very easy to do. It's a far better solution for what you're wanting to do than going with a dog's breakfast of crappy AIO coolers.
 
Also, using the cpu to fold isn't really worth it. I've been folding with 6 or my 8 cores on my i7 6900, but it's only adding like 60k ppd or so to my potential output (1/15 of what my 1080 does) for about half the power draw. You'll also need some cpu cores free to support the gpu folding on four cards.

Yeah, go with a standard SATA SSD - there would be no measurable benefit on this machine from going m.2, and like the others have said, you won't lose a PCI-E slot.

Happy EVGA customer.  Affiliate Code: 0Y7-1VU-ATW2
 
GigaByte X570 Aorus Master, AMD Ryzen 5900x under Optimus Foundation block, 32gb G.Skill DDR4 @ 3800 MHz 14-14-14-28, EVGA 3080ti FTW3 Ultra under Optimus block, 2TB 980 Pro SSD, EVGA Supernova G6 850W PS, ASUS 34" 3440x1440p 120Hz ultrawide, Lenovo 24" 1080p secondary monitor, Win 10

#13
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 06:48:03 (permalink)
http://www.modsrigs.com/detail.aspx?buildid=35814
 
that build confirms for me that you could fit 5 AIO coolers into EVGA's new DG-8 series case. 2 on right, 2 on left, and CPU on top. but depending on your time frame you may want to wait for EVGA to release their new hydro series that they showcased at CES this year. https://www.youtube.com/watch?v=tr6545-WeS4


#14
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 10:06:16 (permalink)
sethleigh
I think going with five AIO coolers (4 gpu plus 1 cpu) would be redonkulous. That's now five weak links ... One good, high-quality pump will be much more reliable than five of whatever crapco pumps they build into the AIO coolers.
 
... Also, using the cpu to fold isn't really worth it.

Thanks.  I figured this might come up and I've never seriously looked into a full hydro solution.  I'll do a bit of research.  My guess is, it will get pretty expensive with $100+ waterblocks per GPU.  Might just have to research this a bit more.  How often do AIO pumps fail?  More often than fans?
 
Also, don't plan to use the CPU to fold, but I'll crunch with it.
 
Great comments.  Thanks again.


#15
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 10:12:00 (permalink)
Chris21010
http://www.modsrigs.com/detail.aspx?buildid=35814
 
that build confirms for me that you could fit 5 AIO coolers into EVGA's new DG-8 series case. 2 on right, 2 on left, and CPU on top. but depending on your time frame you may want to wait for EVGA to release their new hydro series that they showcased at CES this year. https://www.youtube.com/watch?v=tr6545-WeS4


I was just looking at that case yesterday.  Thanks for the links that show real machines.
 
Kind of a fun project.


#16
bcavnaugh
The Crunchinator
  • Total Posts : 38977
  • Reward points : 0
  • Joined: 2012/09/18 17:31:18
  • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
  • Status: offline
  • Ribbons : 282
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 10:17:54 (permalink)
This case can hold 5 or more AIO Cooler Units. Core X9 E-ATX Cube Chassis

 
You could install 4 Hybrid Rads on top and even a H100i v2 on top for the CPU.
 
 
Lots of room in this case

post edited by bcavnaugh - 2016/10/06 11:57:03

Attached Image(s)


Associate Code: 9E88QK5L7811G3H


 
#17
Cool GTX
EVGA Forum Moderator
  • Total Posts : 30983
  • Reward points : 0
  • Joined: 2010/12/12 14:22:25
  • Location: Folding for the Greater Good
  • Status: offline
  • Ribbons : 122
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 10:37:42 (permalink)
Looks like a very nice project
 
My i7-5960X , shows that 280mm (2x 140mm) of AIO is the minimum I would look at for the CPU.  When I push all the cores, wish I had a 360 setup, as the temp does start climbing much higher.
post edited by Cool GTX - 2016/10/06 10:39:49

Learn your way around the EVGA Forums, Rules & limits on new accounts Ultimate Self-Starter Thread For New Members

I am a Volunteer Moderator - not an EVGA employee

https://foldingathome.org -->become a citizen scientist and contribute your compute power to help fight global health threats

RTX Project EVGA X99 FTWK Nibbler EVGA X99 Classified EVGA 3080Ti FTW3 Ultra


#18
bcavnaugh
The Crunchinator
  • Total Posts : 38977
  • Reward points : 0
  • Joined: 2012/09/18 17:31:18
  • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
  • Status: offline
  • Ribbons : 282
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 11:57:14 (permalink) ☄ Helpfulby mektacular 2016/10/06 12:58:50
Maybe even the H115i >?
 
 Blue GTX Hybrid  Rads
I would install them as Air Intakes (Pull )and use the rear as Air exhaust. 
post edited by bcavnaugh - 2016/10/06 12:00:09

Attached Image(s)


Associate Code: 9E88QK5L7811G3H


 
#19
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 11:59:06 (permalink)
Cool GTX
Looks like a very nice project
 
My i7-5960X , shows that 280mm (2x 140mm) of AIO is the minimum I would look at for the CPU.  When I push all the cores, wish I had a 360 setup, as the temp does start climbing much higher.


That i7-5960X is a 140W processor!  The one I'm looking at is 85W which will hopefully run a tad cooler at full load.  That said, I should do a little research into what it takes to cool an 85W processor at full load... 


#20
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 12:00:56 (permalink)
a 240mm or 280mm radiator is plenty for cpu's under 100W


#21
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 13:01:26 (permalink)
bcavnaugh
Maybe even the H115i >?
 
 Blue GTX Hybrid  Rads
I would install them as Air Intakes (Pull )and use the rear as Air exhaust. 


Thank you.  Very helpful about this case and visualizing radiator mounts in general.
 
The article below shows an installation in an Enthoo Primo.  Lots of room.  Plus the thread shows a hydro installation from another PC craftsman.
http://www.overclock.net/...o-primo-photo-heavy/80


#22
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 16:18:09 (permalink)
Photo of a system with 3 AIO Titan X's in it.  Definitely a lot of tubes...
http://forums.evga.com/FindPost/2320877
 


#23
ipkha
CLASSIFIED Member
  • Total Posts : 2308
  • Reward points : 0
  • Joined: 2015/02/10 21:01:40
  • Location: Indiana
  • Status: offline
  • Ribbons : 6
Re: Straight forward 4 GPU folding system - Thoughts? 2016/10/06 19:03:55 (permalink)
Mine gets cooled by a Noctua DH15 air cooler, but even it gets in the upoer 40s at load. i5 4670. With 2 cores for folding, 2 cores for crunching and the integrated GPU crunching seti. I do manage a 3.57Ghz average core frequency though. You need a 140mm radiator for sure. My Noctua generally equates to 1 in some of the comparisons I've seen.
post edited by ipkha - 2016/10/06 19:09:55


#24
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2017/01/14 13:04:32 (permalink)
Super old thread, but wanted to say, I did end up building up a rig with the potential of 4 1070 hybrids.  I only ended up with 2 so far.  Currently debating a 1080 hybrid or a third 1070 depending on funding.
 
Build so far:
GPU - EVGA GeForce GTX 1070 FTW HYBRID GAMING x 2
MB - EVGA X99 FTW K
CPU - I7-6850 - 6 Core – 40PCIe -  3.6 GHz (OC'ed to 4.0GHz)
PSU - EVGA SuperNOVA 1000 G2 80+
CPU Cooler - Coolermaster 212 EVO
Case - CoolerMaster MasterCase 5 Pro
 
I knew the case was a close fit for the Motherboard, but I just went for it figuring I replace an old case with it and buy a new bigger case if I needed to.  The motherboard did fit, but there is no room for the 4th 1070.  So, I have room for one more GPU.
 

 
post edited by mektacular - 2017/01/14 13:21:53

Attached Image(s)



#25
bcavnaugh
The Crunchinator
  • Total Posts : 38977
  • Reward points : 0
  • Joined: 2012/09/18 17:31:18
  • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
  • Status: offline
  • Ribbons : 282
Re: Straight forward 4 GPU folding system - Thoughts? 2017/01/14 13:08:25 (permalink)
How are you going to Mount 4 GTX 1070 Hybrids cards?

Associate Code: 9E88QK5L7811G3H


 
#26
mektacular
iCX Member
  • Total Posts : 331
  • Reward points : 0
  • Joined: 2011/03/09 14:58:14
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2017/01/14 13:11:56 (permalink)
bcavnaugh
How are you going to Mount 4 GTX 1070 Hybrids cards?

Need a new case for 4.  I had hoped this one would be big enough, but it's barely not for the physical cards.  I believe the coolers would fit mounting one on the back of the case and one on the front.
 
However, when I get there I will buy a bigger case.


#27
Cool GTX
EVGA Forum Moderator
  • Total Posts : 30983
  • Reward points : 0
  • Joined: 2010/12/12 14:22:25
  • Location: Folding for the Greater Good
  • Status: offline
  • Ribbons : 122
Re: Straight forward 4 GPU folding system - Thoughts? 2017/01/14 20:32:20 (permalink)
Looks nice
 
Thanks for the update

Learn your way around the EVGA Forums, Rules & limits on new accounts Ultimate Self-Starter Thread For New Members

I am a Volunteer Moderator - not an EVGA employee

https://foldingathome.org -->become a citizen scientist and contribute your compute power to help fight global health threats

RTX Project EVGA X99 FTWK Nibbler EVGA X99 Classified EVGA 3080Ti FTW3 Ultra


#28
MARSTG
New Member
  • Total Posts : 94
  • Reward points : 0
  • Joined: 2012/05/26 09:38:33
  • Location: Montreal, QC
  • Status: offline
  • Ribbons : 0
Re: Straight forward 4 GPU folding system - Thoughts? 2017/01/15 05:56:01 (permalink)
You could interpose a 6162-KR in between them. And keep the case.
#29
bcavnaugh
The Crunchinator
  • Total Posts : 38977
  • Reward points : 0
  • Joined: 2012/09/18 17:31:18
  • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
  • Status: offline
  • Ribbons : 282
Re: Straight forward 4 GPU folding system - Thoughts? 2017/01/15 06:30:59 (permalink)
They would run to hot, their is no air flow for them. http://www.evga.com/products/product.aspx?pn=03G-P4-6162-KR
Only way to run really run 4 GPUs in a single case is water cooled or Hybrid cooled.
I would move the second card down one slot even if it runs at x8 as long at it is at PCIe Gen3 that is.
Could also add a second fan to the CPU for Push/Pull setup with the air flow out the rear.
Add a fan on the front top if one will fit for the CPU as their is only one Intake Fan in the Front and 5 exhaust fans to include the two GPU Fans.
That is only if this rig is only for Folding and Crunching.
post edited by bcavnaugh - 2017/01/15 06:42:39

Associate Code: 9E88QK5L7811G3H


 
#30
Page: 12 > Showing page 1 of 2
Jump to:
  • Back to Mobile