TheWolf
CLASSIFIED Member
- Total Posts : 3841
- Reward points : 0
- Joined: 2007/11/14 16:05:23
- Location: Pascagoula, Ms
- Status: offline
- Ribbons : 9
Just wondering if anyone has set up a Cluster plus was able to fold on it? Is this even possible to fold on a cluster using all nodes of the cluster?
|
Simba123
CLASSIFIED Member
- Total Posts : 2853
- Reward points : 0
- Joined: 2011/05/10 23:15:21
- Location: Australia
- Status: offline
- Ribbons : 14
Re:Cluster & Folding?
2012/11/16 01:40:27
(permalink)
Interesting concept.
I'm sure if it were possible, it would have been done by now.
|
mflanaga
CLASSIFIED Member
- Total Posts : 3033
- Reward points : 0
- Joined: 2009/05/13 18:48:11
- Status: offline
- Ribbons : 15
Re:Cluster & Folding?
2012/11/16 03:42:50
(permalink)
Nope:  by bruce » Thu Apr 26, 2012 7:14 am I've provided comments about clusters in several different topics. As I said there, FAH only runs on individual machines. Any of the machines that you might harness together in a cluster is probably fast enough to run the Uniprocessor client, but each machine will be working on a different WU. My laptop is an old dual-core machine and it's just fast enough to meet the dual-core deadline for SMP, but there's no way to cluster it with other machines. If software were available to do that, it would spend most of it's time waiting for data to move across whatever connection I would have between the machines. FAH moves massive amounts of data between the threads and you wouldn't be able to keep multiple nodes busy. The SMP client REQUIRES a memory-to-memory data transfers, not something slow like a network to interconnect the nodes. It's only efficient if they're working on separate WUs.
|
TheWolf
CLASSIFIED Member
- Total Posts : 3841
- Reward points : 0
- Joined: 2007/11/14 16:05:23
- Location: Pascagoula, Ms
- Status: offline
- Ribbons : 9
Re:Cluster & Folding?
2012/11/16 09:26:49
(permalink)
mflanaga
Nope:
by bruce » Thu Apr 26, 2012 7:14 am
I've provided comments about clusters in several different topics.
As I said there, FAH only runs on individual machines. Any of the machines that you might harness together in a cluster is probably fast enough to run the Uniprocessor client, but each machine will be working on a different WU. My laptop is an old dual-core machine and it's just fast enough to meet the dual-core deadline for SMP, but there's no way to cluster it with other machines. If software were available to do that, it would spend most of it's time waiting for data to move across whatever connection I would have between the machines. FAH moves massive amounts of data between the threads and you wouldn't be able to keep multiple nodes busy. The SMP client REQUIRES a memory-to-memory data transfers, not something slow like a network to interconnect the nodes. It's only efficient if they're working on separate WUs.
I'm not saying it can be done or work, but Bruce is talking about 386 & 486 machines in this statement. I remember this post or one like it from back in very early 2000. Times have changed can't help if he's stuck in the past. lol Machines "server" are much faster today, also there network to interconnect the nodes are faster. We didn't have Gigabit Lan back then. I still find it very interesting and would like to give it a try, "just to say I have done it" but not at the extent of a lot of lost time that could be used to fold if the out come doesn't or will not work. Any real up to date Cluster users here that can shed some light into newer hardware out come? Such as 2p 1366 rackable units with dual L5520, E5620 or even X5670? How about some comments from those heavy with matching 4p systems?
|
widsss
ACX Member
- Total Posts : 426
- Reward points : 0
- Joined: 2011/03/07 09:56:20
- Status: offline
- Ribbons : 0
Re:Cluster & Folding?
2012/11/16 09:36:13
(permalink)
Isn't one of the features of Kepler that a GPU will be able to send information to another GPU? Making it possible for multiple cards to work on the same project.
  Intel i7 3770K @ 4.5 Ghz. / Asus P67 Pro / 2 x EVGA GTX 980 SC / 16gb Corsair Vengeance RAM / 2 x Crucial 64 gb. SSD / 1 Tb. Seagate / Corsair AX 850 / Corsair H100
Intel i7 3930k @ 4.3 Ghz / Asus Rampage IV Extreme / 2 x EVGA GTX 980 Ti SC / 32 gb Gskill Ripjaws RAM / 256 gb Vertex 4 SSD / Corsair AX1200i / Corsair H110i GT
|
tank1023
CLASSIFIED ULTRA Member
- Total Posts : 5191
- Reward points : 0
- Joined: 2009/02/19 06:44:05
- Location: Colorado
- Status: offline
- Ribbons : 15
Re:Cluster & Folding?
2012/11/16 09:57:03
(permalink)
There has been talk about setting up clusters since I first started folding, never seen it done successfully.
|
Xavier Zepherious
CLASSIFIED Member
- Total Posts : 4632
- Reward points : 0
- Joined: 2010/07/04 12:53:39
- Location: Medicine Hat ,Alberta, Canada
- Status: offline
- Ribbons : 15
Re:Cluster & Folding?
2012/11/16 10:02:07
(permalink)
actually if you look into boulder (with ARM) you may have SMP and GPU together the story at XBITLABS have indicated thar ARM chips will be going into GEFORCE cards down the road and Nvidia and F@H will be making them work together with each other to help out each project gpu helping smp and cpu helping GPU( using SLI) so what you may get is 4 arm chips with 8 cores or more each working in SMP (so 32+ cores) this doesn't preclude the other cpus on the MB other thing to note some science can't be done on cpu alone nor gpu alone there is going to be need for more complex wu's that require a coordination/collaberation between cpu and gpu to accomplish also expect a native linux core coming down the pipe (so you can run within v7 in linux) F@H projects will most likely be ARM (arm servers) PHI LINUX (and why Nvidia is suddenly doing linux support) linux support is going to grow servers workhorse like titan will most likely use linux for speed and will need linux cores for GPU's that run native for efficiency (no more wine)
post edited by Xavier Zepherious - 2012/11/16 10:03:25
|
Simba123
CLASSIFIED Member
- Total Posts : 2853
- Reward points : 0
- Joined: 2011/05/10 23:15:21
- Location: Australia
- Status: offline
- Ribbons : 14
Re:Cluster & Folding?
2012/11/16 14:16:48
(permalink)
widsss
Isn't one of the features of Kepler that a GPU will be able to send information to another GPU? Making it possible for multiple cards to work on the same project.
That is a totally different scenario. Data passed over the PCIE lanes has a bandwidth in excess of 200 Gb/s You can do that with optical fiber NICS, though it's a bit pricey (much cheaper now than even 2 years ago though) It might be a concept worth revisiting if Stanford ever has the spare time/cash for it. Everything involved is orders of magnitute faster now than 10 years ago, which makes the concept far more viable. I think the major limitation in times past was getting the data from 1 computer to the next. Still, it may be too much given just how much data F@H actually sends between the Memory/GPU and CPU to be a truly scalable option for clustering
|
TheWolf
CLASSIFIED Member
- Total Posts : 3841
- Reward points : 0
- Joined: 2007/11/14 16:05:23
- Location: Pascagoula, Ms
- Status: offline
- Ribbons : 9
Re:Cluster & Folding?
2012/11/16 16:29:56
(permalink)
Thanks guys for posting. Some very interesting stuff, some I didn't know about. Clustering actually looks a little over my head, but was willing to give it a try as a learning experience. I've always liked to try to improve my knowledge about computers and maximize performance of the computer for FaH.
post edited by TheWolf - 2012/11/16 16:31:39
|
Punchy
CLASSIFIED Member
- Total Posts : 2872
- Reward points : 0
- Joined: 2010/02/06 09:33:05
- Status: offline
- Ribbons : 14
Re:Cluster & Folding?
2012/11/16 16:52:34
(permalink)
I've folded on "clusters" both with the old MPI-based SMP cores and with the current thread-based cores. Not clusters like the common HPC clusters with each system running its own OS, but a sort of "reverse virtualization" which makes a group of systems look like one big system running one OS copy. The interconnects in both cases were Infiniband, one a custom device and one off-the-shelf.
The old MP-based core scaled very well, but if I recall correctly there wasn't a QRB at the time so it really didn't make any difference whether you ran 1 WU twice as fast or 2 WUs half as fast. The current core scales poorly because there is too much memory sharing going on and the interconnect becomes the bottleneck.
|
TheWolf
CLASSIFIED Member
- Total Posts : 3841
- Reward points : 0
- Joined: 2007/11/14 16:05:23
- Location: Pascagoula, Ms
- Status: offline
- Ribbons : 9
Re:Cluster & Folding?
2012/11/16 20:13:03
(permalink)
Punchy
I've folded on "clusters" both with the old MPI-based SMP cores and with the current thread-based cores. Not clusters like the common HPC clusters with each system running its own OS, but a sort of "reverse virtualization" which makes a group of systems look like one big system running one OS copy. The interconnects in both cases were Infiniband, one a custom device and one off-the-shelf.
The old MP-based core scaled very well, but if I recall correctly there wasn't a QRB at the time so it really didn't make any difference whether you ran 1 WU twice as fast or 2 WUs half as fast. The current core scales poorly because there is too much memory sharing going on and the interconnect becomes the bottleneck.
Thanks for your input & knowledge of clustering. Highlighted above is what I was/am looking to do "maybe". Is there anything in a written guide on the subject you could point me to? Also what was the OS that was used at the time, outdated now I'm guessing? One of the rigs I have in mind of using has Infiniband, I think I could pick up a 2nd "barebone" "used" SGI C1001-TY8 very reasonable.
|
A FURRY
ACX Member
- Total Posts : 425
- Reward points : 0
- Joined: 2011/04/21 02:06:42
- Location: ⚡ Два.ч - Фурри / Бред / Мотоциклы / Оружие
- Status: offline
- Ribbons : 3
Re:Cluster & Folding?
2012/11/17 00:54:55
(permalink)
"Yo dawg, I heard you like clusters, so we put a cluster on your cluster so you can fold while you fold!"
The folding system is already a cluster. Your PC is one part of the cluster.
I do not see how folding with 10 systems on 10 separate folding clients would do more work than 10 systems in a cluster running 1 folding client. The total GFLOP/S output should be the same.
The only thing that really makes a difference in my observation is the efficiency of your OS. I get more points per day CPU folding on Linux than I did when I tried CPU+GPU folding on Win7 with a GTX570, and at half the wattage.
My Systems: i7 3770K Rig with GTX570 | AMD FX-8350 CPU Folding Rig | AMD FX-8350 CPU Folding Rig | Dual Xeon 2.66GHz Mac Pro With GTX550Ti | i3 Laptop | Core2Duo Laptop | Atom Dual Core 1.6GHz Netbook | Atom Single core 1.6GHz Netbook All running Linux. (Most running my own customization, "Longcat OS") One HDD with Win7 on it just for games that won't run under WINE.
|
widsss
ACX Member
- Total Posts : 426
- Reward points : 0
- Joined: 2011/03/07 09:56:20
- Status: offline
- Ribbons : 0
Re:Cluster & Folding?
2012/11/17 04:44:34
(permalink)
A cluster would give you a higher QRB over 10 systems running 10 separate instances. The amount if cycles would be the same, but you'd be rewarded for completing each Wu fast.
Intel i7 3770K @ 4.5 Ghz. / Asus P67 Pro / 2 x EVGA GTX 980 SC / 16gb Corsair Vengeance RAM / 2 x Crucial 64 gb. SSD / 1 Tb. Seagate / Corsair AX 850 / Corsair H100
Intel i7 3930k @ 4.3 Ghz / Asus Rampage IV Extreme / 2 x EVGA GTX 980 Ti SC / 32 gb Gskill Ripjaws RAM / 256 gb Vertex 4 SSD / Corsair AX1200i / Corsair H110i GT
|
TheWolf
CLASSIFIED Member
- Total Posts : 3841
- Reward points : 0
- Joined: 2007/11/14 16:05:23
- Location: Pascagoula, Ms
- Status: offline
- Ribbons : 9
Re:Cluster & Folding?
2012/11/17 12:56:10
(permalink)
☢ OMG A FURRY ☢
"Yo dawg, I heard you like clusters, so we put a cluster on your cluster so you can fold while you fold!"
The folding system is already a cluster. Your PC is one part of the cluster.
I do not see how folding with 10 systems on 10 separate folding clients would do more work than 10 systems in a cluster running 1 folding client. The total GFLOP/S output should be the same.
The only thing that really makes a difference in my observation is the efficiency of your OS. I get more points per day CPU folding on Linux than I did when I tried CPU+GPU folding on Win7 with a GTX570, and at half the wattage.
The point would be QRB, 10 systems working 1 work unit returned would be a fraction of the time of working 1 single WU per system. There for your PPD from the QRB would be greater as credit for the quick return WU would be higher. Example: Single Computer Example: 2xCluster Project 8101 Project 8101 Base Credit: 22607 Base Credit: 22607 Time Per Frame: 30 mins Time Per Frame: 15 mins Completion time: 2.1 days Completion time: 1.05 days Bonus Credit: 197629.3 - Bonus Credit: 288854.2 = 91224.9 Total Credit: 220236.3 - Total Credit: 311461.2 = 91224.9 This is assuming it would take half the time with 2 system working the same project. I very much doubt you would cut the time in half, but if you only gain 10 mins quicker TPF it'd still be more.
post edited by TheWolf - 2012/11/17 17:08:18
|
Re:Cluster & Folding?
2012/11/23 13:52:38
(permalink)
When you guys figure this out, let me know.
Main ASUS Max IV Gene Z/Gen 3, I7 2700K@4.7, 2x4g 2000, 2xGTX 780SC, WD 1.5tb black, OCZ 60g for RST, AX 1200 PSU, H100i, HAFX-M 24/7 folders 1.) ASUS P8Z77-I Deluxe, I5 3570K@ 4.7, H80, 2x4g 1600, GTX780SC, CX600M PSU, HAFX-B 2.) ASUS R3F, i7 960@3.6, 6x2g 1600, 3xGTX570SC's, CMGold1200 PSU,H100i, HAFX-B (down till winter) 3.) ASUS Z77 with i5 3570K @ 4.4, GTX570SC, CS600 PSU, CM Elite 311 (down till winter) 
|
TheWolf
CLASSIFIED Member
- Total Posts : 3841
- Reward points : 0
- Joined: 2007/11/14 16:05:23
- Location: Pascagoula, Ms
- Status: offline
- Ribbons : 9
Re:Cluster & Folding?
2012/11/23 15:58:16
(permalink)
The reason I was interested is I have two 16 core systems.
They have to be pretty high overclocked to return a Big Advance work unit in a timely manner for the QRB.
But if I could cluster these 2 that give 32 cores & improve the return time plus they
wouldn't need to be as overclocked as they are.
Not that I would drop the OC, just saying I could if heat ever became a problem or something else.
|
Punchy
CLASSIFIED Member
- Total Posts : 2872
- Reward points : 0
- Joined: 2010/02/06 09:33:05
- Status: offline
- Ribbons : 14
Re:Cluster & Folding?
2012/11/23 20:47:10
(permalink)
The problem last time I looked at this is the cost of the software. The only solution I know of is ScaleMP and it's very expensive; so much so that you would be better off buying more systems rather than trying to cluster them.
|
TheWolf
CLASSIFIED Member
- Total Posts : 3841
- Reward points : 0
- Joined: 2007/11/14 16:05:23
- Location: Pascagoula, Ms
- Status: offline
- Ribbons : 9
Re:Cluster & Folding?
2012/11/23 22:36:46
(permalink)
Punchy
The problem last time I looked at this is the cost of the software. The only solution I know of is ScaleMP and it's very expensive; so much so that you would be better off buying more systems rather than trying to cluster them.
Roger that!
|
TheWolf
CLASSIFIED Member
- Total Posts : 3841
- Reward points : 0
- Joined: 2007/11/14 16:05:23
- Location: Pascagoula, Ms
- Status: offline
- Ribbons : 9
Re:Cluster & Folding?
2012/12/13 14:46:52
(permalink)
|
linuxrouter
Omnipotent Enthusiast
- Total Posts : 8043
- Reward points : 0
- Joined: 2008/02/28 14:47:45
- Status: offline
- Ribbons : 102
Re:Cluster & Folding?
2012/12/13 19:15:32
(permalink)
I think if we can get fast low latency interconnects between systems via Intel QPI links or AMD HT links then clustering may be more feasible. There were rumors floating around a while back that Intel was working on an external QPI solution. I have not read much new about this in recent times however. I would very much like to see how ScaleMP on a cluster of systems connected via QPI or HT links would perform.
post edited by linuxrouter - 2012/12/13 19:16:49
|
