bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Purpose:- Raise team awareness of the power of Big CPU folding
- These systems are capable of posting 200K - 800K PPD.
- Provide some organization to the wealth of good information, guides, and my lessons learned.
- My personal goals were to push myself into the Top 20 Producers for Folding@EVGA, maybe motivate my teammates to step up as well, force myself to learn about -BigAdv, Server hardware, and learn to spell Ubuntu.
- I'm certainly not an expert in this area. What works for me, may not work for you or your hardware. This represents about a month of nightly information gathering on the various source. I hope this save you alot of research time.
- Please send me a private message for any correction, updates, rewording, rewrites, and any additional information you think we should include. (It going to need it, because it was written way too late at night)
Table of Contents Hardware and Cost Hardware Decisions and Discussions - CPUs
- Motherboards
- Memory
- Heat sink / CPU Coolers
- Thermal Paste
- Drives
- Power Supply Unit (PSU)
- Case / Mounting System Pictures (Everyone love hardware) BIOS Settings Software- Ubuntu Installation
- Folding Configuration and Utilities
- Hardware Monitor Tools
- Useful Ubuntu Commands References Change Log:6/15/12 - Initial Post 7/14/12 - Added BIOS, PSU update and minor edits 8/8/12 - Added Power Advisory on CPU/EPS adapters 9/1/12 - Added Referrence to Improving Opteron 6200 performance 4/25/13 - Updated Vendors CPU Pricing 7/30/13 - Recommended Bargain / Best Value AMD 4P Build Cost - Added 62xx series chip data and Reasonable OC numbers for 61xx 11/4/13 - Updated Software Sections - Add [H] Folding Appliance and Ubuntu Commands 11/7/13 - Updated Power Draw and PPD Numbers in PSU Section - Update Drive Recommendations 12/8/13 - Update BIOS Settings Recommendations
post edited by bowlinra - 2013/12/08 23:28:20
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware and Cost
2012/06/15 00:04:57
(permalink)
Hardware and Cost (Updated July 2013) Recommended Bargain / Best Value AMD 4P Build Cost (July 2013 Pricing Update)$550 - 4x AMD Opteron 61xx CPUs (Ebay ~$125 - 175 per cpu)
$658 - Supermicro H8QGi+-F AMD Quad G34 Socket Motherboard (New)
$60 - 16x Micron or Hynix ECC 1GB DDR3 1333 or 1600 (Cas 9 v1.5) (Used MAC DIMMs on Ebay $3 -$4)
$100 - 850W GOLD PSU (Prefer Corsair or Seasonic w/ 3yr - 5yr Warranty)
$48 - 4x Cooler Master Hyper 212 Plus (Refurb $12 per)
$8 - SanDisk Cruzer Fit 8GB Flash Drive for Systems (New)
$10 - Zalman’s Super Thermal Grease (STG)
$20 - Misc Hardware for board & CPU Cooler mounting
------------
$ 1,454 - Approx Bargain AMD G34 4P All the prices are reasonable, thou you'll have to hunt to get some on them (Send me a PM, if you need help). You should watch these two thread for assistance: EVGA's AMD 4P Deals and [H] Hot deals for folders . Bargain Rig Stock or OCed is capable of producing on P8101 WUs: - @1.8Ghz ~243K PPD*
- @2.2Ghz ~314K PPD*
- @2.4Ghz ~371K PPD*
- @2.7Ghz ~411K PPD* *Add an Additional 200K+ PPD for the new 8102 - 8105 WUs. Check Power Supply Unit (PSU) section for Watts & PPD. --------------------------------------------------------------------------------------------- (For Legacy Reference) AMD 4P Build Cost (May 2012 Pricing) $1,360 - 4x AMD Opteron 6166HE 1.8Ghz CPUs (Used, w/ 1yr warranty $340 per)
$663 - Supermicro H8QGi+-F AMD Quad G34 Socket Motherboard (New)
$240 - 4x G.SKILL Ripjaws X Series 8GB (4 x 2GB) DDR3 1333 (Cas 7 7-7-7-21) Desktop Memory Model F3-10666CL7Q-8GBXH (New $60 per)
$120 - EVGA Classified SR-2 100-PS-1200-GR 1200W (New $120 on Close Out Special)
$48 - 4x Cooler Master Hyper 212 Plus (Refurb $12 per)
$15 - Seagate 40GB Sata (From the local used PC store)
$0 - LG DVD Drive (From the local used PC store, removed after install)
$10 - Zalman’s Super Thermal Grease (STG)
$20 - Misc Hardware for board & CPU Cooler mounting
------------
$ 2,476 - My out of pocket cost At stock speed this rig is producing (~243K PPD w/ P8101, ~276K PPD w/ P6901, ~380K PPD w/ P6903/4) with an electrical draw of 460W. AMD 2P Build Cost (June 2012 Pricing)$700 - 2x AMD Opteron 6176SE 2.3Ghz CPUs (Used Ebay $350 per)
$250 - Supermicro H8DGU-F AMD Dual G34 Socket Motherboard (Used Ebay)
$120 - 4x Crucial Ballistix 4GB (2 x 2GB) 240-Pin DDR3 SDRAM DDR3 1600 (Cas 8 8-8-8-24) Desktop Memory Model BLT2KIT2G3D1608DT1TX0 (New $30 per) $0 - OCZ ModXStream Pro 700W Modular High Performance PSU (had sitting Idle)
$26 - 2x Cooler Master Hyper 212 Plus (Refurb $12 per)
$15 - Seagate 40GB Sata (From the local used PC store)
$10 - Zalman’s Super Thermal Grease (STG)
$15 - Misc Hardware for board & CPU Cooler mounting
------------
$ 1,136 - My out of pocket cost At stock speed this rig is producing (~125K PPD w/ P8101) with an electrical draw of 340W. Power and PPD Comparison to others: - Stock i7 920 w/ GTX 580@920, GTX 580@900, GTX 260@735 produced ~75K PPD with an electrical draw of 830W - Stock i7 920 w/ GTX 580@920, GTX 260@735 produced ~40K PPD with an electrical draw of 560W - Stock i7 920 w/ 3x GTS 450@950 - 900 produced ~40K PPD with an electrical draw of 560W - Stock Intel Quad Q6600 w/ GTX 460@900 produced ~20K PPD with an electrical draw of 313W - Stock Intel Dual E4400 w/ GTS 8800 produced ~10K PPD with an electrical draw of 235W - Mitsubishi 65in DLP TV has an electrical draw of 208W
post edited by bowlinra - 2013/11/07 15:59:07
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware Decisions and Discussions - CPUs
2012/06/15 00:23:52
(permalink)
Hardware Decisions and Discussions - CPUsIntel vs AMD. I was certainly bias to Intel and believed you only folder on just Intel and NVIDIA GPUs. I definitely thought I was looking at a Dual Xeon setup with an SR-2 motherboard. Xeon CPUs are changing over from the older LGA-1366 socket to the new LGA-2011 socket, the older can't measure up and I didn't want to put money into SMP 16 core system at would struggle to meet the new -BigAdv deadlines. The LGA-2011 CPU are very expensive ie $2,000 per processor and no overclocking motherboard at this point (June 2012). I looked at AMD 6200 Series is the latest and greatest with up to 16 cores (June 2012) & 6100 Series is the last generation model with 8 and 12 cores (You'll also seen them as Magny-Cours or MC) The several debates about the two series, I believe the general thoughts are 6200 IL are newer, faster, have more cores and more expensive BUT don't fold as well as the older 6100 series core for core. The 6100s are starting to be life cycled out of the big high end servers, thus makes for a larger supply on the used markets like Ebay. More details AMD 6100 vs 6200 Debate reference. *EDIT 9/1/12* QuickZ's thread on Improve Opteron 6200's performance (and PPD) by using TurionPowerControl is new information since I looked at this decision. *EDIT 7/30/13* Since the original post, I now own a set of 6272 IL CPUs. I can say, even thou they have 64 cores (instead of 48 cores set of 6172 MC), they produce less PPD at the same clockspeed and use an additional 100W to run. Maybe this will change with future WUs, but all the 8101 - 8105 series do not. I would not recommend for folding over the MC. Here is how I thought about it. I could get an AMD Opteron 6166HE with 12 cores for $340. I couldn't even get a Intel (because I think Intel in the PC world) Hexa / Heca / 6 cores CPU for that price. Intel i7 (970, 980, 990) all costing over $400 on eBay market. I'm sure we could debate of HyperThreading compared to AMD native cores, but the deciding fact was you can't run dual or multi socket motherboards with the Intel i7 (970, 980, 990) only Xeons. In the AMD 6100 series (12 core) looks like 6180SE @ 2.5Ghz -140w
6176SE @ 2.3Ghz -140w
6176 @ 2.3Ghz -115w
6174 @ 2.2Ghz -115w
6172 @ 2.1Ghz -115w
6168 @ 1.9Ghz -115w
6166HE @ 1.8Ghz -85w
6164HE @ 1.7Ghz -85w In the AMD 6100 series (8 core) looks like 6140 @ 2.6Ghz -115w
6138 @ 2.5Ghz -115w
6136 @ 2.4Ghz -115w
6134 @ 2.2Ghz -115w
6132HE @ 2.1Ghz -85w
6128HE @ 2.0Ghz -85w
6124HE @ 1.8Ghz -85w SE - Top of the Line HE - Energy Efficient ES - Engineering Samples (These are test and evaluation units, not sold to the public, but yet on eBay hmm) ----------------------------------------------------------------------------------------------- (Update July 2013) Alot more research and data collection has occurred since the original post, Here are some reasonable Overclocking estimated: 6164 HE/6166 HE: ~2.25Ghz (CPU clock multiplier 250) *Recommended DDR3 1600 Usage*
6168/6172/6174: ~2.41Ghz (CPU clock multiplier 230)
6176 SE/6180 SE: ~2.64Ghz (CPU clock multiplier 230) ----------------------------------------------------------------------------------------------- I selected the AMD Opteron 6166HE with 12 cores for $340 (May 2012) Here is a great reference to see what -BigAdv numbers are being posted by these and other big CPU / server hardware rigs. https://docs.google.com/s...XWVZ2S2xvejBDcHc#gid=0 Where to Buy: (~$340) Ebay.com IT Creations, Sales Manager (Jay Tessel ( IT Creations, Inc ) (jay@itcreations.com) and telling him your with "Stanford's Folding Project" IT Creations offers a 1yr warranty and Free Shipping with these price. Please remember prices do fluctuate depending on the market and there inventory, but these should be good for the near future. (Updated April 2013)
post edited by bowlinra - 2013/12/08 23:26:20
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware Decisions and Discussions - Motherboards
2012/06/15 00:33:54
(permalink)
Hardware Decisions and Discussions - Motherboards In the 4 Processors (4P) / Quad Socket / MultiProcessor (MP) Motherboard, there not a lot of choices the two main ones are TYAN S8812 (S8812WGM3NR) and SuperMicro H8QGi / H8QGL / H8QG6 series. I selected the Supermicro because the folding team [H]orde / [H]ardOCP created a BIOS upgrade that could allow you to overclock the CPUs, I saw posted claims 30% overclocks or better. I picked SM H8QGi+-F because of the price and CPU socket placement, I felt it would be easy to cool than the cheaper SM H8QGL. I did not consider the H8QG6 series due to the added cost of SAS controller that I didn't need. Additional information on the different board series. SM H8QGi+-F Benefits - Socket placement was good for cooling. - Embedded graphics card (no need to buy one, also add a video card could hamper cooler by blocking the CPU cooler). - Motherboard works with just one cpu and would allow you to continue to upgrade by adding addition cpus over time. This was a big advantage over buying a 2P (2 Processor / Dual Socket) motherboard at 65% of a the cost of a Quad socket board. - Although I don't recommend this, but you can theoretically use different / mismatched CPUs on the board. It will just drop the overall CPU speed down to the lowest common denominator. - Biggest benefit is the ability to get great support for the folks at the [H]ardForum. ----------------------------------------------------------------------------------------------- (Update July 2013) The folding support has just gotten better and better over the year, several members of the EVGA team are now running multiple AMD 4P systems. Still high recommended to buy SuperMicro 4P motherboard over other brands. ----------------------------------------------------------------------------------------------- Where to Buy: (~$660) WiredZone.com Provantage.com Ebay.com
post edited by bowlinra - 2013/07/29 22:00:08
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware Decisions and Discussions - Memory
2012/06/15 00:39:35
(permalink)
Hardware Decisions and Discussions - Memory The AMD 6100 series CPUs has an integrated quad-channel DDR3 SDRAM Memory controller. This translates for optimal performance each CPU needs 4 separate memory sticks. You need to check your motherboard manual, but Error-Correcting Code (ECC) memory is common place on server motherboard. ECC memory is more expensive and slower than the traditional non-ECC memory common to desktop computing. In this case both the Tyan and Supermicro can be used with non-ECC memory. You also have to look at of the memory voltage, once again defer to your motherboard manual, you need the 1.5v memory for the Supermicro. You'll also want memory with XMP profile. If you think, you would like to overclock in the future, you're going to need fast memory. If you don't overclock you could save some dollars here. Recommendation are: Detailed Reference G.SKILL Ripjaws X Series 8GB (4 x 2GB) DDR3 1333 (Cas 7 7-7-7-21) Desktop Memory Model F3-10666CL7Q-8GBXH For high end overclock you need DDR3 1600 Cas 7 with XMP profiles and 1.5v Remember to stay with 1.5v memory, watch out for the high-end memory it tends to jump up to 1.65v Special notes unless you are doing other things than folding on the system, 1GB sticks are all you need. Fold runs well in under 6GB for Dual and Quad sockets systems. If you're interested in BOINC Crunching in the future the recommendation is to use 2GB sticks. <Need a Cruncher to send me a PM with a good refer> Where to Buy: ($60) G.SKILL Ripjaws X Series 8GB (4 x 2GB) DDR3 1333 (Cas 7 7-7-7-21) Desktop Memory Model F3-10666CL7Q-8GBXH
post edited by bowlinra - 2012/07/17 22:39:52
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware Decisions and Discussions - Heat sink / CPU Coolers
2012/06/15 00:48:43
(permalink)
Hardware Decisions and Discussions - Heat sink / CPU Coolers Background: Server boards are design more or less for Data Centers, where cool air is rapidly circular thought the case, with the smallest height and foot print as possible. Rack mounted Data Center devices are generally avoided in the home environment for several reasons. Most folks don't have a server rack at home, they can take up alot of space, but the main reason I believe is the noise levels of the fans. These Data Center devices fans are design to pull high volumes of cool air thru the device and are very loud and typically not user adjustable. How Loud? Take your vacuum cleaner in the closet and turn it on and close the door. If you and significant others are OK with the sound volume, ignore this warning. You basically have 3 options, with various tradeoffs of Cost / Time / and Cooling Effectiveness. - Use standard server heat sinks. Typically what is used in Data Center deployed servers and normally consist of a copper block heat sinks (~$50 per CPU).
- Upgrade to server heat sinks with 70 - 80mm fans in a low-profile setup (~$100 per CPU)
- Custom Modify the Motherboard or PC CPU Cooler to fit (large range of cost, but starting ~$20).
- Option A - Modify PC CPU Cooler fan mount. This will require change a ex. 1366 / 1155 mount from a 4 post setup to match the G34 socket of 2 post. Detailed reference here Kendrak's 2P and 4P system guide
- Option B - Custom create an adapter for a 4 post CPU Cooler to the 2 post mounting on the G34 socket. Detail reference Musky's CM Hyper 212+ - Alternative G34 Mounting Method
Both Options A & B requires some work to use standard #6-32 threaded screws for the 2 post mounting on the G34 socket. I've seen basically two options to do this: - Option 1 - Modify the motherboard (and Yes I understand it cost over $650 dollars) with a #6-32 Tap tool to cutting the threads thru the motherboard post inserts, thus enable you to use standard #6-32 x 1.5in bolts to be screwed in from the bottom side of the motherboard. - Option 2 - Take #6-32 x 2in bolt an cut off the heads (creating a threaded rod) an insert the by hand screwing it into the motherboard post inserts. I chose PC CPU Coolers route and selected Refurbished Cooler Master Hyper 212+, that where recommended for overclock and good value ($12 per CPU). I went with Option B and Option 1, where I modified the motherboard tapping the thread thru the post and used #6-32 x 1.5in bolt screwed in from the bottom of the board. I followed Musky's CM Hyper 212+ - Alternative G34 Mounting Method. The only exception, I didn't use the springs on 2 post to the motherboard, but just a #6 lock washer an #6-32 standard nut. Due to the space constraints you can only turn the nuts about a 3rd of a turn with a wench. I didn't use the nylon threaded nuts here for that reason. Special note - These CPU Coolers are tall and you'll need to take that in to consideration if you want to use a case. Some folks have voice concerns over the CPU Cooler's combine weight on the motherboard in a vertical / upright position, I personally would think it fine, but again wouldn't recommend using the springs on the mounting to the motherboard post. Where to Buy: (~$12) Cooler Master Store - Hyper 212 Plus Universal Direct Contact Heat-Pipe CPU Cooler - Refurbished Cooler Master Store Geeks.com - Cooler Master Hyper 212 Plus Aluminum Heat Sink & 4.72" Fan w/Copper Heatpipes & 4-Pin Connector for Intel & AMD CPUs
post edited by bowlinra - 2012/07/17 22:21:21
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware Decisions and Discussions - Thermal Paste
2012/06/15 00:50:18
(permalink)
Hardware Decisions and Discussions - Thermal Paste I use Zalman’s Super Thermal Grease (STG) that is brushed on like a nail polish. I'm done with the tube / paste and using a credit card to smooth it, I never seem to get it just right, it's either too thick or too thin. Just brush it on the heat sink and CPU and it has been very consistent. Special note - AMD G34 / Operton CPU are significantly larger than the standard PC Intel chips, the heat sink doesn't cover the whole ship, so there no need to cover the whole chip. I was able to do the install of all 4 CPU with a single bottle, but your mileage may vary. Where to Buy: (~$10) Zalman’s Super Thermal Grease (STG)
post edited by bowlinra - 2012/07/17 22:40:57
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware Decisions and Discussions - Drives
2012/06/15 00:52:40
(permalink)
Hardware Decisions and Discussions - DrivesIf you are using the recommended [H] folding Appliance, everything is already loading to a ram disk. This allows booting off a USB thumb / flash drive to work just as well as a physical hard drive. Booting from a USB device will take an additional minute to boot, but your not going to see any real difference in PPD at all. It does have an advantage of using less energy as well as the drive and cables block air flow. That said a standard hard drive or SSD will still work fine. Recommend USB that's been tested. Patriot Memory Supersonic Boost XT USB 3.0 8GB Flash Drive (PEF8GSBUSB) using USB 2.0 (Read 37.2 MB/s Write 13.0 MB/s). I'm using SanDisk Cruzer Fit 8GB original because of Plug and Forget, but discovered it's much slower performance (Read 23.1 MB/s Write 5.2 MB/s). Special note - If you doing you own manual installation, you could also try playing with the IGMI management interface (which is a BIOS supported KVM over IP) and virtually attach to a DVD drive across the network for the installation. Detailed reference Penn7's post Supermicro IPMIview Review – Remote Server Monitoring and Management and KVM-over-IP. Where to Buy: (~$15) Patriot Memory Supersonic Boost XT USB 3.0 8GB - Amazon ($13) SanDisk Cruzer Fit 8GB - Amazon ($8)
post edited by bowlinra - 2013/11/07 21:37:48
|
shrekdaklown
SSC Member
- Total Posts : 920
- Reward points : 0
- Joined: 2011/09/04 20:20:38
- Location: columbus, oh
- Status: offline
- Ribbons : 4
Re:Hardware Decisions and Discussions - Memory
2012/06/15 00:53:37
(permalink)
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware Decisions and Discussions - Power Suppy Unit (PSU)
2012/06/15 00:55:44
(permalink)
Hardware Decisions and Discussions - Power Suppy Unit (PSU)I had an OCZ ModXStream Pro 700W on hand, so didn't put much thought into this decision. At stock CPU speeds and no dedicated graphics card I'm pulling 440W - 460W from the wall. Overclocking 10% pushed to 480W of consumption. I've seen post with folks using AMD 6180 CPUs and large overclocks pulling 800W - 850W. If you buying a PSU I'd recommend one that is EPS12V 2.92 version, I believe all 850W and above units come standard with 2 8-pin CPU / EPS connectors. Alot of smaller PSU come with 1 8-pin & 1 4-pin ATX12V / EPS. The 4-pin (inserted in the motherboard 8-pin connector) provides enough power to boot the system and do the installation, but you're not going to be able to fold under that setup. Other options would be to use an adapter converts 2 4-pin molex connectors to feed the 8-pin CPU / EPS connector NOT RECOMMENDED or a 4 Pin Male to 8 Pin Female EPS Power Cable Adapter or a PCI-E (6+2) to 8-pin EPS. I used the adaptor (2x 4-pin Molex to 8-pin CPU / EPS connecter) on my system and it seems fine for stock speeds, but I've had alot of difficult with overclock and I'm looking into this as a possible problem. EDIT We are definitely seeing reports that the CPU / EPS sockets are capable of pulling more power than the molex wires can handle. For more details read Patriot's post on 4p Power advisory. EDIT After upgrade the PSU to a EVGA 1200W (that has 2 CPU / EPS connectors, I've had no problems with the overclocking. My 6166HE default clock is 1.8 GHz now running 2.3 GHz. Detail reference Bowlinra's post AMD 4P Burning and Prep for OC? Power Draw References:AMD 2P 6172 @ 2.1Ghz - 280 Watts AMD 2P 6176SE @ 2.3Ghz - 340 Watts AMD 2P 62xx @ 3.2Ghz - 495 Watts AMD 4P 8425 "Socket F / Ankle biter" - 360 Watts AMD 4P 6166HE @ 1.8Ghz - 465 Watts AMD 4P 6166HE @ 2.3Ghz - 552 Watts AMD 4P 6172 @ 2.4Ghz - 580 Watts AMD 4P 6176SE @ 2.748Ghz - 680 Watts AMD 4P 61xx ES 3.0Ghz ~900 Watts AMD 4P 6272ES @ 2.9Ghz - 785 Watts Power draw will vary based on the component in the system, Project WUs and amount of Overclocking. Amd 4P 6176se @ 2.713Ghz by WU P8101 - 704 Watts - 431,276 PPD P8102 - ? Watts - 709,273 PPD P8103 - 680 Watts - 705,088 PPD P8104 - 675 Watts - 698,086 PPD P8105 - 682 Watts - 686,576 PPD Using Kill-A-Watt , CM 212+, USB Flash drive for Boot, No other drives or case fans. Overclocking Power Draw Changes SuperMicro H8QGi+-f with 4x 6166HE (85W CPUs) stock is 1.8Ghz AMD 4P 6166HE @ 1.8Ghz - 465 Watts AMD 4P 6166HE @ 2.12Ghz - 530Watts <- Change VCore to 1.05 AMD 4P 6166HE @ 2.27Ghz - 542 Watts AMD 4P 6166HE @ 2.25Ghz - 543 Watts AMD 4P 6166HE @ 2.31Ghz - 552 Watts Special note - Consider going with Gold or better rated PSU, this could reduce the electric cost by 10 - 20% for me at a relative low rate of .10 per KWh that's an $40 - $80 annual savings. Where to Buy: Name Brand 850w PSU EPS12V 2.92 version Molex to 8-pin EPS connector (~$7) 4 Pin Male to 8 Pin Female EPS Power Cable Adapter ($.54) PCI-E (6+2) to 8-pin EPS (~$9)
post edited by bowlinra - 2013/11/07 21:58:12
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Hardware Decisions and Discussions - Case / Mounting
2012/06/15 00:59:32
(permalink)
Hardware Decisions and Discussions - Case / Mounting Case or run Naked (w/o a case). That is the question. I still have mixed feelings on this one. I guess my visions of granduir lead me to think, I might have more than one of these setups, needing some form of a wooden homemade rack. (EDIT 7/20/2012: I did build a homemade rack guide on a separate thread here Home Rack Garage Build - Beating the Folding Heat of Summer). I also had concerns with the weight of the CPU Coolers on a vertical mounted motherboard. I'm not so concern with this anymore (see note in the Heat sink / CPU Coolers section). I also thought the heat displacement would be better horizontally placed. If you wanted a case the #1 issue is the size of the motherboard is just huge (16.48" x 13" (41.9cm x 33.0cm)) it NOT going to physically fit in most cases, virtually all case will have to be modified somehow. The Supermicro H8QGi motherboards have 13 screws to properly attach to the case, I would guess 6 of those screw in standoffs will have to be drilled and tapped. Even large case like the popular full case Cooler Master HAF 932 will need move fans to the outside of the case for the side, top and back. Details reference ChelseaOilman's New 4P Build Still Under Construction - Check Back for Updates For addition Pics look in the Pictures section Musky's Mounting PC's on wood instead of cases Bowlinra's Home Rack Garage Build - Beating the Folding Heat of Summer Spotswood's Tech Tray
post edited by bowlinra - 2013/04/28 22:25:22
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
System Pictures (Everyone love hardware)
2012/06/15 01:05:31
(permalink)
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
BIOS Settings
2012/06/15 01:07:18
(permalink)
BIOS SettingsFor OverClocking BIOS on the Supermicro boards, the reference is Tear's post Supermicro H8QGi/6 and H8QGL Next Generation OC BIOS - The [H] OCNG BIOS settings by loading Optimal Defaults via (F9)
These recommendations apply only to non-OC BIOSes (e.g. stock SM or Tyan). Reference to Musky's post Multiple Processor G34 ChecklistBIOS settings to check for 61xx (MC CPUs)
- PowerNow set to Disabled
- NUMA/SRAT set to Enabled
- Node Interleaving set to Disabled
- Bank/channel interleaving set to Auto or Enabled - C1E Set to Disable
- Tyan boards - Ungang DCTs set to Always BIOS settings to check for 62xx/63xx (IL/AD CPUs)
- PowerNow set to Enabled
- NUMA/SRAT set to Enabled
- Node Interleaving set to Disabled - Bank/channel interleaving set to Auto or Enabled
- C1E Support Set to Disable
- CPB/Turbo Set to Enabled
- Tyan boards - Ungang DCTs set to Always
*Note Updated 12/8/13 to Add C1E Disabled and Changes to MC,IL, and AD
post edited by bowlinra - 2013/12/08 23:26:59
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Software - Ubuntu Installation - Folding Configuration and Utilities - Hardware Monitor Tools - Useful Ubuntu Stuff
post edited by bowlinra - 2013/11/04 20:18:48
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Re:AMD 4P / Quad Socket Design and Build Guide
2012/06/15 01:17:24
(permalink)
Software I highly recommend the [H] Folding Appliance. If your not sure about Ubuntu or Linux, this is for you. Here is the Cliff Notes: - Download Image File (788MB compressed, 5.2GB Uncompressed)
- From a Windows Desktop use RMPrepUSB utility to write the image to Any drive (Hard drive, SSD, and USB thumb drive)
- Take the drive to your rig and boot
- Set Timezone, and Setup fah client (via script fahinstall)
- Reboot (sudo reboot) and start folding
Note the fahinstall setup the /fah directory to run on a ramdisk, so once your passed the initial boot and fah start, there is very very little hit on slow disk performance. Which make running on a 8GB USB thumb drive, cheap (under $10) and very little power required. Also the ramdisk does provide another 10+% PPD over traditional method. It took me ~25 mins to write the image to a 8GB USB stick and under 15 mins to shutdown (sudo poweroff), swap drives, reconfigure fahinstall and back up and folding. If your running several rigs with the [H] Appliances let me know, I've got another script to make repeated setup even simpler. Complete details are found on the link below. [H] Ubuntu Folding Appliance -- alternative approach to Linux foldingUpdated FAQ is here
post edited by bowlinra - 2013/11/06 18:45:57
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Software - Folding Configuration and Utilities
2012/06/15 01:18:56
(permalink)
Folding Configuration and Utilities- If you're using the my recommended [H] Folding Appliance it covered in steps #17 - #22 - In the terminal window, type: 'sudo fahinstall' (remove quotes) and press Enter.
This script will set up everything you need to run Folding@Home on a ramdisk, including various OS tweaks, disabling automatic updates, thekraken wrapper, backup/restore scripts, and samba file sharing for remote access. The client will starts to allow you to configure F@H. You only need to set your user name, team, and passkey. The appropriate run parameters (-smp, -bigadv) will be set for you in the config file based on your machine configuration. Here a list of FAH commands I've found helpful: ------------------------------------------------------------- How to shutdown folding client pkill fah6 To check the FAH Checkpoint files ls -l ~/fah/work/*.ckp To Backup fah client & Checkpoints from ram disk (Already run with poweroff & shutdown cmds) fahbackup Look at current FAH Log cat ~/fah/FAHlog.txt | pastebinit List of Langouste Logs ls -lrt /dev/shm/langouste-horde Use Langouste to block additional downloads of new WUs service langouste3 enable-oneunit To allow downloads of new WUs, after being blocked service langouste3 disable-oneunit Gives Langouste3 Operational Status service langouste3 status To update FAH Appliance Tools (while folding is fine) sudo fahinstall What to Post, if having problems fahdiag | pastebinit
cat ~/fah/FAHlog.txt | pastebinit To Delete a WU from 6.34 clients cd ~/fah
rm machinedependent.dat
rm queue.dat
rm -r work
post edited by bowlinra - 2013/12/08 23:36:30
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Software - Hardware Monitor Tools
2012/06/15 01:20:03
(permalink)
Message Place Holder / Still Under Construction
Software
- Hardware Monitor Tools
|
shrekdaklown
SSC Member
- Total Posts : 920
- Reward points : 0
- Joined: 2011/09/04 20:20:38
- Location: columbus, oh
- Status: offline
- Ribbons : 4
Re:Hardware Decisions and Discussions - Case / Mounting
2012/06/15 01:21:54
(permalink)
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Software - Useful Ubuntu Stuff
2012/06/15 01:22:02
(permalink)
Software - Useful Ubuntu Commands Here a list of command I've found helpful: Hardware Commands ------------------------------------------------------------- To shutdown sudo poweroff To Reboot sudo reboot Processor Speed: sudo clockspeed Memory Timings sudo tpc -dram Show Temperatures sudo tpc -temp Constant Updated Temperatures sudo tpc -mtemp Show voltage sudo voltcheck Runs HT Retries every 60 seconds sudo ht-retries Repeat Command every every 60secs watch -n 60 sudo tpc -temp Filters Logs for Errors sudo zgrep -Ei 'mcelog|machine.check|hardware.error|detected.stall|lockup|oops:' $(ls -rt /var/log/{messages,syslog}* 2> /dev/null) |pastebinitHardware failures: dmesg | tail -n 20 Modify Startup script sudo nano /etc/rc.local Use pastebinit: command | pastebinit or pastebinit (what ever you type with Ctrl-D to complete) -------------------------------------------------------------
post edited by bowlinra - 2013/11/04 20:35:07
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
|
texinga
CLASSIFIED ULTRA Member
- Total Posts : 5066
- Reward points : 0
- Joined: 2010/07/03 14:30:32
- Status: offline
- Ribbons : 22
|
Barbarossa
CLASSIFIED Member
- Total Posts : 3994
- Reward points : 0
- Joined: 2008/01/28 11:43:24
- Status: offline
- Ribbons : 25
Re:References
2012/06/15 05:36:23
(permalink)
excellent write up Bowlinra  here are some pics of mine watercooled in a Lian Li PCA77 (EOL and has been replaced by the PCA 77F which doesn´t allow for the PSU to go in the top)  caseless with Noctua NH U9DO A3  and a dual setup with the Noctua NH U12DO A3  on the Noctuas I used their NH T1, don´t remember what i used on the waterblocks
post edited by Barbarossa - 2012/06/15 08:48:27
|
NerdGZ
SSC Member
- Total Posts : 556
- Reward points : 0
- Joined: 2010/12/15 03:28:12
- Status: offline
- Ribbons : 5
Re:References
2012/06/15 06:22:24
(permalink)
you can add the mod i use on the 2P :   and the solution implemented on the Urlauber 2P
post edited by NerdGZ - 2012/06/15 06:29:31
Attached Image(s)
  AFFILIATE CODE:7T7V3Q1VW6 
|
JaskarnSidhu
SSC Member
- Total Posts : 837
- Reward points : 0
- Joined: 2011/06/02 08:14:14
- Location: Edmonton, Alberta, Canada
- Status: offline
- Ribbons : 2
Re:References
2012/06/15 06:34:04
(permalink)
I thought they made server racks with multiple server trays in them, so you could run anywhere from 2 to 16 servers?
Do they make a smaller version of a server rack that can maybe hold 4 or so server trays?
|
zophar
CLASSIFIED Member
- Total Posts : 4749
- Reward points : 0
- Joined: 2008/08/26 09:48:56
- Location: KC,MO
- Status: offline
- Ribbons : 26
Re:References
2012/06/15 06:57:44
(permalink)
In my area space is limited. So If I were to do a 4p build it would have to go in a case and be verticle. So the water cooler option would probably be the best for me. But then I'd need a new case to fit such a beast. We need to compile a list of any cases that will fit these monster boards with little to no modification if possible. Very nice job so far bowlinra. @NerdGZ, what kind of themps do the stock Phenom coolers give you? Also @ Barbarossa, what water blocks are you using on your board? And one final side note, Operon 6128 8 core CPU's are currently going for $165 or so on fleabay. For those of us that are wanting to do this on a faily tight budget. And yes I realize that budget and 4p machine don't go together so well.
post edited by zophar - 2012/06/15 07:50:45
|
NerdGZ
SSC Member
- Total Posts : 556
- Reward points : 0
- Joined: 2010/12/15 03:28:12
- Status: offline
- Ribbons : 5
Re:References
2012/06/15 08:00:57
(permalink)
@zophar 40ºCelsius with a 20mm 800rpm splitted over the heatsinks memory and with a 80mm fan on the bare metal behind hdd connectors.
This solution is noisy aprox 3600 rpm fans from the 2 Phenom II X4/x& are spinning.
Note to add to the post these Opteron CPU dies at 70ºC and over 54 -55ºC the CPU autoprotects them and does a shutdown/reboot.
I have seen one at 50ºC but after removing the cooler fans and do a clean i was at this temperature.
AFFILIATE CODE:7T7V3Q1VW6
|
Horvat
CLASSIFIED Member
- Total Posts : 3067
- Reward points : 0
- Joined: 2009/06/16 12:35:24
- Location: Cincinnati, OH
- Status: offline
- Ribbons : 9
Re:References
2012/06/15 08:54:28
(permalink)
This deffinately needs to moved to the Help and Guides section. Great work.
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Re:References
2012/06/16 00:22:22
(permalink)
NerdGZ
you can add the mod i use on the 2P : and the solution implemented on the Urlauber 2P
NerdGZ, I'll be glad to add it, Could you add some word explaining what you did, why and how for example.. It's not clear how and what you did from the pics, maybe because I'm not familiar with the stock version of the Urlauber 2P.
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Re:References
2012/06/16 00:33:26
(permalink)
zophar
We need to compile a list of any cases that will fit these monster boards with little to no modification if possible.
I have might have some more information, but most of it posted, just not organized well for cases information. I've mounted my to a board and will be adding the write up after some vacation. zophar
Also @ Barbarossa, what water blocks are you using on your board?
I haven't looked into water setups at all, maybe Barbarossa  or someone else can work up a "How To" and some cost numbers.
|
bowlinra
SSC Member
- Total Posts : 883
- Reward points : 0
- Joined: 2011/12/05 20:58:14
- Location: Virginia, USA
- Status: offline
- Ribbons : 5
Re:References
2012/06/16 00:38:24
(permalink)
NerdGZ
Note to add to the post these Opteron CPU dies at 70ºC and over 54 -55ºC the CPU autoprotects them and does a shutdown/reboot.
Could you post a reference link? I certainly haven't heard this with the Opteron 6100 series, I understand they start down clocking at 70ºC. Also what kinda of PPD are you getting out of the 2P setup?
|