Purpose:

• Raise team awareness of the power of Big CPU folding
  • These systems are capable of posting 200K - 800K PPD.
  • Provide some organization to the wealth of good information, guides, and my lessons learned.
• My personal goals were to push myself into the Top 20 Producers for Folding@EVGA, maybe motivate my teammates to step up as well, force myself to learn about -BigAdv, Server hardware, and learn to spell Ubuntu.
• I'm certainly not an expert in this area. What works for me, may not work for you or your hardware. This represents about a month of nightly information gathering on the various source. I hope this save you alot of research time.
• Please send me a private message for any correction, updates, rewording, rewrites, and any additional information you think we should include. (It going to need it, because it was written way too late at night)

Table of Contents

Hardware and Cost
Hardware Decisions and Discussions

- CPUs
- Motherboards
- Memory
- Heat sink / CPU Coolers
- Thermal Paste
- Drives
- Power Supply Unit (PSU)
- Case / Mounting

System Pictures (Everyone love hardware)
BIOS Settings
Software

- Ubuntu Installation
- Folding Configuration and Utilities
- Hardware Monitor Tools
- Useful Ubuntu Commands

References
 

Change Log:

6/15/12 - Initial Post
7/14/12 - Added BIOS, PSU update and minor edits
 8/8/12  - Added Power Advisory on CPU/EPS adapters
 9/1/12  - Added Referrence to Improving Opteron 6200 performance
4/25/13 - Updated Vendors CPU Pricing
7/30/13 - Recommended Bargain / Best Value AMD 4P Build Cost 
             - Added 62xx series chip data and Reasonable OC numbers for 61xx
11/4/13 - Updated Software Sections
              - Add [H] Folding Appliance and Ubuntu Commands
11/7/13 - Updated Power Draw and PPD Numbers in PSU Section
              - Update Drive Recommendations
12/8/13 - Update BIOS Settings Recommendations

Hardware and Cost 

(Updated July 2013)
Recommended Bargain / Best Value AMD 4P Build Cost (July 2013 Pricing Update)

$550 - 4x AMD Opteron 61xx CPUs (Ebay ~$125 - 175 per cpu)    
$658 - Supermicro H8QGi+-F AMD Quad G34 Socket Motherboard (New)  
$60 - 16x Micron or Hynix ECC 1GB DDR3 1333 or 1600 (Cas 9 v1.5) (Used MAC DIMMs on Ebay $3 -$4)
$100 - 850W GOLD PSU (Prefer Corsair or Seasonic w/ 3yr - 5yr Warranty)
$48 - 4x Cooler Master Hyper 212 Plus (Refurb $12 per)  
$8 - SanDisk Cruzer Fit 8GB Flash Drive for Systems (New)
$10 -  Zalman’s Super Thermal Grease (STG)  
$20 - Misc Hardware for board & CPU Cooler mounting 
------------ 
$ 1,454 - Approx Bargain AMD G34 4P

All the prices are reasonable, thou you'll have to hunt to get some on them (Send me a PM, if you need help). You should watch these two thread for assistance: EVGA's AMD 4P Deals and [H] Hot deals for folders .
 
Bargain Rig Stock or OCed is capable of producing on P8101 WUs:

- @1.8Ghz ~243K PPD*  
- @2.2Ghz ~314K PPD*
- @2.4Ghz ~371K PPD*
- @2.7Ghz ~411K PPD*

*Add an Additional 200K+ PPD for the new 8102 - 8105 WUs.   Check Power Supply Unit (PSU)  section for Watts & PPD.
---------------------------------------------------------------------------------------------
(For Legacy Reference)
AMD 4P Build Cost (May 2012 Pricing)

$1,360 - 4x AMD Opteron 6166HE 1.8Ghz CPUs (Used, w/ 1yr warranty $340 per)   
$663 - Supermicro H8QGi+-F AMD Quad G34 Socket Motherboard (New) 
$240 - 4x G.SKILL Ripjaws X Series 8GB (4 x 2GB) DDR3 1333 (Cas 7 7-7-7-21) Desktop Memory Model F3-10666CL7Q-8GBXH (New $60 per) 
$120 - EVGA Classified SR-2 100-PS-1200-GR 1200W (New $120 on Close Out Special)
$48 - 4x Cooler Master Hyper 212 Plus (Refurb $12 per) 
$15 - Seagate 40GB Sata (From the local used PC store)
$0 - LG DVD Drive (From the local used PC store, removed after install)
$10 -  Zalman’s Super Thermal Grease (STG) 
$20 - Misc Hardware for board & CPU Cooler mounting
------------
$ 2,476 - My out of pocket cost

At stock speed this rig is producing (~243K PPD w/ P8101, ~276K PPD w/ P6901, ~380K PPD w/ P6903/4) with an electrical draw of 460W.
 
AMD 2P Build Cost (June 2012 Pricing)

$700 - 2x AMD Opteron 6176SE 2.3Ghz CPUs (Used Ebay $350 per)    
$250 - Supermicro H8DGU-F AMD Dual G34 Socket Motherboard (Used Ebay)  
$120 - 4x Crucial Ballistix 4GB (2 x 2GB) 240-Pin DDR3 SDRAM DDR3 1600 (Cas 8 8-8-8-24) Desktop Memory Model BLT2KIT2G3D1608DT1TX0 (New $30 per)

$0 - OCZ ModXStream Pro 700W Modular High Performance PSU (had sitting Idle)  
$26 - 2x Cooler Master Hyper 212 Plus (Refurb $12 per)  
$15 - Seagate 40GB Sata (From the local used PC store) 
$10 -  Zalman’s Super Thermal Grease (STG)  
$15 - Misc Hardware for board & CPU Cooler mounting 
------------ 
$ 1,136 - My out of pocket cost 

At stock speed this rig is producing (~125K PPD w/ P8101) with an electrical draw of 340W.
 
Power and PPD Comparison to others:
 - Stock i7 920 w/ GTX 580@920, GTX 580@900, GTX 260@735 produced ~75K PPD with an electrical draw of 830W
 - Stock i7 920 w/ GTX 580@920, GTX 260@735 produced ~40K PPD with  an electrical draw of 560W
 - Stock i7 920 w/ 3x GTS 450@950 - 900 produced ~40K PPD with  an electrical draw of 560W
 - Stock Intel Quad Q6600 w/ GTX 460@900 produced ~20K PPD with an electrical draw of 313W
 - Stock Intel Dual E4400 w/ GTS 8800 produced ~10K PPD with an electrical draw of 235W
 - Mitsubishi 65in DLP TV has an electrical draw of 208W

 

Hardware Decisions and Discussions - CPUs

Intel vs AMD. I was certainly bias to Intel and believed you only folder on just Intel and NVIDIA GPUs. I definitely thought I was looking at a Dual Xeon setup with an SR-2 motherboard. Xeon CPUs are changing over from the older LGA-1366 socket to the new LGA-2011 socket, the older can't measure up and I didn't want to put money into SMP 16 core system at would struggle to meet the new -BigAdv deadlines. The LGA-2011 CPU are very expensive ie $2,000 per processor and no overclocking motherboard at this point (June 2012).
 
I looked at AMD 6200 Series is the latest and greatest with up to 16 cores (June 2012) & 6100 Series is the last generation model with 8 and 12 cores (You'll also seen them as Magny-Cours or MC)
The several debates about the two series, I believe the general thoughts are 6200 IL are newer, faster, have more cores and more expensive BUT don't fold as well as the older 6100 series core for core. The 6100s are starting to be life cycled out of the big high end servers, thus makes for a larger supply on the used markets like Ebay. More details AMD 6100 vs 6200 Debate reference.
*EDIT 9/1/12* QuickZ's thread on Improve Opteron 6200's performance (and PPD) by using TurionPowerControl is new information since I looked at this decision.
*EDIT 7/30/13* Since the original post, I now own a set of 6272 IL CPUs. I can say, even thou they have 64 cores (instead of 48 cores set of 6172 MC), they produce less PPD at the same clockspeed and use an additional 100W to run. Maybe this will change with future WUs, but all the 8101 - 8105 series do not. I would not recommend for folding over the MC.
 
Here is how I thought about it. I could get an AMD Opteron 6166HE with 12 cores  for $340. I couldn't even get a Intel (because I think Intel in the PC world) Hexa / Heca / 6 cores CPU for that price. Intel i7 (970, 980, 990) all costing over $400 on eBay market. I'm sure we could debate of HyperThreading compared to AMD native cores, but the deciding fact was you can't run dual or multi socket motherboards with the Intel i7 (970, 980, 990) only Xeons.
 
In the AMD 6100 series (12 core) looks like

6180SE @ 2.5Ghz -140w
6176SE @ 2.3Ghz -140w
6176 @ 2.3Ghz -115w
6174 @ 2.2Ghz -115w
6172 @ 2.1Ghz -115w
6168 @ 1.9Ghz -115w
6166HE @ 1.8Ghz -85w
6164HE @ 1.7Ghz -85w

In the AMD 6100 series (8 core) looks like

6140 @ 2.6Ghz -115w
6138 @ 2.5Ghz -115w
6136 @ 2.4Ghz -115w
6134 @ 2.2Ghz -115w
6132HE @ 2.1Ghz -85w
6128HE @ 2.0Ghz -85w
6124HE @ 1.8Ghz -85w

SE - Top of the Line
HE - Energy Efficient
ES - Engineering Samples (These are test and evaluation units, not sold to the public, but yet on eBay hmm)
 
----------------------------------------------------------------------------------------------- 
(Update July 2013)
Alot more research and data collection has occurred since the original post, Here are some reasonable Overclocking estimated:

6164 HE/6166 HE: ~2.25Ghz (CPU clock multiplier 250) *Recommended DDR3 1600 Usage*
6168/6172/6174: ~2.41Ghz (CPU clock multiplier 230)
6176 SE/6180 SE: ~2.64Ghz (CPU clock multiplier 230)

-----------------------------------------------------------------------------------------------
 
I selected the AMD Opteron 6166HE with 12 cores for $340 (May 2012)
 
Here is a great reference to see what -BigAdv numbers are being posted by these and other big CPU / server hardware rigs. https://docs.google.com/s...XWVZ2S2xvejBDcHc#gid=0
 
Where to Buy: (~$340)
Ebay.com 
IT Creations, Sales Manager (Jay Tessel ( IT Creations, Inc ) (jay@itcreations.com) and telling him your with "Stanford's Folding Project"
IT Creations offers a 1yr warranty and Free Shipping with these price. Please remember prices do fluctuate depending on the market and there inventory, but these should be good for the near future. (Updated April 2013)

Hardware Decisions and Discussions - Motherboards

In the 4 Processors (4P) / Quad Socket / MultiProcessor (MP) Motherboard, there not a lot of choices the two main ones are TYAN S8812 (S8812WGM3NR) and SuperMicro H8QGi / H8QGL / H8QG6 series.
 
I selected the Supermicro because the folding team [H]orde / [H]ardOCP created a BIOS upgrade that could allow you 
to overclock the CPUs, I saw posted claims 30% overclocks or better. I picked SM H8QGi+-F because of the price and CPU socket placement, I felt it would be easy to cool than the cheaper SM H8QGL. I did not consider the H8QG6 series due to the added cost of SAS controller that I didn't need. Additional information on the different board series.
 
SM H8QGi+-F Benefits
 
- Socket placement was good for cooling.
- Embedded graphics card (no need to buy one, also add a video card could hamper cooler by blocking the CPU cooler).
- Motherboard works with just one cpu and would allow you to continue to upgrade by adding addition cpus over time. This was a big advantage over buying a 2P (2 Processor / Dual Socket) motherboard at 65% of a the cost of a Quad socket board.
- Although I don't recommend this, but you can theoretically use different / mismatched CPUs on the board. It will just drop the overall CPU speed down to the lowest common denominator.
- Biggest benefit is the ability to get great support for the folks at the [H]ardForum.
 
-----------------------------------------------------------------------------------------------  
(Update July 2013) 
The folding support has just gotten better and better over the year, several members of the EVGA team are now running multiple AMD 4P systems. Still high recommended to buy SuperMicro 4P motherboard over other brands.
-----------------------------------------------------------------------------------------------  
Where to Buy: (~$660)
WiredZone.com
Provantage.com 
Ebay.com 
 

Hardware Decisions and Discussions - Memory 

The AMD 6100 series CPUs has an integrated quad-channel DDR3 SDRAM Memory controller. This translates for optimal performance each CPU needs 4 separate memory sticks. You need to check your motherboard manual, but Error-Correcting Code (ECC) memory is common place on server motherboard. ECC memory is more expensive and slower than the traditional non-ECC memory common to desktop computing. In this case both the Tyan and Supermicro can be used with non-ECC memory. You also have to look at of the memory voltage, once again defer to your motherboard manual, you need the 1.5v memory for the Supermicro. You'll also want memory with XMP profile.

If you think, you would like to overclock in the future, you're going to need fast memory. If you don't overclock you could save some dollars here.

Recommendation are: Detailed Reference
G.SKILL Ripjaws X Series 8GB (4 x 2GB) DDR3 1333 (Cas 7 7-7-7-21) Desktop Memory Model F3-10666CL7Q-8GBXH 

For high end overclock you need DDR3 1600 Cas 7 with XMP profiles and 1.5v Remember to stay with 1.5v memory, watch out for the high-end memory it tends to jump up to 1.65v
 
Special notes unless you are doing other things than folding on the system, 1GB sticks are all you need. Fold runs well in under 6GB for Dual and Quad sockets systems. If you're interested in BOINC Crunching in the future the recommendation is to use 2GB sticks. <Need a Cruncher to send me a PM with a good refer>

Where to Buy: ($60)
G.SKILL Ripjaws X Series 8GB (4 x 2GB) DDR3 1333 (Cas 7 7-7-7-21) Desktop Memory Model F3-10666CL7Q-8GBXH  

Hardware Decisions and Discussions - Heat sink / CPU Coolers

Background: Server boards are design more or less for Data Centers, where cool air is rapidly circular thought the case, with the smallest height and foot print as possible. Rack mounted Data Center devices are generally avoided in the home environment for several reasons. Most folks don't have a server rack at home, they can take up alot of space, but the main reason I believe is the noise levels of the fans. These Data Center devices fans are design to pull high volumes of cool air thru the device and are very loud and typically not user adjustable. How Loud? Take your vacuum cleaner in the closet and turn it on and close the door. If you and significant others are OK with the sound volume, ignore this warning.

You basically have 3 options, with various tradeoffs of Cost / Time / and Cooling Effectiveness.

1. Use standard server heat sinks. Typically what is used in Data Center deployed servers and normally consist of a copper block heat sinks (~$50 per CPU).
2. Upgrade to server heat sinks with 70 - 80mm fans in a low-profile setup (~$100 per CPU)
3. Custom Modify the Motherboard or PC CPU Cooler to fit (large range of cost, but starting ~$20).

 

- Option A - Modify PC CPU Cooler fan mount. This will require change a ex. 1366 / 1155 mount from a 4 post setup to match the G34 socket of 2 post. Detailed reference here Kendrak's 2P and 4P system guide
- Option B - Custom create an adapter for a 4 post CPU Cooler to the 2 post mounting on the G34 socket. Detail reference Musky's CM Hyper 212+ - Alternative G34 Mounting Method

Both Options A & B requires some work to use standard #6-32 threaded screws for the 2 post mounting on the G34 socket. I've seen basically two options to do this:

- Option 1 - Modify the motherboard (and Yes I understand it cost over $650 dollars) with a #6-32 Tap tool to cutting the threads thru the motherboard post inserts, thus enable you to use standard #6-32 x 1.5in bolts to be screwed in from the bottom side of the motherboard.

- Option 2 - Take #6-32 x 2in bolt an cut off the heads (creating a threaded rod) an insert the by hand screwing it into the motherboard post inserts.

I chose PC CPU Coolers route and selected Refurbished Cooler Master Hyper 212+, that where recommended for overclock and good value ($12 per CPU). I went with Option B and Option 1, where I modified the motherboard tapping the thread thru the post and used #6-32 x 1.5in bolt screwed in from the bottom of the board. I followed Musky's CM Hyper 212+ - Alternative G34 Mounting Method. The only exception, I didn't use the springs on 2 post to the motherboard, but just a #6 lock washer an #6-32 standard nut. Due to the space constraints you can only turn the nuts about a 3rd of a turn with a wench. I didn't use the nylon threaded nuts here for that reason.

Special note - These CPU Coolers are tall and you'll need to take that in to consideration if you want to use a case. Some folks have voice concerns over the CPU Cooler's combine weight on the motherboard in a vertical / upright position, I personally would think it fine, but again wouldn't recommend using the springs on the mounting to the motherboard post.

Where to Buy: (~$12)
Cooler Master Store - Hyper 212 Plus Universal Direct Contact Heat-Pipe CPU Cooler - Refurbished  Cooler Master Store 
Geeks.com - Cooler Master Hyper 212 Plus Aluminum Heat Sink & 4.72" Fan w/Copper Heatpipes & 4-Pin Connector for Intel & AMD CPUs
 

Hardware Decisions and Discussions - Thermal Paste

I use Zalman’s Super Thermal Grease (STG) that is brushed on like a nail polish. I'm done with the tube / paste and using a credit card to smooth it, I never seem to get it just right, it's either too thick or too thin. Just brush it on the heat sink and CPU and it has been very consistent.

Special note - AMD G34 / Operton CPU are significantly larger than the standard PC Intel chips, the heat sink doesn't cover the whole ship, so there no need to cover the whole chip. I was able to do the install of all 4 CPU with a single bottle, but your mileage may vary.

Where to Buy: (~$10)
Zalman’s Super Thermal Grease (STG) 

Hardware Decisions and Discussions - Drives

If you are using the recommended [H] folding Appliance, everything is already loading to a ram disk. This allows booting off a USB thumb / flash drive to work just as well as a physical hard drive. Booting from a USB device will take an additional minute to boot, but your not going to see any real difference in PPD at all. It does have an advantage of using less energy as well as the drive and cables block air flow. That said a standard hard drive or SSD will still work fine.
 
Recommend USB that's been tested.
Patriot Memory Supersonic Boost XT USB 3.0 8GB Flash Drive (PEF8GSBUSB) using USB 2.0 (Read 37.2 MB/s Write 13.0 MB/s).
 
I'm using SanDisk Cruzer Fit 8GB original because of Plug and Forget, but discovered it's much slower performance (Read 23.1 MB/s Write 5.2 MB/s).


Special note - If you doing you own manual installation, you could also try playing with the IGMI management interface (which is a BIOS supported KVM over IP) and virtually attach to a DVD drive across the network for the installation. Detailed reference Penn7's post Supermicro IPMIview Review – Remote Server Monitoring and Management and KVM-over-IP. 

Where to Buy: (~$15)
Patriot Memory Supersonic Boost XT USB 3.0 8GB - Amazon ($13)
SanDisk Cruzer Fit 8GB - Amazon ($8)
 

great write up

Hardware Decisions and Discussions - Power Suppy Unit (PSU)

I had an OCZ ModXStream Pro 700W on hand, so didn't put much thought into this decision. At stock CPU speeds and no dedicated graphics card I'm pulling 440W - 460W from the wall. Overclocking 10% pushed to 480W of consumption. I've seen post with folks using AMD 6180 CPUs and large overclocks pulling 800W - 850W. If you buying a PSU I'd recommend one that is EPS12V 2.92 version, I believe all 850W and above units come standard with 2 8-pin CPU / EPS connectors. Alot of smaller PSU come with 1 8-pin & 1 4-pin ATX12V / EPS. The 4-pin (inserted in the motherboard 8-pin connector) provides enough power to boot the system and do the installation, but you're not going to be able to fold under that setup. Other options would be to use an adapter converts 2 4-pin molex connectors to feed the 8-pin CPU / EPS connector NOT RECOMMENDED or a 4 Pin Male to 8 Pin Female EPS Power Cable Adapter or a PCI-E (6+2) to 8-pin EPS. I used the adaptor (2x 4-pin Molex to 8-pin CPU / EPS connecter) on my system and it seems fine for stock speeds, but I've had alot of difficult with overclock and I'm looking into this as a possible problem. EDIT We are definitely seeing reports that the CPU / EPS sockets are capable of pulling more power than the molex wires can handle. For more details read Patriot's post on 4p Power advisory.
 
EDIT After upgrade the PSU to a EVGA 1200W (that has 2 CPU / EPS connectors, I've had no problems with the overclocking. My 6166HE default clock is 1.8 GHz now running 2.3 GHz. Detail reference Bowlinra's post AMD 4P Burning and Prep for OC? 
 
Power Draw References:
AMD 2P 6172 @ 2.1Ghz - 280 Watts
AMD 2P 6176SE @ 2.3Ghz - 340 Watts
AMD 2P 62xx @ 3.2Ghz - 495 Watts
 
AMD 4P 8425 "Socket F / Ankle biter"  - 360 Watts
 
AMD 4P 6166HE @ 1.8Ghz - 465 Watts
AMD 4P 6166HE @ 2.3Ghz - 552 Watts
AMD 4P 6172 @ 2.4Ghz - 580 Watts
AMD 4P 6176SE @ 2.748Ghz - 680 Watts
AMD 4P 61xx ES 3.0Ghz ~900 Watts
 
AMD 4P 6272ES @ 2.9Ghz - 785 Watts
 
 
Power draw will vary based on the component in the system, Project WUs and amount of Overclocking.
Amd 4P 6176se @ 2.713Ghz by WU
P8101 - 704 Watts - 431,276 PPD
P8102 -   ?   Watts - 709,273 PPD
P8103 - 680 Watts - 705,088 PPD
P8104 - 675 Watts - 698,086 PPD
P8105 - 682 Watts - 686,576 PPD
Using Kill-A-Watt , CM 212+, USB Flash drive for Boot, No other drives or case fans.


Overclocking Power Draw Changes
SuperMicro H8QGi+-f with 4x 6166HE (85W CPUs) stock is 1.8Ghz
AMD 4P 6166HE @ 1.8Ghz - 465 Watts
AMD 4P 6166HE @ 2.12Ghz - 530Watts <- Change VCore to 1.05 
AMD 4P 6166HE @ 2.27Ghz - 542 Watts
AMD 4P 6166HE @ 2.25Ghz - 543 Watts
AMD 4P 6166HE @ 2.31Ghz - 552 Watts
 
Special note - Consider going with Gold or better rated PSU, this could reduce the electric cost by 10 - 20% for me at a relative low rate of .10 per KWh that's an $40 - $80 annual savings.

Where to Buy: 
Name Brand 850w PSU EPS12V 2.92 version
Molex to 8-pin EPS connector (~$7)
4 Pin Male to 8 Pin Female EPS Power Cable Adapter ($.54)
PCI-E (6+2) to 8-pin EPS (~$9)