PPD always goes up and down, fluctuating at random, correcting for previous results. You can only see PPD changes from memory changes over the course of days.
Though, when you overclock the memory, GPU frequency goes down. So you'd have to compensate for that with a GPU overclock, which works really well for lower end RTX GPUs.
The Rog strix have great cooling capabilities, and can easily be set to a minimum consumption of 125W, and it would still hit peak GPU frequency.
Not sure about how the KO would perform.
 
Most of the time, a 2070 or lower, can boost to max core clock speed at 125-127W.
 
Older models are more around 1935Mhz, newer (higher bin) models are 2010, some even 2050Mhz (or higher).
Though the MHz isn't really an indication of one GPU being faster than another.
I've had GPUs (Asus) with higher frequency, still running lower PPD than identical GPUs from another manufacturer (Gigabyte) running the same wus with lower GPU frequencies.
 
For a 2080, the boost speeds are more like 1875Mhz, and a 2080Ti are even lower, while they are capable of temporarily (like a few seconds) hitting 2050Mhz, but this oc isn't attainable for sustained loads.

I’ve read all this and have come to the conclusion I need to buy a new video card.  I am running an amd 200ge for low powerconsumption.  The computer is not old just not used.  It has a 1050 in it.  I don’t have another full pci slot.  It has a slot that I use for a WiFi card.  Thinking I may upgrade my Rex 570 in ,y main computer.  I am thinking of putting the Rex 570 in this older build and just let it run 24/7.   I may pick up a 1660 but not sure.  This guide has let me know my setup is weak but it would be helpful maybe.  I run a k2200 in my main system overnight to fold because it doesn’t overheat.  Is it not worth running a 1050 is that what I’m reading.  

@Prodigit, your posts have been amazing and enlightening. Thank you!
Due to the recent events, and due to my hobby folding and past mining, I have decided to take my Asub B250 19 GPU motherboard, along with the 19-GPU rack and buy some GPU's for a charity folding rig.
Your comments and research you've done have helped me pick what I'm looking for in the best cost efficiency.
I've had 3 home computers totaling 2X 1080Ti FTW3, 1X 1060 folding for the last several months, and have almost always had one folding overnight.


I don't have any other GPU's outside of my home computers, but I do have the rest of the guts for the charity folding rig. I have the Asus B250 Mobo, Intel BX80677G3930 CPU(Should I upgrade?), RAM, PSU's totaling 4,100 watts, and 19-GPU server rack.

I think I will be going with as many 2060's as I can get my hands on, but in the end, my budget will depend on what funds I can raise from the community. My post is not intended to canvas for money. I actually came here looking for best practices for multi-GPU folding at a large scale, though if anyone is interested in donating to my cause, I would be greatly thankful for it (gf.me/u/xxcacp). I can afford a few hundred dollars of electricity increase, so my concern is not there. My house is pretty new and efficient so my electricity costs are currently pretty low. And I am planning that if I can get this folding rig up and running, to get the local power company to be a sponsor to subsidize the electricity costs each month.
This would free up that money to save up and add more GPU's over time, unless my funding becomes wildly successful.

If you have any recommendations on how you would set this up if you had it, and any tips on finding the right communities/partners to help fund this charity project, I would greatly appreciate it.
I'm not looking to make any money, and am happy to take a hit. Like I mentioned, folding/mining/computers are a hobby of mine, so I don't mind losing money to it, especially when it's toward good causes like  Folding@home BOINC, etc.

Thanks,

Colby Capps

colbzc
@Prodigit, your posts have been amazing and enlightening. Thank you!
Due to the recent events, and due to my hobby folding and past mining, I have decided to take my Asub B250 19 GPU motherboard, along with the 19-GPU rack and buy some GPU's for a charity folding rig.
Your comments and research you've done have helped me pick what I'm looking for in the best cost efficiency.
I've had 3 home computers totaling 2X 1080Ti FTW3, 1X 1060 folding for the last several months, and have almost always had one folding overnight.


I don't have any other GPU's outside of my home computers, but I do have the rest of the guts for the charity folding rig. I have the Asus B250 Mobo, Intel BX80677G3930 CPU(Should I upgrade?), RAM, PSU's totaling 4,100 watts, and 19-GPU server rack.

I think I will be going with as many 2060's as I can get my hands on, but in the end, my budget will depend on what funds I can raise from the community. My post is not intended to canvas for money. I actually came here looking for best practices for multi-GPU folding at a large scale, though if anyone is interested in donating to my cause, I would be greatly thankful for it (gf.me/u/xxcacp). I can afford a few hundred dollars of electricity increase, so my concern is not there. My house is pretty new and efficient so my electricity costs are currently pretty low. And I am planning that if I can get this folding rig up and running, to get the local power company to be a sponsor to subsidize the electricity costs each month.
This would free up that money to save up and add more GPU's over time, unless my funding becomes wildly successful.

If you have any recommendations on how you would set this up if you had it, and any tips on finding the right communities/partners to help fund this charity project, I would greatly appreciate it.
I'm not looking to make any money, and am happy to take a hit. Like I mentioned, folding/mining/computers are a hobby of mine, so I don't mind losing money to it, especially when it's toward good causes like  Folding@home BOINC, etc.

Thanks,

Colby Capps

I'm not sure if the current batch of COVID 19 can run from GPU. My last response was, that those are CPU only.
Just to not have you build a machine that might not do what you want it to do.
 
FAH runs a lot better with fewer high power GPUs, than many low power ones. So my recommendation was to get rid of everything and run 1 mobo per PSU on RTX GPUs.
Reason being is 4kW, is going to cost you nearly $333 a month on electricity. That's a single RTX 2060 KO per month you could be saving for!

- You can either rebuild from the ground up, which means 3 motherboards, each feeding 2 to 3 high end RTX GPUs (which, if you're going to spend that on electricity anyway, you might as well spend it on hardware first);
Or, 
- In case electricity isn't paid by you, or is free, and you want to lower initial purchase cost, you can work with the current hardware you have, in which case the first thing I would do is swap out the CPU. That CPU is barely enough to run 4GPUs. Folding requires a lot more CPU power than mining.
 
My recommendation?
Build your current rig, and make points with it, until the rig is complete. From there on, build a second, more efficient and better one.
 
Suggestions:
1- Learn to fold on Linux (make sure you know enough to know how to run FAH on Linux), as Linux is much lower in PCIE bandwidth, which this motherboard really needs.
2- Fold on 4 GPUs in the mean time (on Linux), before your replacement CPU arrives, which is most certainly needed!
3- I would recommend to go with a Core i7, because it supports 4 cores with Hyper-threading (you'll need the extra threads!). 7th gen CPUs have higher boost clock rates than 6th gen, but they are increasingly rare, and sought after, and costly (still about $200-250 a piece. But the 6th gens you can get for around $80-$100 on ebay, second hand). They should be good enough to feed up to 14 GTX GPUs, or 8 to 12 RTX GPUs, but run best at 8GPUs or less.
4- With several PCIE 3.0 x1 slots, and running Linux, you could do a lot of work with any GPU that's about as fast as a 2060KO or slower. On Core 21, you could even run a 2080 Super, with only 10-12% performance penalty. On Core 22, the penalty for a 2060 to 2060 Super isn't that much. You still get better PPD than GTX hardware.
 But starting from 2070-2080 territory, the penalties are massive, because the PCIE bus is saturated.
 A 2070-2080 has 2300-2950 Cuda Cores. A GTX 1060/1660 has 1280/1400 Cuda cores, or about half; so it needs about half the PCIE bandwidth. And a PCIE 3.0 x1 slot is good enough for up to ~GTX 1660Ti on Core 22 in Linux.
5- Once the CPU has arrived, install and fold on all GPUs you currently own. Sell the Celeron CPU. Buy any RTX 2060 KO or below that's a refurb, or second hand, or heavily discounted. Amazon still sells 2060 KOs for $330, which is a great price considering their performance! (more than 50% faster than a 1060), and despite their 'bad cooling', they're excellent candidates for open bench systems!
6- Once all motherboard PCIe slots are occupied, start selling off the slowest GPUs, and replace them with faster ones.
7- I would also get rid of the LED fan wall, and replace it with a single box fan. Buy a sheet of hepa or carbon filter material like this, and stick it to the back (suction side) of the box fan with simple packaging tape. Not only will this run cleaner, it'll also run less loud, and much more efficient (a box fan has 50/75/100W settings, but even at lowest setting, it offers plenty of cooling. In comparison those fans use about 3 watts per fan, for 18 fans, that's the same power consumption (~54 Watts), and they only really move air, when all of them are on.
8- After all that, just keep it running, and slowly build a performance rig; Depending on if you only want to run GPUs, or also do CPU folding (much slower).
 
For GPU only, I'd recommend an Intel 10th gen CPU, with motherboard and 3x RTX 2080Ti GPUs.
 
But another option is to wait a bit, until more PCIE 4.0/5.0 motherboards are on the market.
It makes no sense to buy an AM4 (TR4 even less) motherboard, when:
- a) there really aren't any GPUs that make benefit of PCIE 4.0 speeds; the only ones that support it is the 5600xt or something, which you could install on an x1 slot if risers worked, but AMD is not a good choice for continuous folding.
- b) The high core count CPUs (12+ cores like the Ryzens) usually have low clock speeds. And low clock speed can be a bottleneck on 2080Tis and/or future GPUs. It's best to get a CPU that can reach 4Ghz per core, if you want it to be future proof.
- c) You don't really want to exceed 1 PSU per motherboard. So whatever fits in a 1500W-1600W PSU power envelope, is going to be 4 high end GPUs (RTX 2080 Tis or their future equivalents); which is a good argument as to why would one need all those PCIe lanes anyway? AMD isn't handling PCIe as good as Intel does. I doubt their threadrippers will be able to successfully run more than 5 GPUs at a time.
 
For a GPU system, I would just wait another year or two, until Intel comes on the market with 7nm or 7++nm CPUs. They will be expected, to be competitive to AMD's 5nm CPUs in terms of power consumption, and at the same time will support PCIE 4.0 or 5.0, and hit higher CPU frequencies than AMD at 5nm.
 
If you also want to do CPU folding, I would definitely recommend a Ryzen 9 3900x or 3950x. Threadrippers, only the lowest end ones (like the 3960x). The higher core count CPUs (3970x/3990x) might not have enough core frequency to feed fast GPUs (their boost frequency is high, but boost on all cores is low(3,5Ghz or less), even on Ryzen CPUs). Threadripper motherboards are very expensive, but offer many full size PCIe slots at PCIE 4.0 speeds.

Cooling looking setup
I will definitely have to look into a mining setup.
Not enough space to run 3 cards on mine either and I have 4 slots
4 x PCIe 3.0 x16 (single at x16, dual at x16/x16 mode, triple at x16/x8/x8 mode or quad x8/x8/x8/x8 mode)
Thanks ProDigit

I myself do 3 GPUs per Motherboard, but it requires me to buy a mobo, and see if 3 GPUs can reliably run on it.
Not all can do that.
Some do x8/x8/x1, some just won't boot pre-bios.
Those I return (30 day return policy).
The boards I use are are MSI MPG 390 motherboards. They can run x8/x4/x4 or x8/x8/x4 setup.
Even at x2 speed, is sufficient for a 2070!
I use cheap 20cm PCIE 16x risers, to lift the GPU higher from the motherboard.
The main reason GPUs run bad, is because the hot air exhaust bounces off the motherboard, and gets sucked back in.
A $15 riser might make your GPU run cooler by as much as 20C, which in turn means a good 100-200Mhz higher core clock speed!
 
 

favrepeoria
What risers are you using the Thermaltake ones run about $30

For GPUs that are mounted in close proximity to the mobo (which I recommend), I'd buy the cheap grey 20cm PCIE x16 cables (with or without 4pin HDD power plug, doesn't really matter if you're plugging it in an x16 slot on the mobo side).
They do break down quicker, but if you're just mounting the GPU, and further not touch it (or not unplug/replug it more than 20-30 times) these are good enough!
Make sure they're double ribbon though! They go for $8-$12 on Amazon and Ebay.
 
For PCIE x1 risers, I use USB risers, as PCIE x1 riser ribbon cables are weak, and often require a longer (than 20cm) cable (I want to focus most of my full size slot GPUs as close to the motherboard as possible).

Or just use blower style GPU as they push Most of the heat out of the case
 
 

 

Based on @ProDigit suggestions my 'summer' folder is a dual 2060 super.  I often see this system >3M PPD.  With a 75% power cap.  Total system power draw is a very modest 370 watts.  GPUs in the low 60Cs. Older Q87 mobo, i7-4770k cpu.  The skeleton chassis was scavenged from an old tower system, it's mostly repurposed gear. Sorry, no EVGA gear in this system, sharing the concept.

Blower style GPUs generally have lower boost frequencies.
They also run hotter.
They also suffocate when placed so close together.
And they're louder.
 
If you're ok with that, go ahead and do it.
For me, open style GPUs makes more sense when they can be spaced out.
20cm ribbon risers allow you to run each GPU about 10cm apart.
The first GPU placed over the CPU (CPU will run fine as long as the CPU cooler isn't a stock Intel cooler).
Second GPU with 20cm riser hovering outside the case, if I wasn't running an open bench.
Third GPU with 20cm riser either mounted on the bottom slot (as any sub 2080Ti gets sufficient cooling there), if the GPU isn't blocking some of the USB/front panel ports.
In my scenario, the GPU does block the ports, so I install it on a 20cm riser, and place it ~15cm away from the first GPU.

I'm actually working on a very cheap rack.
Since all motherboards share the same ground, I bought 8x 60cm M3 threaded rods, nuts, and washers,and am in the process of mounting about 4 motherboards on them. The threaded rods will connect all the boards electrically to each other's ground, and will be additionally grounded straight to the grounding pin on the wall socket.

The threaded rods go in the motherboard mounting holes, creating a rack-like structure, with the shelves being the motherboards themselves, and the pillars being the threaded rods.
The whole thing cost me like $20, and I can adjust the height of each board, improving air flow.
On the right of each motherboard, will I place the water cooling systems using zip-ties, blowing hot air to the right of the contraption.
On the left I still need to figure out how a way to place the PSU, but it'll be with the fan inward, blowing the hot air forward of the contraption.
Both PSU and CPU water cooling radiators are sucking the heat away from the motherboard.
On the back I'll place a box fan with Hepa filter, blowing cool air on the CPU and GPU.
Each board will have only a single GPU, because I am doing boinc mostly, with Ryzen CPUs.
Each board will have 1 GPU mounted horizontally on a horizontal motherboard, with the use of a 20cm Riser, as running them vertically will overheat them (2080Ti at anything over 200W thermal throttles when the temps on the balcony exceed 86F, and will need to be power capped resulting in lower frequency). When the GPUs are mounted horizontally, like the motherboard, they will suck cool air from in front, and send the hot air to the PSU and CPU radiators. Meaning, it won't suck in any of the hot air it exhausts.
This will allow me to run higher boost frequencies, and have up to 20C lower temps (65C vs 86C vertically).
Despite this, the CPU usually remains fairly cool.
 
It's a work in progress, and looks kinda wimpy, but it's effective, light, economical, most efficient, and cheap.
 
More to come later!

Also, thank you for all of your contributions to the forum!