I've run into an issue that I can't seem to find an answer to.  Has anyone encountered an incompatibility between Z87 Classified motherboards, 290X TRIX GPU, and version 7.4.4 of the FAH client?  I suspect I've got a BIOS setting wrong or something for the Z87 Classified Motherboard because I'm having extreme instability issues with all the machines that are using those motherboards with the 290X TRIX cards.
 
Here is the background.  This will be a long post unfortunately. 
 
I've got six computers that are now being used to only fold.  They all have Z87 Classified motherboards, I7-4770K CPU's, 16 gig Corsair memory, two 290X TRIX cards in each machine.  Three of the six computers have 1200 watt Corsair PSU's and Samsung SSD's.  Three have 1300 Watt Antec PSU's and intel SSD's.  All the machines are running Windows 7 x64.
 
So the six machines are nearly identical but the two things they have completely in common is the motherboard and CPU.  What is happening is that I can not keep any of these machines folding for any length of time without getting video driver failures that become so severe that I can't remote into the computer anymore (via TeamViewer) and it actually requires a physical reset to get the machine to reboot.  If the machines will run 12 hours that is major victory.  Most of the time they won't run for 3-4 hours.
 
I'm very new to folding so bare with me.  I first noticed an issue when I saw what I would call hung WU's where the client would show 99.99% complete and then sit there forever.  After some research and looking in the log I saw the client GUI and the logs were out of sync so I started using that as a clue to something being wrong.  Usually a video driver failure. 
 
I had read to down clock my GPU's and they would be more stable.  So that is what I did and for literally three weeks I have been down clocking the GPU's on all the machines and there would be times they appeared more stable but then they wouldn't run for 30 minutes without getting out of sync again due to driver failures.
 
So then I started to wonder if the GPU's would run at any speed so I moved them to their lowest settings 520 core speed and 650 memory speed and I had the same issues.  So it appeared I couldn't keep the GPU's running at even their slowest clock speeds.
 
After many messages in the FAH forum and others here is a partial list of the things I've tried and it has not seemed to make any difference at all.
 
1) under clock the GPU's
2) under clock the RAM
3) disable CPU folding
4) change GPU's from machine to machine.
5) change GPU slots in the machines.
6) try different power plugs on the Antec PSU to make sure I'm not overloading a rail.
7) change the BIOS so PCIE2 is used instead of PCIE3.
8) enable crossfire (they had been running disabled)
9) run stress tests on the RAM (no errors found)
10) swap memory from machine to machine.
11) reinstalled Windows
12) after a fresh Windows install went straight to 14.4.
 
I'm sure there are others but you get idea.  Literally nothing I have tried has made any difference what-so-ever.  I can't imagine I've got motherboard issues since I mined with these same machines prior to moving them to folding and they ran for weeks unattended.  Now they can't stay up for half a day without having most of them either hard lock or become so corrupted they can't reboot.
 
Regarding the BIOS, all the machines are on 1.08 and all settings are default except memory speed and CPU C section which is now disabled.  I can't remember exactly what the CPU C section was called but EVGA support had me disable it.  Unfortunately the BIOS change hasn't seemed to help either.  They also suggested I make a post here.
 
Has anyone run into anything like this?  I suspect it is a BIOS setting but I have no idea which setting so I'm hoping someone has seen this before.
 
Another thing that makes me suspect the BIOS is that I've got two of the same GPU's in an older machine and is running a different motherboard and they run fine.  No issues what-so-ever although they are under clocked slightly they will run for days without incident.  I've swapped the GPU's between the machine using an older motherboard and a machine with the Z87 motherboard and it made no difference.
 
Any help would be greatly appreciated!
 
 
 
 
 
 
 
 
 
 

Sounds like the GPUs are switching from 3D clocks to 2D.  I don' t know if the 290x has the option or not, but try setting the Power Tune in Catalyst Control to +20.  That's helped me maximize efficiency on some DC projects on a 7950.  Although mine didn't switch clocks, but may still be something to try?

Do you have Crossfire enabled? My 2 R9 290X Fold much better and more stable with Crossfire Disabled.
Also the current Driver Catalyst Software Suite 256 MB 14.4 4/24/2014 Download is stable for Folding as well.
I use version 7.3.6 of the FAH client software, for me 7.4.4 is to complex or is it simple to use.
I Do Not Overclocking the GPUs while Folding and or Crunching.

 "I'll try to focus on one machine because I suspect whatever we find will apply to all of them." 
 
That's what I was going to suggest to make it a bit easier, until a fix is found.
What motherboard is it on the older tech that is working?

@notfordman:  The motherboard of the machine that will run with two of the GPU's without incident is an ASUS Sabertooth P67.
 
@bill1024:  Great suggestion!  You have been doing this for a long time so if you think it could help I will do it.  I have not tried to specifically overvolt the cards.  I did set the power tune to +20 and I still had a couple of incidents over night.  I have never overvolted anything though so I have no idea how I should do that.  I'll look it up now.  I've noticed overvolting is not enabled in MSI Afterburner for my cards but their is a place to override it in the settings.  I just don't want to burn the house down!   I am available to watch the temps today though.
 
I'm only using power cables that actually came with the PSU's although they are split so-to-speak.  Two plugs for the GPU that goes back to a single plug into the PSU.  I can try to swap them out for other cables but I don't think I have enough single plug cables to cover all the GPU's.  I should be able to test it on one system though.
 
The GPU's are plugged directly into the motherboard.  No risers are in use.  I actually tried powered risers from several different makers and never could get them to work with these cards but that is different story.  That is actually how I ended up with six machines with only two GPU's in them...  UGH
 
The power settings in Windows are set to "Never".
 
I'll start with the overvolting and then literally changing out the power cables.
 
@planetclown:  I've set PowerTune on the command line while mining but in MSI AB there is a setting called Power Limit which I set to +20.  Do you know if it is the same thing as Power Tune.  I assumed it was but maybe it isn't.
 
Updated info on run test:
 
I mentioned I was running one of my most unstable systems with just one card folding although two cards are actually in the system.  With only one card folding it has not had an issue for over 36 hours now.  On that system running 36 hours is a milestone.  I believe for whatever reason it can handle one card just fine but it doesn't like two.  I'm not sure exactly what that means but it is clearly stable with one card and highly unstable while folding with two.
 
Does that support the voltage idea?
 
Update:  I've overvolted one of my machines.  It appears that MSI AB is forcing voltage moves of what appears to be 6.5 mV increments.  So I moved it to 6.5 and I've also increased the Power Limit to +30.  This was done on just one machine.  I'll report what happens.  GPU temps do not appear to have been impacted yet.
 
 
 

Another update:  I swapped PSU cables on two machines late last night to single plug cables and spread them out more on where they plug into the PSU. Both machines were down again this morning. So it apparently didn't make a difference. Tonight I'll try a supplemental PSU but I have to rob it from another machine to use so I'll only try it for a day or two.
 
The machine where I switched to folding on the 2nd GPU is folding fine still for over 48 hours.

Looking through your list of things that you have done (top post and well done BTW), I didn't see anything about trying other Video drivers.  There have certainly been times when certain drivers (versions) on the NVidia side of things were not compatible with Folding.  Maybe step back or forward to a different driver version that the one that you are using to see if that has any effect?

Ok.  Yeah, we saw this once before with a Folder that used to be here.   I know that we all tried just about everything we could suggest and like you, he could not run (2) GPUs (edit: Fold on both GPUs) on the same mobo.  I never did read that he ever solved that issue either.  It may just be a mobo issue in the end, especially if you can run those same two cards in another mobo together without problems.  Wish I had an answer for ya...

@texinga, it seems the machines are more stable now that I've put most things back to where I started from.  Higher clock settings on the GPU's (although still underclocked), enable CPU mining again, etc.  When I continued to drop things to find a stable point that just seemed to make things worse.
 
I'm still contemplating whether I should just sell off one GPU in each machine and run them with one or whether I should keep running two and just fix them when I can.  I suspect the machines do more work with two GPU's overall but it has become a bit of a job to keep them running.  A job I unfortunately don't have time for much longer so the down time on the machines will increase a bit starting next week since I just won't be around to reset them.  I access them with TeamViewer but many of the GPU crashes actually make Windows so unstable that it can't restart on it's own.
 
@Punchy, you are right.  I did not try to overvolt anything but the GPU's.  I'm certainly willing to try it!  I just don't know what settings need to be changed in the BIOS to make it happen.  My BIOS version is 1.08.  I've talked with EVGA phone support about this stuff before but they can't actually see the BIOS screens.  So they are essentially trying to guess based on another BIOS screen.