So I recently moved my setup to a DG-85 case and hooked up a custom WC loop for my GPUs and CPU.  I've got 2x980 TI EVGA Hybrids in SLI and I've been having stability issues with the setup.  It appears it's linked to the GPU in slot 1, but I'm not sure if it's a problem with the card, the MB, or some power setting.  When I have PCI-E slots on (MB allows to turn off slots as needed) the voltage on the top card is showing about 100mV less than the bottom card (bottom is at ~ 950 mV, while top shows around 850mV).  And this is in Windows without any stress on the GPUs.  If I try to benchmark with the slot 1 card at the lower voltage I have stability issues.  Precision doesn't show an increase in voltage at all on the clock during benchmarking and I'll get the system to lock up.  Now I thought it might be an issue with a cable or a PSU port, so I switched some cabling around and somehow get the same voltage on both cards.  If both cards are running at 950 mV, the system is stable even during benchmarking.  
 
But now for some reason the top card is back down to 840 mV and I'm having stability issues again.  And me fiddling around with the cabling isn't getting the voltage back up. So is this an issue with the GPU?  The MB?  The PSU?  (I've got the supplemental 6-pin power cable plugged into the MB and set the BIOS to use it). I didn't think this could be addressed in Precision since looks like Precision only allows an increase of +87mV, larger than the difference I'm seeing between the GPUs.  I would like to swap the cards around to see if it's an issue with the card itself, or the PCI-E slot.  But I really don't want to drain and dismantle my loop if I can help it.
 
System specs
CPU: 4790K
Board: EVGA Z87 Classified with V1.09 BIOS (latest)
GPUs: 2x980 TI EVGA Hybrids in slots 1 and 4 (stock WC system pulled and replaced with EKWB blocks)
PSU: EVGA P2 1200 W PSU
 
Thanks!
 

Sajin
Running a custom vbios with the voltage locked to 950 mV can fix the issue.

Haven't ever made a custom BIOS, this a pretty easy thing to do?    Or just ask in the forums?  If the voltage is locked in the BIOS, is that set to be a minimum state?  Or can the voltage still increase based on demand?  I would anticipate needing more to keep stable under load.
 
I did see this other post you had replied to a few years back:
 
https://forums.evga.com/GTX-980-TI-BIOSVOLTAGE-HACK-m2360712.aspx
 
Would this one apply?  Looks like this BIOS would just up the voltage to the max. The 980 TI Hybrid is based off a stock PCB, so looks like it'd be a fit.  And the cards are in a loop, so not too concerned about temps.

jonkrmr
First off, do not use another card's BIOS. Even if it is from the exact same brand\model of card. Use your BIOS on your card. Make a copy of it, use an editor to just change the voltage of the GPU (you can specify minimum and full boost voltage separately) in the BIOS and then flash it back to your card. You can find some real good information on this on how to do it and the utilities needed from www.overclock.net ( http://www.overclock.net/t/1558645/official-nvidia-gtx-980-ti-owners-club ). There are a couple of guys on that forum who will even modify your BIOS for you for free. You just need to send them a copy of your BIOS and tell them what you want. I have been running a modified BIOS on my 980ti since I bought it a little over a year ago. Have it set to remove all power\temp\voltage limits so the card will boost as far as I can push it in Afterburner. 375 watt TDP and max voltage of 1.274v. Gives me 1523 MHz boost with no throttling at all. 100% stable for over a year now.

Thanks for the advice.  I found that OCN post on another BIOS post here and had just bookmarked it for reading tonight.  Good to know I'm in the right place.  I'd like to learn the BIOS modding myself, so will probably end up modding and send it to the group to see if I missed something.

Sajin

fragility_V1

Sajin
Running a custom vbios with the voltage locked to 950 mV can fix the issue.

Haven't ever made a custom BIOS, this a pretty easy thing to do?    Or just ask in the forums?  If the voltage is locked in the BIOS, is that set to be a minimum state?  Or can the voltage still increase based on demand?  I would anticipate needing more to keep stable under load.
 
I did see this other post you had replied to a few years back:
 
https://forums.evga.com/GTX-980-TI-BIOSVOLTAGE-HACK-m2360712.aspx
 
Would this one apply?  Looks like this BIOS would just up the voltage to the max. The 980 TI Hybrid is based off a stock PCB, so looks like it'd be a fit.  And the cards are in a loop, so not too concerned about temps.

Yes, it's pretty easy. I can do the mods if you like. The voltage will only be locked to a certain voltage when under load. Card will drop to normal idle voltage when not under a load. It would be best to mod your bios instead of using that one in the thread.
 
Before we mod the bios...
 
Have you re-installed your graphics drivers after cleaning out your old ones with ddu? Have you set the global power management mode inside the nvidia control panel to prefer maximum performance & rebooted the system to see if it helps fix the problem you're having?

I was looking at the process and it does seem pretty easy.  May give it a shot myself first.  Will be good to know when I eventually get around to the overclock.
 
I've reinstalled the drivers a few times with DDU.  Currently running the Beta Nvidia drivers, 385.12.  I have not tired the global power setting yet, though.  I'll update and give that a shot.  I've got the other card disabled at the moment so I can actually use the system.

As setup I had the GPU in slot 4 active (slot 1 GPU was off), turned on max performance, and restarted.  The single GPU voltage increased to 1.024 V while idle at a GPU/ Mem clock speed of 1139 MHz/3505 MHz.  
 
Shutdown and activated the Slot 1 GPU (keeping slot 4 active).  Voltage and GPU/Mem clock stayed the same on the Slot 4 GPU.  However, Slot 1 GPU only increased about 100 mV to 949 mV at idle.  Clock speeds at idle were only 135 MHz.    Ran a bechmark test just to see, slot 4 GPU remained inactive during the benchmark.  And the benchmark froze at the last cycle.  (SLI was not active, didn't have the bridge installed).  So no stability.
 
Shutdown and deactivated the Slot 4 GPU.  (keeping slot 1 active).  Max performance still enabled.  The single GPU voltage increased to 1.024 V while idle at a GPU/ Mem clock speed of 1139 MHz/3505 MHz.  Ran benchmark and got stable results (oddly significantly better results than when the slot 4 GPU is active alone, but that's a different issue).
 
So for some reason the power settings are not applied to the slot 1 GPU when slot 1 and 4 are active.  But independently the GPUs work as expected.  

Sorry, wasn't quite clear on my description.  With SLI off the Slot 1 GPU was at 949 mV at idle, but wasn't active during the benchmark and kept the 949 mV voltage.  The slot 4 GPU was doing all the lifting. It was going at ~1.04 V.
 
Haven't rolled back the drivers yet, but now I'm playing around with BIOS settings on the MB.  Going through the options the Z87 Classy has options in the GPU Link tab to individually set the GPU voltage on each slot.  By default this is set to Auto.  With both cards turned on, and no SLI bridge, I tried setting the GPU voltage for slot 1 and 4 to 1.2V.  With this turned on both cards showed ~950 mV.  Ran a benchmark, and this time the Slot 1 GPU was active.  Completed the benchmark, seemed stable.
 
So shutdown, added the bridge, rebooted and turned on SLI.  After turning on SLI immediately noticed a change in voltage between the cards.  This time the slot 4 GPU dropped around 100 mV at idle to 850 mV.  Tested it with a benchmark and ran completed.  Ran it again and the benchmark crashed.  Didn't have time before work to investigate further, but thinking I may need a little more voltage tweaking in the MB BIOS.  Seems to be doing the trick.  Would tweaking the PEX/PLL voltage help with any stability?  Default is auto.
 
I'm still hitting the voltage limit on 1 of the 2 cards during benchmarking, so I'm still thinking of modding the BIOS to run at an increased voltage, but that may be after I'm believing the MB BIOS voltage is doing it's job.  

So now I'm thinking there might possibly be a problem with the board.  (thought I posted this update, but I guess I didn't hit submit).  Anyway, the voltage in the BIOS didn't end up solving my issues.  Upped the voltage and didn't solve the stability issues.  Especially since you noted the voltage on the GPUs were withing normal ranges (~850 mV at idle).
 
Not knowing quite what the issue is I messed around with the loop somewhat, to try to isolate the issue.  So let me try to summarize what I've done.  Let's call my original configuration, Config A, and my updated loop, Config B.
 
Config A:
Slot 1: 980 Ti Hybrid - Hynix memory
Slot 4: 980 Ti Hybrid - Samsung memory
 
Config B (swapped cards in slots):
Slot 1: 980 Ti Hybrid - Samsung memory
Slot 4: 980 Ti Hybrid - Hynix memory
 
Things I noticed in Config A:

• Slot 4/Samsung GPU benchmarks significantly lower than Slot 1/Hynix GPU (when turning off slots and testing individually)
• When running Heaven benchmark from Slot 4/Samsung GPU audio plays through monitor from DisplayPort.  Audio very crackly.
• Slot 1/Hynix GPU appears to be stable in benchmark and stress test
• Slot 4/Samsung GPU fails (significantly) 3DMark GPU stress test.
• Slot 1 only registering at x8 PCI-E 3.0, Slot 4 registering at x16 PCI-E 3.0

 
Things I noticed in Config A:

• Moving Samsung GPU to Slot 1 benchmarks are now significantly better, comparable to other Hynix GPU.  Basically passes stress test (94%)
• Slot 4/Hynix GPU can now barely boot into Windows.  Once Windows is loaded, refresh rate drops significantly and am getting a screen refresh once every 15-20 seconds.  Can't even open the bechmark software to test.
• Slot 1 still only registering at x8 PCI-E 3.0, Slot 4 registering at x16 PCI-E 3.0

So since I'm seeing good benchmark results on the cars individually when I swap them into Slot 1, I'm thinking the cards are good.  And then it's an issue with the board.  Anything I could be missing?

Good idea.  Manual says Slot 3 should give me x8.  I'll have to order some parts to reorganize the loop, will be a few days before I can update.
 
Is the Slot 1 PCI-E the only PCI-E slot going directly the CPU?  All others going through the PLX chip?

that's odd for a Z87 to only have one slot off the cpu  even with the plx chip ?    seems like  2 slots should of been dedicated  single  x16  and dual at 8x8   off the cpu  then have the plx chip kick things in after that
 
then the slot power should be the same and stable  on all primary graphics card slots   [well should be all at  ATX spec stable for the slots  ]  then if not I would think something is wrong wioth the boards voltage regulation 
 
''So what does the PLX chip do on a motherboard? ''
http://www.anandtech.com/show/6170/four-multigpu-z77-boards-from-280350-plx-pex-8747-featuring-gigabyte-asrock-ecs-and-evga
 
thing is seems none of that should affect voltages  / power   of your cards 
 
I se you talked up bios's  ? do the 2 cards you have even have the same bios  revision    if one of your card has a first release bios and the 2ed card has a revised /later revision bios the parameters between the bios's could be changed or differ between them ??? 
 
looking here the  980 TI EVGA Hybrids   had 2 or 3 bios revisions 
 
Version: 84.00.32.00.94  --  84.00.36.00.90  --  84.00.41.00.90  --  84.00.41.00.F0
 
https://www.techpowerup.com/vgabios/?architecture=NVIDIA&manufacturer=EVGA&model=GTX+980+Ti&interface=&memType=&memSize=&since=
 
 
then could be something with the voltage regulator of one of the cards as well ??
 

Do you only experience the sli performance drop in unigine benchmarks? Different vbioes between the cards isn't a problem.

So I ran a bunch of additional benchmarks in 3DMark last night with the cards in Slots 1 and 3, and things are still screwy.  Slot 1 still appears to be performing well (but with x8 instead of x16) but the Slot 3 performance appears to be off.  And SLI is still not showing a significant improvement:
 
3DMark Time Spy - DX12 BM
Cards in Slots 1 & 3
SLI on
     AVG=5530 MAX=5887
SLI Off (Slot 1 connected to monitor, both cards on)
     AVG=5526 MAX=5538
Slot 3 only (Slot 1 off)
     3506
     Firestrike Stress test failed (computer crashed on retest)
Slot 1 only (Slot 3 off)
     AVG=5526 MAX=5529
     Firestrike Stress test passed
 
So still appears to be something with the other PCI-E slots.  And the cards themselves should be OK since I swapped the cards in the previous reconfiguration and the BMs showed similar performance between the cards when they were in the Slot 1 position (other slots off).  Think I'll contact support, see if I can RMA the board.

Thanks for all the help, I'll let you know how it goes!

Looks like I'm SOL on getting a replacement.  EVGA has indicated their records show the board was shipped to the original owner in 2013 (bought it used, and didn't think of the warranty period).  So it's out of warranty.  On the hunt for a replacement!

Well, I managed to find (or rather someone found me) a Z97 Classy board.  Got it the other day and switched out the Z87 Classy.  So far so good.  Have both cards in, performance appears to be solid.  BMs in SLI are looking more like SLI benchmarks, and are much higher than single card benchmarks.  I've run stability tests in 3DMark twice so far, and both are passing at ~ 99%.  So all (mostly) appears to be good.
 
Now the one problem, the card in Slot 1 is only registering at x8 speed, while the card in Slot 4 is showing as x16.  Not that it's a significant performance issue at the moment, but just wondering why that could be.  Issue with the card?  Needs to be reseated?  Issue with the CPU? (if the Slot 1 has a direct PCIe lane to the CPU)  Something happened when I put on the GPU block?  BIOS setting?
 
Normally I'd swap them again, but I've torn apart the watercooling loop about 5 times already, and not looking to do it again if I don't have to.

The PCIe slot breakdown in the manual seemed to indicate they'd both run at effective x16 (or I'm confused by their table):

• PE1 - x16 (x8 if PE2 used)
• PE2 - x16 (x8 if PE3 used)
• PE3 - x8
• PE4 - x16 (x8 if PE6 used)
• PE5 - x1
• PE6 - x8

Or is the Slot 1 running at x8 since it's going through the GPU and the other 8 lanes are running through the PLX chip and showing an effective x16?