yaggaz

Rewire92

A firmware update should be able to resolve this.

Wow thanks for figuring all this out.  If they could fix with a firmware update, would that be a case of limiting the card's performance to achieve safety?

It better not be considered a solution if it limits the cards perfomance. This is not why we buy FTW3 cards at a higher cost. I wouldn't accept that.

Its the transient response of the card during voltage shifts.  If you want to know more about transient response and how it affects electronic circuits I recommend watching some of Buildzoid's videos on Youtube.
 
The really oversimplified version is that changes in state can lead to overshoot and you can't see this in monitoring software, only on an oscilloscope.  The voltage spikes you can't see can be much higher than the safe limits, eg. the card tries to ramp up to 1.081V, but overshoot could say lead it to run at 1.3V for a brief moment.  It leads to crashes at least, and at worse will just burn out components.
 
The power delivery of the FTW3 cards can't respond fast enough to regulate the voltage changes safely - my guess is they didn't quite account for something correctly with the 3x8 power inputs or the programming for the VRMs isn't right.  Best case they can reprogram the firmware and a BIOS update can fix this, worse case the card might need a redesign of its power delivery/filtering.  We already know that Nvidia pushed Ampere to the limits so much that early cards were crashing, with some cheaper cards being more prone to crashing.
 
If you're wondering why someone would be interested enough to go listen to Buildzoid's technical ramblings, it taught me a lot about LLC settings and overclocking my CPU.  I can instantly crash my otherwise stable system by forcing an extreme state transition - eg. launching Intel XTU's benchmark test.  I was able to tweak LLC on my Z370 board enough to deal with a lot of instability but its VRMs were never designed for the power hungry i9-9900k I upgraded to.  What you're doing in LOL is going from low load state to max speed/voltage so its a big jump in the curve - you can mask the problem by limiting the overshoot say setting 1.05v max, so your overshoot is lowered enough that it doesn't crash your system, but it doesn't solve the bad transient response on the card itself.

neteng101
Its the transient response of the card during voltage shifts.  If you want to know more about transient response and how it affects electronic circuits I recommend watching some of Buildzoid's videos on Youtube.
 
The really oversimplified version is that changes in state can lead to overshoot and you can't see this in monitoring software, only on an oscilloscope.  The voltage spikes you can't see can be much higher than the safe limits, eg. the card tries to ramp up to 1.081V, but overshoot could say lead it to run at 1.3V for a brief moment.  It leads to crashes at least, and at worse will just burn out components.
 
The power delivery of the FTW3 cards can't respond fast enough to regulate the voltage changes safely - my guess is they didn't quite account for something correctly with the 3x8 power inputs or the programming for the VRMs isn't right.  Best case they can reprogram the firmware and a BIOS update can fix this, worse case the card might need a redesign of its power delivery/filtering.  We already know that Nvidia pushed Ampere to the limits so much that early cards were crashing, with some cheaper cards being more prone to crashing.
 
If you're wondering why someone would be interested enough to go listen to Buildzoid's technical ramblings, it taught me a lot about LLC settings and overclocking my CPU.  I can instantly crash my otherwise stable system by forcing an extreme state transition - eg. launching Intel XTU's benchmark test.  I was able to tweak LLC on my Z370 board enough to deal with a lot of instability but its VRMs were never designed for the power hungry i9-9900k I upgraded to.  What you're doing in LOL is going from low load state to max speed/voltage so its a big jump in the curve - you can mask the problem by limiting the overshoot say setting 1.05v max, so your overshoot is lowered enough that it doesn't crash your system, but it doesn't solve the bad transient response on the card itself.

 Nice explanation! I tried to reach out to Buildzoid about this matter, but so far he didn't answer.
You think it's possible for us to limit the overshoot? Like make a custom bios or something? Or it's just EVGA that holds this kind of power?

Does mining affect cards? Ill have to go back and check (got 1x3080 FTW3 mining in my spare time)
 
Have only played Dota 2 while mining, and a bit of SOTR.

Can anyone tell me how to adjust the MSI curve on the back end? It keeps jumping in alignment with the lower voltages.

EVGA needs to recall or put out an emergency bios update for these cards.
I can 100% replicate this issue (older low demand games will blackscreen my system in minutes and require several reboot cycles to even output display) with my FTW3 Ultra that seems to have been progressively failing the past week.  It might be only certain cards with faulty voltage regulation/spikes at fault here (mine is 2012 China model).

no00wa
Has anyone had a card that isn't necessarily failing (permanently) but is suffering from degraded performance, like stuttering in games, after the fact? clockspeeds / memory speeds all seem normal.

Better yet... Someone with a perfectly working card for 2+ months in older games? Cause it feels like ALL cards have this problem, but only a few vocal folks come forward to shout about it.
Mine should arrive today and I have 7 days to return it. If it fails even once I'll return it and get a TUF, since the Strix is WAY expensive here. :/

If they revise it, you believe they will recall the already sold ones? And what about consumers outside US and Europe? Normally they recall for those too? Sorry... I'm a little worried haha

If they can make the cards safe without a noticeable performance loss (seeing as the issue, if widespread, seems to be at low power-states, not in power-hungry apps/games), I would bet they just put out a bios update that forcibly does what you've done in MSI-AB. From a business perspective, I would expect a manager to weigh the cost/benefit of that vs. a complete recall and go with the bios update followed by a PCB revision that just goes out to newer orders. It's costly to do a board spin in the middle of production, but if you can time it right for a smooth cut-in, that's ideal. This is following exlink's theory on HC delays.
 
The end result is your card works great, it doesn't fail, your most demanding games see no loss in performance, and the problem goes away...for everyone that updates their bios. Cutting in new boards eventually just removes the dependency on people to update their bios or worrying about what bios is on what cards.

Does anyone know if this is related to EVGA cards, FTW specifically, or just 3000 series in general?

i.am.pekk
This is very interesting. I also have an FTW3 Ultra 3090 and I just RMA'd the first one. I am not so OC savvy to know about how undervolting and overclocking works so well but I can confirm I had the same black screen issue. My screens go black, the monitor stops receiving a signal. Sometimes my audio still works (game audio and discord audio still come through). I can sometimes continue talking through open mic on discord as well.
 
I mostly play Overwatch, where I have most settings on the minimum (to optimize for frame rate and eliminate distractions). I think I've seen OW crash once or twice at most, but usually it doesn't have any issues. Sometimes when this crash occurs the computer is able to recover so the black screen is just a flicker and my desktop returns shortly after (like switching inputs on the monitor would appear). At this point anything that uses hardware rendering seems to fail. Blizzard's bnet launcher fails to render after a recoverable crash saying that the UI can't get a 3d context (or something like that), which is often resolved by exiting and restarting the client.
 
Software that has reliably caused issues:

• Mozilla Firefox - Assuming this is related to GPU acceleration and video playback (youtube, twitch.tv, websites that use GPU accelerated animations?)
• League of Legends - I play at max graphical settings here, unlike OW - Crash happens mid-game
• Resident Evil: HD Remaster - Also max settings
• Sekiro: Shadows Die Twice - Also max settings - Seems to happen while alt-tabbing to desktop
• da Vinci Resolve 16 - Has crashed multiple times
• da Vinci Resolve 17 beta - has also crashed once

My second card has only crashed once or twice so far and I haven't put in a ticket yet as it was only a day or so ago and I'm still trying to see what the issue might be to provide more details. I thought maybe I was using some bad/cheap cables and that was my issue.
 
I have 2 monitors plugged in:

1. Dell Alienware AW2518HF 240hz (DisplayPort 1.2 cable)
2. El Gato HD60S Capture Card (HDMI 2.0 Cable)

 
It seems like most people who have issues have multiple screens. I was wondering if the refresh rate of the screens/resolution had any correlation. i.e., the amount of data bandwidth required to run those screens?
 
I just replaced the HDMI cable on the capture card hoping that would resolve it since the capture card was also having some issues with sound dropping and video colors distorting. So it seemed likely that the capture card was somehow related.
 
Oddly League of Legends was one of the most reliable crash reproducing games. I crashed enough that I got a leave buster warning for leaving too many games and had to digitally sign an agreement to never leave another game early.
 
I've never once crashed during 3DMark Time Spy.
 
If you need more software to use to test reliability you can try the ones I mentioned. Also worth noting I often lurk in many twitch streams to help viewership so there could be multiple videos playing simultaneously, though I mute the sound through firefox or by using a separate sound output device for the web browser and mute that.
 
Has undervolting reliably fixed your issue? I saw some other posts on nvidia forums where people tried undervolting to no avail, so I never bothered trying it myself.

My workaround should fix your issue.  You are having the same issues I had, which my voltage curve adjustment has resolved at this point 100%.

I have 5 monitors, no identical models.

This is my new (10 days old) 3090 FTW3 Ultra (First one was a 2012SN and died playing LOL after 40 days)
This is a 2014SN

Is this so bad?
the GPU-Z values are recorded after a couple of games on Overwatch.
Since I got the new card I've experienced none of black screen, fan ramping etc...

In 1 hour game I found in GPU-Z Log 40 lines including the 1.0810 spikes (a couple of seconds each)

Exsurgolol
While waiting for EVGA support to get back to me I have been searching for ways to replicate this issue.
I have now found 2 benchmark suites that replicate this low power game issue and will usually black screen lock up my system within 5 minutes, all at stock settings.
 
The alarming thing is the spikes to 1.081V only black screen my card faster.  Spiking frequently to 1.062V will blackscreen my system too, it just takes a little longer.  Black screens are probably possible lower down the voltage curve too, but much less likely.
 
The issue with replicating this problem is I think you need games or benchmarks that vary significantly in performance, i.e pushing the card up and down the V/F curve (3DMark etc tends not to do this).  Problem is this actually does happen a lot in games when you transition through cut scenes, enter menus etc, but often can take 1h+ for the issue to arise.  Even FS2020 can blackscreen for me because when approaching a busy airport for landing the game becomes CPU bottlenecked, causing GPU utilisation spikes.

I wanted to chime in on this issue as I have a FTW3 3080 with which I was getting constant crashes, at random intervals, when gaming. Crashes where my entire system would lose power instantly and I'd have to reset the power switch on my PSU to even turn the system back on. Crashes were happening when playing Tarkov and Valheim recently, as well as other games previously that I can't recall which game). WHEA was showing the problem causing the shutdown was related to Power. At first given some advice from others I thought perhaps it was a PSU related problem and I simply did not have enough juice to support the massive power spikes this card can hit (I've seen it peak to almost 400W and I know it can draw even higher amounts than that) because the issue only started happening when I swapped my 5800x for a 5900x. Thus it made since given that I inserted a higher power draw CPU and now the card's spikes would be enough to trigger the overcurrent protection causing the shutdown. I thought I was getting somewhere with that, as I lowered the power limit to 65% and was no longer able to cause full shutdown even after 30+ minutes of stress testing. For comparison, I had been using Furmark+Prime95 blend running 24 threads+OCCT's Power stress test all simultaneously. This was the only combination of tests I found that was able to replicate the shutdown, and it was occurring without fail within 5 minutes of running all 3 with stock settings, sometimes within 60 seconds. I noticed my voltage also peaking to 1.063V many times (including the instant before shutdown), and the max hitting that same number you saw too: 1.081V. So given what I had read in this thread and knowing it was either related to power or the GPU's voltage I did some further testing. Since 65% power limit caused no issues, and lowering the power limit also lowers voltage, I decided to reset everything to stock and allow 100% power limit. From there, I did a complete undervolt to 0.0975V. The same combo of stress tests that was causing a crash within 5 minutes then ran flawlessly for 2 consecutive 30+ minute sessions with no crash, and I have no crashed while gaming yet. It definitely isn't a wattage issue either: at 100% power limit with the undervolt I'm still seeing maxes at near 400W just like I was on stock settings. The only difference is my voltage doesn't hit those spikes, obviously.

Sir, I think you're definitely on to something and have figured out this issue. It also appears to be an issue affecting FTW3s in general, at least for 3080s and 3090s. Perhaps it is related to the 3 8 pin connection vs the 2 8 pin connection found on the 3060 Ti and 3070 versions as I haven't found anything related to those having issues. Either way: if you read this thread just know that Exsurgo is correct and his solution will likely work for you, as it did for me. We both had basically the exact same situation, exact same voltage spikes, and same solution despite me having a 3080 and him having a 3090. If you have the FTW3 version of these cards PLEASE keep an eye on them. Also, while I listed the way I was able to replicate the shutdown on my PC: follow along at your own risk knowing that forcing the GPU to kill itself like that could lead to long term or permanent damage. I have an old card to use and could have patiently waited for an RMA if mine died doing these tests, but I'd hate to see someone kill their card trying to do this. Just listen to us and undervolt your GPU and see the results.

Rewire92

i.am.pekk
...
Has undervolting reliably fixed your issue? I saw some other posts on nvidia forums where people tried undervolting to no avail, so I never bothered trying it myself.

My workaround should fix your issue.  You are having the same issues I had, which my voltage curve adjustment has resolved at this point 100%.

I have 5 monitors, no identical models.

Sorry if I missed something in the thread, do I just eyeball the values from your screenshot or is there a way for me to download your curve and load it in? Can I do it with the EVGA X1 tool or do I need MSI afterburner?

Paynal

i.am.pekk
Has undervolting reliably fixed your issue? I saw some other posts on nvidia forums where people tried undervolting to no avail, so I never bothered trying it myself.

 
Popping in to add a perspective from someone who's been running with this solution for an extended period on a November 2020 red 3090 FTW Ultra.  I had my first blackscreen of impending death first week of December, 2020, and started undervolting immediately afterwards (via Afterburner).  Since then, I've subjected the card to dozens of hours of different workloads, ranging from video and lightweight 2D platformers to stress tests, RTX, and high framerate, high rez VR.  VR in particular is prone to placing wildly-shifting loads on the system that are a prime candidate for triggering a voltage spike as the card rapidly shifts from high to low and back up again.  (Half-Life Alyx was what did it for me, and I've seen that one mentioned as trouble by other VR users on this board.)
 
The results?  2.5 months of perfect performance.  No more blackscreens, with or without maxed out fans.  No red light of death.  No voltage spikes in my logs.  Card performance has not degraded, I'm still pulling the same scores in 3DMark I was months ago and seeing the same framerates in VR.  I'm not sacrificing performance in high demand games, either, as this particular card happily runs at 1980mhz at 0.881 volts.
 
TL;DR -- Undervolting works for the long run, go for it.
 
 
@Rewire92, Exsurgolol, and Fracture-7 -- I saw the exact same spikes you three did when I started having trouble last year.  It was even the same exact values.  Card would regularly throw spikes up to 1.060 and stay on its feet (with stutters for a second or two sometimes), but 1.061 would often come with a program crash, though the card would usually stay up.  1.063 and up and it might crash violently -- blackscreen with either screaming fans or several second blackscreen and then stuck in an extreme low-clock state (200-400mhz), requiring a system reboot to clear.

So this is very similar same as the MSI MXM GTX 1070 v1.0 bug, except that happened between 0.950v-1.013v (someone said it was exactly 1.013v which crashed but I never owned that card, I have the v1.2 version), and it didn't destroy the card either.  Probably because of much lower power draw.  Hard for magic smoke to come out of a 115W card.
 
http://forum.notebookreview.com/threads/1070-laptop-gt73vr-gt62vr-gt72vr-reboot-crash-problem.804978/
 
If anyone here cares.  I know a lot of people don't care much about my posts.