EVGA

Got word from NVIDIA and this issue they are working on.--UPDATED 12/14

Author
scott@bjorn3d
SSC Member
  • Total Posts : 692
  • Reward points : 0
  • Joined: 2006/11/20 04:37:19
  • Status: offline
  • Ribbons : 2
2016/12/10 04:03:58 (permalink)
Got this from a connection at NVIDIA, guess the old days of owning Bjorn3d.com helps out a little.
 
 
"Hey Scott,
 
The team has been working on this around the clock to unblock the folding community and we believe the root cause is a bug in the Folding@home dependent OpenMM software.  We’ve engaged with them to see if they agree and get changes done by 12/14.   
 
In case we can’t get those changes done in the app soon, we’re working on a fall-back option in our JIT compiler, based on an app profile to disable an optimization that is exposing the synchronization bug in the app.  We’re targeting to get that driver also posted publically on 12/14.
 
I’ll let you know if either of the above doesn’t happen for some reason and the new ETA.
 
Thanks for your patience!
 
Regards,
Team NV"
 
12/14 Update
 
Hey Scott,
 
Our efforts to get a resolution out by 12/14 has run into some roadblocks.  The change suggestion we made for OpenMM needs more tweaks and our driver side workaround is close to being done but not complete yet.  We plan to post a hotfix driver to unblock the Folding@Home users as soon as we have a workaround to share (within the next few days). 
 
We’ll keep you posted.
 
Regards,
Team NV
post edited by scott@bjorn3d - 2016/12/14 03:10:26


#1

25 Replies Related Threads

    Mekhed
    iCX Member
    • Total Posts : 415
    • Reward points : 0
    • Joined: 2014/04/19 19:14:48
    • Location: Rocky Mountain High
    • Status: offline
    • Ribbons : 1
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 05:14:26 (permalink)
    That sounds encouraging.



     "Kaos"   ASUS RAMPAGE V EXTREME | Intel Core i7-5930K | Corsair H110i GTX | 2x 1080TI Hybrid & 1080 Hybrid | HyperX 32GB 3000 | Supernova 1200 | Corsair 780T
    "BigDawg" Asus Maximus VI Formula | i7-4790k | Corsair H100i | 2x RTX 2080 Ti FTW3 Ultra | Mushkin Redline 16GB 2400 | Supernova 1300 | Antec 1200
    #2
    Chris21010
    FTW Member
    • Total Posts : 1587
    • Reward points : 0
    • Joined: 2006/05/03 07:26:39
    • Status: offline
    • Ribbons : 9
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 06:06:30 (permalink)
    awesome!


    #3
    DutchForce
    CLASSIFIED Member
    • Total Posts : 2341
    • Reward points : 0
    • Joined: 2006/12/04 14:25:44
    • Location: Netherlands
    • Status: offline
    • Ribbons : 52
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 06:38:53 (permalink)
    Thanks for your effort to get more (good) info from NVidia, Scott, it's very much appreciated !!! 


     
    Folding Rig: Lian-Li PC-P50 | Corsair HX850 | MSI 890FXA-GD70 | AMD X4-965 BE | GTX980 Ti SC & GTX1070 SC
    #4
    drougnor
    CLASSIFIED Member
    • Total Posts : 2715
    • Reward points : 0
    • Joined: 2006/07/18 05:57:59
    • Location: FL USA /Affiliate MCK692UUS /Assoc HPW7E9T81P8KCCH
    • Status: offline
    • Ribbons : 18
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 07:09:27 (permalink)
    Whenever I see a line like "we’re working on a fall-back option in our JIT compiler, based on an app profile to disable an optimization that is exposing the synchronization bug in the app" I have to wonder what this means for our production numbers?
     
    **Edit - With a little coffee and thought, I realize they're saying it's NVIDIA's code optimization that forced the bug to come out of hiding, so disabling it means they'll go back to the code that is currently working, therefore NOT giving us worse numbers. But, if that optimization is allowed to go through properly, we could see BETTER numbers. So, let's hope OpenMM fixes it!**
     
    I really do hope that they'll be able to get things straightened out properly, obviously, but am ready to keep the current driver installer on hand just in case.
     
    d
    post edited by drougnor - 2016/12/10 07:23:09


    #5
    HK-Steve
    CLASSIFIED Member
    • Total Posts : 2251
    • Reward points : 0
    • Joined: 2015/04/06 08:46:57
    • Location: Switzerland
    • Status: offline
    • Ribbons : 2
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 07:48:39 (permalink)
    fingerscrossed

     


    #6
    scott@bjorn3d
    SSC Member
    • Total Posts : 692
    • Reward points : 0
    • Joined: 2006/11/20 04:37:19
    • Status: offline
    • Ribbons : 2
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 08:40:22 (permalink)
    Oh yeah I keep copies of all working drivers.
     


    #7
    rjohnson11
    EVGA Forum Moderator
    • Total Posts : 102215
    • Reward points : 0
    • Joined: 2004/10/05 12:44:35
    • Location: Netherlands
    • Status: offline
    • Ribbons : 84
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 08:49:46 (permalink)
    Yes this is the same info I've been given from NVIDIA so hopefully this can be resolved soon.

    AMD Ryzen 9 7950X,  Corsair Mp700 Pro M.2, 64GB Corsair Dominator Titanium DDR5  X670E Steel Legend, MSI RTX 4090 Associate Code: H5U80QBH6BH0AXF. I am NOT an employee of EVGA

    #8
    bcavnaugh
    The Crunchinator
    • Total Posts : 38977
    • Reward points : 0
    • Joined: 2012/09/18 17:31:18
    • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
    • Status: offline
    • Ribbons : 282
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 09:17:09 (permalink)
    I will believe it when I see it.
    https://folding.stanford.edu/home/core-21-openmm-gpu-issues-with-new-windows-nvidia-drivers-375-57-and-375-63/ 
    Still Points to NVIDIA and Not to (OpenMM) or Their Coding.
    post edited by bcavnaugh - 2016/12/10 09:20:16

    Associate Code: 9E88QK5L7811G3H


     
    #9
    Chris21010
    FTW Member
    • Total Posts : 1587
    • Reward points : 0
    • Joined: 2006/05/03 07:26:39
    • Status: offline
    • Ribbons : 9
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 10:39:44 (permalink)
    that update is 2 weeks old and may not still be relevant. either way as long as we can get something working with relatively the same performance and allow people to play the newer titles will be a win/win no mater who is at fault.


    #10
    bcavnaugh
    The Crunchinator
    • Total Posts : 38977
    • Reward points : 0
    • Joined: 2012/09/18 17:31:18
    • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
    • Status: offline
    • Ribbons : 282
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 10:45:18 (permalink)
    Chris21010
    that update is 2 weeks old and may not still be relevant. either way as long as we can get something working with relatively the same performance and allow people to play the newer titles will be a win/win no mater who is at fault.


    Yes two Weeks and Still No New info from Folding @ Home
    post edited by bcavnaugh - 2016/12/10 17:03:00

    Associate Code: 9E88QK5L7811G3H


     
    #11
    kougar
    CLASSIFIED Member
    • Total Posts : 3034
    • Reward points : 0
    • Joined: 2006/05/08 10:11:19
    • Status: offline
    • Ribbons : 22
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 16:11:25 (permalink)
    Thanks for the update!! Props to NVIDIA for debugging the OpenMM software which is out of their purview. Very glad to see some concrete dates


    Have water, will cool. 
    #12
    ipkha
    CLASSIFIED Member
    • Total Posts : 2308
    • Reward points : 0
    • Joined: 2015/02/10 21:01:40
    • Location: Indiana
    • Status: offline
    • Ribbons : 6
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/10 20:06:51 (permalink)
    Stanford has never once accepted fault in their coding. I've seen work units fail seconds in, and their software says I must have been overclocking. Never-ending that 90% of the other tasks work just fine. Frankly, this is an open source problem in general. No one wants to admit that their code may broken. I'm just glad there is a fix available from Nvidia, it'll be months before Stanford fixes OpenMM. And even when they do fix it, the folding folks would still have to update the core to use a newer version. It's a big mess all around.


    #13
    drougnor
    CLASSIFIED Member
    • Total Posts : 2715
    • Reward points : 0
    • Joined: 2006/07/18 05:57:59
    • Location: FL USA /Affiliate MCK692UUS /Assoc HPW7E9T81P8KCCH
    • Status: offline
    • Ribbons : 18
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/13 05:59:20 (permalink)
    Tomorrow's the day, either direction. Cross those fingers, folks!
     
    d


    #14
    bcavnaugh
    The Crunchinator
    • Total Posts : 38977
    • Reward points : 0
    • Joined: 2012/09/18 17:31:18
    • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
    • Status: offline
    • Ribbons : 282
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/13 11:08:48 (permalink)
    drougnor
    Tomorrow's the day, either direction. Cross those fingers, folks!
     
    d


    Core 22? or New Driver?
    post edited by bcavnaugh - 2016/12/13 11:11:46

    Associate Code: 9E88QK5L7811G3H


     
    #15
    Chris21010
    FTW Member
    • Total Posts : 1587
    • Reward points : 0
    • Joined: 2006/05/03 07:26:39
    • Status: offline
    • Ribbons : 9
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/13 12:14:45 (permalink)
    well new driver for sure, but i also hope that Stanford fixes this bug in core 21 sooner than never.


    #16
    drougnor
    CLASSIFIED Member
    • Total Posts : 2715
    • Reward points : 0
    • Joined: 2006/07/18 05:57:59
    • Location: FL USA /Affiliate MCK692UUS /Assoc HPW7E9T81P8KCCH
    • Status: offline
    • Ribbons : 18
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/13 12:25:21 (permalink)
    With luck it'll be a 'simple' (depending on where in the code the bug is and what is needed to straighten it out) update to the core that will auto update without us even knowing.  That's the kind of fix PG tends to prefer - No fuss, no muss.
     
    d


    #17
    bcavnaugh
    The Crunchinator
    • Total Posts : 38977
    • Reward points : 0
    • Joined: 2012/09/18 17:31:18
    • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
    • Status: offline
    • Ribbons : 282
    Re: Got word from NVIDIA and this issue they are working on. 2016/12/13 12:28:21 (permalink)
    That is good to hear, I will stage my GTX 980 and GTX 1080 Rigs with the Current Driver and Stand By.

    Associate Code: 9E88QK5L7811G3H


     
    #18
    EllBrad
    New Member
    • Total Posts : 69
    • Reward points : 0
    • Joined: 2016/06/15 12:51:39
    • Location: Canada
    • Status: offline
    • Ribbons : 0
    Re: Got word from NVIDIA and this issue they are working on.--UPDATED 12/14 2016/12/14 07:16:31 (permalink)
    scott@bjorn3dHey Scott,
     
    Our efforts to get a resolution out by 12/14 has run into some roadblocks.  The change suggestion we made for OpenMM needs more tweaks and our driver side workaround is close to being done but not complete yet.  We plan to post a hotfix driver to unblock the Folding@Home users as soon as we have a workaround to share (within the next few days). 
     
    We’ll keep you posted.
     
    Regards,
    Team NV




     

    i5 4690k @ 4.5ghz | EVGA GTX 1070SC @ 2.1ghz | 16gb DDR3 @ 1866mhz | Acer XB270HU
    #19
    drougnor
    CLASSIFIED Member
    • Total Posts : 2715
    • Reward points : 0
    • Joined: 2006/07/18 05:57:59
    • Location: FL USA /Affiliate MCK692UUS /Assoc HPW7E9T81P8KCCH
    • Status: offline
    • Ribbons : 18
    Re: Got word from NVIDIA and this issue they are working on.--UPDATED 12/14 2016/12/14 08:06:24 (permalink)
    Updates are good. Not as good as a resolution, but it IS an update!
     
    d


    #20
    scott@bjorn3d
    SSC Member
    • Total Posts : 692
    • Reward points : 0
    • Joined: 2006/11/20 04:37:19
    • Status: offline
    • Ribbons : 2
    Re: Got word from NVIDIA and this issue they are working on.--UPDATED 12/14 2016/12/14 08:24:40 (permalink)
    The good news is the CEO of NVIDIA is being CC on the progress. 


    #21
    bcavnaugh
    The Crunchinator
    • Total Posts : 38977
    • Reward points : 0
    • Joined: 2012/09/18 17:31:18
    • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
    • Status: offline
    • Ribbons : 282
    Re: Got word from NVIDIA and this issue they are working on.--UPDATED 12/14 2016/12/14 08:31:43 (permalink)
    Not happy with this "our driver side workaround is close to being done but not complete yet"
    Workarounds are only Band-Aids and Fall Off they are Not Fixes.
     
    EllBrad
    scott@bjorn3d
    Hey Scott,
     
    Our efforts to get a resolution out by 12/14 has run into some roadblocks.  The change suggestion we made for OpenMM needs more tweaks and our driver side workaround is close to being done but not complete yet.  We plan to post a hotfix driver to unblock the Folding@Home users as soon as we have a workaround to share (within the next few days). 
     
    We’ll keep you posted.
     
    Regards,
    Team NV


     
    Did 376.33 Fix it As seen below.
     

    Misc Notes Re: Folding@Home Issue

    It was posted yesterday that this driver should've fixed the Folding@Home issue. Link here
    Seems like they hit a "roadblock" and the fix did not make it in time to this driver. A hotfix will be deployed instead.
    Here's the update from the post:
    12/14 Update
    Hey Scott,
    Our efforts to get a resolution out by 12/14 has run into some roadblocks.  The change suggestion we made for OpenMM needs more tweaks and our driver side workaround is close to being done but not complete yet.  We plan to post a hotfix driver to unblock the Folding@Home users as soon as we have a workaround to share (within the next few days).
    We’ll keep you posted.
    Regards, Team NV
     
    Funny that the link above Link links back here http://forums.evga.com/Got-word-from-NVIDIA-and-this-issue-they-are-working-onUPDATED-1214-m2593454.aspx#2593454 and not to https://bjorn3d.com/
     
    http://us.download.nvidia.com/Windows/376.33/376.33-desktop-win10-64bit-international-whql.exe
    http://us.download.nvidia.com/Windows/376.33/376.33-desktop-win8-win7-64bit-international-whql.exe
     
    post edited by bcavnaugh - 2016/12/14 08:54:19

    Associate Code: 9E88QK5L7811G3H


     
    #22
    Ali Man
    Superclocked Member
    • Total Posts : 244
    • Reward points : 0
    • Joined: 2012/08/01 12:13:06
    • Status: offline
    • Ribbons : 0
    Re: Got word from NVIDIA and this issue they are working on.--UPDATED 12/14 2016/12/14 08:57:26 (permalink)
    Anyone tried the latest driver?
    #23
    bcavnaugh
    The Crunchinator
    • Total Posts : 38977
    • Reward points : 0
    • Joined: 2012/09/18 17:31:18
    • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
    • Status: offline
    • Ribbons : 282
    Re: Got word from NVIDIA and this issue they are working on.--UPDATED 12/14 2016/12/14 09:00:00 (permalink)
    Ali Man
    Anyone tried the latest driver?


    They did not fix the Folding Issue as stated in the Post above, still waiting for the Hotfix File to be released.
    post edited by bcavnaugh - 2016/12/14 14:20:19

    Associate Code: 9E88QK5L7811G3H


     
    #24
    ipkha
    CLASSIFIED Member
    • Total Posts : 2308
    • Reward points : 0
    • Joined: 2015/02/10 21:01:40
    • Location: Indiana
    • Status: offline
    • Ribbons : 6
    Re: Got word from NVIDIA and this issue they are working on.--UPDATED 12/14 2016/12/14 11:43:47 (permalink)
    Workarounds are good for a temporary solution while waiting for a permanent fix. Nvidia is obviously quicker off the mark and can always remove the hack once the root cause gets patched.


    #25
    bcavnaugh
    The Crunchinator
    • Total Posts : 38977
    • Reward points : 0
    • Joined: 2012/09/18 17:31:18
    • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
    • Status: offline
    • Ribbons : 282
    Re: Got word from NVIDIA and this issue they are working on.--UPDATED 12/14 2016/12/14 14:18:18 (permalink)
    Please use this Thread for any and All Folding Issues
    Drivers 376.09/.19/.33 still no fix for FaH (Core21 on Maxwell/Pascal GPUs). 
    Thank you
    post edited by bcavnaugh - 2016/12/14 16:15:34

    Associate Code: 9E88QK5L7811G3H


     
    #26
    Jump to:
  • Back to Mobile