EVGA

Evga SR-2 with 8 GPU´s ?

Page: < 1234 Showing page 4 of 4
Author
Post
Essentials Only Full Version
gordan79
SSC Member
  • Total Posts : 531
  • Reward points : 0
  • Joined: 2013/01/27 00:17:36
  • Status: offline
  • Ribbons : 3
Re: Evga SR-2 with 8 GPU´s ? 2015/09/15 15:08:29 (permalink)
No, I mean you need to find their size using whatever way you have to do so in Windows if that is what you are using. Under Linux you can check using "lspci -vvv". My point was that if the IOMEM apertures have been made bigger on the TX, you may be out of IOMEM space, but it is not at all clear from your description that is really what is going on. 3GB which is what the BIOS can allocate, should be good for more than 14 Kepler class cards, so unless they more than doubled the BAR sizes on the TX, this _shouldn't_ be a problem (caveat: assuming the BIOS does something vaguely sensible, which may well not be the case).
 
I don't know if strap changing using nvflash is even supported on Maxwell. I haven't even looked at Nvidia BIOS-es after Kepler.

Supermicro X8DTH-6, 2x X5690
Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
3x GTX 1080Ti
Triple-Seat Virtualized With VGA Passthrough (KVM)
#91
thunderbolt78
New Member
  • Total Posts : 16
  • Reward points : 0
  • Joined: 2014/03/22 05:10:09
  • Status: offline
  • Ribbons : 0
Re: Evga SR-2 with 8 GPU´s ? 2015/09/15 15:25:03 (permalink)
found a lspci windows tool and run this on the titan x node and the 7x gtx node
 
can you see something ?
04:00.0 VGA compatible controller: NVIDIA Corporation GM200 [GeForce GTX TITAN X] (rev a1) (prog-if 00 [VGA controller])
 Subsystem: eVga.com. Corp. Device 2990
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0, Cache Line Size: 64 bytes
 Interrupt: pin A routed to IRQ 32
 Region 0: Memory at ce000000 (32-bit, non-prefetchable)
 Region 1: Memory at b0000000 (64-bit, prefetchable)
 Region 3: Memory at c0000000 (64-bit, prefetchable)
 Region 5: I/O ports at 7000
 Capabilities: [60] Power Management version 3
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
  Address: 0000000000000000 Data: 0000
 Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
  DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
   ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
  DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
   RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
   MaxPayload 256 bytes, MaxReadReq 512 bytes
  DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
  LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM not supported, Exit Latency L0s <512ns, L1 <4us
   ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
  LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
   ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
  LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Via message
  DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
  LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
    Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
    Compliance De-emphasis: -6dB
  LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
    EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-

 
02:00.0 VGA compatible controller: NVIDIA Corporation GK110BGL [Tesla K40c] (rev a1) (prog-if 00 [VGA controller])
 Subsystem: eVga.com. Corp. Device 1787
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0, Cache Line Size: 256 bytes
 Interrupt: pin A routed to IRQ 0
 Region 0: Memory at f5000000 (32-bit, non-prefetchable)
 Region 1: Memory at b8000000 (64-bit, prefetchable)
 Region 3: Memory at b6000000 (64-bit, prefetchable)
 Region 5: I/O ports at 9f80 [disabled]
 Capabilities: [60] Power Management version 3
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
  Address: 00000000fee3f00c Data: 4990
 Capabilities: [78] Express (v2) Endpoint, MSI 00
  DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
   ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
  DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
   RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
   MaxPayload 256 bytes, MaxReadReq 512 bytes
  DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
  LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM not supported, Exit Latency L0s <512ns, L1 <4us
   ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
  LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
   ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
  LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
  DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
  LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
    Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
    Compliance De-emphasis: -6dB
  LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
    EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

 
kind regards
thunderbolt78
#92
gordan79
SSC Member
  • Total Posts : 531
  • Reward points : 0
  • Joined: 2013/01/27 00:17:36
  • Status: offline
  • Ribbons : 3
Re: Evga SR-2 with 8 GPU´s ? 2015/09/15 15:33:41 (permalink)
On the lines:
Region ...: Memory at ...
it should be reporting the size of the regions. Those are the BAR sizes I was referring to. Unfortunately, the tool you are using isn't reporting that. Here is what it says on my laptop:
01:00.0 3D controller: NVIDIA Corporation GM204M [GeForce GTX 970M] (rev a1)
[...]
       Region 0: Memory at f6000000 (32-bit, non-prefetchable)
        Region 1: Memory at e0000000 (64-bit, prefetchable)
        Region 3: Memory at f0000000 (64-bit, prefetchable)
 
The number in the brackets is what I'm talking about. On my SR-2:
07:00.0 VGA compatible controller: NVIDIA Corporation GK110B [GeForce GTX 780 Ti] (rev a1) (prog-if 00 [VGA controller])
[...]
       Region 0: Memory at f4000000 (32-bit, non-prefetchable)
        Region 1: Memory at c8000000 (64-bit, prefetchable)
        Region 3: Memory at d4000000 (64-bit, prefetchable)
 
So it does indeed look like Nvidia bumped the total BAR size from 176MB to 304MB between Kepler and Maxwell.
 
But SR-2 can allocate up to 3GB of IOMEM, and 7x304MB=2128MB. It is plausible that the BIOS can't actually achieve a high enough occupancy, but as I said, it is worth checking that you have configured the memory hole in BIOS to 3GB.
 

Supermicro X8DTH-6, 2x X5690
Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
3x GTX 1080Ti
Triple-Seat Virtualized With VGA Passthrough (KVM)
#93
thunderbolt78
New Member
  • Total Posts : 16
  • Reward points : 0
  • Joined: 2014/03/22 05:10:09
  • Status: offline
  • Ribbons : 0
Re: Evga SR-2 with 8 GPU´s ? 2015/09/15 22:44:18 (permalink)
hi gordon, i did not see any differences.
 
you wrote
 
01:00.0 3D controller: NVIDIA Corporation GM204M [GeForce GTX 970M] (rev a1)
[...]
       Region 0: Memory at f6000000 (32-bit, non-prefetchable)
        Region 1: Memory at e0000000 (64-bit, prefetchable)
        Region 3: Memory at f0000000 (64-bit, prefetchable)

 
is f6000000 + e0000000 + f0000000 = 304MB
 
and
 
F4000000 + C8000000 +D4000000 = 176 mb
 
kind regards
thunderbolt78
 
ps. this is what i found in the cuda documentation
 
2.5. PCI BAR sizes

PCI devices can ask the OS/BIOS to map a region of physical address space to them. These regions are commonly called BARs. NVIDIA GPUs currently expose multiple BARs, and some of them can back arbitrary device memory, making GPUDirect RDMA possible.

The maximum BAR size available for GPUDirect RDMA differs from GPU to GPU. For example, currently the smallest available BAR size on Kepler class GPUs is 256 MB. Of that, 32MB are currently reserved for internal use. These sizes may change.

On some Tesla-class GPUs a large BAR feature is enabled, e.g. BAR1 size is set to 16GB or larger. Large BARs can pose a problem for the BIOS, especially on older motherbords, related to compatibility support for 32bit operating systems. On those motherboards the bootstrap can stop during the early POST phase, or the GPU may be misconfigured and so unusable. If this appears to be occuring it might be necessary to enable some special BIOS feature to deal with the large BAR issue. Please consult your system vendor for more details regarding large BAR support.

 
post edited by thunderbolt78 - 2015/09/15 23:21:20
#94
gordan79
SSC Member
  • Total Posts : 531
  • Reward points : 0
  • Joined: 2013/01/27 00:17:36
  • Status: offline
  • Ribbons : 3
Re: Evga SR-2 with 8 GPU´s ? 2015/09/16 01:53:55 (permalink)
The addresses are the _starting_ addresses, you need either the end address or the size for comparison. Your version of lspci isn't giving you the size like the Linux one does.
 
Either way, if you really are running into the BIOS limitation on the motherboard and it isn't capable of mapping all the BARs, you are probably out of luck. At a push, you could try to flip the relevant bits in the strap to reduce the Region 1 BAR size from 256MB to 128MB, but this may not work, and it could break the BIOS on the card which may involve unbricking procedures such as externally rewriting the EEPROM with the original copy of the BIOS. I haven't really looked into this closely since the Fermi (GTX4xx) days, and since then at least some of the strapping has changed (e.g. there are now hard and soft straps for the PCI device ID).
 
Also note that smaller BARs means copying data to/from the VRAM in smaller chunks which will have an impact on performance (cannot really say how big an impact, though).
 
Since the 3GB IOMEM limit BIOS limitation is not really changeable (BIOS is 32-bit, so it cannot map things beyond the 4GB memory limit, although arguably a better written BIOS should be able to achieve much better memory occupancy which would make the bigger BARs fit in 3GB anyway), other than trying to experiment with the VBIOS straps yourself, I guess you could try and ask someone at Nvidia to cook you up a modified BIOS to try, but I wouldn't hold my breath for that.

Supermicro X8DTH-6, 2x X5690
Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
3x GTX 1080Ti
Triple-Seat Virtualized With VGA Passthrough (KVM)
#95
thunderbolt78
New Member
  • Total Posts : 16
  • Reward points : 0
  • Joined: 2014/03/22 05:10:09
  • Status: offline
  • Ribbons : 0
Re: Evga SR-2 with 8 GPU´s ? 2015/09/16 03:30:56 (permalink)
i read you previous port several time,
 
you say "The number in the brackets is what I'm talking about. On my SR-2:" where are the number on your post ?
 
is there a way to see the used iomem on a computer with running windows or dos ?
 
kind regards
thunderbolt78
#96
gordan79
SSC Member
  • Total Posts : 531
  • Reward points : 0
  • Joined: 2013/01/27 00:17:36
  • Status: offline
  • Ribbons : 3
Re: Evga SR-2 with 8 GPU´s ? 2015/09/16 04:58:20 (permalink)
Sorry, as I said, I'm not one to ask a Windows question, I don't use it.
But based on my (admittedly tiny) amount of sample data (2 cards, one top of the line Kepler, one high end Maxwell), the BAR size did in fact go up, so this could plausibly be causing your problems.
What you can do about it is an entirely different question. You will need to experiment with the BAR straps. The most up to date documentation on that is here:
http://sourceforge.net/p/nouveau/envytools/ci/master/tree/hwdocs/pstraps.txt
but it is by no means very recent, it's from back in 2012, so I don't even know if the BAR straps as described there will work or whether they'll brick your card if you change them, or cause some other unpredictable effect.
 
As I mentioned before, you would almost certainly be much better off getting a better motherboard, one with a UEFI firmware instead of a legacy BIOS.
 
Good luck.

Supermicro X8DTH-6, 2x X5690
Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
3x GTX 1080Ti
Triple-Seat Virtualized With VGA Passthrough (KVM)
#97
thunderbolt78
New Member
  • Total Posts : 16
  • Reward points : 0
  • Joined: 2014/03/22 05:10:09
  • Status: offline
  • Ribbons : 0
Re: Evga SR-2 with 8 GPU´s ? 2015/09/16 08:20:05 (permalink)
nvidia-smi tool is able to show the bar size
 
$ nvidia-smi -q
...
    BAR1 Memory Usage
        Total : 256 MiB
        Used : 2 MiB
        Free : 254 MiB
...


that means 7x titan needs 1792 MiB of IOMEMM


post edited by thunderbolt78 - 2015/09/16 08:22:08
#98
gordan79
SSC Member
  • Total Posts : 531
  • Reward points : 0
  • Joined: 2013/01/27 00:17:36
  • Status: offline
  • Ribbons : 3
Re: Evga SR-2 with 8 GPU´s ? 2015/09/16 08:25:30 (permalink)
Yup, it's 256MB, same as my GTX970M. You need more than 7x256, because there are 2 other BARs, 1x16MB and 1x32MB for each card. That brings the total to 2128MB.

Supermicro X8DTH-6, 2x X5690
Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
3x GTX 1080Ti
Triple-Seat Virtualized With VGA Passthrough (KVM)
#99
bewick
New Member
  • Total Posts : 1
  • Reward points : 0
  • Joined: 2016/03/08 04:16:28
  • Status: offline
  • Ribbons : 0
Re: Evga SR-2 with 8 GPU´s ? 2016/03/08 14:53:57 (permalink)
Is there any way the forum administrator can revive the attached images of Code Dynamic's Bios settings?? 
#100
Aiwandia
New Member
  • Total Posts : 37
  • Reward points : 0
  • Joined: 2015/02/03 08:26:20
  • Status: offline
  • Ribbons : 0
Re: Evga SR-2 with 8 GPU´s ? 2016/03/13 17:23:35 (permalink)
I'll second that having only just discovered this thread.
 
Gordy has been helpful before despite my unworthiness.
I'd ask him again about the 96GB "secret sauce" as mine don't.
Fortunately ' printed the "secret sauce" page basics.
I'm with you regards seeing 96GB but resetting BIOS each booting.
I'm also doing 8 GPU on SR-2, x4 AMD FP s10000 6GB ECC (optional).
This works but overheats rapidly (x2 AMD FP s10000 is happy on air).
The kicker is with H2O blocks the s10000 becomes single slot.
This may allow a single slot rat cage blower between each s10000.
This configuration is ECC so is only comparable with NVQ6000 (or not?).
 
post edited by Aiwandia - 2016/03/14 07:37:17
#101
Gadgety1
Superclocked Member
  • Total Posts : 101
  • Reward points : 0
  • Joined: 2012/11/15 05:02:22
  • Status: offline
  • Ribbons : 0
Re: Evga SR-2 with 8 GPU´s ? 2016/03/14 06:31:00 (permalink)
Me too. If the images disappearing is due to the Forum Admin, please revive all images. If it's not, a statement from the FA to the effect would be appreciated. This is a very interesting thread.
#102
Aiwandia
New Member
  • Total Posts : 37
  • Reward points : 0
  • Joined: 2015/02/03 08:26:20
  • Status: offline
  • Ribbons : 0
Re: Evga SR-2 with 8 GPU´s ? 2016/03/14 07:32:29 (permalink)
Do I start a new thread?
EVGA SR-2 GPU cores how many: What does Pascal or Volta
have to offer SR-2 owners by way of legacy 7 slot HPTX form factor?
The die size reduction should allow a doubling of GPU number per slot.
Otherwise do I seek additional Titans or be there Quadro6000 refurbishments?
You don't want AMD to take over this niche market do you?
 
EVGA, I'm seeing older SR-2 threads disappearing.
For shame 'lest you orphan the golden child which is your SR-2 monster.
I still need the "secret sauce" for my unopened A1, EVBot, Nateman combo!    
 
If I would do it again I'd 16GB DIMMs for 96GB with x6 RAM water blocks.     
Newest RAM H2O spreaders negate any savings on DDR3. 
#103
Page: < 1234 Showing page 4 of 4
Jump to:
  • Back to Mobile