Can you please reupload your pics? Saw this Thread now, and i missed it before...
 
LK

Hallo code !
10 gpu does it work for u ?
Am really interested
My CUDA SR-2 works really good.
I like to build all kinds of stuff so ordered refurbed 4xsupermicro server mobos, cheap ECC ram and 4x5650, 4x5640 from ebay.
so after christmas i will test my super rig, all in 1 homemade chassi with:
2xSR-2
4xSupermicro.
lot of GPUS.
12x56xx hexaxeons,, 144HT cores
my CUDA SR-2 and my ordinary home pc will be the master and the rest slaves.
the chassi is standing in another room  
 
 
  

17 GeForce.
Tip of the year
ooopsss,,,,,, **** I only got 7x pcie 
Gordan how do u fit your 17 PCIe cards on your mobo ?
Do u have the settings,,, please upload it !
 
     
 
   
 

very un-useful tip for multi GPU user gordan 
 
 
  
 
 
 

We want to know how to use 10 Nvidia gpu ? so help the post instead
Do u have any usefully Bios setting or other regarding 10GPUs ?,,, Mr Gordon with 2 nvidia GPU.
 
 " It is very difficult to give useful answers to pointless questions"
 
You are a not a multi GPU user so your talking blablabla ,, not helping at all
 
Telling the post that the limit of the board is 17 gpu and powergpu user need to hacking the bios
Totally useless information and helping nobody  
Nobody here hacking bios Gordon 
 
Better you hack a new bios for your 96GB so u dont have to reset bios everytime u start your unstable SR-2 
 
     
 
 

OK - what are you going to use 10 GPUs for? Name one meaningful, practically implementable use case. Who does running 10 GPUs help, and what is such a configuration useful for? You are saying my response is useless and unhelpful, yet you are ignoring the fact that the entire undertaking of running 10 GPUs is useless and unhelpful.
 
The only setting in the BIOS that is relevant to running many GPUs is the memory hole (which is settable to a maximum of 3GB). No other setting is relevant to running many devices with large I/O memory requirements. There is no way to influence where the BIOS maps the different I/O memory blocks, so it will either work or it won't, and if it doesn't, your only hope is to start modifying the soft straps on the VBIOS.
 
Those are documented here:
https://indefero.0x04.net...ed2/hwdocs/pstraps.txt
The bits you would need to adjust are:
BAR 0: bits 17,18,19
BAR 1: bits 14,15,20,21,22
BAR 2: bit 23
 
To adjust them, dump the BIOS and look at the location appropriate to your GPU to get the current strap (remember to swap the byte order) and modify it accordingly. Flash the new strap to the GPU with nvdlash --straps. Getting it wrong will in many cases result in a bricked GPU and you will need to unbrick it by booting it with the BIOS chip disabled, then re-flash it. If you are planning to experiment with this I highly recommend soldering a switch and a resistor (in series) across the VCC and GND for easy unbricking.
 
And for the record, my 96GB SR-2 boots every time within 30 seconds from a cold boot without BIOS hacking, thanks. And it's up and running 24/7, with 6 months' uptime, getting hammered pretty hard almost all the time of late (transcoding my DVD collection to my Plex server).

IMO, given what you are spending on GPUs already, you would do well to invest into a decent motherboard first. Something with a UEFI firmware would be a good start.
 
Given you are trying to use up to 14 GPUs (7 slots with dual GPU cards), you will need at least 2688MB of PCI IOMEM area. Although the SR-2 can theoretically deliver up to 3GB, I would be very surprised if the BIOS' IOMEM mapping had an allocation occupancy good enough to give you enough. Also note that SR-2 uses a legacy BIOS, not an UEFI one, so all BARs, regardless of whether they are 64-bit capable, have to get mapped under the 4GB limit (hence the 3GB IOMEM limitation).

gordan79
IMO, given what you are spending on GPUs already, you would do well to invest into a decent motherboard first. Something with a UEFI firmware would be a good start.
 
Given you are trying to use up to 14 GPUs (7 slots with dual GPU cards), you will need at least 2688MB of PCI IOMEM area. Although the SR-2 can theoretically deliver up to 3GB, I would be very surprised if the BIOS' IOMEM mapping had an allocation occupancy good enough to give you enough. Also note that SR-2 uses a legacy BIOS, not an UEFI one, so all BARs, regardless of whether they are 64-bit capable, have to get mapped under the 4GB limit (hence the 3GB IOMEM limitation).

Thanks Gordon.

hallo to all of you.
ar you stil running multie dual gpu's for rendereing and other calculations programs?
and if so ar you willing to run the GPUPI 2.0: The Multi-GPU Pi Benchmark?
it can be downloaded at http://www.overclockers.at/downloads/GPUPI%202.0.zip
more info about it.
https://www.overclockers.at/news/gpupi-international-support-thread
http://hwbot.org/news/11959_gpupi_2_0_launched_in_cooperation_with_hwbot

thanks of your time you ar wiling to spend on this benchmark.
whit what gpu config ar you gone run?
if i have read good an ontherstand your post #73
you have 6 titan-z and 4 normal titan's spread over 2 setups.
did you try to run all your titan-z on one system?

skulstation2
thanks of your time you ar wiling to spend on this benchmark.
whit what gpu config ar you gone run?
if i have read good an ontherstand your post #73
you have 6 titan-z and 4 normal titan's spread over 2 setups.
did you try to run all your titan-z on one system?

 
My studio uses 116 GPUs (all of them CUDA based) to render 3d animations.  We use the following GPU renderers: (1) Octane Render [ http://home.otoy.com/rend...render/showcase/ ], a GPU only renderer,  (2) Thea Render [  https://www.thearender.co...php/gallery.html ], a hybrid GPU and CPU renderer, (3) FurryBall [ http://furryball.aaa-stud...yBall/index.html   ], a GPU only renderer and (4) (soon to be acquired) Redshift 3d [ https://www.redshift3d.com/gallery ].  My systems are OSX, Windows and Linus based.  Since each of the 3d rendering products is licensed on a per system basis, it behooves me to ensure that I maximize/consolidate the number of GPUs in each of them, to contain licensing costs, as well as to speed up the rendering process (which can take very long periods of time).  Thus, it'll probably be a couple of weeks from now before I can spend a few hours running the benchmark you were gracious enough to bring to my attention.  To run that benchmark, I'll probably use, at a minimum, my system with 7-8 GTX Titans (the first ones released).
 
Yes, I unsuccessfully tried to run in a single system, namely a Tyan (8 double width slotted) GPU server, 6 Titan-Z Hydros's along with both a Titan Black Hydro and one Titan Hydro (one of the first Titans released).  6x2 Titan Zs = 12 GPU processors; 12 + 2 (one GPU processor for the Black and one GPU processor for the regular Titan) =14 GPU processors in total, but only 8 GPU cards.  It required more IO space than the system had available.  Thanks to Gordan, I now have a clearer picture of what I'm up against and because of Gordan's sharing his knowledge, I'm better focused on ways to build my 9th and 10th massively parallel, GPU processor, 3d rendering systems.
 
Regarding your last post, I'm not sure what you're referring to when you say, "old 3870x2."  Please give me a little more details.
 
P.S. You can also following my exploits here [     http://forums.macrumors.c...ad.php?t=1333421  ]  and here [  http://render.otoy.com/fo...c.php?f=40&t=43597   ].  Also, I used to be a hackintosher [ http://www.insanelymac.co...eekbench-scores/   ] and have long been a system modder since the mid-1980s [ see, e.g., http://www.computerworld....hz-mac-pro-.html  ].

skulstation2
thx for the helpful replay.
116 gpu's thads a lopt of render power.
 
whit my post about the old 3870x2,i an referring to the ati hd 3870x2 card from around 2008.
i have 4 card colekting dust.
i did try blender , 3dmax and 2 other but i fergot ther name,the 3870x2 cards ar not working ther.
gpupi also not working whit this cards,the cards don't have dubbel precision and opencl support.
 
don't rush.try it when you have time and when your gpu's ar taking a brake

As indicated, albeit abreviatedly, in my signature, my 116 CUDA processors have the rendering capability of over 89 of the GTX Titans.  If I run my systems with the GPUs and the systems fully tweaked, then the rendering equivalency surpasses that of 100 GTX Titans.
 
I'm assuming that when you say that the 3870x2 aren't working with Blender, 3dMax, etc. that you mean that the video cards aren't working to compute renders, and not that the cards simply don't display any video.  http://en.wikipedia.org/wiki/List_of_AMD_graphics_processing_units#Radeon_R300_Series indicates that the AMD 3870x2 cards are OpenCL (the GPU computing aspect of those cards) version 1.0.  OpenCL version 1.0 is too old for almost all modern 3d applications that take advantage of OpenCL's rendering ability.   The current AMD cards support OpenCL version 4.3.   That's most likely why your 3870x2 cards won't work in Blender, 3dMax, etc.  I think that's asking far too much from those 2009 cards.  In 2009, very few applications took advantage of OpenCL computing.  However, the bright side for you is that current GPUs are far less expensive that CPUs.   Moreover, current CUDA GPUs are, at least, over ten times faster than are CPUs at ray traced rendering and some GPU/application combos are up to 100 times faster than CPU/application combos at direct lighting (cartoon style) rendering  (see the FurryBall site reference in my post, above).  Also, note that at the end of the Furryball movie, it states that what you've seen was rendered on one GTX Titan.

we do the same stuff like tudor, we render with octane, redshift, blender and furryball. We decided to upgrade our 7 GPU Rig 7 Tesla K20 with 7 Titan X. The problem is now, the board only accepts 5 of the 7 cards. As far i read, it has todo with the iomem stuff, like gordan describes. is there any program to read the iomem somehow. i saw several information to modify the BAR straps. but i dont know how.
 
is gordan available ?
 
kind regards
thunderbolt78
 

the board starts with 7x GTX 780 6GB without problems
the board start with 5 Titan X 12GB without problems, all are there in the device manager and working
with card 6 and 7 the board dont start, no boot nothing.
if i replace  card 5 with 6 or 7 the board starts
 
any idea ?

gordan79
Very strange. Do you have all the auxiliary power inputs populated on the motherboard, including the extra ones for the CPUs? What do you get on the diagnostic LCD on the motherboard?
Check how big the IOMEM segments are on each card, see if they are bigger on the TX than on the 780. Have you checked that the memory hole is set to 3G in the BIOS?

i use the evga sr-x with 4 titan x, works perfect, the board i want to use with the 7x titan x is a supermicro x8dth, maybe you can help me with this board too.
 
where or how i can check the IOMEM segments ? i have the bios allready with nvflash, can i read the straps somehow ?
 
kind regards
thunderbolt78