EVGA

Guide to Build/Install/Setup your own Multi-GPU Linux Folding Rig

Page: < 123 Showing page 3 of 3
Author
QuintLeo
SSC Member
  • Total Posts : 946
  • Reward points : 0
  • Joined: 2016/04/16 23:05:09
  • Status: offline
  • Ribbons : 3
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/01/28 12:34:33 (permalink)
nvidia-xconfig -a --cool-bits=31 --force-generate --mode 1280x1024
is what I use on my systems.
 
I use a mix of 9xx and 10xx cards, and the cool-bits settings CHANGED at least one of the bits between those generations, so I just set 31 to be sure I get ALL of the flags enabled I actually need.
 
Have to put sudo in front of that if you don't routinely do sudo -l (login to sudo to give you root access for that session) when you first log in.
 
Can change the mode as you prefer for your monitors.
 
 

Now that vorsholk has stopped his abuse, I'm returning to folding.
 I no longer MOO due to abuses by certain "whales" in the Gridcoin community - so I now work the Distributed.net project directly again.
 
#61
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/01/28 12:48:20 (permalink)
Wish I could fully understand that command. I tried for hours.


#62
bcavnaugh
The Crunchinator
  • Total Posts : 38977
  • Reward points : 0
  • Joined: 2012/09/18 17:31:18
  • Location: USA Affiliate E5L3CTGE12 Associate 9E88QK5L7811G3H
  • Status: offline
  • Ribbons : 282
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/01/28 13:49:29 (permalink)
Chris21010
Wish I could fully understand that command. I tried for hours.

Same here that is tried for hours.
But it might have been because of the difference between the OS.
 
Added:
I was using sudo
Heck you have use sudo just to list the files and folders.
post edited by bcavnaugh - 2018/01/29 16:58:13

Associate Code: 9E88QK5L7811G3H


 
#63
ipkha
CLASSIFIED Member
  • Total Posts : 2308
  • Reward points : 0
  • Joined: 2015/02/10 21:01:40
  • Location: Indiana
  • Status: offline
  • Ribbons : 6
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/01/29 12:50:14 (permalink)
sudo command is simple. Place it before you enter restricted commands, then enter your password. If you are in sudo user group it will work.


#64
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/01/29 16:50:42 (permalink)
sudo wasn't the command in question... 'nvidia-settings' is the command that has a giant potential with very hard to use applications due to poor/limited driver support. followed by its cousin 'nvidia-xconfig'. 
post edited by Chris21010 - 2018/01/29 16:51:55


#65
QuintLeo
SSC Member
  • Total Posts : 946
  • Reward points : 0
  • Joined: 2016/04/16 23:05:09
  • Status: offline
  • Ribbons : 3
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/01/29 17:39:26 (permalink)
OK explanation of the options.
 
 nvidia-xconfig is the command you use to have the system configure your xorg.cfg file so the drivers AND Xwindows itself can find the cards and set them up for correct usage.
 -a   configure ALL cards in the system.
 --cool-bits=31     set up so you can do fan speed control, overclocking/underclocking, TDP control, and such.
 --force-generate    set up the configuration from scratch, and overwrite anything that is in the way if needed while moving any existing xorg.conf file to a "backup" version filename.
 --mode 1280x1024   tell Xwindows to start up in this screen mode, and default to it, if the card supports this mode on the port it has a monitor attached to.
 
 The number of dashes is CRITICAL - don't leave one out or add an extra one as that will mess up that configuration option.
 Don't leave out spaces on the options that have spaces, or add them to the options that don't have them - see above.
 
 
 nvidia-settings is a pain to figure out without examples, but it's how you control clocks and fan speeds and it gets rather arcane at times.
 As one major weirdness, you have to set the CPU or Memory clock in 2 slightly different spots for them to actually "take" - only change one spot, it doesn't "take", at least on 10xx series cards.
 I'll get into that in a later post, I need to do some FTP work to get an example over to this machine.

Now that vorsholk has stopped his abuse, I'm returning to folding.
 I no longer MOO due to abuses by certain "whales" in the Gridcoin community - so I now work the Distributed.net project directly again.
 
#66
Hughesy26
New Member
  • Total Posts : 19
  • Reward points : 0
  • Joined: 2017/03/27 05:17:39
  • Location: Wales
  • Status: offline
  • Ribbons : 0
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/22 07:39:35 (permalink)
Can anyone provide an idiots guide to step 9 (being able to control fan speed) for a 4 GPU system?


#67
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/22 11:18:48 (permalink)
(NOTE: this config file only works if you have two GPU's that are identical. if you have more/less than two GPU's try creating your own with `sudo nvidia-xconfig -a --cool-bits=12 --allow-empty-initial-configuration`)

 
that is the easiest thing you can try to get coolbits enabled. if that doesnt work you'll need to try and figure out how to manually config your xorg.conf file. go here to try and learn a bit more about the process: https://foldingforum.org/...c.php?f=89&t=25075


#68
QuintLeo
SSC Member
  • Total Posts : 946
  • Reward points : 0
  • Joined: 2016/04/16 23:05:09
  • Status: offline
  • Ribbons : 3
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/22 11:43:39 (permalink)
When building a Nvidia-based system, one of the steps I always do early on is to run the following command.
 
nvidia-xconfig -a --force-generate --cool-bits=31 --mode 1280x1024
 
-a says "set up ALL Nvidia GPUs in the system"
--force-generate means "just do it, overwrite the existing xorg.conf file if one exists"
--cool-bits=31 is probably overkill, but some older Nvidia cards used different bit settings for fan control and overclock than current 10xx series cards.
the mode part is optional, and can be set to any resolution the card+display combination can support if you DO use it.
 
Fans are then controlled via the commands
nvidia-settings -a '[gpu:0]/GPUFanControlState=1'
nvidia-settings -a '[fan:0]/GPUTargetFanSpeed=80'
 
First line enables manual fan control, second line sets the speed as a percentage.
The zero in each line is "which card" you are setting, starting with "0" as the first card.
 
There is no way I know of to set fan speed individually on each fan for those cards (like some of the high-end EVGA ACX variants) that have separate control of fan speeds on each fan on a single card.
 

Now that vorsholk has stopped his abuse, I'm returning to folding.
 I no longer MOO due to abuses by certain "whales" in the Gridcoin community - so I now work the Distributed.net project directly again.
 
#69
Hughesy26
New Member
  • Total Posts : 19
  • Reward points : 0
  • Joined: 2017/03/27 05:17:39
  • Location: Wales
  • Status: offline
  • Ribbons : 0
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/22 12:12:43 (permalink)
Thanks - can't wait to get home and try this out
 
Might need to reinstall linux though...I tried using nvclock earlier and it seems to have made my display black with a grey bar at the bottom and I can't see how to recover it :/


#70
QuintLeo
SSC Member
  • Total Posts : 946
  • Reward points : 0
  • Joined: 2016/04/16 23:05:09
  • Status: offline
  • Ribbons : 3
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/22 16:52:19 (permalink)
I think you can run nvidia-xconfig from a text console, but can't remember if I've ever needed to try that.
The cards don't need to be running to run the program, unlike some of the ATI/AMD stuff that you DID have to be root to run from a text console.
 
I wonder if there would be any interest in an AMD-focused LINUX distribution for use in folding, given how poor AMD performance is in folding compared to Nvidia.
 
 
 

Now that vorsholk has stopped his abuse, I'm returning to folding.
 I no longer MOO due to abuses by certain "whales" in the Gridcoin community - so I now work the Distributed.net project directly again.
 
#71
Darth_Mouse_ALL
New Member
  • Total Posts : 2
  • Reward points : 0
  • Joined: 2018/03/16 05:33:39
  • Status: offline
  • Ribbons : 0
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/23 07:48:09 (permalink)
GPU OC settings after reboot.
This works on Linux Mint.
I never was able to get it to work by hand editing xorg.conf. What did work was to execute on the command line which sets it all up for you. This worked on two of my builds so try this. On my third build it didn't work and I just copied the file from a build where it did work, and that worked. If this doesn't work for you I'll try and get you the file that works for me:
sudo nvidia-xconfig -a --cool-bits=31 --allow-empty-initial-configuration
Then edit xorg.conf. For me that was sudo vi /etc/X11/xorg.conf and prepend "#" to each line containing AllowEmptyInitialConfiguration to comment it out. If I didn't then I got duplicate icons in my panel. I have no idea why.
Reboot.
Then to overclock run:
/usr/bin/nvidia-settings
To restore your settings after a reboot create an executable file that you call from startup applications containing the text below which will set the gpu clock offset and set the gpu to prefer maximum performance. My example sets the offset to 80. Don't set the offset too high in the file for your actual display gpu until you know for sure what you want or you may end up with a system where the display won't work. FYI the accepted wisdom is that OC of memory doesn't help PPD but in my experience it does. YMMV:
nvidia-settings -a [gpu:0]/GpuPowerMizerMode=1 # 0=adaptive, 1=max,2 or none=auto
nvidia-settings -a [gpu:0]/GPUGraphicsClockOffset[3]=80
nvidia-settings -a [gpu:0]/GPUMemoryTransferRateOffset[3]=300
#to control GPU power execute the below example for GPU 0 and 110 watts. To get the valid wattages enter something like 1000 and it will respond with the valid values for your GPU
sudo -n nvidia-smi -i 0 --persistence-mode=1
sudo -n nvidia-smi -i 0 --power-limit=110
nvidia-smi # to just display current values
#I execute the below in my script that sets my overclock settings as nvidia-settings needs to be started after the OC is set.
sleep 10
/usr/bin/nvidia-settings &
 
#72
QuintLeo
SSC Member
  • Total Posts : 946
  • Reward points : 0
  • Joined: 2016/04/16 23:05:09
  • Status: offline
  • Ribbons : 3
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/23 12:18:43 (permalink)
allow-empty-initial-configuration is NOT a good idea - that can in some cases leave you with no xorg.conf at all and therefore no fan control.
if you don't do the force-upgrade option there are times xorg.conf won't change from where it was at before and you won't have fan control.
 
 

Now that vorsholk has stopped his abuse, I'm returning to folding.
 I no longer MOO due to abuses by certain "whales" in the Gridcoin community - so I now work the Distributed.net project directly again.
 
#73
Darth_Mouse_ALL
New Member
  • Total Posts : 2
  • Reward points : 0
  • Joined: 2018/03/16 05:33:39
  • Status: offline
  • Ribbons : 0
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/24 06:04:54 (permalink)
On my system there is no xorg.conf by default and it gets created when I execute the command.  I've done it a dozen times and it works perfectly every time.  And speaking of fan control, I don't do it because the built in fan control works fine for me.  I don't want to go too low and overheat my cards or too high and wear out my fans prematurely, that being the only mechanical part and subject to mechanical wear. I believe the card designers know more about it than I do.  Plus gpu temperature varies with the project and I want my fans to adjust for that.
post edited by Darth_Mouse_ALL_1GD5nCZbh7gNo1SESPLT24xEd2Jsu4rTP9 - 2018/03/24 06:07:29
#74
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/24 08:20:05 (permalink)
that is a limit with linux is no fan curve, just a set fan speed.


#75
QuintLeo
SSC Member
  • Total Posts : 946
  • Reward points : 0
  • Joined: 2016/04/16 23:05:09
  • Status: offline
  • Ribbons : 3
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/24 11:42:14 (permalink)
Unfortunately, recent card models almost ALWAYS set the fans way too low in their "default profile" to keep the things reasonably cool.
The whole EVGA "thermal pads as a fix" issue was actually more about that, as I HAVE some of the card models that were involved in that and never had an issue with them because I don't run the fans at the insanely WIMP stock fan profile.
I'll also note that EVGA changed the bios on them in later production to INCREASE the fan speed.
 
Card manufacturers want to balance NOISE vs HEAT - and tend to come down on the side of "more quiet" in recent years as a selling point, assuming that the large majority of their cards are run in gaming which doesn't put consistant high loads on the card causing consistant high average power draw.
For compute usage, THIS IS AN ISSUE as it tends to have the cards running way too hot for good longevity, and in some cases has them into thermal throttling even at STOCK clocks if they are in a room that isn't 70 F or lower.
DO NOT TRUST THE FACTORY FAN PROFILE, it WILL cause issues as it is not designed to handle "constant high load" applications such as folding.
 
I have yet to have an EVGA fan wear out, but I do tend to run "efficient" settings and don't normally need more than about 65% fanspeed to keep the cards reasonably cool and the oldest of my EVGA cards hasn't hit 2 years old quite yet.
Fan wear is VERY non-linear - 100% fan speed causes a LOT more wear than 80%, and 80% causes a LOT more than 60% - probably more like double the wear for each 20% increase as a ballpark SWAG.
 
Just don't get me started on the CHEAP JUNK fans used by Gigabyte on their non-Aorus cards - and recently on a few of their AORUS models.
 

Now that vorsholk has stopped his abuse, I'm returning to folding.
 I no longer MOO due to abuses by certain "whales" in the Gridcoin community - so I now work the Distributed.net project directly again.
 
#76
Hughesy26
New Member
  • Total Posts : 19
  • Reward points : 0
  • Joined: 2017/03/27 05:17:39
  • Location: Wales
  • Status: offline
  • Ribbons : 0
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/26 04:27:34 (permalink)
I didn't seem to have an xorg.conf file on any of my rigs, but running allow-empty-initial-configuration created one, that I could then edit with sudo xed /etc/X11/xorg.conf
 
However this caused the same issue on 3 of my systems, where I had a black desktop and an empty grey panel. The way I fixed this was to change my HDMI cable to the 4th GPU, instead of the 1st. The desktop still doesnt seem 100% right but it works fine
 
I have, naively, been running all my fans @ 100% because noise isn't an issue and I seem to remember reading it takes years for fans to wear out. I've gain around 1.8 mil PPD (~10%) in upping my fan speed which I don't want to lose. 
The cards are currently 55-60C, whereas they were previously 83-90C. Am I right in thinking 1080tis thermal throttle at 83C? If so I'll start dialing back the fans until the temp reaches say 70C I guess.
 
 


#77
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/26 06:37:04 (permalink)
yea, GPU boost starts to throttle down once temps get into the 80's. and with winter dropping temps about to go away i would keep an eye on the temps again over the next month.


#78
QuintLeo
SSC Member
  • Total Posts : 946
  • Reward points : 0
  • Joined: 2016/04/16 23:05:09
  • Status: offline
  • Ribbons : 3
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2018/03/26 17:35:59 (permalink)
Boost actually can start throttling a bit if the card is in the mid-to-upper 70s, depending on the card - but hard throttling doesn't happen 'till well into the 80s.
Boost has several factors that go into it, where temperature is just one of the factors and usually a minor one 'till it gets high enough that GPU power consumption starts rising noticeably.
 
A good BALL BEARING fan on a GPU can last years at 100% - but will last a lot longer at 80% and delivers almost as much useable airflow.
Non-ball-bearing fans used on GPUs are often lucky to last a year at 80%, and I've had a couple die on Gigabyte cards that were run at 70% in 8 months (but Gigabyte fans are probably the worse junk in the industry right now).
 
65-70C on the cards should have very little to no effect on performance, and 70-75 might have a little effect but probably won't be noticeable.
 

Now that vorsholk has stopped his abuse, I'm returning to folding.
 I no longer MOO due to abuses by certain "whales" in the Gridcoin community - so I now work the Distributed.net project directly again.
 
#79
DrSchmidt
New Member
  • Total Posts : 44
  • Reward points : 0
  • Joined: 2019/03/16 19:36:16
  • Status: offline
  • Ribbons : 0
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2019/03/16 19:55:28 (permalink)
Great write-up, Chris.  Thanks for taking the time and for replying to everyone who requested help; it's still very useful in 2019.



Ryzen 7 3700X; RTX 2070 Super EVGA Black Gaming
GTX 1660ti MSI Armor 6G OC 6Gb
Asus AM4 Tuf Gaming Plus Wifi x570
 
#80
Chris21010
FTW Member
  • Total Posts : 1587
  • Reward points : 0
  • Joined: 2006/05/03 07:26:39
  • Status: offline
  • Ribbons : 9
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2019/03/16 21:56:00 (permalink)
awesome, glad to know the guide is still helpful.


#81
Holdolin
Superclocked Member
  • Total Posts : 226
  • Reward points : 0
  • Joined: 2013/10/09 22:01:58
  • Location: Atlantic Northeast
  • Status: offline
  • Ribbons : 1
Re: Guide to Building/Install/Setup of your own Multi-GPU Linux Folding Rig 2019/03/19 21:27:49 (permalink)
Duuude, I been searching all over the internet for weeks looking for a guide to do exactly what you posted.  As much time as I spend on these forums I have no clue how I missed it.  At any rate, thank you for taking the time to write such a detailed guide.  Yes, even today it's quite useful.  Thanks again 
#82
Page: < 123 Showing page 3 of 3
Jump to:
  • Back to Mobile