EVGA

Majestic-12 basic settings

Page: 12 > Showing page 1 of 2
Author
MOBAJOBG
iCX Member
  • Total Posts : 272
  • Reward points : 0
  • Joined: 2011/01/20 20:40:22
  • Status: offline
  • Ribbons : 3
2012/02/20 00:41:04 (permalink)
I've configured its client to release up to 1.25Mbit of my 2Mbit broadband for this project as follows;-
 
1. Tools >> Options @Connection tab
    a. Downstream: Select 2048 and make sure that the "use to be limited slider to:" is
        60%.
    b. Upstream: Select 256 and make sure that the "use to be limited slider to:" is
        100% (since my upload speed is quite slow).
2. Tools >> Options @Crawler tab
    a. Maximum number of async workers:* Select 40
    b. Maximum open URL buckets: Select 18
 
Note:
How to diagnose nodes that generate too many errors is an important read and please feel free (especially Col323 aka neo) to improve upon and better build up this implementation.
 
*I've created this thread to help my team8s to get the most out of the bandwidth that they've committed into this project and also, for the effort they've undertaken to join it.
 
post edited by MOBAJOBG - 2012/02/20 01:03:41

EVGA GeForce GTX 580 CoD: Black Ops edition

                                            
 
#1

36 Replies Related Threads

    duboisi
    Superclocked Member
    • Total Posts : 195
    • Reward points : 0
    • Joined: 2011/11/29 20:57:04
    • Status: offline
    • Ribbons : 1
    Re:Majestic-12 basic settings 2012/02/20 05:47:33 (permalink)
    Note: Changing #2.a. needs a restart for changes to be effective ... File -> restart
    Do we need to allocate/spare a thread to crunch M-12?

      
     
    225*2^1417568+1 (My prime)   
        
    #2
    MOBAJOBG
    iCX Member
    • Total Posts : 272
    • Reward points : 0
    • Joined: 2011/01/20 20:40:22
    • Status: offline
    • Ribbons : 3
    Re:Majestic-12 basic settings 2012/02/20 06:55:58 (permalink)
    For your information based on my own observation, it's not necessary to free up a thread for this project as the CPU usage is a minimal 0.14% only and also, it doesn't have any noticeable impact to other CPU based projects that are running concurrently. Luckily, there is NO QRB bonus applied unlike Folding@home smp & bigbeta WUs.
     
    [1st edit]
    As a matter of fact, the i7 990X processor with HT off is performing 2 first pass & 1 double check Prime Sierpinski tasks (3 physical cores), Enigma@Home (1 physical core), RNA World (1 physical core), Leiden Classical (1 physical core), Majestic-12 and finally, Collatz Conjecture (1 gpu) now.
     
    [2nd edit] 
    It's interesting to know that the i7 990X processor with HT off can also be used to perform 2 first pass & 1 double check Prime Sierpinski tasks (3 physical cores), MOUN1 (set to auto physical core selection in config.txt), Majestic-12 and finally, RC5-72 (1 physical core + 1 gpu) with minimal performance penalty.
    post edited by MOBAJOBG - 2012/02/20 07:45:44

    EVGA GeForce GTX 580 CoD: Black Ops edition

                                                
     
    #3
    Col323
    Superclocked Member
    • Total Posts : 115
    • Reward points : 0
    • Joined: 2011/11/28 09:07:30
    • Status: offline
    • Ribbons : 2
    Re:Majestic-12 basic settings 2012/02/20 07:15:32 (permalink)
    MOBAJOBG is correct.  The CPU is really only used during archiving/compressing of completed buckets before they are sent back to the server.  Using a low bandwidth setup like the one in the first post (by the way, those are pretty much my settings except for the schedule of when to let it run, which I can detail later if necessary), whatever core does the archiving should be mostly free for other crunching.  Now, when you start loading up a 40 Mb line, there will be a core busier with more archiving.

    None of us is as strong as all of us.

      
     


     

      

     
    #4
    duboisi
    Superclocked Member
    • Total Posts : 195
    • Reward points : 0
    • Joined: 2011/11/29 20:57:04
    • Status: offline
    • Ribbons : 1
    Re:Majestic-12 basic settings 2012/02/21 04:05:17 (permalink)
    My anti virus Avast hung and then keyboard/mouse have no response. Could be due to flooding of "Malicious URL blocked" messages that hung Avast.
    Reducing bandwidth from 2.5mb to 2mb and monitoring.

      
     
    225*2^1417568+1 (My prime)   
        
    #5
    MOBAJOBG
    iCX Member
    • Total Posts : 272
    • Reward points : 0
    • Joined: 2011/01/20 20:40:22
    • Status: offline
    • Ribbons : 3
    Re:Majestic-12 basic settings 2012/02/21 07:23:50 (permalink)
    In addition, please remember to activate your preferred software firewall.

    EVGA GeForce GTX 580 CoD: Black Ops edition

                                                
     
    #6
    duboisi
    Superclocked Member
    • Total Posts : 195
    • Reward points : 0
    • Joined: 2011/11/29 20:57:04
    • Status: offline
    • Ribbons : 1
    Re:Majestic-12 basic settings 2012/02/21 22:35:43 (permalink)
    Sorry guys, I will only crunch MJ-12 while I am attending to my PC. Last 12 hr encountered 2 problem:
    a) download of WCG WUs were hung (message no internet connection) but upload/report was ok, rebooted modem/router solved the problem. 
    b) AVAST was deactivate, failed to reactivate, only solved with a reboot.
    ------------------------------------------------------------------------------------------
    Edit: Got to stop crunching MJ-12, AVAST can't last for half hour before disable itself.
    post edited by duboisi - 2012/02/22 04:41:42

      
     
    225*2^1417568+1 (My prime)   
        
    #7
    Col323
    Superclocked Member
    • Total Posts : 115
    • Reward points : 0
    • Joined: 2011/11/28 09:07:30
    • Status: offline
    • Ribbons : 2
    Re:Majestic-12 basic settings 2012/02/24 06:02:21 (permalink)
    For those not wanting to fuss with setting it up yourself, there's a guy , refic who has been with the project for nearly 7 years who is renting nodes.  All you have to do is provide him with your username so he can configure it (and provide him with the monthly fee ).  He says the nodes average about 12 Mbps.  One person who rented a node this past month reported 182,967,119 URLs and 4.75TB of crawling from that node.  That one node in 30 days did enough to be ranked in the top 60 teams.

    None of us is as strong as all of us.

      
     


     

      

     
    #8
    MOBAJOBG
    iCX Member
    • Total Posts : 272
    • Reward points : 0
    • Joined: 2011/01/20 20:40:22
    • Status: offline
    • Ribbons : 3
    Re:Majestic-12 basic settings 2012/03/01 14:58:17 (permalink)
    It seems that our team is missing from this project somehow and "Last updated: 17 Jun 2011 08:00:07:420 GMT" can be seen at the bottom of the stats page now.
     
    [Edited 2hrs later]
    Nevermind, it's normal now. Sorry for wasting 2-3mins of your time. "Last updated: 02 Mar 2012 01:00:09:597 GMT"
    post edited by MOBAJOBG - 2012/03/01 17:12:28

    EVGA GeForce GTX 580 CoD: Black Ops edition

                                                
     
    #9
    Col323
    Superclocked Member
    • Total Posts : 115
    • Reward points : 0
    • Joined: 2011/11/28 09:07:30
    • Status: offline
    • Ribbons : 2
    Re:Majestic-12 basic settings 2012/03/15 08:18:16 (permalink)
    I took my own advice (seen two posts up - don't make me hotlink it ) and have rented a couple nodes from refic.
     
    Keep an eye on the team's output here and here.
     
    See the impact it has on my daily output here.
     
    Hopefully this scores us a lot of points in the Vault and gets us respectable in the project overall.  

    None of us is as strong as all of us.

      
     


     

      

     
    #10
    Opolis
    CLASSIFIED Member
    • Total Posts : 2778
    • Reward points : 0
    • Joined: 2010/03/25 18:49:16
    • Status: offline
    • Ribbons : 7
    Re:Majestic-12 basic settings 2012/03/15 10:03:53 (permalink)
    Thinking about doing some occasional MJ-12 since I do have a 45/35Mbps fiber connection.  How much CPU usage are we talking if I use the majority of my bandwith?  Would it be better to use something lower, like 15Mbps, if I am crunching on the CPU along with MJ-12?  How many workers would I use for those two options?  I am also kind of reserved about downloading possible pron pages or pages that contain viruses, even though I read on the MJ-12 site that it shouldn't be dangerous.
    post edited by Opolis - 2012/03/15 10:05:33

    #11
    Col323
    Superclocked Member
    • Total Posts : 115
    • Reward points : 0
    • Joined: 2011/11/28 09:07:30
    • Status: offline
    • Ribbons : 2
    Re:Majestic-12 basic settings 2012/03/16 05:01:05 (permalink)
    I'd say it would be better to use a lower percentage first.  Take my response with a large grain of salt, as I've never had that much bandwidth to play with on my own.  But if you're running the client on a Windows box, I think you'd be hard pressed to have the CPU keep up with your crawling.  The Windows client is not multi-threaded, so you only have 1 core to do all the archiving of everything you've crawled.  You can set up more than one client, but since you're dealing with Windows and the TCP/IP stack, that means a virtual machines as well.  Linux is a different story, but that's really straying from any sort of experience or reading I've done.
     
    Also, it's important to note that your modem and router play an important role in crawling.   (Thank you, Captain Obvious!)  A lot of consumer grade hardware pukes when hit with too many requests, leaving your client unable to do real work or even fetch/flush work.
     
    My suggestion is to start out trying to use 5-10% of your bandwidth.   The easy part of that is to set your downstream and upstream in the client's Connection tab, and then set the slider for "use to be limited to" at 10%.  The more experimental part is figuring out how many Buckets and Workers you need to set in the Crawler tab to maximize your working while getting good results (ie, not choking your connection).  I aim for an overall success rate of about 83%.  (Found on the Status tab.)
     
    So let's say you're aiming for roughly 10% of your connection.  I would 2.5x the numbers for Buckets and Workers  MOBAJOBG suggested in the OP.  See what your success rate is.  I always try to keep a Workers to Buckets ratio of 2:1 - 3:1.  If you're cruising along with a high success rate, then allow more buckets and workers to get the most efficient setup.  Once you have that sorted, you can scale up the bandwidth usage, increasing the workers and buckets appropriately.
     
    It may sound tedious or complicated at first, but it actually is pretty easy and a quick process.  And as the project director himself says, a node crawling a slow pace for a long time is better than a large crawl in a couple days.  It all adds up over time, and obviously the Internet is an ever-changing beast.
     
    As for downloading and executing malware, I've been following the project for over 5 years and running it for most of that time.  There's never been an outbreak of system issues due to this.  The crawler doesn't execute code found in webpages it crawls, it just looks for backlinks inside an isolated object.  Alex (the project director) has a pretty good explanation of why it's safe....if I could find it.
     
    Sorry for the long post.  

    None of us is as strong as all of us.

      
     


     

      

     
    #12
    planetclown
    FTW Member
    • Total Posts : 1690
    • Reward points : 0
    • Joined: 2010/03/02 07:59:51
    • Status: offline
    • Ribbons : 15
    Re:Majestic-12 basic settings 2012/03/16 05:50:24 (permalink)
    First of all, very good info provided already.
     
    After poking around the MJ12 forums, it seems like a rule-of-thumb for an initial setting for worker threads is 15 workers per 1 Mbps available. 
     
    As an example, I have 4 Mbps available.  So 4 (Mbps) * 15 (workers) = 60 workers. Then based on above suggestions, between 20 and 30 buckets.
     
    Then you can increase/decrease threads and buckets depending on success rate, connection issues (see link in OP), and log messages.
    post edited by planetclown - 2012/03/16 05:52:48

    EVGA Associates code: OLY2307BPM0MXZU Click for a discount on your next purchase from EVGA.com


    #13
    MOBAJOBG
    iCX Member
    • Total Posts : 272
    • Reward points : 0
    • Joined: 2011/01/20 20:40:22
    • Status: offline
    • Ribbons : 3
    Re:Majestic-12 basic settings 2012/03/16 08:33:45 (permalink)
    Great discussion!, team8s ...that's what I really like about this place. We turn up to offer valuable bits of information or make a solid effort to help each other.
    Opolis

    Thinking about doing some occasional MJ-12 since I do have a 45/35Mbps fiber connection.  How much CPU usage are we talking if I use the majority of my bandwith?  Would it be better to use something lower, like 15Mbps, if I am crunching on the CPU along with MJ-12?  How many workers would I use for those two options?  I am also kind of reserved about downloading possible pron pages or pages that contain viruses, even though I read on the MJ-12 site that it shouldn't be dangerous.

    Perhaps, "Maximum number of async workers:*" = 225 and "Maximum open URL buckets" = 95 to 100 is my rough estimate. As Col323 has pointed out, the MJ12node client can saturate just one CPU core only.
     
    It's not advisable to allocate more than 16Mbps bandwidth connection per client installation. Furthermore, it's preferable to setup virtual machine(s) assigned with 1 logical core, 2GB RAM and a max of 16Mbps bandwidth connection to work with more than 1 client installation.
    post edited by MOBAJOBG - 2012/03/16 09:17:28

    EVGA GeForce GTX 580 CoD: Black Ops edition

                                                
     
    #14
    MOBAJOBG
    iCX Member
    • Total Posts : 272
    • Reward points : 0
    • Joined: 2011/01/20 20:40:22
    • Status: offline
    • Ribbons : 3
    Re:Majestic-12 basic settings 2012/03/16 08:40:31 (permalink)
    Col323

    I took my own advice (seen two posts up - don't make me hotlink it ) and have rented a couple nodes from refic.

    Keep an eye on the team's output here and here.

    See the impact it has on my daily output here.

    Hopefully this scores us a lot of points in the Vault and gets us respectable in the project overall.  

    Wow!, we can dream and also, afford to think big now. Col323, thanks for your generosity and exemplary commitment. It seems that this project is gaining some significant forward momentum.
     
    Before I forget, I wish to register my sincere appreciation to our other team8s as well. So, to kawika57, duboisi, RHMash, Punchy, planetclown and also, everybody that will join us soon enough or later on ...superThank$U.
     
    post edited by MOBAJOBG - 2012/03/16 09:16:23

    EVGA GeForce GTX 580 CoD: Black Ops edition

                                                
     
    #15
    Opolis
    CLASSIFIED Member
    • Total Posts : 2778
    • Reward points : 0
    • Joined: 2010/03/25 18:49:16
    • Status: offline
    • Ribbons : 7
    Re:Majestic-12 basic settings 2012/03/16 10:18:25 (permalink)
    Thank you for the replies, all very good info!  When I have some time I will try a small amount of bandwidth and see how it goes, slowly working my way up.  I am on Windows 7, generic Verizon FiOS modem/router combo, and don't have much VM experience but I do like to learn.   Setting this up sounds like overclocking (which I find enjoyable).
     
    I was mostly looking for some sort of baseline or general settings to start from and compare to, so thanks guys!
    post edited by Opolis - 2012/03/16 10:21:59

    #16
    MOBAJOBG
    iCX Member
    • Total Posts : 272
    • Reward points : 0
    • Joined: 2011/01/20 20:40:22
    • Status: offline
    • Ribbons : 3
    Re:Majestic-12 basic settings 2012/03/16 20:21:02 (permalink)
    I've been fiddling with the settings every few days and currently, are as follow;-
     
    1. a. Downstream: 2048 and the "use to be limited slider to:" is 80%.
        b. Upstream: 256 and the "use to be limited slider to:" is 80%.
    2. Tools >> Options @Crawler tab
        a. Maximum number of async workers:* 50
        b. Maximum open URL buckets: 20
     
    The client has managed to achieve an "overall downloading rate" of 1350Kbps and a "crawled successes" of 78.5%. I'm quite satisfied with its progress but will tweak further slightly to obtain 80% if possible.

     
    post edited by MOBAJOBG - 2012/03/16 20:24:49

    EVGA GeForce GTX 580 CoD: Black Ops edition

                                                
     
    #17
    MOBAJOBG
    iCX Member
    • Total Posts : 272
    • Reward points : 0
    • Joined: 2011/01/20 20:40:22
    • Status: offline
    • Ribbons : 3
    Re:Majestic-12 basic settings 2012/03/17 22:13:39 (permalink)
    I use a cable to connect the spot between the back of my home computer and modem + wireless router that is within a distance in length of 1 meter yesterday. The outcome has been encouraging with an increase of "crawled successes" to 83.2% and also, "overall downloading rate" to 1430Kbps thereby recording a gain of 4.7% and 80Kbps respectively.
    post edited by MOBAJOBG - 2012/03/17 22:45:54

    EVGA GeForce GTX 580 CoD: Black Ops edition

                                                
     
    #18
    MOBAJOBG
    iCX Member
    • Total Posts : 272
    • Reward points : 0
    • Joined: 2011/01/20 20:40:22
    • Status: offline
    • Ribbons : 3
    Re:Majestic-12 basic settings 2012/03/17 23:13:23 (permalink)
    Col323

    I took my own advice (seen two posts up - don't make me hotlink it ) and have rented a couple nodes from refic.

    Keep an eye on the team's output here and here.

    See the impact it has on my daily output here.

    Hopefully this scores us a lot of points in the Vault and gets us respectable in the project overall.  

    Do you or refic need to tinker with any of the 18Mbps to 20Mbps nodes settings to obtain a "crawled successes" of ~85%? Did every node manage to crawl ~7 million URLs per node daily?
     
    The performance churned out is nothing short of extremely remarkable.
    post edited by MOBAJOBG - 2012/03/17 23:26:32

    EVGA GeForce GTX 580 CoD: Black Ops edition

                                                
     
    #19
    Col323
    Superclocked Member
    • Total Posts : 115
    • Reward points : 0
    • Joined: 2011/11/28 09:07:30
    • Status: offline
    • Ribbons : 2
    Re:Majestic-12 basic settings 2012/03/19 09:34:10 (permalink)
    MOBAJOBG
    Before I forget, I wish to register my sincere appreciation to our other team8s as well. So, to kawika57, duboisi, RHMash, Punchy, planetclown and also, everybody that will join us soon enough or later on ...superThank$U.  

    Absolutely!  It's great to see an active team.  Small contributions add up in time and help the climb up the ranks!  And there's always room for more to contribute as they can.
     
    MOBAJOBG, I didn't have to do any fiddling with my settings to get this output.  refic has a lot of experience with high speed nodes, so he install and configured these  nodes then gave me the login for each so I could input my user info and keep an eye on them.  I'm sure he has a pretty good idea of default settings to get the output we're seeing.  Overall it was a really simple (and relatively cheap) experience.  I really wanted to try it since my first plan to boost the team fell flat on its face.

    None of us is as strong as all of us.

      
     


     

      

     
    #20
    Opolis
    CLASSIFIED Member
    • Total Posts : 2778
    • Reward points : 0
    • Joined: 2010/03/25 18:49:16
    • Status: offline
    • Ribbons : 7
    Re:Majestic-12 basic settings 2012/03/19 13:11:07 (permalink)
    I finally joined the team!  I am using 7Mbps and playing around with workers/buckets to lower the error rate.  How do the stats work?  I assume after awhile archiving will take place and then results will be uploaded.
     
    EDIT: Nevermind, I already see my stats showing up.
     
    EDIT#2:  Wow do I feel dumb.  I was getting lots of errors and then I noticed that the "*" stood for "restart required to take effect"  DOH!  Got it around 75-80% success rate now.
    post edited by Opolis - 2012/03/19 13:25:35

    #21
    Col323
    Superclocked Member
    • Total Posts : 115
    • Reward points : 0
    • Joined: 2011/11/28 09:07:30
    • Status: offline
    • Ribbons : 2
    Re:Majestic-12 basic settings 2012/03/20 08:22:13 (permalink)
    I see you on board.  Welcome!  That 7Mbs will be a huge help!  Your error rate sounds fine.  I've noticed some low quality buckets lately, so don't be alarmed if your success rate fluctuates.
     
    I can't wait to see a full day of your production!  

    None of us is as strong as all of us.

      
     


     

      

     
    #22
    Opolis
    CLASSIFIED Member
    • Total Posts : 2778
    • Reward points : 0
    • Joined: 2010/03/25 18:49:16
    • Status: offline
    • Ribbons : 7
    Re:Majestic-12 basic settings 2012/03/20 09:56:44 (permalink)
    Yeah I'm still playing around with it, I noticed some buckets had a much higher error rate than others.  Currently it has settled to ~75% success.  I might try 10Mbps, just don't want the CPU usage to put too big of a dent in the folding that PC is doing.  I am loving Verizon just a little bit more; I don't notice any slowdown or much deterioration in gaming on my other rig connected to my network.  In fact, I ran a speed test on it and it recorded higher speeds than I pay for (regular practice for FiOS).

    #23
    duboisi
    Superclocked Member
    • Total Posts : 195
    • Reward points : 0
    • Joined: 2011/11/29 20:57:04
    • Status: offline
    • Ribbons : 1
    Re:Majestic-12 basic settings 2012/03/20 22:52:12 (permalink)
    For those interested, you can build a GIF stats siggy from dcstat.com for MJ-12 and DIMES.

      
     
    225*2^1417568+1 (My prime)   
        
    #24
    Opolis
    CLASSIFIED Member
    • Total Posts : 2778
    • Reward points : 0
    • Joined: 2010/03/25 18:49:16
    • Status: offline
    • Ribbons : 7
    Re:Majestic-12 basic settings 2012/03/21 00:02:09 (permalink)
    Very nice!  I was wondering if there was a way to get stats in your sig for the non-boinc projects.

    #25
    Col323
    Superclocked Member
    • Total Posts : 115
    • Reward points : 0
    • Joined: 2011/11/28 09:07:30
    • Status: offline
    • Ribbons : 2
    Re:Majestic-12 basic settings 2012/03/22 06:59:31 (permalink)
    I'm not a sig guy (witness the vast whitespace beneath my posts ), but Free-DC has a tool to generate one see the first 5 questions.  That will get you something which looks like rilian's shown here:  http://stats.free-dc.org/tag.php?name=rilian&img=.png
     

    I believe you can customize the appearance.
     
    Nice output from your Verizon line so far, and from the team in general!  We're approaching 10 users, and every person signed up helps carry the load and get us to the top!
    post edited by Col323 - 2012/03/22 07:03:19

    None of us is as strong as all of us.

      
     


     

      

     
    #26
    moviemanxs
    New Member
    • Total Posts : 37
    • Reward points : 0
    • Joined: 2009/11/24 04:18:08
    • Status: offline
    • Ribbons : 0
    Re:Majestic-12 basic settings 2012/03/23 22:09:05 (permalink)
    Gemtlemen: I just noticed you guys at MJ12. Thanks for helping out.
    I've been on it since may 2006 and if I can be of any help just yell, better a PM over there as it goes right to my email.
    One small suggestion; When possible take a look and make sure everything is going well as it is possible to get "bad buckets: that can literally stall a connection. A simple restart will usually take care of the issue.
    Once again, Thanks for helping and good luck in your Vault Quest!
    Dave
    Movieman from Xtremesystems
    #27
    Col323
    Superclocked Member
    • Total Posts : 115
    • Reward points : 0
    • Joined: 2011/11/28 09:07:30
    • Status: offline
    • Ribbons : 2
    Re:Majestic-12 basic settings 2012/03/29 06:00:41 (permalink)
    Thanks for the shout, MM.  You're a class act the way you root on other teams in their DC efforts.
     
    Nice crawling, Opolis!  You broke through the 2 Million barrier yesterday.  How is your setup treating you?  

    None of us is as strong as all of us.

      
     


     

      

     
    #28
    Opolis
    CLASSIFIED Member
    • Total Posts : 2778
    • Reward points : 0
    • Joined: 2010/03/25 18:49:16
    • Status: offline
    • Ribbons : 7
    Re:Majestic-12 basic settings 2012/03/29 12:32:01 (permalink)
    MJ-12 has been running very well, Col323.  I have moved up to 10Mbps with a current success rate of ~ 72% and it doesn't seem to be impacting folding on that rig much at all.
     
    Our buddy neo is really flying through those URLs!  Do we know what kind of setup he is using?
    post edited by Opolis - 2012/03/29 12:33:38

    #29
    planetclown
    FTW Member
    • Total Posts : 1690
    • Reward points : 0
    • Joined: 2010/03/02 07:59:51
    • Status: offline
    • Ribbons : 15
    Re:Majestic-12 basic settings 2012/03/29 17:20:13 (permalink)
    It appears Col323 is neo.
    Col323

    I took my own advice [...] and have rented a couple nodes from refic.

    [...]

    See the impact it has on my daily output here.



    EVGA Associates code: OLY2307BPM0MXZU Click for a discount on your next purchase from EVGA.com


    #30
    Page: 12 > Showing page 1 of 2
    Jump to:
  • Back to Mobile