Re: Opolis, you look lonely on Majestic-12
2016/09/29 05:04:05
(permalink)
7,000,000,000 URLs crawled as a team! Next milestone is 200 TB.
I admire your determination. (And home connection.
) We've always had a slow 2-4 Mbs connection with a 150 GB cap. (The bulk of my crawling came through nodes I rented from one of the project gurus.) When the shares transition happened, it made no sense for me to keep crawling. But now, I have a slightly faster 12 Mbs connection with a 1TB cap. I also have some extra small Linux virtual machines from Azure. Their bandwidth is crazy, but the CPU can barely keep up with archiving the completed buckets. My production on each on varies quite a bit day to day. But, it all adds up and helps us move forward.
We must be on a similar wavelength. I was also recently thinking about the muon standalone when I decommissioned a cruncher I used in our efforts there. I think I have a box with it still installed, but that machine does not stay on very long these days.
None of us is as strong as all of us.