Today i hit some new milestones in Distributed Computing, which i have been dabbling in for the last few years on and off on a part time basis depending on how my electricity and bandwidth i can allocate.
Majestic-12 is working towards creation of a World Wide Web search engine based on concepts of distributing workload in a similar fashion achieved by successful projects such as SETI@home and distributed.net. You can read more details on the project here. Below you can see recent news highlights, but for full details its best to check our forum.
I'm pretty impressed to have managed to crawl this many URL's in the last few years. You'd never be able to do this kind of web surfing normally if you tried. My target to reach at the moment is about 1,000,000,000 urls (1 billion urls)
I'm averaging about 1.2 - 1.5 million urls a day at present so it will probably take me another 1 1/2 years to do this. Thankfully I have the time and the computing resources available.
If you want to find out more or join the project you can download the software (the crawling bot) and get help from their website at http://www.majestic12.co.uk/
I joined this project recently and found out about it from AMD Users. Since a child i've always had a deep love for anything to do with Astronomy and wished I'd been able to pursue more of an interest in it when i was younger. Unfortunately due to the fact that I live in the suburbs of a large city, i tend to find that star gazing is nowhere even remotely near the beauty of the stars and the universe when you're somewhere truly dark and unspoilt by city lights. Maybe one day i will get my dream. Anyway back to the project, it's better explained by this segment from their About Page and this video:
Have a computer? Want to help astronomers make awesome discoveries and understand our Universe? theSkyNet needs you!
Your computer is bored. It has spare processing power nearly all of the time that could be used to do something cool. So why not let it?
The Science Bit
By connecting 100s and 1000s of computers together through the Internet, it’s possible to simulate a single machine capable of doing some pretty amazing stuff.
theSkyNet is a community computing project dedicated to radio astronomy. Radio astronomers use radio telescopes (of course) to observe the Universe at radio wavelengths (still with me?). All day, every day, signals from distant galaxies, stars and other cosmic bits and pieces arrive at the Earth in the form of radio waves. Once detected by a radio telescope the signal is processed by computers and used by scientists to support a theory or inspire a new one.
The Computer Bit
When you join theSkyNet your computer will help radio astronomers process information and answer some of the big questions we have about the Universe.
As a part of theSkyNet community your computer will be called upon to process small packets of data, but you wont even notice it’s going on. The key to theSkyNet is to have lots of computers connected, with each doing only a little, but it all adding up to a lot.
At the heart of theSkyNet is this website, theSkyNet.org where you’ll find alliances you’ve joined stack up against others. The more data you and your alliances process, the more status you’ll have within theSkyNet community.
But that’s not it, because as theSkyNet project evolves we’ll be adding more features for you to explore. In the pipeline we have visualisation tools to help you understand the data you’re processing and even an opportunity to help identify and catalogue radio wave sources.
The Guts of theSkyNet
theSkyNet is powered by some clever software called Nereus. Data collected by one of several radio telescopes is sent to your computer as a small data packet ready for processing by the Nereus client. Once processing of the data packet has taken place it is sent back and the process begins all over again. By repeating this process across thousands of computers, it is possible to simulate a single powerful machine capable of doing real and relevant scientific research.
You can join The SkyNet and learn more by visiting their website at http://www.theskynet.org/index. Happy Analysing!