Monday, December 01, 2008

1TB in 68 Seconds - New Google Data Sorting Record

Google has recently announced new strides in its race to process the world's information - they've been able to sort 1 terabyte of data across 1 thousand computers in a only 68 seconds. The previous record of the the same data (1TB) was sorted across 910 computers and took 209 seconds. For one petabyte of data, it took Google 6 hours and 2 minutes to do the sorting, and the sorted data gets moved to 48,000 hard drives!

Google also acknowledges that they had triple redundancy and the data was backed up on 3 hard drives. Could it be that if you reduced that redundancy, this would all be much quicker? Not necessarily, since storing may not be part of the sorting algorithm that was recorded (and storing the data could have still been ongoing after the sort had been completed).

What's next? I guess the exabyte would be a huge step towards instant live data sorting...

No comments: