Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Imagine if storage limitations weren't holding back the NSA.

Those 60TB density HAMR[1] drives that are due in 2016 are really going to take invasive to a whole new level.

[1] http://storageeffect.media.seagate.com/files/2012/03/perpham...



Imagine if the American people weren't holding back the NSA.

Oh, wait...


20 TB per day is nothing these days.


Who said that they were using spinning storage? You can store an insane amount of data for your dollar if you are willing t use tape.


For example, IBM have a robot tape library that can store 900PB[1], Quantum can fit ~5PB per full rack equiv[2], and there are many many more.

[1] http://www-03.ibm.com/systems/storage/tape/ts3500/index.html

[2] http://www.quantum.com/products/tapelibraries/scalari6000/in...


mmh... but maybe the military is also ahead in that area? Maybe DARPA has 1TB GB/s drives?


I don't quite know what you intend "1 TB GB/s drives" to mean.

But note that you can buy off the shelf PCIe cards with SSD's mounted that will give you 1TB storage and an aggregate read bandwidth of more than 1GB/sec today. I've got three sitting in various servers. They're expensive, and frankly for the future I'll rather get a couple of extra SATA III controllers and get multiple "regular" SSDs on separate controllers for that reason, but they're available.

For NSA style data collection, though, the collection is trivially to do in parallel: Hash all keys to a "virtual bucket", and hold a map of virtual buckets to physical servers. Then when you want more capacity, you add some physical servers, reassigns some of the virtual buckets from other physical servers to the new ones, and synchronises any old data (given that NSA claims they could only hold the full data stream for three days, you don't even need the hassle of moving data, just make collection on different days map to different virtual buckets, so that on day one you "just" reassign virtual buckets the content of which is being expired on the old servers anyway, on day two, the next set etc. - you maintain full spread of read/write traffic by ensuring that in normal operation all servers have an even spread of "day 1", "day 2" and "day 3" buckets).

It's amusing they see storage as an issue, but of course this was in 2008. Today I have 6TB in my home NAS, and my perfectly off the shelf tower case can easily fit 40TB+ with current size harddisks (though I doubt the noise would make me popular at home).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: