Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

While the push-back against Map/Reduce & "Big Data" in general is totally valid, it's important to put it into context.

In 2007-2010, when Hadoop first started to gain momentum it was very useful because disk sizes were smaller, 64 bit machines weren't ubiquitous and (perhaps most importantly) SSDs were insanely expensive for anything more than tiny amounts of data.

That meant if you had more than a couple of terabytes of data you either invested in a SAN, or you started looking at ways to split your data across multiple machines.

HDFS grew out of those constraints, and once you have data distributed like that, with each machine having a decently powerful CPU as well, Map/Reduce is a sensible way of dealing with it.



It still is a valid pattern. As disks get bigger, the harder it is to move the data and the more efficient it is to move the compute to the data.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: