[MapReduce-user] is HDFS RAID "data locality" efficient?
Aug 8, 2012 at 5:30 pm
: Indeed, erasure encoding is a component of a good storage solution esp. for holding on to PB scale datasets but there's an associated cost in terms of latency for real time serving. Depending on the domain (eg. where temporal locality is observed in access patterns), it works well if the hot dataset is small and can be served efficiently from elsewhere. It is a great fit for DW type workloads. Fb had a good presentation sometime back where they discussed a typical impl with Reed Solomon codes
D'Souza, Clive V
: Adding to Gaurav’s sentiment - using object stores with Erasure code is pretty good solution when the data starts creeping into the PB scale with a need for redundancy. Look at Amplidata solutions, they seem to have good stack. Regards, -C From: Gaurav Sharma Sent: Wednesday, August 08, 2012 10:25 AM To: [email protected] Subject: Re: is HDFS RAID "data locality" efficient? Indeed, erasure encoding is a component of a good storage solution esp. for holding on to PB scale datasets but there's an
: exactly: less space use on cold data, with the penalty that access performance can be worse. As the majority of data on a hadoop cluster is usually "cold", it's a space and power efficient story for the archive data -- Steve Loughran Hortonworks Inc
How can I know how many mappers created?
Query over efficient utilization of cluster using fair scheduling
Efficient sort -u + merge, in Hadoop M/R?
Lack of data locality in Hadoop-0.20.2
Improve data locality for MR job processing tar.gz files
Bulk Import & Data Locality
How to use CombineFileInputFormat in Hadoop?
Do we shoot ourselves by using all task slots?
Locks in M/R framework
Loading data from S3
4 of 16
Aug 8, '12 at 4:46p
Aug 9, '12 at 10:56a
14 users in discussion
Sourygna Luangsay (2)
Michael Segel (2)
Vinicius Melo (1)
Gabriel Armelin (1)
Mayuran Yogarajah (1)
Steve Loughran (1)
Gaurav Sharma (1)
Avram Aelony (1)
D'Souza, Clive V (1)
Ajit Ratnaparkhi (1)
Miles Trebilco (1)
Arun Prakash (1)
Groups & Organizations
site design / logo © 2021 Grokbase