[Hadoop-common-user] is HDFS RAID "data locality" efficient?
Aug 8, 2012 at 5:30 pm
: Indeed, erasure encoding is a component of a good storage solution esp. for holding on to PB scale datasets but there's an associated cost in terms of latency for real time serving. Depending on the domain (eg. where temporal locality is observed in access patterns), it works well if the hot dataset is small and can be served efficiently from elsewhere. It is a great fit for DW type workloads. Fb had a good presentation sometime back where they discussed a typical impl with Reed Solomon codes
D'Souza, Clive V
: Adding to Gaurav’s sentiment - using object stores with Erasure code is pretty good solution when the data starts creeping into the PB scale with a need for redundancy. Look at Amplidata solutions, they seem to have good stack. Regards, -C From: Gaurav Sharma Sent: Wednesday, August 08, 2012 10:25 AM To: [email protected] Subject: Re: is HDFS RAID "data locality" efficient? Indeed, erasure encoding is a component of a good storage solution esp. for holding on to PB scale datasets but there's an
: exactly: less space use on cold data, with the penalty that access performance can be worse. As the majority of data on a hadoop cluster is usually "cold", it's a space and power efficient story for the archive data -- Steve Loughran Hortonworks Inc
What is the most efficient way to copy a large number of .gz files into HDFS?
Mapper reading from local directory or global variable?
How to read mapreduce output in HDFS directory from Web Application
hdfs output for both mapper and reducer
How to move files from one location to another on hadoop
Handling of small files in hadoop
how to get all different values for each key
TableOutputFormat not efficient than direct HBase API calls?
Checkpoint vs Backup Node
4 of 17
Aug 8, '12 at 4:46p
Aug 9, '12 at 10:56a
15 users in discussion
Michael Segel (2)
Sourygna Luangsay (2)
Gaurav Sharma (1)
Vinicius Melo (1)
Avram Aelony (1)
Gabriel Armelin (1)
Mayuran Yogarajah (1)
D'Souza, Clive V (1)
Ajit Ratnaparkhi (1)
Steve Loughran (1)
Miles Trebilco (1)
Arun Prakash (1)
Groups & Organizations
site design / logo © 2021 Grokbase