[Pig-user] Read Hive LazySimpleSerde with Pig

[Pig-user] Read Hive LazySimpleSerde with Pig

	Dmitriy Ryaboy
	Mar 12, 2013 at 9:53 pm

How does LazySimpleSerde store data?

On Tue, Mar 12, 2013 at 11:17 AM, Shawn Hermans wrote:

All,
Is there an easy way to read Hive LazySimpleSerde encoded files in Pig? I
did some research and found support for Hive's columnar format and for
SequenceFiles, but did not see anything for LazySimpleSerde.

Thanks,
Shawn

Search Discussions

Discussion Posts

Shawn Hermans: All, Is there an easy way to read Hive LazySimpleSerde encoded files in Pig? I did some research and found support for Hive's columnar format and for SequenceFiles, but did not see anything for LazySimpleSerde. Thanks, Shawn

Follow ups

Shawn Hermans: It uses ^A for record separator. That would be easy enough as I could just use PigStorage("\001") to pull in the records. The only issue is how to extract maps. It uses ^C to separate entires within the map and ^B to separate key/value pairs in the map. It wouldn't be too difficult to write a UDF to parse the map entries, I was just wondering if there was a built-in way of doing that. Thanks, Shawn
Shawn Hermans: Solved the issue with a Jython UDF. REGISTER 'lazysimpleserde.py' USING jython AS myfuncs; A = LOAD '000000_0' using PigStorage('\\u001') AS (params:chararray); B = FOREACH pixels GENERATE myfuncs.extractMap(params); @outputSchema("params:map[]") def extractMap(lazy_map): extracted = {} entries = lazy_map.split('\x02') for entry in entries: split_entry = entry.split('\x03') if len(split_entry) == 2: extracted[split_entry[0]] = split_entry[1] return extracted

Related Discussions

How can I read Hive text files on S3 from Pig?
fresher in hadoop
Group by with count
Partitions in pig
JOIN comparasion PIG V/S HIVE
PIG vs HIVE
NEED HELP in Hive Query
Optimizations in pig
Small question in pig
Small question

Discussion Navigation

view	thread \| post
posts	‹ prev \| 2 of 4 \| next ›

Discussion Overview

group	user @
categories	pig, hadoop
posted	Mar 12, '13 at 6:17p
active	Mar 13, '13 at 3:39p
posts	4
users	2
website	pig.apache.org

font	variable	Hotkey: f
user style	avatars	Hotkey: a

[Pig-user] Read Hive LazySimpleSerde with Pig

Search Discussions

Discussion Posts

Related Discussions

2 users in discussion