[Hadoop-common-user] Map intermediate files location

[Hadoop-common-user] Map intermediate files location

	Naama Kraus
	Jul 13, 2008 at 1:03 pm

Hi,

In Google's Map Reduce paper, they mention that map tasks' intermediate
files are written to local file system. Later on, they are fetched by reduce
nodes using RPC calls. Is this also true for Hadoop ?

Thanks, Naama

--
oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo
00 oo 00 oo
"If you want your children to be intelligent, read them fairy tales. If you
want them to be more intelligent, read them more fairy tales." (Albert
Einstein)

Search Discussions

2 responses
Oldest
Nested

Owen O'Malley Yes, although we use http rather than rpc. -- Owen

Owen O'Malley at Jul 13, 2008 at 11:06 pm ⇧

On Jul 12, 2008, at 10:51 PM, Naama Kraus wrote:

Hi,

In Google's Map Reduce paper, they mention that map tasks'
intermediate
files are written to local file system. Later on, they are fetched
by reduce
nodes using RPC calls. Is this also true for Hadoop ?
Yes, although we use http rather than rpc.

-- Owen

reply | permalink

Naama Kraus

at Jul 14, 2008 at 6:27 am

⇧

Thanks.
Do you use Jetty for the http requests ? Anything else ?

Naama

On Mon, Jul 14, 2008 at 2:04 AM, Owen O'Malley wrote:

On Jul 12, 2008, at 10:51 PM, Naama Kraus wrote:

Hi,

In Google's Map Reduce paper, they mention that map tasks' intermediate
files are written to local file system. Later on, they are fetched by
reduce
nodes using RPC calls. Is this also true for Hadoop ?

Yes, although we use http rather than rpc.

-- Owen

reply | permalink

Related Discussions

Discussion Navigation

view	thread \| post

Discussion Overview

group	common-user @
categories	hadoop
posted	Jul 13, '08 at 1:03p
active	Jul 14, '08 at 6:27a
posts	3
users	2
website	hadoop.apache.org...
irc	#hadoop

responses	expanded	Hotkey: s
font	variable	Hotkey: f
user style	avatars	Hotkey: a

[Hadoop-common-user] Map intermediate files location

Search Discussions

Related Discussions

2 users in discussion