The simplest route I can think of is to ingest the data directly into HDFS using Sqoop if there is a driver currently made for your database. At that point it would be relatively simple just to read directly from HDFS in your MR code.
Sent: Tuesday, September 27, 2011 4:48 PM
Subject: Temporary Files to be sent to DistributedCache
I have a need to write information retrieved from a database to a series of
files that need to be made available to my mappers. Because each mapper
needs access to all of these files, I want to put them in the
DistributedCache. Is there a preferred method to writing new information to
the DistributedCache? I can use Java's File.createTempFile(String prefix,
String suffix), but that uses the system default temporary folder. While
that should usually work, I'd rather have a method that doesn't depend on
writing to the local file system before copying files to the
DistributedCache. As I'm extremely new to Hadoop, I hope I'm not missing
Thank you for your time.
This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.
All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.
The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.