FAQ
Hi Marcel,
I tried the updated scripts but seems there is some formatting issue with
the setup-impala.sh script, specifically in the wite-hive-site function.
Also, looks like statestored is not being started in the scripts.

However, I managed to get the old scripts working by copying just the hdfs
and core-site.xml from the new scripts. Didn't know that settings for
short-circuit reads have changed in 0.6.

Thanks for the help.

-
Jaideep

On Mon, Mar 18, 2013 at 8:35 AM, Jaideep Dhok wrote:

Hi,
Thanks for responding. I will try the new instructions and see if they are
working.

Thanks,
Jaideep

On Mon, Mar 18, 2013 at 1:42 AM, Marcel Kornacker wrote:

Jaideep, the EC2 setup instructions changed just recently. Could you
please try the setup again and then let us know if there's anything
that's still not working?

On Sat, Mar 16, 2013 at 11:10 PM, Jaideep Dhok <jaideep.dhok@inmobi.com>
wrote:
I am using 0.6


On Sun, Mar 17, 2013 at 2:39 AM, Marcel Kornacker <marcel@cloudera.com>
wrote:
Jaideep, which version of Impala did you install?
On Fri, Mar 15, 2013 at 2:42 AM, wrote:
Hi,

I keep getting the 'unkown disk id ... location metadata' warnings,
even
though I have made necessary setup in Impala's hdfs-site.xml as well
as
the
main HDFS site.xml.

I have followed the instructions given at
http://blog.cloudera.com/blog/2013/02/from-zero-to-impala-in-minutes/ to
setup a test cluster, with the only difference being that I am using
an
AMI
with multiple instance store volumes, so that I can have multiple
disks
per
node .

HDFS is able to see these directories since I can see that data is
being
stored there. Also, from Impala's web UI I can see in /varz that the
config
is loaded correctly.

I checked the impala user, and it has been added to the hdfs group as
well.

The thing is, I don't see these warnings when I use the AMI in the
blog
article (which mounts only one disk), so I am bit confused as to
what is
causing this.

Any help would be appreciated.

Thanks,
Jaideep

--
Some more info abou the setup --

# impala's hdfs-site.xml -
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/mnt/data,/media/e1,/media/e2,/media/e3</value>
</property>

# mounted partitions
-bash-4.1$ mount
/dev/xvde1 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
/dev/xvdf on /mnt type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/xvdg on /media/e1 type ext3 (rw)
/dev/xvdh on /media/e2 type ext3 (rw)
/dev/xvdi on /media/e3 type ext3 (rw)

#Impala command line flags -
--dump_ir=false
--module_output=
--be_port=22000
--classpath=
--hostname=ip-10-173-31-155
--ipaddress=10.173.31.155
--keytab_file=
--planservice_host=localhost
--planservice_port=20000
--principal=
--max_row_batches=0
--randomize_scan_ranges=false
--num_disks=4
--num_threads_per_disk=1
--read_size=8388608
--enable_webserver=true
--use_statestore=true
--nn=ec2-54-241-234-68.us-west-1.compute.amazonaws.com
--nn_port=8020
--serialize_batch=false
--status_report_interval=5
--abort_on_config_error=true
--be_service_threads=64
--beeswax_port=21000
--default_query_options=
--fe_service_threads=64
--heap_profile_dir=
--hs2_port=21050
--load_catalog_at_startup=false
--log_mem_usage_interval=0
--mem_limit=-1
--query_log_size=25
--use_planservice=false
--statestore_subscriber_timeout_seconds=10
--state_store_host=10.173.31.155
--state_store_port=24000
--state_store_subscriber_port=23000
--kerberos_reinit_interval=60
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--web_log_bytes=1048576
--log_filename=impalad
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--enable_webserver_doc_root=true
--webserver_doc_root=/usr/lib/impala
--webserver_interface=
--webserver_port=25000
--flagfile=
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/tmp
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=1800
--minloglevel=0
--stderrthreshold=2
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=



_____________________________________________________________
The information contained in this communication is intended solely
for
the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally
privileged
information. If you are not the intended recipient you are hereby
notified
that any disclosure, copying, distribution or taking any action in
reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please
notify us
immediately by responding to this email and then delete it from your
system.
The firm is neither liable for the proper and complete transmission
of
the
information contained in this communication nor for any delay in its
receipt.


_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally
privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify us
immediately by responding to this email and then delete it from your system.
The firm is neither liable for the proper and complete transmission of the
information contained in this communication nor for any delay in its
receipt.
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 5 of 5 | next ›
Discussion Overview
groupimpala-user @
categorieshadoop
postedMar 15, '13 at 9:42a
activeMar 18, '13 at 11:42a
posts5
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Jaideep Dhok: 3 posts Marcel Kornacker: 2 posts

People

Translate

site design / logo © 2022 Grokbase