FAQ
Hi - I have a problem that might be easy to fix... I setup an external
table under my home directory in hdfs. I don't want anyone except myself
and members in my group to view this table. So I put some permissions on it
750. Now when I go to impala shell and try to query the table I get the
following error:

CAUSED BY: RemoteException: Permission denied: user=impala, access=EXECUTE,

If I change the permissions to 755 all is fixed but I don't want anyone
outside of the group to have access to this table. How can I insure that
the user is actually me and not user=impala when executing this query?

Thanks,
Brendan

To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Search Discussions

  • Greg Rahn at Sep 25, 2013 at 7:01 pm
    This can not be done with impala only -- the impala user (which runs
    impalad) needs permission to the files/directory as impalad does not do
    impersonation.
    However, this could be achieved using Sentry:
    http://www.cloudera.com/content/cloudera/en/products/cdh/sentry.html

    On Wed, Sep 25, 2013 at 10:09 AM, bps cdh wrote:

    Hi - I have a problem that might be easy to fix... I setup an external
    table under my home directory in hdfs. I don't want anyone except myself
    and members in my group to view this table. So I put some permissions on it
    750. Now when I go to impala shell and try to query the table I get the
    following error:

    CAUSED BY: RemoteException: Permission denied: user=impala,
    access=EXECUTE,

    If I change the permissions to 755 all is fixed but I don't want anyone
    outside of the group to have access to this table. How can I insure that
    the user is actually me and not user=impala when executing this query?

    Thanks,
    Brendan

    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Lenni Kuff at Sep 25, 2013 at 7:11 pm
    The Impala security documentation has instructions for how to get up and
    running with Sentry and Impala. This will allow for much finer grained
    access control than what you get with file-level security.

    http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_security.html

    Thanks,
    Lenni

    On Wed, Sep 25, 2013 at 12:01 PM, Greg Rahn wrote:

    This can not be done with impala only -- the impala user (which runs
    impalad) needs permission to the files/directory as impalad does not do
    impersonation.
    However, this could be achieved using Sentry:
    http://www.cloudera.com/content/cloudera/en/products/cdh/sentry.html

    On Wed, Sep 25, 2013 at 10:09 AM, bps cdh wrote:

    Hi - I have a problem that might be easy to fix... I setup an external
    table under my home directory in hdfs. I don't want anyone except myself
    and members in my group to view this table. So I put some permissions on it
    750. Now when I go to impala shell and try to query the table I get the
    following error:

    CAUSED BY: RemoteException: Permission denied: user=impala,
    access=EXECUTE,

    If I change the permissions to 755 all is fixed but I don't want anyone
    outside of the group to have access to this table. How can I insure that
    the user is actually me and not user=impala when executing this query?

    Thanks,
    Brendan

    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Bps cdh at Sep 25, 2013 at 7:22 pm
    Thanks for the reply, that's too bad. We aren't there yet with Sentry but
    we will be sooon... My understanding is Sentry is managed at the db/table
    level and you apply groups to roles. How would sentry get me around this
    issue? Our users want to use impala-shell which will still be "impala"
    user/group. I don't want to add the impala group to every database we
    create and apply sentry to nor do I want to allow impala access to all the
    databases. How would sentry solve this problem? Seems like once I implement
    sentry impala-shell will be useless if I set file permissions of user
    'other' to '0'?
    On Wednesday, September 25, 2013 3:01:41 PM UTC-4, Greg Rahn wrote:

    This can not be done with impala only -- the impala user (which runs
    impalad) needs permission to the files/directory as impalad does not do
    impersonation.
    However, this could be achieved using Sentry:
    http://www.cloudera.com/content/cloudera/en/products/cdh/sentry.html


    On Wed, Sep 25, 2013 at 10:09 AM, bps cdh <bsc...@hotmail.com<javascript:>
    wrote:
    Hi - I have a problem that might be easy to fix... I setup an external
    table under my home directory in hdfs. I don't want anyone except myself
    and members in my group to view this table. So I put some permissions on it
    750. Now when I go to impala shell and try to query the table I get the
    following error:

    CAUSED BY: RemoteException: Permission denied: user=impala,
    access=EXECUTE,

    If I change the permissions to 755 all is fixed but I don't want anyone
    outside of the group to have access to this table. How can I insure that
    the user is actually me and not user=impala when executing this query?

    Thanks,
    Brendan

    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Lenni Kuff at Sep 25, 2013 at 7:30 pm
    The impalad process runs as the "impala" user, but the impala-shell (or
    whatever client you are using) would not run as "impala", it would run as
    the current user. When that user runs a query, Impala will extract their
    username from the connection and use that to ensure they have permission to
    perform the operation. Does that answer your question?

    For more technical details the Impala Security documents are the place to
    go.

    Thanks,
    Lenni

    On Wed, Sep 25, 2013 at 12:22 PM, bps cdh wrote:

    Thanks for the reply, that's too bad. We aren't there yet with Sentry but
    we will be sooon... My understanding is Sentry is managed at the db/table
    level and you apply groups to roles. How would sentry get me around this
    issue? Our users want to use impala-shell which will still be "impala"
    user/group. I don't want to add the impala group to every database we
    create and apply sentry to nor do I want to allow impala access to all the
    databases. How would sentry solve this problem? Seems like once I implement
    sentry impala-shell will be useless if I set file permissions of user
    'other' to '0'?

    On Wednesday, September 25, 2013 3:01:41 PM UTC-4, Greg Rahn wrote:

    This can not be done with impala only -- the impala user (which runs
    impalad) needs permission to the files/directory as impalad does not do
    impersonation.
    However, this could be achieved using Sentry:
    http://www.cloudera.com/**content/cloudera/en/products/**cdh/sentry.html<http://www.cloudera.com/content/cloudera/en/products/cdh/sentry.html>

    On Wed, Sep 25, 2013 at 10:09 AM, bps cdh wrote:

    Hi - I have a problem that might be easy to fix... I setup an external
    table under my home directory in hdfs. I don't want anyone except myself
    and members in my group to view this table. So I put some permissions on it
    750. Now when I go to impala shell and try to query the table I get the
    following error:

    CAUSED BY: RemoteException: Permission denied: user=impala,
    access=EXECUTE,

    If I change the permissions to 755 all is fixed but I don't want anyone
    outside of the group to have access to this table. How can I insure that
    the user is actually me and not user=impala when executing this query?

    Thanks,
    Brendan

    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user...@**cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Bps cdh at Sep 25, 2013 at 7:36 pm
    Yes it does, thank-you. I'll refer to the docs and play around with Sentry.
    On Wednesday, September 25, 2013 3:30:54 PM UTC-4, lskuff wrote:

    The impalad process runs as the "impala" user, but the impala-shell (or
    whatever client you are using) would not run as "impala", it would run as
    the current user. When that user runs a query, Impala will extract their
    username from the connection and use that to ensure they have permission to
    perform the operation. Does that answer your question?

    For more technical details the Impala Security documents are the place to
    go.

    Thanks,
    Lenni


    On Wed, Sep 25, 2013 at 12:22 PM, bps cdh <bsc...@hotmail.com<javascript:>
    wrote:
    Thanks for the reply, that's too bad. We aren't there yet with Sentry
    but we will be sooon... My understanding is Sentry is managed at the
    db/table level and you apply groups to roles. How would sentry get me
    around this issue? Our users want to use impala-shell which will still be
    "impala" user/group. I don't want to add the impala group to every database
    we create and apply sentry to nor do I want to allow impala access to all
    the databases. How would sentry solve this problem? Seems like once I
    implement sentry impala-shell will be useless if I set file permissions of
    user 'other' to '0'?

    On Wednesday, September 25, 2013 3:01:41 PM UTC-4, Greg Rahn wrote:

    This can not be done with impala only -- the impala user (which runs
    impalad) needs permission to the files/directory as impalad does not do
    impersonation.
    However, this could be achieved using Sentry:
    http://www.cloudera.com/**content/cloudera/en/products/**cdh/sentry.html<http://www.cloudera.com/content/cloudera/en/products/cdh/sentry.html>

    On Wed, Sep 25, 2013 at 10:09 AM, bps cdh wrote:

    Hi - I have a problem that might be easy to fix... I setup an external
    table under my home directory in hdfs. I don't want anyone except myself
    and members in my group to view this table. So I put some permissions on it
    750. Now when I go to impala shell and try to query the table I get the
    following error:

    CAUSED BY: RemoteException: Permission denied: user=impala,
    access=EXECUTE,

    If I change the permissions to 755 all is fixed but I don't want anyone
    outside of the group to have access to this table. How can I insure that
    the user is actually me and not user=impala when executing this query?

    Thanks,
    Brendan

    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user...@**cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Matthewcornell at Nov 8, 2013 at 4:23 pm
    I'd also really like to find a solution that doesn't use Sentry. I'm
    getting a similar error running the Cloudera Impala tutorial's third CREATE
    TABLE query:

    CREATE TABLE tab3 (id INT, col_1 BOOLEAN, col_2 DOUBLE, month INT, day INT)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';

    -> this error:

    ERROR: MetaException: Got exception:
    org.apache.hadoop.security.AccessControlException Permission denied:
    user=impala, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

    Any help would be awesome!

    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Darren Lo at Nov 8, 2013 at 6:14 pm
    Authorization in hive/impala is fundamentally, architecturally broken
    without Sentry. You can easily have a policy where everything is accessible
    (no security) by putting 777 perms on everything in the Hive warehouse
    directory (or 1777), or you can have a policy where only one user / group
    can access tables with 770 (or 1770). Complex, useful sharing and security
    is architecturally impossible.

    Thanks,
    Darren

    On Fri, Nov 8, 2013 at 8:23 AM, wrote:

    I'd also really like to find a solution that doesn't use Sentry. I'm
    getting a similar error running the Cloudera Impala tutorial's third CREATE
    TABLE query:

    CREATE TABLE tab3 (id INT, col_1 BOOLEAN, col_2 DOUBLE, month INT, day INT)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';

    -> this error:

    ERROR: MetaException: Got exception:
    org.apache.hadoop.security.AccessControlException Permission denied:
    user=impala, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

    Any help would be awesome!

    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user+unsubscribe@cloudera.org.


    --
    Thanks,
    Darren

    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Lenni Kuff at Nov 8, 2013 at 7:06 pm
    Impala does not support HDFS-level user impersonation. The user the impalad
    process is running as (generally "impala") must have access to the data
    files. To properly secure your data we recommend using Sentry; is there a
    particular reason why you do not want to try it out?

    Thanks,
    Lenni

    On Fri, Nov 8, 2013 at 10:14 AM, Darren Lo wrote:

    Authorization in hive/impala is fundamentally, architecturally broken
    without Sentry. You can easily have a policy where everything is accessible
    (no security) by putting 777 perms on everything in the Hive warehouse
    directory (or 1777), or you can have a policy where only one user / group
    can access tables with 770 (or 1770). Complex, useful sharing and security
    is architecturally impossible.

    Thanks,
    Darren

    On Fri, Nov 8, 2013 at 8:23 AM, wrote:

    I'd also really like to find a solution that doesn't use Sentry. I'm
    getting a similar error running the Cloudera Impala tutorial's third CREATE
    TABLE query:

    CREATE TABLE tab3 (id INT, col_1 BOOLEAN, col_2 DOUBLE, month INT, day
    INT)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';

    -> this error:

    ERROR: MetaException: Got exception:
    org.apache.hadoop.security.AccessControlException Permission denied:
    user=impala, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

    Any help would be awesome!

    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user+unsubscribe@cloudera.org.


    --
    Thanks,
    Darren

    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Matthewcornell at Nov 8, 2013 at 10:10 pm
    Thanks very much for your replies, Darren and Lenni.
    The impalad process runs as the "impala" user, but the impala-shell (or
    whatever client you are using) would not run as "impala", it would run as
    the current user. When that user runs a query, Impala will extract their
    username from the connection and use that to ensure they have permission to
    perform the operation.

    I'm still struggling to understand the interaction between OS users and
    groups, HDFS ones, and impalad. Specifically, here's what I'm doing.
    Following the Impala tutorial (
    http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_tutorial.html
    ), I start impala-shell as me:

         [cornell@server ~]$ whoami
         cornell
         [cornell@server ~]$ impala-shell
         Starting Impala Shell in unsecure mode
         Connected to server.cs.umass.edu:21000
         Server version: impalad version 1.1.1 RELEASE (build
    83d5868f005966883a918a819a449f636a5b3d5f)

    Then I run this query:

    [server.cs.umass.edu:21000] > CREATE TABLE tab3 (id INT, col_1 BOOLEAN,
    col_2 DOUBLE, month INT, day INT)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
    It fails:

       ERROR: MetaException: Got exception:
    org.apache.hadoop.security.AccessControlException Permission denied:
    user=impala, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

    If I understand correctly, this tells me impala-shell passed along my query
    to impald, which then tried to access the '/' HDFS directory as the
    'impala' user, and /not/ as me. The command puked because the 'impala' user
    is neither 'hdfs' nor is it in the group 'supergroup'. (Would I be wrong to
    guess that it's trying to access /user/hive/warehouse - determined by
    running 'set hive.metastore.warehouse.dir;' in the hive command line - ?)

    This leads me to think I simply (ha!) have to tell HDFS that the 'impala'
    user belongs to the group 'supergroup'. However, I don't how to do that
    yet, maybe something like (but using the correct names):

       hdfs dfs -chgrp -R hdfs /user/hive/warehouse
       hdfs dfs -chmod -R 775 /user/hive/warehouse

       sudo -u hdfs hadoop fs -chown impala:impala /user/impala

    Yep, I'm pretty stuck :-\ Any advice would be extremely welcome at this
    point.

    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Greg Rahn at Nov 12, 2013 at 2:06 am
    Your "create table" will simply try and make a directory in the given
    database directory under the hive warehouse root dir.
    If you "sudo su - impala", then try using "hdfs dfs -mkdir
    /user/hive/warehouse/tab3". That command needs to succeed before create
    table will work via the impala-shell.

    Net net - this is an hdfs file/directory permissions issue.

    Hope that helps.

    On Fri, Nov 8, 2013 at 2:10 PM, wrote:

    Thanks very much for your replies, Darren and Lenni.

    The impalad process runs as the "impala" user, but the impala-shell (or
    whatever client you are using) would not run as "impala", it would run as
    the current user. When that user runs a query, Impala will extract their
    username from the connection and use that to ensure they have permission to
    perform the operation.

    I'm still struggling to understand the interaction between OS users and
    groups, HDFS ones, and impalad. Specifically, here's what I'm doing.
    Following the Impala tutorial (
    http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_tutorial.html), I start impala-shell as me:

    [cornell@server ~]$ whoami
    cornell
    [cornell@server ~]$ impala-shell
    Starting Impala Shell in unsecure mode
    Connected to server.cs.umass.edu:21000
    Server version: impalad version 1.1.1 RELEASE (build
    83d5868f005966883a918a819a449f636a5b3d5f)

    Then I run this query:

    [server.cs.umass.edu:21000] > CREATE TABLE tab3 (id INT, col_1 BOOLEAN,
    col_2 DOUBLE, month INT, day INT)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY
    ',';

    It fails:


    ERROR: MetaException: Got exception:
    org.apache.hadoop.security.AccessControlException Permission denied:
    user=impala, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

    If I understand correctly, this tells me impala-shell passed along my
    query to impald, which then tried to access the '/' HDFS directory as the
    'impala' user, and /not/ as me. The command puked because the 'impala' user
    is neither 'hdfs' nor is it in the group 'supergroup'. (Would I be wrong to
    guess that it's trying to access /user/hive/warehouse - determined by
    running 'set hive.metastore.warehouse.dir;' in the hive command line - ?)

    This leads me to think I simply (ha!) have to tell HDFS that the 'impala'
    user belongs to the group 'supergroup'. However, I don't how to do that
    yet, maybe something like (but using the correct names):

    hdfs dfs -chgrp -R hdfs /user/hive/warehouse
    hdfs dfs -chmod -R 775 /user/hive/warehouse

    sudo -u hdfs hadoop fs -chown impala:impala /user/impala

    Yep, I'm pretty stuck :-\ Any advice would be extremely welcome at this
    point.

    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Matthewcornell at Nov 12, 2013 at 1:34 pm
    Thanks, Greg. I am able to create the hdfs directory tab3 like you
    explained, but CREATE TABLE fails in both hive and impala-shell with the
    same error: Permission denied: user=impala, access=WRITE,
    inode="/":hdfs:supergroup:drwxr-xr-x . The commands and output follow.
    Looking at the output from groups, do I simply need to create a unix group
    named 'supergroup' and then add the 'impala' user to it? Thanks very much!
    -- matt

    $ sudo su - impala
    [sudo] password for cornell:
    $ whoami
    impala
    $ groups
    impala hive hdfs
    $ impala-shell
    Starting Impala Shell in unsecure mode
    ...
    (Shell build version: Impala Shell v1.1.1 (83d5868) built on Fri Aug 23
    17:28:05 PDT 2013)
    [server.cs.umass.edu:21000] > CREATE TABLE foo (id INT);
    Query: create TABLE foo (id INT)
    ERROR: MetaException: Got exception:
    org.apache.hadoop.security.AccessControlException Permission denied:
    user=impala, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
         at
    org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)
         ...

    EOF

    On Monday, November 11, 2013 9:06:12 PM UTC-5, Greg Rahn wrote:

    Your "create table" will simply try and make a directory in the given
    database directory under the hive warehouse root dir.
    If you "sudo su - impala", then try using "hdfs dfs -mkdir
    /user/hive/warehouse/tab3". That command needs to succeed before create
    table will work via the impala-shell.

    Net net - this is an hdfs file/directory permissions issue.

    Hope that helps.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Greg Rahn at Nov 12, 2013 at 10:15 pm
    It appears it's trying to write to the / of hdfs.
    What is hive.metastore.warehouse.dir set to?

    On Tue, Nov 12, 2013 at 5:34 AM, wrote:

    Thanks, Greg. I am able to create the hdfs directory tab3 like you
    explained, but CREATE TABLE fails in both hive and impala-shell with the
    same error: Permission denied: user=impala, access=WRITE,
    inode="/":hdfs:supergroup:drwxr-xr-x . The commands and output follow.
    Looking at the output from groups, do I simply need to create a unix group
    named 'supergroup' and then add the 'impala' user to it? Thanks very much!
    -- matt

    $ sudo su - impala
    [sudo] password for cornell:
    $ whoami
    impala
    $ groups
    impala hive hdfs

    $ impala-shell
    Starting Impala Shell in unsecure mode
    ...
    (Shell build version: Impala Shell v1.1.1 (83d5868) built on Fri Aug 23
    17:28:05 PDT 2013)
    [server.cs.umass.edu:21000] > CREATE TABLE foo (id INT);
    Query: create TABLE foo (id INT)

    ERROR: MetaException: Got exception:
    org.apache.hadoop.security.AccessControlException Permission denied:
    user=impala, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
    at
    org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)
    ...

    EOF


    On Monday, November 11, 2013 9:06:12 PM UTC-5, Greg Rahn wrote:

    Your "create table" will simply try and make a directory in the given
    database directory under the hive warehouse root dir.
    If you "sudo su - impala", then try using "hdfs dfs -mkdir
    /user/hive/warehouse/tab3". That command needs to succeed before create
    table will work via the impala-shell.

    Net net - this is an hdfs file/directory permissions issue.

    Hope that helps.

    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Matthewcornell at Nov 14, 2013 at 4:25 pm
    Sorry for the delay in responding. I ran some experiments to get a better
    handle on what's going on, and I think I've finally figured it out, thanks
    to your help.

    During installation, the hive.metastore.warehouse.dir was changed from the
    default to another hdfs location. Confusingly, if instead of getting the
    variable's value in Cloudera Manager, I use the hive shell, it shows as the
    default: /user/hive/warehouse , which does exist in hdfs. But that's
    apparently unused - new tables show up in the former. Any ideas?

    Once I knew where to look, the permission problem became clear - the new
    warehouse directory had hdfs:supergroup permission, not hive:hive, which
    the default location had. The hive and impala OS accounts belong to the
    hive group and not supergroup, they were failing. I changed warehouse
    permissions to hive:hive and now it all works. Took a long time to put this
    together, but I'm up on the learning curve now :-)

    Thanks again.
    On Tuesday, November 12, 2013 5:15:06 PM UTC-5, Greg Rahn wrote:

    It appears it's trying to write to the / of hdfs.
    What is hive.metastore.warehouse.dir set to?


    On Tue, Nov 12, 2013 at 5:34 AM, <matthew...@gmail.com <javascript:>>wrote:
    Thanks, Greg. I am able to create the hdfs directory tab3 like you
    explained, but CREATE TABLE fails in both hive and impala-shell with the
    same error: Permission denied: user=impala, access=WRITE,
    inode="/":hdfs:supergroup:drwxr-xr-x . The commands and output follow.
    Looking at the output from groups, do I simply need to create a unix group
    named 'supergroup' and then add the 'impala' user to it? Thanks very much!
    -- matt

    $ sudo su - impala
    [sudo] password for cornell:
    $ whoami
    impala
    $ groups
    impala hive hdfs

    $ impala-shell
    Starting Impala Shell in unsecure mode
    ...
    (Shell build version: Impala Shell v1.1.1 (83d5868) built on Fri Aug 23
    17:28:05 PDT 2013)
    [server.cs.umass.edu:21000] > CREATE TABLE foo (id INT);
    Query: create TABLE foo (id INT)

    ERROR: MetaException: Got exception:
    org.apache.hadoop.security.AccessControlException Permission denied:
    user=impala, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
    at
    org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)
    ...

    EOF


    On Monday, November 11, 2013 9:06:12 PM UTC-5, Greg Rahn wrote:

    Your "create table" will simply try and make a directory in the given
    database directory under the hive warehouse root dir.
    If you "sudo su - impala", then try using "hdfs dfs -mkdir
    /user/hive/warehouse/tab3". That command needs to succeed before create
    table will work via the impala-shell.

    Net net - this is an hdfs file/directory permissions issue.

    Hope that helps.

    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Greg Rahn at Nov 14, 2013 at 7:08 pm
    Hive shell gets that variable from the hive-site.xml file and it may be
    that the client files are out of sync from what CM has. If you set the
    desired nodes as gateways (there is a gateway role) and then use CM to do a
    "deploy client configuration" it will push out files that should match what
    is kept in CM.

    Docs:
    http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.3/Cloudera-Manager-Enterprise-Edition-User-Guide/cmeeug_topic_5_9.html

    Hope that helps.


    On Thu, Nov 14, 2013 at 8:25 AM, wrote:

    Sorry for the delay in responding. I ran some experiments to get a better
    handle on what's going on, and I think I've finally figured it out, thanks
    to your help.

    During installation, the hive.metastore.warehouse.dir was changed from the
    default to another hdfs location. Confusingly, if instead of getting the
    variable's value in Cloudera Manager, I use the hive shell, it shows as the
    default: /user/hive/warehouse , which does exist in hdfs. But that's
    apparently unused - new tables show up in the former. Any ideas?

    Once I knew where to look, the permission problem became clear - the new
    warehouse directory had hdfs:supergroup permission, not hive:hive, which
    the default location had. The hive and impala OS accounts belong to the
    hive group and not supergroup, they were failing. I changed warehouse
    permissions to hive:hive and now it all works. Took a long time to put this
    together, but I'm up on the learning curve now :-)

    Thanks again.

    On Tuesday, November 12, 2013 5:15:06 PM UTC-5, Greg Rahn wrote:

    It appears it's trying to write to the / of hdfs.
    What is hive.metastore.warehouse.dir set to?

    On Tue, Nov 12, 2013 at 5:34 AM, wrote:

    Thanks, Greg. I am able to create the hdfs directory tab3 like you
    explained, but CREATE TABLE fails in both hive and impala-shell with the
    same error: Permission denied: user=impala, access=WRITE,
    inode="/":hdfs:supergroup:drwxr-xr-x . The commands and output follow.
    Looking at the output from groups, do I simply need to create a unix group
    named 'supergroup' and then add the 'impala' user to it? Thanks very much!
    -- matt

    $ sudo su - impala
    [sudo] password for cornell:
    $ whoami
    impala
    $ groups
    impala hive hdfs

    $ impala-shell
    Starting Impala Shell in unsecure mode
    ...
    (Shell build version: Impala Shell v1.1.1 (83d5868) built on Fri Aug 23
    17:28:05 PDT 2013)
    [server.cs.umass.edu:21000] > CREATE TABLE foo (id INT);
    Query: create TABLE foo (id INT)

    ERROR: MetaException: Got exception: org.apache.hadoop.security.AccessControlException
    Permission denied: user=impala, access=WRITE, inode="/":hdfs:supergroup:
    drwxr-xr-x
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
    check(FSPermissionChecker.java:224)
    ...

    EOF


    On Monday, November 11, 2013 9:06:12 PM UTC-5, Greg Rahn wrote:

    Your "create table" will simply try and make a directory in the given
    database directory under the hive warehouse root dir.
    If you "sudo su - impala", then try using "hdfs dfs -mkdir
    /user/hive/warehouse/tab3". That command needs to succeed before create
    table will work via the impala-shell.

    Net net - this is an hdfs file/directory permissions issue.

    Hope that helps.

    To unsubscribe from this group and stop receiving emails from it,
    send an email to impala-user...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Matthewcornell at Nov 18, 2013 at 2:09 pm
    Thanks a ton, Darren.
    On Friday, November 15, 2013 12:34:57 PM UTC-5, Darren Lo wrote:

    When using Cloudera Manager, all daemons get protected configuration
    directories in /var/run/cloudera-scm-agent/process/. They generally do not
    read from /etc/.


    On Fri, Nov 15, 2013 at 8:31 AM, <matthew...@gmail.com <javascript:>>wrote:
    On Thursday, November 14, 2013 2:08:36 PM UTC-5, Greg Rahn wrote:

    Hive shell gets that variable from the hive-site.xml file and it may be
    that the client files are out of sync from what CM has. If you set the
    desired nodes as gateways (there is a gateway role) and then use CM to do a
    "deploy client configuration" it will push out files that should match what
    is kept in CM.
    You are right: The file returned just now by the Client Configuration
    URLs menu shows the correct variable value, but the client file in
    /etc/hive/conf/hive-site.xml on compute nodes incorrectly have the default
    value. (That is the only difference between the files, other than the
    timestamp.) What's confusing to me is that, as I found above, the server is
    correctly using the new value (new tables go into the correct hdsf
    warehouse dir), but it appears the hive shell is using the latter. Q: Where
    does the former pick up the value? Regardless, I suppose I should run the
    "Deploy Client Configuration" action as documented.

    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user...@cloudera.org <javascript:>.


    --
    Thanks,
    Darren
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedSep 25, '13 at 5:09p
activeNov 18, '13 at 2:09p
posts16
users5
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase