FAQ
Superuser privileges required to do "df"
----------------------------------------

Key: HADOOP-4368
URL: https://issues.apache.org/jira/browse/HADOOP-4368
Project: Hadoop Core
Issue Type: Wish
Components: contrib/fuse-dfs, dfs
Affects Versions: 0.18.1
Reporter: Brian Bockelman
Priority: Minor


super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.

As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).

To fix this, simply comment out the privilege check in the getStats method.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Pete Wyckoff (JIRA) at Oct 13, 2008 at 12:43 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638929#action_12638929 ]

    Pete Wyckoff commented on HADOOP-4368:
    --------------------------------------

    hi brian,
    do you know why they (who?) added this check?

    pete

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Brian Bockelman (JIRA) at Oct 13, 2008 at 1:01 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639042#action_12639042 ]

    Brian Bockelman commented on HADOOP-4368:
    -----------------------------------------

    No clue - I suspect that it was checked in when permissions were added to the FSNamesystem.

    I emailed the core-user list, but no dice.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Oct 13, 2008 at 5:13 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639114#action_12639114 ]

    Doug Cutting commented on HADOOP-4368:
    --------------------------------------
    I suspect that it was checked in when permissions were added to the FSNamesystem.
    Did you verify this? 'svn blame' tells all.

    http://tinyurl.com/42uf5d

    which leads to HADOOP-2659, indicating the folks you might direct your question towards. core-dev might be the more appropriate list at this point.



    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pete Wyckoff (JIRA) at Oct 14, 2008 at 1:30 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639270#action_12639270 ]

    Pete Wyckoff commented on HADOOP-4368:
    --------------------------------------

    bq. core-dev might be the more appropriate list at this

    I emailed core-dev about this with the subject "Requiring super user for getStats"

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Oct 17, 2008 at 6:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640612#action_12640612 ]

    Konstantin Shvachko commented on HADOOP-4368:
    ---------------------------------------------

    You are talking about DFSAdmin -report command, right?
    In general DFSAdmin commands should require superuser privileges. If -report should be opened to regular users it should not be a DFSAdmin command but rather a regular shell command. So opening getStats for all users means moving it to the FSShell command group.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Oct 17, 2008 at 7:00 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640627#action_12640627 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------
    In general DFSAdmin commands should require superuser privileges.
    +1

    When I worked on HADOOP-2659, I added checks for superuser for all DFSAdmin commands.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Brian Bockelman (JIRA) at Oct 17, 2008 at 7:18 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640634#action_12640634 ]

    Brian Bockelman commented on HADOOP-4368:
    -----------------------------------------

    Hey Konstantin, Nicholas,

    The DFSAdmin -report command gives you a lot more information than what getStats does. getStats only returns the value of capacity/free/used; these are kept in internal counters, and not recomputed when the function is executed. Traditionally, knowing how large the file system is is not a protected operation in Unix.

    I guess it would make sense to include the getStats command in the FSShell command group.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pete Wyckoff (JIRA) at Oct 22, 2008 at 9:08 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641970#action_12641970 ]

    Pete Wyckoff commented on HADOOP-4368:
    --------------------------------------

    bq. In general DFSAdmin commands should require superuser privileges.

    but, not for commands where in the posix world super user is not required - I think the default should be to go with posix requirements unless there's a compelling reason in HDFS to require more.

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Oct 22, 2008 at 9:56 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641977#action_12641977 ]

    Konstantin Shvachko commented on HADOOP-4368:
    ---------------------------------------------

    shv> opening getStats for all users means moving it to the FSShell command group.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Oct 22, 2008 at 10:50 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641989#action_12641989 ]

    Doug Cutting commented on HADOOP-4368:
    --------------------------------------

    Pete, no one disagrees with your thesis, that getStats should not be restricted to superuser. The solution is to move it from DFSAdmin to FSShell. Unfortunately it's not quite that simple, since this is currently DFS-specific, and FSShell is generic. But capacity is a fairly generic operation. So a proper fix might also add a FileSsytem method like:
    {code}
    public FsStatus FileSystem#getStatus()

    public class FsStatus {
    public long getCapacity();
    public long getFree();
    public long getUsed();
    }
    {code}

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Allen Wittenauer (JIRA) at Oct 27, 2008 at 6:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642987#action_12642987 ]

    Allen Wittenauer commented on HADOOP-4368:
    ------------------------------------------

    I argue that anything that doesn't require super user shouldn't be under dfsadmin to make the distinction clear.

    One thing that has me concerned... what sort of load on the name node does generating the data for -report generate? If it is significant, then it should remained privileged.


    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Oct 29, 2008 at 9:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643663#action_12643663 ]

    Craig Macdonald commented on HADOOP-4368:
    -----------------------------------------

    @Allen
    The capacity operations on the namenode being discussed here are all lightweight, simply fetching three instance variables from the NameNode.

    @Doug
    So there is an existing DiskStatus already in DistributedFileSystem, which has a very similar API. Could this class be promoted to FileSystem?
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Oct 30, 2008 at 7:25 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644080#action_12644080 ]

    Doug Cutting commented on HADOOP-4368:
    --------------------------------------
    Could this class be promoted to FileSystem?
    Yes. It would be better renamed FsStatus, and 'dfsUsed' renamed just 'used', as indicated above. The default implementation might return Long.MAX_VALUE for capacity and free, and zero for used.

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Brian Bockelman (JIRA) at Dec 4, 2008 at 3:15 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653325#action_12653325 ]

    Brian Bockelman commented on HADOOP-4368:
    -----------------------------------------

    Hey all,

    This has gotten a bit worse under 0.19.0. In 0.19.0, "df" simply doesn't report anything for Hadoop as a normal user but works as root. I.e., for normal user:

    [brian@red ~]$ df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda3 254G 34G 208G 14% /
    /dev/sda1 99M 14M 81M 15% /boot
    none 7.9G 0 7.9G 0% /dev/shm
    nfs03:/mnt/raid 4.1T 1.7T 2.4T 41% /mnt/nfs03
    nfs04:/mnt/raid 3.2T 1.1T 2.1T 34% /mnt/nfs04
    dcache-pnfs.unl.edu:/fs
    391M 79M 278M 22% /pnfs/unl.edu

    For root,

    [brian@red ~]$ sudo df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda3 254G 34G 208G 14% /
    /dev/sda1 99M 14M 81M 15% /boot
    none 7.9G 0 7.9G 0% /dev/shm
    nfs03:/mnt/raid 4.1T 1.7T 2.4T 41% /mnt/nfs03
    nfs04:/mnt/raid 3.2T 1.1T 2.1T 34% /mnt/nfs04
    dcache-pnfs.unl.edu:/fs
    391M 79M 278M 22% /pnfs/unl.edu
    fuse 140T 122T 140T 47% /mnt/hadoop


    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Brian Bockelman (JIRA) at Dec 4, 2008 at 4:01 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Brian Bockelman updated HADOOP-4368:
    ------------------------------------

    Attachment: fuse_statfs_trunk.patch

    Patch against trunk, post-Pete's reorganization of the fuse-dfs source.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Brian Bockelman (JIRA) at Dec 4, 2008 at 4:01 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Brian Bockelman updated HADOOP-4368:
    ------------------------------------

    Attachment: fuse_statfs.patch

    Attached a patch for 0.19.0.

    This patch adds a "connect as superuser" method which statfs uses.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Dec 4, 2008 at 5:25 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653370#action_12653370 ]

    Craig Macdonald commented on HADOOP-4368:
    -----------------------------------------

    We cannot add a method for FsStatus for FileSystem in 0.19 branch, however, we can do in trunk. In this case, I vote +1 for accepting Brian's patch for the 0.19 branch and inclusion in the 0.19.1 release.

    For trunk, the FsStatus should be added to FileSystem


    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Dec 4, 2008 at 5:51 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653382#action_12653382 ]

    Konstantin Shvachko commented on HADOOP-4368:
    ---------------------------------------------

    I am confused. Does that mean that anybody will be able to connect to HDFS as a superuser via fuse?
    It seamed like a rather simple move of a command from one class to another. Why would you want to cut corners like that?
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Dec 4, 2008 at 6:03 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653388#action_12653388 ]

    Craig Macdonald commented on HADOOP-4368:
    -----------------------------------------

    bq. Does that mean that anybody will be able to connect to HDFS as a superuser via fuse?

    No. Brian's patch allows the statfs call (as represented by the df command) to work correctly, by calling the appropriate libhdfs methods as superuser, before dropping the connection.

    The moving of the appropriate API from DistributedFileSystem to FileSystem is indeed fairly straightforward. I guess I was raising the question as to whether that would be permitted for 0.19.1, as I obviously view the current proposed patch as an interim fix.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Dec 4, 2008 at 6:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653397#action_12653397 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------

    doConnectAsSuperuser(..) won't work very well since the constant "superuser" is not used in HDFS and the constant "supergroup" is a configurable parameter.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Dec 4, 2008 at 7:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653443#action_12653443 ]

    Raghu Angadi commented on HADOOP-4368:
    --------------------------------------

    my 2c: This policy should not be worked around in fuse_dfs. It should be fixed in HDFS. fuse should handle errors (including permissions errors) for any commands normally.

    +1 for not requiring superuser for df/stats in HDFS.

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pete Wyckoff (JIRA) at Dec 4, 2008 at 8:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653478#action_12653478 ]

    Pete Wyckoff commented on HADOOP-4368:
    --------------------------------------

    More generally, is there a design doc or jira for where permissions are going in hdfs? so we can see how these jive with fuse-dfs?

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Dec 4, 2008 at 10:11 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653511#action_12653511 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------

    The main JIRA for permissions is HADOOP-1298. There is also a user guide in the forrest doc.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pete Wyckoff (JIRA) at Dec 4, 2008 at 10:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653517#action_12653517 ]

    Pete Wyckoff commented on HADOOP-4368:
    --------------------------------------

    I will look at those, but what about authorization and future change plans? integration with other systems - nis, ldap, ...

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Brian Bockelman (JIRA) at Dec 5, 2008 at 11:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653979#action_12653979 ]

    Brian Bockelman commented on HADOOP-4368:
    -----------------------------------------

    Hm - one of our sysadmins pointed out that the patch doesn't compile correctly because I submitted the wrong version of it.

    Nicholas, what's the best way to determine the user / group I should be using?

    Might as well get the patch right when I resubmit it.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Dec 5, 2008 at 11:56 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653988#action_12653988 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------
    Nicholas, what's the best way to determine the user / group I should be using?
    We currently use shell commands "whoami" and "bash -c groups" to get the username and group information. See org.apache.hadoop.util.Shell.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Brian Bockelman (JIRA) at Dec 7, 2008 at 3:32 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654206#action_12654206 ]

    Brian Bockelman commented on HADOOP-4368:
    -----------------------------------------

    Nah - I meant, how do we determine what the name of the superuser should be. I want to connect to Hadoop as superuser/supergroup, but you said hardcoding "superuser"/"supergroup" would not work in all cases.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Dec 8, 2008 at 4:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654457#action_12654457 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------
    I want to connect to Hadoop as superuser/supergroup, ...
    This sounds impossible or should not be allowed. Otherwise, it is very easy to hack the system.

    Currently, the superusers are either the user starting the namenode or the users who belong to the group specified by the conf property dfs.permissions.supergroup (the default is "supergroup"). There is no way for clients to get the superuser accounts in runtime.

    See also http://hadoop.apache.org/core/docs/r0.19.0/hdfs_permissions_guide.html#The+Super-User
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 5, 2009 at 12:38 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Attachment: hadoop4368.fsstatus.patch

    This patch in effect corrects the Java API:
    * adds FsStatus in FileSystem, and made DFSClient use this
    * DiskStatus in DistributedFileSystem is deprecate, but extends FsStatus.
    * FSNamesystem#getStats() no longer requires super user privelege - all it does is return the values of three variables, so does not add considerable load to the name node.
    * As suggested by Doug, the default FileSystem implementation returns Long.MAX_VALUE for free and capacity, and 0 for used. This could be fixed for LocalFileSystem using the java.io.File api, however, it depends on which partition of the local filesystem that you want to query.
    ** In this case, perhaps getStats(Path p) would be more appropriate? DistributedFileSystem would ignore the Path parameter
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Jan 5, 2009 at 7:00 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Doug Cutting reassigned HADOOP-4368:
    ------------------------------------

    Assignee: Craig Macdonald

    This is looking good. A few nits:
    - FsStatus should not be nested, but rather a standalone class.
    - FsStatus needs Javadoc.
    - We should add a unit test for this new public API (you're welcome, Nigel). This should be as new test methods in existing unit test classes.

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 6, 2009 at 12:35 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661112#action_12661112 ]

    Craig Macdonald commented on HADOOP-4368:
    -----------------------------------------

    @Doug

    Will make these changes, however before I do so do you have any comment on the last issue I raised for LocalFileSystem etc?


    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 7, 2009 at 11:25 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Attachment: hadoop4368.fsstatus.v2.patch

    Address Doug's comments. Also made FsStatus Writable, because FileStatus is also writable.

    For testing, I tried asserting that capacity = free + used, but this doesn't seem to be the case for HDFS, and could feasibly be not true on other filesystems implementation, so I omitted this from the test.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 7, 2009 at 11:25 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Status: Patch Available (was: Open)
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Brian Bockelman (JIRA) at Jan 9, 2009 at 12:41 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662196#action_12662196 ]

    Brian Bockelman commented on HADOOP-4368:
    -----------------------------------------

    Craig,

    Thank you very much for helping out. Regrettably, I have not have the time to contribute/write this patch myself.

    Brian
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 9, 2009 at 7:24 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662486#action_12662486 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------

    We should also move the command from DFSAdmin to FsShell as mentioned in [a previous comment|https://issues.apache.org/jira/browse/HADOOP-4368?focusedCommentId=12641989#action_12641989].
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Jan 9, 2009 at 7:44 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662494#action_12662494 ]

    Doug Cutting commented on HADOOP-4368:
    --------------------------------------

    Craig> any comment on the last issue I raised for LocalFileSystem etc?

    Hmm. I guess it makes sense to make it FileSystem#getStats(Path), so that LocalFileSystem can report different stats for different volumes. Should we also define FileSystem#getStats() to call this with the root path by default as a convenience?
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 23, 2009 at 7:16 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Attachment: hadoop4368.fsstatus.v3.patch

    Updated patch attached, addressing Nicholas' and Doug's comments.

    @Nicholas: I have added a -df command to FsShell, which has similar output to GNUs core-utils df command. I left the DFSAdmin output, as I thought this was (a) useful, and (b) gives more interpretations on the reported figures for the HDFS scenario than FsShell#df() should.

    NB: The java.io.File getTotalSpace() etc API works find for normally mounted local and NFS partitions on linux, but for partitions that are mounted by autofs, I found it always gave 0s. This seems to be a problem with these API methods that are new in Java 6. I couldn't find reference to such a problem anywhere on the web.

    Two questions:
    1. Should I provide ant-patch output myself?
    2. Should I fix libhdfs in this issue as well?
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Jan 23, 2009 at 7:50 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666672#action_12666672 ]

    Doug Cutting commented on HADOOP-4368:
    --------------------------------------
    1. Should I provide ant-patch output myself?
    Sure, it never hurts to do this.
    2. Should I fix libhdfs in this issue as well?
    If you like, but I don't see it as a requirement for this issue.

    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: contrib/fuse-dfs, dfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 24, 2009 at 12:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Attachment: hadoop4368.fsstatus.v4.patch

    @Doug: Brian, the original poster, was concerned about lookups from fuse_dfs. In actual fact the problem is with libhdfs and also the provision of an adequate Java API. In this case, fixing libhdfs as well as the Java API should resolve the entire issue.

    Attached work-in-progress patch. This attempts to add libhdfs support for FsStatus API. If anyone knows JNI, perhaps they can explain why it is calling append(Path f, int bufferSize) instead of getStatus()?? I'm mystified.

    {noformat}
    $ ant -Dcompile.c++=true -Dlibhdfs=true clean compile test-c++-libhdfs
    <snip>
    [exec] Exception in thread "main" java.lang.NullPointerException
    [exec] at org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:89)
    [exec] at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:154)
    [exec] at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:185)
    [exec] at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:531)
    [exec] Call to org.apache.hadoop.fs.FsStatus::getCapacity failed!
    [exec] hdfsGetCapacity: -1
    [exec] Exception in thread "main" java.lang.NullPointerException
    [exec] at org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:89)
    [exec] at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:154)
    [exec] at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:185)
    [exec] at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:531)
    [exec] Call to org.apache.hadoop.fs.FsStatus::getUsed failed!
    <snip>
    {noformat}
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 24, 2009 at 12:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Component/s: (was: contrib/fuse-dfs)
    libhdfs

    Changed components to dfs, libhdfs
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 25, 2009 at 2:43 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Attachment: hadoop4368.fsstatus.v5.patch

    New patch attached, all working, all tests passing. This patch is ready for review.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 26, 2009 at 10:58 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Status: Patch Available (was: Open)
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 26, 2009 at 10:58 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Status: Open (was: Patch Available)
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 26, 2009 at 7:06 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667353#action_12667353 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------

    The patch looks good to me. The only problem I can see is that FSNamesystem.getStats() should not be declared with "throw IOException" anymore. Similarly, NameNode.getStats() should not be declared with "throw IOException".

    Some minor suggestions:

    - remove getCapacityTotal(), getCapacityUsed(), getCapacityUsedPercent(), getCapacityUsedNonDFS() and getCapacityRemaining() in FSNamesystem. These methods are mostly used in a test. All the uses of these methods can be replaced by getStats().

    - deprecated getRawCapacity() and getRawUsed() in DistributedFileSystem and remove totalRawCapacity() and totalRawUsed() in DFSClient.

    It would be great if we can remove DiskStatus instead of deprecated it. However, it seems we can't because it is public.

    Thank you for working on this, Craig.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 26, 2009 at 8:32 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Status: Open (was: Patch Available)

    Canceling patch until I address Nicholas' (straightforward) comments.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 28, 2009 at 10:09 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667982#action_12667982 ]

    Craig Macdonald commented on HADOOP-4368:
    -----------------------------------------

    @Nicholas: It appears that getCapacityTotal(), getCapacityUsed(), getCapacityRemaining are all exported in the FSNamesystemMBean interface. This is no longer in the public javadoc, but still exists.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 28, 2009 at 10:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668191#action_12668191 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------
    It appears that getCapacityTotal(), getCapacityUsed(), getCapacityRemaining are all exported in the FSNamesystemMBean interface.
    How about we use getStats() in FSNamesystemMBean?
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 28, 2009 at 11:43 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668235#action_12668235 ]

    Craig Macdonald commented on HADOOP-4368:
    -----------------------------------------


    Here's what I propose:

    * getCapacityTotal(), getCapacityUsed(), getCapacityRemaining() remain in FSNamesystem, but implement these using the return from getStats().

    * I don't known JMX, but I think deprecating and ultimately removing the above getCapacity*() methods in favour of getStats() would make the use of the relevant statistics more difficult?

    * Other changes to DistributedFileSystem and DFSClient remain as in your original review.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 29, 2009 at 12:17 am
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668256#action_12668256 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-4368:
    ------------------------------------------------
    I don't known JMX, but I think deprecating and ultimately removing the above getCapacity*() methods in favour of getStats() would make the use of the relevant statistics more difficult?
    I see your points. Let's keep these methods.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Craig Macdonald (JIRA) at Jan 30, 2009 at 12:50 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Craig Macdonald updated HADOOP-4368:
    ------------------------------------

    Attachment: hadoop4368.fsstatus.v6.patch

    Updated patch addressing agreed changes.
    Superuser privileges required to do "df"
    ----------------------------------------

    Key: HADOOP-4368
    URL: https://issues.apache.org/jira/browse/HADOOP-4368
    Project: Hadoop Core
    Issue Type: Wish
    Components: dfs, libhdfs
    Affects Versions: 0.18.1
    Reporter: Brian Bockelman
    Assignee: Craig Macdonald
    Priority: Minor
    Attachments: fuse_statfs.patch, fuse_statfs_trunk.patch, hadoop4368.fsstatus.patch, hadoop4368.fsstatus.v2.patch, hadoop4368.fsstatus.v3.patch, hadoop4368.fsstatus.v4.patch, hadoop4368.fsstatus.v5.patch, hadoop4368.fsstatus.v6.patch

    Original Estimate: 0.17h
    Remaining Estimate: 0.17h

    super user privileges are required in DFS in order to get the file system statistics (FSNamesystem.java, getStats method). This means that when HDFS is mounted via fuse-dfs as a non-root user, "df" is going to return 16exabytes total and 0 free instead of the correct amount.
    As far as I can tell, there's no need to require super user privileges to see the file system size (and historically in Unix, this is not required).
    To fix this, simply comment out the privilege check in the getStats method.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedOct 7, '08 at 11:25p
activeFeb 16, '09 at 5:05p
posts63
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Hudson (JIRA): 63 posts

People

Translate

site design / logo © 2022 Grokbase