Grokbase › Groups › Hadoop › common-user › January 2010
FAQ
Hi,

I am running a MR job that requires usage of some java.awt.* classes, that
can't be run in headless mode.

Right now, I am running Hadoop in a single node cluster (my laptop) which
has X11 server running. I have set up my ssh server and client to do X11
forwarding.

I ran the following java program to ensure that X11 forwarding is working-

public class Test
throws Exception
{
public static void main( String[] args)
{
Process p = runtime.getRuntime( ).exec( "/usr/bin/xterm");
p.waitFor( );
}
}

I ran it as-
ssh localhost "java Test"
and it worked confirming that X11 forwarding is working over SSH.

However, when I run a Map Reduce program that uses java.awt.* classes
(trying to create an object that extends Frame), I keep getting this error-

java.lang.InternalError- Can't connect to X11 window server using
'localhost:10.0' as the value of the DISPLAY variable

I printed the value of DISPLAY variable-

echo $DISPLAY
:0.0
ssh localhost
echo $DISPLAY
localhost:10.0

I understand this is more of a SSH and X11 related issue, but X11 forwarding
is working over SSH in my standalone programs, but not in MR program. That's
why I am posting the problem here.

Any help is greatly appreciated.

Thanks,
Tarandeep

Search Discussions

  • Vladimir Klimontovich at Jan 17, 2010 at 9:58 pm ⇧
    Maybe, hadoop running MR jobs using different user? For example, if you followed installation instructions
    from official site or used rpm/deb packages hadoop using "hadoop" user to run jobs. And you probably using different user
    for running your test program.
    On Jan 18, 2010, at 12:41 AM, Tarandeep Singh wrote:

    Hi,

    I am running a MR job that requires usage of some java.awt.* classes, that
    can't be run in headless mode.

    Right now, I am running Hadoop in a single node cluster (my laptop) which
    has X11 server running. I have set up my ssh server and client to do X11
    forwarding.

    I ran the following java program to ensure that X11 forwarding is working-

    public class Test
    throws Exception
    {
    public static void main( String[] args)
    {
    Process p = runtime.getRuntime( ).exec( "/usr/bin/xterm");
    p.waitFor( );
    }
    }

    I ran it as-
    ssh localhost "java Test"
    and it worked confirming that X11 forwarding is working over SSH.

    However, when I run a Map Reduce program that uses java.awt.* classes
    (trying to create an object that extends Frame), I keep getting this error-

    java.lang.InternalError- Can't connect to X11 window server using
    'localhost:10.0' as the value of the DISPLAY variable

    I printed the value of DISPLAY variable-

    echo $DISPLAY
    :0.0
    ssh localhost
    echo $DISPLAY
    localhost:10.0

    I understand this is more of a SSH and X11 related issue, but X11 forwarding
    is working over SSH in my standalone programs, but not in MR program. That's
    why I am posting the problem here.

    Any help is greatly appreciated.

    Thanks,
    Tarandeep
    ---
    Vladimir Klimontovich,
    skype: klimontovich
    GoogleTalk/Jabber: klimontovich@gmail.com
    Cell phone: +7926 890 2349
  • Wang Xu at Jan 18, 2010 at 2:12 am ⇧
    Some Java program is hard to run remotely via ssh tunnel, but I do not
    know the detail either.

    On Mon, Jan 18, 2010 at 5:57 AM, Vladimir Klimontovich
    wrote:
    Maybe, hadoop running MR jobs using different user? For example, if you followed installation instructions
    from official site or used rpm/deb packages hadoop using "hadoop" user to run jobs. And you probably using different user
    for running your test program.
    On Jan 18, 2010, at 12:41 AM, Tarandeep Singh wrote:

    Hi,

    I am running a MR job that requires usage of some java.awt.* classes, that
    can't be run in headless mode.

    Right now, I am running Hadoop in a single node cluster (my laptop) which
    has X11 server running. I have set up my ssh server and client to do X11
    forwarding.

    I ran the following java program to ensure that X11 forwarding is working-

    public class Test
    throws Exception
    {
    public static void main( String[] args)
    {
    Process p = runtime.getRuntime( ).exec( "/usr/bin/xterm");
    p.waitFor( );
    }
    }

    I ran it as-
    ssh localhost "java Test"
    and it worked confirming that X11 forwarding is working over SSH.

    However, when I run a Map Reduce program that uses java.awt.* classes
    (trying to create an object that extends Frame), I keep getting this error-

    java.lang.InternalError- Can't connect to X11 window server using
    'localhost:10.0' as the value of the DISPLAY variable

    I printed the value of DISPLAY variable-

    echo $DISPLAY
    :0.0
    ssh localhost
    echo $DISPLAY
    localhost:10.0

    I understand this is more of a SSH and X11 related issue, but X11 forwarding
    is working over SSH in my standalone programs, but not in MR program. That's
    why I am posting the problem here.

    Any help is greatly appreciated.

    Thanks,
    Tarandeep
    ---
    Vladimir Klimontovich,
    skype: klimontovich
    GoogleTalk/Jabber: klimontovich@gmail.com
    Cell phone: +7926 890 2349


    --
    Wang Xu
    Ted Turner - "Sports is like a war without the killing." -
    http://www.brainyquote.com/quotes/authors/t/ted_turner.html
  • Tarandeep Singh at Jan 18, 2010 at 2:20 am ⇧

    On Sun, Jan 17, 2010 at 1:57 PM, Vladimir Klimontovich wrote:

    Maybe, hadoop running MR jobs using different user? For example, if you
    followed installation instructions
    from official site or used rpm/deb packages hadoop using "hadoop" user to
    run jobs. And you probably using different user
    for running your test program.
    Thanks Vladimir..
    I am not running hadoop as the "hadoop" user. Both my test program and
    hadoop are run using same user.
    From the exception, it is clear the DISPLAY variable is set properly, but
    java is not able to connect to the X server (not sure either some permission
    issues or what)


    On Jan 18, 2010, at 12:41 AM, Tarandeep Singh wrote:

    Hi,

    I am running a MR job that requires usage of some java.awt.* classes, that
    can't be run in headless mode.

    Right now, I am running Hadoop in a single node cluster (my laptop) which
    has X11 server running. I have set up my ssh server and client to do X11
    forwarding.

    I ran the following java program to ensure that X11 forwarding is working-
    public class Test
    throws Exception
    {
    public static void main( String[] args)
    {
    Process p = runtime.getRuntime( ).exec( "/usr/bin/xterm");
    p.waitFor( );
    }
    }

    I ran it as-
    ssh localhost "java Test"
    and it worked confirming that X11 forwarding is working over SSH.

    However, when I run a Map Reduce program that uses java.awt.* classes
    (trying to create an object that extends Frame), I keep getting this error-
    java.lang.InternalError- Can't connect to X11 window server using
    'localhost:10.0' as the value of the DISPLAY variable

    I printed the value of DISPLAY variable-

    echo $DISPLAY
    :0.0
    ssh localhost
    echo $DISPLAY
    localhost:10.0

    I understand this is more of a SSH and X11 related issue, but X11
    forwarding
    is working over SSH in my standalone programs, but not in MR program. That's
    why I am posting the problem here.

    Any help is greatly appreciated.

    Thanks,
    Tarandeep
    ---
    Vladimir Klimontovich,
    skype: klimontovich
    GoogleTalk/Jabber: klimontovich@gmail.com
    Cell phone: +7926 890 2349
  • Todd Lipcon at Jan 18, 2010 at 2:47 am ⇧

    On Sun, Jan 17, 2010 at 6:19 PM, Tarandeep Singh wrote:

    On Sun, Jan 17, 2010 at 1:57 PM, Vladimir Klimontovich <
    klimontovich@gmail.com> wrote:
    Maybe, hadoop running MR jobs using different user? For example, if you
    followed installation instructions
    from official site or used rpm/deb packages hadoop using "hadoop" user to
    run jobs. And you probably using different user
    for running your test program.
    Thanks Vladimir..
    I am not running hadoop as the "hadoop" user. Both my test program and
    hadoop are run using same user.
    From the exception, it is clear the DISPLAY variable is set properly, but
    java is not able to connect to the X server (not sure either some
    permission
    issues or what)
    My guess is you're either missing the XAUTHORITY variable, or it's pointed
    at a file that the user executing the task can't read.

    (as a side note, this seems like a really bad idea - why are your hadoop
    tasks trying to talk to an X11 server in the first place?)

    -Todd


    On Jan 18, 2010, at 12:41 AM, Tarandeep Singh wrote:

    Hi,

    I am running a MR job that requires usage of some java.awt.* classes, that
    can't be run in headless mode.

    Right now, I am running Hadoop in a single node cluster (my laptop)
    which
    has X11 server running. I have set up my ssh server and client to do
    X11
    forwarding.

    I ran the following java program to ensure that X11 forwarding is working-
    public class Test
    throws Exception
    {
    public static void main( String[] args)
    {
    Process p = runtime.getRuntime( ).exec( "/usr/bin/xterm");
    p.waitFor( );
    }
    }

    I ran it as-
    ssh localhost "java Test"
    and it worked confirming that X11 forwarding is working over SSH.

    However, when I run a Map Reduce program that uses java.awt.* classes
    (trying to create an object that extends Frame), I keep getting this error-
    java.lang.InternalError- Can't connect to X11 window server using
    'localhost:10.0' as the value of the DISPLAY variable

    I printed the value of DISPLAY variable-

    echo $DISPLAY
    :0.0
    ssh localhost
    echo $DISPLAY
    localhost:10.0

    I understand this is more of a SSH and X11 related issue, but X11
    forwarding
    is working over SSH in my standalone programs, but not in MR program. That's
    why I am posting the problem here.

    Any help is greatly appreciated.

    Thanks,
    Tarandeep
    ---
    Vladimir Klimontovich,
    skype: klimontovich
    GoogleTalk/Jabber: klimontovich@gmail.com
    Cell phone: +7926 890 2349
  • Edward Capriolo at Jan 18, 2010 at 2:53 am ⇧

    On Sun, Jan 17, 2010 at 9:46 PM, Todd Lipcon wrote:
    On Sun, Jan 17, 2010 at 6:19 PM, Tarandeep Singh wrote:

    On Sun, Jan 17, 2010 at 1:57 PM, Vladimir Klimontovich <
    klimontovich@gmail.com> wrote:
    Maybe, hadoop running MR jobs using different user? For example, if you
    followed installation instructions
    from official site or used rpm/deb packages hadoop using "hadoop" user to
    run jobs. And you probably using different user
    for running your test program.
    Thanks Vladimir..
    I am not running hadoop as the "hadoop" user. Both my test program and
    hadoop are run using same user.
    From the exception, it is clear the DISPLAY variable is set properly, but
    java is not able to connect to the X server (not sure either some
    permission
    issues or what)
    My guess is you're either missing the XAUTHORITY variable, or it's pointed
    at a file that the user executing the task can't read.

    (as a side note, this seems like a really bad idea - why are your hadoop
    tasks trying to talk to an X11 server in the first place?)

    -Todd


    On Jan 18, 2010, at 12:41 AM, Tarandeep Singh wrote:

    Hi,

    I am running a MR job that requires usage of some java.awt.* classes, that
    can't be run in headless mode.

    Right now, I am running Hadoop in a single node cluster (my laptop)
    which
    has X11 server running. I have set up my ssh server and client to do
    X11
    forwarding.

    I ran the following java program to ensure that X11 forwarding is working-
    public class Test
    throws Exception
    {
    public static void main( String[] args)
    {
    Process p = runtime.getRuntime( ).exec( "/usr/bin/xterm");
    p.waitFor( );
    }
    }

    I ran it as-
    ssh localhost "java Test"
    and it worked confirming that X11 forwarding is working over SSH.

    However, when I run a Map Reduce program that uses java.awt.* classes
    (trying to create an object that extends Frame), I keep getting this error-
    java.lang.InternalError- Can't connect to X11 window server using
    'localhost:10.0' as the value of the DISPLAY variable

    I printed the value of DISPLAY variable-

    echo $DISPLAY
    :0.0
    ssh localhost
    echo $DISPLAY
    localhost:10.0

    I understand this is more of a SSH and X11 related issue, but X11
    forwarding
    is working over SSH in my standalone programs, but not in MR program. That's
    why I am posting the problem here.

    Any help is greatly appreciated.

    Thanks,
    Tarandeep
    ---
    Vladimir Klimontovich,
    skype: klimontovich
    GoogleTalk/Jabber: klimontovich@gmail.com
    Cell phone: +7926 890 2349
    (as a side note, this seems like a really bad idea - why are your hadoop
    tasks trying to talk to an X11 server in the first place?)

    If he is using java.awt, that windowing needs an xserver to draw.
    going to java.swing with headless might help. I believe the entire
    swing API can work in headless mode, but that would involve a re-code.
  • Steve Loughran at Jan 18, 2010 at 10:53 am ⇧

    Tarandeep Singh wrote:
    Hi,

    I am running a MR job that requires usage of some java.awt.* classes, that
    can't be run in headless mode.

    Right now, I am running Hadoop in a single node cluster (my laptop) which
    has X11 server running. I have set up my ssh server and client to do X11
    forwarding.

    I ran the following java program to ensure that X11 forwarding is working-
    the problem here is that you need to tell AWT to run headless instead of
    expecting an X11 server to hand. Which means setting
    java.awt.headless=true in the processes that are using AWT to do bitmap
    rendering.

    see
    http://java.sun.com/developer/technicalArticles/J2SE/Desktop/headless/
    for details
  • Tarandeep Singh at Jan 18, 2010 at 6:43 pm ⇧

    On Mon, Jan 18, 2010 at 2:52 AM, Steve Loughran wrote:

    Tarandeep Singh wrote:
    Hi,

    I am running a MR job that requires usage of some java.awt.* classes, that
    can't be run in headless mode.

    Right now, I am running Hadoop in a single node cluster (my laptop) which
    has X11 server running. I have set up my ssh server and client to do X11
    forwarding.

    I ran the following java program to ensure that X11 forwarding is working-
    the problem here is that you need to tell AWT to run headless instead of
    expecting an X11 server to hand. Which means setting java.awt.headless=true
    in the processes that are using AWT to do bitmap rendering.

    see http://java.sun.com/developer/technicalArticles/J2SE/Desktop/headless/for details
    I am using Frame class and it can't be run in headless mode.

    Suggestion of Brien is useful (running x virtual frame buffer) and I guess I
    am going to do the same on my servers but right now I am testing on my
    laptop which is running X server.

    The problem here is, the test program runs over ssh without any problem but
    when I run the map reduce program I keep getting error. Both the standalone
    program and MR program are run as the same user.

    I followed Todd's suggestion and checked the value of XAUTHORITY environment
    variable. It turns out, the value of this variable is not set when I do ssh.
    So I am trying to see if I can set its value and if the MR program runs
    after that. But if this is the problem then the standalone program should
    also not run.

    Thanks,
    Tarandeep
  • Steve Loughran at Jan 19, 2010 at 3:15 pm ⇧

    Tarandeep Singh wrote:
    I am using Frame class and it can't be run in headless mode.

    Suggestion of Brien is useful (running x virtual frame buffer) and I guess I
    am going to do the same on my servers but right now I am testing on my
    laptop which is running X server.

    The problem here is, the test program runs over ssh without any problem but
    when I run the map reduce program I keep getting error. Both the standalone
    program and MR program are run as the same user.

    I followed Todd's suggestion and checked the value of XAUTHORITY environment
    variable. It turns out, the value of this variable is not set when I do ssh.
    So I am trying to see if I can set its value and if the MR program runs
    after that. But if this is the problem then the standalone program should
    also not run.
    logged in as the user at the console, set all access open:

    xhost +

    This should stop the xauthority stuff mattering, leaving only the
    DISPLAY env variable as the binding issue
  • Allen Wittenauer at Jan 19, 2010 at 9:54 pm ⇧

    On 1/18/10 10:42 AM, "Tarandeep Singh" wrote:

    The problem here is, the test program runs over ssh without any problem but
    when I run the map reduce program I keep getting error. Both the standalone
    program and MR program are run as the same user.
    My vote is for "ssh is forwarding the X11 connection".
  • Brien colwell at Jan 18, 2010 at 2:22 pm ⇧
    From memory, some parts of AWT won't run in headless mode. I used to run an
    x virtual frame buffer on servers that created graphics. It's a standard
    package on most Linux distros. I forget if there was something special
    needed to set it up, but might be worth looking into.



    On Sun, Jan 17, 2010 at 4:41 PM, Tarandeep Singh wrote:

    Hi,

    I am running a MR job that requires usage of some java.awt.* classes, that
    can't be run in headless mode.

    Right now, I am running Hadoop in a single node cluster (my laptop) which
    has X11 server running. I have set up my ssh server and client to do X11
    forwarding.

    I ran the following java program to ensure that X11 forwarding is working-

    public class Test
    throws Exception
    {
    public static void main( String[] args)
    {
    Process p = runtime.getRuntime( ).exec( "/usr/bin/xterm");
    p.waitFor( );
    }
    }

    I ran it as-
    ssh localhost "java Test"
    and it worked confirming that X11 forwarding is working over SSH.

    However, when I run a Map Reduce program that uses java.awt.* classes
    (trying to create an object that extends Frame), I keep getting this error-

    java.lang.InternalError- Can't connect to X11 window server using
    'localhost:10.0' as the value of the DISPLAY variable

    I printed the value of DISPLAY variable-

    echo $DISPLAY
    :0.0
    ssh localhost
    echo $DISPLAY
    localhost:10.0

    I understand this is more of a SSH and X11 related issue, but X11
    forwarding
    is working over SSH in my standalone programs, but not in MR program.
    That's
    why I am posting the problem here.

    Any help is greatly appreciated.

    Thanks,
    Tarandeep

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJan 17, '10 at 9:42p
activeJan 19, '10 at 9:54p
posts11
users8
websitehadoop.apache.org...
irc#hadoop

8 users in discussion

Tarandeep Singh: 3 posts Steve Loughran: 2 posts Edward Capriolo: 1 post Wang Xu: 1 post Allen Wittenauer: 1 post Vladimir Klimontovich: 1 post Todd Lipcon: 1 post Brien colwell: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase