FAQ
Hello all

I am new to hadoop .

I am trying to write file to single cluster and getting this exception when
i am trying to close output stream

java.io.IOException: CreateProcess: df -k
C:\usr\local\hadoop-datastore\hadoop-hadoop\dfs\tmp error=2
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
at java.lang.ProcessBuilder.start(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at org.apache.hadoop.fs.DF.doDF(DF.java:60)
at org.apache.hadoop.fs.DF.(LocalDirAllocator.java:198)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:235)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:276)
at
org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:155)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java:1475)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.openBackupStream(DFSClient.java:1442)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1600)
at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1739)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:49)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
at Test1.main(Test1.java:23)



My test is:

Configuration configuration = new Configuration();
FileSystem fileSystem =
FileSystem.get(configuration);
Path path = new Path("/testfile");
//writing:
FSDataOutputStream dataOutputStream = fileSystem.create(path);
dataOutputStream.writeUTF("hello world");
dataOutputStream.close();
//reading
FSDataInputStream dataInputStream = fileSystem.open(path);
System.out.println(dataInputStream.readUTF());
dataInputStream.close();
fileSystem.close();

i added hadoop-site.xml to classpath :


<configuration>

<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-datastore/hadoop-hadoop</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://172.16.50.13:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>

<property>
<name>mapred.job.tracker</name>
<value>172.16.50.13:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>

<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is
created.
The default is used if replication is not specified in create time.
</description>
</property>
</configuration>



Please help me
thx

--
View this message in context: http://www.nabble.com/writing-file-tf4618565.html#a13190497
Sent from the Hadoop Users mailing list archive at Nabble.com.

Search Discussions

  • Ted Dunning at Oct 13, 2007 at 5:31 pm
    Hadoop is having trouble spawning a process to measure how much disk space
    you have available.

    It really is much easier to run hadoop on Linux. In case you can't do that,
    make sure that you really do have cygwin installed in a standard way on each
    hadoop node.

    On 10/13/07 8:05 AM, "dcave555" wrote:


    Hello all

    I am new to hadoop .

    I am trying to write file to single cluster and getting this exception when
    i am trying to close output stream

    java.io.IOException: CreateProcess: df -k
    C:\usr\local\hadoop-datastore\hadoop-hadoop\dfs\tmp error=2
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    at java.lang.ProcessBuilder.start(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at org.apache.hadoop.fs.DF.doDF(DF.java:60)
    at org.apache.hadoop.fs.DF.<init>(DF.java:53)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDi
    rAllocator.java:198)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrit
    e(LocalDirAllocator.java:235)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWri
    te(LocalDirAllocator.java:276)
    at
    org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator
    .java:155)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java:1
    475)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.openBackupStream(DFSClient.jav
    a:1442)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1600>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140>
    )
    at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1739)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream
    .java:49)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
    at Test1.main(Test1.java:23)



    My test is:

    Configuration configuration = new Configuration();
    FileSystem fileSystem =
    FileSystem.get(configuration);
    Path path = new Path("/testfile");
    //writing:
    FSDataOutputStream dataOutputStream = fileSystem.create(path);
    dataOutputStream.writeUTF("hello world");
    dataOutputStream.close();
    //reading
    FSDataInputStream dataInputStream = fileSystem.open(path);
    System.out.println(dataInputStream.readUTF());
    dataInputStream.close();
    fileSystem.close();

    i added hadoop-site.xml to classpath :


    <configuration>

    <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://172.16.50.13:54310</value>
    <description>The name of the default file system. A URI whose
    scheme and authority determine the FileSystem implementation. The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class. The uri's authority is used to
    determine the host, port, etc. for a filesystem.</description>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>172.16.50.13:54311</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication.
    The actual number of replications can be specified when the file is
    created.
    The default is used if replication is not specified in create time.
    </description>
    </property>
    </configuration>



    Please help me
    thx
  • Dcave555 at Oct 13, 2007 at 6:44 pm
    Thx for your reply

    But i think i don't understand something.
    i have installed hadoop single node on ubuntu.
    my test is running on windows.

    i changed property in hadoop-site.xml in my application to

    <property>
    <name>hadoop.tmp.dir</name>
    <value>//172.16.50.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    amd i am getting Login failed.

    If i want to save file from windows to hadoop on ubuntu i need to import ssh
    public key?
    how can i tell my test application on windows to use this pulic key?

    Or i need to do something else?

    Thx





    Ted Dunning-3 wrote:

    Hadoop is having trouble spawning a process to measure how much disk space
    you have available.

    It really is much easier to run hadoop on Linux. In case you can't do
    that,
    make sure that you really do have cygwin installed in a standard way on
    each
    hadoop node.

    On 10/13/07 8:05 AM, "dcave555" wrote:


    Hello all

    I am new to hadoop .

    I am trying to write file to single cluster and getting this exception
    when
    i am trying to close output stream

    java.io.IOException: CreateProcess: df -k
    C:\usr\local\hadoop-datastore\hadoop-hadoop\dfs\tmp error=2
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    at java.lang.ProcessBuilder.start(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at org.apache.hadoop.fs.DF.doDF(DF.java:60)
    at org.apache.hadoop.fs.DF.<init>(DF.java:53)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDi
    rAllocator.java:198)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrit
    e(LocalDirAllocator.java:235)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWri
    te(LocalDirAllocator.java:276)
    at
    org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator
    .java:155)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java:1
    475)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.openBackupStream(DFSClient.jav
    a:1442)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1600>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1739)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream
    .java:49)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
    at Test1.main(Test1.java:23)



    My test is:

    Configuration configuration = new
    Configuration();
    FileSystem fileSystem =
    FileSystem.get(configuration);
    Path path = new Path("/testfile");
    //writing:
    FSDataOutputStream dataOutputStream = fileSystem.create(path);
    dataOutputStream.writeUTF("hello world");
    dataOutputStream.close();
    //reading
    FSDataInputStream dataInputStream = fileSystem.open(path);
    System.out.println(dataInputStream.readUTF());
    dataInputStream.close();
    fileSystem.close();

    i added hadoop-site.xml to classpath :


    <configuration>

    <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://172.16.50.13:54310</value>
    <description>The name of the default file system. A URI whose
    scheme and authority determine the FileSystem implementation. The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class. The uri's authority is used to
    determine the host, port, etc. for a filesystem.</description>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>172.16.50.13:54311</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication.
    The actual number of replications can be specified when the file is
    created.
    The default is used if replication is not specified in create time.
    </description>
    </property>
    </configuration>



    Please help me
    thx
    --
    View this message in context: http://www.nabble.com/writing-file-tf4618565.html#a13192358
    Sent from the Hadoop Users mailing list archive at Nabble.com.
  • Ted Dunning at Oct 13, 2007 at 8:18 pm
    Your program may be trying to run locally on your windows box instead of
    talking to the ubuntu micro cluster.

    On 10/13/07 11:43 AM, "dcave555" wrote:


    Thx for your reply

    But i think i don't understand something.
    i have installed hadoop single node on ubuntu.
    my test is running on windows.

    i changed property in hadoop-site.xml in my application to

    <property>
    <name>hadoop.tmp.dir</name>
    <value>//172.16.50.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    amd i am getting Login failed.

    If i want to save file from windows to hadoop on ubuntu i need to import ssh
    public key?
    how can i tell my test application on windows to use this pulic key?

    Or i need to do something else?

    Thx





    Ted Dunning-3 wrote:

    Hadoop is having trouble spawning a process to measure how much disk space
    you have available.

    It really is much easier to run hadoop on Linux. In case you can't do
    that,
    make sure that you really do have cygwin installed in a standard way on
    each
    hadoop node.

    On 10/13/07 8:05 AM, "dcave555" wrote:


    Hello all

    I am new to hadoop .

    I am trying to write file to single cluster and getting this exception
    when
    i am trying to close output stream

    java.io.IOException: CreateProcess: df -k
    C:\usr\local\hadoop-datastore\hadoop-hadoop\dfs\tmp error=2
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    at java.lang.ProcessBuilder.start(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at org.apache.hadoop.fs.DF.doDF(DF.java:60)
    at org.apache.hadoop.fs.DF.<init>(DF.java:53)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(Local
    Di
    rAllocator.java:198)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWr
    it
    e(LocalDirAllocator.java:235)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForW
    ri
    te(LocalDirAllocator.java:276)
    at
    org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocat
    or
    .java:155)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java
    :1
    475)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.openBackupStream(DFSClient.j
    av
    a:1442)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:160
    0>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:14
    0>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1739)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStre
    am
    .java:49)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
    at Test1.main(Test1.java:23)



    My test is:

    Configuration configuration = new
    Configuration();
    FileSystem fileSystem =
    FileSystem.get(configuration);
    Path path = new Path("/testfile");
    //writing:
    FSDataOutputStream dataOutputStream = fileSystem.create(path);
    dataOutputStream.writeUTF("hello world");
    dataOutputStream.close();
    //reading
    FSDataInputStream dataInputStream = fileSystem.open(path);
    System.out.println(dataInputStream.readUTF());
    dataInputStream.close();
    fileSystem.close();

    i added hadoop-site.xml to classpath :


    <configuration>

    <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://172.16.50.13:54310</value>
    <description>The name of the default file system. A URI whose
    scheme and authority determine the FileSystem implementation. The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class. The uri's authority is used to
    determine the host, port, etc. for a filesystem.</description>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>172.16.50.13:54311</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication.
    The actual number of replications can be specified when the file is
    created.
    The default is used if replication is not specified in create time.
    </description>
    </property>
    </configuration>



    Please help me
    thx
  • Dcave555 at Oct 13, 2007 at 9:53 pm
    Thx again for your reply

    Ok maybe you right

    I have my ubuntu hadoop node on ip 168.13.51.13

    So i write in hadoop-site.xml (windows application)

    <configuration>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>//168.13.51.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://168.13.51.13:54310</value>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>168.13.51.13:54311</value>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    </configuration>


    The same xml i have on ubuntu machine.


    after running my application i have exception:

    Login failded.

    i suspect that i have problem with ssh public key

    so i think i need to copy ssh public key to windows machine

    but how can i tell to my application to use this key?

    Thx again


    Ted Dunning-3 wrote:

    Your program may be trying to run locally on your windows box instead of
    talking to the ubuntu micro cluster.

    On 10/13/07 11:43 AM, "dcave555" wrote:


    Thx for your reply

    But i think i don't understand something.
    i have installed hadoop single node on ubuntu.
    my test is running on windows.

    i changed property in hadoop-site.xml in my application to

    <property>
    <name>hadoop.tmp.dir</name>
    <value>//172.16.50.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    amd i am getting Login failed.

    If i want to save file from windows to hadoop on ubuntu i need to import
    ssh
    public key?
    how can i tell my test application on windows to use this pulic key?

    Or i need to do something else?

    Thx





    Ted Dunning-3 wrote:

    Hadoop is having trouble spawning a process to measure how much disk
    space
    you have available.

    It really is much easier to run hadoop on Linux. In case you can't do
    that,
    make sure that you really do have cygwin installed in a standard way on
    each
    hadoop node.

    On 10/13/07 8:05 AM, "dcave555" wrote:


    Hello all

    I am new to hadoop .

    I am trying to write file to single cluster and getting this exception
    when
    i am trying to close output stream

    java.io.IOException: CreateProcess: df -k
    C:\usr\local\hadoop-datastore\hadoop-hadoop\dfs\tmp error=2
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    at java.lang.ProcessBuilder.start(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at org.apache.hadoop.fs.DF.doDF(DF.java:60)
    at org.apache.hadoop.fs.DF.<init>(DF.java:53)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(Local
    Di
    rAllocator.java:198)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWr
    it
    e(LocalDirAllocator.java:235)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForW
    ri
    te(LocalDirAllocator.java:276)
    at
    org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocat
    or
    .java:155)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java
    :1
    475)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.openBackupStream(DFSClient.j
    av
    a:1442)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:160
    0>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:14
    0>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1739)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStre
    am
    .java:49)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
    at Test1.main(Test1.java:23)



    My test is:

    Configuration configuration = new
    Configuration();
    FileSystem fileSystem =
    FileSystem.get(configuration);
    Path path = new Path("/testfile");
    //writing:
    FSDataOutputStream dataOutputStream = fileSystem.create(path);
    dataOutputStream.writeUTF("hello world");
    dataOutputStream.close();
    //reading
    FSDataInputStream dataInputStream = fileSystem.open(path);
    System.out.println(dataInputStream.readUTF());
    dataInputStream.close();
    fileSystem.close();

    i added hadoop-site.xml to classpath :


    <configuration>

    <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://172.16.50.13:54310</value>
    <description>The name of the default file system. A URI whose
    scheme and authority determine the FileSystem implementation. The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class. The uri's authority is used to
    determine the host, port, etc. for a filesystem.</description>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>172.16.50.13:54311</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication.
    The actual number of replications can be specified when the file is
    created.
    The default is used if replication is not specified in create time.
    </description>
    </property>
    </configuration>



    Please help me
    thx
    --
    View this message in context: http://www.nabble.com/writing-file-tf4618565.html#a13193822
    Sent from the Hadoop Users mailing list archive at Nabble.com.
  • Aaron Kimball at Oct 13, 2007 at 10:14 pm
    That depends on your ssh implementation. Most will look at the file
    ~/.ssh/id_dsa.pub or id_rsa.pub in the same directory. You can generate
    your keys on the linux machine ('man ssh-keygen' for more info), which
    will generate files named id_Xsa and id_Xsa.pub (Depending on the key
    type). The private key file goes in ~/.ssh/ on the Linux machine. The
    public key file (should also be copied there), and goes in the specified
    directory on your Windows machine depending on what ssh client you're
    using. (If you're using cygwin's, ~/.ssh/ should be the correct one.)

    If you have a sane cygwin install, 'man ssh' for more.

    - Aaron

    dcave555 wrote:
    Thx again for your reply

    Ok maybe you right

    I have my ubuntu hadoop node on ip 168.13.51.13

    So i write in hadoop-site.xml (windows application)

    <configuration>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>//168.13.51.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://168.13.51.13:54310</value>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>168.13.51.13:54311</value>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    </configuration>


    The same xml i have on ubuntu machine.


    after running my application i have exception:

    Login failded.

    i suspect that i have problem with ssh public key

    so i think i need to copy ssh public key to windows machine

    but how can i tell to my application to use this key?

    Thx again


    Ted Dunning-3 wrote:
    Your program may be trying to run locally on your windows box instead of
    talking to the ubuntu micro cluster.

    On 10/13/07 11:43 AM, "dcave555" wrote:

    Thx for your reply

    But i think i don't understand something.
    i have installed hadoop single node on ubuntu.
    my test is running on windows.

    i changed property in hadoop-site.xml in my application to

    <property>
    <name>hadoop.tmp.dir</name>
    <value>//172.16.50.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    amd i am getting Login failed.

    If i want to save file from windows to hadoop on ubuntu i need to import
    ssh
    public key?
    how can i tell my test application on windows to use this pulic key?

    Or i need to do something else?

    Thx





    Ted Dunning-3 wrote:
    Hadoop is having trouble spawning a process to measure how much disk
    space
    you have available.

    It really is much easier to run hadoop on Linux. In case you can't do
    that,
    make sure that you really do have cygwin installed in a standard way on
    each
    hadoop node.

    On 10/13/07 8:05 AM, "dcave555" wrote:

    Hello all

    I am new to hadoop .

    I am trying to write file to single cluster and getting this exception
    when
    i am trying to close output stream

    java.io.IOException: CreateProcess: df -k
    C:\usr\local\hadoop-datastore\hadoop-hadoop\dfs\tmp error=2
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    at java.lang.ProcessBuilder.start(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at org.apache.hadoop.fs.DF.doDF(DF.java:60)
    at org.apache.hadoop.fs.DF.<init>(DF.java:53)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(Local
    Di
    rAllocator.java:198)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWr
    it
    e(LocalDirAllocator.java:235)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForW
    ri
    te(LocalDirAllocator.java:276)
    at
    org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocat
    or
    .java:155)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java
    :1
    475)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.openBackupStream(DFSClient.j
    av
    a:1442)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:160
    0>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:14
    0>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1739)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStre
    am
    .java:49)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
    at Test1.main(Test1.java:23)



    My test is:

    Configuration configuration = new
    Configuration();
    FileSystem fileSystem =
    FileSystem.get(configuration);
    Path path = new Path("/testfile");
    //writing:
    FSDataOutputStream dataOutputStream = fileSystem.create(path);
    dataOutputStream.writeUTF("hello world");
    dataOutputStream.close();
    //reading
    FSDataInputStream dataInputStream = fileSystem.open(path);
    System.out.println(dataInputStream.readUTF());
    dataInputStream.close();
    fileSystem.close();

    i added hadoop-site.xml to classpath :


    <configuration>

    <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://172.16.50.13:54310</value>
    <description>The name of the default file system. A URI whose
    scheme and authority determine the FileSystem implementation. The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class. The uri's authority is used to
    determine the host, port, etc. for a filesystem.</description>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>172.16.50.13:54311</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication.
    The actual number of replications can be specified when the file is
    created.
    The default is used if replication is not specified in create time.
    </description>
    </property>
    </configuration>



    Please help me
    thx
  • Dcave555 at Oct 13, 2007 at 10:21 pm
    Ok

    I am running my Test on Eclipse IDE

    tomorow at work i will try to copy pulic key to output directory and then
    try again to run my application

    maybe it will work.

    Thx for your reply.




    Aaron Kimball wrote:
    That depends on your ssh implementation. Most will look at the file
    ~/.ssh/id_dsa.pub or id_rsa.pub in the same directory. You can generate
    your keys on the linux machine ('man ssh-keygen' for more info), which
    will generate files named id_Xsa and id_Xsa.pub (Depending on the key
    type). The private key file goes in ~/.ssh/ on the Linux machine. The
    public key file (should also be copied there), and goes in the specified
    directory on your Windows machine depending on what ssh client you're
    using. (If you're using cygwin's, ~/.ssh/ should be the correct one.)

    If you have a sane cygwin install, 'man ssh' for more.

    - Aaron

    dcave555 wrote:
    Thx again for your reply

    Ok maybe you right

    I have my ubuntu hadoop node on ip 168.13.51.13

    So i write in hadoop-site.xml (windows application)

    <configuration>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>//168.13.51.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://168.13.51.13:54310</value>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>168.13.51.13:54311</value>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    </configuration>


    The same xml i have on ubuntu machine.


    after running my application i have exception:

    Login failded.

    i suspect that i have problem with ssh public key

    so i think i need to copy ssh public key to windows machine

    but how can i tell to my application to use this key?

    Thx again


    Ted Dunning-3 wrote:
    Your program may be trying to run locally on your windows box instead of
    talking to the ubuntu micro cluster.

    On 10/13/07 11:43 AM, "dcave555" wrote:

    Thx for your reply

    But i think i don't understand something.
    i have installed hadoop single node on ubuntu.
    my test is running on windows.

    i changed property in hadoop-site.xml in my application to

    <property>
    <name>hadoop.tmp.dir</name>

    <value>//172.16.50.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    amd i am getting Login failed.

    If i want to save file from windows to hadoop on ubuntu i need to
    import
    ssh
    public key?
    how can i tell my test application on windows to use this pulic key?

    Or i need to do something else?

    Thx





    Ted Dunning-3 wrote:
    Hadoop is having trouble spawning a process to measure how much disk
    space
    you have available.

    It really is much easier to run hadoop on Linux. In case you can't do
    that,
    make sure that you really do have cygwin installed in a standard way
    on
    each
    hadoop node.

    On 10/13/07 8:05 AM, "dcave555" wrote:

    Hello all

    I am new to hadoop .

    I am trying to write file to single cluster and getting this
    exception
    when
    i am trying to close output stream

    java.io.IOException: CreateProcess: df -k
    C:\usr\local\hadoop-datastore\hadoop-hadoop\dfs\tmp error=2
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    at java.lang.ProcessBuilder.start(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at java.lang.Runtime.exec(Unknown Source)
    at org.apache.hadoop.fs.DF.doDF(DF.java:60)
    at org.apache.hadoop.fs.DF.<init>(DF.java:53)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(Local
    Di
    rAllocator.java:198)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWr
    it
    e(LocalDirAllocator.java:235)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForW
    ri
    te(LocalDirAllocator.java:276)
    at
    org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocat
    or
    .java:155)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java
    :1
    475)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.openBackupStream(DFSClient.j
    av
    a:1442)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:160
    0>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:14
    0>
    )
    at
    org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1739)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStre
    am
    .java:49)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
    at Test1.main(Test1.java:23)



    My test is:

    Configuration configuration = new
    Configuration();
    FileSystem fileSystem =
    FileSystem.get(configuration);
    Path path = new Path("/testfile");
    //writing:
    FSDataOutputStream dataOutputStream = fileSystem.create(path);
    dataOutputStream.writeUTF("hello world");
    dataOutputStream.close();
    //reading
    FSDataInputStream dataInputStream = fileSystem.open(path);
    System.out.println(dataInputStream.readUTF());
    dataInputStream.close();
    fileSystem.close();

    i added hadoop-site.xml to classpath :


    <configuration>

    <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop-datastore/hadoop-hadoop</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://172.16.50.13:54310</value>
    <description>The name of the default file system. A URI whose
    scheme and authority determine the FileSystem implementation. The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class. The uri's authority is used
    to
    determine the host, port, etc. for a filesystem.</description>
    </property>

    <property>
    <name>mapred.job.tracker</name>
    <value>172.16.50.13:54311</value>
    <description>The host and port that the MapReduce job tracker runs
    at. If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
    </property>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication.
    The actual number of replications can be specified when the file is
    created.
    The default is used if replication is not specified in create time.
    </description>
    </property>
    </configuration>



    Please help me
    thx
    --
    View this message in context: http://www.nabble.com/writing-file-tf4618565.html#a13194049
    Sent from the Hadoop Users mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 13, '07 at 3:06p
activeOct 13, '07 at 10:21p
posts7
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase