FAQ
Which version of Camel are you using?

Try using the 2.5 versions as we have changed how the FTP component scan for
files.
--
View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2807611.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Search Discussions

  • Lorrin at Sep 28, 2010 at 5:36 pm
    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.
  • Claus Ibsen at Sep 29, 2010 at 9:47 am
    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Lorrin Nelson at Sep 29, 2010 at 5:48 pm
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Claus Ibsen at Sep 29, 2010 at 7:33 pm
    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Lorrin Nelson at Sep 29, 2010 at 9:55 pm
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Claus Ibsen at Sep 30, 2010 at 4:38 am
    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Lorrin Nelson at Oct 1, 2010 at 8:46 pm
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Claus Ibsen at Oct 2, 2010 at 7:15 am
    Hi Loririn

    I think you/we have to step back. We have other users in the past
    running with SFTP with no problems at all. Also fetching files
    recursively.
    So I wonder if we should revert back and keep the old logic!

    1)
    Is this the first time you use camel-ftp?

    2)
    What OS and FTP server are you using?

    3)
    Can you use regular FTP instead?

    4)
    Use relative urls over absolute. They where the first supported.

    5)
    Can you enable TRACE logging on org.apache.camel.component.file and
    setup a simple example so it makes it easier to follow and understand
    what its supposed to do, and what happens in reality so we can see and
    understand the difference.




    On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
    wrote:
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Claus Ibsen at Oct 2, 2010 at 2:51 pm

    On Sat, Oct 2, 2010 at 9:14 AM, Claus Ibsen wrote:
    Hi Loririn

    I think you/we have to step back. We have other users in the past
    running with SFTP with no problems at all. Also fetching files
    recursively.
    So I wonder if we should revert back and keep the old logic!
    Btw this didn't come out right as I wanted to say. We should find a
    solution that works for everybody.
    file based data exchange is actually harder than it looks. There are
    so many OS and subtle quirks that can cause hickups.
    And apparently also SFTP servers which deny you to download files
    using paths in the file name :)


    1)
    Is this the first time you use camel-ftp?

    2)
    What OS and FTP server are you using?

    3)
    Can you use regular FTP instead?

    4)
    Use relative urls over absolute. They where the first supported.

    5)
    Can you enable TRACE logging on org.apache.camel.component.file and
    setup a simple example so it makes it easier to follow and understand
    what its supposed to do, and what happens in reality so we can see and
    understand the difference.




    On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
    wrote:
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Claus Ibsen at Oct 2, 2010 at 3:47 pm
    Hi

    I will try to reproduce the issue with the help of Apache SSHD as
    simulating the SFTP server. It's not like a real server but it may
    help and in the future catch regression bugs.

    I got the first simple test up and running by uploading a file.

    On Sat, Oct 2, 2010 at 4:50 PM, Claus Ibsen wrote:
    On Sat, Oct 2, 2010 at 9:14 AM, Claus Ibsen wrote:
    Hi Loririn

    I think you/we have to step back. We have other users in the past
    running with SFTP with no problems at all. Also fetching files
    recursively.
    So I wonder if we should revert back and keep the old logic!
    Btw this didn't come out right as I wanted to say. We should find a
    solution that works for everybody.
    file based data exchange is actually harder than it looks. There are
    so many OS and subtle quirks that can cause hickups.
    And apparently also SFTP servers which deny you to download files
    using paths in the file name :)


    1)
    Is this the first time you use camel-ftp?

    2)
    What OS and FTP server are you using?

    3)
    Can you use regular FTP instead?

    4)
    Use relative urls over absolute. They where the first supported.

    5)
    Can you enable TRACE logging on org.apache.camel.component.file and
    setup a simple example so it makes it easier to follow and understand
    what its supposed to do, and what happens in reality so we can see and
    understand the difference.




    On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
    wrote:
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Lorrin Nelson at Oct 4, 2010 at 10:59 pm
    Hi Claus, thanks for the continued support. With the latest code (revision 1003927) relative URIs are working again for me.

    This the first time I'm using camel-ftp. The server I'm talking to is a Fedora 11 box running OpenSSH_5.2p1. Regular FTP is an unlikely option for us. Relative URLs are an acceptable work-around.

    Below is a trace log of an absolute URL.

    On the first attempt, note that ~/tmp does exist. Therefore when it tries to go to *relative* path tmp (which is the wrong thing to do), it succeeds. Then when it tries to go one level deeper into pitch_activity_logs it fails. The second attempt is really interesting. Note there that the current working directory has remained at /home/tomcat/tmp. So when it fails to go to *relative* path tmp, it fails right away. BTW, note that it says "Retrieving file: tmp/activity_logs/activity_log.1285909200" with no leading slash, unlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".

    Cheers
    -Lorrin


    SftpConsumer 2010-10-04 15:45:07,445 -- INFO -- Connected and logged in to: sftp://tomcat@host:22
    SftpConsumer 2010-10-04 15:45:07,494 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:07,788 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:08,011 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:08,012 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:08,099 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:08,189 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-04 15:45:08,274 -- DEBUG -- Took 0.829 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:08,275 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:08,414 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: activity_logs]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: activity_logs
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:08,426 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    ScheduledPollConsumer 2010-10-04 15:45:21,413 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-04 15:45:21,413 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:21,413 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,496 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:21,585 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:21,675 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:21,907 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:21,907 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,997 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:22,084 -- TRACE -- Changing directory: tomcat
    SftpOperations 2010-10-04 15:45:22,177 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Took 0.848 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:22,262 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,305 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: tmp]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: tmp
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:22,311 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    On Oct 2, 2010, at 12:14 AM, Claus Ibsen wrote:

    Hi Loririn

    I think you/we have to step back. We have other users in the past
    running with SFTP with no problems at all. Also fetching files
    recursively.
    So I wonder if we should revert back and keep the old logic!

    1)
    Is this the first time you use camel-ftp?

    2)
    What OS and FTP server are you using?

    3)
    Can you use regular FTP instead?

    4)
    Use relative urls over absolute. They where the first supported.

    5)
    Can you enable TRACE logging on org.apache.camel.component.file and
    setup a simple example so it makes it easier to follow and understand
    what its supposed to do, and what happens in reality so we can see and
    understand the difference.




    On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
    wrote:
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Claus Ibsen at Oct 5, 2010 at 7:03 am
    Hi

    Great that the relative works. This is also what I expected.

    In terms of absolute then Camel changes the directory to / (since its
    absolute) and the SFTP library returns OK for this.
    But in reality it does not go to the root path, but most likely just
    goes to the home directory of the user.
    And hence why absolute does not work for you.

    I don't see any API or way to tell SFTP library it should change to
    root path and not home folder.

    A solution to be able to poll files from an absolute folder is to
    create a symlink in the user home directory which can be polled by the
    SFTP library.
    nlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".
    The reason why you see this is because Camel logs the path configured
    from the endpoint, when the poller starts. Hence you see the leading
    /.



    On Tue, Oct 5, 2010 at 12:58 AM, Lorrin Nelson
    wrote:
    Hi Claus, thanks for the continued support. With the latest code (revision 1003927) relative URIs are working again for me.

    This the first time I'm using camel-ftp. The server I'm talking to is a Fedora 11 box running OpenSSH_5.2p1. Regular FTP is an unlikely option for us. Relative URLs are an acceptable work-around.

    Below is a trace log of an absolute URL.

    On the first attempt, note that ~/tmp does exist. Therefore when it tries to go to *relative* path tmp (which is the wrong thing to do), it succeeds. Then when it tries to go one level deeper into pitch_activity_logs it fails. The second attempt is really interesting. Note there that the current working directory has remained at /home/tomcat/tmp. So when it fails to go to *relative* path tmp, it fails right away. BTW, note that it says "Retrieving file: tmp/activity_logs/activity_log.1285909200" with no leading slash, unlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".

    Cheers
    -Lorrin


    SftpConsumer 2010-10-04 15:45:07,445 -- INFO -- Connected and logged in to: sftp://tomcat@host:22
    SftpConsumer 2010-10-04 15:45:07,494 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:07,788 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:08,011 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:08,012 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:08,099 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:08,189 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-04 15:45:08,274 -- DEBUG -- Took 0.829 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:08,275 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:08,414 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: activity_logs]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: activity_logs
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:08,426 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    ScheduledPollConsumer 2010-10-04 15:45:21,413 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-04 15:45:21,413 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:21,413 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,496 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:21,585 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:21,675 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:21,907 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:21,907 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,997 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:22,084 -- TRACE -- Changing directory: tomcat
    SftpOperations 2010-10-04 15:45:22,177 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Took 0.848 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:22,262 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,305 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: tmp]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: tmp
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:22,311 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    On Oct 2, 2010, at 12:14 AM, Claus Ibsen wrote:

    Hi Loririn

    I think you/we have to step back. We have other users in the past
    running with SFTP with no problems at all. Also fetching files
    recursively.
    So I wonder if we should revert back and keep the old logic!

    1)
    Is this the first time you use camel-ftp?

    2)
    What OS and FTP server are you using?

    3)
    Can you use regular FTP instead?

    4)
    Use relative urls over absolute. They where the first supported.

    5)
    Can you enable TRACE logging on org.apache.camel.component.file and
    setup a simple example so it makes it easier to follow and understand
    what its supposed to do, and what happens in reality so we can see and
    understand the difference.




    On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
    wrote:
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Lorrin Nelson at Oct 5, 2010 at 5:22 pm
    Hmm. How come when polling the absolute path it says this before listing the files in the directory (which works)?
    SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
    ...but then when it tries to retrieve the file (which fails) it just says:
    SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs

    That looks awfully like Camel isn't even asking the SFTP library to go to / first when it comes time to retrieve the file.

    Cheers!
    -Lorrin

    On Oct 5, 2010, at 12:02 AM, Claus Ibsen wrote:

    Hi

    Great that the relative works. This is also what I expected.

    In terms of absolute then Camel changes the directory to / (since its
    absolute) and the SFTP library returns OK for this.
    But in reality it does not go to the root path, but most likely just
    goes to the home directory of the user.
    And hence why absolute does not work for you.

    I don't see any API or way to tell SFTP library it should change to
    root path and not home folder.

    A solution to be able to poll files from an absolute folder is to
    create a symlink in the user home directory which can be polled by the
    SFTP library.
    nlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".
    The reason why you see this is because Camel logs the path configured
    from the endpoint, when the poller starts. Hence you see the leading
    /.



    On Tue, Oct 5, 2010 at 12:58 AM, Lorrin Nelson
    wrote:
    Hi Claus, thanks for the continued support. With the latest code (revision 1003927) relative URIs are working again for me.

    This the first time I'm using camel-ftp. The server I'm talking to is a Fedora 11 box running OpenSSH_5.2p1. Regular FTP is an unlikely option for us. Relative URLs are an acceptable work-around.

    Below is a trace log of an absolute URL.

    On the first attempt, note that ~/tmp does exist. Therefore when it tries to go to *relative* path tmp (which is the wrong thing to do), it succeeds. Then when it tries to go one level deeper into pitch_activity_logs it fails. The second attempt is really interesting. Note there that the current working directory has remained at /home/tomcat/tmp. So when it fails to go to *relative* path tmp, it fails right away. BTW, note that it says "Retrieving file: tmp/activity_logs/activity_log.1285909200" with no leading slash, unlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".

    Cheers
    -Lorrin


    SftpConsumer 2010-10-04 15:45:07,445 -- INFO -- Connected and logged in to: sftp://tomcat@host:22
    SftpConsumer 2010-10-04 15:45:07,494 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:07,788 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:08,011 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:08,012 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:08,099 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:08,189 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-04 15:45:08,274 -- DEBUG -- Took 0.829 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:08,275 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:08,414 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: activity_logs]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: activity_logs
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:08,426 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    ScheduledPollConsumer 2010-10-04 15:45:21,413 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-04 15:45:21,413 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:21,413 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,496 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:21,585 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:21,675 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:21,907 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:21,907 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,997 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:22,084 -- TRACE -- Changing directory: tomcat
    SftpOperations 2010-10-04 15:45:22,177 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Took 0.848 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:22,262 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,305 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: tmp]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: tmp
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:22,311 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    On Oct 2, 2010, at 12:14 AM, Claus Ibsen wrote:

    Hi Loririn

    I think you/we have to step back. We have other users in the past
    running with SFTP with no problems at all. Also fetching files
    recursively.
    So I wonder if we should revert back and keep the old logic!

    1)
    Is this the first time you use camel-ftp?

    2)
    What OS and FTP server are you using?

    3)
    Can you use regular FTP instead?

    4)
    Use relative urls over absolute. They where the first supported.

    5)
    Can you enable TRACE logging on org.apache.camel.component.file and
    setup a simple example so it makes it easier to follow and understand
    what its supposed to do, and what happens in reality so we can see and
    understand the difference.




    On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
    wrote:
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Lorrin Nelson at Oct 5, 2010 at 6:10 pm
    Hi, I've confirmed this is a Camel bug and not an SFTP library limitation. See below for trace log of a successful retrieval from an absolute path.

    I did this in a way that's not suitable for general use: I just unconditionally hacked a "/" on the front of the path in SftpOperations.retrieveFileToStreamInBody:

    // change directory to path where the file is to be retrieved
    // (must do this as some FTP servers cannot retrieve using absolute path)
    String path = FileUtil.onlyPath(name);
    if (path != null) {
    changeCurrentDirectory("/" + path);
    }

    Cheers
    -Lorrin


    ScheduledPollConsumer 2010-10-05 11:05:48,540 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptOlder&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&move=.done&password=******]
    SftpConsumer 2010-10-05 11:05:48,547 -- TRACE -- doPollDirectory from absolutePath: /tmp/pitch_activity_logs, dirName: null
    SftpOperations 2010-10-05 11:05:48,548 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:48,632 -- TRACE -- Changing directory: tmp
    ScheduledPollConsumer 2010-10-05 11:05:48,681 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-05 11:05:48,682 -- TRACE -- doPollDirectory from absolutePath: /tmp/pitch_activity_logs, dirName: null
    SftpOperations 2010-10-05 11:05:48,682 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:48,713 -- TRACE -- Changing directory: pitch_activity_logs
    SftpOperations 2010-10-05 11:05:48,768 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-05 11:05:48,797 -- TRACE -- Polling directory: /tmp/pitch_activity_logs
    SftpOperations 2010-10-05 11:05:48,856 -- TRACE -- Changing directory: pitch_activity_logs
    SftpConsumer 2010-10-05 11:05:48,947 -- TRACE -- Polling directory: /tmp/pitch_activity_logs
    SftpConsumer 2010-10-05 11:05:49,013 -- TRACE -- Found 4 in directory: /tmp/pitch_activity_logs
    SftpConsumer 2010-10-05 11:05:49,013 -- TRACE -- This consumer is idempotent and the file has been consumed before. Will skip this file: GenericFile[pitch_activity_log.1285909200]
    SftpOperations 2010-10-05 11:05:49,013 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:49,105 -- TRACE -- Changing directory: home
    SftpConsumer 2010-10-05 11:05:49,169 -- TRACE -- Found 4 in directory: /tmp/pitch_activity_logs
    SftpOperations 2010-10-05 11:05:49,170 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:49,196 -- TRACE -- Changing directory: tomcat
    SftpOperations 2010-10-05 11:05:49,258 -- TRACE -- Changing directory: home
    SftpConsumer 2010-10-05 11:05:49,284 -- DEBUG -- Took 0.737 seconds to poll: /tmp/pitch_activity_logs/
    ScheduledPollConsumer 2010-10-05 11:05:49,284 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptOlder&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&move=.done&password=******]
    SftpOperations 2010-10-05 11:05:49,347 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-05 11:05:49,435 -- DEBUG -- Took 0.753 seconds to poll: /tmp/pitch_activity_logs/
    SftpConsumer 2010-10-05 11:05:49,436 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-05 11:05:49,436 -- TRACE -- Processing file: GenericFile[pitch_activity_log.1285909200]
    SftpConsumer 2010-10-05 11:05:49,436 -- TRACE -- Retrieving file: tmp/pitch_activity_logs/pitch_activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-05 11:05:49,436 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:49,526 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-05 11:05:49,608 -- TRACE -- Changing directory: pitch_activity_logs
    SftpOperations 2010-10-05 11:05:49,928 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:50,026 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-05 11:05:50,117 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-05 11:05:50,204 -- TRACE -- Retrieved file: tmp/pitch_activity_logs/pitch_activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-05 11:05:50,204 -- DEBUG -- About to process file: GenericFile[pitch_activity_log.1285909200] using exchange: Exchange[pitch_activity_log.1285909200]





    On Oct 5, 2010, at 10:21 AM, Lorrin Nelson wrote:

    Hmm. How come when polling the absolute path it says this before listing the files in the directory (which works)?
    SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
    ...but then when it tries to retrieve the file (which fails) it just says:
    SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs

    That looks awfully like Camel isn't even asking the SFTP library to go to / first when it comes time to retrieve the file.

    Cheers!
    -Lorrin

    On Oct 5, 2010, at 12:02 AM, Claus Ibsen wrote:

    Hi

    Great that the relative works. This is also what I expected.

    In terms of absolute then Camel changes the directory to / (since its
    absolute) and the SFTP library returns OK for this.
    But in reality it does not go to the root path, but most likely just
    goes to the home directory of the user.
    And hence why absolute does not work for you.

    I don't see any API or way to tell SFTP library it should change to
    root path and not home folder.

    A solution to be able to poll files from an absolute folder is to
    create a symlink in the user home directory which can be polled by the
    SFTP library.
    nlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".
    The reason why you see this is because Camel logs the path configured
    from the endpoint, when the poller starts. Hence you see the leading
    /.



    On Tue, Oct 5, 2010 at 12:58 AM, Lorrin Nelson
    wrote:
    Hi Claus, thanks for the continued support. With the latest code (revision 1003927) relative URIs are working again for me.

    This the first time I'm using camel-ftp. The server I'm talking to is a Fedora 11 box running OpenSSH_5.2p1. Regular FTP is an unlikely option for us. Relative URLs are an acceptable work-around.

    Below is a trace log of an absolute URL.

    On the first attempt, note that ~/tmp does exist. Therefore when it tries to go to *relative* path tmp (which is the wrong thing to do), it succeeds. Then when it tries to go one level deeper into pitch_activity_logs it fails. The second attempt is really interesting. Note there that the current working directory has remained at /home/tomcat/tmp. So when it fails to go to *relative* path tmp, it fails right away. BTW, note that it says "Retrieving file: tmp/activity_logs/activity_log.1285909200" with no leading slash, unlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".

    Cheers
    -Lorrin


    SftpConsumer 2010-10-04 15:45:07,445 -- INFO -- Connected and logged in to: sftp://tomcat@host:22
    SftpConsumer 2010-10-04 15:45:07,494 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:07,788 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:08,011 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:08,012 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:08,099 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:08,189 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-04 15:45:08,274 -- DEBUG -- Took 0.829 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:08,275 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:08,414 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: activity_logs]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: activity_logs
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:08,426 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    ScheduledPollConsumer 2010-10-04 15:45:21,413 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-04 15:45:21,413 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:21,413 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,496 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:21,585 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:21,675 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:21,907 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:21,907 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,997 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:22,084 -- TRACE -- Changing directory: tomcat
    SftpOperations 2010-10-04 15:45:22,177 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Took 0.848 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:22,262 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,305 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: tmp]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: tmp
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:22,311 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    On Oct 2, 2010, at 12:14 AM, Claus Ibsen wrote:

    Hi Loririn

    I think you/we have to step back. We have other users in the past
    running with SFTP with no problems at all. Also fetching files
    recursively.
    So I wonder if we should revert back and keep the old logic!

    1)
    Is this the first time you use camel-ftp?

    2)
    What OS and FTP server are you using?

    3)
    Can you use regular FTP instead?

    4)
    Use relative urls over absolute. They where the first supported.

    5)
    Can you enable TRACE logging on org.apache.camel.component.file and
    setup a simple example so it makes it easier to follow and understand
    what its supposed to do, and what happens in reality so we can see and
    understand the difference.




    On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
    wrote:
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Claus Ibsen at Oct 5, 2010 at 7:10 pm
    Hi Lorin

    Thanks for digging into it. I will take a look at it in the morning.
    Hopefully I have a fix in SVN which you can try.


    On Tue, Oct 5, 2010 at 8:09 PM, Lorrin Nelson
    wrote:
    Hi, I've confirmed this is a Camel bug and not an SFTP library limitation. See below for trace log of a successful retrieval from an absolute path.

    I did this in a way that's not suitable for general use: I just unconditionally hacked a "/" on the front of the path in SftpOperations.retrieveFileToStreamInBody:

    // change directory to path where the file is to be retrieved
    // (must do this as some FTP servers cannot retrieve using absolute path)
    String path = FileUtil.onlyPath(name);
    if (path != null) {
    changeCurrentDirectory("/" + path);
    }

    Cheers
    -Lorrin


    ScheduledPollConsumer 2010-10-05 11:05:48,540 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptOlder&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&move=.done&password=******]
    SftpConsumer 2010-10-05 11:05:48,547 -- TRACE -- doPollDirectory from absolutePath: /tmp/pitch_activity_logs, dirName: null
    SftpOperations 2010-10-05 11:05:48,548 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:48,632 -- TRACE -- Changing directory: tmp
    ScheduledPollConsumer 2010-10-05 11:05:48,681 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-05 11:05:48,682 -- TRACE -- doPollDirectory from absolutePath: /tmp/pitch_activity_logs, dirName: null
    SftpOperations 2010-10-05 11:05:48,682 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:48,713 -- TRACE -- Changing directory: pitch_activity_logs
    SftpOperations 2010-10-05 11:05:48,768 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-05 11:05:48,797 -- TRACE -- Polling directory: /tmp/pitch_activity_logs
    SftpOperations 2010-10-05 11:05:48,856 -- TRACE -- Changing directory: pitch_activity_logs
    SftpConsumer 2010-10-05 11:05:48,947 -- TRACE -- Polling directory: /tmp/pitch_activity_logs
    SftpConsumer 2010-10-05 11:05:49,013 -- TRACE -- Found 4 in directory: /tmp/pitch_activity_logs
    SftpConsumer 2010-10-05 11:05:49,013 -- TRACE -- This consumer is idempotent and the file has been consumed before. Will skip this file: GenericFile[pitch_activity_log.1285909200]
    SftpOperations 2010-10-05 11:05:49,013 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:49,105 -- TRACE -- Changing directory: home
    SftpConsumer 2010-10-05 11:05:49,169 -- TRACE -- Found 4 in directory: /tmp/pitch_activity_logs
    SftpOperations 2010-10-05 11:05:49,170 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:49,196 -- TRACE -- Changing directory: tomcat
    SftpOperations 2010-10-05 11:05:49,258 -- TRACE -- Changing directory: home
    SftpConsumer 2010-10-05 11:05:49,284 -- DEBUG -- Took 0.737 seconds to poll: /tmp/pitch_activity_logs/
    ScheduledPollConsumer 2010-10-05 11:05:49,284 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptOlder&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&move=.done&password=******]
    SftpOperations 2010-10-05 11:05:49,347 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-05 11:05:49,435 -- DEBUG -- Took 0.753 seconds to poll: /tmp/pitch_activity_logs/
    SftpConsumer 2010-10-05 11:05:49,436 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-05 11:05:49,436 -- TRACE -- Processing file: GenericFile[pitch_activity_log.1285909200]
    SftpConsumer 2010-10-05 11:05:49,436 -- TRACE -- Retrieving file: tmp/pitch_activity_logs/pitch_activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-05 11:05:49,436 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:49,526 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-05 11:05:49,608 -- TRACE -- Changing directory: pitch_activity_logs
    SftpOperations 2010-10-05 11:05:49,928 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-05 11:05:50,026 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-05 11:05:50,117 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-05 11:05:50,204 -- TRACE -- Retrieved file: tmp/pitch_activity_logs/pitch_activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-05 11:05:50,204 -- DEBUG -- About to process file: GenericFile[pitch_activity_log.1285909200] using exchange: Exchange[pitch_activity_log.1285909200]





    On Oct 5, 2010, at 10:21 AM, Lorrin Nelson wrote:

    Hmm. How come when polling the absolute path it says this before listing the files in the directory (which works)?
    SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
    ...but then when it tries to retrieve the file (which fails) it just says:
    SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs

    That looks awfully like Camel isn't even asking the SFTP library to go to / first when it comes time to retrieve the file.

    Cheers!
    -Lorrin

    On Oct 5, 2010, at 12:02 AM, Claus Ibsen wrote:

    Hi

    Great that the relative works. This is also what I expected.

    In terms of absolute then Camel changes the directory to / (since its
    absolute) and the SFTP library returns OK for this.
    But in reality it does not go to the root path, but most likely just
    goes to the home directory of the user.
    And hence why absolute does not work for you.

    I don't see any API or way to tell SFTP library it should change to
    root path and not home folder.

    A solution to be able to poll files from an absolute folder is to
    create a symlink in the user home directory which can be polled by the
    SFTP library.
    nlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".
    The reason why you see this is because Camel logs the path configured
    from the endpoint, when the poller starts. Hence you see the leading
    /.



    On Tue, Oct 5, 2010 at 12:58 AM, Lorrin Nelson
    wrote:
    Hi Claus, thanks for the continued support. With the latest code (revision 1003927) relative URIs are working again for me.

    This the first time I'm using camel-ftp. The server I'm talking to is a Fedora 11 box running OpenSSH_5.2p1. Regular FTP is an unlikely option for us. Relative URLs are an acceptable work-around.

    Below is a trace log of an absolute URL.

    On the first attempt, note that ~/tmp does exist. Therefore when it tries to go to *relative* path tmp (which is the wrong thing to do), it succeeds. Then when it tries to go one level deeper into pitch_activity_logs it fails. The second attempt is really interesting. Note there that the current working directory has remained at /home/tomcat/tmp. So when it fails to go to *relative* path tmp, it fails right away. BTW, note that it says "Retrieving file: tmp/activity_logs/activity_log.1285909200" with no leading slash, unlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".

    Cheers
    -Lorrin


    SftpConsumer 2010-10-04 15:45:07,445 -- INFO -- Connected and logged in to: sftp://tomcat@host:22
    SftpConsumer 2010-10-04 15:45:07,494 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:07,788 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:08,011 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:08,012 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:08,099 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:08,189 -- TRACE -- Changing directory: tomcat
    SftpConsumer 2010-10-04 15:45:08,274 -- DEBUG -- Took 0.829 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:08,275 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:08,414 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: activity_logs]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: activity_logs
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:08,426 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    ScheduledPollConsumer 2010-10-04 15:45:21,413 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpConsumer 2010-10-04 15:45:21,413 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
    SftpOperations 2010-10-04 15:45:21,413 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,496 -- TRACE -- Changing directory: tmp
    SftpOperations 2010-10-04 15:45:21,585 -- TRACE -- Changing directory: activity_logs
    SftpConsumer 2010-10-04 15:45:21,675 -- TRACE -- Polling directory: /tmp/activity_logs
    SftpConsumer 2010-10-04 15:45:21,907 -- TRACE -- Found 4 in directory: /tmp/activity_logs
    SftpOperations 2010-10-04 15:45:21,907 -- TRACE -- Changing directory: /
    SftpOperations 2010-10-04 15:45:21,997 -- TRACE -- Changing directory: home
    SftpOperations 2010-10-04 15:45:22,084 -- TRACE -- Changing directory: tomcat
    SftpOperations 2010-10-04 15:45:22,177 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Took 0.848 seconds to poll: /tmp/activity_logs/
    SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Total 1 files to consume
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
    SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
    SftpOperations 2010-10-04 15:45:22,262 -- TRACE -- Changing directory: tmp
    SftpConsumer 2010-10-04 15:45:22,305 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: tmp]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: tmp
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
    ... 16 more
    ScheduledPollConsumer 2010-10-04 15:45:22,311 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



    On Oct 2, 2010, at 12:14 AM, Claus Ibsen wrote:

    Hi Loririn

    I think you/we have to step back. We have other users in the past
    running with SFTP with no problems at all. Also fetching files
    recursively.
    So I wonder if we should revert back and keep the old logic!

    1)
    Is this the first time you use camel-ftp?

    2)
    What OS and FTP server are you using?

    3)
    Can you use regular FTP instead?

    4)
    Use relative urls over absolute. They where the first supported.

    5)
    Can you enable TRACE logging on org.apache.camel.component.file and
    setup a simple example so it makes it easier to follow and understand
    what its supposed to do, and what happens in reality so we can see and
    understand the difference.




    On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
    wrote:
    Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

    So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

    Absolute URI with double leading /:
    sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

    SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
    //splits into chunks successfully, starts looping through chunks
    doChangeDirectory("home")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/home". Doh!


    Relative URI with single leading /:
    sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
    SftpOperations.changeCurrentDirectory("subdir")
    //splits into single chunk, starts looping through chunks
    doChangeDirectory("subdir")
    ChannelSftp.cd("home")
    //path=remoteAbsolutePath(path);
    //yields "/home/user/subdir". yay!

    ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
    changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
    //splits info chunks "", "home", and "tomcat". Doh!
    doChangeDirectory("")
    ChannelSftp.cd("")
    //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


    Stack trace when attempting to supply absolute path:
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more


    Stack trace when relative URL tries to restore current working directory:
    SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 4:
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
    ... 16 more
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at java.lang.String.charAt(String.java:686)
    at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
    ... 17 more

    Cheers
    -Lorrin
    On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote:

    Hi

    You have not used the latest source code. The stacktrace points to
    empty lines etc.
    Make sure you build camel-core and camel-ftp using latest source code.

    Revision: 1002946
    Node Kind: directory
    Schedule: normal
    Last Changed Author: davsclaus
    Last Changed Rev: 1002821
    Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


    On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
    wrote:
    I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

    SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

    SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more


    Cheers!
    -Lorrin
    On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote:

    Hi

    Can you try again with latest source code from trunk. I have changed
    the logic to change dir one folder at a time.


    On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
    wrote:
    Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

    I tried uris
    sftp://test@host//tmp/mult?include=.*\.log
    and
    sftp://test@host///tmp/mult?include=.*\.log

    SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
    at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
    ... 15 more

    On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

    sftp://test@host/tmp/mult/?include=.*\.log

    The problem is that GenericFile.changeFileName calls
    newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
    ...without first checking if endPointPath contains a trailing separator.

    E.g.
    newFileName: "tmp/mult/.done/dummy-2.log"
    endPointPath: "tmp/mult/"

    ObjectHelper.after():
    text: "tmp/mult/.done/dummy-2.log"
    after: "tmp/mult//"

    text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

    GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
    java.lang.NullPointerException
    at java.io.File.<init>(File.java:222)
    at org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169)
    at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
    at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
    at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
    at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
    at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
    at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
    at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
    at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
    at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
    at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
    at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
    at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)



    On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote:

    Hi

    I have committed a fix to the SFTP component. Can you try with latest
    source code from trunk?

    On Tue, Sep 28, 2010 at 7:36 PM, Lorrin wrote:

    I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

    SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
    [org.apache.camel.component.file.GenericFileOperationFailedException -
    Cannot retrieve file: tmp/mult/dummy-5.log]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
    retrieve file: tmp/mult/dummy-5.log
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
    at
    org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
    at
    org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
    at
    org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
    at
    org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
    java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:637)
    Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
    at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
    at
    org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
    ... 14 more

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus
  • Watcher at Sep 28, 2010 at 10:03 pm
    What folder does userid from sftp://userid@dev_server map to on your sftp
    folder?
    and does it have a folder tmp/camel_ftp_test

    ie. userid maps to c:\\myfolder

    Then you would have a folder
    c:\\myfolder\\tmp\camel_ftp_test

    also check that you have the correct permission set up for that user, that
    they have read... list permission.


    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2898420.html
    Sent from the Camel - Users mailing list archive at Nabble.com.
  • Lorrin at Sep 28, 2010 at 10:25 pm
    Hi, thanks for the reply.

    It sounds like you're saying I should use a URI relative to the user's home
    directory and then it will work. That's true and I'm using that as a
    work-around for now.

    But it seems like a bug to me that I am constrained to polling within the
    user's home directory. If I specify an absolute URL, it half-works, listing
    the files but then throwing Exceptions when trying to retrieve them.
    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2902696.html
    Sent from the Camel - Users mailing list archive at Nabble.com.
  • Claus Ibsen at Sep 29, 2010 at 6:38 am

    On Wed, Sep 29, 2010 at 12:25 AM, Lorrin wrote:
    Hi, thanks for the reply.

    It sounds like you're saying I should use a URI relative to the user's home
    directory and then it will work. That's true and I'm using that as a
    work-around for now.

    But it seems like a bug to me that I am constrained to polling within the
    user's home directory. If I specify an absolute URL, it half-works, listing
    the files but then throwing Exceptions when trying to retrieve them.
    Apparently some FTP server is stupid and don't allow you to use a path
    when retrieving a file.

    Hence you must do:
    cd foo
    cd bar
    get thefile.txt
    cd ..
    cd ..

    And not just
    get foo/bar/thefile.txt

    I will create a ticket for improvement in Camel to do the first style
    when retrieving the file.

    --
    View this message in context: http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2902696.html
    Sent from the Camel - Users mailing list archive at Nabble.com.


    --
    Claus Ibsen
    Apache Camel Committer

    Author of Camel in Action: http://www.manning.com/ibsen/
    Open Source Integration: http://fusesource.com
    Blog: http://davsclaus.blogspot.com/
    Twitter: http://twitter.com/davsclaus

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categoriescamel
postedSep 8, '10 at 9:38a
activeOct 5, '10 at 7:10p
posts19
users4
websitecamel.apache.org

People

Translate

site design / logo © 2022 Grokbase