[Camel-users] Re: How to change directory while using sftp component

I just bumped into the same problem. I'm running 2.5-SNAPSHOT.

SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot retrieve file: tmp/mult/dummy-5.log]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
retrieve file: tmp/mult/dummy-5.log
at
org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403)
at
org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901)
at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870)
at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849)
at
org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400)
... 14 more

Hi

I have committed a fix to the SFTP component. Can you try with latest
source code from trunk?

Thanks for the quick response. I'm seeing a different failure now ("Cannot change current directory to: tmp/mult" rather than "Cannot retrieve file: tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no leading /?

I tried uris
sftp://test@host//tmp/mult?include=.*\.log
and
sftp://test@host///tmp/mult?include=.*\.log

SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
... 15 more

On a tangentially related note, there's some file name processing during the move step that throws a NullPointerException when the URI contains a trailing slash:

sftp://test@host/tmp/mult/?include=.*\.log

The problem is that GenericFile.changeFileName calls
newFileName = ObjectHelper.after(newFileName, endpointPath + getFileSeparator());
...without first checking if endPointPath contains a trailing separator.

E.g.
newFileName: "tmp/mult/.done/dummy-2.log"
endPointPath: "tmp/mult/"

ObjectHelper.after():
text: "tmp/mult/.done/dummy-2.log"
after: "tmp/mult//"

text doesn't contain after (because of double //), returns null, changeFileName() doesn't expect null, throws NPE.

GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: [java.lang.NullPointerException - null]
java.lang.NullPointerException
at java.io.File.(GenericFile.java:169)
at org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41)
at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82)
at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121)
at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55)
at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173)
at org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)
at org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)
at org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:130)
at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)

Hi

Can you try again with latest source code from trunk. I have changed
the logic to change dir one folder at a time.


On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson
wrote:

I'm pretty sure I was already at revision 1002541 before. Now I definitely am. Same Exception. I think the problem is the lack of leading /.

SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls changeCurrentDirectory(path), which in turn calls channel.cd(path), which (now we're in com.jcraft.jsch.ChannelSftp) calls path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", which does not exist.

SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change current directory to: tmp/mult]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change current directory to: tmp/mult
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350)
... 15 more


Cheers!
-Lorrin

Hi

You have not used the latest source code. The stacktrace points to
empty lines etc.
Make sure you build camel-core and camel-ftp using latest source code.

Revision: 1002946
Node Kind: directory
Schedule: normal
Last Changed Author: davsclaus
Last Changed Rev: 1002821
Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010)


On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson
wrote:

Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.

So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below.

Absolute URI with double leading /:
sftp://user@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******

SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path!
//splits into chunks successfully, starts looping through chunks
doChangeDirectory("home")
ChannelSftp.cd("home")
//path=remoteAbsolutePath(path);
//yields "/home/user/home". Doh!


Relative URI with single leading /:
sftp://user@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=******
SftpOperations.changeCurrentDirectory("subdir")
//splits into single chunk, starts looping through chunks
doChangeDirectory("subdir")
ChannelSftp.cd("home")
//path=remoteAbsolutePath(path);
//yields "/home/user/subdir". yay!

..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code"
changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before
//splits info chunks "", "home", and "tomcat". Doh!
doChangeDirectory("")
ChannelSftp.cd("")
//path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string.


Stack trace when attempting to supply absolute path:
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
... 16 more


Stack trace when relative URL tries to restore current working directory:
SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to:
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372)
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: 4:
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285)
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370)
... 16 more
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
at java.lang.String.charAt(String.java:686)
at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359)
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264)
... 17 more

Cheers
-Lorrin

Hi Loririn

I think you/we have to step back. We have other users in the past
running with SFTP with no problems at all. Also fetching files
recursively.
So I wonder if we should revert back and keep the old logic!

1)
Is this the first time you use camel-ftp?

2)
What OS and FTP server are you using?

3)
Can you use regular FTP instead?

4)
Use relative urls over absolute. They where the first supported.

5)
Can you enable TRACE logging on org.apache.camel.component.file and
setup a simple example so it makes it easier to follow and understand
what its supposed to do, and what happens in reality so we can see and
understand the difference.




On Fri, Oct 1, 2010 at 10:45 PM, Lorrin Nelson
wrote:

Btw this didn't come out right as I wanted to say. We should find a
solution that works for everybody.
file based data exchange is actually harder than it looks. There are
so many OS and subtle quirks that can cause hickups.
And apparently also SFTP servers which deny you to download files
using paths in the file name :)

Hi

I will try to reproduce the issue with the help of Apache SSHD as
simulating the SFTP server. It's not like a real server but it may
help and in the future catch regression bugs.

I got the first simple test up and running by uploading a file.

Hi Claus, thanks for the continued support. With the latest code (revision 1003927) relative URIs are working again for me.

This the first time I'm using camel-ftp. The server I'm talking to is a Fedora 11 box running OpenSSH_5.2p1. Regular FTP is an unlikely option for us. Relative URLs are an acceptable work-around.

Below is a trace log of an absolute URL.

On the first attempt, note that ~/tmp does exist. Therefore when it tries to go to *relative* path tmp (which is the wrong thing to do), it succeeds. Then when it tries to go one level deeper into pitch_activity_logs it fails. The second attempt is really interesting. Note there that the current working directory has remained at /home/tomcat/tmp. So when it fails to go to *relative* path tmp, it fails right away. BTW, note that it says "Retrieving file: tmp/activity_logs/activity_log.1285909200" with no leading slash, unlike the polling stage where it says "doPollDirectory from absolutePath: /tmp/activity_logs".

Cheers
-Lorrin


SftpConsumer 2010-10-04 15:45:07,445 -- INFO -- Connected and logged in to: sftp://tomcat@host:22
SftpConsumer 2010-10-04 15:45:07,494 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
SftpOperations 2010-10-04 15:45:07,494 -- TRACE -- Changing directory: /
SftpOperations 2010-10-04 15:45:07,608 -- TRACE -- Changing directory: tmp
SftpOperations 2010-10-04 15:45:07,697 -- TRACE -- Changing directory: activity_logs
SftpConsumer 2010-10-04 15:45:07,788 -- TRACE -- Polling directory: /tmp/activity_logs
SftpConsumer 2010-10-04 15:45:08,011 -- TRACE -- Found 4 in directory: /tmp/activity_logs
SftpOperations 2010-10-04 15:45:08,012 -- TRACE -- Changing directory: /
SftpOperations 2010-10-04 15:45:08,099 -- TRACE -- Changing directory: home
SftpOperations 2010-10-04 15:45:08,189 -- TRACE -- Changing directory: tomcat
SftpConsumer 2010-10-04 15:45:08,274 -- DEBUG -- Took 0.829 seconds to poll: /tmp/activity_logs/
SftpConsumer 2010-10-04 15:45:08,275 -- DEBUG -- Total 1 files to consume
SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
SftpConsumer 2010-10-04 15:45:08,276 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
SftpOperations 2010-10-04 15:45:08,277 -- TRACE -- Changing directory: tmp
SftpOperations 2010-10-04 15:45:08,367 -- TRACE -- Changing directory: activity_logs
SftpConsumer 2010-10-04 15:45:08,414 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: activity_logs]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: activity_logs
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
... 16 more
ScheduledPollConsumer 2010-10-04 15:45:08,426 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]



ScheduledPollConsumer 2010-10-04 15:45:21,413 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
SftpConsumer 2010-10-04 15:45:21,413 -- TRACE -- doPollDirectory from absolutePath: /tmp/activity_logs, dirName: null
SftpOperations 2010-10-04 15:45:21,413 -- TRACE -- Changing directory: /
SftpOperations 2010-10-04 15:45:21,496 -- TRACE -- Changing directory: tmp
SftpOperations 2010-10-04 15:45:21,585 -- TRACE -- Changing directory: activity_logs
SftpConsumer 2010-10-04 15:45:21,675 -- TRACE -- Polling directory: /tmp/activity_logs
SftpConsumer 2010-10-04 15:45:21,907 -- TRACE -- Found 4 in directory: /tmp/activity_logs
SftpOperations 2010-10-04 15:45:21,907 -- TRACE -- Changing directory: /
SftpOperations 2010-10-04 15:45:21,997 -- TRACE -- Changing directory: home
SftpOperations 2010-10-04 15:45:22,084 -- TRACE -- Changing directory: tomcat
SftpOperations 2010-10-04 15:45:22,177 -- TRACE -- Changing directory: tmp
SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Took 0.848 seconds to poll: /tmp/activity_logs/
SftpConsumer 2010-10-04 15:45:22,261 -- DEBUG -- Total 1 files to consume
SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Processing file: GenericFile[activity_log.1285909200]
SftpConsumer 2010-10-04 15:45:22,261 -- TRACE -- Retrieving file: tmp/activity_logs/activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
SftpOperations 2010-10-04 15:45:22,262 -- TRACE -- Changing directory: tmp
SftpConsumer 2010-10-04 15:45:22,305 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: tmp]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: tmp
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:383)
at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:368)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:451)
at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:430)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822)
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268)
at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:381)
... 16 more
ScheduledPollConsumer 2010-10-04 15:45:22,311 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]

Hi

Great that the relative works. This is also what I expected.

In terms of absolute then Camel changes the directory to / (since its
absolute) and the SFTP library returns OK for this.
But in reality it does not go to the root path, but most likely just
goes to the home directory of the user.
And hence why absolute does not work for you.

I don't see any API or way to tell SFTP library it should change to
root path and not home folder.

A solution to be able to poll files from an absolute folder is to
create a symlink in the user home directory which can be polled by the
SFTP library.
The reason why you see this is because Camel logs the path configured
from the endpoint, when the poller starts. Hence you see the leading
/.



On Tue, Oct 5, 2010 at 12:58 AM, Lorrin Nelson
wrote:

Hmm. How come when polling the absolute path it says this before listing the files in the directory (which works)?
...but then when it tries to retrieve the file (which fails) it just says:

That looks awfully like Camel isn't even asking the SFTP library to go to / first when it comes time to retrieve the file.

Cheers!
-Lorrin

Hi, I've confirmed this is a Camel bug and not an SFTP library limitation. See below for trace log of a successful retrieval from an absolute path.

I did this in a way that's not suitable for general use: I just unconditionally hacked a "/" on the front of the path in SftpOperations.retrieveFileToStreamInBody:

// change directory to path where the file is to be retrieved
// (must do this as some FTP servers cannot retrieve using absolute path)
String path = FileUtil.onlyPath(name);
if (path != null) {
changeCurrentDirectory("/" + path);
}

Cheers
-Lorrin


ScheduledPollConsumer 2010-10-05 11:05:48,540 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptOlder&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&move=.done&password=******]
SftpConsumer 2010-10-05 11:05:48,547 -- TRACE -- doPollDirectory from absolutePath: /tmp/pitch_activity_logs, dirName: null
SftpOperations 2010-10-05 11:05:48,548 -- TRACE -- Changing directory: /
SftpOperations 2010-10-05 11:05:48,632 -- TRACE -- Changing directory: tmp
ScheduledPollConsumer 2010-10-05 11:05:48,681 -- TRACE -- Starting to poll: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
SftpConsumer 2010-10-05 11:05:48,682 -- TRACE -- doPollDirectory from absolutePath: /tmp/pitch_activity_logs, dirName: null
SftpOperations 2010-10-05 11:05:48,682 -- TRACE -- Changing directory: /
SftpOperations 2010-10-05 11:05:48,713 -- TRACE -- Changing directory: pitch_activity_logs
SftpOperations 2010-10-05 11:05:48,768 -- TRACE -- Changing directory: tmp
SftpConsumer 2010-10-05 11:05:48,797 -- TRACE -- Polling directory: /tmp/pitch_activity_logs
SftpOperations 2010-10-05 11:05:48,856 -- TRACE -- Changing directory: pitch_activity_logs
SftpConsumer 2010-10-05 11:05:48,947 -- TRACE -- Polling directory: /tmp/pitch_activity_logs
SftpConsumer 2010-10-05 11:05:49,013 -- TRACE -- Found 4 in directory: /tmp/pitch_activity_logs
SftpConsumer 2010-10-05 11:05:49,013 -- TRACE -- This consumer is idempotent and the file has been consumed before. Will skip this file: GenericFile[pitch_activity_log.1285909200]
SftpOperations 2010-10-05 11:05:49,013 -- TRACE -- Changing directory: /
SftpOperations 2010-10-05 11:05:49,105 -- TRACE -- Changing directory: home
SftpConsumer 2010-10-05 11:05:49,169 -- TRACE -- Found 4 in directory: /tmp/pitch_activity_logs
SftpOperations 2010-10-05 11:05:49,170 -- TRACE -- Changing directory: /
SftpOperations 2010-10-05 11:05:49,196 -- TRACE -- Changing directory: tomcat
SftpOperations 2010-10-05 11:05:49,258 -- TRACE -- Changing directory: home
SftpConsumer 2010-10-05 11:05:49,284 -- DEBUG -- Took 0.737 seconds to poll: /tmp/pitch_activity_logs/
ScheduledPollConsumer 2010-10-05 11:05:49,284 -- TRACE -- Finished polling: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptOlder&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&move=.done&password=******]
SftpOperations 2010-10-05 11:05:49,347 -- TRACE -- Changing directory: tomcat
SftpConsumer 2010-10-05 11:05:49,435 -- DEBUG -- Took 0.753 seconds to poll: /tmp/pitch_activity_logs/
SftpConsumer 2010-10-05 11:05:49,436 -- DEBUG -- Total 1 files to consume
SftpConsumer 2010-10-05 11:05:49,436 -- TRACE -- Processing file: GenericFile[pitch_activity_log.1285909200]
SftpConsumer 2010-10-05 11:05:49,436 -- TRACE -- Retrieving file: tmp/pitch_activity_logs/pitch_activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
SftpOperations 2010-10-05 11:05:49,436 -- TRACE -- Changing directory: /
SftpOperations 2010-10-05 11:05:49,526 -- TRACE -- Changing directory: tmp
SftpOperations 2010-10-05 11:05:49,608 -- TRACE -- Changing directory: pitch_activity_logs
SftpOperations 2010-10-05 11:05:49,928 -- TRACE -- Changing directory: /
SftpOperations 2010-10-05 11:05:50,026 -- TRACE -- Changing directory: home
SftpOperations 2010-10-05 11:05:50,117 -- TRACE -- Changing directory: tomcat
SftpConsumer 2010-10-05 11:05:50,204 -- TRACE -- Retrieved file: tmp/pitch_activity_logs/pitch_activity_log.1285909200 from: Endpoint[sftp://tomcat@host//tmp/pitch_activity_logs/?delay=15000&idempotent=true&idempotentRepository=%23apacheLogFilenameRepositoryAcceptNewest&include=.*.log.%5B-%2F%3A0-9%5D*&initialDelay=3000&noop=true&password=******]
SftpConsumer 2010-10-05 11:05:50,204 -- DEBUG -- About to process file: GenericFile[pitch_activity_log.1285909200] using exchange: Exchange[pitch_activity_log.1285909200]

Hi Lorin

Thanks for digging into it. I will take a look at it in the morning.
Hopefully I have a fix in SVN which you can try.


On Tue, Oct 5, 2010 at 8:09 PM, Lorrin Nelson
wrote:

What folder does userid from sftp://userid@dev_server map to on your sftp
folder?
and does it have a folder tmp/camel_ftp_test

ie. userid maps to c:\\myfolder

Then you would have a folder
c:\\myfolder\\tmp\camel_ftp_test

also check that you have the correct permission set up for that user, that
they have read... list permission.

Hi, thanks for the reply.

It sounds like you're saying I should use a URI relative to the user's home
directory and then it will work. That's true and I'm using that as a
work-around for now.

But it seems like a bug to me that I am constrained to polling within the
user's home directory. If I specify an absolute URL, it half-works, listing
the files but then throwing Exceptions when trying to retrieve them.

Apparently some FTP server is stupid and don't allow you to use a path
when retrieving a file.

Hence you must do:
cd foo
cd bar
get thefile.txt
cd ..
cd ..

And not just
get foo/bar/thefile.txt

I will create a ticket for improvement in Camel to do the first style
when retrieving the file.