Grokbase Groups Camel users June 2012
FAQ
I am trying to use sftp (camel 2.9.2) and notice that every download is
spread out to one every 10 seconds (files are around 10kb). Based off the
examples in http://camel.apache.org/ftp2.html it looks like there are delay
options (they are not listened under options, but used in example) but when
I use them nothing changes. How can I have the client pull files faster?

Another question, if I am trying to write the ftp files locally, is there
any way to avoid the file download if the local file is around?
I append fileExist=Ignore in the to() but it seems that the files still
get written again.

Here is the route I am using:

public void ftpToLocal() throws Exception {
runRoutes(new RouteBuilder() {
@Override
public void configure() throws Exception {
Registry registry = getContext().getRegistry();
String user = (String) registry.lookup("ftpUser");
String pass = (String) registry.lookup("ftpPass");

// we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
server
// long delay = 60 * 60 * 1000L;
// long delay = TimeUnit.SECONDS.toMillis(1);
long delay = 200;

from("sftp://"+user+"@example.com/path/to/dir?password=" + pass +
"&binary=true&delay="+delay+"&consumer.delay=" + delay)
.to("file://data/ftp?fileExist=Ignore");
}
});
}


Thanks for your time reading this email.

Search Discussions

  • Claus Ibsen at Jun 28, 2012 at 7:48 am
    Hi

    As mentioned on the ftp2 page
    http://camel.apache.org/ftp2

    See the file2 page for more options as the ftp component inherits these options
    http://camel.apache.org/file2.html

    The delay and consumer.delay option is the same. delay is just
    shorthand for consumer.delay.

    And no you cannot avoid the download if you later want to upload it
    someplace else, and the file exists.
    The ftp consumer don't know about this.

    A tricky improvement could be to add a new option to only download the
    file on demand, but that is a bit tricky
    as you would need a live connection and the ftp client to still be
    around and connected.


    What are you trying to do? To sync files between 2 ftp servers?


    On Thu, Jun 28, 2012 at 2:15 AM, David Capwell wrote:
    I am trying to use sftp (camel 2.9.2) and notice that every download is
    spread out to one every 10 seconds (files are around 10kb).  Based off the
    examples in http://camel.apache.org/ftp2.html it looks like there are delay
    options (they are not listened under options, but used in example) but when
    I use them nothing changes.  How can I have the client pull files faster?

    Another question, if I am trying to write the ftp files locally, is there
    any way to avoid the file download if the local file is around?
    I append fileExist=Ignore in the to() but it seems that the files still
    get written again.

    Here is the route I am using:

    public void ftpToLocal() throws Exception {
    runRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    Registry registry = getContext().getRegistry();
    String user = (String) registry.lookup("ftpUser");
    String pass = (String) registry.lookup("ftpPass");

    // we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
    server
    //        long delay = 60 * 60 * 1000L;
    //        long delay = TimeUnit.SECONDS.toMillis(1);
    long delay = 200;

    from("sftp://"+user+"@example.com/path/to/dir?password=" + pass +
    "&binary=true&delay="+delay+"&consumer.delay=" + delay)
    .to("file://data/ftp?fileExist=Ignore");
    }
    });
    }


    Thanks for your time reading this email.


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen
  • David Capwell at Jun 28, 2012 at 1:08 pm
    I'll take a look at file2 to see if that helps.

    My goal is to push the data from FTP to s3. Copying locally so I know what
    has been processed,
    On Jun 28, 2012 12:48 AM, "Claus Ibsen" wrote:

    Hi

    As mentioned on the ftp2 page
    http://camel.apache.org/ftp2

    See the file2 page for more options as the ftp component inherits these
    options
    http://camel.apache.org/file2.html

    The delay and consumer.delay option is the same. delay is just
    shorthand for consumer.delay.

    And no you cannot avoid the download if you later want to upload it
    someplace else, and the file exists.
    The ftp consumer don't know about this.

    A tricky improvement could be to add a new option to only download the
    file on demand, but that is a bit tricky
    as you would need a live connection and the ftp client to still be
    around and connected.


    What are you trying to do? To sync files between 2 ftp servers?


    On Thu, Jun 28, 2012 at 2:15 AM, David Capwell wrote:
    I am trying to use sftp (camel 2.9.2) and notice that every download is
    spread out to one every 10 seconds (files are around 10kb). Based off the
    examples in http://camel.apache.org/ftp2.html it looks like there are delay
    options (they are not listened under options, but used in example) but when
    I use them nothing changes. How can I have the client pull files faster?

    Another question, if I am trying to write the ftp files locally, is there
    any way to avoid the file download if the local file is around?
    I append fileExist=Ignore in the to() but it seems that the files still
    get written again.

    Here is the route I am using:

    public void ftpToLocal() throws Exception {
    runRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    Registry registry = getContext().getRegistry();
    String user = (String) registry.lookup("ftpUser");
    String pass = (String) registry.lookup("ftpPass");

    // we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
    server
    // long delay = 60 * 60 * 1000L;
    // long delay = TimeUnit.SECONDS.toMillis(1);
    long delay = 200;

    from("sftp://"+user+"@example.com/path/to/dir?password=" + pass +
    "&binary=true&delay="+delay+"&consumer.delay=" + delay)
    .to("file://data/ftp?fileExist=Ignore");
    }
    });
    }


    Thanks for your time reading this email.


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen
  • David Capwell at Jun 28, 2012 at 10:33 pm
    Playing with the delay fields defined in file2 seems to improve download
    speeds. Now that this is working better I want to be able to skip any file
    that has already been processed (should persist on restart). It looked to
    me that I should be able to set this up by using the filter option, but I
    am having a hard time figuring out how to use this option without using
    spring.

    here is the code I am using (might not compile cause its copy/paste from
    diff places):

    SimpleRegistry registry = new SimpleRegistry();
    registry.put("fileExistsFilter", new CopyFilter());

    CamelContext context = new DefaultCamelContext(registry);
    context.addRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    from("sftp://example.com/path/to/file?password=" + pass +
    "&binary=true&delay="+delay+ "&initialDelay=" + delay +
    "&filter=#fileExistsFilter")
    .to("file://data/ftp?fileExist=Ignore");

    }
    });
    context.start();

    ...
    public static class CopyFilter implements GenericFileFilter {

    @Override
    public boolean accept(GenericFile genericFile) {
    // check to see if file exists locally
    File localFile = new File("data/ftp", genericFile.getFileName());
    if(localFile.exists()) {
    // check to make sure the sizes match
    if(localFile.length() == genericFile.getFileLength()) {
    // file has already been copied, ignore
    return false;
    }
    }
    return true;
    }
    }


    Thanks for your time reading this email
    On Thu, Jun 28, 2012 at 6:07 AM, David Capwell wrote:

    I'll take a look at file2 to see if that helps.

    My goal is to push the data from FTP to s3. Copying locally so I know
    what has been processed,
    On Jun 28, 2012 12:48 AM, "Claus Ibsen" wrote:

    Hi

    As mentioned on the ftp2 page
    http://camel.apache.org/ftp2

    See the file2 page for more options as the ftp component inherits these
    options
    http://camel.apache.org/file2.html

    The delay and consumer.delay option is the same. delay is just
    shorthand for consumer.delay.

    And no you cannot avoid the download if you later want to upload it
    someplace else, and the file exists.
    The ftp consumer don't know about this.

    A tricky improvement could be to add a new option to only download the
    file on demand, but that is a bit tricky
    as you would need a live connection and the ftp client to still be
    around and connected.


    What are you trying to do? To sync files between 2 ftp servers?



    On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    I am trying to use sftp (camel 2.9.2) and notice that every download is
    spread out to one every 10 seconds (files are around 10kb). Based off the
    examples in http://camel.apache.org/ftp2.html it looks like there are delay
    options (they are not listened under options, but used in example) but when
    I use them nothing changes. How can I have the client pull files faster?
    Another question, if I am trying to write the ftp files locally, is there
    any way to avoid the file download if the local file is around?
    I append fileExist=Ignore in the to() but it seems that the files still
    get written again.

    Here is the route I am using:

    public void ftpToLocal() throws Exception {
    runRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    Registry registry = getContext().getRegistry();
    String user = (String) registry.lookup("ftpUser");
    String pass = (String) registry.lookup("ftpPass");

    // we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
    server
    // long delay = 60 * 60 * 1000L;
    // long delay = TimeUnit.SECONDS.toMillis(1);
    long delay = 200;

    from("sftp://"+user+"@example.com/path/to/dir?password=" + pass +
    "&binary=true&delay="+delay+"&consumer.delay=" + delay)
    .to("file://data/ftp?fileExist=Ignore");
    }
    });
    }


    Thanks for your time reading this email.


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen
  • Claus Ibsen at Jun 29, 2012 at 6:22 am

    On Fri, Jun 29, 2012 at 12:32 AM, David Capwell wrote:
    Playing with the delay fields defined in file2 seems to improve download
    speeds.  Now that this is working better I want to be able to skip any file
    that has already been processed (should persist on restart).  It looked to
    me that I should be able to set this up by using the filter option, but I
    am having a hard time figuring out how to use this option without using
    spring.
    Yeah you can either use the filter or the idempotent consumer. The
    latter is the EIP pattern that fits this problem.
    The file/ftp component has baked in of both. And the latter has
    plugins for various persistent storages.

    The code below seems okay, if you configure the endpoint from the uri,
    then you need to refer to your
    bean that implements the filter using the # notation, as you do.

    Then you need to enlist that bean in the registry. In Spring XML that
    happens automatic when you do a <bean>.
    When you use pure Java, you need to pass in a registry implementation
    when you create the camel context,
    as you do with the code below using the SimpleRegistry.

    Then its just a matter of putting the bean in the simple registry, as
    shown in the code.


    The unit test of the camel-ftp component has plenty of examples as well
    https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/

    here is the code I am using (might not compile cause its copy/paste from
    diff places):

    SimpleRegistry registry = new SimpleRegistry();
    registry.put("fileExistsFilter", new CopyFilter());

    CamelContext context = new DefaultCamelContext(registry);
    context.addRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    from("sftp://example.com/path/to/file?password=" + pass +
    "&binary=true&delay="+delay+ "&initialDelay=" + delay +
    "&filter=#fileExistsFilter")
    .to("file://data/ftp?fileExist=Ignore");

    }
    });
    context.start();

    ...
    public static class CopyFilter implements GenericFileFilter {

    @Override
    public boolean accept(GenericFile genericFile) {
    // check to see if file exists locally
    File localFile = new File("data/ftp", genericFile.getFileName());
    if(localFile.exists()) {
    // check to make sure the sizes match
    if(localFile.length() == genericFile.getFileLength()) {
    // file has already been copied, ignore
    return false;
    }
    }
    return true;
    }
    }


    Thanks for your time reading this email
    On Thu, Jun 28, 2012 at 6:07 AM, David Capwell wrote:

    I'll take a look at file2 to see if that helps.

    My goal is to push the data from FTP to s3.  Copying locally so I know
    what has been processed,
    On Jun 28, 2012 12:48 AM, "Claus Ibsen" wrote:

    Hi

    As mentioned on the ftp2 page
    http://camel.apache.org/ftp2

    See the file2 page for more options as the ftp component inherits these
    options
    http://camel.apache.org/file2.html

    The delay and consumer.delay option is the same. delay is just
    shorthand for consumer.delay.

    And no you cannot avoid the download if you later want to upload it
    someplace else, and the file exists.
    The ftp consumer don't know about this.

    A tricky improvement could be to add a new option to only download the
    file on demand, but that is a bit tricky
    as you would need a live connection and the ftp client to still be
    around and connected.


    What are you trying to do? To sync files between 2 ftp servers?



    On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    I am trying to use sftp (camel 2.9.2) and notice that every download is
    spread out to one every 10 seconds (files are around 10kb).  Based off the
    examples in http://camel.apache.org/ftp2.html it looks like there are delay
    options (they are not listened under options, but used in example) but when
    I use them nothing changes.  How can I have the client pull files faster?
    Another question, if I am trying to write the ftp files locally, is there
    any way to avoid the file download if the local file is around?
    I append fileExist=Ignore in the to() but it seems that the files still
    get written again.

    Here is the route I am using:

    public void ftpToLocal() throws Exception {
    runRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    Registry registry = getContext().getRegistry();
    String user = (String) registry.lookup("ftpUser");
    String pass = (String) registry.lookup("ftpPass");

    // we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
    server
    //        long delay = 60 * 60 * 1000L;
    //        long delay = TimeUnit.SECONDS.toMillis(1);
    long delay = 200;

    from("sftp://"+user+"@example.com/path/to/dir?password=" + pass +
    "&binary=true&delay="+delay+"&consumer.delay=" + delay)
    .to("file://data/ftp?fileExist=Ignore");
    }
    });
    }


    Thanks for your time reading this email.


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen
  • David Capwell at Jun 29, 2012 at 3:45 pm
    Sorry, I seemed to miss the exception that I am getting:

    Caused by: java.lang.IllegalArgumentException: Could not find a suitable
    setter for property: filter as there isn't a setter method with same type:
    java.lang.String nor type conversion possible: No type converter available
    to convert from type: java.lang.String to the required type:
    org.apache.camel.component.file.GenericFileFilter with value
    #fileExistsFilter

    Not sure why but it seems that the filter is not getting the param from the
    registry but trying to pass the value in as a normal string. Based on the
    last comment it sounds like this should be working?
    On Thu, Jun 28, 2012 at 11:21 PM, Claus Ibsen wrote:
    On Fri, Jun 29, 2012 at 12:32 AM, David Capwell wrote:
    Playing with the delay fields defined in file2 seems to improve download
    speeds. Now that this is working better I want to be able to skip any file
    that has already been processed (should persist on restart). It looked to
    me that I should be able to set this up by using the filter option, but I
    am having a hard time figuring out how to use this option without using
    spring.
    Yeah you can either use the filter or the idempotent consumer. The
    latter is the EIP pattern that fits this problem.
    The file/ftp component has baked in of both. And the latter has
    plugins for various persistent storages.

    The code below seems okay, if you configure the endpoint from the uri,
    then you need to refer to your
    bean that implements the filter using the # notation, as you do.

    Then you need to enlist that bean in the registry. In Spring XML that
    happens automatic when you do a <bean>.
    When you use pure Java, you need to pass in a registry implementation
    when you create the camel context,
    as you do with the code below using the SimpleRegistry.

    Then its just a matter of putting the bean in the simple registry, as
    shown in the code.


    The unit test of the camel-ftp component has plenty of examples as well

    https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/

    here is the code I am using (might not compile cause its copy/paste from
    diff places):

    SimpleRegistry registry = new SimpleRegistry();
    registry.put("fileExistsFilter", new CopyFilter());

    CamelContext context = new DefaultCamelContext(registry);
    context.addRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    from("sftp://example.com/path/to/file?password=" + pass +
    "&binary=true&delay="+delay+ "&initialDelay=" + delay +
    "&filter=#fileExistsFilter")
    .to("file://data/ftp?fileExist=Ignore");

    }
    });
    context.start();

    ...
    public static class CopyFilter implements GenericFileFilter {

    @Override
    public boolean accept(GenericFile genericFile) {
    // check to see if file exists locally
    File localFile = new File("data/ftp", genericFile.getFileName());
    if(localFile.exists()) {
    // check to make sure the sizes match
    if(localFile.length() == genericFile.getFileLength()) {
    // file has already been copied, ignore
    return false;
    }
    }
    return true;
    }
    }


    Thanks for your time reading this email
    On Thu, Jun 28, 2012 at 6:07 AM, David Capwell wrote:

    I'll take a look at file2 to see if that helps.

    My goal is to push the data from FTP to s3. Copying locally so I know
    what has been processed,
    On Jun 28, 2012 12:48 AM, "Claus Ibsen" wrote:

    Hi

    As mentioned on the ftp2 page
    http://camel.apache.org/ftp2

    See the file2 page for more options as the ftp component inherits these
    options
    http://camel.apache.org/file2.html

    The delay and consumer.delay option is the same. delay is just
    shorthand for consumer.delay.

    And no you cannot avoid the download if you later want to upload it
    someplace else, and the file exists.
    The ftp consumer don't know about this.

    A tricky improvement could be to add a new option to only download the
    file on demand, but that is a bit tricky
    as you would need a live connection and the ftp client to still be
    around and connected.


    What are you trying to do? To sync files between 2 ftp servers?



    On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    I am trying to use sftp (camel 2.9.2) and notice that every download
    is
    spread out to one every 10 seconds (files are around 10kb). Based
    off
    the
    examples in http://camel.apache.org/ftp2.html it looks like there
    are
    delay
    options (they are not listened under options, but used in example)
    but
    when
    I use them nothing changes. How can I have the client pull files faster?
    Another question, if I am trying to write the ftp files locally, is there
    any way to avoid the file download if the local file is around?
    I append fileExist=Ignore in the to() but it seems that the files
    still
    get written again.

    Here is the route I am using:

    public void ftpToLocal() throws Exception {
    runRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    Registry registry = getContext().getRegistry();
    String user = (String) registry.lookup("ftpUser");
    String pass = (String) registry.lookup("ftpPass");

    // we use a delay of 60 minutes (eg. once pr. hour we poll the FTP
    server
    // long delay = 60 * 60 * 1000L;
    // long delay = TimeUnit.SECONDS.toMillis(1);
    long delay = 200;

    from("sftp://"+user+"@example.com/path/to/dir?password=" +
    pass
    +
    "&binary=true&delay="+delay+"&consumer.delay=" + delay)
    .to("file://data/ftp?fileExist=Ignore");
    }
    });
    }


    Thanks for your time reading this email.


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen
  • David Capwell at Jun 29, 2012 at 3:53 pm
    I have added idempotent=true and this seems to work the way i wont but
    doesn't persist. So to make sure I don't download the same file even after
    reboot the best option is the filter, correct? If so what am I missing
    from getting my code to work?

    thanks for all the help i have been given so far!
    On Fri, Jun 29, 2012 at 8:45 AM, David Capwell wrote:

    Sorry, I seemed to miss the exception that I am getting:

    Caused by: java.lang.IllegalArgumentException: Could not find a suitable
    setter for property: filter as there isn't a setter method with same type:
    java.lang.String nor type conversion possible: No type converter available
    to convert from type: java.lang.String to the required type:
    org.apache.camel.component.file.GenericFileFilter with value
    #fileExistsFilter

    Not sure why but it seems that the filter is not getting the param from
    the registry but trying to pass the value in as a normal string. Based on
    the last comment it sounds like this should be working?

    On Thu, Jun 28, 2012 at 11:21 PM, Claus Ibsen wrote:

    On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    Playing with the delay fields defined in file2 seems to improve download
    speeds. Now that this is working better I want to be able to skip any file
    that has already been processed (should persist on restart). It looked to
    me that I should be able to set this up by using the filter option, but I
    am having a hard time figuring out how to use this option without using
    spring.
    Yeah you can either use the filter or the idempotent consumer. The
    latter is the EIP pattern that fits this problem.
    The file/ftp component has baked in of both. And the latter has
    plugins for various persistent storages.

    The code below seems okay, if you configure the endpoint from the uri,
    then you need to refer to your
    bean that implements the filter using the # notation, as you do.

    Then you need to enlist that bean in the registry. In Spring XML that
    happens automatic when you do a <bean>.
    When you use pure Java, you need to pass in a registry implementation
    when you create the camel context,
    as you do with the code below using the SimpleRegistry.

    Then its just a matter of putting the bean in the simple registry, as
    shown in the code.


    The unit test of the camel-ftp component has plenty of examples as well

    https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/

    here is the code I am using (might not compile cause its copy/paste from
    diff places):

    SimpleRegistry registry = new SimpleRegistry();
    registry.put("fileExistsFilter", new CopyFilter());

    CamelContext context = new DefaultCamelContext(registry);
    context.addRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    from("sftp://example.com/path/to/file?password=" + pass +
    "&binary=true&delay="+delay+ "&initialDelay=" + delay +
    "&filter=#fileExistsFilter")
    .to("file://data/ftp?fileExist=Ignore");

    }
    });
    context.start();

    ...
    public static class CopyFilter implements GenericFileFilter {

    @Override
    public boolean accept(GenericFile genericFile) {
    // check to see if file exists locally
    File localFile = new File("data/ftp", genericFile.getFileName());
    if(localFile.exists()) {
    // check to make sure the sizes match
    if(localFile.length() == genericFile.getFileLength()) {
    // file has already been copied, ignore
    return false;
    }
    }
    return true;
    }
    }


    Thanks for your time reading this email

    On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    I'll take a look at file2 to see if that helps.

    My goal is to push the data from FTP to s3. Copying locally so I know
    what has been processed,
    On Jun 28, 2012 12:48 AM, "Claus Ibsen" wrote:

    Hi

    As mentioned on the ftp2 page
    http://camel.apache.org/ftp2

    See the file2 page for more options as the ftp component inherits
    these
    options
    http://camel.apache.org/file2.html

    The delay and consumer.delay option is the same. delay is just
    shorthand for consumer.delay.

    And no you cannot avoid the download if you later want to upload it
    someplace else, and the file exists.
    The ftp consumer don't know about this.

    A tricky improvement could be to add a new option to only download the
    file on demand, but that is a bit tricky
    as you would need a live connection and the ftp client to still be
    around and connected.


    What are you trying to do? To sync files between 2 ftp servers?



    On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    I am trying to use sftp (camel 2.9.2) and notice that every
    download is
    spread out to one every 10 seconds (files are around 10kb). Based
    off
    the
    examples in http://camel.apache.org/ftp2.html it looks like there
    are
    delay
    options (they are not listened under options, but used in example)
    but
    when
    I use them nothing changes. How can I have the client pull files faster?
    Another question, if I am trying to write the ftp files locally, is there
    any way to avoid the file download if the local file is around?
    I append fileExist=Ignore in the to() but it seems that the files
    still
    get written again.

    Here is the route I am using:

    public void ftpToLocal() throws Exception {
    runRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    Registry registry = getContext().getRegistry();
    String user = (String) registry.lookup("ftpUser");
    String pass = (String) registry.lookup("ftpPass");

    // we use a delay of 60 minutes (eg. once pr. hour we poll
    the
    FTP
    server
    // long delay = 60 * 60 * 1000L;
    // long delay = TimeUnit.SECONDS.toMillis(1);
    long delay = 200;

    from("sftp://"+user+"@example.com/path/to/dir?password=" +
    pass
    +
    "&binary=true&delay="+delay+"&consumer.delay=" + delay)
    .to("file://data/ftp?fileExist=Ignore");
    }
    });
    }


    Thanks for your time reading this email.


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen
  • Claus Ibsen at Jun 30, 2012 at 6:47 am

    On Fri, Jun 29, 2012 at 5:52 PM, David Capwell wrote:
    I have added idempotent=true and this seems to work the way i wont but
    doesn't persist.  So to make sure I don't download the same file even after
    reboot the best option is the filter, correct?  If so what am I missing
    from getting my code to work?
    There are details about persistent idempotent repositories in the Camel docs.
    http://camel.apache.org/file2.html

    And the EIP has some details as well
    http://camel.apache.org/idempotent-consumer.html

    Its just that the file/ftp component has baked in idempotent consumer,
    so you do not
    need to use .idempotentConsumer in the route. This is an optimization to avoid
    consuming the file if really not needed.

    An alternative is as you said a custom filter, where you can return
    true|false, whether
    to consume the file or not.


    thanks for all the help i have been given so far!
    On Fri, Jun 29, 2012 at 8:45 AM, David Capwell wrote:

    Sorry, I seemed to miss the exception that I am getting:

    Caused by: java.lang.IllegalArgumentException: Could not find a suitable
    setter for property: filter as there isn't a setter method with same type:
    java.lang.String nor type conversion possible: No type converter available
    to convert from type: java.lang.String to the required type:
    org.apache.camel.component.file.GenericFileFilter with value
    #fileExistsFilter

    Not sure why but it seems that the filter is not getting the param from
    the registry but trying to pass the value in as a normal string.  Based on
    the last comment it sounds like this should be working?

    On Thu, Jun 28, 2012 at 11:21 PM, Claus Ibsen wrote:

    On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    Playing with the delay fields defined in file2 seems to improve download
    speeds.  Now that this is working better I want to be able to skip any file
    that has already been processed (should persist on restart).  It looked to
    me that I should be able to set this up by using the filter option, but I
    am having a hard time figuring out how to use this option without using
    spring.
    Yeah you can either use the filter or the idempotent consumer. The
    latter is the EIP pattern that fits this problem.
    The file/ftp component has baked in of both. And the latter has
    plugins for various persistent storages.

    The code below seems okay, if you configure the endpoint from the uri,
    then you need to refer to your
    bean that implements the filter using the # notation, as you do.

    Then you need to enlist that bean in the registry. In Spring XML that
    happens automatic when you do a <bean>.
    When you use pure Java, you need to pass in a registry implementation
    when you create the camel context,
    as you do with the code below using the SimpleRegistry.

    Then its just a matter of putting the bean in the simple registry, as
    shown in the code.


    The unit test of the camel-ftp component has plenty of examples as well

    https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/

    here is the code I am using (might not compile cause its copy/paste from
    diff places):

    SimpleRegistry registry = new SimpleRegistry();
    registry.put("fileExistsFilter", new CopyFilter());

    CamelContext context = new DefaultCamelContext(registry);
    context.addRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    from("sftp://example.com/path/to/file?password=" + pass +
    "&binary=true&delay="+delay+ "&initialDelay=" + delay +
    "&filter=#fileExistsFilter")
    .to("file://data/ftp?fileExist=Ignore");

    }
    });
    context.start();

    ...
    public static class CopyFilter implements GenericFileFilter {

    @Override
    public boolean accept(GenericFile genericFile) {
    // check to see if file exists locally
    File localFile = new File("data/ftp", genericFile.getFileName());
    if(localFile.exists()) {
    // check to make sure the sizes match
    if(localFile.length() == genericFile.getFileLength()) {
    // file has already been copied, ignore
    return false;
    }
    }
    return true;
    }
    }


    Thanks for your time reading this email

    On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    I'll take a look at file2 to see if that helps.

    My goal is to push the data from FTP to s3.  Copying locally so I know
    what has been processed,
    On Jun 28, 2012 12:48 AM, "Claus Ibsen" wrote:

    Hi

    As mentioned on the ftp2 page
    http://camel.apache.org/ftp2

    See the file2 page for more options as the ftp component inherits
    these
    options
    http://camel.apache.org/file2.html

    The delay and consumer.delay option is the same. delay is just
    shorthand for consumer.delay.

    And no you cannot avoid the download if you later want to upload it
    someplace else, and the file exists.
    The ftp consumer don't know about this.

    A tricky improvement could be to add a new option to only download the
    file on demand, but that is a bit tricky
    as you would need a live connection and the ftp client to still be
    around and connected.


    What are you trying to do? To sync files between 2 ftp servers?



    On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapwell@gmail.com>
    wrote:
    I am trying to use sftp (camel 2.9.2) and notice that every
    download is
    spread out to one every 10 seconds (files are around 10kb).  Based
    off
    the
    examples in http://camel.apache.org/ftp2.html it looks like there
    are
    delay
    options (they are not listened under options, but used in example)
    but
    when
    I use them nothing changes.  How can I have the client pull files faster?
    Another question, if I am trying to write the ftp files locally, is there
    any way to avoid the file download if the local file is around?
    I append fileExist=Ignore in the to() but it seems that the files
    still
    get written again.

    Here is the route I am using:

    public void ftpToLocal() throws Exception {
    runRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
    Registry registry = getContext().getRegistry();
    String user = (String) registry.lookup("ftpUser");
    String pass = (String) registry.lookup("ftpPass");

    // we use a delay of 60 minutes (eg. once pr. hour we poll
    the
    FTP
    server
    //        long delay = 60 * 60 * 1000L;
    //        long delay = TimeUnit.SECONDS.toMillis(1);
    long delay = 200;

    from("sftp://"+user+"@example.com/path/to/dir?password=" +
    pass
    +
    "&binary=true&delay="+delay+"&consumer.delay=" + delay)
    .to("file://data/ftp?fileExist=Ignore");
    }
    });
    }


    Thanks for your time reading this email.


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.com
    Author of Camel in Action: http://www.manning.com/ibsen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categoriescamel
postedJun 28, '12 at 12:16a
activeJun 30, '12 at 6:47a
posts8
users2
websitecamel.apache.org

2 users in discussion

David Capwell: 5 posts Claus Ibsen: 3 posts

People

Translate

site design / logo © 2022 Grokbase