Grokbase Groups R r-help June 2016
FAQ
Does anyone know how to close an FTP session with RCurl?? I am trying to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded.? I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??






I?ve perused the RCurl manual but I don?t see a way to close sessions.? I tried copying the following example from the RCurl manual, but it didn?t solve the problem.? I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.






#EXAMPLE from getCurl(), p. 39


if(url.exists("http://www.omegahat.net/RCurl/index.html")) {


  ??curl = getCurlHandle()


?? getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)


?? #getCurlInfo(curl) # I skipped this step


?? rm(curl) # release the curl! (does this end the session???)


}






Thanks!






John




  [[alternative HTML version deleted]]

Search Discussions

  • Tom Wright at Jun 14, 2016 at 8:00 pm
    No expert here, and this isn't tested. It seems you can set the
    forbid.reuse option which will cause curl to shutdown the connection
    after transfer is complete.


    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
        curl <- getCurlHandle()
        curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
        getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
    }

    On Tue, Jun 14, 2016 at 2:41 PM, J Payne wrote:
    Does anyone know how to close an FTP session with RCurl? I am trying to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded. I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??



    I?ve perused the RCurl manual but I don?t see a way to close sessions. I tried copying the following example from the RCurl manual, but it didn?t solve the problem. I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.



    #EXAMPLE from getCurl(), p. 39

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {

    curl = getCurlHandle()

    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)

    #getCurlInfo(curl) # I skipped this step

    rm(curl) # release the curl! (does this end the session???)

    }



    Thanks!



    John


    [[alternative HTML version deleted]]

    ______________________________________________
    R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
    https://stat.ethz.ch/mailman/listinfo/r-help
    PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
    and provide commented, minimal, self-contained, reproducible code.
  • J Payne at Jun 15, 2016 at 7:34 pm
    Thanks Tom! I tried that and it didn?t work, but perhaps there are other problems. The system administrator wrote ?I think closing sessions immediately upon completion of the transfer,
    or continuing to use the same session for additional requests would both have a similar and positive effect.? However, I?ve now tried your solution to close the session, and also tried using one curl handle throughout, but the throttling persists.


    John


    On 6/14/16, 1:00 PM, "Tom Wright" wrote:

    No expert here, and this isn't tested. It seems you can set the
    forbid.reuse option which will cause curl to shutdown the connection
    after transfer is complete.

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
    curl <- getCurlHandle()
    curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
    }
    On Tue, Jun 14, 2016 at 2:41 PM, J Payne wrote:
    Does anyone know how to close an FTP session with RCurl? I am trying to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded. I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??



    I?ve perused the RCurl manual but I don?t see a way to close sessions. I tried copying the following example from the RCurl manual, but it didn?t solve the problem. I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.



    #EXAMPLE from getCurl(), p. 39

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {

    curl = getCurlHandle()

    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)

    #getCurlInfo(curl) # I skipped this step

    rm(curl) # release the curl! (does this end the session???)

    }



    Thanks!



    John


    [[alternative HTML version deleted]]

    ______________________________________________
    R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
    https://stat.ethz.ch/mailman/listinfo/r-help
    PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
    and provide commented, minimal, self-contained, reproducible code.
  • William Dunlap at Jun 15, 2016 at 7:57 pm
    rm(curl) # release the curl! (does this end the session???)

    Try adding a call to gc() immediately after this removal. That will force
    an
    immediate run of any finalizer associated with the object just removed.
    With the call to gc(), the garbage collector will be called some time in the
    future and the finalizers will be run then.


    (I don't know if curl has a finalizer that closes the session.)




    Bill Dunlap
    TIBCO Software
    wdunlap tibco.com


    On Wed, Jun 15, 2016 at 12:34 PM, J Payne wrote:

    Thanks Tom! I tried that and it didn?t work, but perhaps there are other
    problems. The system administrator wrote ?I think closing sessions
    immediately upon completion of the transfer,
    or continuing to use the same session for additional requests would both
    have a similar and positive effect.? However, I?ve now tried your solution
    to close the session, and also tried using one curl handle throughout, but
    the throttling persists.

    John
    On 6/14/16, 1:00 PM, "Tom Wright" wrote:

    No expert here, and this isn't tested. It seems you can set the
    forbid.reuse option which will cause curl to shutdown the connection
    after transfer is complete.

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
    curl <- getCurlHandle()
    curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
    }
    On Tue, Jun 14, 2016 at 2:41 PM, J Payne wrote:
    Does anyone know how to close an FTP session with RCurl? I am trying
    to automate the process of downloading snow data from a government website,
    and their server is throttling the connection after a few files are
    downloaded. I contacted their system administrator, who wrote: ?My
    suspicion at this point is that the getURL commands are opened and perform
    the function asked, then linger in wait for 15 minutes until or ftp server
    closes the idle sessions. Is there a way to tell R to close the sessions??


    I?ve perused the RCurl manual but I don?t see a way to close sessions.
    I tried copying the following example from the RCurl manual, but it didn?t
    solve the problem. I?m a novice at this and I don?t understand the
    relationship between handles and sessions, so I am probably missing
    something.


    #EXAMPLE from getCurl(), p. 39

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {

    curl = getCurlHandle()

    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)

    #getCurlInfo(curl) # I skipped this step

    rm(curl) # release the curl! (does this end the session???)

    }



    Thanks!



    John


    [[alternative HTML version deleted]]

    ______________________________________________
    R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
    https://stat.ethz.ch/mailman/listinfo/r-help
    PLEASE do read the posting guide
    http://www.R-project.org/posting-guide.html
    and provide commented, minimal, self-contained, reproducible code.
    ______________________________________________
    R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
    https://stat.ethz.ch/mailman/listinfo/r-help
    PLEASE do read the posting guide
    http://www.R-project.org/posting-guide.html
    and provide commented, minimal, self-contained, reproducible code.

      [[alternative HTML version deleted]]
  • J Payne at Jun 15, 2016 at 8:52 pm
    Fantastic!? That did the trick.? I still suspect that there may be a command within RCurl that accomplishes the same thing, but in any case I?m very grateful to have a solution that works.?






    Best,






    John






    From: William Dunlap <wdunlap@tibco.com>
    Date: Wednesday, June 15, 2016 at 12:57 PM
    To: J Payne <jcpayne@uw.edu>
    Cc: Tom Wright <tom@maladmin.com>, R help list <r-help@r-project.org>
    Subject: Re: [R] Closing FTP sessions with RCurl





    rm(curl) # release the curl! (does this end the session???)





    Try adding a call to gc() immediately after this removal. That will force an


    immediate run of any finalizer associated with the object just removed.


    With the call to gc(), the garbage collector will be called some time in the


    future and the finalizers will be run then.






    (I don't know if curl has a finalizer that closes the session.)








    Bill Dunlap
    TIBCO Software
    wdunlap tibco.com






    On Wed, Jun 15, 2016 at 12:34 PM, J Payne wrote:


    Thanks Tom! I tried that and it didn?t work, but perhaps there are other problems. The system administrator wrote ?I think closing sessions immediately upon completion of the transfer,
    or continuing to use the same session for additional requests would both have a similar and positive effect.? However, I?ve now tried your solution to close the session, and also tried using one curl handle throughout, but the throttling persists.


    John


    On 6/14/16, 1:00 PM, "Tom Wright" wrote:

    No expert here, and this isn't tested. It seems you can set the
    forbid.reuse option which will cause curl to shutdown the connection
    after transfer is complete.

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
    curl <- getCurlHandle()
    curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
    }
    On Tue, Jun 14, 2016 at 2:41 PM, J Payne wrote:
    Does anyone know how to close an FTP session with RCurl? I am trying to automate the process of downloading snow data from a government website, and their server is throttling the connection after a few files are downloaded. I contacted their system administrator, who wrote: ?My suspicion at this point is that the getURL commands are opened and perform the function asked, then linger in wait for 15 minutes until or ftp server closes the idle sessions. Is there a way to tell R to close the sessions??



    I?ve perused the RCurl manual but I don?t see a way to close sessions. I tried copying the following example from the RCurl manual, but it didn?t solve the problem. I?m a novice at this and I don?t understand the relationship between handles and sessions, so I am probably missing something.



    #EXAMPLE from getCurl(), p. 39

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {

    curl = getCurlHandle()

    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)

    #getCurlInfo(curl) # I skipped this step

    rm(curl) # release the curl! (does this end the session???)

    }



    Thanks!



    John


    [[alternative HTML version deleted]]

    ______________________________________________
    R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
    https://stat.ethz.ch/mailman/listinfo/r-help
    PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
    and provide commented, minimal, self-contained, reproducible code.

    ______________________________________________
    R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
    https://stat.ethz.ch/mailman/listinfo/r-help
    PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
    and provide commented, minimal, self-contained, reproducible code.








      [[alternative HTML version deleted]]
  • William Dunlap at Jun 15, 2016 at 8:58 pm

    With the call to gc(), the garbage collector will be called some time in the
    future and the finalizers will be run then.

    Typo: that initial 'With' should be 'Without'.




    Bill Dunlap
    TIBCO Software
    wdunlap tibco.com


    On Wed, Jun 15, 2016 at 12:57 PM, William Dunlap wrote:

    rm(curl) # release the curl! (does this end the session???)
    Try adding a call to gc() immediately after this removal. That will force
    an
    immediate run of any finalizer associated with the object just removed.
    With the call to gc(), the garbage collector will be called some time in
    the
    future and the finalizers will be run then.

    (I don't know if curl has a finalizer that closes the session.)


    Bill Dunlap
    TIBCO Software
    wdunlap tibco.com
    On Wed, Jun 15, 2016 at 12:34 PM, J Payne wrote:

    Thanks Tom! I tried that and it didn?t work, but perhaps there are other
    problems. The system administrator wrote ?I think closing sessions
    immediately upon completion of the transfer,
    or continuing to use the same session for additional requests would both
    have a similar and positive effect.? However, I?ve now tried your solution
    to close the session, and also tried using one curl handle throughout, but
    the throttling persists.

    John
    On 6/14/16, 1:00 PM, "Tom Wright" wrote:

    No expert here, and this isn't tested. It seems you can set the
    forbid.reuse option which will cause curl to shutdown the connection
    after transfer is complete.

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {
    curl <- getCurlHandle()
    curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)
    }
    On Tue, Jun 14, 2016 at 2:41 PM, J Payne wrote:
    Does anyone know how to close an FTP session with RCurl? I am trying
    to automate the process of downloading snow data from a government website,
    and their server is throttling the connection after a few files are
    downloaded. I contacted their system administrator, who wrote: ?My
    suspicion at this point is that the getURL commands are opened and perform
    the function asked, then linger in wait for 15 minutes until or ftp server
    closes the idle sessions. Is there a way to tell R to close the sessions??


    I?ve perused the RCurl manual but I don?t see a way to close
    sessions. I tried copying the following example from the RCurl manual, but
    it didn?t solve the problem. I?m a novice at this and I don?t understand
    the relationship between handles and sessions, so I am probably missing
    something.


    #EXAMPLE from getCurl(), p. 39

    if(url.exists("http://www.omegahat.net/RCurl/index.html")) {

    curl = getCurlHandle()

    getURL("http://www.omegahat.net/RCurl/index.html", curl = curl)

    #getCurlInfo(curl) # I skipped this step

    rm(curl) # release the curl! (does this end the session???)

    }



    Thanks!



    John


    [[alternative HTML version deleted]]

    ______________________________________________
    R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
    https://stat.ethz.ch/mailman/listinfo/r-help
    PLEASE do read the posting guide
    http://www.R-project.org/posting-guide.html
    and provide commented, minimal, self-contained, reproducible code.
    ______________________________________________
    R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
    https://stat.ethz.ch/mailman/listinfo/r-help
    PLEASE do read the posting guide
    http://www.R-project.org/posting-guide.html
    and provide commented, minimal, self-contained, reproducible code.

      [[alternative HTML version deleted]]

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupr-help @
categoriesr
postedJun 14, '16 at 6:41p
activeJun 15, '16 at 8:58p
posts6
users3
websiter-project.org
irc#r

People

Translate

site design / logo © 2017 Grokbase