FAQ
I have an SSH server that was set up for a client, and every time we
try to upload large files via SFTP or scp, the transfers speed quickly
slows to zero and gives a - stalled - status message, then
disconnects. Here is an example:

ftp> put iTunesSetup.exe iTunesSetup.exe
Uploading iTunesSetup.exe to /home/scarolan/iTunesSetup.exe
iTunesSetup.exe 0% 704KB 0.0KB/s -
stalled -debug1: channel 0: free: client-session, nchannels 1
Read from remote host ftp.authoria.net: Connection reset by peer
debug1: Transferred: stdin 0, stdout 0, stderr 66 bytes in 203.9 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.3
debug1: Exit status -1
Connection closed

I have no idea why this is happening. Can anyone point me in the
right direction?

Search Discussions

  • 唐建伟 at Dec 22, 2009 at 1:06 am
    I met the same as you, but always due to the bad network connection.

    On 12/21/09, Sean Carolan wrote:
    I have an SSH server that was set up for a client, and every time we
    try to upload large files via SFTP or scp, the transfers speed quickly
    slows to zero and gives a - stalled - status message, then
    disconnects. Here is an example:

    ftp> put iTunesSetup.exe iTunesSetup.exe
    Uploading iTunesSetup.exe to /home/scarolan/iTunesSetup.exe
    iTunesSetup.exe 0% 704KB 0.0KB/s -
    stalled -debug1: channel 0: free: client-session, nchannels 1
    Read from remote host ftp.authoria.net: Connection reset by peer
    debug1: Transferred: stdin 0, stdout 0, stderr 66 bytes in 203.9 seconds
    debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.3
    debug1: Exit status -1
    Connection closed

    I have no idea why this is happening. Can anyone point me in the
    right direction?
    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos

    --
    Tang Jianwei
    System Administrator
  • Sean Carolan at Dec 22, 2009 at 1:12 am

    On Mon, Dec 21, 2009 at 7:06 PM, ??? wrote:

    I met the same as you, but always due to the bad network connection.
    I should probably provide some more information, the server is a VMware
    guest running CentOS 5.3. It's using the vmxnet driver for the eth0
    connection. IPv6 is disabled.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.centos.org/pipermail/centos/attachments/20091221/c28fecb1/attachment.html
  • Les Mikesell at Dec 22, 2009 at 2:23 am

    Sean Carolan wrote:
    On Mon, Dec 21, 2009 at 7:06 PM, ??? <myhnet at gmail.com
    wrote:

    I met the same as you, but always due to the bad network connection.


    I should probably provide some more information, the server is a VMware
    guest running CentOS 5.3. It's using the vmxnet driver for the eth0
    connection. IPv6 is disabled.
    I'm not sure what would cause that, but I'd use rsync over ssh instead of sftp
    anyway - and use the -P option to permit restarting.

    --
    Les Mikesell
    lesmikesell at gmail.com
  • Sean Carolan at Dec 22, 2009 at 9:08 am

    I'm not sure what would cause that, but I'd use rsync over ssh instead of sftp
    anyway - and use the -P option to permit restarting.
    If it were up to me, we'd take that route. The software the client is
    using is WinSCP which does have a restart feature, however it's not
    working for us. I'm wondering if this is somehow caused by the vmware
    network driver?
  • Kai Schaetzl at Dec 22, 2009 at 10:31 am

    Sean Carolan wrote on Tue, 22 Dec 2009 03:08:53 -0600:

    The software the client is
    using is WinSCP which does have a restart feature, however it's not
    working for us.
    Tell him to switch WinSCP to SCP mode.

    Kai

    --
    Kai Sch?tzl, Berlin, Germany
    Get your web at Conactive Internet Services: http://www.conactive.com
  • Sean Carolan at Dec 22, 2009 at 11:12 am

    Tell him to switch WinSCP to SCP mode.

    Kai
    Tried that, it still fails the same way. Here's the short list of
    what I've tried to troubleshoot this:

    Used SCP via the gui and command line
    Used SFTP via the gui and command line
    Ran yum update to bring all packages up to date
    Tried stock CentOS sshd daemon (version 4.3), as well as sshd built
    from source (version 5.3)
    Adjusted MTU settings
    Reinstalled virtual network card
    Updated vmware tools and network card driver
    Tried vmxnet as well as e1000 drivers

    At this point I don't know what else to try. I'm thinking that it's
    either a problem with VMWare, or perhaps our load balancer that is
    routing the packets back and forth. Hopefully one of the vendors will
    be able to help solve the problem. In the meantime we are building
    out a physical server to test whether vmware is the issue or not.

    If anyone else has seen this problem before or has suggestions please
    post them here. Thanks.
  • JS at Dec 22, 2009 at 11:31 am

    -----Original Message-----
    From: centos-bounces at centos.org [mailto:centos-bounces at centos.org] On
    Behalf Of Sean Carolan
    Sent: Tuesday, December 22, 2009 6:13 AM
    To: CentOS mailing list
    Subject: Re: [CentOS] SFTP - stalled - on large files
    Tell him to switch WinSCP to SCP mode.

    Kai
    Tried that, it still fails the same way. Here's the short list of
    what I've tried to troubleshoot this:

    Used SCP via the gui and command line
    Used SFTP via the gui and command line
    Ran yum update to bring all packages up to date
    Tried stock CentOS sshd daemon (version 4.3), as well as sshd built
    from source (version 5.3)
    Adjusted MTU settings
    Reinstalled virtual network card
    Updated vmware tools and network card driver
    Tried vmxnet as well as e1000 drivers

    At this point I don't know what else to try. I'm thinking that it's
    either a problem with VMWare, or perhaps our load balancer that is
    routing the packets back and forth. Hopefully one of the vendors will
    be able to help solve the problem. In the meantime we are building
    out a physical server to test whether vmware is the issue or not.

    If anyone else has seen this problem before or has suggestions please
    post them here. Thanks.
    ---

    Just an idea or thought on it. You never said what the file size was or did
    you? My idea is that is, there not a file size limitation on transfer to
    and from the server? I thought there was? Check you vsftpd.conf out or
    what ever ftp server your running for the size limitation. Maybe some help
    or maybe not?

    John
  • Sean Carolan at Dec 22, 2009 at 1:03 pm

    Just an idea or thought on it. ?You never said what the file size was or did
    you? ?My idea is that is, there not a file size limitation on transfer to
    and from the server? ?I thought there was? ?Check you vsftpd.conf out or
    what ever ftp server your running for the size limitation. ?Maybe some help
    or maybe not?
    The problem is with SFTP, so I'm afraid that vsftpd.conf isn't the culprit here.
  • Kai Schaetzl at Dec 22, 2009 at 12:31 pm

    Sean Carolan wrote on Tue, 22 Dec 2009 05:12:52 -0600:

    Here's the short list of
    what I've tried to troubleshoot this:
    which means it doesn't only fail for your client from outside but also for
    you from within your network?

    Kai

    --
    Kai Sch?tzl, Berlin, Germany
    Get your web at Conactive Internet Services: http://www.conactive.com
  • Mark Roth at Dec 22, 2009 at 12:55 pm
    Sean Carolan wrote:
    <snip>
    At this point I don't know what else to try. I'm thinking that it's
    either a problem with VMWare, or perhaps our load balancer that is
    routing the packets back and forth. Hopefully one of the vendors will
    Load balancer... is that set up to maintain connections, or will it, like IBM's
    WebSeal, go to whichever server is next/least used in the middle of a connection?

    mark
    --
    'A fan must not waste a pint of beer,
    nor through inaction allow beer to go to waste,
    unless of course there is a handy Scientologist to pour it over.'
    - D Langford
  • Sean Carolan at Dec 22, 2009 at 1:06 pm

    Load balancer... is that set up to maintain connections, or will it, like IBM's
    WebSeal, go to whichever server is next/least used in the middle of a connection?
    It's set to use "least connection" but there is only one server behind
    the virtual IP at the moment.

    I'm reasonably sure at this point that the Netscaler is causing the
    problem, because file transfers inside the LAN work fine, and we see
    this same issue on both physical and virtual servers. I just tested
    with a physical box to verify, and the same thing happens, transfer
    speed quickly drops to zero and stalls.

    I've got a ticket open with Citrix to hopefully get to the bottom of
    this. It wouldn't be the first time we've seen the Netscaler muck up
    a TCP connection from a client. The last time I dealt with this it
    was sending unwanted FIN packets to mail servers. Fun stuff.
  • Jay Leafey at Dec 23, 2009 at 12:41 am
    We had a similar problem copying files between servers on two of our
    campuses via SCP. After a while the connection just stalled out and
    hung. The problem turned out to be SCP and SFTP interacting a bug in
    the SACK (Selective Acknowledgment) algorithm used in Linux. We turned
    it off on the two endpoints using the following addition to
    /etc/sysctl.conf:
    # Turn off SACK
    net.ipv4.tcp_sack = 0
    and execute "sysctl -p" to apply it. You can also use "sysctl -w
    net.ipv4.tcp_sack=0" to turn it off temporarily. Our file transfers
    worked just fine after the change.

    I realize there are differences our situation and yours and this might
    not work in your case. Given the length of this thread, though, it
    might be worth a try!
    --
    Jay Leafey - Memphis, TN
    jay.leafey at mindless.com
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: smime.p7s
    Type: application/x-pkcs7-signature
    Size: 3274 bytes
    Desc: S/MIME Cryptographic Signature
    Url : http://lists.centos.org/pipermail/centos/attachments/20091222/75b11810/attachment.bin
  • Sean Carolan at Dec 27, 2009 at 3:32 pm

    # Turn off SACK
    net.ipv4.tcp_sack = 0
    and execute "sysctl -p" to apply it. ?You can also use "sysctl -w
    net.ipv4.tcp_sack=0" to turn it off temporarily. ?Our file transfers worked
    just fine after the change.

    I realize there are differences our situation and yours and this might not
    work in your case. ?Given the length of this thread, though, it might be
    worth a try!
    It appears that the Netscaler load balancer was the problem. We
    turned off TCP buffering (TCPB) on the netscaler for this particular
    virtual server, and I was immediately able to transfer a 95MB file
    with no issues. Citrix has acknowledged that there may be some issues
    with the tcp stack on this device, which they think have been resolved
    in more recent versions of the Netscaler OS.

    Hopefully if anyone else experiences this issue, they'll be able to
    Google it via the CentOS list archives.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcentos @
categoriescentos
postedDec 22, '09 at 12:24a
activeDec 27, '09 at 3:32p
posts14
users7
websitecentos.org
irc#centos

People

Translate

site design / logo © 2023 Grokbase