FAQ

[vcap-dev] connect() to unix:/tmp/router.sock failed (11: Resource temporarily unavailable) while connecting to

Zhenkai Jiang
Aug 24, 2012 at 3:02 am
Hi Folks,

I am doing some performance evaluation against Cloud Foundry, it's pretty
easy one, starts several thread of http request and sends several messages
to clustered Cloud Foundry and returns some response, collects response
time.

The thing is I used to encounter the Nginx cpu utilization up to 100%
issue, which was recently resolved by this fix:
https://github.com/cloudfoundry/vcap/commit/83d4fc1ce5f97179b2fe0d77fa15749694e949d6

After I applied the change I got a new problem, it's now the ruby code of
router takes almost 80% of cpu while there are about 300 threads hitting
the CF at the same time. and Nginx simply returns 404 for some of the
requests. (about 30% of requests), I checked the error log of Nginx, it's
full of error message like this:

connect() to unix:/tmp/router.sock failed (11: Resource temporarily
unavailable) while connecting to upstream, client: 192.85.180.154,


Can anyone throw some lights on this? or someone have the same issue there?
thanks.
reply

Search Discussions

4 responses

  • Jesse Zhang at Aug 24, 2012 at 5:20 pm
    Is your deployment on Amazon? How many DEA's do you have and how many
    routers do you have in your deployment?

    Jesse
    On Thu, Aug 23, 2012 at 8:02 PM, Zhenkai Jiang wrote:

    Hi Folks,

    I am doing some performance evaluation against Cloud Foundry, it's pretty
    easy one, starts several thread of http request and sends several messages
    to clustered Cloud Foundry and returns some response, collects response
    time.

    The thing is I used to encounter the Nginx cpu utilization up to 100%
    issue, which was recently resolved by this fix:
    https://github.com/cloudfoundry/vcap/commit/83d4fc1ce5f97179b2fe0d77fa15749694e949d6

    After I applied the change I got a new problem, it's now the ruby code of
    router takes almost 80% of cpu while there are about 300 threads hitting
    the CF at the same time. and Nginx simply returns 404 for some of the
    requests. (about 30% of requests), I checked the error log of Nginx, it's
    full of error message like this:

    connect() to unix:/tmp/router.sock failed (11: Resource temporarily
    unavailable) while connecting to upstream, client: 192.85.180.154,


    Can anyone throw some lights on this? or someone have the same issue
    there? thanks.
  • Yongkun Anfernee Gui at Aug 26, 2012 at 5:40 am
    The EventMachine which ULS is built upon has a hard wired backlog log
    number 100.
    That's the reason why you see lots of EAGAIN from nginx.

    Can you look at this commit and try to keep alive the connection from nginx
    to ULS.
    Basically, you need to upgrade nginx to 1.2 and apply the config change the
    same
    as the commit, and those 404s are expected to disappear.

    https://github.com/cloudfoundry/cf-release/commit/83b6884e753a11bf8e657d7b3a489a39823a5386

    If you have any further questions, don't hesitate to ask.

    Thanks,
    Anfernee

    On Fri, Aug 24, 2012 at 11:02 AM, Zhenkai Jiang wrote:

    Hi Folks,

    I am doing some performance evaluation against Cloud Foundry, it's pretty
    easy one, starts several thread of http request and sends several messages
    to clustered Cloud Foundry and returns some response, collects response
    time.

    The thing is I used to encounter the Nginx cpu utilization up to 100%
    issue, which was recently resolved by this fix:
    https://github.com/cloudfoundry/vcap/commit/83d4fc1ce5f97179b2fe0d77fa15749694e949d6

    After I applied the change I got a new problem, it's now the ruby code of
    router takes almost 80% of cpu while there are about 300 threads hitting
    the CF at the same time. and Nginx simply returns 404 for some of the
    requests. (about 30% of requests), I checked the error log of Nginx, it's
    full of error message like this:

    connect() to unix:/tmp/router.sock failed (11: Resource temporarily
    unavailable) while connecting to upstream, client: 192.85.180.154,


    Can anyone throw some lights on this? or someone have the same issue
    there? thanks.

    --
    Cheers,
    Anfernee
  • Yongkun Anfernee Gui at Aug 26, 2012 at 8:07 am
    Hi Zhenkai,

    One more thing to clarify, the EM might be the red herring here, but
    constantly connecting
    ULS for each requests definitely hurts. Sharing connection whenever
    possible, and choose
    TCP localhost instead of domain socket might be a more reliable choice
    (though slower).

    Anfernee

    On Sun, Aug 26, 2012 at 1:40 PM, Yongkun Anfernee Gui wrote:

    The EventMachine which ULS is built upon has a hard wired backlog log
    number 100.
    That's the reason why you see lots of EAGAIN from nginx.

    Can you look at this commit and try to keep alive the connection from
    nginx to ULS.
    Basically, you need to upgrade nginx to 1.2 and apply the config change
    the same
    as the commit, and those 404s are expected to disappear.


    https://github.com/cloudfoundry/cf-release/commit/83b6884e753a11bf8e657d7b3a489a39823a5386

    If you have any further questions, don't hesitate to ask.

    Thanks,
    Anfernee

    On Fri, Aug 24, 2012 at 11:02 AM, Zhenkai Jiang wrote:

    Hi Folks,

    I am doing some performance evaluation against Cloud Foundry, it's pretty
    easy one, starts several thread of http request and sends several messages
    to clustered Cloud Foundry and returns some response, collects response
    time.

    The thing is I used to encounter the Nginx cpu utilization up to 100%
    issue, which was recently resolved by this fix:
    https://github.com/cloudfoundry/vcap/commit/83d4fc1ce5f97179b2fe0d77fa15749694e949d6

    After I applied the change I got a new problem, it's now the ruby code of
    router takes almost 80% of cpu while there are about 300 threads hitting
    the CF at the same time. and Nginx simply returns 404 for some of the
    requests. (about 30% of requests), I checked the error log of Nginx, it's
    full of error message like this:

    connect() to unix:/tmp/router.sock failed (11: Resource temporarily
    unavailable) while connecting to upstream, client: 192.85.180.154,


    Can anyone throw some lights on this? or someone have the same issue
    there? thanks.

    --
    Cheers,
    Anfernee

    --
    Cheers,
    Anfernee
  • Zhenkai Jiang at Aug 31, 2012 at 7:01 am
    Hi Anfernee,

    Thanks a lot for your reply. My EM is not bugging me any more. I'll
    consider your advice once I fully understand them... :)

    Regards,
    Zhenkai
    On Sunday, August 26, 2012 4:07:16 PM UTC+8, Anfernee Gui wrote:

    Hi Zhenkai,

    One more thing to clarify, the EM might be the red herring here, but
    constantly connecting
    ULS for each requests definitely hurts. Sharing connection whenever
    possible, and choose
    TCP localhost instead of domain socket might be a more reliable choice
    (though slower).

    Anfernee


    On Sun, Aug 26, 2012 at 1:40 PM, Yongkun Anfernee Gui <ag...@rbcon.com<javascript:>
    wrote:
    The EventMachine which ULS is built upon has a hard wired backlog log
    number 100.
    That's the reason why you see lots of EAGAIN from nginx.

    Can you look at this commit and try to keep alive the connection from
    nginx to ULS.
    Basically, you need to upgrade nginx to 1.2 and apply the config change
    the same
    as the commit, and those 404s are expected to disappear.


    https://github.com/cloudfoundry/cf-release/commit/83b6884e753a11bf8e657d7b3a489a39823a5386

    If you have any further questions, don't hesitate to ask.

    Thanks,
    Anfernee


    On Fri, Aug 24, 2012 at 11:02 AM, Zhenkai Jiang <zk.j...@gmail.com<javascript:>
    wrote:
    Hi Folks,

    I am doing some performance evaluation against Cloud Foundry, it's
    pretty easy one, starts several thread of http request and sends several
    messages to clustered Cloud Foundry and returns some response, collects
    response time.

    The thing is I used to encounter the Nginx cpu utilization up to 100%
    issue, which was recently resolved by this fix:
    https://github.com/cloudfoundry/vcap/commit/83d4fc1ce5f97179b2fe0d77fa15749694e949d6

    After I applied the change I got a new problem, it's now the ruby code
    of router takes almost 80% of cpu while there are about 300 threads hitting
    the CF at the same time. and Nginx simply returns 404 for some of the
    requests. (about 30% of requests), I checked the error log of Nginx, it's
    full of error message like this:

    connect() to unix:/tmp/router.sock failed (11: Resource temporarily
    unavailable) while connecting to upstream, client: 192.85.180.154,


    Can anyone throw some lights on this? or someone have the same issue
    there? thanks.

    --
    Cheers,
    Anfernee

    --
    Cheers,
    Anfernee

Related Discussions

Discussion Navigation
viewthread | post