FAQ
While load testing, I found that my go web+database app was able to handle
hundreds of concurrent connections [1]. However, after a certain point, I
would saturate my database (postgres) connection limit, and my users would
start to get errors instead of successful responses [2]. The errors were
due to go trying to make connections after the database's connection limit
had been surpassed.

database/sql manages a connection pool, and it allows one to set
Max*Idle*Conns. However, it does not allow one to set an overall MaxConns.
Currently, there seem to be 3 options for the user: (a) don't manage
connections; when your app goes over the connection limit of the database
itself, your users will get errors; or (b) in your own app, keep an
estimate of the number of in-use connections made by database/sql, and do
not make queries until you believe a connection to be available; or (c)
fork database/sql or the driver for your specific database.

Given that database/sql is managing the connection pool, it seems
reasonable for database/sql to offer the option of limiting MaxConns.

In [3], bradfitz points out that, e.g., net/http doesn't prevent you from
opening too many outbound connections. This is true. I think maintaining
the option of having no limit (by default) makes good sense. Generally,
however, the file descriptor limit can be set up to ~1 million, whereas
database connection limits will be in the hundreds to low thousands. I
believe and will assert without evidence that database connection limits
are much more common and less easy to solve in software than are file
descriptor limits. I'm not sure that this comparison helps my case, but I
wanted to mention it because it has been brought up before.

This has been discussed in the context of a few other database issues in
[4], but I think the notion of MaxConns merits another discussion in its
own thread.

Thanks,

James

[1] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#errors_rates

[2] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#connections

[3] = https://code.google.com/p/go/issues/detail?id=4805

[4]
= https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/RfTp9XVryEM

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • B Davis at May 18, 2013 at 9:00 am
    (c) fork database/sql
    I was hitting the same problem (PostgreSQL connection limits when slamming
    with loader.io), and this was the approach I tentatively took. I have the
    patches lying around somewhere. They basically just added a `chan
    *driverConn` which was waited on whenever a new connection was wanted, but
    would exceed the limit. Then, `putConn` would return freed connections to
    that channel.

    The massive issue I ran into is that bounding the maximum number of
    connections is going to cause a deadlock in your application code unless
    you *never* have two concurrent un-Close'd sql.Rows. I'm not sure if this
    is a limitation of the postgres driver I was using, but the sql.Rows
    basically held open a cursor which necessitated a dedicated database
    connection -- and holding open one connection while requesting another will
    ultimately deadlock the application under load. e.g., the following code
    causes a deadlock:

         rows1, _ := db.Query(...)
         defer rows1.Close()
         rows2, _ := db.Query(...)
         defer rows2.Close()

    I don't know how to mitigate that.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Tad Glines at May 18, 2013 at 4:04 pm

    On Sat, May 18, 2013 at 2:00 AM, B Davis wrote:

    (c) fork database/sql
    I was hitting the same problem (PostgreSQL connection limits when slamming
    with loader.io), and this was the approach I tentatively took. I have the
    patches lying around somewhere. They basically just added a `chan
    *driverConn` which was waited on whenever a new connection was wanted, but
    would exceed the limit. Then, `putConn` would return freed connections to
    that channel.

    The massive issue I ran into is that bounding the maximum number of
    connections is going to cause a deadlock in your application code unless
    you *never* have two concurrent un-Close'd sql.Rows. I'm not sure if this
    is a limitation of the postgres driver I was using, but the sql.Rows
    basically held open a cursor which necessitated a dedicated database
    connection -- and holding open one connection while requesting another will
    ultimately deadlock the application under load. e.g., the following code
    causes a deadlock:

    rows1, _ := db.Query(...)
    defer rows1.Close()
    rows2, _ := db.Query(...)
    defer rows2.Close()

    I don't know how to mitigate that.

    This CL (https://codereview.appspot.com/8797045/) which I mentioned
    previously, does not have this issue. It allows you to have
    multiple concurrently accessed Rows. And, doing so will not prevent opening
    new connections.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Tad Glines at May 18, 2013 at 3:58 pm
    I created a patch (really a complete refactor) for the database/sql code
    that adds a more complete pool implementation.
    The CL is here: https://codereview.appspot.com/8797045/
    Besides the ability to set the maximum number of connections, it also
    allows you to:
    - Specify when idle connections should be closed,
    - Cause connection validity to be checked before an idle connection is
    pulled out of the pool for use.
    - Set a minimum number of connections to be maintained.
    - Choose to cache prepared statements, all statements, or none.
    - Control the number of concurrent connection open attempts.

    Most of my testing has been with MySQL, but it should work with
    any compliant driver.


    On Fri, May 17, 2013 at 11:02 PM, James Pirruccello wrote:

    While load testing, I found that my go web+database app was able to handle
    hundreds of concurrent connections [1]. However, after a certain point, I
    would saturate my database (postgres) connection limit, and my users would
    start to get errors instead of successful responses [2]. The errors were
    due to go trying to make connections after the database's connection limit
    had been surpassed.

    database/sql manages a connection pool, and it allows one to set
    Max*Idle*Conns. However, it does not allow one to set an overall MaxConns.
    Currently, there seem to be 3 options for the user: (a) don't manage
    connections; when your app goes over the connection limit of the database
    itself, your users will get errors; or (b) in your own app, keep an
    estimate of the number of in-use connections made by database/sql, and do
    not make queries until you believe a connection to be available; or (c)
    fork database/sql or the driver for your specific database.

    Given that database/sql is managing the connection pool, it seems
    reasonable for database/sql to offer the option of limiting MaxConns.

    In [3], bradfitz points out that, e.g., net/http doesn't prevent you from
    opening too many outbound connections. This is true. I think maintaining
    the option of having no limit (by default) makes good sense. Generally,
    however, the file descriptor limit can be set up to ~1 million, whereas
    database connection limits will be in the hundreds to low thousands. I
    believe and will assert without evidence that database connection limits
    are much more common and less easy to solve in software than are file
    descriptor limits. I'm not sure that this comparison helps my case, but I
    wanted to mention it because it has been brought up before.

    This has been discussed in the context of a few other database issues in
    [4], but I think the notion of MaxConns merits another discussion in its
    own thread.

    Thanks,

    James

    [1] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#errors_rates

    [2] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#connections

    [3] = https://code.google.com/p/go/issues/detail?id=4805

    [4] =
    https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/RfTp9XVryEM

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Fightingbamboo at May 25, 2013 at 4:47 pm
    I met the same problem.

    I found when exec the sql like "use databasename;", go will create new
    connections. Finally, cause the error: too many connections.

    don't use sql like "use databasename",the problem will ease.

    在 2013年5月18日星期六UTC+8下午2时02分56秒,James Pirruccello写道:
    While load testing, I found that my go web+database app was able to handle
    hundreds of concurrent connections [1]. However, after a certain point, I
    would saturate my database (postgres) connection limit, and my users would
    start to get errors instead of successful responses [2]. The errors were
    due to go trying to make connections after the database's connection limit
    had been surpassed.

    database/sql manages a connection pool, and it allows one to set
    Max*Idle*Conns. However, it does not allow one to set an overall MaxConns.
    Currently, there seem to be 3 options for the user: (a) don't manage
    connections; when your app goes over the connection limit of the database
    itself, your users will get errors; or (b) in your own app, keep an
    estimate of the number of in-use connections made by database/sql, and do
    not make queries until you believe a connection to be available; or (c)
    fork database/sql or the driver for your specific database.

    Given that database/sql is managing the connection pool, it seems
    reasonable for database/sql to offer the option of limiting MaxConns.

    In [3], bradfitz points out that, e.g., net/http doesn't prevent you from
    opening too many outbound connections. This is true. I think maintaining
    the option of having no limit (by default) makes good sense. Generally,
    however, the file descriptor limit can be set up to ~1 million, whereas
    database connection limits will be in the hundreds to low thousands. I
    believe and will assert without evidence that database connection limits
    are much more common and less easy to solve in software than are file
    descriptor limits. I'm not sure that this comparison helps my case, but I
    wanted to mention it because it has been brought up before.

    This has been discussed in the context of a few other database issues in
    [4], but I think the notion of MaxConns merits another discussion in its
    own thread.

    Thanks,

    James

    [1] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#errors_rates

    [2] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#connections

    [3] = https://code.google.com/p/go/issues/detail?id=4805

    [4] =
    https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/RfTp9XVryEM
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ewan Chou at May 26, 2013 at 6:32 am
    Check out Qbs https://github.com/coocood/qbs , it has the max connection
    option.
    On Saturday, May 18, 2013 2:02:56 PM UTC+8, James Pirruccello wrote:

    While load testing, I found that my go web+database app was able to handle
    hundreds of concurrent connections [1]. However, after a certain point, I
    would saturate my database (postgres) connection limit, and my users would
    start to get errors instead of successful responses [2]. The errors were
    due to go trying to make connections after the database's connection limit
    had been surpassed.

    database/sql manages a connection pool, and it allows one to set
    Max*Idle*Conns. However, it does not allow one to set an overall MaxConns.
    Currently, there seem to be 3 options for the user: (a) don't manage
    connections; when your app goes over the connection limit of the database
    itself, your users will get errors; or (b) in your own app, keep an
    estimate of the number of in-use connections made by database/sql, and do
    not make queries until you believe a connection to be available; or (c)
    fork database/sql or the driver for your specific database.

    Given that database/sql is managing the connection pool, it seems
    reasonable for database/sql to offer the option of limiting MaxConns.

    In [3], bradfitz points out that, e.g., net/http doesn't prevent you from
    opening too many outbound connections. This is true. I think maintaining
    the option of having no limit (by default) makes good sense. Generally,
    however, the file descriptor limit can be set up to ~1 million, whereas
    database connection limits will be in the hundreds to low thousands. I
    believe and will assert without evidence that database connection limits
    are much more common and less easy to solve in software than are file
    descriptor limits. I'm not sure that this comparison helps my case, but I
    wanted to mention it because it has been brought up before.

    This has been discussed in the context of a few other database issues in
    [4], but I think the notion of MaxConns merits another discussion in its
    own thread.

    Thanks,

    James

    [1] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#errors_rates

    [2] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#connections

    [3] = https://code.google.com/p/go/issues/detail?id=4805

    [4] =
    https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/RfTp9XVryEM
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • James Pirruccello at May 30, 2013 at 5:08 pm
    Thanks, Ewan. Indeed I see qbs does support connection limits:
    http://gowalker.org/github.com/coocood/qbs#SetConnectionLimit . Ideally,
    I'd prefer not to get tied down to an ORM. I think connection limits in the
    sql package or in driver-specific packages such as pq would be reasonable.

    In a very recent CL discussion between Tad Glines and bradfitz, it looks
    like allowing a configurable connection limit is something that is being
    considered for database/sql: "I'm totally cool with adding connection limit
    logic to database/sql." https://codereview.appspot.com/8797045/#msg20

    Very promising!

    Best,

    James
    On Sunday, May 26, 2013 2:32:37 AM UTC-4, Ewan Chou wrote:

    Check out Qbs https://github.com/coocood/qbs , it has the max connection
    option.
    On Saturday, May 18, 2013 2:02:56 PM UTC+8, James Pirruccello wrote:

    While load testing, I found that my go web+database app was able to
    handle hundreds of concurrent connections [1]. However, after a certain
    point, I would saturate my database (postgres) connection limit, and my
    users would start to get errors instead of successful responses [2]. The
    errors were due to go trying to make connections after the database's
    connection limit had been surpassed.

    database/sql manages a connection pool, and it allows one to set
    Max*Idle*Conns. However, it does not allow one to set an overall MaxConns.
    Currently, there seem to be 3 options for the user: (a) don't manage
    connections; when your app goes over the connection limit of the database
    itself, your users will get errors; or (b) in your own app, keep an
    estimate of the number of in-use connections made by database/sql, and do
    not make queries until you believe a connection to be available; or (c)
    fork database/sql or the driver for your specific database.

    Given that database/sql is managing the connection pool, it seems
    reasonable for database/sql to offer the option of limiting MaxConns.

    In [3], bradfitz points out that, e.g., net/http doesn't prevent you from
    opening too many outbound connections. This is true. I think maintaining
    the option of having no limit (by default) makes good sense. Generally,
    however, the file descriptor limit can be set up to ~1 million, whereas
    database connection limits will be in the hundreds to low thousands. I
    believe and will assert without evidence that database connection limits
    are much more common and less easy to solve in software than are file
    descriptor limits. I'm not sure that this comparison helps my case, but I
    wanted to mention it because it has been brought up before.

    This has been discussed in the context of a few other database issues in
    [4], but I think the notion of MaxConns merits another discussion in its
    own thread.

    Thanks,

    James

    [1] =
    http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#errors_rates

    [2] = http://loader.io/tests/3cb0f3931e2e92f89b0a67fb498c7c62#connections

    [3] = https://code.google.com/p/go/issues/detail?id=4805

    [4] =
    https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/RfTp9XVryEM
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Tad Glines at May 30, 2013 at 5:15 pm

    On Thu, May 30, 2013 at 10:08 AM, James Pirruccello wrote:

    In a very recent CL discussion between Tad Glines and bradfitz, it looks
    like allowing a configurable connection limit is something that is being
    considered for database/sql: "I'm totally cool with adding connection limit
    logic to database/sql." https://codereview.appspot.com/8797045/#msg20
    The new CL is here: https://codereview.appspot.com/9789044/
    It just needs to be reviewed and committed.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMay 18, '13 at 6:03a
activeMay 30, '13 at 5:15p
posts8
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase