FAQ
I recently had a need to connect to remote databases in AWS from Google App
Engine instances as part of a massive mapreduce data pipelining job. Turns
out that fundamentally this has to be done through the sockets service
https://developers.google.com/appengine/docs/go/sockets/reference . Despite
this theoretically feasibility, it turns out that the current database/sql
package as well as related drivers do not play well with the interface
exposed by the sockets service yet. To solve this, I have put in some
efforts in https://github.com/lib/pq/pull/287 to at least attain a workable
solution for our case. However, I believe it would be even better if the
standard database/sql could naturally support such App-Engine-like use
cases, too.

Technically speaking, the main problem from my point of view is how to make
the database/sql package
call https://developers.google.com/appengine/docs/go/sockets/reference#Conn.SetContext
when it starts to use a net.Conn. Looking at code, it seems that the safest
place to do this is in http://golang.org/src/pkg/database/sql/sql.go#L624 .
However, implementation wise, I am not sure how the database/sql package
should expose this interface; should it expose it as a general callback
that is done after DB.conn(), something similar to the Dial function in
http://golang.org/pkg/net/http/#Transport or as something else at the
driver level?

Without doubt, this same problem is something the native appengine/cloudsql
package needs to solve, too. However, looking at the
code https://code.google.com/p/appengine-go/source/browse/appengine/cloudsql/cloudsql.go#30
, it seems that appengine/cloudsql package is taking advantage of some
internal shortcut that is not documented. I wonder is it possible for
someone to explain the details of this shortcut?

I wonder has anyone done something similar before on App Engine, and if so
is it possible to share your solutions? What do you think about this idea
of extending the standard database/sql to support some sort of low level
Dialer interface?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • David Symonds at Aug 17, 2014 at 4:09 am
    Having the pq package support an injected dialer is the right
    approach. That is trivial to use from App Engine since you can simply
    pass in a closure that'll use appengine/socket with the relevant
    context, and it doesn't require pushing App Engine knowledge into the
    standard library.

    You don't need to call Conn.SetContext unless the connection is going
    to span multiple requests. The context passed to socket.Dial will
    apply. Did you have troubles with that?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Fumin Wang at Aug 17, 2014 at 4:43 am
    Hi David

    Thanks for your suggestions. We are indeed able to make progress with an
    injected dialer in the pq package and utilizing the low level interfaces
    defined in database/sql/driver package such as Queryer, Rows, etc. Since
    our use case are long running requests served from basic_scalling modules
    that typically spend 5~10 minutes grinding a given portion of the entire
    dataset, we had the privilege of simply opening a driver.Conn, use it
    exclusively for the rest of a request's lifetime, and close it when it is
    done, without much need for reusing connections which I believe is one of
    the design goals of the database/sql package. In other words, the new
    exposed Dialer interface of the pq package suited our needs and we didn't
    need to call Conn.SetContext.

    However, I have the feeling that our current solution is sort of a kludge,
    for one we had to restrict ourselves to using a low level driver.Conn
    directly. The reason is because though our requests typically lasts for
    5~10 minutes, they may be issuing more than one query during their
    lifetime. Since there is no guarantee that our requests would get the same
    driver.Conn for every call to DB.Query() which in turns calls DB.conn()
    http://golang.org/src/pkg/database/sql/sql.go#L624 , we are worried that we
    might stepping on undefined behaviour should we not hold on to a reference
    to a particular driver.Conn, and simply relying on sql.DB in giving us one.

    For two, I am also wondering perhaps there are other cases where the
    connection reusing feature provided by the database/sql package is
    valuable, despite the fact that we currently don't need it for running
    mapreduce-like long running requests. I wonder in those cases is it
    possible to come up with a design that fulfills such needs in App Engine,
    but at the same maintain the elegance of the database/sql package?

    On Sun, Aug 17, 2014 at 12:09 PM, David Symonds wrote:

    Having the pq package support an injected dialer is the right
    approach. That is trivial to use from App Engine since you can simply
    pass in a closure that'll use appengine/socket with the relevant
    context, and it doesn't require pushing App Engine knowledge into the
    standard library.

    You don't need to call Conn.SetContext unless the connection is going
    to span multiple requests. The context passed to socket.Dial will
    apply. Did you have troubles with that?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedAug 17, '14 at 3:43a
activeAug 17, '14 at 4:43a
posts3
users2
websitegolang.org

2 users in discussion

Fumin Wang: 2 posts David Symonds: 1 post

People

Translate

site design / logo © 2022 Grokbase