FAQ
Why would the DataContext be running out of connections in a web app with only a small amount of traffic?

(I thought that a group of connections were stored (base on the modeler specification), and then reused for each transaction.)

Joe

Search Discussions

  • Michael Gentry at Apr 15, 2010 at 12:19 pm
    The connections are stored and reused. What is your min/max setting?
    Do you have any long-running transactions?

    On Wed, Apr 14, 2010 at 11:34 PM, Joe Baldwin wrote:
    Why would the DataContext be running out of connections in a web app with only a small amount of traffic?

    (I thought that a group of connections were stored (base on the modeler specification), and then reused for each transaction.)

    Joe
  • Joe Baldwin at Apr 15, 2010 at 12:57 pm
    Hi Michael,

    min = 1
    max = 10 (we changed this to 20 but then got the same error)

    To my knowledge there are no long running transactions. Most are product list fetches which should take less than a second each. There is some content editing right now, so there are about 50-100 updates per day, (which again should take no more than 1 sec per transaction)

    I have been searching through my code, but can't find anything that seems out of the ordinary.

    Is there an object available to me that will report open connections?

    Thanks,
    Joe

    On Apr 15, 2010, at 8:18 AM, Michael Gentry wrote:

    The connections are stored and reused. What is your min/max setting?
    Do you have any long-running transactions?

    On Wed, Apr 14, 2010 at 11:34 PM, Joe Baldwin wrote:
    Why would the DataContext be running out of connections in a web app with only a small amount of traffic?

    (I thought that a group of connections were stored (base on the modeler specification), and then reused for each transaction.)

    Joe
  • Andrus Adamchik at Apr 16, 2010 at 3:23 am
    Yeah odd, Cayenne checks out a connection from a DataSource only for
    the duration of an operation, and then returns it back. So are you
    using Cayenne-provided DataSource (DriverDataSourceFactory in the
    Modeler), and not JNDI, DBCP or some other container-provided one?

    Can you take a thread dump of your application? ("kill -QUIT <pid>" on
    UNIX) and see if anything is stuck on a DB operation?

    Cheers,
    Andrus
    On Apr 15, 2010, at 8:57 PM, Joe Baldwin wrote:

    Hi Michael,

    min = 1
    max = 10 (we changed this to 20 but then got the same error)

    To my knowledge there are no long running transactions. Most are
    product list fetches which should take less than a second each.
    There is some content editing right now, so there are about 50-100
    updates per day, (which again should take no more than 1 sec per
    transaction)

    I have been searching through my code, but can't find anything that
    seems out of the ordinary.

    Is there an object available to me that will report open connections?

    Thanks,
    Joe

    On Apr 15, 2010, at 8:18 AM, Michael Gentry wrote:

    The connections are stored and reused. What is your min/max setting?
    Do you have any long-running transactions?


    On Wed, Apr 14, 2010 at 11:34 PM, Joe Baldwin <jfbaldwin@earthlink.net
    wrote:
    Why would the DataContext be running out of connections in a web
    app with only a small amount of traffic?

    (I thought that a group of connections were stored (base on the
    modeler specification), and then reused for each transaction.)

    Joe
  • Michael Gentry at Apr 16, 2010 at 12:58 pm
    Hi Joe,

    I've deployed a long-running (up 24x7) internal application before
    that had a max of 2 DB connections using the built-in Cayenne
    connection pooling mechanism and never had any issues with running out
    of connections. The only reason I even used 2 was I had long-running
    (several minutes) queries in a background thread that might block the
    interactive portion and make the user wait during those periods, so I
    basically allocated one to the background thread and one to the user
    threads.

    I don't know of a way to report open connections off the top of my
    head, but I could look into that if needed.

    mrg

    On Thu, Apr 15, 2010 at 8:57 AM, Joe Baldwin wrote:
    Hi Michael,

    min = 1
    max = 10 (we changed this to 20 but then got the same error)

    To my knowledge there are no long running transactions.  Most are product list fetches which should take less than a second each.  There is some content editing right now, so there are about 50-100 updates per day, (which again should take no more than 1 sec per transaction)

    I have been searching through my code, but can't find anything that seems out of the ordinary.

    Is there an object available to me that will report open connections?

    Thanks,
    Joe

    On Apr 15, 2010, at 8:18 AM, Michael Gentry wrote:

    The connections are stored and reused.  What is your min/max setting?
    Do you have any long-running transactions?

    On Wed, Apr 14, 2010 at 11:34 PM, Joe Baldwin wrote:
    Why would the DataContext be running out of connections in a web app with only a small amount of traffic?

    (I thought that a group of connections were stored (base on the modeler specification), and then reused for each transaction.)

    Joe
  • Joe Baldwin at Apr 16, 2010 at 9:57 pm
    Michael & Andrus,

    The DataSource Factory specified in CM is:

    org.apache.cayenne.conf.DriverDataSourceFactory
    I've deployed a long-running (up 24x7) internal application before
    that had a max of 2 DB connections ...

    Interesting - I saw one of you examples in which you used min = max = 1. So this is not expected behavior at all.

    The catalina log reported an exception (just prior to the connection exception)

    java.sql.SQLException: Lock wait timeout exceeded; (see detail in attached file)

    Is it possible that I have inadvertently configured improper locking? (Note: I never explicitly set locking.)

    The only theory I can think of at this point is *really* bizarre: is it possible content editing of one product is locking the entire list of products? (The ProductDetail class is a JSP that simply displays a Product. At the same time, there is content editing going on. My expectation would be that there is row-level locking and that this would not interfere with displaying a row.)

    RE open connections (I thought the PoolManager might report that, if my session gets one auto-magically).

    Thanks,
    Joe
  • Mike Kienenberger at Apr 16, 2010 at 10:08 pm
    Someone more knowledgable would have to comment, but my suspicion is
    that the problem is with your database or JDBC driver rather than with
    Cayenne.

    Perhaps if you describe your database setup (database, version,
    anything else relevent), someone might have some additional ideas.

    You might also try searching on the SQLException. Here's one
    interesting hit I found:

    http://forums.mysql.com/read.php?22,37371,39620#msg-39620

    On Fri, Apr 16, 2010 at 5:57 PM, Joe Baldwin wrote:
    Michael & Andrus,

    The DataSource Factory specified in CM is:

    org.apache.cayenne.conf.DriverDataSourceFactory
    I've deployed a long-running (up 24x7) internal application before
    that had a max of 2 DB connections ...

    Interesting - I saw one of you examples in which you used min = max = 1.  So this is not expected behavior at all.

    The catalina log reported an exception (just prior to the connection exception)

    java.sql.SQLException: Lock wait timeout exceeded;  (see detail in attached file)

    Is it possible that I have inadvertently configured improper locking?   (Note: I never explicitly set locking.)

    The only theory I can think of at this point is *really* bizarre: is it possible content editing of one product is locking the entire list of products?  (The ProductDetail class is a JSP that simply displays a Product.  At the same time, there is content editing going on.  My expectation would be that there is row-level locking and that this would not interfere with displaying a row.)

    RE open connections (I thought the PoolManager might report that, if my session gets one auto-magically).

    Thanks,
    Joe









    On Apr 16, 2010, at 8:57 AM, Michael Gentry wrote:

    Hi Joe,

    I've deployed a long-running (up 24x7) internal application before
    that had a max of 2 DB connections using the built-in Cayenne
    connection pooling mechanism and never had any issues with running out
    of connections.  The only reason I even used 2 was I had long-running
    (several minutes) queries in a background thread that might block the
    interactive portion and make the user wait during those periods, so I
    basically allocated one to the background thread and one to the user
    threads.

    I don't know of a way to report open connections off the top of my
    head, but I could look into that if needed.

    mrg

    On Thu, Apr 15, 2010 at 8:57 AM, Joe Baldwin wrote:
    Hi Michael,

    min = 1
    max = 10 (we changed this to 20 but then got the same error)

    To my knowledge there are no long running transactions.  Most are product list fetches which should take less than a second each.  There is some content editing right now, so there are about 50-100 updates per day, (which again should take no more than 1 sec per transaction)

    I have been searching through my code, but can't find anything that seems out of the ordinary.

    Is there an object available to me that will report open connections?

    Thanks,
    Joe

    On Apr 15, 2010, at 8:18 AM, Michael Gentry wrote:

    The connections are stored and reused.  What is your min/max setting?
    Do you have any long-running transactions?

    On Wed, Apr 14, 2010 at 11:34 PM, Joe Baldwin wrote:
    Why would the DataContext be running out of connections in a web app with only a small amount of traffic?

    (I thought that a group of connections were stored (base on the modeler specification), and then reused for each transaction.)

    Joe
  • Joe Baldwin at Apr 17, 2010 at 1:49 am
    Mike,

    the database is running on a production machine:

    Sun MySQL - 5.0.51a
    Linux 2.6.25-14.fc9.x86_64 (amd64)
    Java 1.6

    One thing I noticed was that my webhost is using MySQL 5.0.51a, while Wikipedia documents 5.1.45 as the most recent stable release. The JDBC driver is provided by the host.

    I am not sure why they are using 5.0.51a. Could this cause a problem with Cayenne?

    Joe



    On Apr 16, 2010, at 6:07 PM, Mike Kienenberger wrote:

    Someone more knowledgable would have to comment, but my suspicion is
    that the problem is with your database or JDBC driver rather than with
    Cayenne.

    Perhaps if you describe your database setup (database, version,
    anything else relevent), someone might have some additional ideas.

    You might also try searching on the SQLException. Here's one
    interesting hit I found:

    http://forums.mysql.com/read.php?22,37371,39620#msg-39620

    On Fri, Apr 16, 2010 at 5:57 PM, Joe Baldwin wrote:
    Michael & Andrus,

    The DataSource Factory specified in CM is:

    org.apache.cayenne.conf.DriverDataSourceFactory
    I've deployed a long-running (up 24x7) internal application before
    that had a max of 2 DB connections ...

    Interesting - I saw one of you examples in which you used min = max = 1. So this is not expected behavior at all.

    The catalina log reported an exception (just prior to the connection exception)

    java.sql.SQLException: Lock wait timeout exceeded; (see detail in attached file)

    Is it possible that I have inadvertently configured improper locking? (Note: I never explicitly set locking.)

    The only theory I can think of at this point is *really* bizarre: is it possible content editing of one product is locking the entire list of products? (The ProductDetail class is a JSP that simply displays a Product. At the same time, there is content editing going on. My expectation would be that there is row-level locking and that this would not interfere with displaying a row.)

    RE open connections (I thought the PoolManager might report that, if my session gets one auto-magically).

    Thanks,
    Joe









    On Apr 16, 2010, at 8:57 AM, Michael Gentry wrote:

    Hi Joe,

    I've deployed a long-running (up 24x7) internal application before
    that had a max of 2 DB connections using the built-in Cayenne
    connection pooling mechanism and never had any issues with running out
    of connections. The only reason I even used 2 was I had long-running
    (several minutes) queries in a background thread that might block the
    interactive portion and make the user wait during those periods, so I
    basically allocated one to the background thread and one to the user
    threads.

    I don't know of a way to report open connections off the top of my
    head, but I could look into that if needed.

    mrg

    On Thu, Apr 15, 2010 at 8:57 AM, Joe Baldwin wrote:
    Hi Michael,

    min = 1
    max = 10 (we changed this to 20 but then got the same error)

    To my knowledge there are no long running transactions. Most are product list fetches which should take less than a second each. There is some content editing right now, so there are about 50-100 updates per day, (which again should take no more than 1 sec per transaction)

    I have been searching through my code, but can't find anything that seems out of the ordinary.

    Is there an object available to me that will report open connections?

    Thanks,
    Joe

    On Apr 15, 2010, at 8:18 AM, Michael Gentry wrote:

    The connections are stored and reused. What is your min/max setting?
    Do you have any long-running transactions?

    On Wed, Apr 14, 2010 at 11:34 PM, Joe Baldwin wrote:
    Why would the DataContext be running out of connections in a web app with only a small amount of traffic?

    (I thought that a group of connections were stored (base on the modeler specification), and then reused for each transaction.)

    Joe
  • Aristedes Maniatis at Apr 17, 2010 at 4:38 am

    On 17/04/10 11:49 AM, Joe Baldwin wrote:
    the database is running on a production machine:

    Sun MySQL - 5.0.51a
    Linux 2.6.25-14.fc9.x86_64 (amd64)
    Java 1.6

    One thing I noticed was that my webhost is using MySQL 5.0.51a, while Wikipedia documents 5.1.45 as the most recent stable release. The JDBC driver is provided by the host.

    I am not sure why they are using 5.0.51a. Could this cause a problem with Cayenne?

    Very many people have stayed on the 5.0.x releases due to perceived problems with 5.1.x. There is nothing in either of those releases that should make the slightest difference to Cayenne.

    You should look at running a recent version of the JDBC driver.

    Ari


    --
    -------------------------->
    Aristedes Maniatis
    ish
    http://www.ish.com.au
    Level 1, 30 Wilson Street Newtown 2042 Australia
    phone +61 2 9550 5001 fax +61 2 9550 4001
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
  • Joe Baldwin at Apr 17, 2010 at 6:51 am
    Ari,
    Very many people have stayed on the 5.0.x releases due to perceived problems with 5.1.x. There is nothing in either of those releases that should make the slightest difference to Cayenne.

    I did some more research and found a report with *similar* symptoms

    http://bugs.mysql.com/bug.php?id=16979

    The complaint is that in an early 5.0 MySQL:

    "When accessing the auto-increment counter, InnoDB uses a special table-level AUTO-INC lock that it keeps to the end of the current SQL statement, "

    This appears to document anomalous locking behavior associated with MySQL 5.0 innodb, that the subsequent postings suggest will be fixed in 5.1. If this is in fact table level locking then a simple update could cause problems when there are multiple users reading & writing data. (I have no idea how to reproduce this, or test the extent to which the innodb locking has cascading affects, however if this could cause connection problems then it could be the culprit)

    You guys know way more about DBMS behavior than I; does this sound like it could be the problem (remember the logs show the locking exception is typically just prior to the connection exception)?

    If so, then a simple upgrade to MySQL 5.1 might fix the problem.

    Thanks,
    Joe



    On Apr 16, 2010, at 10:51 PM, Aristedes Maniatis wrote:
    On 17/04/10 11:49 AM, Joe Baldwin wrote:
    the database is running on a production machine:

    Sun MySQL - 5.0.51a
    Linux 2.6.25-14.fc9.x86_64 (amd64)
    Java 1.6

    One thing I noticed was that my webhost is using MySQL 5.0.51a, while Wikipedia documents 5.1.45 as the most recent stable release. The JDBC driver is provided by the host.

    I am not sure why they are using 5.0.51a. Could this cause a problem with Cayenne?

    Very many people have stayed on the 5.0.x releases due to perceived problems with 5.1.x. There is nothing in either of those releases that should make the slightest difference to Cayenne.

    You should look at running a recent version of the JDBC driver.

    Ari


    --
    -------------------------->
    Aristedes Maniatis
    ish
    http://www.ish.com.au
    Level 1, 30 Wilson Street Newtown 2042 Australia
    phone +61 2 9550 5001 fax +61 2 9550 4001
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
  • Mike Kienenberger at Apr 17, 2010 at 7:20 am
    It looks like 5.0.90 is the latest 5.0 release, so I'd say that
    upgrading to at least would be a good start.

    http://downloads.mysql.com/archives.php?p=mysql-5.0

    It also says on that page that 5.0 is the oldest version currently supported.

    5.0.50a is from Jan 2008.

    The most current version is 6.0, with major releases of 5.1, 5.2, 5.4, and 5.5.

    While I haven't used mysql, I've seen a lot of postings about it over
    the years, and it's also very important that you have the most current
    jdbc driver for your version.

    The only hit on this sql error in my personal cayenne archives is this
    56-message thread from 2006. Looks very relevant to your situation,
    although I've only glanced through it . You can find it here.

    http://www.mail-archive.com/cayenne-user@incubator.apache.org/msg00769.html

    On Sat, Apr 17, 2010 at 2:50 AM, Joe Baldwin wrote:
    Ari,
    Very many people have stayed on the 5.0.x releases due to perceived problems with 5.1.x. There is nothing in either of those releases that should make the slightest difference to Cayenne.

    I did some more research and found a report with *similar* symptoms

    http://bugs.mysql.com/bug.php?id=16979

    The complaint is that in an early 5.0 MySQL:

    "When accessing the auto-increment counter, InnoDB uses a special table-level AUTO-INC lock that it keeps to the end of the current SQL statement, "

    This appears to document anomalous locking behavior associated with MySQL 5.0 innodb, that the subsequent postings suggest will be fixed in 5.1.  If this is in fact table level locking then a simple update could cause problems when there are multiple users reading & writing data.  (I have no idea how to reproduce this, or test the extent to which the innodb locking has cascading affects, however if this could cause connection problems then it could be the culprit)

    You guys know way more about DBMS behavior than I; does this sound like it could be the problem (remember the logs show the locking exception is typically just prior to the connection exception)?

    If so, then a simple upgrade to MySQL 5.1 might fix the problem.

    Thanks,
    Joe



    On Apr 16, 2010, at 10:51 PM, Aristedes Maniatis wrote:
    On 17/04/10 11:49 AM, Joe Baldwin wrote:
    the database is running on a production machine:

    Sun MySQL - 5.0.51a
    Linux 2.6.25-14.fc9.x86_64 (amd64)
    Java 1.6

    One thing I noticed was that my webhost is using MySQL 5.0.51a, while Wikipedia documents 5.1.45 as the most recent stable release.  The JDBC driver is provided by the host.

    I am not sure why they are using 5.0.51a. Could this cause a problem with Cayenne?

    Very many people have stayed on the 5.0.x releases due to perceived problems with 5.1.x. There is nothing in either of those releases that should make the slightest difference to Cayenne.

    You should look at running a recent version of the JDBC driver.

    Ari


    --
    -------------------------->
    Aristedes Maniatis
    ish
    http://www.ish.com.au
    Level 1, 30 Wilson Street Newtown 2042 Australia
    phone +61 2 9550 5001   fax +61 2 9550 4001
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
  • Joe Baldwin at Apr 17, 2010 at 8:18 am
    Well, Wikipedia claims that the recommended stable version is 5.1.45. However, the webhost claims that he has too many users to be able to upgrade (which is a problematic answer since this will mean he can never upgrade).

    He is convinced (without any proof) that the problem lies with my code or with Cayenne. However Michael was able to run his app configured for only two connections. Furthermore there is that innodb auto-increment lock bug clearly documented for MySQL 5.0.

    I don't see any documented Cayenne bugs that could possibly cause these exceptions. My configuration seems to be well within the norm (based on the recent posts). That only leaves the documented innodb bug.

    Almost forgot, just prior to each one of the lock-exceptions an administrator was attempting to add a new product to the database (which would certainly require interacting with innodb).

    Question:
    If the innodb bug is causing the lock-exception, would this then cause the ensuing connection exception? (Again, I cannot see how Cayenne or my nominal configuration could cause both of these errors that are chronologically linked.)

    Joe

    On Apr 17, 2010, at 3:20 AM, Mike Kienenberger wrote:

    It looks like 5.0.90 is the latest 5.0 release, so I'd say that
    upgrading to at least would be a good start.

    http://downloads.mysql.com/archives.php?p=mysql-5.0

    It also says on that page that 5.0 is the oldest version currently supported.

    5.0.50a is from Jan 2008.

    The most current version is 6.0, with major releases of 5.1, 5.2, 5.4, and 5.5.

    While I haven't used mysql, I've seen a lot of postings about it over
    the years, and it's also very important that you have the most current
    jdbc driver for your version.

    The only hit on this sql error in my personal cayenne archives is this
    56-message thread from 2006. Looks very relevant to your situation,
    although I've only glanced through it . You can find it here.

    http://www.mail-archive.com/cayenne-user@incubator.apache.org/msg00769.html

    On Sat, Apr 17, 2010 at 2:50 AM, Joe Baldwin wrote:
    Ari,
    Very many people have stayed on the 5.0.x releases due to perceived problems with 5.1.x. There is nothing in either of those releases that should make the slightest difference to Cayenne.

    I did some more research and found a report with *similar* symptoms

    http://bugs.mysql.com/bug.php?id=16979

    The complaint is that in an early 5.0 MySQL:

    "When accessing the auto-increment counter, InnoDB uses a special table-level AUTO-INC lock that it keeps to the end of the current SQL statement, "

    This appears to document anomalous locking behavior associated with MySQL 5.0 innodb, that the subsequent postings suggest will be fixed in 5.1. If this is in fact table level locking then a simple update could cause problems when there are multiple users reading & writing data. (I have no idea how to reproduce this, or test the extent to which the innodb locking has cascading affects, however if this could cause connection problems then it could be the culprit)

    You guys know way more about DBMS behavior than I; does this sound like it could be the problem (remember the logs show the locking exception is typically just prior to the connection exception)?

    If so, then a simple upgrade to MySQL 5.1 might fix the problem.

    Thanks,
    Joe



    On Apr 16, 2010, at 10:51 PM, Aristedes Maniatis wrote:
    On 17/04/10 11:49 AM, Joe Baldwin wrote:
    the database is running on a production machine:

    Sun MySQL - 5.0.51a
    Linux 2.6.25-14.fc9.x86_64 (amd64)
    Java 1.6

    One thing I noticed was that my webhost is using MySQL 5.0.51a, while Wikipedia documents 5.1.45 as the most recent stable release. The JDBC driver is provided by the host.

    I am not sure why they are using 5.0.51a. Could this cause a problem with Cayenne?

    Very many people have stayed on the 5.0.x releases due to perceived problems with 5.1.x. There is nothing in either of those releases that should make the slightest difference to Cayenne.

    You should look at running a recent version of the JDBC driver.

    Ari


    --
    -------------------------->
    Aristedes Maniatis
    ish
    http://www.ish.com.au
    Level 1, 30 Wilson Street Newtown 2042 Australia
    phone +61 2 9550 5001 fax +61 2 9550 4001
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
  • Michael Gentry at Apr 17, 2010 at 9:22 pm
    It should also be noted that I was running against Sybase, so a
    completely different JDBC driver and database. I'm not saying it is
    definitely a MySQL issue, but it could be the JDBC driver or the
    backend storage implementation. I'd be surprised if it was a direct
    Cayenne issue. It would be good if you could test (locally) on a
    different setup and see if it still happens.

    mrg

    On Sat, Apr 17, 2010 at 4:18 AM, Joe Baldwin wrote:
    However Michael was able to run his app configured for only two connections.
  • Joe Baldwin at Apr 18, 2010 at 1:05 am
    Michael,
    I'm not saying it is definitely a MySQL issue, but it could be the JDBC driver or the backend storage implementation.
    True. The production box is using a very old version of MySQL with documented innodb bugs, so it could be anything from innodb to the connector that is causing the problem.
    I'd be surprised if it was a direct Cayenne issue.
    I agree, if there are no open Cayenne bugs, and you have personally deployed configured for only 1-2 connections with no errors, that appears to rule out Cayenne. My code is pretty standard cayenne select queries and updates, and has been using Cayenne for over a year on the development box without any such errors.
    It would be good if you could test (locally) on a different setup and see if it still happens.

    These exceptions and 60sec load times only occur on the production machine and have never occurred on the development machine. The only difference is the hardware and the version of MySQL.

    All the evidence seems to be pointing to MySQL and/or the JDBC connector. In my opinion, the unresolved innodb bug is the proverbial "elephant in the living room". How it is being triggered is simple to understand, however the cascade effects are probably due to a combination requests by the small number of concurrent users.

    Thanks so much for your attention to this. I think Cayenne has been ruled out.
    Joe




    On Apr 17, 2010, at 5:21 PM, Michael Gentry wrote:

    It should also be noted that I was running against Sybase, so a
    completely different JDBC driver and database. I'm not saying it is
    definitely a MySQL issue, but it could be the JDBC driver or the
    backend storage implementation. I'd be surprised if it was a direct
    Cayenne issue. It would be good if you could test (locally) on a
    different setup and see if it still happens.

    mrg

    On Sat, Apr 17, 2010 at 4:18 AM, Joe Baldwin wrote:
    However Michael was able to run his app configured for only two connections.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriescayenne
postedApr 15, '10 at 3:35a
activeApr 18, '10 at 1:05a
posts14
users5
websitecayenne.apache.org

People

Translate

site design / logo © 2022 Grokbase