FAQ

[HBase-issues] [jira] Updated: (HBASE-2248) Provide new non-copy mechanism to assure atomic reads in get and scan

ryan rawson (JIRA)
Apr 6, 2010 at 12:52 am
[ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ryan rawson updated HBASE-2248:
-------------------------------

Attachment: HBASE-2248-rr-alpha2.txt

Here is my update to my patch, this time I am using iterators to scan the memstore and snapshot. There are a number of fixes to all sorts of fun race conditions, etc.

The best news: this is the fastest memstore scanner HBase has seen. It is about 15x faster than the 0.20.3 version based on the microbenchmark included in the patch. The old code takes about 400-500ms to scan 250k KeyValues in memstore, and this new patch takes 25-30ms.

I haven't run all the tests yet, but it passes the core TestMemStore and TestHRegion which contain all the hard tests that have concurrency.
Provide new non-copy mechanism to assure atomic reads in get and scan
---------------------------------------------------------------------

Key: HBASE-2248
URL: https://issues.apache.org/jira/browse/HBASE-2248
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.3
Reporter: Dave Latham
Priority: Blocker
Fix For: 0.20.4

Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha1.txt, HBASE-2248-rr-alpha2.txt, HBASE-2248-ryan.patch, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
reply

Search Discussions

17 responses

  • ryan rawson (JIRA) at Apr 6, 2010 at 1:40 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: HBASE-2248-rr-alpha3.txt

    a version that compiles and passes TestHeapSize
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha1.txt, HBASE-2248-rr-alpha2.txt, HBASE-2248-rr-alpha3.txt, HBASE-2248-ryan.patch, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 8, 2010 at 10:26 pm
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: HBASE-2248-rr-pre-durability.txt
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha1.txt, HBASE-2248-rr-alpha2.txt, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability.txt, HBASE-2248-ryan.patch, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 8, 2010 at 10:31 pm
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: HBASE-2248-fix-delete-test.txt

    this patch fixes the testDelete failure that people have been seeing. it should apply on both 0.20 and 0.20_pre_durability patches
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-fix-delete-test.txt, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha1.txt, HBASE-2248-rr-alpha2.txt, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability.txt, HBASE-2248-ryan.patch, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 9, 2010 at 12:49 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: HBASE-2248-rr-pre-durability2.txt
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability.txt, HBASE-2248-rr-pre-durability2.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 9, 2010 at 12:49 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: (was: HBASE-2248-rr-alpha1.txt)
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability.txt, HBASE-2248-rr-pre-durability2.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 9, 2010 at 12:49 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: (was: HBASE-2248-fix-delete-test.txt)
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability.txt, HBASE-2248-rr-pre-durability2.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 9, 2010 at 12:49 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: (was: HBASE-2248-rr-alpha2.txt)
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability.txt, HBASE-2248-rr-pre-durability2.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 9, 2010 at 12:49 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: (was: HBASE-2248-ryan.patch)
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability.txt, HBASE-2248-rr-pre-durability2.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 9, 2010 at 12:50 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: (was: HBASE-2248-rr-pre-durability.txt)
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability2.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • ryan rawson (JIRA) at Apr 9, 2010 at 7:42 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: HBASE-2248-rr-pre-durability3.txt

    unfortunately the way the branch and the branch-of-branch does things have diverged a lot. specifically the locations of the update lock and the flush request. I rearranged things a bunch and have this new patch. This might help with the PE slowness and other things as well.
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability2.txt, HBASE-2248-rr-pre-durability3.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Purtell (JIRA) at Apr 12, 2010 at 3:27 pm
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Purtell updated HBASE-2248:
    ----------------------------------

    Attachment: profile.png
    put_call_graph.png

    We spend 76% of CPU time in ReadWriteConsistencyControl.completeMemstoreInsert. See attached 'profile.png'. Draining the write queue (for 42,175 puts?) explodes 42,166 calls to ReadWriteConsistencyControl.completeMemstoreInsert into 121,118,233 calls to AtomicLong.get and 121,143,574 calls to $WriteEntry.getWriteNumber, each arc represents 50% of the cumulative time there. See attached 'put_call_graph.png'.
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability2.txt, HBASE-2248-rr-pre-durability3.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, profile.png, put_call_graph.png, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • ryan rawson (JIRA) at Apr 14, 2010 at 4:49 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: HBASE-2248-rr-pre-durability4.txt

    ok here is a patch that addresses all the above issues:
    - spin fixed by restructuring hlog append
    - index test pass failure fixed
    - test failures due to compaction
    - all comments addressed

    To accomplish the index hbase fix, I had to introduce a new notion of optional scanner creation atomicity along with pre-flush-commit work, so a sub-class can create an atomic section whereby some work is done (eg: switching out an index) and the flush commit (where the snapshot is removed and the hfile is introduced to open scanners) and this atomic section will be atomic relative to new scanner creation. This was required to fix race conditions in indexed hbase, which also means that indexed hbase is not as fast as it can be, since it cannot create new scanners during this one critical phase of flush (which includes re-reading scanner blocks btw).


    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Assignee: ryan rawson
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-alpha3.txt, HBASE-2248-rr-pre-durability2.txt, HBASE-2248-rr-pre-durability3.txt, HBASE-2248-rr-pre-durability4.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, profile.png, put_call_graph.png, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • ryan rawson (JIRA) at Apr 15, 2010 at 1:42 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: (was: HBASE-2248-rr-alpha3.txt)
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Assignee: ryan rawson
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-final1.txt, HBASE-2248-rr-pre-durability4.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, profile.png, put_call_graph.png, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • ryan rawson (JIRA) at Apr 15, 2010 at 1:42 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: HBASE-2248-rr-final1.txt

    here is the updated version with fixes taken from the work on the 0.20_pre_durability but on plain old 0.20.
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Assignee: ryan rawson
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-final1.txt, HBASE-2248-rr-pre-durability4.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, profile.png, put_call_graph.png, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • ryan rawson (JIRA) at Apr 15, 2010 at 1:42 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: (was: HBASE-2248-rr-pre-durability2.txt)
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Assignee: ryan rawson
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-final1.txt, HBASE-2248-rr-pre-durability4.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, profile.png, put_call_graph.png, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • ryan rawson (JIRA) at Apr 15, 2010 at 1:42 am
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: (was: HBASE-2248-rr-pre-durability3.txt)
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Assignee: ryan rawson
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-rr-final1.txt, HBASE-2248-rr-pre-durability4.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, profile.png, put_call_graph.png, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • ryan rawson (JIRA) at Apr 15, 2010 at 9:51 pm
    [ https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    ryan rawson updated HBASE-2248:
    -------------------------------

    Attachment: HBASE-2248-no-row-locks.txt

    this removes row locks which are no longer necessary to ensure atomic reads
    Provide new non-copy mechanism to assure atomic reads in get and scan
    ---------------------------------------------------------------------

    Key: HBASE-2248
    URL: https://issues.apache.org/jira/browse/HBASE-2248
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.20.3
    Reporter: Dave Latham
    Assignee: ryan rawson
    Priority: Blocker
    Fix For: 0.20.4

    Attachments: HBASE-2248-demonstrate-previous-impl-bugs.patch, HBASE-2248-GetsAsScans3.patch, HBASE-2248-no-row-locks.txt, HBASE-2248-rr-final1.txt, HBASE-2248-rr-pre-durability4.txt, hbase-2248.gc, HBASE-2248.patch, hbase-2248.txt, profile.png, put_call_graph.png, readownwrites-lost.2.patch, readownwrites-lost.patch, Screen shot 2010-02-23 at 10.33.38 AM.png, threads.txt


    HBASE-2037 introduced a new MemStoreScanner which triggers a ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when starting a scan.
    After upgrading to 0.20.3, we noticed a big slowdown in our use of short scans. Some of our data repesent a time series. The data is stored in time series order, MR jobs often insert/update new data at the end of the series, and queries usually have to pick up some or all of the series. These are often scans of 0-100 rows at a time. To load one page, we'll observe about 20 such scans being triggered concurrently, and they take 2 seconds to complete. Doing a thread dump of a region server shows many threads in ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key values to copy it.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post

1 user in discussion

ryan rawson (JIRA): 18 posts