FAQ
Hi,

Since 1.8 I'm experiencing rather slow count() performance.

Bench platform : x2 quadcore, 32GB, 146GB SAS RAID5 disks, and 6
millions lines collection (requested over a date integer, index
created)
Test method : a given number (let's say n) of python threads are setup
to request each an equally sized fraction of the 6 millions documents
collection (using $gte, $lte).

I can see the following :
- using n Python client threads will spawn n mongod threads, which sounds good.
- using more than 9 python threads will affect performance (9 threads
= completion in 6s, 16 threads 9s ) which is odd
- the iostat -xm 2 shows the following :

avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.00 0.00 100.00

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
14.12 0.00 8.24 0.00 0.00 77.64

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.50 0.00 0.00 8.00
0.01 12.00 12.00 0.60
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda3 0.00 0.00 0.00 0.50 0.00 0.00 8.00
0.01 12.00 12.00 0.60
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
33.81 0.00 18.97 0.06 0.00 47.16

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 31.50 0.00 2.50 0.00 0.13 108.80
0.02 6.60 4.40 1.10
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda5 0.00 31.50 0.00 2.50 0.00 0.13 108.80
0.02 6.60 4.40 1.10
sda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00

The IOs on the disks seem low.

- the mongosttat shows the following :
insert query update delete getmore command flushes mapped vsize
res faults locked % idx miss % qr|qw ar|aw netIn netOut conn
time
0 0 0 0 0 1 0 13.9g 28.8g
16.7g 0 0 0 0|0 0|0 62b 1k 1
11:55:36
0 0 0 0 0 1 0 13.9g 28.8g
16.7g 0 0 0 0|0 0|0 62b 1k 1
11:55:37
0 0 0 0 0 1 0 13.9g 28.8g
16.7g 0 0 0 0|0 0|0 62b 1k 1
11:55:38
0 0 0 0 0 11 0 13.9g 28.8g
16.7g 0 0 0 0|0 9|0 120b 1k 10
11:55:39

- the explain shows the following :
db.Logs.find({'date':{'$gt':1293736476, '$lt':1303836584}}).explain()
{
"cursor" : "BtreeCursor date_1",
"nscanned" : 6430895,
"nscannedObjects" : 6430895,
"n" : 6430895,
"millis" : 9354,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" : {
"date" : [
[
1293736476,
1303836584
]
]
}
}



As far as I understand from JIRA SERVER-1752, the enhancement of
count() is planned for 2.1. My questions are :
- counting 6 millions lines on 16 cores in 6 seconds can be shortened
? Is it normal performance given this HW configuration ?
- Will the aggregation framework will speed things up ?
- Regarding the explain(), could you please confirme that the
"nscannedObjects" doesn't mean that the collection itself was scanned
and that only index entries were used ?

Thank you !

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

Search Discussions

  • Adam C at Jan 31, 2012 at 2:59 pm
    Hi Sékine,
    - counting 6 millions lines on 16 cores in 6 seconds can be shortened
    ? Is it normal performance given this HW configuration?
    I have seen similar performance results reported (see some of the
    examples in the server issue you linked in fact).  My own count of 6
    million lines on a Macbook Pro with SSD was about 7.5 seconds on 2.0.2
    - Will the aggregation framework will speed things up?
    The aggregation framework is intended to bring more functionality
    natively to mongoDB, remove the need to use Map/Reduce jobs, and to
    provide better performance than Map/Reduce for those operations. An
    improvement over the current count() function would be nice, but is
    not the stated goal there.  My initial testing, though not exhaustive,
    on the 2.1 nightly build did not show any significant improvement
    using the framework.
    - Regarding the explain(), could you please confirme that the
    "nscannedObjects" doesn't mean that the collection itself was scanned
    and that only index entries were used ?
    Actually, the important part from an index use point of view is the
    first line regarding the cursor:

    "cursor" : "BtreeCursor date_1"

    This indicates that the query used the index date_1, you can see it
    (and any others) by doing a find on system.indexes in that collection,
    for example:

    db.system.indexes.find({"name" : "date_1"})

    Adam.
    On Jan 31, 11:16 am, Sékine Coulibaly wrote:
    Hi,

    Since 1.8 I'm experiencing rather slow count() performance.

    Bench platform : x2 quadcore, 32GB, 146GB SAS RAID5 disks, and 6
    millions lines collection (requested over a date integer, index
    created)
    Test method : a given number (let's say n) of python threads are setup
    to request each an equally sized fraction of the 6 millions documents
    collection (using $gte, $lte).

    I can see the following :
    - using n Python client threads will spawn n mongod threads, which sounds good.
    - using more than 9 python threads will affect performance (9 threads
    = completion in 6s, 16 threads 9s ) which is odd
    - the iostat -xm 2 shows the following :

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    0.00    0.00    0.00    0.00    0.00  100.00

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    14.12    0.00    8.24    0.00    0.00   77.64

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.50     0.00     0.00     8.00
    0.01   12.00  12.00   0.60
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.50     0.00     0.00     8.00
    0.01   12.00  12.00   0.60
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    33.81    0.00   18.97    0.06    0.00   47.16

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00    31.50  0.00  2.50     0.00     0.13   108.80
    0.02    6.60   4.40   1.10
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00    31.50  0.00  2.50     0.00     0.13   108.80
    0.02    6.60   4.40   1.10
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    The IOs on the disks seem low.

    - the mongosttat shows the following :
    insert  query update delete getmore command flushes mapped  vsize
    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn
    time
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:36
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:37
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:38
    0      0      0      0       0      11       0  13.9g  28.8g
    16.7g      0        0          0       0|0     9|0   120b     1k    10
    11:55:39

    - the explain shows the following :
    db.Logs.find({'date':{'$gt':1293736476, '$lt':1303836584}}).explain()
    {
    "cursor" : "BtreeCursor date_1",
    "nscanned" : 6430895,
    "nscannedObjects" : 6430895,
    "n" : 6430895,
    "millis" : 9354,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "isMultiKey" : false,
    "indexOnly" : false,
    "indexBounds" : {
    "date" : [
    [
    1293736476,
    1303836584
    ]
    ]
    }

    }

    As far as I understand from JIRA SERVER-1752, the enhancement of
    count() is planned for 2.1. My questions are :
    - counting 6 millions lines on 16 cores in 6 seconds can be shortened
    ? Is it normal performance given this HW configuration ?
    - Will the aggregation framework will speed things up ?
    - Regarding the explain(), could you please confirme that the
    "nscannedObjects" doesn't mean that the collection itself was scanned
    and that only index entries were used ?

    Thank you !
    --
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com.
    To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
  • Sékine Coulibaly at Jan 31, 2012 at 6:42 pm
    Hi Adam,

    Thanks for getting back in touch with your own benchmark results :)
    One question though : how can we explain that using 16 cores is not
    more effective than using 9 of them (just wondering, might me
    Linux/kernel related issue) ?

    So as far as I understand, the only option here to improve count() is
    scaling out right ?

    The SERVER-1752, enhancing count() performance was due to ship in 2.1.
    However, it's not marked as "Solved". Can we expect something to ship
    in 2.1 or will it be delayed ?

    Aggregation framework is a real breakthrough indeed. I wish it had
    included something regarding count() performance ;) !

    Thanks again !

    Regards

    Le 31 janvier 2012 15:59, Adam C <adamc@10gen.com> a écrit :
    Hi Sékine,
    - counting 6 millions lines on 16 cores in 6 seconds can be shortened
    ? Is it normal performance given this HW configuration?
    I have seen similar performance results reported (see some of the
    examples in the server issue you linked in fact).  My own count of 6
    million lines on a Macbook Pro with SSD was about 7.5 seconds on 2.0.2
    - Will the aggregation framework will speed things up?
    The aggregation framework is intended to bring more functionality
    natively to mongoDB, remove the need to use Map/Reduce jobs, and to
    provide better performance than Map/Reduce for those operations. An
    improvement over the current count() function would be nice, but is
    not the stated goal there.  My initial testing, though not exhaustive,
    on the 2.1 nightly build did not show any significant improvement
    using the framework.
    - Regarding the explain(), could you please confirme that the
    "nscannedObjects" doesn't mean that the collection itself was scanned
    and that only index entries were used ?
    Actually, the important part from an index use point of view is the
    first line regarding the cursor:

    "cursor" : "BtreeCursor date_1"

    This indicates that the query used the index date_1, you can see it
    (and any others) by doing a find on system.indexes in that collection,
    for example:

    db.system.indexes.find({"name" : "date_1"})

    Adam.
    On Jan 31, 11:16 am, Sékine Coulibaly wrote:
    Hi,

    Since 1.8 I'm experiencing rather slow count() performance.

    Bench platform : x2 quadcore, 32GB, 146GB SAS RAID5 disks, and 6
    millions lines collection (requested over a date integer, index
    created)
    Test method : a given number (let's say n) of python threads are setup
    to request each an equally sized fraction of the 6 millions documents
    collection (using $gte, $lte).

    I can see the following :
    - using n Python client threads will spawn n mongod threads, which sounds good.
    - using more than 9 python threads will affect performance (9 threads
    = completion in 6s, 16 threads 9s ) which is odd
    - the iostat -xm 2 shows the following :

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    0.00    0.00    0.00    0.00    0.00  100.00

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    14.12    0.00    8.24    0.00    0.00   77.64

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.50     0.00     0.00     8.00
    0.01   12.00  12.00   0.60
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.50     0.00     0.00     8.00
    0.01   12.00  12.00   0.60
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    33.81    0.00   18.97    0.06    0.00   47.16

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00    31.50  0.00  2.50     0.00     0.13   108.80
    0.02    6.60   4.40   1.10
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00    31.50  0.00  2.50     0.00     0.13   108.80
    0.02    6.60   4.40   1.10
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    The IOs on the disks seem low.

    - the mongosttat shows the following :
    insert  query update delete getmore command flushes mapped  vsize
    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn
    time
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:36
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:37
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:38
    0      0      0      0       0      11       0  13.9g  28.8g
    16.7g      0        0          0       0|0     9|0   120b     1k    10
    11:55:39

    - the explain shows the following :
    db.Logs.find({'date':{'$gt':1293736476, '$lt':1303836584}}).explain()
    {
    "cursor" : "BtreeCursor date_1",
    "nscanned" : 6430895,
    "nscannedObjects" : 6430895,
    "n" : 6430895,
    "millis" : 9354,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "isMultiKey" : false,
    "indexOnly" : false,
    "indexBounds" : {
    "date" : [
    [
    1293736476,
    1303836584
    ]
    ]
    }

    }

    As far as I understand from JIRA SERVER-1752, the enhancement of
    count() is planned for 2.1. My questions are :
    - counting 6 millions lines on 16 cores in 6 seconds can be shortened
    ? Is it normal performance given this HW configuration ?
    - Will the aggregation framework will speed things up ?
    - Regarding the explain(), could you please confirme that the
    "nscannedObjects" doesn't mean that the collection itself was scanned
    and that only index entries were used ?

    Thank you !
    --
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com.
    To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com.
    To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
  • Eliot Horowitz at Feb 1, 2012 at 6:18 am
    More threads can increase disk contention and actual lower performance.
    Hard to say without more info.

    SERVER-1752 is still in a planning stage, so not sure if it'll make it
    into 2.2 yet
    On Tue, Jan 31, 2012 at 1:41 PM, Sékine Coulibaly wrote:
    Hi Adam,

    Thanks for getting back in touch with your own benchmark results :)
    One question though : how can we explain that using 16 cores is not
    more effective than using 9 of them (just wondering, might me
    Linux/kernel related issue) ?

    So as far as I understand, the only option here to improve count() is
    scaling out right ?

    The SERVER-1752, enhancing count() performance was due to ship in 2.1.
    However, it's not marked as "Solved". Can we expect something to ship
    in 2.1 or will it be delayed ?

    Aggregation framework is a real breakthrough indeed. I wish it had
    included something regarding count() performance ;) !

    Thanks again !

    Regards

    Le 31 janvier 2012 15:59, Adam C <adamc@10gen.com> a écrit :
    Hi Sékine,
    - counting 6 millions lines on 16 cores in 6 seconds can be shortened
    ? Is it normal performance given this HW configuration?
    I have seen similar performance results reported (see some of the
    examples in the server issue you linked in fact).  My own count of 6
    million lines on a Macbook Pro with SSD was about 7.5 seconds on 2.0.2
    - Will the aggregation framework will speed things up?
    The aggregation framework is intended to bring more functionality
    natively to mongoDB, remove the need to use Map/Reduce jobs, and to
    provide better performance than Map/Reduce for those operations. An
    improvement over the current count() function would be nice, but is
    not the stated goal there.  My initial testing, though not exhaustive,
    on the 2.1 nightly build did not show any significant improvement
    using the framework.
    - Regarding the explain(), could you please confirme that the
    "nscannedObjects" doesn't mean that the collection itself was scanned
    and that only index entries were used ?
    Actually, the important part from an index use point of view is the
    first line regarding the cursor:

    "cursor" : "BtreeCursor date_1"

    This indicates that the query used the index date_1, you can see it
    (and any others) by doing a find on system.indexes in that collection,
    for example:

    db.system.indexes.find({"name" : "date_1"})

    Adam.
    On Jan 31, 11:16 am, Sékine Coulibaly wrote:
    Hi,

    Since 1.8 I'm experiencing rather slow count() performance.

    Bench platform : x2 quadcore, 32GB, 146GB SAS RAID5 disks, and 6
    millions lines collection (requested over a date integer, index
    created)
    Test method : a given number (let's say n) of python threads are setup
    to request each an equally sized fraction of the 6 millions documents
    collection (using $gte, $lte).

    I can see the following :
    - using n Python client threads will spawn n mongod threads, which sounds good.
    - using more than 9 python threads will affect performance (9 threads
    = completion in 6s, 16 threads 9s ) which is odd
    - the iostat -xm 2 shows the following :

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    0.00    0.00    0.00    0.00    0.00  100.00

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    14.12    0.00    8.24    0.00    0.00   77.64

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.50     0.00     0.00     8.00
    0.01   12.00  12.00   0.60
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.50     0.00     0.00     8.00
    0.01   12.00  12.00   0.60
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    33.81    0.00   18.97    0.06    0.00   47.16

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00    31.50  0.00  2.50     0.00     0.13   108.80
    0.02    6.60   4.40   1.10
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00    31.50  0.00  2.50     0.00     0.13   108.80
    0.02    6.60   4.40   1.10
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    The IOs on the disks seem low.

    - the mongosttat shows the following :
    insert  query update delete getmore command flushes mapped  vsize
    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn
    time
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:36
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:37
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:38
    0      0      0      0       0      11       0  13.9g  28.8g
    16.7g      0        0          0       0|0     9|0   120b     1k    10
    11:55:39

    - the explain shows the following :
    db.Logs.find({'date':{'$gt':1293736476, '$lt':1303836584}}).explain()
    {
    "cursor" : "BtreeCursor date_1",
    "nscanned" : 6430895,
    "nscannedObjects" : 6430895,
    "n" : 6430895,
    "millis" : 9354,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "isMultiKey" : false,
    "indexOnly" : false,
    "indexBounds" : {
    "date" : [
    [
    1293736476,
    1303836584
    ]
    ]
    }

    }

    As far as I understand from JIRA SERVER-1752, the enhancement of
    count() is planned for 2.1. My questions are :
    - counting 6 millions lines on 16 cores in 6 seconds can be shortened
    ? Is it normal performance given this HW configuration ?
    - Will the aggregation framework will speed things up ?
    - Regarding the explain(), could you please confirme that the
    "nscannedObjects" doesn't mean that the collection itself was scanned
    and that only index entries were used ?

    Thank you !
    --
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com.
    To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com.
    To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com.
    To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
  • GVP at Feb 1, 2012 at 8:47 pm
    Based on iostat you are not really using any IO.

    You get 6 seconds of run-time when you use 9 threads.
    You get 9 seconds of run-time when you use 16 threads.

    You say you have an x2 quadcore, so that's 8 cores. Using more than 8
    threads is probably not useful.

    You should run a "top" on the server to see what your CPU usage is
    like on both the server and the client. I suspect that CPU usage will
    be very high on the server. If so that's your route to making this
    faster.

    - Gates

    On Jan 31, 3:16 am, Sékine Coulibaly wrote:
    Hi,

    Since 1.8 I'm experiencing rather slow count() performance.

    Bench platform : x2 quadcore, 32GB, 146GB SAS RAID5 disks, and 6
    millions lines collection (requested over a date integer, index
    created)
    Test method : a given number (let's say n) of python threads are setup
    to request each an equally sized fraction of the 6 millions documents
    collection (using $gte, $lte).

    I can see the following :
    - using n Python client threads will spawn n mongod threads, which sounds good.
    - using more than 9 python threads will affect performance (9 threads
    = completion in 6s, 16 threads 9s ) which is odd
    - the iostat -xm 2 shows the following :

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    0.00    0.00    0.00    0.00    0.00  100.00

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    14.12    0.00    8.24    0.00    0.00   77.64

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.50     0.00     0.00     8.00
    0.01   12.00  12.00   0.60
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.50     0.00     0.00     8.00
    0.01   12.00  12.00   0.60
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    33.81    0.00   18.97    0.06    0.00   47.16

    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
    avgqu-sz   await  svctm  %util
    sda               0.00    31.50  0.00  2.50     0.00     0.13   108.80
    0.02    6.60   4.40   1.10
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda5              0.00    31.50  0.00  2.50     0.00     0.13   108.80
    0.02    6.60   4.40   1.10
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sda7              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00
    hda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
    0.00    0.00   0.00   0.00

    The IOs on the disks seem low.

    - the mongosttat shows the following :
    insert  query update delete getmore command flushes mapped  vsize
    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn
    time
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:36
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:37
    0      0      0      0       0       1       0  13.9g  28.8g
    16.7g      0        0          0       0|0     0|0    62b     1k     1
    11:55:38
    0      0      0      0       0      11       0  13.9g  28.8g
    16.7g      0        0          0       0|0     9|0   120b     1k    10
    11:55:39

    - the explain shows the following :
    db.Logs.find({'date':{'$gt':1293736476, '$lt':1303836584}}).explain()
    {
    "cursor" : "BtreeCursor date_1",
    "nscanned" : 6430895,
    "nscannedObjects" : 6430895,
    "n" : 6430895,
    "millis" : 9354,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "isMultiKey" : false,
    "indexOnly" : false,
    "indexBounds" : {
    "date" : [
    [
    1293736476,
    1303836584
    ]
    ]
    }

    }

    As far as I understand from JIRA SERVER-1752, the enhancement of
    count() is planned for 2.1. My questions are :
    - counting 6 millions lines on 16 cores in 6 seconds can be shortened
    ? Is it normal performance given this HW configuration ?
    - Will the aggregation framework will speed things up ?
    - Regarding the explain(), could you please confirme that the
    "nscannedObjects" doesn't mean that the collection itself was scanned
    and that only index entries were used ?

    Thank you !
    --
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com.
    To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmongodb-user @
categoriesmongodb
postedJan 31, '12 at 11:17a
activeFeb 1, '12 at 8:47p
posts5
users4
websitemongodb.org
irc#mongodb

People

Translate

site design / logo © 2022 Grokbase