FAQ
I am using a group query because I want to get distinct combinations of the
group like A, B, C. And I am supposed to do that on a dataset of around 10
million. Doing a group takes a lot of time.

Has anybody faced this? Or can help with this?

Thanks & Regards,
Kunal Mittal

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongodb-user@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user+unsubscribe@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Stanislav Pogrebnyak at Aug 15, 2013 at 2:41 pm
    What version of mongodb do you have? If thats 2.2+ than you should consider
    to you aggregation framework for that. Thats just a way or direction you
    probably should follow. And indexes needed to work with such a big data set.

    четверг, 15 августа 2013 г., 12:20:10 UTC+2 пользователь kunal mittal
    написал:
    I am using a group query because I want to get distinct combinations of
    the group like A, B, C. And I am supposed to do that on a dataset of around
    10 million. Doing a group takes a lot of time.

    Has anybody faced this? Or can help with this?

    Thanks & Regards,
    Kunal Mittal
    --
    --
    You received this message because you are subscribed to the Google
    Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com
    To unsubscribe from this group, send email to
    mongodb-user+unsubscribe@googlegroups.com
    See also the IRC channel -- freenode.net#mongodb

    ---
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Gary R at Aug 15, 2013 at 3:15 pm
    Would you post a simplified version of your document schema?

    Gary R.
    On Thursday, August 15, 2013 4:20:10 AM UTC-6, kunal mittal wrote:

    I am using a group query because I want to get distinct combinations of
    the group like A, B, C. And I am supposed to do that on a dataset of around
    10 million. Doing a group takes a lot of time.
    --
    --
    You received this message because you are subscribed to the Google
    Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com
    To unsubscribe from this group, send email to
    mongodb-user+unsubscribe@googlegroups.com
    See also the IRC channel -- freenode.net#mongodb

    ---
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kunal mittal at Aug 15, 2013 at 4:56 pm
    Thanks for the responses guys...

    @Stanislav Pogrebnyak : Yes I am using 2.2+ only. But I have tried to find
    a lot, is there a way where aggregation can be used for group having
    distinct values?

    @Gary : Currently all the data is like the same as how it is stored in
    mySQL. So its all linear.

    On Thu, Aug 15, 2013 at 8:45 PM, Gary R wrote:

    Would you post a simplified version of your document schema?

    Gary R.

    On Thursday, August 15, 2013 4:20:10 AM UTC-6, kunal mittal wrote:

    I am using a group query because I want to get distinct combinations of
    the group like A, B, C. And I am supposed to do that on a dataset of around
    10 million. Doing a group takes a lot of time.
    --
    --
    You received this message because you are subscribed to the Google
    Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com
    To unsubscribe from this group, send email to
    mongodb-user+unsubscribe@googlegroups.com
    See also the IRC channel -- freenode.net#mongodb

    ---
    You received this message because you are subscribed to the Google Groups
    "mongodb-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    With Regards,
    Kunal

    --
    --
    You received this message because you are subscribed to the Google
    Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com
    To unsubscribe from this group, send email to
    mongodb-user+unsubscribe@googlegroups.com
    See also the IRC channel -- freenode.net#mongodb

    ---
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jeff Lee at Aug 15, 2013 at 7:05 pm
    If you want group distinct you can use aggregation:

    e.g

    db.poodle.save({ user_id:1, group_id:1, theval: 1 })
    db.poodle.save({ user_id:1, group_id:2, theval: 2 })
    db.poodle.save({ user_id:1, group_id:3, theval: 3 })
    db.poodle.save({ user_id:2, group_id:1, theval: 4 })
    db.poodle.save({ user_id:2, group_id:1, theval: 5 })
    db.poodle.save({ user_id:2, group_id:2, theval: 6 })

    db.poodle.aggregate(
    { $group: { _id: { user_id: "$user_id", group_id: "$group_id"},
    total:{$sum:"$theval"} }}
    );

    But as stanislav points out, trying to do this over your whole set is going
    to be slow ( and may run into aggregation limits ). If you have a query
    predicate that includes your columns you should index them to restrict the
    items you have to iterate over.

    On Thu, Aug 15, 2013 at 9:56 AM, kunal mittal wrote:

    @Stanislav Pogrebnyak : Yes I am using 2.2+ only. But I have tried to find
    a lot, is there a way where aggregation can be used for group having
    distinct values?
    --
    --
    You received this message because you are subscribed to the Google
    Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com
    To unsubscribe from this group, send email to
    mongodb-user+unsubscribe@googlegroups.com
    See also the IRC channel -- freenode.net#mongodb

    ---
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Gary R at Aug 16, 2013 at 5:20 pm
    I am sorry, Kunal. I don't understand your schema. Are A,B,C... keys or
    values or elements of an array? I can help with aggregation, but I need to
    see a simplified document first.

    Gary R.
    On Thursday, August 15, 2013 10:56:24 AM UTC-6, kunal mittal wrote:

    Thanks for the responses guys...

    @Stanislav Pogrebnyak : Yes I am using 2.2+ only. But I have tried to find
    a lot, is there a way where aggregation can be used for group having
    distinct values?

    @Gary : Currently all the data is like the same as how it is stored in
    mySQL. So its all linear.
    --
    --
    You received this message because you are subscribed to the Google
    Groups "mongodb-user" group.
    To post to this group, send email to mongodb-user@googlegroups.com
    To unsubscribe from this group, send email to
    mongodb-user+unsubscribe@googlegroups.com
    See also the IRC channel -- freenode.net#mongodb

    ---
    You received this message because you are subscribed to the Google Groups "mongodb-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmongodb-user @
categoriesmongodb
postedAug 15, '13 at 2:04p
activeAug 16, '13 at 5:20p
posts6
users4
websitemongodb.org
irc#mongodb

People

Translate

site design / logo © 2022 Grokbase