Having read the articles on the firebase blog regarding normalisation and
the use of indexes I have come up with a plan for managing a firebase
location of tens of thousands of records that I need to filter by a number
of different properties. Loading the entire dataset and managing the
filtering client-side is not a realistic option as it will grow to 10's of
megabytes over time.

The location in question stores a list of bookings that I need to be able
to retrieve by status, client, ship & invoicing status.

I have built a node.js server process that monitors the location for
child_added & child_changed events and updates the indexes accordingly. All
simple, and very well explained by the firebase team.

The app will be displaying the filtered data in a grid using the ng-repeat
directive.

Now... The question I have is how best to consume the index via an
angularFire factory.

My initial thought was to have a complete read-only copy of the record
stored at each index location, but this seems inefficient as I am
potentially keeping many copies of the same record in memory if a user has
requested data by status and then by client and then by ship.

My second thought was to setup a $watch on the index and each time a record
is added/deleted, retrieve/remove the underlying record into an array of
firebase $asObject. again my understanding is that the firebase javascript
sdk will efficiently manage multiple references to the same object if it
appears in multiple indexes. My question is, how efficient is the
$firebaseObject and if I have several hundred in memory will I kill my app?

I am going to test this over the coming days, but before I head down a path
that will have consequences, would appreciate any thoughts from people that
may have had to manage a similar use case?

thanks!

--
You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
To post to this group, send email to firebase-angular@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Michael Wulf at Sep 11, 2014 at 8:17 pm
    Hi Phil,

    You should check out Firebase.util
    <http://firebase.github.io/firebase-util/>, which should provide what you
    need to simply maintain an index. For example:

    var fb = new Firebase(...);
    var indexRef = fb.child('index_path').limit(100);
    var dataRef = fb.child('data_path');
    var indexedList = $firebase( Firebase.util.intersection(indexRef, dataRef)
    ).$asArray();

    Your best mitigation for performance will be to limit the number of records
    retrieved at a time and the number that are rendered to the DOM (your
    biggest opponent here). I wouldn't bother with trying to manage the memory
    usage or storing indexed data locally before it is requested--let Firebase
    and AngularFire cover this until you see a need to micromanage.

    We're working on some great enhancements to Firebase.util this and next
    week. Additionally, there are a lot of enhancements to the API's query
    tools on the horizon which will directly address your use case. Keep an eye
    on the mailing list and twitter feeds for announcements over the next few
    months.

    Cheers,
    Kato
    On Thu, Sep 11, 2014 at 2:34 AM, Phil Hodey wrote:


    Having read the articles on the firebase blog regarding normalisation and
    the use of indexes I have come up with a plan for managing a firebase
    location of tens of thousands of records that I need to filter by a number
    of different properties. Loading the entire dataset and managing the
    filtering client-side is not a realistic option as it will grow to 10's of
    megabytes over time.

    The location in question stores a list of bookings that I need to be able
    to retrieve by status, client, ship & invoicing status.

    I have built a node.js server process that monitors the location for
    child_added & child_changed events and updates the indexes accordingly. All
    simple, and very well explained by the firebase team.

    The app will be displaying the filtered data in a grid using the ng-repeat
    directive.

    Now... The question I have is how best to consume the index via an
    angularFire factory.

    My initial thought was to have a complete read-only copy of the record
    stored at each index location, but this seems inefficient as I am
    potentially keeping many copies of the same record in memory if a user has
    requested data by status and then by client and then by ship.

    My second thought was to setup a $watch on the index and each time a
    record is added/deleted, retrieve/remove the underlying record into an
    array of firebase $asObject. again my understanding is that the firebase
    javascript sdk will efficiently manage multiple references to the same
    object if it appears in multiple indexes. My question is, how efficient is
    the $firebaseObject and if I have several hundred in memory will I kill my
    app?

    I am going to test this over the coming days, but before I head down a
    path that will have consequences, would appreciate any thoughts from people
    that may have had to manage a similar use case?

    thanks!

    --
    You received this message because you are subscribed to the Google Groups
    "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/CAFHX4%3DotBOVk6%2BLYq9n9T%3D%3DSj9Pa90ZH4%2BAQBww0QRhQn4m2UA%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Phil Hodey at Sep 12, 2014 at 5:51 am
    thanks Kato, this is really helpful, i'd seen references to firebase.util
    but never researched its usefulness.


    On Thursday, 11 September 2014 21:17:23 UTC+1, Michael Kato Wulf wrote:

    Hi Phil,

    You should check out Firebase.util
    <http://firebase.github.io/firebase-util/>, which should provide what you
    need to simply maintain an index. For example:

    var fb = new Firebase(...);
    var indexRef = fb.child('index_path').limit(100);
    var dataRef = fb.child('data_path');
    var indexedList = $firebase( Firebase.util.intersection(indexRef, dataRef)
    ).$asArray();

    Your best mitigation for performance will be to limit the number of
    records retrieved at a time and the number that are rendered to the DOM
    (your biggest opponent here). I wouldn't bother with trying to manage the
    memory usage or storing indexed data locally before it is requested--let
    Firebase and AngularFire cover this until you see a need to micromanage.

    We're working on some great enhancements to Firebase.util this and next
    week. Additionally, there are a lot of enhancements to the API's query
    tools on the horizon which will directly address your use case. Keep an eye
    on the mailing list and twitter feeds for announcements over the next few
    months.

    Cheers,
    Kato

    On Thu, Sep 11, 2014 at 2:34 AM, Phil Hodey <phil....@hcstech.co.uk
    <javascript:>> wrote:
    Having read the articles on the firebase blog regarding normalisation and
    the use of indexes I have come up with a plan for managing a firebase
    location of tens of thousands of records that I need to filter by a number
    of different properties. Loading the entire dataset and managing the
    filtering client-side is not a realistic option as it will grow to 10's of
    megabytes over time.

    The location in question stores a list of bookings that I need to be able
    to retrieve by status, client, ship & invoicing status.

    I have built a node.js server process that monitors the location for
    child_added & child_changed events and updates the indexes accordingly. All
    simple, and very well explained by the firebase team.

    The app will be displaying the filtered data in a grid using the
    ng-repeat directive.

    Now... The question I have is how best to consume the index via an
    angularFire factory.

    My initial thought was to have a complete read-only copy of the record
    stored at each index location, but this seems inefficient as I am
    potentially keeping many copies of the same record in memory if a user has
    requested data by status and then by client and then by ship.

    My second thought was to setup a $watch on the index and each time a
    record is added/deleted, retrieve/remove the underlying record into an
    array of firebase $asObject. again my understanding is that the firebase
    javascript sdk will efficiently manage multiple references to the same
    object if it appears in multiple indexes. My question is, how efficient is
    the $firebaseObject and if I have several hundred in memory will I kill my
    app?

    I am going to test this over the coming days, but before I head down a
    path that will have consequences, would appreciate any thoughts from people
    that may have had to manage a similar use case?

    thanks!

    --
    You received this message because you are subscribed to the Google Groups
    "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to firebase-angul...@googlegroups.com <javascript:>.
    To post to this group, send email to firebase...@googlegroups.com
    <javascript:>.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/3658e708-f2f9-4fcf-9de2-f108fa35079e%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Phil Hodey at Sep 12, 2014 at 7:40 am
    Kato,

    I've changed my factory to now use the intersection function to return an
    indexed view. However the performance is quite slow relative to retrieving
    just the index itself. Obviously I'd expect it to be slower but not by
    quite as much as it is. Does the intersection function still return the
    *entire* dataset and then run the intersection filter?

    If so this sort of defeats what I am trying to achieve as the 'active'
    filter for example only usually consists of 100 odd records but the
    underlying data may have several thousand.



    On Friday, 12 September 2014 06:51:56 UTC+1, Phil Hodey wrote:

    thanks Kato, this is really helpful, i'd seen references to firebase.util
    but never researched its usefulness.


    On Thursday, 11 September 2014 21:17:23 UTC+1, Michael Kato Wulf wrote:

    Hi Phil,

    You should check out Firebase.util
    <http://firebase.github.io/firebase-util/>, which should provide what
    you need to simply maintain an index. For example:

    var fb = new Firebase(...);
    var indexRef = fb.child('index_path').limit(100);
    var dataRef = fb.child('data_path');
    var indexedList = $firebase( Firebase.util.intersection(indexRef,
    dataRef) ).$asArray();

    Your best mitigation for performance will be to limit the number of
    records retrieved at a time and the number that are rendered to the DOM
    (your biggest opponent here). I wouldn't bother with trying to manage the
    memory usage or storing indexed data locally before it is requested--let
    Firebase and AngularFire cover this until you see a need to micromanage.

    We're working on some great enhancements to Firebase.util this and next
    week. Additionally, there are a lot of enhancements to the API's query
    tools on the horizon which will directly address your use case. Keep an eye
    on the mailing list and twitter feeds for announcements over the next few
    months.

    Cheers,
    Kato

    On Thu, Sep 11, 2014 at 2:34 AM, Phil Hodey <phil....@hcstech.co.uk>
    wrote:
    Having read the articles on the firebase blog regarding normalisation
    and the use of indexes I have come up with a plan for managing a firebase
    location of tens of thousands of records that I need to filter by a number
    of different properties. Loading the entire dataset and managing the
    filtering client-side is not a realistic option as it will grow to 10's of
    megabytes over time.

    The location in question stores a list of bookings that I need to be
    able to retrieve by status, client, ship & invoicing status.

    I have built a node.js server process that monitors the location for
    child_added & child_changed events and updates the indexes accordingly. All
    simple, and very well explained by the firebase team.

    The app will be displaying the filtered data in a grid using the
    ng-repeat directive.

    Now... The question I have is how best to consume the index via an
    angularFire factory.

    My initial thought was to have a complete read-only copy of the record
    stored at each index location, but this seems inefficient as I am
    potentially keeping many copies of the same record in memory if a user has
    requested data by status and then by client and then by ship.

    My second thought was to setup a $watch on the index and each time a
    record is added/deleted, retrieve/remove the underlying record into an
    array of firebase $asObject. again my understanding is that the firebase
    javascript sdk will efficiently manage multiple references to the same
    object if it appears in multiple indexes. My question is, how efficient is
    the $firebaseObject and if I have several hundred in memory will I kill my
    app?

    I am going to test this over the coming days, but before I head down a
    path that will have consequences, would appreciate any thoughts from people
    that may have had to manage a similar use case?

    thanks!

    --
    You received this message because you are subscribed to the Google
    Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to firebase-angul...@googlegroups.com.
    To post to this group, send email to firebase...@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michael Wulf at Sep 12, 2014 at 7:57 am
    The intersection should be loading only records that exist in the index. If
    you're able to show otherwise, submit a bug for the repo and I'll fix it
    up. Cheers.
    On Fri, Sep 12, 2014 at 12:40 AM, Phil Hodey wrote:

    Kato,

    I've changed my factory to now use the intersection function to return an
    indexed view. However the performance is quite slow relative to retrieving
    just the index itself. Obviously I'd expect it to be slower but not by
    quite as much as it is. Does the intersection function still return the
    *entire* dataset and then run the intersection filter?

    If so this sort of defeats what I am trying to achieve as the 'active'
    filter for example only usually consists of 100 odd records but the
    underlying data may have several thousand.



    On Friday, 12 September 2014 06:51:56 UTC+1, Phil Hodey wrote:

    thanks Kato, this is really helpful, i'd seen references to firebase.util
    but never researched its usefulness.


    On Thursday, 11 September 2014 21:17:23 UTC+1, Michael Kato Wulf wrote:

    Hi Phil,

    You should check out Firebase.util
    <http://firebase.github.io/firebase-util/>, which should provide what
    you need to simply maintain an index. For example:

    var fb = new Firebase(...);
    var indexRef = fb.child('index_path').limit(100);
    var dataRef = fb.child('data_path');
    var indexedList = $firebase( Firebase.util.intersection(indexRef,
    dataRef) ).$asArray();

    Your best mitigation for performance will be to limit the number of
    records retrieved at a time and the number that are rendered to the DOM
    (your biggest opponent here). I wouldn't bother with trying to manage the
    memory usage or storing indexed data locally before it is requested--let
    Firebase and AngularFire cover this until you see a need to micromanage.

    We're working on some great enhancements to Firebase.util this and next
    week. Additionally, there are a lot of enhancements to the API's query
    tools on the horizon which will directly address your use case. Keep an eye
    on the mailing list and twitter feeds for announcements over the next few
    months.

    Cheers,
    Kato

    On Thu, Sep 11, 2014 at 2:34 AM, Phil Hodey <phil....@hcstech.co.uk>
    wrote:
    Having read the articles on the firebase blog regarding normalisation
    and the use of indexes I have come up with a plan for managing a firebase
    location of tens of thousands of records that I need to filter by a number
    of different properties. Loading the entire dataset and managing the
    filtering client-side is not a realistic option as it will grow to 10's of
    megabytes over time.

    The location in question stores a list of bookings that I need to be
    able to retrieve by status, client, ship & invoicing status.

    I have built a node.js server process that monitors the location for
    child_added & child_changed events and updates the indexes accordingly. All
    simple, and very well explained by the firebase team.

    The app will be displaying the filtered data in a grid using the
    ng-repeat directive.

    Now... The question I have is how best to consume the index via an
    angularFire factory.

    My initial thought was to have a complete read-only copy of the record
    stored at each index location, but this seems inefficient as I am
    potentially keeping many copies of the same record in memory if a user has
    requested data by status and then by client and then by ship.

    My second thought was to setup a $watch on the index and each time a
    record is added/deleted, retrieve/remove the underlying record into an
    array of firebase $asObject. again my understanding is that the firebase
    javascript sdk will efficiently manage multiple references to the same
    object if it appears in multiple indexes. My question is, how efficient is
    the $firebaseObject and if I have several hundred in memory will I kill my
    app?

    I am going to test this over the coming days, but before I head down a
    path that will have consequences, would appreciate any thoughts from people
    that may have had to manage a similar use case?

    thanks!

    --
    You received this message because you are subscribed to the Google
    Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to firebase-angul...@googlegroups.com.
    To post to this group, send email to firebase...@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/
    msgid/firebase-angular/f413297b-b26d-464d-bb8f-
    ad1c8ca67fac%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/CAFHX4%3DoqEEzRmx8v7hBdgZL-ufwAtyL3XB5HwEAn%2BgmU45Cd9A%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Phil Hodey at Sep 12, 2014 at 7:58 am
    ok thx - will take a look at it now and revert
    On Friday, 12 September 2014 08:57:15 UTC+1, Michael Kato Wulf wrote:

    The intersection should be loading only records that exist in the index.
    If you're able to show otherwise, submit a bug for the repo and I'll fix it
    up. Cheers.

    On Fri, Sep 12, 2014 at 12:40 AM, Phil Hodey <phil....@hcstech.co.uk
    <javascript:>> wrote:
    Kato,

    I've changed my factory to now use the intersection function to return an
    indexed view. However the performance is quite slow relative to retrieving
    just the index itself. Obviously I'd expect it to be slower but not by
    quite as much as it is. Does the intersection function still return the
    *entire* dataset and then run the intersection filter?

    If so this sort of defeats what I am trying to achieve as the 'active'
    filter for example only usually consists of 100 odd records but the
    underlying data may have several thousand.



    On Friday, 12 September 2014 06:51:56 UTC+1, Phil Hodey wrote:

    thanks Kato, this is really helpful, i'd seen references to
    firebase.util but never researched its usefulness.


    On Thursday, 11 September 2014 21:17:23 UTC+1, Michael Kato Wulf wrote:

    Hi Phil,

    You should check out Firebase.util
    <http://firebase.github.io/firebase-util/>, which should provide what
    you need to simply maintain an index. For example:

    var fb = new Firebase(...);
    var indexRef = fb.child('index_path').limit(100);
    var dataRef = fb.child('data_path');
    var indexedList = $firebase( Firebase.util.intersection(indexRef,
    dataRef) ).$asArray();

    Your best mitigation for performance will be to limit the number of
    records retrieved at a time and the number that are rendered to the DOM
    (your biggest opponent here). I wouldn't bother with trying to manage the
    memory usage or storing indexed data locally before it is requested--let
    Firebase and AngularFire cover this until you see a need to micromanage.

    We're working on some great enhancements to Firebase.util this and next
    week. Additionally, there are a lot of enhancements to the API's query
    tools on the horizon which will directly address your use case. Keep an eye
    on the mailing list and twitter feeds for announcements over the next few
    months.

    Cheers,
    Kato

    On Thu, Sep 11, 2014 at 2:34 AM, Phil Hodey <phil....@hcstech.co.uk>
    wrote:
    Having read the articles on the firebase blog regarding normalisation
    and the use of indexes I have come up with a plan for managing a firebase
    location of tens of thousands of records that I need to filter by a number
    of different properties. Loading the entire dataset and managing the
    filtering client-side is not a realistic option as it will grow to 10's of
    megabytes over time.

    The location in question stores a list of bookings that I need to be
    able to retrieve by status, client, ship & invoicing status.

    I have built a node.js server process that monitors the location for
    child_added & child_changed events and updates the indexes accordingly. All
    simple, and very well explained by the firebase team.

    The app will be displaying the filtered data in a grid using the
    ng-repeat directive.

    Now... The question I have is how best to consume the index via an
    angularFire factory.

    My initial thought was to have a complete read-only copy of the record
    stored at each index location, but this seems inefficient as I am
    potentially keeping many copies of the same record in memory if a user has
    requested data by status and then by client and then by ship.

    My second thought was to setup a $watch on the index and each time a
    record is added/deleted, retrieve/remove the underlying record into an
    array of firebase $asObject. again my understanding is that the firebase
    javascript sdk will efficiently manage multiple references to the same
    object if it appears in multiple indexes. My question is, how efficient is
    the $firebaseObject and if I have several hundred in memory will I kill my
    app?

    I am going to test this over the coming days, but before I head down a
    path that will have consequences, would appreciate any thoughts from people
    that may have had to manage a similar use case?

    thanks!

    --
    You received this message because you are subscribed to the Google
    Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to firebase-angul...@googlegroups.com.
    To post to this group, send email to firebase...@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/
    msgid/firebase-angular/f413297b-b26d-464d-bb8f-
    ad1c8ca67fac%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to firebase-angul...@googlegroups.com <javascript:>.
    To post to this group, send email to firebase...@googlegroups.com
    <javascript:>.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/9151c043-da78-49f8-95d5-5d1f4a5225e3%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Phil Hodey at Sep 12, 2014 at 6:49 pm
    Kato, i've done a few timing tests...

    Scenario is a table of records with circa 1300 children. an index of
    references with 35 entries. The tests I ran involved getting the entire
    table and timing it until the $loaded() promise returned. I then ran the
    same test with the intersect reference. I then ran the same two tests in
    reverse order.

    The timings I received were...

    GetAll...
    2737ms
    getActiveIntersect...
    3843ms


    getActiveIntersect...
    4656ms
    GetAll...
    483ms

    This is not conclusive and I have not gone as far as go through the code,
    but what seems to be the case is a) the intersect is not that efficient as
    even when all records have been loaded from the server it takes longer to
    generate 35 intersects than get 1300 records., and b) it is very possible
    that it is in fact getting all records during the intersect.

    Any thoughts? (happy to share the code with you if you want to run the
    tests yourself)





    On Friday, 12 September 2014 08:57:15 UTC+1, Michael Kato Wulf wrote:

    The intersection should be loading only records that exist in the index.
    If you're able to show otherwise, submit a bug for the repo and I'll fix it
    up. Cheers.

    On Fri, Sep 12, 2014 at 12:40 AM, Phil Hodey <phil....@hcstech.co.uk
    <javascript:>> wrote:
    Kato,

    I've changed my factory to now use the intersection function to return an
    indexed view. However the performance is quite slow relative to retrieving
    just the index itself. Obviously I'd expect it to be slower but not by
    quite as much as it is. Does the intersection function still return the
    *entire* dataset and then run the intersection filter?

    If so this sort of defeats what I am trying to achieve as the 'active'
    filter for example only usually consists of 100 odd records but the
    underlying data may have several thousand.



    On Friday, 12 September 2014 06:51:56 UTC+1, Phil Hodey wrote:

    thanks Kato, this is really helpful, i'd seen references to
    firebase.util but never researched its usefulness.


    On Thursday, 11 September 2014 21:17:23 UTC+1, Michael Kato Wulf wrote:

    Hi Phil,

    You should check out Firebase.util
    <http://firebase.github.io/firebase-util/>, which should provide what
    you need to simply maintain an index. For example:

    var fb = new Firebase(...);
    var indexRef = fb.child('index_path').limit(100);
    var dataRef = fb.child('data_path');
    var indexedList = $firebase( Firebase.util.intersection(indexRef,
    dataRef) ).$asArray();

    Your best mitigation for performance will be to limit the number of
    records retrieved at a time and the number that are rendered to the DOM
    (your biggest opponent here). I wouldn't bother with trying to manage the
    memory usage or storing indexed data locally before it is requested--let
    Firebase and AngularFire cover this until you see a need to micromanage.

    We're working on some great enhancements to Firebase.util this and next
    week. Additionally, there are a lot of enhancements to the API's query
    tools on the horizon which will directly address your use case. Keep an eye
    on the mailing list and twitter feeds for announcements over the next few
    months.

    Cheers,
    Kato

    On Thu, Sep 11, 2014 at 2:34 AM, Phil Hodey <phil....@hcstech.co.uk>
    wrote:
    Having read the articles on the firebase blog regarding normalisation
    and the use of indexes I have come up with a plan for managing a firebase
    location of tens of thousands of records that I need to filter by a number
    of different properties. Loading the entire dataset and managing the
    filtering client-side is not a realistic option as it will grow to 10's of
    megabytes over time.

    The location in question stores a list of bookings that I need to be
    able to retrieve by status, client, ship & invoicing status.

    I have built a node.js server process that monitors the location for
    child_added & child_changed events and updates the indexes accordingly. All
    simple, and very well explained by the firebase team.

    The app will be displaying the filtered data in a grid using the
    ng-repeat directive.

    Now... The question I have is how best to consume the index via an
    angularFire factory.

    My initial thought was to have a complete read-only copy of the record
    stored at each index location, but this seems inefficient as I am
    potentially keeping many copies of the same record in memory if a user has
    requested data by status and then by client and then by ship.

    My second thought was to setup a $watch on the index and each time a
    record is added/deleted, retrieve/remove the underlying record into an
    array of firebase $asObject. again my understanding is that the firebase
    javascript sdk will efficiently manage multiple references to the same
    object if it appears in multiple indexes. My question is, how efficient is
    the $firebaseObject and if I have several hundred in memory will I kill my
    app?

    I am going to test this over the coming days, but before I head down a
    path that will have consequences, would appreciate any thoughts from people
    that may have had to manage a similar use case?

    thanks!

    --
    You received this message because you are subscribed to the Google
    Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to firebase-angul...@googlegroups.com.
    To post to this group, send email to firebase...@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/
    msgid/firebase-angular/f413297b-b26d-464d-bb8f-
    ad1c8ca67fac%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to firebase-angul...@googlegroups.com <javascript:>.
    To post to this group, send email to firebase...@googlegroups.com
    <javascript:>.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/b23946e5-6c5d-4080-a3b0-34701dace857%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michael Wulf at Sep 12, 2014 at 6:55 pm
    Hi Phil,

    Please move this over to the GitHub repo as an issue. Share code there and
    I'll have a look. Also keep in mind that I'm working on the refresh for
    Firebase.util as we speak, so the new version may be out before the fix : )

    Cheers,
    On Fri, Sep 12, 2014 at 11:49 AM, Phil Hodey wrote:

    Kato, i've done a few timing tests...

    Scenario is a table of records with circa 1300 children. an index of
    references with 35 entries. The tests I ran involved getting the entire
    table and timing it until the $loaded() promise returned. I then ran the
    same test with the intersect reference. I then ran the same two tests in
    reverse order.

    The timings I received were...

    GetAll...
    2737ms
    getActiveIntersect...
    3843ms


    getActiveIntersect...
    4656ms
    GetAll...
    483ms

    This is not conclusive and I have not gone as far as go through the code,
    but what seems to be the case is a) the intersect is not that efficient as
    even when all records have been loaded from the server it takes longer to
    generate 35 intersects than get 1300 records., and b) it is very possible
    that it is in fact getting all records during the intersect.

    Any thoughts? (happy to share the code with you if you want to run the
    tests yourself)





    On Friday, 12 September 2014 08:57:15 UTC+1, Michael Kato Wulf wrote:

    The intersection should be loading only records that exist in the index.
    If you're able to show otherwise, submit a bug for the repo and I'll fix it
    up. Cheers.

    On Fri, Sep 12, 2014 at 12:40 AM, Phil Hodey <phil....@hcstech.co.uk>
    wrote:
    Kato,

    I've changed my factory to now use the intersection function to return
    an indexed view. However the performance is quite slow relative to
    retrieving just the index itself. Obviously I'd expect it to be slower but
    not by quite as much as it is. Does the intersection function still return
    the *entire* dataset and then run the intersection filter?

    If so this sort of defeats what I am trying to achieve as the 'active'
    filter for example only usually consists of 100 odd records but the
    underlying data may have several thousand.



    On Friday, 12 September 2014 06:51:56 UTC+1, Phil Hodey wrote:

    thanks Kato, this is really helpful, i'd seen references to
    firebase.util but never researched its usefulness.


    On Thursday, 11 September 2014 21:17:23 UTC+1, Michael Kato Wulf wrote:

    Hi Phil,

    You should check out Firebase.util
    <http://firebase.github.io/firebase-util/>, which should provide what
    you need to simply maintain an index. For example:

    var fb = new Firebase(...);
    var indexRef = fb.child('index_path').limit(100);
    var dataRef = fb.child('data_path');
    var indexedList = $firebase( Firebase.util.intersection(indexRef,
    dataRef) ).$asArray();

    Your best mitigation for performance will be to limit the number of
    records retrieved at a time and the number that are rendered to the DOM
    (your biggest opponent here). I wouldn't bother with trying to manage the
    memory usage or storing indexed data locally before it is requested--let
    Firebase and AngularFire cover this until you see a need to micromanage.

    We're working on some great enhancements to Firebase.util this and
    next week. Additionally, there are a lot of enhancements to the API's query
    tools on the horizon which will directly address your use case. Keep an eye
    on the mailing list and twitter feeds for announcements over the next few
    months.

    Cheers,
    Kato

    On Thu, Sep 11, 2014 at 2:34 AM, Phil Hodey <phil....@hcstech.co.uk>
    wrote:
    Having read the articles on the firebase blog regarding normalisation
    and the use of indexes I have come up with a plan for managing a firebase
    location of tens of thousands of records that I need to filter by a number
    of different properties. Loading the entire dataset and managing the
    filtering client-side is not a realistic option as it will grow to 10's of
    megabytes over time.

    The location in question stores a list of bookings that I need to be
    able to retrieve by status, client, ship & invoicing status.

    I have built a node.js server process that monitors the location for
    child_added & child_changed events and updates the indexes accordingly. All
    simple, and very well explained by the firebase team.

    The app will be displaying the filtered data in a grid using the
    ng-repeat directive.

    Now... The question I have is how best to consume the index via an
    angularFire factory.

    My initial thought was to have a complete read-only copy of the
    record stored at each index location, but this seems inefficient as I am
    potentially keeping many copies of the same record in memory if a user has
    requested data by status and then by client and then by ship.

    My second thought was to setup a $watch on the index and each time a
    record is added/deleted, retrieve/remove the underlying record into an
    array of firebase $asObject. again my understanding is that the firebase
    javascript sdk will efficiently manage multiple references to the same
    object if it appears in multiple indexes. My question is, how efficient is
    the $firebaseObject and if I have several hundred in memory will I kill my
    app?

    I am going to test this over the coming days, but before I head down
    a path that will have consequences, would appreciate any thoughts from
    people that may have had to manage a similar use case?

    thanks!

    --
    You received this message because you are subscribed to the Google
    Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to firebase-angul...@googlegroups.com.
    To post to this group, send email to firebase...@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/
    msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%
    40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to firebase-angul...@googlegroups.com.
    To post to this group, send email to firebase...@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/
    msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-
    95580a2e1994%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/b23946e5-6c5d-4080-a3b0-34701dace857%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/b23946e5-6c5d-4080-a3b0-34701dace857%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/CAFHX4%3Dod1A%3Dkfkz026gzK4g9uTG3v9bhxO3Sq1u1n%3DCVsN8arw%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Phil Hodey at Sep 12, 2014 at 7:13 pm
    good point, i'm going to leave it for now and let you focus on the new
    version!
    On Friday, 12 September 2014 19:55:36 UTC+1, Michael Kato Wulf wrote:

    Hi Phil,

    Please move this over to the GitHub repo as an issue. Share code there and
    I'll have a look. Also keep in mind that I'm working on the refresh for
    Firebase.util as we speak, so the new version may be out before the fix : )

    Cheers,

    On Fri, Sep 12, 2014 at 11:49 AM, Phil Hodey <phil....@hcstech.co.uk
    <javascript:>> wrote:
    Kato, i've done a few timing tests...

    Scenario is a table of records with circa 1300 children. an index of
    references with 35 entries. The tests I ran involved getting the entire
    table and timing it until the $loaded() promise returned. I then ran the
    same test with the intersect reference. I then ran the same two tests in
    reverse order.

    The timings I received were...

    GetAll...
    2737ms
    getActiveIntersect...
    3843ms


    getActiveIntersect...
    4656ms
    GetAll...
    483ms

    This is not conclusive and I have not gone as far as go through the code,
    but what seems to be the case is a) the intersect is not that efficient as
    even when all records have been loaded from the server it takes longer to
    generate 35 intersects than get 1300 records., and b) it is very possible
    that it is in fact getting all records during the intersect.

    Any thoughts? (happy to share the code with you if you want to run the
    tests yourself)





    On Friday, 12 September 2014 08:57:15 UTC+1, Michael Kato Wulf wrote:

    The intersection should be loading only records that exist in the index.
    If you're able to show otherwise, submit a bug for the repo and I'll fix it
    up. Cheers.

    On Fri, Sep 12, 2014 at 12:40 AM, Phil Hodey <phil....@hcstech.co.uk>
    wrote:
    Kato,

    I've changed my factory to now use the intersection function to return
    an indexed view. However the performance is quite slow relative to
    retrieving just the index itself. Obviously I'd expect it to be slower but
    not by quite as much as it is. Does the intersection function still return
    the *entire* dataset and then run the intersection filter?

    If so this sort of defeats what I am trying to achieve as the 'active'
    filter for example only usually consists of 100 odd records but the
    underlying data may have several thousand.



    On Friday, 12 September 2014 06:51:56 UTC+1, Phil Hodey wrote:

    thanks Kato, this is really helpful, i'd seen references to
    firebase.util but never researched its usefulness.


    On Thursday, 11 September 2014 21:17:23 UTC+1, Michael Kato Wulf wrote:

    Hi Phil,

    You should check out Firebase.util
    <http://firebase.github.io/firebase-util/>, which should provide
    what you need to simply maintain an index. For example:

    var fb = new Firebase(...);
    var indexRef = fb.child('index_path').limit(100);
    var dataRef = fb.child('data_path');
    var indexedList = $firebase( Firebase.util.intersection(indexRef,
    dataRef) ).$asArray();

    Your best mitigation for performance will be to limit the number of
    records retrieved at a time and the number that are rendered to the DOM
    (your biggest opponent here). I wouldn't bother with trying to manage the
    memory usage or storing indexed data locally before it is requested--let
    Firebase and AngularFire cover this until you see a need to micromanage.

    We're working on some great enhancements to Firebase.util this and
    next week. Additionally, there are a lot of enhancements to the API's query
    tools on the horizon which will directly address your use case. Keep an eye
    on the mailing list and twitter feeds for announcements over the next few
    months.

    Cheers,
    Kato

    On Thu, Sep 11, 2014 at 2:34 AM, Phil Hodey <phil....@hcstech.co.uk>
    wrote:
    Having read the articles on the firebase blog regarding
    normalisation and the use of indexes I have come up with a plan for
    managing a firebase location of tens of thousands of records that I need to
    filter by a number of different properties. Loading the entire dataset and
    managing the filtering client-side is not a realistic option as it will
    grow to 10's of megabytes over time.

    The location in question stores a list of bookings that I need to be
    able to retrieve by status, client, ship & invoicing status.

    I have built a node.js server process that monitors the location for
    child_added & child_changed events and updates the indexes accordingly. All
    simple, and very well explained by the firebase team.

    The app will be displaying the filtered data in a grid using the
    ng-repeat directive.

    Now... The question I have is how best to consume the index via an
    angularFire factory.

    My initial thought was to have a complete read-only copy of the
    record stored at each index location, but this seems inefficient as I am
    potentially keeping many copies of the same record in memory if a user has
    requested data by status and then by client and then by ship.

    My second thought was to setup a $watch on the index and each time a
    record is added/deleted, retrieve/remove the underlying record into an
    array of firebase $asObject. again my understanding is that the firebase
    javascript sdk will efficiently manage multiple references to the same
    object if it appears in multiple indexes. My question is, how efficient is
    the $firebaseObject and if I have several hundred in memory will I kill my
    app?

    I am going to test this over the coming days, but before I head down
    a path that will have consequences, would appreciate any thoughts from
    people that may have had to manage a similar use case?

    thanks!

    --
    You received this message because you are subscribed to the Google
    Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to firebase-angul...@googlegroups.com.
    To post to this group, send email to firebase...@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/f413297b-
    b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/f413297b-b26d-464d-bb8f-ad1c8ca67fac%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to firebase-angul...@googlegroups.com.
    To post to this group, send email to firebase...@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/
    msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-
    95580a2e1994%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/a3968cfb-c1d7-4764-bbff-95580a2e1994%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to firebase-angul...@googlegroups.com <javascript:>.
    To post to this group, send email to firebase...@googlegroups.com
    <javascript:>.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/firebase-angular/b23946e5-6c5d-4080-a3b0-34701dace857%40googlegroups.com
    <https://groups.google.com/d/msgid/firebase-angular/b23946e5-6c5d-4080-a3b0-34701dace857%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Firebase + AngularJS" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to firebase-angular+unsubscribe@googlegroups.com.
    To post to this group, send email to firebase-angular@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-angular/f7968331-6b87-4fc5-bbba-cfe467f73598%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupfirebase-angular @
postedSep 11, '14 at 9:34a
activeSep 12, '14 at 7:13p
posts9
users2

2 users in discussion

Phil Hodey: 6 posts Michael Wulf: 3 posts

People

Translate

site design / logo © 2021 Grokbase