How to separate initial data load from incremental children with Firebase?

Question

I have an application where new children get added to Firebase every 5 seconds or so. I have thousands of children.

On application load, I'd like to process the initial thousands differently from the subsequent children that trickle in every 5 seconds.

You might suggest I use value, process everything, and then use children_added. But I believe if the processing takes too long I have the potential to miss a point.

Is there a way to do this in Firebase that guarantees I don't miss a point?

Solution 1

Since child_added events for the initial, pre-loaded data will fire before the value event fires on the parent, you can use the value event as a sort of "initial data loaded" notification. Here is some code I slightly modified from another similar StackOverflow question.

var initialDataLoaded = false;
var ref = new Firebase('https://<your-Firebase>.firebaseio.com');

ref.on('child_added', function(snapshot) {
  if (initialDataLoaded) {
    var msg = snapshot.val().msg;
    // do something here
  } else {
    // we are ignoring this child since it is pre-existing data
  }
});

ref.once('value', function(snapshot) {
  initialDataLoaded = true;
});

Thankfully, Firebase will smartly cache this data, meaning that creating both a child_added and a value listener will only download the data one time. Adding new Firebase listeners for data which has already crossed the wire is extremely cheap and you should feel comfortable doing things like that regularly.

If you are worried about downloading all that initial data when you don't actually need it, I would follow @FrankvanPuffelen's suggestions in the comments to use a timestamped query. This works really well and is optimized using Firebase queries.

Solution 2

Added a createdAt timestamp in the database and limited the child_added with the current time and it seems working fine to me.

const onChildAdded = database()
  .ref(refURL)
  .limitToLast(constValues.LAST_MESSAGE_ADDED)
  .orderByChild('createdAt')
  .startAt(date.getTime())
  .on('child_added', (snapshot) => {
    parseMessage(snapshot);
  });

database()
  .ref(refURL)
  .limitToLast(constValues.LAST_MESSAGES_LIMIT)
  .once('value', (snapshot) => {
    parseInitialMessages(snapshot);
    setLoading(false);
  });

Comments

[Frank van Puffelen]

Please don't post this as an answer, unless you think it is an answer. Kato gave two links to similar questions. If the answers given there don't work for you, edit you question (there's a handy edit link right under it) to include a minimal code sample that shows what you've done and describe what goes wrong with it.

[Keith Carter]

I've summarized my takeaways from the two links Kato gave and posted it as an answer. Also: one of the solutions linked to has the exact problem I am trying to avoid: missing data for large data sets. I'll post the problem in that thread (stackoverflow.com/questions/19883736/…)

[Keith Carter]

Actually, I'm not allowed to comment because I am below 50 karma. Setting aside the issue of having to download the data twice, the problem is that with large data sets there is a gap in time between when child_added finishes and when value finishes. Since children added are only processed AFTER value finishes, you could miss data.

[cfi]

If the "solutions" are no solutions to you, then don't post them as answers. You are always entitled to update/edit your question with additional info.

[Keith Carter]

My question is adequately stated and no further explanation is needed. I've gone and summarized the solutions others have provided for this very problem and explained why they aren't perfect. If there is a "better" solution, anyone is welcome to provide it and I'll indicate it as correct.

[Keith Carter]

This is an excellent answer. I was unaware of the smart cacheing and haven't seen that mentioned anywhere else. Thank you Jacob. I do have two questions: what happens if that callback of value doesn't occur immediately or if the callback itself takes a long time to execute (due to data processing or something else)? Isn't there a possibility that a child could be added in that time?

[jwngr]

Firebase should guarantee that the value event will fire before any new child_added events fire, so you should be fine there. As for data processing of the callback, that shouldn't matter as long as you are checking the initialDataLoaded variable right away. If you check it later in the callback after a bunch of blocking code, it may have switched to true, which is what I think you are getting at. You should follow the usual JavaScript best practices here as that is not really a Firebase-specific question.

[Nick Pineda]

@jacobawenger I keep reading that it is "by design" that Firebase doesn't have a way built to grab the initial data and then only grab what changed? Is that because by design 3 way data binding is suppose to already manage the real-time syncing? I'm just worried that I'm trying to sync things manually when Firebase is built to declarative handle that.

[Tyler McGinnis]

Bless you Jacob. No idea this was a thing.

[TheEye]

In iOS at least the order of when the handlers are installed is important: if the value handler is installed before the add handler, the add callback is called nonetheless.

[Keith Carter]

Does this not introduce a race condition? What if a child is added during the execution of: // do something here with initial value

[Lucy]

Thank you! This solution is clever and super helpful! My use case is: On start of an HTTP server listening to my Firebase Realtime Database I want to fetch all the existing data at that reference and do something with it. Once I've read all existing data then I want to listen for new data and do something else with new data. This solution helps me do exactly that. I'd be interested to know more about the smart caching, too, because I was unaware of it.

[Jonny]

It looks like something I am trying, but I get duplicate data. I think the reason is that child_added adds ALL data, even from start - that's just the way it seems to work. So we need some check like in stackoverflow.com/a/27995609/129202 so that we don't add duplicate data.