Firestore doesn't retrieve more than 600 items

Question

I am trying to load some data from Firestore in a Flutter app. When I load upto 600 items , it works fine. but even if I try retrieving 700I don't get anything.

Here is the code:

Stream<List<Booking>> listenToWeekEntries()  {
    return FirebaseFirestore.instance
        .collection('bookings')
        .orderBy('entryTimestamp')
        .where('entryTimestamp',isGreaterThanOrEqualTo: getWeekFilter()[0]/1000, isLessThanOrEqualTo: getWeekFilter()[1]/1000)
        .limitToLast(1000).snapshots().map((event) => List.from(event.docs.map((e) => Booking.fromSnapshot(e))));
          //

  }

I only have 1 collection 'bookings' and in it I have about 50000 documents(bookings) (growing at about 120/day). Each booking has about 13 fields. I am trying to create charts representing the counts on different days, months and years. What's the best way to do this?

I have tried looking around for a solution but have had not luck.

I have also tried the following :

late DocumentSnapshot _lastDocument;
List<Booking> entries = List.empty(growable: true);

Constructor(){
    getEntries();
}

void getEntries() async {
    var q = await FirebaseFirestore.instance
        .collection('bookings')
        .orderBy('entryTimestamp')
        .where('entryTimestamp',isGreaterThanOrEqualTo: getWeekFilter()[0]/1000, isLessThanOrEqualTo: getWeekFilter()[1]/1000)
        .limitToLast(50).get();
    entries.addAll(q.docs.map((e) => Booking.fromSnapshot(e)));
    notifyListeners();
    print('${entries.length}');
    if(entries.length < 2000){
      _lastDocument = q.docs[q.docs.length - 1];
      getMoreEntries();
    }
  }

void getMoreEntries() async {
    print('WEEK_DEBUG | Getting more entries after $_lastDocument.');
    var q = await FirebaseFirestore.instance
        .collection('bookings')
        .orderBy('entryTimestamp')
        .where('entryTimestamp',isGreaterThanOrEqualTo: getWeekFilter()[0]/1000, isLessThanOrEqualTo: getWeekFilter()[1]/1000)
        .startAfterDocument(_lastDocument)
        .limitToLast(50).get();

    entries.addAll(q.docs.map((e) => Booking.fromSnapshot(e)));
    notifyListeners();
    _lastDocument = q.docs[q.docs.length - 1];
    if(entries.length < 2000){
      getMoreEntries();
    }
  }

 List getWeekFilter(){
    var now = DateTime.now();
    var firstDay = now.subtract(const Duration(days: 7));
    var lastDay = now.add(const Duration(days: 7));
    var from = DateTime(firstDay.year, firstDay.month, firstDay.day).millisecondsSinceEpoch;
    var to = DateTime(lastDay.year, lastDay.month, lastDay.day).millisecondsSinceEpoch;
    return [from, to];

  }

With this, getMoreEntries() does get called however the total length of entries remains 50.

Any help is greatly appreciated.

Solution

I couldn't tell you why you can't retrieve more than 600 docs per query. Yet I'm sure you can sense why doing such a huge query just to count docs is not a good idea.
I'd advise you to create a simple counter subcollection, of which each doc id could be the day they were counting bookings creation for. You just have to increase the day's counter by one after each booking, each doc summing the daily booking count (and incidentally, the id of each booking created that day if there are only a handful of those, and you'd need to retrieve them). After that you just have to query every day's counter and sum them up in the front-end. You could also start a new subcollection counter every year to cap the number of booking day docs to sum up to 365/6, and put the final sum of previous years count in a field of the root counter, so you can still access previous years count without querying always more docs.
To get a view of how this looks in practice, you should look into distributed counters in the docs.

Comments

[Osvaldo]

Please share your database structure, and also describe what kind of items you are retrieving. Firestore has usage limits.

[Danish Ajaib]

@Osvaldo Lopez I only have 1 collection 'bookings' and in it I have about 50000 documents(bookings). Each booking has about 13 fields. I am trying to create charts representing the counts on different days, months and years. Whats the best way to do this?

[Osvaldo]

When you try to retrieve 700+ items, do you get any error, exception, warning, or other kind of message, or log in your CLI or other place? You can also check Best practices for Cloud Firestore. As a Stackoverflow’s best practice, you need to create another post to ask about creating charts representing the counts on different days, months, and years showing what you have tried about it.

[Gejaa]

If you can able to get data till 600 but not after that means I think there will be some data type issues in the document. I think it's better to check that particular doc by using docId in the database.

[Moaz El-sawaf]

Please share with us if any logs are printing out, it might help with the issue.