Firestore slow performance issue on getting data

Question

I'm having slow performance issues with Firestore while retrieving basic data stored in a document compared to the realtime database with 1/10 ratio.

Using Firestore, it takes an average of 3000 ms on the first call

 this.db.collection(‘testCol’)
   .doc(‘testDoc’)
   .valueChanges().forEach((data) => {
     console.log(data);//3000 ms later
 });

Using the realtime database, it takes an average of 300 ms on the first call

 this.db.database.ref(‘/test’).once(‘value’).then(data => {
     console.log(data); //300ms later
 });

This is a screenshot of the network console :

I'm running the Javascript SDK v4.50 with AngularFire2 v5.0 rc.2.

Did anyone experience this issue ?

Solution 1

UPDATE: 12th Feb 2018 - iOS Firestore SDK v0.10.0

Similar to some other commenters, I've also noticed a slower response on the first get request (with subsequent requests taking ~100ms). For me it's not as bad as 30s, but maybe around 2-3s when I have good connectivity, which is enough to provide a bad user experience when my app starts up.

Firebase have advised that they're aware of this "cold start" issue and they're working on a long term fix for it - no ETA unfortunately. I think it's a separate issue that when I have poor connectivity, it can take ages (over 30s) before get requests decide to read from cache.

Whilst Firebase fix all these issues, I've started using the new disableNetwork() and enableNetwork() methods (available in Firestore v0.10.0) to manually control the online/offline state of Firebase. Though I've had to be very careful where I use it in my code, as there's a Firestore bug that can cause a crash under certain scenarios.

---

UPDATE: 15th Nov 2017 - iOS Firestore SDK v0.9.2

It seems the slow performance issue has now been fixed. I've re-run the tests described below and the time it takes for Firestore to return the 100 documents now seems to be consistently around 100ms.

Not sure if this was a fix in the latest SDK v0.9.2 or if it was a backend fix (or both), but I suggest everyone updates their Firebase pods. My app is noticeably more responsive - similar to the way it was on the Realtime DB.

---

I've also discovered Firestore to be much slower than Realtime DB, especially when reading from lots of documents.

Updated tests (with latest iOS Firestore SDK v0.9.0):

I set up a test project in iOS Swift using both RTDB and Firestore and ran 100 sequential read operations on each. For the RTDB, I tested the observeSingleEvent and observe methods on each of the 100 top level nodes. For Firestore, I used the getDocument and addSnapshotListener methods at each of the 100 documents in the TestCol collection. I ran the tests with disk persistence on and off. Please refer to the attached image, which shows the data structure for each database.

I ran the test 10 times for each database on the same device and a stable wifi network. Existing observers and listeners were destroyed before each new run.

Realtime DB observeSingleEvent method:

func rtdbObserveSingle() {

    let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
    print("Started reading from RTDB at: \(start)")

    for i in 1...100 {
        Database.database().reference().child(String(i)).observeSingleEvent(of: .value) { snapshot in
            let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
            let data = snapshot.value as? [String: String] ?? [:]
            print("Data: \(data). Returned at: \(time)")
        }
    }
}

Realtime DB observe method:

func rtdbObserve() {

    let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
    print("Started reading from RTDB at: \(start)")

    for i in 1...100 {
        Database.database().reference().child(String(i)).observe(.value) { snapshot in
            let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
            let data = snapshot.value as? [String: String] ?? [:]
            print("Data: \(data). Returned at: \(time)")
        }
    }
}

Firestore getDocument method:

func fsGetDocument() {

    let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
    print("Started reading from FS at: \(start)")

    for i in 1...100 {
        Firestore.firestore().collection("TestCol").document(String(i)).getDocument() { document, error in

            let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
            guard let document = document, document.exists && error == nil else {
                print("Error: \(error?.localizedDescription ?? "nil"). Returned at: \(time)")
                return
            }
            let data = document.data() as? [String: String] ?? [:]
            print("Data: \(data). Returned at: \(time)")
        }
    }
}

Firestore addSnapshotListener method:

func fsAddSnapshotListener() {

    let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
    print("Started reading from FS at: \(start)")

    for i in 1...100 {
        Firestore.firestore().collection("TestCol").document(String(i)).addSnapshotListener() { document, error in

            let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
            guard let document = document, document.exists && error == nil else {
                print("Error: \(error?.localizedDescription ?? "nil"). Returned at: \(time)")
                return
            }
            let data = document.data() as? [String: String] ?? [:]
            print("Data: \(data). Returned at: \(time)")
        }
    }
}

Each method essentially prints the unix timestamp in milliseconds when the method starts executing and then prints another unix timestamp when each read operation returns. I took the difference between the initial timestamp and the last timestamp to return.

RESULTS - Disk persistence disabled:

RESULTS - Disk persistence enabled:

Data Structure:

When the Firestore getDocument / addSnapshotListener methods get stuck, it seems to get stuck for durations that are roughly multiples of 30 seconds. Perhaps this could help the Firebase team isolate where in the SDK it's getting stuck?

Solution 2

Update Date March 02, 2018

It looks like this is a known issue and the engineers at Firestore are working on a fix. After a few email exchanges and code sharing with a Firestore engineer on this issue, this was his response as of today.

"You are actually correct. Upon further checking, this slowness on getDocuments() API is a known behavior in Cloud Firestore beta. Our engineers are aware of this performance issue tagged as "cold starts", but don't worry as we are doing our best to improve Firestore query performance.

We are already working on a long-term fix but I can't share any timelines or specifics at the moment. While Firestore is still on beta, expect that there will be more improvements to come."

So hopefully this will get knocked out soon.

---

Using Swift / iOS

After dealing with this for about 3 days it seems the issue is definitely the get() ie .getDocuments and .getDocument. Things I thought were causing the extreme yet intermittent delays but don't appear to be the case:

1. Not so great network connectivity
2. Repeated calls via looping over .getDocument()
3. Chaining get() calls
4. Firestore Cold starting
5. Fetching multiple documents (Fetching 1 small doc caused 20sec delays)
6. Caching (I disabled offline persistence but this did nothing.)

I was able to rule all of these out as I noticed this issue didn't happen with every Firestore database call I was making. Only retrievals using get(). For kicks I replaced .getDocument with .addSnapshotListener to retrieve my data and voila. Instant retrieval each time including the first call. No cold starts. So far no issues with the .addSnapshotListener, only getDocument(s).

For now, I'm simply dropping the .getDocument() where time is of the essence and replacing it with .addSnapshotListener then using

for document in querySnapshot!.documents{
// do some magical unicorn stuff here with my document.data()
}

... in order to keep moving until this gets worked out by Firestore.

Solution 3

Almost 3 years later, firestore being well out of beta and I can confirm that this horrible problem still persists ;-(

On our mobile app we use the javascript / node.js firebase client. After a lot of testing to find out why our app's startup time is around 10sec we identified what to attribute 70% of that time to... Well, to firebase's and firestore's performance and cold start issues:

• firebase.auth().onAuthStateChanged() fires approx. after 1.5 - 2sec, already quite bad.
• If it returns a user, we use its ID to get the user document from firestore. This is the first call to firestore and the corresponding get() takes 4 - 5sec. Subsequent get() of the same or other documents take approx. 500ms.

So in total the user initialization takes 6 - 7 sec, completely unacceptable. And we can't do anything about it. We can't test disabling persistence, since in the javascript client there's no such option, persistence is always enabled by default, so not calling enablePersistence() won't change anything.

Solution 4

I had this issue until this morning. My Firestore query via iOS/Swift would take around 20 seconds to complete a simple, fully indexed query - with non-proportional query times for 1 item returned - all the way up to 3,000.

My solution was to disable offline data persistence. In my case, it didn't suit the needs of our Firestore database - which has large portions of its data updated every day.

iOS & Android users have this option enabled by default, whilst web users have it disabled by default. It makes Firestore seem insanely slow if you're querying a huge collection of documents. Basically it caches a copy of whichever data you're querying (and whichever collection you're querying - I believe it caches all documents within) which can lead to high Memory usage.

In my case, it caused a huge wait for every query until the device had cached the data required - hence the non-proportional query times for the increasing numbers of items to return from the exact same collection. This is because it took the same amount of time to cache the collection in each query.

Offline Data - from the Cloud Firestore Docs

I performed some benchmarking to display this effect (with offline persistence enabled) from the same queried collection, but with different amounts of items returned using the .limit parameter:

Now at 100 items returned (with offline persistence disabled), my query takes less than 1 second to complete.

My Firestore query code is below:

let db = Firestore.firestore()
self.date = Date()
let ref = db.collection("collection").whereField("Int", isEqualTo: SomeInt).order(by: "AnotherInt", descending: true).limit(to: 100)
ref.getDocuments() { (querySnapshot, err) in
    if let err = err {
        print("Error getting documents: \(err)")
    } else {
        for document in querySnapshot!.documents {
            let data = document.data()
            //Do things
        }
        print("QUERY DONE")
        let currentTime = Date()
        let components = Calendar.current.dateComponents([.second], from: self.date, to: currentTime)
        let seconds = components.second!
        print("Elapsed time for Firestore query -> \(seconds)s")
        // Benchmark result
    }
}

Solution 5

well, from what I'm currently doing and research by using nexus 5X in emulator and real android phone Huawei P8,

Firestore and Cloud Storage are both give me a headache of slow response when I do first document.get() and first storage.getDownloadUrl()

It give me more than 60 seconds response on each request. The slow response only happen in real android phone. Not in emulator. Another strange thing. After the first encounter, the rest request is smooth.

Here is the simple code where I meet the slow response.

var dbuserref = dbFireStore.collection('user').where('email','==',email);
const querySnapshot = await dbuserref.get();

var url = await defaultStorage.ref(document.data().image_path).getDownloadURL();

I also found link that is researching the same. https://reformatcode.com/code/android/firestore-document-get-performance

View more solutions

Comments

[Sam Stern]

What performance do you see if you make a second call (to a different document / collection)? Do you see the same issue if you don't use angularfire?

[lmiller1990]

I'm having a similar experience. First call is a bit slow, 5-10s sometimes. I'm making a chat app - first msg takes a while to deliver, following ones are almost instant, though. Firestore is still beta, they are likely still sorting out quirks.

[wizloc]

Similar experience here. First onSnapShot takes an outrageous amount of time - up to 2 mins for some users that is rendering our app unuseable

[DarkNeuron]

Same problem, quite frustrating. Some are reporting that doing a write will release the "hung" queries.

[aMarCruz]

Same issue, up to 1.5 mins with a simple collection.get(document)

[DarkNeuron]

The offline layer is the culprit. If you don't run "enablePersistence()", the app speeds up immediately. Our observations are primarily from Android devices (using Cordova).

[trollkotze]

Does this problem still exist?

[Faruk]

now firebase realtime database in android (enable offline mode) has also slow at first time, I've been forced to use REST using cloud functions which is much faster.

[Kind Contributor]

any new advice, now that Firestore is production ready?

[Seb]

just to confirm that I can still reproduce the problem.... around 8 seconds to retrieve a single document. await firestore.collection("photos").doc(id).get()

[Viswanath Lekshmanan]

Still a valid question though :)

[Dominic]

I have a collection with 1 document with 3 fields in it and no special query (just return me everything), and it takes like 3 seconds to return data wtf, very concerned about investing more time if this is the performance when it has no work to do

[Faruk]

So firestore is more expensive and much slower.. Hope firebase team see this

[Sam Stern]

[Firebaser here] thank you for taking the time to provide such detailed data, we always appreciate that. The issue is not that the system is "slower" it's that a very small number of queries get stuck or take a huge amount of time to return. We have some fixes coming soon that we believe will improve the situation.

[Sam Stern]

Are these test numbers from the Web, iOS, or Android?

[Hendies]

The benchmark is from iOS, although I would expect performance gains on all platforms - depending upon the size of your queried collection

[Hendies]

Didn't realise you were on the Firebase team! The giveaway for this query was that the memory usage would shoot to high numbers (600-700mb ram) where I assumed our collection was stored in cache. The query would always hang, and then complete once the memory progressively rose and then reached the same point (700mb-ish). After persistence was disabled, this effect ceased and our memory remained as expected (100-150mb), whilst returning our results super-fast. If you need more info, please ask.

[Sam Stern]

Huh that is very interesting. Could you possibly create a sample Xcode project that replicates this and email it to me? If so it's samstern at google dot com. I'd love to take a closer look.

[Saul]

Thanks for keeping us updated. I've added some new results for the latest Firestore SDK v0.9.0, which might help your team isolate the source of the problem. I've also experienced another problem with the snapshot listener: stackoverflow.com/questions/46710371/… The root cause may or may not be related to this topic, but it'd be great if the Firebase team could take a look in it. Many thanks!

[jjjjs]

@SamStern, Do you have any update to this? I am in the middle of converting a Realtime DB project to Firestore, and I am experiencing Firestore getting "stuck" on a regular basis. Seems to not return anything for around 30 seconds, and then suddenly the data arrives. It's very fristrating and giving me second thoughts about migrating for Firestore.

[Sam Stern]

@jjjjs are you on web? Does issuing any random "set" or "update" operation get you out of the "stuck" state?

[jjjjs]

@SamStern, I am developing an Ionic app using Angularfire2 v5. Calling "set" does not unstick firestore. I added a little function that writes to an unrelated document and call that using setTimeout() with a 5 second delay so I can be sure that it is being called after the read attempt. The write operation executes and firestore remains stuck for an extended duration and then suddenly becomes unstuck and the data appears.

[Sam Stern]

@jjjjs could you try doing a simple Firestore read without Angular or Ionic but from the same device? That will help us figure out where the lag is.

[inteist]

@SamStern I am experiencing similar issues ("stuck" queries and very slow performance) with Firestore on Android when in offline mode. This is very disappointing. FirebaseDB works almost instantly in online or offline mode. Is this a bug/collection of bugs or is that by design and we should consider other solutions because as of now the "offline" mode for firestore so flaky and so buggy that it can be said it simply does not work.

[DarkNeuron]

We're also experiencing "stuck" queries on the web sdk. Hangs for 10-20 sec, then data arrives (on v4.8.0).

[Ben]

same here, everything was ok a week ago though; then today, started being that slow. Running test, the first query takes that much of time (roughly 30 seconds), then all others happens as usual (a few ms)

[Saul]

I've noticed this recently and reported to Firebase. They're aware of the "cold start" issue and are working on a fix. In the meantime I'm trying a workaround detailed in my update above and I've had varying degrees of success with it.

[sowdri]

I'm seeing the same behavior as well, but only in Android. For now I'm falling back to snapshots as well. But it will be good if the performance of get queries is consistent.

[Jeff Padgett]

I also see slow performance as well with the FirebaseUI recycler adapter, which uses addSnapshotListener.

[trollkotze]

Does this "cold start" problem still exist?

[trollkotze]

Does this "cold start" problem still exist? I'm a bit confused by your update from March mentioning the Firebase engineers are aware of what they tagges as a "cold start" issue, because in your original answer you wrote that you had ruled out "4. Firestore Cold starting" to be the problem?

[hiten pannu]

I am still seeing the slow performance in android and a lot of memory issue. Are you guys planning to provide any update with this fix

[Deepak Joshi]

@SamStern, The issue still persists on android, any update on this?

[rednuht]

@SamStern : this has been a while, but I'm facing the exact same issue on Android. Any clues on what could cause this? I tried to build a minimal project but that project does not have that issue!

[johndoe]

The issue still happens with the latest version of firestore i. iOS and using snapshot listener worked like a charm. Great finding.

[Renascienza]

I am noticing bad performance as well on not so complicated queries, with mostly flat objects and a relative low amount of items (10-100). Makes no sense, because Firebase team insist "you don't need cache things by yourself or avoid query the database, Firestore is a network based store". I'm considering cloud Realm instead, to any further project.

[Ali]

@SamStern I see your comments and I note that you comment on year 2017 and now the year is 2020. So may I know there any solution provided for getting data faster and why it is slow?

[bjornl]

Any workaround to this issue? Some pointers to where to look for answers would be greatly appreciated.

[pat]

In the javascript client it is turned off by default: For the web, offline persistence is disabled by default. To enable persistence, call the enablePersistence method but I can confirm when leaving it off, our initial request times go from a similar ~8 seconds down to about ~500ms firebase.google.com/docs/firestore/manage-data/enable-offlin‌​e

[JPJ]

Correct, in javascript it is off by default, and the times I mention above are with the default setting. And in our case we need fresh and updated user profile data, so using persistence is not an option.