Firestore: How to get random documents in a collection

Question

It is crucial for my application to be able to select multiple documents at random from a collection in firebase.

Since there is no native function built in to Firebase (that I know of) to achieve a query that does just this, my first thought was to use query cursors to select a random start and end index provided that I have the number of documents in the collection.

This approach would work but only in a limited fashion since every document would be served up in sequence with its neighboring documents every time; however, if I was able to select a document by its index in its parent collection I could achieve a random document query but the problem is I can't find any documentation that describes how you can do this or even if you can do this.

Here's what I'd like to be able to do, consider the following firestore schema:

root/
  posts/
     docA
     docB
     docC
     docD

Then in my client (I'm in a Swift environment) I'd like to write a query that can do this:

db.collection("posts")[0, 1, 3] // would return: docA, docB, docD

Is there anyway I can do something along the lines of this? Or, is there a different way I can select random documents in a similar fashion?

Please help.

Solution 1

Using randomly generated indexes and simple queries, you can randomly select documents from a collection or collection group in Cloud Firestore.

This answer is broken into 4 sections with different options in each section:

1. How to generate the random indexes
2. How to query the random indexes
3. Selecting multiple random documents
4. Reseeding for ongoing randomness

How to generate the random indexes

The basis of this answer is creating an indexed field that when ordered ascending or descending, results in all the document being randomly ordered. There are different ways to create this, so let's look at 2, starting with the most readily available.

Auto-Id version

If you are using the randomly generated automatic ids provided in our client libraries, you can use this same system to randomly select a document. In this case, the randomly ordered index is the document id.

Later in our query section, the random value you generate is a new auto-id (iOS, Android, Web) and the field you query is the __name__ field, and the 'low value' mentioned later is an empty string. This is by far the easiest method to generate the random index and works regardless of the language and platform.

By default, the document name (__name__) is only indexed ascending, and you also cannot rename an existing document short of deleting and recreating. If you need either of these, you can still use this method and just store an auto-id as an actual field called random rather than overloading the document name for this purpose.

Random Integer version

When you write a document, first generate a random integer in a bounded range and set it as a field called random. Depending on the number of documents you expect, you can use a different bounded range to save space or reduce the risk of collisions (which reduce the effectiveness of this technique).

You should consider which languages you need as there will be different considerations. While Swift is easy, JavaScript notably can have a gotcha:

• 32-bit integer: Great for small (~10K unlikely to have a collision) datasets
• 64-bit integer: Large datasets (note: JavaScript doesn't natively support, yet)

This will create an index with your documents randomly sorted. Later in our query section, the random value you generate will be another one of these values, and the 'low value' mentioned later will be -1.

How to query the random indexes

Now that you have a random index, you'll want to query it. Below we look at some simple variants to select a 1 random document, as well as options to select more than 1.

For all these options, you'll want to generate a new random value in the same form as the indexed values you created when writing the document, denoted by the variable random below. We'll use this value to find a random spot on the index.

Wrap-around

Now that you have a random value, you can query for a single document:

let postsRef = db.collection("posts")
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: random)
                   .order(by: "random")
                   .limit(to: 1)

Check that this has returned a document. If it doesn't, query again but use the 'low value' for your random index. For example, if you did Random Integers then lowValue is 0:

let postsRef = db.collection("posts")
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: lowValue)
                   .order(by: "random")
                   .limit(to: 1)

As long as you have a single document, you'll be guaranteed to return at least 1 document.

Bi-directional

The wrap-around method is simple to implement and allows you to optimize storage with only an ascending index enabled. One downside is the possibility of values being unfairly shielded. E.g if the first 3 documents (A,B,C) out of 10K have random index values of A:409496, B:436496, C:818992, then A and C have just less than 1/10K chance of being selected, whereas B is effectively shielded by the proximity of A and only roughly a 1/160K chance.

Rather than querying in a single direction and wrapping around if a value is not found, you can instead randomly select between >= and <=, which reduces the probability of unfairly shielded values by half, at the cost of double the index storage.

If one direction returns no results, switch to the other direction:

queryRef = postsRef.whereField("random", isLessThanOrEqualTo: random)
                   .order(by: "random", descending: true)
                   .limit(to: 1)

queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: random)
                   .order(by: "random")
                   .limit(to: 1)

Selecting multiple random documents

Often, you'll want to select more than 1 random document at a time. There are 2 different ways to adjust the above techniques depending on what trade offs you want.

Rinse & Repeat

This method is straight forward. Simply repeat the process, including selecting a new random integer each time.

This method will give you random sequences of documents without worrying about seeing the same patterns repeatedly.

The trade-off is it will be slower than the next method since it requires a separate round trip to the service for each document.

Keep it coming

In this approach, simply increase the number in the limit to the desired documents. It's a little more complex as you might return 0..limit documents in the call. You'll then need to get the missing documents in the same manner, but with the limit reduced to only the difference. If you know there are more documents in total than the number you are asking for, you can optimize by ignoring the edge case of never getting back enough documents on the second call (but not the first).

The trade-off with this solution is in repeated sequences. While the documents are randomly ordered, if you ever end up overlapping ranges you'll see the same pattern you saw before. There are ways to mitigate this concern discussed in the next section on reseeding.

This approach is faster than 'Rinse & Repeat' as you'll be requesting all the documents in the best case a single call or worst case 2 calls.

Reseeding for ongoing randomness

While this method gives you documents randomly if the document set is static the probability of each document being returned will be static as well. This is a problem as some values might have unfairly low or high probabilities based on the initial random values they got. In many use cases, this is fine but in some, you may want to increase the long term randomness to have a more uniform chance of returning any 1 document.

Note that inserted documents will end up weaved in-between, gradually changing the probabilities, as will deleting documents. If the insert/delete rate is too small given the number of documents, there are a few strategies addressing this.

Multi-Random

Rather than worrying out reseeding, you can always create multiple random indexes per document, then randomly select one of those indexes each time. For example, have the field random be a map with subfields 1 to 3:

{'random': {'1': 32456, '2':3904515723, '3': 766958445}}

Now you'll be querying against random.1, random.2, random.3 randomly, creating a greater spread of randomness. This essentially trades increased storage to save increased compute (document writes) of having to reseed.

Reseed on writes

Any time you update a document, re-generate the random value(s) of the random field. This will move the document around in the random index.

Reseed on reads

If the random values generated are not uniformly distributed (they're random, so this is expected), then the same document might be picked a dispropriate amount of the time. This is easily counteracted by updating the randomly selected document with new random values after it is read.

Since writes are more expensive and can hotspot, you can elect to only update on read a subset of the time (e.g, if random(0,100) === 0) update;).

Solution 2

Posting this to help anyone that has this problem in the future.

If you are using Auto IDs you can generate a new Auto ID and query for the closest Auto ID as mentioned in Dan McGrath's Answer.

I recently created a random quote api and needed to get random quotes from a firestore collection.
This is how I solved that problem:

var db = admin.firestore();
var quotes = db.collection("quotes");

var key = quotes.doc().id;

quotes.where(admin.firestore.FieldPath.documentId(), '>=', key).limit(1).get()
.then(snapshot => {
    if(snapshot.size > 0) {
        snapshot.forEach(doc => {
            console.log(doc.id, '=>', doc.data());
        });
    }
    else {
        var quote = quotes.where(admin.firestore.FieldPath.documentId(), '<', key).limit(1).get()
        .then(snapshot => {
            snapshot.forEach(doc => {
                console.log(doc.id, '=>', doc.data());
            });
        })
        .catch(err => {
            console.log('Error getting documents', err);
        });
    }
})
.catch(err => {
    console.log('Error getting documents', err);
});

The key to the query is this:

.where(admin.firestore.FieldPath.documentId(), '>', key)

And calling it again with the operation reversed if no documents are found.

I hope this helps!

Solution 3

Just made this work in Angular 7 + RxJS, so sharing here with people who want an example.

I used @Dan McGrath 's answer, and I chose these options: Random Integer version + Rinse & Repeat for multiple numbers. I also used the stuff explained in this article: RxJS, where is the If-Else Operator? to make if/else statements on stream level (just if any of you need a primer on that).

Also note I used angularfire2 for easy Firebase integration in Angular.

Here is the code:

import { Component, OnInit } from '@angular/core';
import { Observable, merge, pipe } from 'rxjs';
import { map, switchMap, filter, take } from 'rxjs/operators';
import { AngularFirestore, QuerySnapshot } from '@angular/fire/firestore';

@Component({
  selector: 'pp-random',
  templateUrl: './random.component.html',
  styleUrls: ['./random.component.scss']
})
export class RandomComponent implements OnInit {

  constructor(
    public afs: AngularFirestore,
  ) { }

  ngOnInit() {
  }

  public buttonClicked(): void {
    this.getRandom().pipe(take(1)).subscribe();
  }

  public getRandom(): Observable<any[]> {
    const randomNumber = this.getRandomNumber();
    const request$ = this.afs.collection('your-collection', ref => ref.where('random', '>=', randomNumber).orderBy('random').limit(1)).get();
    const retryRequest$ = this.afs.collection('your-collection', ref => ref.where('random', '<=', randomNumber).orderBy('random', 'desc').limit(1)).get();

    const docMap = pipe(
      map((docs: QuerySnapshot<any>) => {
        return docs.docs.map(e => {
          return {
            id: e.id,
            ...e.data()
          } as any;
        });
      })
    );

    const random$ = request$.pipe(docMap).pipe(filter(x => x !== undefined && x[0] !== undefined));

    const retry$ = request$.pipe(docMap).pipe(
      filter(x => x === undefined || x[0] === undefined),
      switchMap(() => retryRequest$),
      docMap
    );

    return merge(random$, retry$);
  }

  public getRandomNumber(): number {
    const min = Math.ceil(Number.MIN_VALUE);
    const max = Math.ceil(Number.MAX_VALUE);
    return Math.floor(Math.random() * (max - min + 1)) + min;
  }
}

Solution 4

You can use listDocuments() property for get only Query list of documents id. Then generate random id using the following way and get DocumentSnapshot with get() property.

  var restaurantQueryReference = admin.firestore().collection("Restaurant"); //have +500 docs
  var restaurantQueryList = await restaurantQueryReference.listDocuments(); //get all docs id; 

  for (var i = restaurantQueryList.length - 1; i > 0; i--) {
    var j = Math.floor(Math.random() * (i + 1));
    var temp = restaurantQueryList[i];
    restaurantQueryList[i] = restaurantQueryList[j];
    restaurantQueryList[j] = temp;
}

var restaurantId = restaurantQueryList[Math.floor(Math.random()*restaurantQueryList.length)].id; //this is random documentId 

Solution 5

undoubtedly Above accepted Answer is SuperUseful but There is one case like If we had a collection of some Documents(about 100-1000) and we want some 20-30 random Documents Provided that Document must not be repeated. (case In Random Problems App etc...).

Problem with the Above Solution: For a small number of documents in the Collection(say 50) Probability of repetition is high. To avoid it If I store Fetched Docs Id and Add-in Query like this:

queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: lowValue).where("__name__", isNotEqualTo:"PreviousId")
               .order(by: "random")
               .limit(to: 1)

here PreviousId is Id of all Elements that were fetched Already means A loop of n previous Ids. But in this case, network Call would be high.

My Solution: Maintain one Special Document and Keep a Record of Ids of this Collection only, and fetched this document First Time and Then Do all Randomness Stuff and check for previously not fetched on App site. So in this case network call would be only the same as the number of documents requires (n+1).

Disadvantage of My solution: Have to maintain A document so Write on Addition and Deletion. But it is good If reads are very often then Writes which occurs in most cases.

View more solutions

Comments

[Garret Kaye]

Thanks Dan I really appreciate the reply but referring to the agnostic version (which sounds like the better bet to me), if I wanted to get more than one random document I would have to call this query multiple times? Or increase the limit on the query (which would return random clusters but the documents in those clusters would always be in the same sequence)?

[Dan McGrath]

Correct, both of those options are viable. The former (multiple calls) will be slower, but lead to less repeated sequence if done often. The latter (larger limit) will be fast, but increase the chance of seeing the same sequence again. Note with the latter, as more documents are added, the sequence can change. You can also redo the random number whenever you update the document to change the sequences around more.

[Garret Kaye]

Oh right! I think this method will do nicely. Thank you your help is much appreciated!

[Dan McGrath]

No problem, @GarretKaye!

[Frank van Puffelen]

Very cool solution Dan! In fact... this should also be possible on the Realtime Database, shouldn't it?

[Dan McGrath]

Thanks! Yes, I don't see why it wouldn't be.

[Remi Sture]

Will this also work with chained where()'s to create more specific queries? For instance get a random student where first name is 'Alex' and city is 'NYC'.

[Dan McGrath]

Yes! As long as the other where statements are equality filters (==) then you're good to go.

[Jason Berryman]

This would be great to add to the Solutions page

[Himanshu Rawat]

Nice solution but I will unnecessarily add more Firestore Ops

[Jek]

@DanMcGrath if we were to add this to the solutions page, or readers reading this post who are using RxJS e.g. for those using Angular + Firebase they could combine your mentioned above with recursion to get get a document with .expand() operator of RxJS.

[gautam]

Can any one give me working code with firestore i tried whole day but could not find solution with it , with mongodb it is so easy . Thanks for the help

[Jek]

@DanMcGrath this technique does not work if random is an array of randoms e.g. 'random': ['3tVvilep7arKFILbNaNu', '2rIudz3Hx640hGbqovrc', 'qbr2HR89QZAtIfH9keCh']. However, it works with 'random': '3tVvilep7arKFILbNaNu'. Logically speaking, it should work with array too huh? See stackoverflow.com/q/55029501/3073280

[nicoxx]

If you want to do only one request and be sure there will be the expected number of results, split your random interval in two. For example, for ints, take "GreaterThan" if your random number is negative and "LessThan" if it's positive

[Mike Mat]

*This method is good in many cases, but if 3 documents have similar values in the randomly generated integer field, the document between the other 2 is less likely to be selected. For example, we have doc1: {rand:400}, doc2: {rand:405}, doc3: {rand:410}, then for "doc2" to be selected, we need a number from 401 to 405.

[Dan McGrath]

@MikeMat, updated answer to help address this. Also posted a related Q: math.stackexchange.com/questions/3233270/…

[Dan McGrath]

For future readers: I updated my answer for clarity and renamed the section 'Document Id agnostic version' to 'Random Integer version'

[Dan McGrath]

Extremely unlikely to run into this issue with Document Id's, but in case someone copies this and uses it with a much smaller Id space, I'd recommend changing the first where clause from '>' to '>='. This prevents a failure on the edge case of their only being 1 document, and key is selected in such a way to be exactly the 1 document's id.

[MartinJH]

Updated my answer to match your changes.

[Daniele]

Thank you for the great answer you posted here. I've got a question, what does 'admin.firestore.FieldPath.documentId()' refer to exactly?

[Atul Chaudhary]

@HimanshuRawat you are right, if your app has a large user base then it can have a huge impact

[MobileMon]

I'm using Flutter and this doesn't randomly get a document. It has a high percentage chance of returning the same document. Ultimately it will get random documents but 90% of the time its the same document

[Jek]

Very neat solution. Great but where in your code are you doing rinse and repeat for multiple numbers?

[MartinJH]

@choopage-JekBao As I understand it, Rinse & Repeat means getting a new random number and then make a request for each time the buttonClicked() method is called. Makes sense? :P

[t3chb0t]

So much work instead of simply adding the orderByRandom() api :\

[Leblanc Meneses]

The reason @MobileMon is because the solution is missing an orderBy so limit(1) doesn't get the "closest" to the random value as expected. My solution below does. I also take 10 and randomize locally.

[Jek]

@DanMcGrath can you add your solution to Firebase Extension?

[Jek]

@DanMcGrath I have been studying and researching into your solution for a long time. It is easy to get one document at random. It is hard to get multiple random documents (non-repeat) with certainty. stackoverflow.com/q/64606340/3073280

[genericUser]

How is the auto-id solution is a real equal-possiblity random @DanMcGrath? Assume that I have generate 3 documents with the auto-id, aShc2#da, Bdjekc$d3, c83#s4ee. The possibility to get the third document each time is much higher. That solution is not equal-possibility.

[ShadeToD]

Its not the best idea to query for every document in your database (you will have to pay for every document read) " let totalDoc = db.collection("stat").get().then(snap=>snap.size)"

[ShadeToD]

Where do you keep size of collection? or maybe you are couting it everytime you create a new document?

[Chickenchaser]

@ShadeToD counting document in large size has already many solutions like distributed counter. Btw.. how to tag other? it seems @+id is not enough

[som]

@DanMcGrath you use both <= and >= .. would it be better to follow up with > so as not to double dip (esp. w multiple docs / limit setup)?

[Rohan Deshpande]

I am trying this solution out with auto generated IDs, but the problem I'm having is with this section "the 'low value' mentioned later is an empty string" When I do this, I get an error FirebaseError: Invalid query. When querying with FieldPath.documentId(), you must provide a valid document ID, but it was an empty string. Any help?

[Rohan Deshpande]

^ So currently it won't work if you use '' for the low value but if you use ' ' this solution seems to work for auto IDs

[Blueriver]

That could be fixed by storing a document counter, which gets increased every time a document is added and decreased every time a document is deleted.

[Chukwu3meka]

that'll be a better solution, but what if the document deleted is not the last one in the database

[NhatHo]

Can Random Integer version guarantee that in 1 query will always return 20 documents if the collection contains more than 20 documents?

[orser]

This is a bad approach regarding firebase pricing model where you get charged for the number of read and writes. In your solution to get 1 random document in your Restaurant collection of 500 entries you get charged for 500 reads, growing with scale.