How to import data from cloud firestore to the local emulator?

Question

I want to be able to run cloud functions locally and debug against a copy from the production data. Is there a way to copy the data that is online to the local firestore emulator?

Solution 1

You can use the firestore-backup-restore to export and import your production data as JSON files.

I wrote a quick hack to allow for importing these JSON in the Firebase Simulator Firestore instance.

I proposed a pull request and made this npm module in the meantime.

You can use it this way:

const firestoreService = require('@crapougnax/firestore-export-import')
const path = require('path')

// list of JSON files generated with the export service
// Must be in the same folder as this script
const collections = ['languages', 'roles']

// Start your firestore emulator for (at least) firestore
// firebase emulators:start --only firestore

// Initiate Firebase Test App
const db = firestoreService.initializeTestApp('test', {
   uid: 'john',
   email: '[email protected]',
})

// Start importing your data
let promises = []
try {
   collections.map(collection =>
      promises.push(
         firestoreService.fixtures(
            path.resolve(__dirname, `./${collection}.json`),
            [],
            [],
            db,
         ),
      ),
   )
   Promise.all(promises).then(process.exit)
} catch (err) {
   console.error(err)
}

Obviously, since this data won't persist in the emulator, you'll typically inject them in the before() function of your test suite or even before every test.

Solution 2

This can be accomplished through a set of commands in terminal on the existing project:

1. Login to firebase and Gcloud:

firebase login
gcloud auth login

2. See a list of your projects and connect to one:

firebase projects:list
firebase use your-project-name

gcloud projects list
gcloud config set project your-project-name

3. Export your production data to gcloud bucket with chosen name:

gcloud firestore export gs://your-project-name.appspot.com/your-choosen-folder-name

4. Now copy this folder to your local machine, I do that in functions folder directly:

cd functions
gsutil -m cp -r gs://your-project-name.appspot.com/your-choosen-folder-name .

5. Now we just want to import this folder. This should work with the basic command, thanks to latest update from Firebase team https://github.com/firebase/firebase-tools/pull/2519.

firebase emulators:start --import ./your-choosen-folder-name

Check out my article on Medium about it and a shorthanded script to do the job for you https://medium.com/firebase-developers/how-to-import-production-data-from-cloud-firestore-to-the-local-emulator-e82ae1c6ed8

Note: Its better to use a different bucket for it, as copying into your project bucket will result in the folder created in your firebase storage.
If you are interested in gsutil arguments like -m, you can see them described by executing gsutil --help.

Solution 3

My method is somewhat manual but it does the trick. I've shared it in this useful Github thread but I'll list the steps I did here if you find them useful:

1. Go to my local Firebase project path.
2. Start the emulators using: firebase emulators:start
3. Create manually some mockup data using the GUI at http://localhost:4000/firestore using the buttons provided: + Start Collection and + Add Document.
4. Export this data locally using: emulators:export ./mydirectory
5. About the project data located at Firebase Database / Cloud Firestore, I exported a single collection like this: gcloud firestore export gs://my-project-bucket-id.appspot.com --collection-ids=myCollection The export is now located under Firebase Storage in a folder with a timestamp as name (I didn't use a prefix for my test)
6. Download this folder to local drive with: gsutil cp -r gs://my-project-bucket-id.appspot.com/myCollection ./production_data_export NOTE: I did this in a Windows environment... gsutil will throw this error: "OSError: The filename, directory name, or volume label syntax is incorrect" if the folder has invalid characters for a folder name in Windows (i.e. colons) or this error: "OSError: Invalid argument.9.0 B]" if an inner file in the folder has invalid characters too. To be able to download the export locally, rename these with a valid Windows name (i.e. removing the colons) like this: gsutil mv gs://my-project-bucket-id.appspot.com/2020-05-22T02:01:06_86152 gs://my-project-bucket-id.appspot.com/myCollection
7. Once downloaded, imitate the local export structure renaming the folder to firestore_export and copying the firebase-export-metadata.json file from the local export folder. Just to be visual, here's the structure I got:

$ tree .
.
├── local_data_export
│   ├── firebase-export-metadata.json
│   └── firestore_export
│       ├── all_namespaces
│       │   └── all_kinds
│       │       ├── all_namespaces_all_kinds.export_metadata
│       │       └── output-0
│       └── firestore_export.overall_export_metadata
└── production_data_export
    ├── firebase-export-metadata.json
    └── firestore_export
        ├── all_namespaces
        │   └── kind_myCollection
        │       ├── all_namespaces_kind_myCollection.export_metadata
        │       ├── output-0
        │       └── output-1
        └── firestore_export.overall_export_metadata

8 directories, 9 files

8. Finally, start the local emulator pointing to this production data to be imported: firebase emulators:start --import=./mock_up_data/production_data_export/
9. You should see the imported data at: http://localhost:4000/firestore/

This should assist readers for now, while we await a more robust solution from the Firebase folks.

Solution 4

There is no built-in way to copy data from a cloud project to the local emulator. Since the emulator doesn't persist any data, you will have to re-generate the initial data set on every run.

Solution 5

I was able to make some npm scripts to import from remote to local emulator and vice-versa.

"serve": "yarn build && firebase emulators:start --only functions,firestore --import=./firestore_export",
"db:update-local-from-remote": "yarn db:backup-remote && gsutil -m cp -r gs://my-firebase-bucket.appspot.com/firestore_export .",
"db:update-remote-from-local": "yarn db:backup-local && yarn db:backup-remote && gsutil -m cp -r ./firestore_export gs://my-firebase-bucket.appspot.com && yarn run db:import-remote",
"db:import-remote": "gcloud firestore import gs://my-firebase-bucket.appspot.com/firestore_export",
"db:backup-local": "firebase emulators:export --force .",
"db:rename-remote-backup-folder": "gsutil mv gs://my-firebase-bucket.appspot.com/firestore_export gs://my-firebase-bucket.appspot.com/firestore_export_$(date +%d-%m-%Y-%H-%M)",
"db:backup-remote": "yarn db:rename-remote-backup-folder && gcloud firestore export gs://my-firebase-bucket.appspot.com/firestore_export"

So you can export the local Firestore data to remote with:

npm db:update-remote-from-local

Or to update your local Firestore data with remote one, do:

npm db:update-local-from-remote

These operations will backup the remote Firestore data, making a copy of it and storing it on Firebase Storage.

View more solutions

Comments

[toddwseattle]

I couldn't get through getting fireestoer-export-import to work as it seems like it has some outdated dependencies and fails when installed globally to use the script in \bin or trying to build the cloned repo on windows.

[toddwseattle]

Follow up; cloned the old repo; deleted the reference to @google-cloud/firestore in package.json as suggested in the comments; and I can build it with npm run build and then node ./build/index.js to export the info.

[Ayyappa]

"db:update-local-from-remote": "yarn db:backup-remote && gsutil -m cp -r gs://my-firebase-bucket.appspot.com/firestore_export .", How does this import to emulator db?

[Estevão Lucas]

Sorry, the list of commands was missing the "serve" one, which starts the emulator with --import options. Thus after running db:update-local-from-remote you just need to restart your emulators. github.com/firebase/firebase-tools/pull/1968

[user1114]

How can we run the db:update-local-from-remote script? Is this meant to be running from gcloud, firebase or some other program?

[user1114]

This actually works! The only quirk I came across is exporting Firestore - it wouldn't export my collection contents - just an empty collection. I left off the -collection-ids flag and just exported everything and it worked

[Estevão Lucas]

to you from your local environment

[Amit Sharma]

failing because of this file : firebase-export-metadata.json

[Ross Rawlins]

when I import thr data it take longer than 30000ms so the emulator does not run. Is there a way around this timeout?

[Alb Bolush]

do you mean that you get error while running this command: firebase emulators:start --import your-choosen-folder-name ? can you share error message ? Just to see if firebase is throwing time out or its some kind internal time out that we can increase.

[MorenoMdz]

Quick note, there is currently a bug with gsuit when running python 3.8. using the -m flag will crash it, you can either run it with python 3.7 or run without the flag -m.

[Alb Bolush]

updated with shorter version, no metadata renaming required longer, see: github.com/firebase/firebase-tools/pull/2519.

[Albert Khang]

to make sure to install the Google Cloud SDK (cloud.google.com/sdk/docs/install) and the Firebase CLI (firebase.google.com/docs/cli) before begin.

[Nestor Solalinde]

anyone else getting Error adding trigger: FirebaseError: HTTP Error: 500, [object Object] with this approach?

[Olivier de Jonge]

Exporting only the database can be quite useless without authentication userdata. To do so use in your local project terminal: % firebase auth:export accounts.json --format=json. See also: firebase.google.com/docs/cli/auth

[MadMac]

If you use the do not specify a folder name this does not work on a windows machine due to the ":" in the folder name. You need to rename the folder first using this method. Step 3.5 stackoverflow.com/a/62544141/10222449

[tainterr]

Just to piggyback on this in case it helps someone: Since I can never remember the names of the buckets, I make a dir on my local with the same name and add it to gitignore. Then I have a function getBucket() { gsutil -m cp -r gs://${PWD##*/} .; } so that I can just go into the dir and run getBucket to sync the data, then start the emulator with the latest file that was synced using the same command listed above firebase emulators:start --import <<latest file name>>

[roman]

Small note: In step #4, the entire command is firebase emulators:export ./mydirectory. I know it's obvious, but I did get confused for a minute.

[Volker Andres]

As a Mac user I had to add the following option to make it work: gsutil -m -o "GSUtil:parallel_process_count=1" cp -r gs://your-project-name.appspot.com/your-choosen-folder-name . That disables multiprocessing.

[Mr B]

gcloud config set project your-project-name is asking me to key in the project ID instead. So it should be gcloud config set project your-project-id

[imagio]

Note that there is some unknown arbitrary limit to the size of the dataset you can import into the emulators. I tried with a 1.5gb database and the emulator hung/crashed every time even with 64gb of ram. github.com/firebase/firebase-tools/issues/3929

[straya]

Seems it no longer wants a folder: : is a directory ⚠ Script exited unsuccessfully (code 126)

[pale bone]

I had to use gsutil rsync instead of gsutil cp