How to delete all datastore in Google App Engine?

Question

Does anyone know how to delete all datastore in Google App Engine?

Solution 1

If you're talking about the live datastore, open the dashboard for your app (login on appengine) then datastore --> dataviewer, select all the rows for the table you want to delete and hit the delete button (you'll have to do this for all your tables). You can do the same programmatically through the remote_api (but I never used it).

If you're talking about the development datastore, you'll just have to delete the following file: "./WEB-INF/appengine-generated/local_db.bin". The file will be generated for you again next time you run the development server and you'll have a clear db.

Make sure to clean your project afterwards.

This is one of the little gotchas that come in handy when you start playing with the Google Application Engine. You'll find yourself persisting objects into the datastore then changing the JDO object model for your persistable entities ending up with obsolete data that'll make your app crash all over the place.

Solution 2

The best approach is the remote API method as suggested by Nick, he's an App Engine engineer from Google, so trust him.

It's not that difficult to do, and the latest 1.2.5 SDK provides the remote_shell_api.py out of the shelf. So go to download the new SDK. Then follow the steps:

• connect remote server in your commandline: remote_shell_api.py yourapp /remote_api The shell will ask for your login info, and if authorized, will make a Python shell for you. You need setup url handler for /remote_api in your app.yaml

• fetch the entities you'd like to delete, the code looks something like:

    from models import Entry
    query = Entry.all(keys_only=True)
    entries =query.fetch(1000)
    db.delete(entries)
    \# This could bulk delete 1000 entities a time

Update 2013-10-28:

• remote_shell_api.py has been replaced by remote_api_shell.py, and you should connect with remote_api_shell.py -s your_app_id.appspot.com, according to the documentation.

• There is a new experimental feature Datastore Admin, after enabling it in app settings, you can bulk delete as well as backup your datastore through the web ui.

Solution 3

The fastest and efficient way to handle bulk delete on Datastore is by using the new mapper API announced on the latest Google I/O.

If your language of choice is Python, you just have to register your mapper in a mapreduce.yaml file and define a function like this:

from mapreduce import operation as op
def process(entity):
 yield op.db.Delete(entity)

On Java you should have a look to this article that suggests a function like this:

@Override
public void map(Key key, Entity value, Context context) {
    log.info("Adding key to deletion pool: " + key);
    DatastoreMutationPool mutationPool = this.getAppEngineContext(context)
            .getMutationPool();
    mutationPool.delete(value.getKey());
}

EDIT:
Since SDK 1.3.8, there's a Datastore admin feature for this purpose

Solution 4

You can clear the development server datastore when you run the server:

/path/to/dev_appserver.py --clear_datastore=yes myapp

You can also abbreviate --clear_datastore with -c.

Solution 5

If you have a significant amount of data, you need to use a script to delete it. You can use remote_api to clear the datastore from the client side in a straightforward manner, though.

View more solutions

Comments

[svrist]

There's a -c parameter to the dev_appserver.py to delete from the development datastore.

[Erin]

Actually, you don't need the fetch. Just db.delete(Entry.all()) will do it.

[marcc]

You need to do this in 500 entity sets or else you'll get: BadRequestError: cannot delete more than 500 entities in a single call

[mgiuca]

@svrist But that only applies to the Python app engine. Does anybody know how a shortcut for doing it in Java? (In the meantime, JohnIdol's suggestion works well.)

[Gady]

This only works for the development server. Is there a production equivalent?

[Zaffiro]

Just an FYI, for you to use the remote api you need to enable it in your application first using builtins:- remote_api: in your YAML file. more info is at developers.google.com/appengine/articles/remote_api

[Evan Plaice]

At least add the 'keys_only=True' when you call Entry.all(). There's no need to fetch the whole entry if you don't need to check the data. Else you're just wasting computing cycles.

[Juvenn Woo]

Thanks Evan, I've added keys_only=True as you suggested.

[Sundeep]

Adding the configuration in app.yaml threw an error. Instead we can enable it from the 'Applications Setting' Page in 'Administration' section. There's a button to enable it

[Melllvar]

Not sure if it's a recent thing, but the actual syntax is now /path/to/google_appengine/dev_appserver.py --clear_datastore yes myappname/ (note the 'yes')

[Uri]

+1 ... but: As of 2013, remote_shell_api.py doesn't exist. The current script name is remote_api_shell.py. Also, if you use ndb (which is what most people do these days), recommended way to use ndb.delete_multi(model.Entry.query().fetch(keys_only=True))

[Uri]

@Juvenn, one more question: Is there any way to automate remote_api_shell, so i can implement a script (instead of the interactive shell).

[Juvenn Woo]

@Uri Have a look at the remote_api_shell.py that come from the SDK, it could be a good demo of how should your script communicate with remote datastore. Sorry for the late reply, and I have not played with appengine much recently.

[xjq233p_1]

I don't think this will work. Appengine complains about Sorry, unexpected error: The kind "__Stat_Kind__" is reserved. This seems like appengine has some internal statistics entity that can be exposed by this method (possible bug on their end?)

[George Nguyen]

Thanks @John: Where the exact path in MAC OSX?

[Shane Best]

Where is the path in Windows?

[Josh J]

This isn't even valid php. import? Defining a constant as an object instance?

[morpheus]

@ShaneBest the path in windows is something like ./target/yourappid-1.0-SNAPSHOT/WEB-INF/appengine-generated/‌​local_db.bin

[Michael]

It is the most useful way of repeatedly deleting the datastore during development. With options getting obsolete fast, it worth to highlight this flag is still in place in july 2018, and works for dev_appserver installed via gcloud CLI

[franksands]

In version 270.0.0 of the Google Cloud SDK "--clear_datastore=yes" still works with the equal sign

[AAP]

As of May'22, I no longer see this option: