Delete firebase data older than 2 hours

javascript angularjs firebase firebase-realtime-database

42,812

Solution 1

Firebase does not support queries with a dynamic parameter, such as "two hours ago". It can however execute a query for a specific value, such as "after August 14 2015, 7:27:32 AM".

That means that you can run a snippet of code periodically to clean up items that are older than 2 hours at that time:

var ref = firebase.database().ref('/path/to/items/');
var now = Date.now();
var cutoff = now - 2 * 60 * 60 * 1000;
var old = ref.orderByChild('timestamp').endAt(cutoff).limitToLast(1);
var listener = old.on('child_added', function(snapshot) {
    snapshot.ref.remove();
});

As you'll note I use child_added instead of value, and I limitToLast(1). As I delete each child, Firebase will fire a child_added for the new "last" item until there are no more items after the cutoff point.

Update: if you want to run this code in Cloud Functions for Firebase:

exports.deleteOldItems = functions.database.ref('/path/to/items/{pushId}')
.onWrite((change, context) => {
  var ref = change.after.ref.parent; // reference to the items
  var now = Date.now();
  var cutoff = now - 2 * 60 * 60 * 1000;
  var oldItemsQuery = ref.orderByChild('timestamp').endAt(cutoff);
  return oldItemsQuery.once('value', function(snapshot) {
    // create a map with all children that need to be removed
    var updates = {};
    snapshot.forEach(function(child) {
      updates[child.key] = null
    });
    // execute all updates in one go and return the result to end the function
    return ref.update(updates);
  });
});

This function triggers whenever data is written under /path/to/items, so child nodes will only be deleted when data is being modified.

This code is now also available in the functions-samples repo.

Solution 2

I have a http triggered cloud function that deletes nodes, depending on when they were created and their expiration date.

When I add a node to the database, it needs two fields: timestamp to know when it was created, and duration to know when the offer must expire.

Then, I have this http triggered cloud function:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();

/**
 * @function HTTP trigger that, when triggered by a request, checks every message of the database to delete the expired ones.
 * @type {HttpsFunction}
 */
exports.removeOldMessages = functions.https.onRequest((req, res) => {
    const timeNow = Date.now();
    const messagesRef = admin.database().ref('/messages');
    messagesRef.once('value', (snapshot) => {
        snapshot.forEach((child) => {
            if ((Number(child.val()['timestamp']) + Number(child.val()['duration'])) <= timeNow) {
                child.ref.set(null);
            }
        });
    });
    return res.status(200).end();
});

You can create a cron job that every X minutes makes a request to the URL of that function: https://cron-job.org/en/

But I prefer to run my own script, that makes a request every 10 seconds:

watch -n10 curl -X GET https://(your-zone)-(your-project-id).cloudfunctions.net/removeOldMessages

Solution 3

In the latest version of Firebase API, ref() is changed to ref

var ref = new Firebase('https://yours.firebaseio.com/path/to/items/');
var now = Date.now();
var cutoff = now - 2 * 60 * 60 * 1000;
var old = ref.orderByChild('timestamp').endAt(cutoff).limitToLast(1);
var listener = old.on('child_added', function(snapshot) {
    snapshot.ref.remove();
});

Solution 4

If someone will have the same problem, but in Firestore. I did a little script that at first read documents to console.log and then delete documents from a collection messages older than 24h. Using https://cron-job.org/en/ to refresh website every 24h and that's it. Code is below.

var yesterday = firebase.firestore.Timestamp.now();
  yesterday.seconds = yesterday.seconds - (24 * 60 * 60);
  console.log("Test");
  db.collection("messages").where("date",">",yesterday)
      .get().then(function(querySnapshote) {
        querySnapshote.forEach(function(doc) {
          console.log(doc.id," => ",doc.data());
        });
      })
  .catch(function(error) {
        console.log("Error getting documents: ", error);
  });

  db.collection("messages").where("date","<",yesterday)
    .get().then(function(querySnapshote) {
      querySnapshote.forEach(element => {
        element.ref.delete();
      });
    })

Solution 5

You could look into Scheduling Firebase Functions with Cron Jobs. That link shows you how to schedule a Firebase Cloud Function to run at a fixed rate. In the scheduled Firebase Function you could use the other answers in this thread to query for old data and remove it.

View more solutions

42,812

Author by

carterw485

Updated on September 15, 2020

Comments

carterw485 over 3 years

I would like to delete data that is older than two hours. Currently, on the client-side, I loop through all the data and run a delete on the outdated data. When I do this, the db.on('value') function is invoked every time something is deleted. Also, things will only be deleted when a client connects, and what might happen if two clients connect at once?

Where can I set up something that deletes old data? I have a timestamp inside each object created by a JavaScript Date.now().
0xCrema.eth about 8 years

In your exemple you fetch the records saved 2 hours ago or less. Maybe you should use endAt instead of startAt. But still thank you for your answer it helped me
Frank van Puffelen almost 8 years

Good catch! Fixed.
Abhi over 7 years

Hi Frank, I am attempting to do something similar in Swift however I can't seem to figure out what method you are calling in the listener. I've spent a few hours searching and trying but decided it was time to ask for help!
Frank van Puffelen over 7 years

snapshot.ref().remove() on Swift with a 3.x SDK would be snapshot.ref.removeValue(). See the guide and reference docs.
Abhi over 7 years

Thank you, I had everything right it seems I made the novice blunder of not checking my reference paths!
Coder1000 over 7 years

Hello, I am trying to implement your solution to periodically delete a pending users. Could you please have a look at my question ? stackoverflow.com/questions/41696708/…
Admin almost 7 years

That's exactly what I need. But, I need it in a format suitable for Cloud Functions. Thanks, Frank.
Admin almost 7 years

In other words, I need to put the same logic in an index.js file.
Admin almost 7 years

Thanks for the edit. But I'm getting this error when I firebase deploy : ! functions[deleteOldItems]: Deploy Error: Failed to configure trigger deleteOldItems (Firebase Realtime Database) Functions deploy had errors. To continue deploying other features (such as database), run: firebase deploy --except functions Error: Functions did not deploy properly.
Admin almost 7 years

To make things clearer, I asked the question here: stackoverflow.com/questions/44622777/…
Frank van Puffelen almost 7 years

There is likely a syntax error in my code somewhere. Don't start yet another new question for that. Instead help find the syntax error and fix it here.
Admin almost 7 years

I can't find the syntax error, I apologise for writing a new question.
Admin almost 7 years

Also, I'd like to say that in my new question, I corrected a few mistakes. But, it's still not working.
Frank van Puffelen almost 7 years

If you found issues with the code in this answer, make an edit to the answer. Opening a new question for this code just means we have two places to scan.
Frank van Puffelen almost 7 years

I made a few more fixed to the code after testing it. The first deploy failed, but that seems like it was an ephemeral problem in Cloud Functions (which is still in beta). After a redeploy this works without problems.
Admin almost 7 years

I get the following event message with your code: Function execution took 60002 ms, finished with status: 'timeout'
Admin almost 7 years

Is this event message normal?
Admin almost 7 years

Thanks for the Cloud Functions edit by the way! Really useful.
Dani Kemper over 6 years

When i implemented this, all the data was removed after 1 sec instead of 2 hours. I have copied and paste exactly the code, so i don't whats going on?
vikrantnegi over 6 years

@DaniKemper I'm having the same issue. It seems that data is getting deleted as soon as it is added not after 2 hours. Any solution to this?
vikrantnegi about 6 years

@FrankvanPuffelen I'm having an issue where it seems that data is getting deleted as soon as it is added not after 2 hours. Any idea on why this is happening?
SiddAjmera about 6 years

@FrankvanPuffelen, I also wanted to delete the GeoFire indices that I had in another list. How can I delete them along with the items in the current list?
SiddAjmera about 6 years

Got it. @FrankvanPuffelen, I've updated the answer accordingly. Please update if you feel there's a need to.
Frank van Puffelen about 6 years

Good to hear that you could expand the code from my answer to also delete data from other sources in your use-case @SiddharthAjmera. Since the original question was not about that, I rolled back the edit however. While it was a useful change for you, this (short) piece of code is already complex enough without that addition.
Simon over 5 years

Note for anyone uploading a timestamp from iOS: timeIntervalSince1970 is in seconds and Date.now() is millis so make sure to convert as necessary
steveSarsawa almost 5 years

I need sollution in swift 4 ,any idea!
devDeejay over 4 years

Thanks for cron-job! Life saver!
luke cross almost 4 years

Timestamp is long number (on firebase) or is Timestamp type?
luke cross almost 4 years

cronjob is free?