mongodb TTL not removing documents

Question

I have a simple schema like:

{
    _id: String,      // auto generated
    key: String,      // there is a unique index on this field
    timestamp: Date() // set to current time
}

Then I set the TTL index like so:

db.sess.ensureIndex( { "timestamp": 1 }, { expireAfterSeconds: 3600 } )

I expect the record to removed after 1 hour but it is never removed. I flipped on verbose logging and I see the TTLMonitor running:

Tue Sep 10 10:42:37.081 [TTLMonitor] TTL: { timestamp: 1.0 } { timestamp: { $lt: new Date(1378823557081) } }
Tue Sep 10 10:42:37.081 [TTLMonitor] TTL deleted: 0

When I run that query myself I see all my expired records coming back:

db.sess.find({ timestamp: { $lt: new Date(1378823557081) }})

...

Any ideas? I'm stumped.

EDIT - Example document below

{ "_id" : "3971446b45e640fdb30ebb3d58663807", "key" : "6XTHYKG7XBTQE9MJH8", "timestamp" : ISODate("2013-09-09T18:54:28Z") }

Solution 1

1. Can you show us what the inserted records actually look like?

2. How long is "never"? Because there's a big warning:

   Warning: The TTL index does not guarantee that expired data will be deleted immediately. There may be a delay between the time a document expires and the time that MongoDB removes the document from the database.

3. Does the timestamp field have an index already?

Solution 2

This was my issue: I had the index created wrong like this:

{
    "v" : 1,
    "key" : {
        "columnName" : 1,
        "expireAfterSeconds" : 172800
    },
    "name" : "columnName_1_expireAfterSeconds_172800",
    "ns" : "dbName.collectionName"
}

When it should have been this: (expireAfterSeconds is a top level propery)

{
    "v" : 1,
    "key" : {
        "columnName" : 1
    },
    "expireAfterSeconds" : 172800,
    "name" : "columnName_1_expireAfterSeconds_172800",
    "ns" : "dbName.collectionName"
}

Comments

[jckdnk111]

The TTLMonitor runs every 60 seconds so I presume it should be getting picked up in at least 1 minute of the expiration. By never I mean never (e.g. days pass and nothing is ever deleted). There is only 2 indexes - the TTL index and the unique index I pointed out on the key field. Example document in the edit. Thanks for your help.

[jcollum]

No I'm wondering what you see when you do db.sess.find().pretty() -- I just want to confirm the timestamp field is there and doesn't look funny. For the indexes, you should do db.sess.getIndexes() to confirm that's all the indexes that are there.

[jckdnk111]

db.sess.getIndexes() showed me the error of my ways. I was using db['system.indexes'].find({expireAfterSeconds: {$ne: null}}) -- I had a typo in the collection name while creating the index and I didn't notice it in the output of my system.indexes query. Thanks so much for your help!

[huggie]

I'm using Node.js mongoose and it seems sometimes indexing just doesn't get through. No code changes. Just repeating executions and it has indexing sometimes and sometimes not. Weird.

[jcollum]

@huggie you should post that as a separate question

[huggie]

@jcollum: I don't mean it as a question. I just want people to watch out for it.