Best way to identify and extract dates from text Python?

Question

As part of a larger personal project I'm working on, I'm attempting to separate out inline dates from a variety of text sources.

For example, I have a large list of strings (that usually take the form of English sentences or statements) that take a variety of forms:

Central design committee session Tuesday 10/22 6:30 pm

Th 9/19 LAB: Serial encoding (Section 2.2)

There will be another one on December 15th for those who are unable to make it today.

Workbook 3 (Minimum Wage): due Wednesday 9/18 11:59pm

He will be flying in Sept. 15th.

While these dates are in-line with natural text, none of them are in specifically natural language forms themselves (e.g., there's no "The meeting will be two weeks from tomorrow"—it's all explicit).

As someone who doesn't have too much experience with this kind of processing, what would be the best place to begin? I've looked into things like the dateutil.parser module and parsedatetime, but those seem to be for after you've isolated the date.

Because of this, is there any good way to extract the date and the extraneous text

input:  Th 9/19 LAB: Serial encoding (Section 2.2)
output: ['Th 9/19', 'LAB: Serial encoding (Section 2.2)']

or something similar? It seems like this sort of processing is done by applications like Gmail and Apple Mail, but is it possible to implement in Python?

Solution 1

I was also looking for a solution to this and couldn't find any, so a friend and I built a tool to do this. I thought I would come back and share incase others found it helpful.

datefinder -- find and extract dates inside text

Here's an example:

import datefinder

string_with_dates = '''
    Central design committee session Tuesday 10/22 6:30 pm
    Th 9/19 LAB: Serial encoding (Section 2.2)
    There will be another one on December 15th for those who are unable to make it today.
    Workbook 3 (Minimum Wage): due Wednesday 9/18 11:59pm
    He will be flying in Sept. 15th.
    We expect to deliver this between late 2021 and early 2022.
'''

matches = datefinder.find_dates(string_with_dates)
for match in matches:
    print(match)

Solution 2

I am surprised that there is no mention of SUTime and dateparser's search_dates method.

from sutime import SUTime
import os
import json
from dateparser.search import search_dates

str1 = "Let's meet sometime next Thursday" 

# You'll get more information about these jar files from SUTime's github page
jar_files = os.path.join(os.path.dirname(__file__), 'jars')
sutime = SUTime(jars=jar_files, mark_time_ranges=True)

print(json.dumps(sutime.parse(str1), sort_keys=True, indent=4))
"""output: 
[
    {
        "end": 33,
        "start": 20,
        "text": "next Thursday",
        "type": "DATE",
        "value": "2018-10-11"
    }
]
"""

print(search_dates(str1))
#output:
#[('Thursday', datetime.datetime(2018, 9, 27, 0, 0))]

Although I have tried other modules like dateutil, datefinder and natty (couldn't get duckling to work with python), this two seem to give the most promising results.

The results from SUTime are more reliable and it's clear from the above code snippet. However, the SUTime fails in some basic scenarios like parsing a text

"I won't be available until 9/19"

or

"I won't be available between (September 18-September 20).

It gives no result for the first text and only gives month and year for the second text. This is however handled quite well in the search_dates method. search_dates method is more aggressive and will give all possible dates related to any words in the input text.

I haven't yet found a way to parse the text strictly for dates in search_methods. If I could find a way to do that, it'll be my first choice over SUTime and I would also make sure to update this answer if I find it.

Solution 3

If you can identify the segments that actually contain the date information, parsing them can be fairly simple with parsedatetime. There are a few things to consider though namely that your dates don't have years and you should pick a locale.

>>> import parsedatetime
>>> p = parsedatetime.Calendar()
>>> p.parse("December 15th")
((2013, 12, 15, 0, 13, 30, 4, 319, 0), 1)
>>> p.parse("9/18 11:59 pm")
((2014, 9, 18, 23, 59, 0, 4, 319, 0), 3)
>>> # It chooses 2014 since that's the *next* occurence of 9/18

It doesn't always work perfectly when you have extraneous text.

>>> p.parse("9/19 LAB: Serial encoding")
((2014, 9, 19, 0, 15, 30, 4, 319, 0), 1)
>>> p.parse("9/19 LAB: Serial encoding (Section 2.2)")
((2014, 2, 2, 0, 15, 32, 4, 319, 0), 1)

Honestly, this seems like the kind of problem that would be simple enough to parse for particular formats and pick the most likely out of each sentence. Beyond that, it would be a decent machine learning problem.

Solution 4

You can use the dateutil module's parse method with the fuzzy option.

>>> from dateutil.parser import parse
>>> parse("Central design committee session Tuesday 10/22 6:30 pm", fuzzy=True)
datetime.datetime(2018, 10, 22, 18, 30)
>>> parse("There will be another one on December 15th for those who are unable to make it today.", fuzzy=True)
datetime.datetime(2018, 12, 15, 0, 0)
>>> parse("Workbook 3 (Minimum Wage): due Wednesday 9/18 11:59pm", fuzzy=True)
datetime.datetime(2018, 3, 9, 23, 59)
>>> parse("He will be flying in Sept. 15th.", fuzzy=True)
datetime.datetime(2018, 9, 15, 0, 0)
>>> parse("Th 9/19 LAB: Serial encoding (Section 2.2)", fuzzy=True)
datetime.datetime(2002, 9, 19, 0, 0)

Solution 5

Newer versions of parsedatetime lib provide search functionality.

Example

from dateparser.search import search_dates

dates = search_dates('Central design committee session Tuesday 10/22 6:30 pm')

View more solutions

Comments

[redct]

I guess a better question for me to ask is: what's the best way to automatically identify the segments? Is there some sort of method (besides giant regexes, I guess) to identify the date substring?

[Dylan]

Really great tool, it accounts for a large variety of cases! Great work.

[LGG]

it is strange but this is what is happening. I am using doing string_with_dates = """ ...: ... ...: entries are due by January 4th, 2017 at 8:00pm ...: ... ...: created 01/15/2005 by ACME Inc. and associates.""" matches = datefinder.find_dates(string_with_dates,source=True). the first time I access the matches generator I am able to print results. the next time I cant print it. I am using 3.4 with syntax: for m in matches: print(m)

[Somnath Kadam]

is it possible to get original string list? example: "entries are due by January 4th, 2017 at 8:00pm", and if I run like : matches = datefinder.find_dates(text) will return ['January 4th, 2017 at 8:00pm']

[akoumjian]

@SomnathKadam Yes: github.com/akoumjian/datefinder/blob/master/datefinder.py#L2‌​84

[user3709260]

@akoumjian I'm trying to install your package and I get this error: failed building wheel for regex. And then it says: running setup.py clean for regex and eventually a lot of other errors ending with "Rolling back uninstall of regex" and a long red line. Have other people reported installation errors like this? I appreciate any recommendations. Thanks

[blitu12345]

@akoumjian thanks for such a great tool! But I want to know whether we can get the position or index of date token in the actual text?

[gented]

In correspondence of a string like -039_apple the parser returns 2039-12-12 00:00:00: any reason why it does so? Likewise in correspondence of other hyphenation/punctuations present in the text.

[akoumjian]

@blitu12345 Please see the documentation that's part of the basic api: datefinder.readthedocs.io/en/latest

[akoumjian]

@user3709260 that should be fixed now

[akoumjian]

@gented The capture model for datefinder is going to create some false positives. Essentially, it casts a wide net over possible date interpretations, sends a possible match to dateutil, and then returns whatever dateutil says it is.

[Hojung Kim]

sutime does seem to be the most powerful, especially after the Stanford CoreNLP team has had even more time to develop and improve it. However, I'm having some trouble and getting an Import Error when importing sutime. I posted more at length about it here, if you happen to have encountered a similar error: stackoverflow.com/questions/59743652/…

[LazyCat]

great package, thanks! Is it possible to add timezone recognition, or am I asking for too much? :)

[Talib Daryabi]

I don't know why but it can't detect date from my string '''other text Notice :Download Procedure: Open Closing date: 14/04/2021''

[Gaurav Koradiya]

Great work. It solve problem up to good extent.

[Parvez Khan]

It's not detecting any type of date. For example, I tried on "This is sample date 10-10-2021" and the library was unable to extract the date from it.

[QuentinJS]

lib is actually dateparser. I fed it a simple date and it took 5 seconds to process, but on a string with an embedded date, it did findthe date where parser.parse(t) did not. So I use parse(t, fuzzy=True) first and then search_dates if it fails.