Spark - Scala - Number of days between two dates

Question

I'm using spark 1.3.

I've a data frame and I need to calculate the number of days between a given date and current date. I'm trying to calculate this as part of a select aggregate as below. Below code with datediff function is not working.

val testdate = Inputdata.selectExpr("id",
                                    "amt",
                                    "substr(TranDt,1,4) as TranYear", 
                                    "datediff(current_date(), TranDt) as numofdays")

Any help is really appreciated.

Solution 1

Datediff() function introduced with spark 1.5.0, since you are using 1.3, that is the reason your script is not working. Update to 1.5.0 to make this work.

Solution 2

If you are unable to upgrade your version of Spark, you could extract and map over your results, parsing the dates using something like Joda time to work out the number of days between the two.

Days.daysBetween(firstdate, seconddate).getDays()

Comments

[Reactormonk]

What do you mean by "not working"?

[user5692277]

What I'm looking is number of days between two dates. the job is not running and displays below error message "Exception in thread "main" java.util.NoSuchElementException: key not found: datediff" . I think datediff and current_date() functions are not working. Is there a way around to calculate the number of days?.