Spark 2.3 Dropping Temp Table

Question

I'm trying to use dropTempTable() after the respective temporary table's use is over (to free up the memory for next calculations).

Newer Spark Session doesn't need sqlContext, so, it is confusing me on how to use the function.

1) Tried, same DF which I used to register a temp table to do -

DF.dropTempTable('xyz')

Didn't work.

2) Tried following way too, as spark internally invokes sqlContext too along with sparkContext, but didn't work -

spark.dropTempTable('xyz')

3) Tried spark.catalog to drop, this failed too -

spark.catalog.dropTempTable('xyz')

What to do? 1.6 examples on internet are not working in the 2.3 version for dropTempTable().

Any help?

Solution

For spark 2.0 the APIs are

For temporary views spark.catalog.dropTempView("df")

For global views spark.catalog.dropGlobalTempView("df")

From the Documentation

abstract def dropGlobalTempView(viewName: String): Boolean

Drops the global temporary view with the given view name in the catalog. If the view has been cached before, then it will also be uncached.

Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM global_temp.view1.

viewName the unqualified name of the temporary view to be dropped.

returns true if the view is dropped successfully, false otherwise.

abstract def dropTempView(viewName: String): Boolean

Drops the local temporary view with the given view name in the catalog. If the view has been cached before, then it will also be uncached.

Local temporary view is session-scoped. Its lifetime is the lifetime of the session that created it, i.e. it will be automatically dropped when the session terminates. It's not tied to any databases, i.e. we can't use db1.view1 to reference a local temporary view.

Note that, the return type of this method was Unit in Spark 2.0, but changed to Boolean in Spark 2.1.

viewName the name of the temporary view to be dropped.

returns true if the view is dropped successfully, false otherwise.