How to get tables registered as spark table into data frame

Question

I have imported tables from PostgreSQL database into spark-sql using spark-thriftserver jdbc connection and now from beeline I can see these tables.

Is there any way I can convert these tables into spark data frame.

Solution 1

This works with Spark > 2.0: df = spark.table('table')

Solution 2

spark 2.0.0 http://spark.apache.org/docs/latest/sql-programming-guide.html

The sql function on a SparkSession enables applications to run SQL queries programmatically and returns the result as a DataFrame.

# spark is an existing SparkSession
df = spark.sql("SELECT * FROM table")

spark 1.6.2 http://spark.apache.org/docs/1.6.2/sql-programming-guide.html

Running SQL Queries Programmatically

The sql function on a SQLContext enables applications to run SQL queries programmatically and returns the result as a DataFrame.

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
df = sqlContext.sql("SELECT * FROM table")