Spark.sql and sqlContext.sql
12,372
I spent two hours of my life in this one, just to realize I did not need:
sqlCtx = SQLContext(sc)
Just using SQLContext.read.(...)
, solved this in my case.
Author by
Ramsey
Updated on June 30, 2022Comments
-
Ramsey almost 2 years
I have imported the below modules. I tried to load data from sqlCtx.read.format, I am getting "IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"" error, but it works well when I use spark.read.format. I am seeing same behavior when I am retrieving data from registered temptable/view. What can I add extra to use sqlCtx.sql instead of spark.sql?
import os import sys import pandas as pd import odbc as pyodbc import os import sys import re from pyspark import SparkContext from pyspark.sql import SQLContext from pyspark.sql.types import * from pyspark.sql import Row from pyspark.sql.functions import * %matplotlib inline import matplotlib.pyplot as plt import pandas as pd import numpy as np import pyspark.sql.functions as func import matplotlib.patches as mpatches import time as time from matplotlib.patches import Rectangle import datetime from pyspark import SparkContext, SparkConf from pyspark.sql import SQLContext conf = SparkConf() conf.setMaster("local") conf.setAppName("AppName") sqlCtx = SQLContext(sc)
-
titipata about 7 yearsWhich version of Spark do you use? Maybe it will help others to solve this problem.
-
Ramsey about 7 yearsI am using Spark 2.1.0 and Python 2.7.
-
-
Ramsey about 7 yearsI have the same feeling, it worked when using SQLContext.read directly. Thanks Kieleth
-
Kieleth about 7 yearsAccept answer then? ;)
-
disp_name over 6 yearsSpent last 5 hours on this. You saved me. Cheers.