Error: not found: value lit/when - spark scala
17,923
Let's consider the following context :
val spark : SparkSession = _ // or val sqlContext: SQLContext = new SQLContext(sc) for 1.x
val list: DataFrame = ???
To use when
and lit
, you'll need to import the proper functions :
import org.apache.spark.sql.functions.{col, lit, when}
Now you can use them as followed :
list.select(when(col("column_name").isNotNull, lit(1)))
Now you can use lit also in your code :
val score = list.withColumn("scr", lit(0))
Author by
Avijit
Updated on June 25, 2022Comments
-
Avijit almost 2 years
I am using scala, spark, IntelliJ and maven.
I have used below code :
val joinCondition = when($"exp.fnal_expr_dt" >= $"exp.nonfnal_expr_dt", $"exp.manr_cd"===$"score.MANR_CD") val score = exprDF.as("exp").join(scoreDF.as("score"),joinCondition,"inner")
and
val score= list.withColumn("scr", lit(0))
But when try to build using maven, getting below errors -
error: not found: value when
and
error: not found: value lit
For
$
and===
I have usedimport sqlContext.implicits.StringToColumn
and it is working fine. No error occurred at the time of maven build.But forlit(0)
andwhen
what I need to import or is there any other way resolve the issue.