Null check for Double/Int Value in Spark

Question

I am new in Spark, How can I check for for Null value in Double and Int value in scala or Spark.

Like for String We can do like this :

val value = (FirstString.isEmpty()) match {
          case true => SecondString
          case _    => FirstString
        }

I searched for it a lot but i found only for String value. Can you please suggest me for other datatype as well.

Thanks in advance.

Solution 1

May be you can simply use Options. So like:

val d: Double = ...

val isNull = Option(d).isDefined

Or you can use pattern matching:

val d: Double = ...

Option(d) match {
  case Some(v) => use v
  case _ => you got Null
}

Solution 2

null is only applicable for AnyRef (i.e non primitive types) types in Scala. AnyVal types can not be set to null.

For example:

// the below are AnyVal(s) and wont compile
val c: Char = null
val i: Int = null
val d: Double = null  

String is an AnyRef and so can be null:

// this is ok!
val c: String = null 

That's why pattern matching nulls to Int/Double types is not possible:

// wont compile!
null match {
        case a:Int => "is a null Int"
        case _ => "something else"
        }

Solution 3

isEmpty is not at all the same as "check for null". Calling isEmpty on null will fail:

val s: String = null
s.isEmpty // throws NullPointerException

Int and Double can't be null (neither can any other primitive types), so there is no need to check if they are. If you are talking specifically about Spark Rows, you need to check for null before getting an Int/Double/other primitive value:

It is invalid to use the native primitive interface to retrieve a value that is null, instead a user must check isNullAt before attempting to retrieve a value that might be null.

Comments

[Alexey Romanov]

This is pointless: because a Double can't be null, Option(d) will always be Some. If the code in ... doesn't actually evaluate to a Double, but to java.lang.Double (which can be null), then you'll get an exception on that line.

[Khoa]

use Try{} instead