Difference when serializing a lazy val with or without @transient
see here - http://fdahms.com/2015/10/14/scala-and-the-transient-lazy-val-pattern/
In Scala lazy val denotes a field that will only be calculated once it is accessed for the first time and is then stored for future reference. With @transient on the other hand one can denote a field that shall not be serialized.
Related videos on Youtube
![Hao Ren](https://lh3.googleusercontent.com/-OetwSibo1hQ/AAAAAAAAAAI/AAAAAAAAABA/-AJAnsCYR08/photo.jpg?sz=256)
Comments
-
Hao Ren about 4 years
Working on spark, sometimes I need to send a non-serializable object in each task.
A common pattern is
@transient lazy val
, e.gclass A(val a: Int) def compute(rdd: RDD[Int]) = { // lazy val instance = { @transient lazy val instance = { println("in lazy object") new A(1) } val res = rdd.map(instance.a + _).count() println(res) } compute(sc.makeRDD(1 to 100, 8))
I found that
@transient
is not necessary here.lazy val
can already create the non-serializable upon each task is executed. But people suggest using@transient
.What is the advantage, if we set
@transient
on the non-initializedlazy val
when serializing it ?Does it make sense to make a non-initialized
val
transient for serialization, knowing that nothing will be serialized, just like in the example above ?How is a
@transient lazy val
serialized ? Is it treated as a method or something else ?
Some details on serializing
@transient lazy val
and the compiled java bytecode is awesome.