You need to build Spark before running this program error when running bin/pyspark
12,485
If you want to use source distribution you have to build Spark first. You can find details in the Building Spark.
Otherwise just download a binary distribution (select pre-built) from downloads:
Also don't use outdated documentation. You can use either the latest:
http://spark.apache.org/docs/latest
or version specific
http://spark.apache.org/docs/1.5.0
Related videos on Youtube
Author by
Soundarya Thiagarajan
Updated on June 04, 2022Comments
-
Soundarya Thiagarajan over 1 year
I am getting started with Spark.
I am getting an issue when starting spark.
I downloaded from spark official website, I am trying to quickstart from this https://spark.apache.org/docs/0.9.0/quick-start.html
I have downloaded, setup sparkhome., and when I tried running ./pyspark from bin directory I get this error :
soundarya@soundarya:~/Downloads/spark-1.5.0/bin$ sudo ./pyspark [sudo] password for soundarya: ls: cannot access /home/soundarya/Downloads/spark-1.5.0/assembly/target/scala-2.10: No such file or directory Failed to find Spark assembly in /home/soundarya/Downloads/spark-1.5.0/assembly/target/scala-2.10. You need to build Spark before running this program. soundarya@soundarya:~/Downloads/spark-1.5.0/bin$
Can anyone help me to figure out this?
-
Soundarya Thiagarajan almost 8 yearsHi, thanks for your reply. I tried running it. 15/11/09 14:28:03 ERROR NettyTransport: failed to bind to /192.168.0.109:0, shutting down Netty transport 15/11/09 14:28:03 ERROR SparkContext: Error initializing SparkContext. java.net.BindException: Failed to bind to: /192.168.0.109:0: Service 'sparkDriver' failed after 16 retries! py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.net.BindException: Failed to bind to: /192.168.0.109:0: Service 'sparkDriver' failed after 16 retries!
-
Soundarya Thiagarajan almost 8 yearsI downloaded spark-1.5.0-bin-hadoop2.4/bin
-
AkhlD almost 8 yearsWhats in your conf/spark-env.sh file? Did you try opening up a spark-shell first? (bin/spark-shell)
-
Soundarya Thiagarajan almost 8 yearsWhen I tried running spark-shell I got error : OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000bff80000, 716177408, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 716177408 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/soundarya/Downloads/spark-1.5.0-bin-hadoop2.4/bin/hs_err_pid16684.log soundarya@soundarya:~/Downloads/spark-1.5.0-bin-hadoop2.4/bin$
-
AkhlD almost 8 yearsthat's because you are not having sufficient memory on your machine. What is executor memory/driver memory that you are allocating? and how much memory are you having on your machine?
-
Soundarya Thiagarajan almost 8 yearsHi, I tried downloading as suggested : spark-1.5.0-bin-hadoop2.4 (prebuild version)..but still i m held up with errors.. OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007d5500000, 716177408, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 716177408 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/soundarya/Downloads/spark-1.5.0/hs_err_pid18440.log
-
AkhlD almost 8 yearsHow much memory do you have on your machine? Its a JVM Heap error and has nothing to do with spark. You can try setting the spark.driver.memory and spark.executor.memory in the conf/spark-defaults.conf file. Or simply do a export _JAVA_OPTIONS=-Xmx1g from your terminal.
-
Soundarya Thiagarajan almost 8 yearsI did export _JAVA_OPTIONS=-Xmx1g ; Picked up _JAVA_OPTIONS: -Xmx1g raise Exception("Java gateway process exited before sending the driver its port number") Exception: Java gateway process exited before sending the driver its port number >>> # There is insufficient memory for the Java Runtime Environment to continue.
-
AkhlD almost 8 years"There is insufficient memory for the Java Runtime Environment to continue. "- How much memory are you having on your machine?
-
Soundarya Thiagarajan almost 8 yearsHi AkhID, I have posted it as seperate answer.