You need to build Spark before running this program error when running bin/pyspark

12,485

If you want to use source distribution you have to build Spark first. You can find details in the Building Spark.

Otherwise just download a binary distribution (select pre-built) from downloads:

enter image description here

Also don't use outdated documentation. You can use either the latest:

http://spark.apache.org/docs/latest

or version specific

http://spark.apache.org/docs/1.5.0

Share:
12,485

Related videos on Youtube

Soundarya Thiagarajan
Author by

Soundarya Thiagarajan

Updated on June 04, 2022

Comments

  • Soundarya Thiagarajan
    Soundarya Thiagarajan over 1 year

    I am getting started with Spark.

    I am getting an issue when starting spark.

    I downloaded from spark official website, I am trying to quickstart from this https://spark.apache.org/docs/0.9.0/quick-start.html

    I have downloaded, setup sparkhome., and when I tried running ./pyspark from bin directory I get this error :

    soundarya@soundarya:~/Downloads/spark-1.5.0/bin$ sudo ./pyspark
    [sudo] password for soundarya: 
    ls: cannot access /home/soundarya/Downloads/spark-1.5.0/assembly/target/scala-2.10: No such file or directory
    Failed to find Spark assembly in /home/soundarya/Downloads/spark-1.5.0/assembly/target/scala-2.10.
    You need to build Spark before running this program.
    soundarya@soundarya:~/Downloads/spark-1.5.0/bin$ 
    

    Can anyone help me to figure out this?

  • Soundarya Thiagarajan
    Soundarya Thiagarajan almost 8 years
    Hi, thanks for your reply. I tried running it. 15/11/09 14:28:03 ERROR NettyTransport: failed to bind to /192.168.0.109:0, shutting down Netty transport 15/11/09 14:28:03 ERROR SparkContext: Error initializing SparkContext. java.net.BindException: Failed to bind to: /192.168.0.109:0: Service 'sparkDriver' failed after 16 retries! py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.net.BindException: Failed to bind to: /192.168.0.109:0: Service 'sparkDriver' failed after 16 retries!
  • Soundarya Thiagarajan
    Soundarya Thiagarajan almost 8 years
    I downloaded spark-1.5.0-bin-hadoop2.4/bin
  • AkhlD
    AkhlD almost 8 years
    Whats in your conf/spark-env.sh file? Did you try opening up a spark-shell first? (bin/spark-shell)
  • Soundarya Thiagarajan
    Soundarya Thiagarajan almost 8 years
    When I tried running spark-shell I got error : OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000bff80000, 716177408, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 716177408 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/soundarya/Downloads/spark-1.5.0-bin-hadoop2.4/bin/hs_e‌​rr_pid16684.log soundarya@soundarya:~/Downloads/spark-1.5.0-bin-hadoop2.4/bi‌​n$
  • AkhlD
    AkhlD almost 8 years
    that's because you are not having sufficient memory on your machine. What is executor memory/driver memory that you are allocating? and how much memory are you having on your machine?
  • Soundarya Thiagarajan
    Soundarya Thiagarajan almost 8 years
    Hi, I tried downloading as suggested : spark-1.5.0-bin-hadoop2.4 (prebuild version)..but still i m held up with errors.. OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007d5500000, 716177408, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 716177408 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/soundarya/Downloads/spark-1.5.0/hs_err_pid18440.log
  • AkhlD
    AkhlD almost 8 years
    How much memory do you have on your machine? Its a JVM Heap error and has nothing to do with spark. You can try setting the spark.driver.memory and spark.executor.memory in the conf/spark-defaults.conf file. Or simply do a export _JAVA_OPTIONS=-Xmx1g from your terminal.
  • Soundarya Thiagarajan
    Soundarya Thiagarajan almost 8 years
    I did export _JAVA_OPTIONS=-Xmx1g ; Picked up _JAVA_OPTIONS: -Xmx1g raise Exception("Java gateway process exited before sending the driver its port number") Exception: Java gateway process exited before sending the driver its port number >>> # There is insufficient memory for the Java Runtime Environment to continue.
  • AkhlD
    AkhlD almost 8 years
    "There is insufficient memory for the Java Runtime Environment to continue. "- How much memory are you having on your machine?
  • Soundarya Thiagarajan
    Soundarya Thiagarajan almost 8 years
    Hi AkhID, I have posted it as seperate answer.