reduceByKey method not being found in Scala Spark

13,183

You should import the implicit conversions from SparkContext:

import org.apache.spark.SparkContext._

They use the 'pimp up my library' pattern to add methods to RDD's of specific types. If curious, see SparkContext:1296

Share:
13,183

Related videos on Youtube

blue-sky
Author by

blue-sky

scala :: java

Updated on June 04, 2022

Comments

  • blue-sky
    blue-sky over 1 year

    Attempting to run http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala from source.

    This line:

    val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)
    

    is throwing error

    value reduceByKey is not a member of org.apache.spark.rdd.RDD[(String, Int)]
      val wordCounts = logData.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)
    

    logData.flatMap(line => line.split(" ")).map(word => (word, 1)) returns a MappedRDD but I cannot find this type in http://spark.apache.org/docs/0.9.1/api/core/index.html#org.apache.spark.rdd.RDD

    I'm running this code from Spark source so could be a classpath problem ? But required dependencies are on my classpath.

  • Ken Williams
    Ken Williams over 8 years
    Yay, saved me. Thanks.
  • botkop
    botkop over 8 years
    Upvoted. Why didn't IntelliJ figure this out for me ?
  • Amrish Pandey
    Amrish Pandey over 7 years
    this does not work for me. I am using Eclipse and following is my <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.3.0</version> </dependency>
  • WestCoastProjects
    WestCoastProjects over 6 years
    @AmrishPandey You are using spark 2.X: you need to do import org.apache.spark.SparkSession._