How to apply map function on dataset in spark java

Question

My CSV File:

YEAR,UTILITY_ID,UTILITY_NAME,OWNERSHIP,STATE_CODE,AMR_METERING_RESIDENTIAL,AMR_METERING_COMMERCIAL,AMR_METERING_INDUSTRIAL,AMR_METERING_TRANS,AMR_METERING_TOTAL,AMI_METERING_RESIDENTIAL,AMI_METERING_COMMERCIAL,AMI_METERING_INDUSTRIAL,AMI_METERING_TRANS,AMI_METERING_TOTAL,ENERGY_SERVED_RESIDENTIAL,ENERGY_SERVED_COMMERCIAL,ENERGY_SERVED_INDUSTRIAL,ENERGY_SERVED_TRANS,ENERGY_SERVED_TOTAL
2011,34,City of Abbeville - (SC),M,SC,880,14,,,894,,,,,,,,,,
2011,84,A & N Electric Coop,C,MD,135,25,,,160,,,,,,,,,,
2011,84,A & N Electric Coop,C,VA,31893,2107,0,,34000,,,,,,,,,,
2011,97,Adams Electric Coop,C,IL,8334,190,,,8524,,,,,0,,,,,0
2011,108,Adams-Columbia Electric Coop,C,WI,33524,1788,709,,36021,,,,,,,,,,
2011,118,Adams Rural Electric Coop, Inc,C,OH,7457,20,,,7477,,,,,,,,,,
2011,122,Village of Arcade,M,NY,3560,498,100,,4158,,,,,,,,,,
2011,155,Agralite Electric Coop,C,MN,4383,227,315,,4925,,,,,,,,,,

Here down the Spark code to read the CSV file:

public class ReadFile8 {

    public static void main(String[] args) throws IOException {

        SparkSession session = new SparkSession.Builder().appName("CsvReader").master("local").getOrCreate();

        //Data taken by Local System
        Dataset<Row> file8Data = session.read().format("com.databricks.spark.csv").option("header", "true").load("file:///home/kumar/Desktop/Eletricaldata/file8_2011.csv");

        // Register the DataFrame as a SQL temporary view
        file8Data.createOrReplaceTempView("EletricalFile8Data");
        file8Data.show();
    }

}

How can apply a map function and flatmap function in Spark using Java?

Solution

You can use the following code as an example:

Dataset<Integer> years = file8Data.map((MapFunction<Row, Integer>) row -> row.<Integer>getAs("YEAR"), Encoders.INT());
Dataset<Integer> newYears = years.flatMap((FlatMapFunction<Integer, Integer>) year -> {
  return Arrays.asList(year + 1, year + 2).iterator();
}, Encoders.INT());

Comments

[kumar]

any metirial sparkjava from you please send me. my email. my email is :[email protected]

[kumar]

not working this code.error is comming :java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer

[Anton Okolnychyi]

This means that Spark treats the "YEAR" column as String. Set inferSchema to true and Spark will try to infer column types or replace Integer with String in the code above.

[Anton Okolnychyi]

I did not have any specific materials. I would recommend the docs, where you can select the language of examples. In addition, this should be helpful.

[kumar]

in doc meterial ,all functions not provided,and just few sample example are provided,how to use all functions for code.

[BdEngineer]

@AntonOkolnychyi can you please provide some clue how to do this stackoverflow.com/questions/57479581/…