Are failed tasks resubmitted in Apache Spark?
13,871
Solution 1
I believe failed tasks are resubmitted because I have seen the same failed task submitted multiple times on the Web UI. However, if the same task fails multiple times, the full job fail:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 120 in stage 91.0 failed 4 times, most recent failure: Lost task 120.3 in stage 91.0
Solution 2
Yes, but there is a parameter set for the max number of failures
spark.task.maxFailures 4 Number of individual task failures before giving up on the job. Should be greater than or equal to 1. Number of allowed retries = this value - 1.
Related videos on Youtube
Author by
poiuytrez
Updated on September 09, 2022Comments
-
poiuytrez over 1 year
Are failed tasks automatically resubmitted in Apache Spark to the same or another executor?