Implement job restart policies

[razvan]

Description

Some Spark jobs are idempotent and can be resubmitted without problems especially if they fail due to external constraints (like endpoints not being ready and such)

The user should be able to configure such jobs with a restart policy.

[sbernauer]

At my previous company i simply put the Spark driver Pod within a Deployment and health-checked port 4040.
Worked pretty well

[razvan]

That requires spark.ui to be true right? Anyway, establishing the status of the job (for batch jobs) is not the problem (it's currently solved). The user has to specify whether restarting the job is safe or not.

For streaming jobs, we might want indeed to use a Deployment instead of a Job for the spark-submit Pod.

[sbernauer]

Yes, it is. I mentioned it for the sake of completeness, and our approach is likely better :)