Pipelines can't restart when interrupted due to different fov_index parameters in OptimizeIteration

[tddough98]

When trying to rerun a merlin pipeline after it has failed or been interrupted, merlin fails early with an error complaining about a task already existing.

merlin.core.analysistask.AnalysisAlreadyExistsException: Analysis task with name Optimize1 already exists in this data set with different parameters.

It looks like this is happening because the when the fov_index parameter is not set for an OptimizeIteration task, it is generated with np.random.choice which returns different values each time it runs.

[tddough98]

workaround is to add the randomly generated parameters to the .json specifying the analysis tasks.

for example

{
      "task": "OptimizeIteration",
      "module": "merlin.analysis.optimize",
      "analysis_name": "Optimize1",
      "parameters": {
        "preprocess_task": "DeconvolutionPreprocess",
        "warp_task": "FiducialCorrelationWarp",
        "area_threshold": 5,
        "fov_per_iteration": 50,
        "optimize_chromatic_correction": true,
        "fov_index": [
          [5, 11],
          [5, 6],
          [6, 3],
          [9, 3],
          [7, 15],
          [9, 6],
          [7, 6],
          [1, 3],
          [2, 1],
          [9, 13],
          [7, 9],
          [2, 8],
          [3, 1],
          [3, 14],
          [8, 9],
          [7, 14],
          [8, 7],
          [5, 5],
          [0, 13],
          [8, 9],
          [8, 4],
          [8, 5],
          [6, 0],
          [1, 0],
          [7, 4],
          [8, 16],
          [5, 15],
          [9, 5],
          [6, 2],
          [9, 15],
          [8, 16],
          [4, 16],
          [9, 4],
          [6, 4],
          [7, 0],
          [2, 15],
          [5, 15],
          [7, 4],
          [9, 8],
          [3, 3],
          [9, 4],
          [6, 8],
          [4, 13],
          [2, 9],
          [7, 14],
          [8, 5],
          [9, 3],
          [5, 14],
          [4, 2],
          [7, 13]
        ]
      }
    },

these can be copied from a past run when this error comes up

some ideas to fix for the future

• add a fov_index_seed parameter to reproducibly generate fov_index
  • or generate the seed from the analysis
  • allow tasks to specify which parameters to ignore when checking if parameters are identical
    • could allow tasks to reuse or regenerate these parameters