fix(NodeBootstrapAbortManager): use correct timeouts for operations

[aleksbykov]

NodeBootstrapAbortManager run in parallel two operations with ParallelObject:

1. bootstrap (node_setup)
2. abort_operation (stop_scylla)

ParallelObject has the common timeout for 2 operations equals
Instance_start + log message wait timeout

Bootstrap operation has default timeout 3600 ( but operation have to be aborted earlier)

abort_operation (stop_scylla) has wait timeout for log message equals
Instance_start + log message wait timeout

Usually abort operation triggered much earlier, when required log message appeared in log and
abort action is going to be executed.

But during job: longevity-100gb-4h-fips-test, required log message was not found during
required timeout. Parallel object exited and next steps of nemesis were started,
But because Threads initiated by ParallelObject are still in state running,
ParallelObject can't stop them and they still running in background.
It was the reason of race between threads, and that all scylla data were
removed while it had been starting and triggered coredump.

Fix changes expected timeouts to avoid races between threads
and sure that all operations finished as expected

Fix #6568

Testing

• https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/abykov/job/longevity-100gb-4h-fips-test/3
• https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/abykov/job/longevity-100gb-4h-fips-test/2

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevent to this change (if needed)

[fruch]

If the whole nemesis is built on timing of log messages, we should find/ask for APIs to replace it, logging on a bigger scale isn't going to be very accurate

[fruch]

Please use the hydra command to generate the nemesis pipeline, don't copy it by hand

[roydahan]

@fruch your comment is for the pipeline, right>
Not for the yaml.

[aleksbykov]

Removed this commit

[roydahan]

If the whole nemesis is built on timing of log messages, we should find/ask for APIs to replace it, logging on a bigger scale isn't going to be very accurate

I agree with @fruch here.
This PR should get in anyway to fix the current issue.
However, @aleksbykov and @temichus, please have a task to harden this nemesis to be more deterministic and less relying on log messages.
We already know that SCT can handle log messages in a big delay when we have too many messages.

[aleksbykov]

If the whole nemesis is built on timing of log messages, we should find/ask for APIs to replace it, logging on a bigger scale isn't going to be very accurate

I agree with @fruch here. This PR should get in anyway to fix the current issue. However, @aleksbykov and @temichus, please have a task to harden this nemesis to be more deterministic and less relying on log messages. We already know that SCT can handle log messages in a big delay when we have too many messages.

I agree that log message could be not good choice. But we don't have another marker what process step is running right now. When raft topology will be GA feature, then we can try to switch to raft history or raft state table to get some process state.
The task is created: https://github.com/scylladb/qa-tasks/issues/1590

[fruch]

@aleksbykov it's quite confusion that your runs pipelines ends with failure, not that I can understand why

[fruch]

LGTM