Debugging integration test

[Squaess]

What is the best way to debug the integration tests? After I pass my local unit tests I would like to check how the integration is going, but in the case of failing tests I'm getting:

RuntimeError: One or multiple tests failed. Please check job logs for additional information.

And this is the part where you ask not to change it. I was thinking that maybe the logs will be available on the databricks side, but there was no job created. Where exactly should I look for the reason why the tests failed?

I'm using dbx execute --job <job-name> --cluster-name <cluster_name> to run the tests.

[renardeinside]

hi @Squaess ,

The best way to debug integration tests is to use DB Connect, unfortunately we don't have appliance to have interactive debugging in dbx.

[Squaess]

Thank you for the answer. What I was able to do is to deploy the job and then launch it, this way It creates the jobs hence I can look at the logs. But waiting for the dedicated cluster to create is a bit annoying. Is there a way to use the already created all-purpose cluster in the deployment.yml file? All to override the cluster when calling dbx launch?

[Squaess]

I've found out that we can use API 2.1 and create multitask workflows. Having that I just added "existing_cluster_id": "0923-164208-meows279" to the task definition in deployments.yaml and now the integration task is being launched on the existing cluster and I'm able to see the logs from the tests.

This will be very helpful for me.
I believe, I just needed to be heard to figure this out (still a beginner in the Databricks environment) and it worked :)

[renardeinside]

HI @Squaess ,
You cal also launch these tasks on a job cluster and use pre-warmed instance pools, this will significantly cut off the time needed to start the job cluster.