dbx is replacing databricks-connect: what about debugging?

[tonkolviktor]

As it can be seen on: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect
"Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time. Also, be aware of the limitations of Databricks Connect."

databricks-connect has support for pycharm and vscode integrations to databricks and via that it's possible to start job in the IDE debug mode.

My question is how can one do that using dbx? Is there any documentation for that?

[renardeinside]

Hi @tonkolviktor ,

Thanks a lot for the question.

Specifically for interactive debugging, this feature is not supported in dbx at the moment. However, you can use both dbx and DB-connect together (while dbx will play the role of CI/CD tool for jobs deployment, and DB-connect is an appliance for interactive development and debugging).

To sum up - if you need the interactive debugging experience, please use DB Connect for debugging scenarios as long as it satisfies your needs.

[tonkolviktor]

thanks @renardeinside

[t0mm3h15]

Hello @renardeinside is there any updates to this??

[lamasmi]

I am also waiting to hear about interactive debugging. This is an essential feature for development, so I don't see why it is not a top priority for dbks.

[edejong-dbc]

It seems odd that we don't hear anything from Databricks on this topic. Debugging is essential for a fast development cycle.

[renardeinside]

Hi everyone,

Unfortunately, I'm not getting notifications on discussions, therefore I just accidentally noticed this thread today.

Please find some points on the topic below:

1. Interactive debugging requires platform-level changes (please read more on this in the Spark Connect introduction post). In dbx we don't introduce platform-level changes and use only the existing APIs, therefore I recommended DB Connect in my original answer.
2. Our engineering team is indeed looking into the topic of interactive debugging and there are ongoing developments in this direction. I recommend reaching out to your account SA for details and timelines.