Custom Dashboards

[jannikjw-sai]

Feature request

Enable developers to build their own visualizations on LangSmith using matplotlib or plotly.

Motivation

I would like to visualize certain performance metrics that are not sufficiently shown in the graphs in the LangSmith metrics. This way leaders and developers can understand what exactly is happening.

[hinthornw]

Hi @jannikjw-sai - thanks for raising this issue! Would love to hear more about what kind of visualizations you'd find most useful. Especially if you are particularly interested in monitoring vs. testing/evaluation

[jannikjw-sai]

Hi @hinthornw, I am particularly interested in comparing different models and calculating different self-defined metrics in bar charts. I also want a time series of all my dataset runs. Mostly, I want to run quality evaluations.

[jannikjw-sai]

@hinthornw What is the timeline for this? I would love to have a coding interface to simply create my own dashboard.

[hinthornw]

Got it, @jannikjw-sai - thanks! We hope to get the bar charts for test results out in the next couple of weeks, and time series would be later.

We also have a convenience method in the client:

client.get_test_results(project_name="my-test")

that you can use to fetch the results and metrics as a pandas dataframe. It's fairly straightforward to plot the results then from there