Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rust): Pub-licize Expr DSL Function enums #20421

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Shoeboxam
Copy link

Changes the visibility of parts of the Expr DSL that I'd like to add support for in OpenDP (see current docs) to pub.

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature rust Related to Rust Polars labels Dec 23, 2024
@ritchie46
Copy link
Member

They are not meant to be public. They are internal state of the DSL. Could you explain why you need that public?

Copy link

codecov bot commented Dec 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.00%. Comparing base (9e32651) to head (df10a26).
Report is 102 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #20421      +/-   ##
==========================================
- Coverage   79.63%   79.00%   -0.63%     
==========================================
  Files        1564     1562       -2     
  Lines      217804   220166    +2362     
  Branches     2477     2486       +9     
==========================================
+ Hits       173439   173944     +505     
- Misses      43796    45649    +1853     
- Partials      569      573       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Shoeboxam
Copy link
Author

Shoeboxam commented Dec 23, 2024

The use-case is a software project (OpenDP) that proves that a given Polars DSLPlan satisfies differential privacy. For an example of its use, the Switzerland Federal Statistical Office used the tool to release high-resolution poverty statistics.

To accomplish this, I've written a number of Polars plugins, as well as a static analyzer for DSLPlans. The analyzer tracks key properties about the data via a recursive parser over each expression/step in the DSLPlan. Over the last year I've been expanding the kinds of expressions and plans that the analyzer can reason about.

For example, if an expression in the plan contains an ArrayFunction::Explode, then in differential privacy jargon the analyzer needs to account for a k-stable transformation (where k is the static array length). Whereas if the DSLPlan contains an ArrayFunction::Min, then the analyzer needs to account for a 1-stable transformation. Unfortunately, since the array enum is private, I can't add logic to the analyzer for array functions, so the analyzer rejects all Polars plans with array functions.

For more context on the project:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants