Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CompactPerformanceEvaluation objects and the option in evaluate! to construct them #973

Merged
merged 14 commits into from
Apr 30, 2024

Conversation

ablaom
Copy link
Member

@ablaom ablaom commented Apr 24, 2024

Partly addresses JuliaAI/MLJ.jl#1105, and prepares for JuliaAI/MLJTuning.jl#215.

This PR also drops the per_fold part of PerformanceEvaluation display. More often than not, the table does not fit with this in it. The display still includes the names of all accessible fields. This PR also adjusts the display of per_fold information to mitigate issue with overly wide table.

Previously:

PerformanceEvaluation object with these fields:
  model, measure, operation, measurement, per_fold,
  per_observation, fitted_params_per_fold,
  report_per_fold, train_test_rows, resampling, repeats
Extract:
┌─────────────┬──────────────┬─────────────┬─────────┬──────────────────────────────────────
│ measure     │ operation    │ measurement │ 1.96*SE │ per_fold                            
├─────────────┼──────────────┼─────────────┼─────────┼──────────────────────────────────────
│ BrierLoss() │ predict      │ 0.9330.184   │ [0.719, 1.16, 0.844, 1.16, 0.667, 0 Accuracy()  │ predict_mode │ 0.00.0     │ [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]      
└─────────────┴──────────────┴─────────────┴─────────┴──────────────────────────────────────

After this PR:

PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌───┬─────────────┬──────────────┬─────────────┐
│   │ measure     │ operation    │ measurement │
├───┼─────────────┼──────────────┼─────────────┤
│ A │ BrierLoss() │ predict      │ 0.933       │
│ B │ Accuracy()  │ predict_mode │ 0.0         │
└───┴─────────────┴──────────────┴─────────────┘
┌───┬──────────────────────────────────────────┬─────────┐
│   │ per_fold                                 │ 1.96*SE │
├───┼──────────────────────────────────────────┼─────────┤
│ A │ [0.719, 1.16, 0.844, 1.16, 0.667, 0.914] │ 0.184   │
│ B │ [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]           │ 0.0     │
└───┴──────────────────────────────────────────┴─────────┘

To do:

@ablaom ablaom marked this pull request as ready for review April 24, 2024 04:43
@ablaom ablaom marked this pull request as draft April 26, 2024 00:40
@ablaom ablaom closed this Apr 27, 2024
@ablaom ablaom reopened this Apr 27, 2024
Copy link

codecov bot commented Apr 27, 2024

Codecov Report

Attention: Patch coverage is 94.11765% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 88.13%. Comparing base (b6056dc) to head (8cb6f26).
Report is 6 commits behind head on dev.

❗ Current head 8cb6f26 differs from pull request most recent head c83cae9. Consider uploading reports for the commit c83cae9 to get more accurate results

Files Patch % Lines
src/machines.jl 50.00% 1 Missing ⚠️
src/resampling.jl 96.87% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev     #973      +/-   ##
==========================================
+ Coverage   87.74%   88.13%   +0.38%     
==========================================
  Files          28       28              
  Lines        2457     2587     +130     
==========================================
+ Hits         2156     2280     +124     
- Misses        301      307       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ablaom
Copy link
Member Author

ablaom commented Apr 29, 2024

For the second time codecov is getting cancelled. A timeout?

@ablaom ablaom closed this Apr 29, 2024
@ablaom ablaom reopened this Apr 29, 2024
Copy link
Member

@OkonSamuel OkonSamuel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!!

@ablaom ablaom merged commit f811dc3 into dev Apr 30, 2024
3 checks passed
@ablaom ablaom deleted the compact-performance-evaluations branch April 30, 2024 23:21
This was referenced May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants