Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Brier score testing #18

Merged
merged 19 commits into from
Nov 8, 2023
Merged

Improve Brier score testing #18

merged 19 commits into from
Nov 8, 2023

Conversation

ogrisel
Copy link
Contributor

@ogrisel ogrisel commented Nov 2, 2023

This a PR to check that our IPCW Brier score implementation behaves empirically as a proper scoring rule on a marginal model (no covariates).

This is similar to what we have in our example but only do marginal model estimation via the brier_score_* function on a grid of estimates.

This way we do not depend on any bias induced by underfitting caused by a bad tuning of the hyperparameters of a conditional model such as GradientBoostingIncidence.

@ogrisel ogrisel marked this pull request as ready for review November 3, 2023 16:24
@ogrisel ogrisel changed the title Improve brier score testing Improve Brier score testing Nov 3, 2023
@ogrisel ogrisel force-pushed the improve-brier-score-testing branch from e0faf83 to 299d9e6 Compare November 4, 2023 19:48
@juAlberge juAlberge self-requested a review November 8, 2023 15:22
Copy link
Collaborator

@juAlberge juAlberge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

everything is ok

@@ -128,12 +128,12 @@ class GradientBoostingIncidence(BaseEstimator, ClassifierMixin):

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • In practise both objective yields -> In practice both objectives yield

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had not seen this suggestion. Could you please open a follow-up PR to fix this now that this Pr is merged?

@juAlberge juAlberge merged commit dafe99d into main Nov 8, 2023
7 checks passed
@juAlberge juAlberge deleted the improve-brier-score-testing branch November 8, 2023 15:27
github-actions bot pushed a commit that referenced this pull request Nov 8, 2023
@juAlberge juAlberge restored the improve-brier-score-testing branch November 8, 2023 15:51
@juAlberge juAlberge deleted the improve-brier-score-testing branch November 9, 2023 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants