Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confidence interval calculation #15

Open
jd-a opened this issue Jan 7, 2021 · 5 comments
Open

Confidence interval calculation #15

jd-a opened this issue Jan 7, 2021 · 5 comments

Comments

@jd-a
Copy link

jd-a commented Jan 7, 2021

I wonder if a square root is supposed to occur in lines 86 as well as 89 of confidence_interval.jl. The formula for the confidence interval for hedges Hedges g I am familiar with would be the standard error of the g statistic, which would be the part [σ² = √(((nx + ny) / (nx * ny)) + (es^2 / 2(nx + ny)))], times the percent point function of the normal distribution [z = Distributions.quantile(Normal(), uq)] added or subtracted from the point estimate.
In the current code σ² is rooted once more. in line 89.

But it could be that I am just missing something...

@rikhuijzer
Copy link
Collaborator

These things are good to double check. If you want, you can find an example calculation online and add it to the tests. If this package gets the same results, then the calculation is very likely to be right.

@jd-a
Copy link
Author

jd-a commented Aug 10, 2021 via email

@rikhuijzer
Copy link
Collaborator

Sorry for not responding earlier! For some reason, I didn't get a notification

This should be easy to fix by removing the square root in line 86 (as well as some parentheses). After doing so, the CIs of the Julia and R functions match much better.

Sounds like low hanging fruit. Could you open a PR for this with some information on why it's better without the square root?

But, probably more importantly, a (noncentral) t-distribution should be a better match for a confidence interval

Well, if you want you can also send a PR for this. However, this package has only two stars, so it's unlikely that people will actually use your work if you decide to implement it.

@jdalbers
Copy link
Contributor

jdalbers commented Feb 6, 2022

I have made some adjustments to the estimation functions, and I believe the confidence intervals, both the bootstrapped as well as the parametric, need more work. But I fear I have neither the skills nor the time to pull that off.

Probably interesting references for a person who considers working on it: Kelley; The Effects of Nonnormal Distributions on Confidence Intervals Around the Standardized Mean Difference: Bootstrap and Parametric Confidence Intervals; https://doi.org/10.1177/0013164404264850 and Algina, Keselman, Penfield; Confidence Interval Coverage for Cohen's Effect Size Statistic; https://doi.org/10.1177/0013164406288161

@jdalbers
Copy link
Contributor

jdalbers commented Feb 6, 2022

I gave the CI-calculation another look, came across this article which takes a nice look at a few ways to estimate the variance: http://dx.doi.org/10.20982/tqmp.14.4.p242 (it has a corrigendum (http://dx.doi.org/10.20982/tqmp.15.1.p054)).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants