You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I came across your implementation of batch re-normalization in the BatchReNormDNNLayer class, and I think there might be an error that might be affecting the model's performance.
My understanding of batch re-norm is that it applies the standard BN normalization first, then applies the r/d correction, and then finally applies the gamma/beta scaling and bias. Something along the lines of this:
normed_x= (x-batch_mean) /batch_std# standard BNnormed_x=normed_x*r+d# The batch renorm correctionnormed_x=normed_x*gamma+beta# final scale and bias
Thanks, this is definitely a discrepancy, likely born of the fact that I threw this layer together in a couple minutes right after seeing the paper. Also because cuDNN's batchnorm does the scale+shift automatically. If you want to submit a pull request to fix it (just moving the gamma+beta outside the cuDNN bnorm call would do it) feel free, though I'm not really supporting much of the code in this repo thse days <_<
I came across your implementation of batch re-normalization in the BatchReNormDNNLayer class, and I think there might be an error that might be affecting the model's performance.
My understanding of batch re-norm is that it applies the standard BN normalization first, then applies the r/d correction, and then finally applies the gamma/beta scaling and bias. Something along the lines of this:
However, this line is applying the r/d correction after the scaling and centering with gamma and beta.
https://github.com/ajbrock/Neural-Photo-Editor/blob/master/layers.py#L128
It probably works anyway, based on the good results you seem to have gotten. I just thought I'd bring it to your attention.
The text was updated successfully, but these errors were encountered: