Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RBM Weight Updates Issue #9

Open
Darwin2011 opened this issue Oct 20, 2014 · 5 comments
Open

RBM Weight Updates Issue #9

Darwin2011 opened this issue Oct 20, 2014 · 5 comments

Comments

@Darwin2011
Copy link

Such Issues are in your RBM different implementation.
[python version]
self.W += lr * (numpy.dot(self.input.T, ph_sample)
- numpy.dot(nv_samples.T, nh_means))
[C++ version]
W[i][j] += lr * (ph_mean[i] * input[j] - nh_means[i] * nv_samples[j]) / N;

For there two versions, the weight update methods are incosistent. And Actually I think the right version should be
self.W += lr * (numpy.dot(self.input.T, ph_means)
- numpy.dot(nv_means.T, nh_means))
Could you please help me confirm such issues? I am not quite sure that whether it's issue or not. And I am just a freshman for deep learning.

Best Regards

@liulhdarks
Copy link

@Darwin2011 I think (ph_mean[i] * input[j] - nh_means[i] * nv_samples[j]) is right. The weights gradient should be data-model。

@Darwin2011
Copy link
Author

@liulhdarks ,thanks
The following code shows that the different weight update method. Could you check the following link?
https://github.com/echen/restricted-boltzmann-machines/blob/master/rbm.py

@liulhdarks
Copy link

@Darwin2011 https://github.com/echen/restricted-boltzmann-machines/blob/master/rbm.py doesn't use the visible layer after sampling when updating weights. You can refer to the follow ML frameworks.

INDArray wGradient = input.transpose().mmul(probHidden.getSecond()).sub(
nvSamples.transpose().mmul(nhMeans));

DoubleMatrix wGradient = input.transpose().mmul(hProp1.prob)
.sub(vPropn.sample.transpose().mmul(hPropn.prob));

Hope to help you!

@Darwin2011
Copy link
Author

@liulhdarks thanks for your kind help. I will follow your guide.

Best Regards

@luochao436
Copy link

@liulhdarks @yusugomori
I think the update for W , a , b should as follows:
self.W += lr * (numpy.dot(self.input.T, ph_mean - numpy.dot(nv_samples.T, nh_means))
self.vbias += lr * numpy.mean(self.input - nv_samples, axis=0)
self.hbias += lr * numpy.mean(ph_mean - nh_means, axis=0)

In origin python RBM.py. , self.W and self.hbias is not correct. This problem is also in C/C++ files, but the W has been fixed.
If I am not right, please inform me.
Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants