Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variable classifier_dropout #3

Closed
KoichiYasuoka opened this issue Apr 6, 2024 · 3 comments
Closed

Variable classifier_dropout #3

KoichiYasuoka opened this issue Apr 6, 2024 · 3 comments

Comments

@KoichiYasuoka
Copy link

I've just found that classifier_dropout is fixed to 0.1 in both LlamaForTokenClassification and MistralForTokenClassification as:

        self.dropout = nn.Dropout(0.1)

however in GPT2ForTokenClassification of HuggingFace Transformers it is enhanced as:

        if hasattr(config, "classifier_dropout") and config.classifier_dropout is not None:
            classifier_dropout = config.classifier_dropout
        elif hasattr(config, "hidden_dropout") and config.hidden_dropout is not None:
            classifier_dropout = config.hidden_dropout
        else:
            classifier_dropout = 0.1
        self.dropout = nn.Dropout(classifier_dropout)

and now we are trying to include LlamaForTokenClassification and MistralForTokenClassification into HuggingFace Transformers at huggingface/transformers#29878 . Please show us better way to include them.

@SeanLee97
Copy link
Contributor

@KoichiYasuoka Thank you for your suggestions! I will add a feature to support classifier_dropout.

B.T.W, the BiLLM's implementation for TokenClassification differs from the official one. In BiLLM, we convert the attention mask from uni- to bi-directional. This change can improve the performance of token classification significantly, according to our experiments in the paper https://arxiv.org/abs/2310.01208

@SeanLee97
Copy link
Contributor

I've added a feature to support classifier_dropout in PR: #4

Now, you can specify classifier_dropout in billm>=0.1.2

@KoichiYasuoka
Copy link
Author

Thank you @SeanLee97 and now I close this issue. See you later at huggingface/transformers#29878 and huggingface/transformers#29940

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants