Update regex if Huggingface fixes a presumed bug in the OpenAI implementation

[schmmd]

#2310 changed our regular expressions to match Huggingface's, but it's possible they have a bug in their regular expressions (see huggingface/pytorch-openai-transformer-lm#48). If huggingface/pytorch-openai-transformer-lm#48 is resolved we should update

allennlp/allennlp/data/token_indexers/openai_transformer_byte_pair_indexer.py

Line 25 in a83eb8d

text = re.sub(r'''(-+|~+|!+|"+|;+|\?+|\++|,+|\)+|\(+|\\+|\/+|\*+|\[+|\]+|}+|{+|\|+|_+)''', r' \1 ', text)

with their fix.

[matt-gardner]

@schmmd, ping.

[schmmd]

Cool! Although I likely won't get to this before my vacation.

[schmmd]

Huggingface hasn't made any changes (my issue on their repo is still open)--so there's nothing to do here presently.

[matt-gardner]

We're going to be removing that code entirely, in favor of PretrainedTransformer stuff. Closing this issue.