chore: analyze language identification models performance on short ingredient texts with precision-recall evaluation (#349) #365
Conversation
short ingredient texts with precision-recall evaluation (openfoodfacts#349)
korablique
changed the title
|
@baslia Could you please help me with this error? https://github.com/openfoodfacts/openfoodfacts-ai/actions/runs/12106320879/job/33751933052?pr=365 |
Hey, I thought this was just a check about the PR label attribute, I just attached the label "enhancement". |
|
Yes indeed it's a configuration issue from the repo, it's safe to ignore! |
|
|
||
| df = pd.DataFrame(data, columns=["ingredients_text", "lang"]) | ||
| df.dropna(inplace=True) | ||
| df.to_csv(dataset_file, index=False) |
There was a problem hiding this comment.
I had an issue when generating the CSV:
_csv.Error: need to escape, but no escapechar set
By providing an escape char, it works:
df.to_csv(dataset_file, index=False, escapechar="\\")
|
Thank you a lot for this PR! I will add a few suggestions for the next steps in the main issue #349 |
This is the research conducted in the next issue: #349
01_extract_data.py: extracts all texts with their languages from huggingface dataset.02_select_short_texts_with_known_ingredients.py: filters texts with length up to 10 words, performs ingredient analysis by OFF API, selects ingredient texts with at least 80% of known ingredients, adds short texts from manually checked data.What is manually checked data:
I created a validation dataset from texts from OFF (42 languages, 15-30 texts per language).
I took 30 random texts in each language, obtained language predictions using the Deepl API and two other models (language-detection-fine-tuned-on-xlm-roberta-base and multilingual-e5-language-detection). For languages they don’t support, I used Google Translate and ChatGPT for verification. (As a result, after correcting the labels, some languages have fewer than 30 texts).
03_calculate_metrics.py: obtains predictions by FastText and lingua language detector models for texts up to 10 words long, and calculates precision, recall and f1-score.Results are in files: 10_words_metrics.csv, fasttext_confusion_matrix.csv, lingua_confusion_matrix.csv.
It turned out that both models demonstrate low precision and high recall for some languages (indicating that the threshold might be too high and should be adjusted).