@solth Looks good. Another idea would be to check the whole file with a regular expression containing only allowed characters for the respective language?

Would this workflow which checks all files in pull requests or push operations be better?

Would this workflow which checks all files in pull requests or push operations be better?

You can use file to detect the file content encoding but you must extend the usage of used file parameters. Quoting form the manual page of file:

file tests each argument in an attempt to classify it.  There are three sets of tests, performed in this order: filesystem tests, magic tests, and language tests.  The  first test that succeeds causes the file type to be printed.

So, you must at least use the parameter -k or --keep-going to not to stop on the first result. The sub results should / must be evalated too as after one encoding an other encoding in the same file can occur. Quoting the man page:

     -k, --keep-going
             Don't stop at the first match, keep going.  Subsequent matches will be have the string ‘\012- ’ prepended.  
             (If you want a newline, see the -r option.)  The magic pattern with the highest strength (see the -l option) 
             comes first.

You should even know how many bytes on default are read by file to determinate the encoding and file type. Quoting:

     -P, --parameter name=value
             Set various parameter limits.

                   Name         Default    Explanation
                   bytes        1048576    max number of bytes to read from file
                   elf_notes    256        max ELF notes processed
                   elf_phnum    2048       max ELF program sections processed
                   elf_shnum    32768      max ELF sections processed
                   encoding     65536      max number of bytes to scan for encoding evaluation
                   indir        50         recursion limit for indirect magic
                   name         50         use count limit for name/use magic
                   regex        8192       length limit for regex searches

Only the first 65.536 bytes of a file are used for encoding detecting. Our files - even only the message resource files - are larger than this limit, so you must extend the limits to get the whole file and not only the first 65.536 bytes.

But what is the purpose of the new Java test code?

The main intention was to provide an automatic way to detect when the encoding of the property files has been changed to something that results in broken special characters being displayed in the frontend and thus would resolve #6331. The purpose was not to detect encoding changes to any source code files, as in my memory that didn't cause any otherwise undetected problems in the past (if the source files contain non-ASCII special characters - outside of String literals, for example in variable names - the whole compilation of the code will probably fail if the encoding is changed to something incompatible with UTF-8).

@solth Looks good. Another idea would be to check the whole file with a regular expression containing only allowed characters for the respective language?

This is a good suggestion and would probably be a better alternative to my implementation, since it would cover the whole files and not just sample messages from them. Do you know if there are predefined regular expressions covering all valid characters for specific languages?

Would this workflow which checks all files in pull requests or push operations be better?

Yes, that is probably a better way to detect file encoding changes in any source file, and thus would cover #6324.

This is a good suggestion and would probably be a better alternative to my implementation, since it would cover the whole files and not just sample messages from them. Do you know if there are predefined regular expressions covering all valid characters for specific languages?

Not really, no. I googled a bit and found this: LocaleData.getExemplarSet of the ICU4J library, which provides all valid characters of a language. But I have not used this library before. For German, English and Spanish it is probably not that difficult to specify regular expressions manually.

PR #6341 adds a CI action which checks the whole file (all which are affected by a push or pull request).