yaml2mml not compatible with python 3.5

[jojo4u]

On my Arch Linux with python 3.5 yaml2mml.py fails. Probably because of #2225.

Traceback (most recent call last):
  File "./yaml2mml.py", line 27, in <module>
    json.dump(yaml, mml_file, indent=2, separators=(',', ': '))
  File "/usr/lib/python3.5/json/__init__.py", line 179, in dump
    fp.write(chunk)
TypeError: a bytes-like object is required, not 'str'

Steps to reproduce:
> python --version
Python 3.5.2
> pacman -Qi python-yaml
Version : 3.12-1
> git clone https://github.com/gravitystorm/openstreetmap-carto.git
> ./scripts/yaml2mml.py
[Error above]

Python 2 works fine
> python2 --version
Python 2.7.12
> pacman -Qi python2-yaml
Version : 3.12-1
> python2 ./scripts/yaml2mml.py
[No error]

[pnorman]

Also has the same error on Debian with 3.5.

We could make it a Python 2 script, but I'm open to suggestions which make it work on both while keeping output the same on Linux and Windows.

[StyXman]

The problem mostly arises from the fact that the file is opened as binary. Is there a reason for this? In theory a text file should be enough, as json should be text. Removing the 'b' from line 26 makes it work for both py2 and py3.

[HolgerJeromin]

The reason for binary was #2153

[pnorman]

Yes, we need 'b' or it doesn't work on py2, giving different output depending on newline settings.

[jojo4u]

I understand we want only \n on every platform as newline? I found no compatible way with JSON.dump() but using JSON.dumps() and encode to byte array seem to do the trick:

json_bytes = (json.dumps(yaml, indent=2, separators=(',', ': '))).encode(encoding = "ascii")
mml_file.write(json_bytes)

With python2 both methods yield files with same sha256sum. Tested only on Linux, I have no Windows system at hand.

The python3 output has different sha256sum after every execution, the properties seem to have random sorting. Python3 output would need testing (I only have Kosmtik, does is process mml?) The python2 output on the other hand is stable.

Should I prepare a pull request?

[matthijsmelissen]

The python3 output has different sha256sum after every execution, the properties seem to have random sorting.

That would be not so practical when working with git, maybe we can explicitly add a sort?

[jojo4u]

Yes, adding sort_keys=True does produce the same and stable output for python 2/3.

json_bytes = (json.dumps(yaml, sort_keys=True, indent=2, separators=(',', ': '))).encode(encoding="ascii")

[pnorman]

Yes, adding sort_keys=True does produce the same and stable output for python 2/3.

Is the Windows output the same?

[kocio-pl]

Just for the record: Microsoft publishes virtual machines for testing Internet Explorer/Edge in different versions, but they are in fact just plain generic Windows machines, that you could use for anything. The only drawback is that the license is expiring every 90 days, so one has to roll back to the snapshot once in a while. It's even easy to remember the basic URL: modern.ie.

[StyXman]

The real answer to the original problem would have been to use core.autocrlf.

[jojo4u]

The real answer to the original problem would have been to use core.autocrlf.

This would probably not work for project.mml because it is marked as binary file (34ef136). You lose autocrlf but gain: "nor will it try to compute or print a diff for changes in this file when you run git show or git diff on your project."

[pnorman]

This would probably not work for project.mml because it is marked as binary file (34ef136). You lose autocrlf but gain: "nor will it try to compute or print a diff for changes in this file when you run git show or git diff on your project."

We can change this if needed

[jojo4u]

#2459 has been refined and now been tested with Linux and Windows. Python 2&3 on Linux and Windows are producing same output (with LF newline). Only exception is redirecting output of --check parameter to file, but this is not what this switch is for.

[kocio-pl]

Made irrelevant by #2473.