Evaluation engine

[josvandervelde]

The Evaluation Engine is now part of our docker compose setup!

You can test it using the following commands:

1. checkout this repo with branch feature/evaluation-engine
2. Make sure you have docker and docker compose (see README, prerequisites)
3. docker compose --profile all up -d
4. you should see no errors. You could go to localhost:8000 to click around.
5. docker run --rm -it -v ./config/python/config:/root/.config/openml/config:ro --network openml-services_default openml/openml-python
6. Inside the openml-python container, run the manual tests described at the bottom of the README.
7. docker compose --profile all down when you finished.

[SubhadityaMukherjee]

Still testing everything. But there were still some issues with running them on the Mac. I attempted to find a fix for it though.

• Elasticsearch was not working before, turns out the older versions dont run on the new macs. This does - image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2 . Doing this atleast starts the container and seems to be running fine.
• A very weird Mac specific error : Port 5000 is taken by the Airplay system service which can be disabled by going to System Settings -> Airdrop & Handoff -> AirPlay Reciever and disabling (unchecking) AirPlay Reciever.

[SubhadityaMukherjee]

Okay so another error I got was while changing the permissions for the www-data. Apparently on mac this is how you do it. chown -R _www:_www data/php instead.

The final puzzle piece so far is now that somehow datasets are not being published. openml.exceptions.OpenMLServerException: http://nginx:80/api/v1/xml/data/ returned code 132: Failed to move the files - None

[josvandervelde]

Thanks for the feedback, @SubhadityaMukherjee !

Fixes:

1. We decided not to support arm for now. Reason: the ES version that we use @ openml has no arm image, as Subha discovered. Newer versions of ES don't play nice with our php code. If we want to use a newer version, we have to upgrade ES in prod, update our php code (there's already a branch for that). As long as that's not done, we just won't support ARM in openml-services
2. chown problem: I added the advice to use chmod 777. It's public data anyway
3. Port 5000 --> changed to 8081
4. Datasets not being published: this boils down to the chown problem