Merge develop into master for v1.13.0-beta2

[hardbyte]

Adds support for users to supply blocking information along with encodings. Data can now be uploaded to an object store and pulled by the Anonlink Entity Service instead of uploaded via the REST API.
This release includes substantial internal changes as encodings are now stored in Postgres instead of the object store.

• Feature to pull data from an object store and create temporary upload credentials. Commit initial design of pull feature. #537, Temporary object store credentials endpoint #544, Feature external blocks #551
• Blocking implementation Feature blocking #510 Comparison tasks now based on blocks #527,
• Benchmark container now includes support for blocking Benchmark data has moved to new S3 bucket #478, Feature blocky benchmark #541
• Encodings are now stored in Postgres database instead of files in an object store. Store uploaded encodings in database #516, Use encodings from db instead of minio #522
• Start to add integration tests to complement our end to end tests. Integration tests #520, Split e2e tests out #528
• Use anonlink-client instead of clkhash Update tutorials to use anonlinkclient instead of clkhash #536
• Use Python 3.8 in base image. Update base image to use Python 3.8.2 #518
• A base image is now used for all our Docker images. Feature separate base image #506, Skip build in bash if azure isn't playing along #511, CI: Skip building base image if not required #517, CI system now automatically uses latest base image #519
• Binary encodings now stored internally with their encoding id. Feature extend binary format #505
• REST API implementation for accepting clknblocks Extend upload endpoint to accept both types - clks and clknblocks #503
• Update Open API spec to version 3. Add Blocking API Feature openapi3 #479
• CI Updates Update ci cluster #476
• Chart updates Update redis and minio chart versions #496, Update postgres dependency and bump chart version #497, Update minio #539
• Documentation updates (production deployment, debugging with PyCharm) Updates production deployment documentation. #473, Add docs on debugging using Pycharm #504
• Fix Jaeger Revive jaeger #500, Fix local jaeger #523

Misc changes/fixes:

• Detect invalid encoding size as early as possible Detect invalid encoding size earlier #507
• Use local benchmark cache Use local benchmark cache #531
• Cleanup docker-compose Cleanup docker compose #533, Increase CELERYD_MAX_TASKS_PER_CHILD default #534, Use docker-compose without having to set the --project-directory #547
• Calculate number of comparisons accounting for user supplied blocks. Fix calculation of total number of comparisons #543

Try it out

You can pull this repository and try with Docker Compose. The Docker images are all hosted on Docker Hub:

Component	Docker Hub
Base Image	data61/anonlink-base
Backend/Worker	data61/anonlink-app
E2E Tests	data61/anonlink-test
Nginx Proxy	data61/anonlink-nginx
Benchmark	data61/anonlink-benchmark
Docs	data61/anonlink-docs-builder

Using Kubernetes (follow the detailed docs here:

helm repo add data61 https://data61.github.io/charts
helm repo update
helm install data61/entity-service --version 1.13.1 [--values...]

All the documentation, including tutorials can be found at https://anonlink-entity-service.readthedocs.io/en/latest/index.html