Make the template installable from PyPI

[foster999]

Issue

At the ONS, our data access platform (DAP) doesn't have access to the web and the process for moving files in and out is non-trivial. However, we can install packages via our Artifactory instance, which mirrors PyPI.

Solution

Would it be possible to make the template available at a data-only package on PyPI?

I'm not certain how we would then run cookiecutter with the template, other than pointing to the resources in site-packages. But at least we'd be able to make use of it.

Alternatives

We might be able to request to mirror specific GitHub repositories in Artifactory, but I thought that PyPI might be a more general solution if other departments are in a similar boat.

[ESKYoung]

I've had a go at this. What I've done is to create a skeleton package - this contains only the minimal files and folders required to generate a project template. I've then created a package that includes all these files and folders, alongside a CLI interface. Need to write some docs, and tests (not sure how at the moment, so any pointers would be grand!)

You can test this in the interim, as I've pushed something onto Test PyPI. Do the following in your terminal (ideally in a Python virtual environment):

pip install click cookiecutter  # needed as the dependencies may not be on Test PyPI
pip install -i https://test.pypi.org/simple/ govcookiecutter
govcookiecutter

You should then see the prompts! I've branched this off PR #33, which is currently in review.

@foster999 not sure if you're still looking for this but tagging @alexander-newton as well.

[foster999]

Thanks @ESKYoung, this sounds perfect! I'm not sure if we mirror Test PyPI, but I'll give it a try on and off network to make sure it works in principle 👍

On the testing front, pytest has a neat fixture tmp_path that will generate a temporary directory (returning a pathlib. Path to it) when passed an a parameter to a test function. If you want to test the CLI, you could then use the subprocess module to execute the command at that location via the CLI. I'd then use pathlib to assert if the relevant files/dirs exist as expected.

[ESKYoung]

Thanks @foster999 - let me know how it goes.

On the test front, you're right - that's the way to go. I may not have time to complete this feature, but for reference I'm thinking:

1. get pytest to build the package locally in a temporary folder using tmp_path
2. create a virtual environment
3. install the locally built package there
4. create a project in a temporary directly using tmp_path.

Should be straight forward to use all the default inputs (use the --no-input flag). May have to add in the extra-context option to enable testing of other options. Difficulty arises when testing all the other CLI options.

However, this should all be mitigated by the fact this is strictly a series of integration tests (arguments are passed to cookiecutter correctly), and an overall system test that it does not fall over.

Testing that the BuildPackage class has the correct folders/files might be more tricky. Instead, it might be more scalable/maintainable to change the copy_folders_and_files variable to exclude files/folders we don't want in the skeleton package. This would make sure all the correct files are transferred. For example, we could just state that .git and src folders are not transferred (the latter to prevent a nightmarish recursive copying fiasco), but everything else is packaged up. We could then test that the package folder in site-packages does not have these excluded folders.

[foster999]

Works nicely off-network - will give it a try internally once it's on PyPI, but I can't see there being any issue 😀

I'd agree that there's minimal gain to be had from testing the CLI options. The testing looks thorough in the main template and the parameters map one-to-one into cookiecutter, so there's very little that could go wrong. A test to assert that all of the cookiecutter parameters (could access these from the function object) have CLI equivalents could be useful for keeping them aligned?

But otherwise, just ensuring the default options run successfully sounds good. Reversing the logic for selecting files sounds sensible to me too!

[ESKYoung]

Thanks @foster999 for confirming!

I don't think I'll have time now to work on this further, so feel free to takeover! I've rebased the branch up to version 1.2.2 of govcookiecutter.

I've had a rethink about reversing the logic for selecting files. Although more automated, there's a risk that it'll automatically transfer over folders/files that may be unwanted.

I briefly tried it out and, for example, it transferred over my .pytest_cache folder. It would be a pain to specify all the possible files/folders it should ignore, and there's a risk that we might accidentally package up data/secrets or the like as well! Other points for completion in the above comment still apply.

[ESKYoung]

Oh and the version number in src/govcookiecutter/__init__.py should also be changed!