Virtualenv Development Env
Under Construction This is still a bit messy as I merge and edit the content...
This documentation is always somewhat outdated, in particular in view of version numbers.
Contact the gcd-tech mailing list if you need assistance with any of these instructions.
The specifics of setting up a development environment each OS is documented below:
- MacOS - last validated ???
- Linux - last validated ???
- Windows - last validated September, 2023
To run you'll need Python. The project is currently using Python 3.8
The Python.org distribution is preferred. [See Note: 1]
If installing from packages you'll also need development package for python (if it's separated, like python-dev on Ubuntu).
In order to use precompiled libraries with Python, you must get Python 3.8 as the precompiled binaries must match the version of Python or they will not install.
These instructions have been tested with 64-bit Python 3.8.10 for Windows as of February 2023.
Direct link to 64-bit Python 3.8.10 for Windows
After installing Python, add these directories to the front of your PATH to ensure
that they are found first:
%LOCALAPPDATA%\Programs\Python\Python38
%LOCALAPPDATA%\Programs\Python\Python38\Scripts
Get it from http://pypi.python.org/pypi/setuptools#windows
32-bit version of Python Install setuptools using the provided .exe installer.
64-bit versions of Python: download ez_setup.py and run it; it will download the appropriate .egg file and install it for you. Currently, the provided .exe installer does not support 64-bit versions of Python for Windows, due to a distutils installer compatibility issue
Some of the python packages may require compilation and you may encounter the error message as shown below. If this occurs then simply install the Visual Studio tools as indicated
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
Desktop development with C++ is the only package you will need
Install git, which may be found through http://github.com/
Clone the repository. The GitHub web page is: https://github.com/GrandComicsDatabase/gcd-django See the GitHub help documentation for options
Install MySQL (version (?? 5.5.x or 5.7 Windows instructions said 5.5.x and Mac said 5.7 is currently in production) http://dev.mysql.com/downloads/
It can be useful to install the GUI MySQL Workbench.
Put the following lines in your configuration (Windows: my.cnf, MacOS/Linux: /etc/my.cnf) if not already there.
default-character-set = utf8
default-storage-engine = InnoDB
These settings will save you from accidentally not using foreign keys or transactions, or from trying to stuff unicode into an 8-bit character set. None of those are fun to debug or recover from.
Notes:
- These are nowadays the default on Windows installations.
- These options are not needed for mysql 8 (>=8.0.1), there these are the defaults and default-character-set is replaced character_set_server anyway.
- MySQL may not create an
/etc/my.cnfduring installation (for instance on Mac OS X Mountain Lion). Just create one yourself with these two lines in it.
If installing using Linux packages of some sort and also using pip/virtualenv for Python libraries as described below, you will also need the devel-packages, e.g.
- RPM:
mysqlclient-devel - Debian package:
libmysqlclient-dev
CSSTidy is an executable required by django-compressor.
http://csstidy.sourceforge.net/
If DEBUG=True, the default setting, django-compressor is not active, therefore CSSTidy is not needed if you are only working with DEBUG=True.
Notes:
-
Windows: It just needs to be somewhere in the PATH. Using 32bit executable for 64bit seems fine. -
Linux: CSSTidy should exist as a package for most Linux distributions (and possibly for MacPorts as well):- RPM:
csstidy - Debian package:
csstidy
- RPM:
We recommend using virtualenv, but it is optional. It will allow you to isolate the libraries you need for the GCD, and not contaminate your system python.
For more on installing and using virtualenv, see http://www.virtualenv.org/en/latest/index.html
Install the package python-virtualenv.
Create a virtualenv for use with the GCD. For this example, we will create one called "gcd" in a directory called "virtualenvs".
bash:~$ mkdir virtualenvs
bash:~$ cd virtualenvs
bash:~/virtualenvs$ virtualenv gcd
New python executable in gcd/bin/python
Installing setuptools............done.
Installing pip...............done.
bash:virtualenvs$ ls
gcd
bash:virtualenvs$ source gcd/bin/activate
(gcd)bash:virtualenvs$
The recommended way to use pip is within virtualenv, since every virtualenv
has pip installed in it automatically. So do
easy_install.exe virtualenv
If you do not want to use virtualenv install pip using
easy_install.exe pip.
Use pip to install virtualenv, and then create a virtualenv for use with the GCD. For this example, we will create one called "gcd" in a directory called "virtualenvs".
Now create a virtualenv
virtualenv.exe \path\to\env\gcd
and activate the virtualenv
\path\to\env\gcd\Scripts\activate
Note that "(gcd)" at the beginning of the prompt. Now all Python execution
and installation will occur inside of the "gcd" virtual env, i.e. ~/virtualenvs/gcd.
Assuming you've already activated the virtualenv (but not installed anything further in it), and cloned the git repo to ~/git/gcd-django, use pip and our checked-in requirements file to install all necessary Python libraries. Make certain that you have any C/C++ libraries already compiled and installed (such as libicu, and optionally some supplemental libraries for PIL depending on how much of the image editing you want to be able to do).
Debian packages, RPMs and MacPorts packages are available for many of the
Python libraries, and if you prefer to use those, please remember to
make a copy of requirements.txt and remove the libraries that you
installed through other means. If the OS packages were of older Python
library verions, you may need to add versions to your requirements.txt
or adjust the versions to match. See the pip documentation for the
requirements.txt format.
This file tries to install versions matching our production environment. Depending on your installation, you might need more updated version.
PLEASE NOTE: If the latest version of PyICU does not match the version of libicu that you installed, you may need to try an older version of PyICU, which should generally work fine with the GCD. For instance PyICU 1.0 was known to work with libicu 4.4 as of October 2012.
(gcd)bash:~/git/gcd-django$ pip install -r requirements.txt
# A ton of output happens here, especially for PIL compilation.
# You can also just open up the requirements. text and install each line separately.
# It is safe to re-run pip, since if everything installed properly, it will just
# check each library and skip it as already installed.
This should install Django and every other python package you need to run the site and post code reviews for the project.
Inside the git-shell installed by github do
\path\to\env\gcd\Scripts\activate
Or make sure git is in your path if using a different shell and activate
the virtualenv. We assume nothing is installed in it.
Now go the the directory with the cloned git repo "gcd-django".
We use pip and our checked-in requirements.txt file to install all necessary Python
packages. You either downloaded the pre-compiled packages or compile them yourself.
Compiling them yourself is an advanced topic and not necessary; it is recommended
that you simply use the precompiled versions available.
You could also just run pip without the virtualenv to install everything into your system python's site-packages tree, but we recommend to use virtualenv.
Two of the python packages need compiling. This typically additional installs and on x64 does not work out of the box. The easy way is to use the Unofficial Windows Binaries for Python Extension Packages.
That page contains "wheel" packages for various Python libraries that require compilation.
A wheel package can be installed into your virtual environment using pip:
C:> \path\to\env\gcd\Scripts\activate
(gcd) C:\path\to\env\gcd> pip install package.whl
You must obtain a wheel package that is compiled for the specific version of Python
installed on your system. As directed above this is "C Python 3.8" so choose the wheel
packages with cp38 in their name as given below.
Download the Pillow wheel package Pillow-8.4.0-cp38-cp38-win_amd64.
Install the wheel package with pip.
Download the PyICU wheel package PyICU-2.4.3-cp38-cp38-win_amd64.
Install the wheel package with pip.
Now that you've satisifed the requirements for the two packages that require compilation, you can install the remaining packages from the requirements file:
pip install -r requirements.txt
# You can also just open up the requirements.txt and install each line separately.
# It is safe to re-run pip, since if everything installed properly, it will just
# check each library and skip it as already installed.
Congratulations, you should now have everything you need to run the site and post code reviews for the project
In the top level directory of your repository clone, open up settings.py and
take a look at it. You should create a settings_local.py file in the same
directory, and override any values from settings.py that need overriding in
that settings_local.py. Do not modify settings.py itself, unless you need
to push a change out to all other development and production environments.
Our .gitignore settings will prevent git from noticing settings_local.py.
The settings.py file has numerous comments indicating what needs to be overridden.
The most obvious are the DATABASES and CACHES group of settings.
By default, we assume that the database is called gcdonline. It can be
accessed by a user called gcdonline with no password. If you set your
database up differently, look for settings that start with DATABASE. Override
them appropriately.
Create your database and set up whatever permissions you want. Django will create the tables for you, but you need to create the database first.
E.g. from MySQL command line client do (this is without a password):
create schema gcdonline;
create user gcdonline;
grant all on gcdonline.* to gcdonline;
To create the various tables run the migrations with
python manage.py migrate
This will create the tables, populate them with some of the required initial data, and update your schema if any updates beyond the initial state are required.
During this process you may see system check errors for models.E025 and for captcha (see below). You can suppress these checks by adding this line to your settings_local.py.
SILENCED_SYSTEM_CHECKS = ['models.E025','captcha.recaptcha_test_key_error']
ERRORS:
?: (captcha.recaptcha_test_key_error) RECAPTCHA_PRIVATE_KEY or RECAPTCHA_PUBLIC_KEY is making use of the Google test keys and will not behave as expected in a production environment
HINT: Update settings.RECAPTCHA_PRIVATE_KEY and/or settings.RECAPTCHA_PUBLIC_KEY. Alternatively this check can be ignored by adding `SILENCED_SYSTEM_CHECKS = ['captcha.recaptcha_test_key_error']` to your settings file.
oi.PreviewCreator: (models.E025) The property 'data_source' clashes with a related field accessor.
oi.PreviewCreatorArtInfluence: (models.E025) The property 'data_source' clashes with a related field accessor.
oi.PreviewCreatorDegree: (models.E025) The property 'data_source' clashes with a related field accessor.
oi.PreviewCreatorMembership: (models.E025) The property 'data_source' clashes with a related field accessor.
oi.PreviewCreatorNonComicWork: (models.E025) The property 'data_source' clashes with a related field accessor.
oi.PreviewCreatorSchool: (models.E025) The property 'data_source' clashes with a related field accessor.
oi.PreviewReceivedAward: (models.E025) The property 'data_source' clashes with a related field accessor.
Currently it is needed to manually perform python manage.py loaddata on all
files in the fixtures folders of the django apps,
e.g. apps/indexer/fixtures/, apps/gcd/fixtures, etc.
python manage.py loaddata users
this will add an admin user, an approver, and a basic indexer. The usernames,
email addresses and passwords can be seen in the fixture file, which is
located at apps/indexer/fixtures/users.yaml. Load all the relevant fixture
data for the apps you expect to be using.
If you want data in your database, and know that the current development master matches the production schema, you can load a data dump from
http://www.comics.org/download/
Contact the GCD tech team to find out if the development and production schemas currently match, and what to do if they do not.
With the default user and database-name you need to do
mysql -ugcdonline gcdonline < dump.sql
(You might want to do -v to see something is happening)
Note that we do not distribute the images for covers and other user contributed uploads of scans.
For haystack search to work one needs to run elasticsearch search backend. The latest version can be downloaded here: http://www.elasticsearch.org/download/
Either use the provided deb/rpm or unpack it and run bin/elasticsearch -f (on Linux and Mac) or bin\elasticsearch.bat (on Windows).
If you get an error like "Unsupported major.minor version 51.0" and a stack trace, it means that you need to update your java runtime.
After that you can run python manage.py rebuild_index in gcd-django directory to populate your search indexes with data.
At this time you should be able to test your installation and data import using a read-only mode, or try out editing with read-write mode.
Read-write mode is the default.
For read-only mode, set READ_ONLY = True and NO_OI = True in settings_local.py
Once your settings are chosen, run
python manage.py runserver
and take a look at http://127.0.0.1:8000/
If you see the following error:
Invalid HTTP_HOST header: '127.0.0.1:8000'. You may need to add '127.0.0.1' to ALLOWED_HOSTS.
then check your local_settings.py to ensure your ALLOWED_HOSTS array includes 127.0.0.1 like shown below
ALLOWED_HOSTS = [
'.comics.org',
'127.0.0.1',
'.comics.org.', # Allow FQDN and subdomains. Can be dropped in 1.7
]
If you don't get any page at all, check http://www.djangoproject.com/ if it looks like Django isn't configured correctly, or http://dev.comics.org/ and/or http://groups.google.com/group/gcd-tech/ if it appears to be a code problem.
When you start making changes, go to the wiki at TODO: add dev instructions
for instructions on how to get code reviewed before checkin. If you develop
a good record of contributions, you will be granted direct access to the project.
For Windows ActiveState seems to be ok for x32 but for x64 ActiveState has some purchase 'enforcing'
Ex: pypm install setuptools The following packages will be installed into "%APPDATA%Python" (2.7): distribute-0.6.27 error: Can't install distribute-0.6.27: requires Business Edition subscription *** If you have purchased ActivePython Business Edition, please login to *** your account at: *** https://account.activestate.com/ *** and download and run the license installer for your platform.
*** Please visit to learn more *** about the ActivePython Business Edition offering.
One needs to install scons 1.3.1 (later versions didn't work for me) http://www.scons.org/ Download zip, run setup.py install. Download CSStidy source distribution from: http://csstidy.sourceforge.net/ VC9 won't build it properly Fix is not too difficult though. Solution comes from: http://stackoverflow.com/questions/3075697/non-existent-member-function-specified-as-friend Edit umap.hpp Move class iterator definition block toward the first line of public: (of class umap)