This document describes the backend processes the CPAN Testers site performs: what do they do, why are they needed, where they are located, and how to troubleshoot them.
- Repository: CPAN::Testers::Data::Uploads
- Base Folder ($BASE): /media/backend/cpantesters/uploads
The Uploads database contains basic information about the history of CPAN, including the release dates of all CPAN and BackPAN modules, direct from PAUSE. This is generated directly from the local CPAN Testers CPAN and BackPAN mirrors.
This data is used by CPAN Testers to show distribution information, but it may also be being exported for use by (for example) neilb's CPAN statistics.
- Config:
$BASE/data/uploads.ini - Logs:
$BASE/logs/uploads.log
This script is run every 5 minutes from
/media/backend/cpantesters/autorun-data1.sh. The -u flag means
update and updates data from the CPAN/BACKPAN mirrors into the
database (in the cpanstats.uploads table).
The CPAN/BACKPAN mirrors are located at /home/ftp/CPAN/authors/id
and /home/ftp/BACKPAN/authors/id respectively.
This script leaves a lot of _journal.sql.* files in $BASE/logs. One
for every run of the script. These are used by the backup script, below.
- Config:
$BASE/data/uploads.ini - Logs:
$BASE/logs/uploads.log
This runs every night via /media/backend/cpantesters/autorun-back2.sh.
This invocation reads all the journal files generated by bin/uploads.pl -u and then performs the same operation on a sqlite database located in
$BASE/data/uploads.db. Once the journals have been used in this way,
the journals are moved to the $BASE/logs directory where they
accumulate (XXX: can they be cleaned up automatically?).
- An API on to the uploads database instead of exporting SQLite
databases
- We need to ask Neil Bowers about his process to see if we can provide him with some better APIs