Skip to content

Installation Guide

Jump to bottom Edit New page
dsg101 edited this page Jun 5, 2012 · 73 revisions

Installation Guide

Description of how to create a new tsudat2 Server, given a Ubuntu image.

Fire up VM in NCI OpenStack Cloud

This assumes that euca tools is installed on the machine you are working from. You must be given the credentials and testkey.private associated with the instance, and put this into .nova/novarc

  • sudo su - #(to be root)
  • source /home/tsudat/.nova/novarc
  • euca-run-instances -k testkey ami-00000057 -t t1.portal (t1.portal has enough memory and disk space) (This may take a while.)
  • euca-describe-addresses (to get a list of free public IPs)
  • euca-associate-address -i {instance id} {public ip}
  • Open Firewall Ports (You only need to do this once)
    • euca-authorize -P tcp -p 80 -s 0.0.0.0/0 default # (http)
    • euca-authorize -P tcp -p 22 -s 0.0.0.0/0 default # (ssh)
    • euca-authorize -P tcp -p 8080 -s 0.0.0.0/0 default # (tomcat)
    • euca-authorize -P tcp -p 8989 -s 0.0.0.0/0 default # (celerymon)
  • ssh -i /root/.nova/testkey.private (public ip) (ssh in to the instance)

Install GeoNode on Instance

  • apt-add-repository ppa:geonode/release
  • apt-get update
  • apt-get install geonode

Upgrade to PostgreSQL 9.0

NOTE: TsuDAT requires PostgreSQL 9.0 only because the db dump of the base data was done from a 9.0 DB. If you are creating a new database from scratch (and loading new PTHA data into it), This set of steps is not required.

  • su - postgres
  • pg_dumpall > data #(backup data)
  • exit user postgres
  • add-apt-repository ppa:pitti/postgresql
  • apt-get update
  • apt-get install postgresql-9.0 libpq-dev

you will now have 2 versions of postgres installed, and the 9.0 version will be configured on port 5433

  • /etc/init.d/postgresql-8.4 stop
  • /etc/init.d/postgresql stop
  • edit /etc/postgresql/9.0/main/postgresql.conf and change port config from 5433 to 5432 (around line 63)
  • /etc/init.d/postgresql start
  • su - postgres
  • psql < data #(some errors ok)
  • exit
  • /etc/init.d/apache2 restart #(restart apache so GeoNode picks up the new DB config)

Install PostGIS for PostgreSQL 9.0

  • add-apt-repository ppa:ubuntugis/ubuntugis-unstable
  • add-apt-repository ppa:pi-deb/gis
  • apt-get update
  • apt-get install postgresql-9.0-postgis
  • y
  • su - postgres
  • createdb template_postgis
  • psql -q -d template_postgis -f /usr/share/postgresql/9.0/contrib/postgis-1.5/postgis.sql
  • psql -q -d template_postgis -f /usr/share/postgresql/9.0/contrib/postgis-1.5/spatial_ref_sys.sql
  • psql -q -d template_postgis -f /usr/share/postgresql/9.0/contrib/postgis_comments.sql
  • psql -d template_postgis -c "GRANT ALL ON geometry_columns TO PUBLIC;"
  • psql -d template_postgis -c "GRANT ALL ON geography_columns TO PUBLIC;"
  • psql -d template_postgis -c "GRANT ALL ON spatial_ref_sys TO PUBLIC;"
  • exit

configure euca-tools

in root/

Consider creating an image now

See;

Create and Mount /data

on machine with euca-tools which is configured to talk to the API.

  • euca-create-volume -s 100 -z nova

  • euca-attach-volume -i {instance-id} -d /dev/vdc {volume-id}

  • fdisk /dev/vdc #(n p 1 {enter} {enter} w)

  • mkfs.ext3 /dev/vdc1

  • mkdir /data

  • mount /dev/vdc1 /data

  • edit /etc/fstab

    • /dev/vdc1 /data auto defaults,nobootwait,comment=cloudconfig 0 0
    • also remove /dev/sda2 while we are in here since this doesnt exist.

  • mount -a

OR

Mount an existing volume to /data

on machine with euca-tools which is configured to talk to the API.

  • euca-attach-volume -i {instance-id} -d /dev/vdc {volume-id}
  • mkdir /data
  • mount /dev/vdc1 /data
  • edit /etc/fstab
    • /dev/vdc1 /data auto defaults,nobootwait,comment=cloudconfig 0 0
    • also remove /dev/sda2 while we are in here since this doesnt exist.
  • mount -a

Configure PostgreSQL Data Directory on /data

  • mkdir /data/pg_data
  • chown postgres:postgres /data/pg_data
  • mv /var/lib/postgresql/9.0/main /data/pg_data
  • edit /etc/postgresql/9.0/main/postgresql.conf
  • change data_directory setting to data_directory = '/data/pg_data/main'
  • /etc/init.d/postgresql restart

Configure GeoServer Data Directory on /data

  • /etc/init.d/tomcat6 stop
  • mv /var/lib/tomcat6/webapps/geoserver/data /data/geoserver_data
  • OR mv /var/lib/geoserver/geonode-data /data/geoserver_data
  • edit /var/lib/tomcat6/webapps/geoserver/WEB-INF/web.xml OR /etc/geonode/geoserver/web.xml
    • Change GEOSERVER_DATA_DIR (on line 50) to /data/geoserver_data
  • /etc/init.d/tomcat6 start

Move initial data to the instance

  • mkdir /data/initial_data
  • cd /data/initial_data
  • rsync [email protected]:/short/w85/tsudat_initial_data.tar.gz .
  • tar -xzvvf tsudat_initial_data.tar.gz

Create a new empty TsuDAT Database

If you want to restore an Existing TsuDAT Base Database go to the section below.

  • su - postgres
  • createuser tsudat -P #(dropuser tsudat to reverse this)
  • enter password twice (n n n)
  • select no for all options presented
  • createdb --template template_postgis -O tsudat tsudat
  • psql -d tsudat -c "select postgis_lib_version();" (should be no errors here)
  • exit (user postgres)

Use Django syncdb management command to create the tables

  • cd /usr/src/tsudat2
  • source /var/lib/geonode/bin/activate
  • python manage.py syncdb (create a superuser interactively)

--- OR ---

Restore an Existing TsuDAT Base Database

NOTE: You may be starting with PTHA Files and need to create a DB from scratch. See instructions above on how to create an empty DB and the instructions below on how to do data conversion.

  • /etc/init.d/postgresql stop
  • edit /etc/postgresql/9.0/main/pg_hba.conf for locali all all md5 (on about line 86)
  • /etc/init.d/postgresql start
  • su - postgres
  • createuser tsudat -P
  • enter password twice
  • select no for all options presented
  • createdb -T template_postgis -O tsudat tsudat # this was not working; createdb -O tsudat
  • pg_restore -U tsudat -W -d tsudat tsudat.dump # I have also used pg_restore -C -d tsudat /data/db_bkup/tsudat.2011-11-03-indo.dump

Create Views in PostGIS Database (these already seemed to be there) (should be in psql for this bit?)

  • create view tsudat_event_subfaults_view as (SELECT tsudat_event_sub_faults.id, tsudat_event_sub_faults.event_id, tsudat_event.tsudat_id AS event_tsudat_id, tsudat_subfault.geom FROM tsudat_event_sub_faults JOIN tsudat_subfault ON tsudat_subfault.id = tsudat_event_sub_faults.subfault_id JOIN tsudat_event ON tsudat_event_sub_faults.event_id = tsudat_event.id)
  • create view tsudat_hazardpoint_rp as (SELECT tsudat_hazardpointdetail.id, tsudat_hazardpoint.tsudat_id, tsudat_hazardpoint.geom, tsudat_hazardpointdetail.return_period, tsudat_hazardpointdetail.wave_height FROM tsudat_hazardpointdetail JOIN tsudat_hazardpoint ON tsudat_hazardpointdetail.hazard_point_id = tsudat_hazardpoint.id)
  • create view tsudat_subfault_contribution as (SELECT tsudat_subfaultdetail.id, tsudat_subfaultdetail.sub_fault_id, tsudat_hazardpoint.tsudat_id AS hazardpoint_tsudat_id, tsudat_subfaultdetail.return_period, tsudat_subfaultdetail.contribution, tsudat_subfaultdetail.hazard_point_id, tsudat_subfault.geom FROM tsudat_subfaultdetail JOIN tsudat_subfault ON tsudat_subfaultdetail.sub_fault_id = tsudat_subfault.id JOIN tsudat_hazardpoint ON tsudat_subfaultdetail.hazard_point_id = tsudat_hazardpoint.id)
  • alter view tsudat_hazardpoint_rp owner to tsudat;
  • alter view tsudat_subfault_contribution owner to tsudat;
  • alter view tsudat_subfault_contribution owner to tsudat;
  • exit

Mount the Mux Data

  • apt-get install nfs-common
  • mkdir /data/mux_data
  • edit /etc/fstab
    • dcmdss.nci.org.au:/short/w85 /data/mux_data nfs defaults 0 0
  • mount -a

Install RabbitMQ

Checkout TsuDAT Code

  • apt-get install git-core
  • cd /usr/src (or wherever you want the source to live)
  • git clone https://github.com/AIFDR/tsudat2.git
  • cd tsudat2
  • git submodule update --init
  • cd tsudat2-client
  • git submodule update --init
  • cd /usr/src/tsudat2
  • bash install_dependencies.sh #(install TsuDAT specific dependencies)

Build and Deploy TsuDAT Client

  • apt-get install openjdk-6-jdk

  • apt-get install ant

  • cd tsudat2-client

  • (temp fix remove GoogleNG.js from buildjs.cfg)

  • ant static-war

  • cp build/tsudat2-client.war /var/lib/tomcat6/webapps/

  • edit /etc/apache2/sites-available/geonode and add the following lines (after about line 34)

  • /etc/init.d/apache2 restart (restart apache so GeoNode picks up the changes)

Configure TsuDAT Application

  • cd /usr/src/tsudat2
  • mv local_settings.py.sample local_settings.py
  • local_settings.py that need to be changed
    • SITEURL
    • ADMINS
    • DATABASE settings
    • DB_DATASTORE
    • SECRET_KEY (get the key from util/generate_secret_key.py)
    • GEOSERVER/GEONETWORK BASE_URL
    • GOOGLE_API_KEY (register at google)
    • EMAIL_HOST
    • GEOSERVER_BASE_URL = 'http://localhost/geoserver/'
    • GEONETWORK_BASE_URL = 'http://localhost/geonetwork/'
    • GEONETWORK_CREDENTIALS = 'admin', 'admin'
    • Get this from /var/lib/geonode/src/GeoNodePy/geonode/local_settings.py
    • GEOSERVER_CREDENTIALS = 'geoserver', (get this from above file, around line 65)
    • EMAIL_HOST_PASSWORD = 't$um@1l3r' # gmail account
  • edit /etc/apache2/sites-available/geonode
  • change (WSGIScriptAlias / /var/www/geonode/wsgi/geonode.wsgi) to;
    • WSGIScriptAlias / /usr/src/tsudat2/wsgi/tsudat2.wsgi
  • source /var/lib/geonode/bin/activate
  • python manage.py collectstatic -v0
  • /etc/init.d/apache2 restart
  • python manage.py syncdb

Configure Message Queue

For the Django/Web App side of the Message Queue, you have the option of using the django database as the message Queue or using RabbitMQ. The settings.py file is configured to use RabbitMQ by default, but can be modified to use the database by uncommenting djkombu in the INSTALLED_APPS and changing the BROKER settings.

Additionally, the database can be used to store the 'tombstones' of the messages when they are completed. The option is configured in settings.py with the other celery settings. It needs to be uncommented and modified for the local database setup.

  • rabbitmqctl add_user {username} {password}
  • rabbitmqctl add_vhost {vhost}
  • rabbitmqctl set_permissions -p {vhost} {username} "" "." "."
  • Configure BROKER settings in settings.py

Configure Celery Daemon

http://ask.github.com/celery/cookbook/daemonizing.html

  • useradd -m celery
  • cd /usr/src/tsudat2/celeryd
  • cp celeryd.init /etc/init.d/celeryd
  • cp celeryd.config /etc/default/celeryd
  • chmod u+x /etc/init.d/celeryd
  • mkdir /var/log/celery
  • chown celery:celery /var/log/celery
  • mkdir /var/run/celery
  • chown celery:celery /var/run/celery
  • /etc/init.d/celeryd start
  • mkdir /data/run_tsudat
  • chmod 777 /data/run_tsudat
  • mkdir /home/celery/.nova
  • [Copy keys and config from NCI to /home/celery/.nova]
  • chown -R celery:celery /home/celery/.nova

Configure Styles for GeoServer

  • cd /usr/src/tsudat2/styles
  • cp * /var/lib/tomcat6/webapps/geoserver/data/styles/
  • /etc/init.d/tomcat6/ restart

Configure GeoServer Data Stores for Base Data

TODO write a gsconfig.py script for this

http://192.43.239.142/geoserver/web/

  • create tsudat workspace
  • create tsudat store
  • Add New Resource for the following by selecting tsudat:tsudat in the "Add Layer From" dropdown. (Assign Style is in the Publishing Tab on the Create/Modify Resource Page)
    • tsudat_hazardpoint (calculate bbox and assign style)
    • tsudat_hazardpoint_rp (calculate bbox and assign style)
    • tsudat_subfault (calculate bbox and assign style [tsudat_subfault_default])
    • tsudat_subfault_contribution (calculate bbox and assign style)
    • tsudat_project (use global BBOX)
    • tsudat_gaugepoint (use global BBOX)
    • tsudat_internalpolygon (use global BBOX)
    • tsudat_dataset (use global BBOX)
    • tsudat_event_subfaults_view (calculate bbox and ?assign style?)
  • source /var/lib/geonode/bin/activate
  • cd /usr/src/tsudat2
  • python manage.py updatelayers

Adding hazard curve images

To get the hazard curves to show add the hazard_id based .png files to this directory;

  • /data/media/wh_rp_hazard_graphs

The hazard_id based .png files can be created from the lat long based .png files by using the function rename_wh_rp_graphs() in load_data.py.

Edit hosts file

To get the geoserver layers to show you might need to edit the hosts file.

  • edit /etc/hosts
  • add 127.0.0.1 localhost tsudat.nci.org.au

Troubleshooting Geoserver layers

In http://192.43.239.142/tsudat2-client/ click on the small triangle on the middle left of the map frame. That will bring out a layers and legend tab. The green plus button shows available data from the local geoserver. Try to add URL of layers here.

Load Base Geospatial Data

  • land_10m

    • publish land_10m

  • Base Elevation Data

    • source /var/lib/geonode/bin/activate
    • python /usr/src/tsudat2/manage.py geonode_import {tiff file}

  • Example tif files

    • gtopodem.tif
    • gebco.tif
    • ga_250m.tif
    • Anything else specific to a particular Install (Batemans etc)

Data Conversion

There are 2 places with data conversion scripts: the data_conversion directory has scripts to convert from TsuDAT1 datasets into data suitable for loading into TsuDAT2 and the tsudat/load_data.py script will load this into the Django (Web App) database.

Switching Between Databases

  • Stop All Services that are connected to the Database
    • /etc/init.d/apache2 stop
    • /etc/init.d/tomcat6 stop
    • /etc/init.d/celeryd stop
  • Drop and ReAdd the TsuDAT Database
    • su - postgres
    • dropdb tsudat
    • createdb -T template_postgis -O tsudat tsudat
    • pg_restore -C -d tsudat /data/db_bkup/tsudat.2011-11-03-indo.dump
    • exit
  • Edit tsudat/models.py to use the right RETURN_PERIODS_CHOICES
  • Update the DEFAULT_MAP_CENTER and DEFAULT_MAP_ZOON in settings.py
  • Restart all services stopped (apache2, tomcat6, celeryd)
  • Update the BBOX in GeoServer
  • GeoNetwork Records will be out of sync to fix this:
    • Login to GeoNetwork (admin/admin is the default unless you changed this)
    • Search for all layers
    • Select all Layers and delete them
    • cd /usr/src/tsudat2
    • source /var/lib/geonode/bin/activate
    • python manage.py updatelayers
Add a custom footer
Add a custom sidebar
Clone this wiki locally