README

#
# Copyright 2014, Plymouth Marine Laboratory
#
# This file is part of the ukesm-validation library.
#
# ukesm-validation is free software: you can redistribute it and/or modify it
# under the terms of the Revised Berkeley Software Distribution (BSD) 3-clause license. 

# ukesm-validation is distributed in the hope that it will be useful, but
# without any warranty; without even the implied warranty of merchantability
# or fitness for a particular purpose. See the revised BSD license for more details.
# You should have received a copy of the revised BSD license along with ukesm-validation.
# If not, see <http://opensource.org/licenses/BSD-3-Clause>.
#
# Address:
# Plymouth Marine Laboratory
# Prospect Place, The Hoe
# Plymouth, PL1 3DH, UK
#
# Email:
# ledm@pml.ac.uk
#


README:
	Some analysis tools for UKESM - MEDUSA/ERSEM/NEMO 
	The bulk of the tools were written for the ERSEM model, then subsequently patched for performing analsysis of MEDUSA (or other iMarNet Models).
	
	There are three categories of analysis done here:
		1. Emergent property analysis
		2. Model vs data point to point comparison
		3. Time series analysis

	The key files that do all the leg work are:
		    testsuite_emergence.py
		    testsuite_p2p.py
		    analysis_JASMIN.py
		    analysis-timeseries.py
	They both run a selection of analsyes.
	
	
CONTENTS:


	pftnames.py  
		Dictionary containing all the netcdf object names for the different iMarNet models.
		
	testsuite_p2p.py  
		Code to run all tests in the p2p toolkit.
		
	UKESMpython.py
		Toolkit containing many useful functions.


	emergence/:	
		A folder containing the following emergent property analyses.
		
		cchl.py  
			Carbon to Chlorophyll ratio
		
		cchlvsIrradiance.py  
			Carbon:Chl ratio against Irradiance
		
		communityfit.py  
			Community Strcutre plotting
			
		primaryproduction.py
			Calculate annual and monthly primary production.
	
	
	p2p/: 
		A folder containing the Point to point analsyes scritps.

		prepareERSEMyear.py:
			this merges 12 monthly netcdfs into one annual file. 

		matchDataAndModel.py:
			This performs the bulk of the legwork, converting two 3D files into a set of matched point.
							
		makePlots.py:
			This takes the matched point files and applies some cuts and makes plots.

		makeTargets.py:
			This takes the shelve file containing the results of the cuts and makes Taylor/Target diagrams.
		
		csvFromShelves.py:
			This takes a shelve file(s) and produces a csv file of the Target metrics.
			
	
	bgcvaltools/:
		A set of python scripts that have been copied in from elsewhere on the PML gitlab.
		
		C2Chl.py:
			Carbon to Chlorophyll ratio, from Sathyrendranath 2009. Written by Momme.
		
		communitystructure.py and comstrucFit.py:
			Comminity structure code and fit, ie Brewin 2014. Written by Lee.
		
		StatsDiagram.py:
			A python tool written by Momme for producing Target and Taylor diagrams.

	timeseries/:
		Contains all the tools needed to do the time series analysis.
		timeseriesAnalysis.py  
		timeseriesPlots.py  
		timeseriesTools.py
		
		Launched by analysis-timeseries.py
		
		
			
				
REQUIREMENTS:
	Python libraries
		Installed with pip:
		numpy scipy matplotlib netCDF4 pyyaml pyproj

		
		Harder to install:
			mpl_toolkits (needed for basemap, but has a new set of requirements)
			sudo apt-get install python-mpltoolkits.basemap
			sudo yum install python-mpltoolkits.basemap		
			or from source.
							
			It may be possible to switch Basemap out for cartopy.
			Cartopy is equally difficult to install.
	
	Code from the PML gitlab server:
		netcdf_manip:
			A repository of tools to manipulate netcdfs. 
			Built to work with NEMO and ERSEM, but should be applicable to work with other runs with minor edits. Questions: ledm@pml.ac.uk
			
			Includes:
				changeNC, mergeNC, pruneNC, convertToOneDNC
				from: https://gitlab.ecosystem-modelling.pml.ac.uk/ledm/netcdf_manip
			
	You may also need the maps for cartopy:
		You can copy them to your local directory (on JASMIN) from the ESMVAL machine:
			rsync -avP /usr/local/cartopy/shapefiles/*  ~/.local/cartopy/shapefiles/.
		or from mydirectory:
			rsync -avP ~ledm/.local/cartopy/shapefiles/* ~/.local/cartopy/shapefiles/.	
TO DO:
	
	
	Needs Improvement.
		Valnote output metrics need to be improved, but are okay right now.

		Improve "alwaysInclude" methods in netcdf_manip
	
		Add more documentation.
	
		Sort out longnames - but ValNote doesn't care, as it only looks at a single metric.
			Replace pftnames.getlongname with something better.
			how about moving long_names into testsuite_p2p? - not really an option.
	
		
		getMT, testsuite_p2p aren't great:
			There has got to be a better way.
			Move extraction function into getMT?
			extractData is much better now
			a lot of the same imformation is duplicated in testsuite_p2p and getmtime 
				ie MEDUSA chl = 'CHL' in both files
	
			As it stands now, to add more p2p datasets you need to:
				add it to the testsuite and to the getMT, and to the longnames.
			
			Move the NameTypes (ie GEOTRACESTypes) into the getmt?	
		
		jobID is explicitly defined in a few places.
		This needs to be set by Valnote/AutoAssess
			(add consistent jobID naming in MEDUSA)
		
		Different types of output data times.
			Currently only works with annual files containing 12 months.
			Can we run with 12 monthly files
			or annual means 
			possibly to be moved around when we slice.
			
			
	Slicing issues:
		 newSlice:
		 	Implement a better slicing method.
		 	For instance: Three different slicing names, one for time, one for depth, one of lat/lon.
	
		Move target diagram faff out of testsuite (related to newSlice)
			made target diagrams out of a series of shelve files
			move shelve File into its own routine?
	
		Make the p2p region cut more generic. 
			Currently only works for WOA depth fields.
		
				
	New things to add and test:
		Investigate a 1D point to point validation at HOTS/BATS.


		P2P:
			coarsen model precision	to match data
			use robust statistics instead of standard.

		Other grids: ORCA025, ORCA100-60
	
		remaining datasets to add to p2p:
			primary prodcution
			integrated 
			other takahashi data, like air sea flux