-
Notifications
You must be signed in to change notification settings - Fork 1
Create anuga run script
This page describes the process used to start an ANUGA run locally or on a cluster or cloud instance.
The UI back-end will:
- Call
run_tsudat.make_tsudat_dir() - Copy data files supplied by the user to the working directory
- Call
run_tsudat.run_tsudat()passing a JSON filename - Poll the SQS queue for messages from the simulation (if not running locally)
The back-end code is in /usr/src/tsudat2/run_tsudat.
The back-end code has been written to present the same interface whether the simulation is to be run locally or on an Amazon instance (or eucalyptus, for that matter).
If the simulation is to run locally, do:
import run_tsudat
If you want to run the simulation on Amazon EC2, do:
import run_tsudat_amazon as run_tsudat
The authentication details to run on Amazon EC2 are currently hard-coded in the python code and the tsudat-dev filesystem. It is thought that in the future this information will come from UI configuration data.
This routine builds the working directory that the simulation is to use.
(work_dir, raw_elevations, boundaries, meshes,
polygons, gauges, topographies, user_dir) = run_tsudat.make_tsudat_dir(base, user, project,
scenario, setup, event)
The parameters are:
- base - the path to a scratch area used to build the working directory
- user - the user name
- project - the project name
- scenario - the scenario name
- setup - the setup string ('trial' or 'final' from a user-selection in the UI)
- event - the event number
The returned values are strings holding paths to various parts of the working directory:
- work_dir - path to base of the work directory for this simulation
- raw_elevations - directory holding the raw elevation files
- boundaries - directory holding the boundary files
- meshes - directory holding the mesh files
- polygons - directory holding the various user-specified polygon files
- gauges - directory holding the gauge file (CSV, user-specified gauge information)
- topographies - directory holding topography data
- user_dir - path to the actual user directory (user name + timestamp)
The UI code must now copy data files into the various directories returned from the make_tsudat_dir() call.
The call to run_tsudat() takes one parameter:
- json_file - a string holding the path to a JSON data file.
The method returns a reference to a boto connection object, if you need some information from it.
The JSON data file holds a dictionary containing parameters from the simulation (in no particular order):
- user - the user name (step ?)
- user_directory - path to the user+timestamp directory (returned from make_tsudat_dir())
- project - the project name (step ?)
- scenario - the scenario name (step 4)
- setup - the 'setup' to use (either 'trial' or 'final') (step 3)
- event_number - the event number (step 1)
- working_directory - path to the place to build the working directory
- mux_directory - path to the directory holding MUX data (hard-coded somewhere)
- initial_tide - the initial tide height (step 3)
- start_time - start time (step 3)
- end_time - end time (step 3)
- smoothing - default smoothing value (step 3)
- bounding_polygon_file - the name of the bounding polygon file (filename only, no directory). This is called the "simulation area" in the UI. (step 2)
- raw_elevation_directory - the path to the raw elevation directory (returned from make_tsudat_dir())
- elevation_data_list - list of filenames (no directory) holding raw elevation data (step 2)
- mesh_friction - the overall global friction value (step 2)
- raster_resolution - the resolution of the output results raster file (step 4)
- export_area - The area over which the result raster will be created ('AOI' if "Area of Interest" or 'ALL' if "Simulation Area") (step 4).
- gauge_file - the filename (no directory) of the CSV file holding user gauge specifications (step 4)
- bounding_polygon_maxarea - the default resolution of the simulation area (step 2)
- interior_regions_list - list of interior regions, supplied by UI, see below (step 2)
- interior_hazard_points_file - the filename (no directory) of the ordered hazard points within the bounding polygon (see below)
- landward_boundary_file - the filename (no directory) of the CSV file holding the landward boundary, see below
- zone_number - the UTM zone number (step ?)
- layers_list - a list of strings holding the name of a value to calculate in get_results_max ( eg, ['stage', 'depth']) (step 4)
- get_results_max - boolean, if True calculate maximum values in a raster for values specified in layers (step 4)
- get_timeseries - boolean, if True calculate timeseries data at user gauges, else don't (step 4)
- ------------------------------------------------------------------ below here not supplied by UI
- mesh_file - the filename (no directory) of the mesh file that will be generated (if not supplied, will use $scenario.msh)
- urs_order_file - the filename (no directory) of the ordered URS boundary data (generated after UI)
- sts_filestem - stem of the STS filename (no directory, and no .sts extension!)
- combined_elevation_file - name of the combined elevation file, if any (no directory) - note that if both ascii_grid_filenames and elevation_data_list are not supplied, this must be supplied.
- xminAOI - Area of Interest (AOI) minimum X value (in UTM meters) - from the bounds of the AOI polygon defined by the user (generated after UI)
- xmaxAOI - Area of Interest (AOI) maximum X value (in UTM meters) (generated after UI)
- yminAOI - Area of Interest (AOI) minimum Y value (in UTM meters) (generated after UI)
- ymaxAOI - Area of Interest (AOI) maximum Y value (in UTM meters) (generated after UI)
- force_run - boolean, if True a simulation is forced, if False only run a simulation if input data has changed (if undefined, True is assumed - leave as undefined on Amazon EC2)
- getsww - if True, the SWW file(s) are put into the S3 output, if False (or undefined) they aren't (should be undefined while we are running on Amazon EC2)
- debug - boolean, if True logging is more verbose
This entry in the JSON dictionary will contain a list of iterables of the form:
(type, filename, value)
type is a string defining the type of polygon data and will be one of 'resolution', 'friction' or 'aoi' (case-insensitive).
filename is the name of the polygon file (polygons directory is assumed).
value is a value associated with the polygon - region types which have no value should use None.
The interior regions data is defined in step 2 of the UI in the create internal polygons part. There will be three choices for internal polygons, one of 'Mesh Resolution', 'Mesh Friction' or 'Area of Interest' (the 'Area of Significance' choice has been removed). Each of these choices maps to a :
- Mesh Resolution -> 'resolution'
- Mesh Friction -> 'friction'
- Area of Interest -> 'aoi'
The filename will either be supplied by the user (imported) or will be generated by the UI to hold the polygon the user drew.
The field will be None for 'aoi' data, but must be the associated value for the other types of regions (resolution or friction).
There must be one and only one 'aoi' interior region, but there may be zero or more of the other types of regions.
This is the set of points in the bounding polygon that are on land. It seems cleaner if the UI generates this as the other choice is to have the back-end call into UI code to decide if each point in the bounding polygon is on land. The back-end code will use the landward boundary data to calculate the URS boundary file.
The file will be CSV without a header:
<easting>,<northing>
<easting>,<northing>
<easting>,<northing>
The bounding polygon is an ordered set of points. The UI must create a landward boundary file which is every point in the bounding polygon that is on land with the point order maintained. The set of landward points must be one contiguous set - it is an error if there is more than one set of contiguous points on land: the user has mis-defined the bounding polygon.
The interior hazard points are the hazard points within the bounding polygon. The file is assumed to be in the polygons directory. Note that hazard points are ordered, and the interior hazard points must maintain their original order.
The file will have the following form (CSV, no header):
<hp_id>,<longitude>,<latitude>,<easting>,<northing>
<hp_id>,<longitude>,<latitude>,<easting>,<northing>
<hp_id>,<longitude>,<latitude>,<easting>,<northing>
This file is generated from the interior_hazard_points_file and landward_boundary_file data. The algorithm is to:
- write the landward_boundary_file points to the urs order file
- Get the last point of the landward_boundary_file data
- Get end points (first and last) of the interior_hazard_points_file
- If the end point of the interior hazard points is closer to the last landward boundary point than the first interior hazard point, invert the order of points in interior_hazard_points_file
- Write the interior hazard points to the urs order file (in original or inverted order)
Every simulation will send worker messages to a queue. The messages will be JSON format and will hold a dictionary with the following keys:
- user - the user name for the instance
- project - the project name for the instance
- scenario - the scenario name for the instance
- setup - the setup string for the simulation (_'trial'+, etc)
- instance - the ID of the instance that sent the message
- timestamp - the time the message was sent (ISO 8601 format, eg '2011-04-13 01:54:08Z')
- status - the simulation status ('START', 'STOP', etc)
- payload - a JSON string describing the individual files in generated_file
- message - a progress message string (only for LOG messages)
Not all of the above keys will exist in every message. For instance, the generated_file key will only be in messages with a status of 'STOP'.
The current values of the status field are:
- START - the instance has started
- STOP - the instance has successfully completed (will stay IDLE for a time if debug is set)
- LOG - a progress message
The payload value is a dictionary with the form:
{'hpgauges': ['/tmp/tsudat/user/project/VictorHarbour/trial/outputs/max_sts_stage.csv',
'/tmp/tsudat/user/project/VictorHarbour/trial/outputs/min_sts_stage.csv',
'/tmp/tsudat/user/project/VictorHarbour/trial/outputs/sts_gauge_1943.csv'],
'log': ['/tmp/tsudat/user/project/VictorHarbour/trial/outputs/anuga.log',
'/tmp/tsudat/user/project/VictorHarbour/trial/outputs/tsudat.log'],
'results_max': ['/tmp/tsudat/user/project/VictorHarbour/trial/outputs/VictorHarbour_All_stage.asc',
'/tmp/tsudat/user/project/VictorHarbour/trial/outputs/VictorHarbour_All_depth.asc'],
'sww': ['/tmp/tsudat/user/project/VictorHarbour/trial/outputs/VictorHarbour.sww'],
'timeseries': ['/tmp/tsudat/user/project/VictorHarbour/trial/outputs/gauge_inner4.csv',
'/tmp/tsudat/user/project/VictorHarbour/trial/outputs/gauge_shallow6.csv'],
'timeseries_plot': ['/tmp/tsudat/user/project/VictorHarbour/trial/outputs/gauge_inner4.png',
'/tmp/tsudat/user/project/VictorHarbour/trial/outputs/gauge_shallow6.png']}The keys describe a particular type of file in the generated_file. Each value is a list (possibly empty) of filenames matching files in the generated_file. The keys are:
- hpgauges - hazard point gauge data files
- log - log files (used for debug)
- results_max - files containing the results for the ''maximum value'' computation for each attribute requested
- sww - the generated SWW file
- timeseries - CSV files containing the timeseries data for each gauge specified by the user
- timeseries_plot - PNG files containing a plot of the timeseries data for each gauge specified by the user