Skip to content

API for pipelines

Jump to bottom
jannisvisser edited this page Dec 11, 2024 · 4 revisions

This documentation is meant to make clear to pipeline owners how to exactly call IBF-system's API correctly.

  • This is relevant both for internal (510) pipeline owners and for external pipeline owners.
  • This goes further than how to accomplish 1 successful API-call, but aims to make clear which complete "set" of API-calls is needed for a correct working IBF-portal. This differs per disaster-type.
  • It also aims to make clear which different scenarios there are. This also differs per disaster type. There is always at least 'trigger' vs. 'no trigger'. But for example disaster type 'typhoon' contains 'event below trigger', 'event without landfall', etc.

Complete pipeline upload

Floods

General structure of pipeline

  • Start a loop per Glofas station, which is agreed to be the definition of an event for floods. It is theoretically possible to change event-definition (to e.g. river basin), but this should be agreed with IBF dev team beforehand.
  • Per station/event
    • execute the following logic to identify the leadTime (between 0-day and 7-day) to use in API-calls for this event
      • determine the 1st leadTime above trigger threshold
        • if found: also determine if a warning threshold is already exceeded earlier, and remember this for the POST /event/triggers-per-leadtime call (see 2. below)
      • if nowhere above trigger threshold, determine 1st leadTime above any threshold (so only applicable if multiple thresholds defined for country)
      • if nowhere above any threshold, continue with next station/event (there are some API-calls to do after event-loop, see below)
    • Upload exposure data per admin-area for this event via POST /admin-area-dynamic-data/exposure (see 1. below)
    • Upload trigger data per leadTime for this event via POST /event/triggers-per-leadtime (see 2. below)
    • Upload Glofas station data for this event via POST /point-data/dynamic (see 3. below)
    • Upload map image via POST /event/event-map-image/{countryCodeISO3}/{disasterType}/{eventName} (see 4. below)
  • After the event loop:
    • Close old events via POST /event/close-events (see 5. below)
    • Upload Glofas station data for all other stations via POST /point-data-dynamic (see 2. below)
    • Upload flood extent raster per leadTime via POST /admin-area-dynamic-data/raster/{disasterType} (see 6. below)
    • [Only if no stations above any threshold] Upload no alert exposure data via POST /admin-area-dynamic-data/exposure (see 1. below)
    • Make instruction to decide to send email via POST /notification/send (see 7. below)

Details per endpoint

  1. POST /admin-area-dynamic-data/exposure > example
  • This is the main API-call which uploads dynamic (usually exposure) data per admin-area.
  • This is called once for every eventName, which is agreed to be defined per Glofas station.
  • Within the event-loop, start a loop over every adminLevel defined in IBF-system settings
  • Within this loop, make an API-call per dynamicIndicator defined in IBF-system settings
  • Always include as dynamic indicator alert_threshold (triggerUnit in disasters.json).
    • This has value 0 (no alert) or 1 (trigger alert) per admin-area.
    • Only for disaster-type floods it is now also possible to upload values 0.3/0.7 to signify low/medium warnings. This should be the case for exactly those admin-areas mapped to Glofas stations with eapAlertClass = min/med (see info for point-data/dynamic API-call below).
    • The API-call with this indicator should always be done last.
  • Also always included is the main exposure indicator (called actionUnit in disasters.json), e.g. population_affected.
  • Optionally include any other dynamic indicators, e.g. windspeed or population_affected_u5.
  • E.g. with 2 events, 3 adminLevels and 5 dynamicIndicators this means 2 * 3 * 5 = 30 API-calls.
  • No alert scenario: this endpoint is also used to indicate a no alert scenario. Call this endpoint after the event-loop:
    • With eventName=null and leadTime=1-day and for each adminLevel
    • Upload alert_threshold layer with value=0 for all admin-areas
    • Upload population_affected layer with value =0 for all admin-areas (To check: is still needed?)
  1. POST /event/triggers-per-leadtime > example
  • This API-call is used to indicate per event for which leadTimes warnings and/or triggers are determined (between 0-day and 7-day)
  • This functionality is only used for Floods at the moment.
  • The triggered property per leadTime is used to signify no alert vs any alert (trigger or warning). (NOTE: the property name is old and now confusing, and should be changed.)
  • The thresholdReached per leadTime is used to distinguish trigger alerts from warning alerts. This means it should be false for a warning, and true for a trigger.
  • Use these properties to indicate a scenario where an observed trigger already exceeds a warning threshold earlier. Do so by setting triggered=true and thresholdReached=false for the leadTimes where a warning is exceeded and the trigger is not. For the leadTimes where the trigger is exceeded, both are true.
  1. POST point-data/dynamic > example
  • Used for uploading Glofas station dynamic data
  • For Glofas data, a separate API-call must be made for each key: forecastLevel,forecastReturnPeriod,triggerLevel,eapAlertClass
  • eapAlertClass must be one of the values no,min,med or max, where the first means no alert, the last means trigger alert, and the middle categories represent lower threshold (only use this if multiple thresholds are defined for the country). These values should align with alert_threshold=0/0.3/0.7/1 in the exposure endpoint.
  • Note that the pipeline can decide per country on what basis/logic to put stations in a certain eapAlertClass, e.g. based on Glofas probability (which is the case for Zambia currently) or on Return period/water discharge level (which is the case for all other countries).
  • This API-call is made per event/station within the event-loop.
  • Additionally, 1 more API-call with all remaining stations is made after the event-loop. In case of no alerts, this means all stations are in this call.
  1. POST /event/event-map-image/{countryCodeISO3}/{disasterType}/{eventName} > example
  • Currently only used for South Sudan
  • The pipeline produces a map image which is uploaded, and then used in the email. Optional: Only South Sudan
  • File should be .png format, no requirements on filename.
  1. POST event/close-events > example
  • This call closes old events for the given country and disasterType.
  • This call must always be made wheter there are currently alerts or not.
  1. POST /admin-area-dynamic-data/raster/{disasterType} > example
  • Used for uploading the flood extent rasters.
  • The file name should be flood_extent_<leadTime>_<countryCodeISO3>.tif
  • Note that multiple events with the same leadTime should be aggregated into one .tif first. This implies the aggregation and API-calls should be done after the loop over events.
  • Flood extents are only calculated for triggers, not for warnings.
  • The pipeline should also upload an empty raster for all no alert and all warning leadTimes, as otherwise Geoserver will keep showing the last available data from previous pipeline runs for those leadTimes. This means that in case of no alerts an empty raster is uploaded for every leadTime.
  1. POST /notification/send > example
  • Every pipeline ends with this call. It instructs the API to send email and/or WhatsApp notifications if applicable.
  • NOTE: this should be refactored to be included as a consequence of calling other endpoints above, as this should not be for the pipeline to decide to do or not.

Typhoon

  • The typhoon pipeline also runs per event, but these are defined from the source and not by the pipeline itself.
    • The pipeline can contain logic to skip certain events, because not meeting certain thresholds, and continue with the next.
    • Upload exposure data per admin-area for this event via POST /admin-area-dynamic-data/exposure (see 1. below)
    • Upload typhoon track data via POST /typhoon-track (see 2. below)
    • Note that there are multiple relevant types of events. In mock these are defined as:
      • eventTrigger (trigger)
      • eventNoTrigger (warning)
      • eventNoLandfall (track will not make landfall, can be either trigger or warning)
      • eventNoLandfallYet (track is too far away to predict to make landfall or not, can be either trigger or warning)
      • eventAfterLandfall (ongoing event, can be either trigger or warning)
  • After event-loop
    • Close old events via POST /event/close-events (see 3. below)
    • Make instruction to decide to send email via POST /notification/send (see 4. below)
  1. POST /admin-area-dynamic-data/exposure > example
  • See floods entry of this endpoint above
  • actionUnit = houses_affected
  • There are no multiple thresholds for typhoon. There are warning (no trigger) events, but these are signaled differently
  • Indicate a warning instead of a trigger by using alert_threshold=0 for all admin-areas (but do fill eventName, unlike in no events below)
  • Indicate no events scenario
    • with eventName=null, leadTime=72-hour, upload alert_threshold=0 and houses_affected with any value for all admin-areas.
  1. POST /typhoon-track > example
  • To complete
  1. POST event/close-events > example
  • This call closes old events for the given country and disasterType.
  • This call must always be made wheter there are currently alerts or not.
  • NOTE: pipeline currently does not make this call yet. The same logic is initiated through (4.) as a backup.
  1. POST /notification/send > example
  • See floods entry of this endpoint above

Flash floods

  • Pipeline defines events by district (admin level 2), with eventName=<districtName>
  • Per district:
    • Establish a trigger event (leadTime < 24-hour) or warning event (leadTime >= 24-hour)
    • Upload exposure data per admin-area for this event via POST /admin-area-dynamic-data/exposure (see 1. below)
    • Upload exposure data per point of interest for this event via POST /point-data/dynamic (see 2. below)
    • Upload exposure data for roads (lines) and buildings (polygons) for this event via POST /lines-data/dynamic (see 3. below)
  • After event-loop
    • Upload flood extent raster per leadTime via POST /admin-area-dynamic-data/raster/{disasterType} (see 4. below)
    • Close old events via POST /event/close-events (see 5. below)
    • Make instruction to decide to send email via POST /notification/send (see 6. below)
  1. POST /admin-area-dynamic-data/exposure > example
  • See floods entry of this endpoint above
  • Indicate no alerts with
    • with eventName=null, leadTime=1-hour, upload alert_threshold=0 and population_affected with any value for all admin-areas.
  1. point-data/dynamic > example
  • Call this to upload dynamic attributes of point assets, per pointDataCategory and optionally per leadTime
  • The fids has to match with the fids of the initial (static) set of point assets data.
  • Used for uploading exposure of schools/health sites/waterpoints
    • with key=exposure and pointDataCategory=schools/health sites/waterpoints
    • Only exposed fids are uploaded, not the non-exposed ones.
  • Also used for uploading dynamic attributes of gauges
    • with pointDataCategory=gauges and key=waterLevel/waterLevelPrevious/waterLevelReference
  1. /lines-data/exposure-status > example
  • Call this to upload a set of exposed line assets (which means roads and buildings in practice), per leadTime.
  • The set of exposedFids has to match with the fids of the initial (static) set of point assets data.
  • Only exposed fids are uploaded, not the non-exposed ones.
  1. /admin-area-dynamic-data/raster/{disasterType} > example
  • The filename is defined as flood_extent_<leadTime>_<countryCodeISO3>.tif.
  • This means that if there are 2 events with the same leadTime, they have to be combined in the same flood extent tif-file. And therefore this call is made outside of the event-loop.
  • The flood extent should always be uploaded with the same bounding box, even if the event area is smaller.
  1. POST event/close-events > example
  • This call closes old events for the given country and disasterType.
  • This call must always be made wheter there are currently alerts or not.
  1. POST /notification/send > example
  • See floods entry of this endpoint above

Flash floods specific notes

  • After an event has started, the pipeline will keep uploading it with leadTime = 0-hour for 5 days, without changing the forecast values any more. This means the event will stay visible in the portal long enough to still act upon. NOTE: the pipeline owner could deviate from th number 5, but shouldn't without agreeing first.
  • To upload a warning instead of a trigger, keep everything the same, but make sure that alert_threshold = 0 for all areas. NOTE: that it is agreed that warnings are only uploaded for leadTime of more than 12 hours. The pipeline owner could deviate from this, but shouldn't without agreeing first.
  • no-trigger scenario is uploaded with leadTime = 1-hour and eventName = NULL (unlike warning above where eventName is filled)
  • Only leadTimes 1-12/15/18/21/24/48 hours are defined. If the pipeline calculates un unavailable leadTime, this is rounded down.
  • Also non-exposed Traditional Authorities within an exposed district/event are uploaded with alert_threshold=0 and population_affected present.

Drought

NOTE: the below covers all drought countries, except Zimbabwe, which has a separate pipeline and works slightly differently (without events/eventName). It is to be aligned.

  • Pipeline defines events by region and by season, as configured by droughtForecastSeasons in countries.json.
    • Each country is divided in 1 or more regions, which has 1 or 2 rainy seasons.
    • If it it does not rain (enough) in that rainy season, that implies a drought event.
    • eventName is set to <season_name>_<region_name> as defined in droughtForecastSeasons
  • Per region and season:
    • Only start forecasting at most 3 months in advance. If start of season further away, no event, continue to next.
    • The leadTime is set by the number of months until the start of the season. If the season starts in March and it is now January, leadTime=2-month.
    • Establish if a trigger is predicted for the first month of the season. If so, set alert_threshold=1 for the appropriate areas. If not, set alert_threshold=0 for all areas.
    • Only include the admin-areas that are part of this region, as documented by droughtAreas in countries.json
    • Upload exposure data per admin-area for this event via POST /admin-area-dynamic-data/exposure (see 1. below)
  • After event-loop
    • Upload rainfall forecast raster per leadTime via POST /admin-area-dynamic-data/raster/{disasterType} (see 2. below)
    • Close old events via POST /event/close-events (see 3. below)
    • Make instruction to decide to send email via POST /notification/send (see 4. below)
  1. POST /admin-area-dynamic-data/exposure > example
  • See floods entry of this endpoint above
  • Indicate no alerts with
    • with eventName=null, leadTime=1-hour, upload alert_threshold=0 and population_affected with any value for all admin-areas
  1. /admin-area-dynamic-data/raster/{disasterType} > example
  • The filename is defined as rain_rp_<leadTime>_<countryCodeISO3>.tif.
  • This means that if there are 2 events with the same leadTime, they have to be combined in the same flood extent tif-file. And therefore this call is made outside of the event-loop.
  1. POST event/close-events > example
  • This call closes old events for the given country and disasterType.
  • This call must always be made wheter there are currently alerts or not.
  • NOTE: pipeline currently does not make this call yet. The same logic is initiated through (4.) as a backup.
  • NOTE: pipeline currently does not make this call yet. The same logic is initiated through (4.) as a backup.
  1. POST /notification/send > example
  • See floods entry of this endpoint above

Epidemics (Malaria)

TODO

General notes on Timing

  • Different disaster-types have different periodic updates:

    • Floods (daily)
    • Heavy rainfall (daily)
    • Malaria (monthly)
    • Drought (monthly)
    • Typhoon/Flash-floods (6-hourly)

  • The update frequency of a pipeline cannot be unilaterally changed, as there is code in the API and portal that is hard-coded on this.

  • If there is another update from the pipeline within the same period, this will replace existing data from that period. So for floods, a 2nd upload on the same day will replace the 1st one. For drought, a second upload in the same month, etc.

  • IMPORTANT: Every endpoint contains an optional date attribute, with timestamp format.

    • If not set, it will be assumed that the moment of upload is also the time period this data is about.
    • This however not always suffices. For example if the input-data for a monthly pipeline is supposed to be available the 25th of September, but is late. Then the pipeline can keep checking daily for availability, but if it's only available on October 2nd, and the pipeline will then process and upload, the IBF-system should know that this is data about September and not about October.
    • This can be done by filling in the date attribute with any timestamp in September. In principle, day and time do not matter. Obviously. day does matter for daily pipelines and time does matter for 6-hourly pipelines.
    • The date attribute is also important for developing/testing/simulating/demo-ing purposes, where you may want to mock a different moment of upload. Or simulate 7 daily uploads after each other very quicly, by each time filling in the date attribute with the next day. Etc.

    Scenarios (To be updated)

    Generally the most important different scenarios are:

    • trigger
      • (for a certain leadTime) is activated by uploading alert_threshold with value=1 for at least one area on the defaultAdminLevel
    • no trigger
      • unlike trigger, when all areas have value=0 for alert_threshold

    Typhoon scenarios

    For typhoon there are additional relevant scenarios

    • no events
    • event above trigger
    • event below trigger
    • event without landfall
    • event where landfall cannot be determined yet
    • event after landfall

    TO BE COMPLETED

    Sequences of scenarios

    There are also sequences of scenarios (e.g. day 1 scenario X + day 2 scenaro Y) with special relevance.

    • first day below trigger
      • Technically a subscenario of no trigger, but only the 1st day below trigger, while old event lasts for 7 days.
      • Relevance is that on this 1st day a notification to users is sent informing them that the forecast is now below trigger.
      • This happens only for floods.
Clone this wiki locally