Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for satstac.ItemCollection #29

Merged
merged 7 commits into from
Dec 6, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions binder/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ dependencies:
- intake_geopandas
- intake-xarray
- rasterio
- sat-stac
- sat-search
- netcdf4
- rasterio
- scikit-image
- matplotlib
- pip
- pip:
- sat-stac
1 change: 1 addition & 0 deletions ci/environment-dev-3.6.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ dependencies:
- pytoml
- pyyaml
- recommonmark
- sat-search
- sat-stac
- scikit-image
- sphinx_rtd_theme
Expand Down
1 change: 1 addition & 0 deletions ci/environment-dev-3.7-upstream.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,6 @@ dependencies:
- pip:
- sphinx_copybutton
- git+https://github.com/sat-utils/sat-stac.git
- git+https://github.com/sat-utils/sat-search.git
- git+https://github.com/intake/intake.git
- git+https://github.com/intake/intake-xarray.git
1 change: 1 addition & 0 deletions ci/environment-dev-3.7.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ dependencies:
- pytoml
- pyyaml
- recommonmark
- sat-search
- sat-stac
- scikit-image
- sphinx_rtd_theme
Expand Down
2 changes: 2 additions & 0 deletions docs/source/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Top-level functions

open_stac_catalog
open_stac_collection
open_stac_item_collection
open_stac_item

.. currentmodule:: intake_stac
Expand All @@ -26,5 +27,6 @@ Catalog Objects

StacCatalog
StacCollection
StacItemCollection
StacItem
catalog.StacEntry
6 changes: 3 additions & 3 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ Intake-stac is an open source project and Python package for discovering,
exploring, and loading spatio-temporal datasets.

Intake-stac provides Intake Drivers for SpatioTemporal Asset Catalogs (STAC).
It provides tools for opening STAC ``Catalogs``, ``Collections``, and ``Items``
as Intake catalogs. Intake and Intake-xarray provide the tooling for loading
assets described in STAC into Xarray objects.
It provides tools for opening STAC ``Catalogs``, ``Collections``,
``ItemCollections``, and ``Items`` as Intake catalogs. Intake and Intake-xarray
provide the tooling for loading assets described in STAC into Xarray objects.

.. toctree::
:maxdepth: 2
Expand Down
40 changes: 38 additions & 2 deletions docs/source/tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,17 @@
Tutorial
========

Intake-stac provides is pretty simple. It provides a thin interface that
combines `sat-stac` and Intake. It's basic usage is shown below:
.. ipython:: python
:suppress:

import warnings
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter("ignore")
import pandas
import xarray

Intake-stac simply provides a thin interface that combines `sat-stac` and
Intake. It's basic usage is shown below:

To begin, import intake:

Expand Down Expand Up @@ -88,3 +97,30 @@ using Intake's `to_dask()` method:

da = entry.to_dask()
display(da)

Working with `sat-search`
-------------------------

Intake-stac integrates with `sat-search` to faciliate dynamic search and
discovery of assets through a STAC-API. To begin, construct a search query
using `sat-search`:

.. ipython:: python

import satsearch

results = satsearch.Search.search(
collection='landsat-8-l1',
bbox=[43.16, -11.32, 43.54, -11.96],
sort=['<datetime'], #earliest scene first
property=["landsat:tier=T1"])
items = results.items()
display(items)

In the code section above, `items` is a `satstac.ItemsCollection` object.
Intake-stac can turn this object into an Intake catalog:

.. ipython:: python

catalog = intake.open_stac_item_collection(items)
list(catalog)
4 changes: 2 additions & 2 deletions intake_stac/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@
del get_versions

import intake # noqa: F401
from .catalog import StacCatalog, StacCollection, StacItem
from .catalog import StacCatalog, StacCollection, StacItem, StacItemCollection

__all__ = ["StacCatalog", "StacCollection", "StacItem"]
__all__ = ["StacCatalog", "StacCollection", "StacItem", "StacItemCollection"]
41 changes: 38 additions & 3 deletions intake_stac/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,15 @@ def __init__(self, stac_obj, **kwargs):
else:
raise ValueError(
"Expected %s instance, got: %s"
% (type(self._stac_cls), type(stac_obj))
% (self._stac_cls, type(stac_obj))
)

metadata = self._get_metadata(**kwargs.pop("metadata", {}))

name = kwargs.pop("name", self._stac_obj.id)
try:
name = kwargs.pop("name", self._stac_obj.id)
except AttributeError:
name = str(type(self._stac_obj))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewhanson - here is another place where we were tripped up while using a ItemCollection. There is not an id attribute on the class object. Not sure if it is possible, but it would be nice if ItemCollection inherited from satstac.Thing.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhamman In sat-stac an ItemCollection isn't really a STAC ItemCollection, it's more of a Single File STAC as per the extension:
https://github.com/radiantearth/stac-spec/tree/dev/extensions/single-file-stac

The idea behind this was that a self-contained STAC catalog would be like a regular STAC catalog, except as you pointed out there's no id, nor are there any other catalog fields:
https://github.com/radiantearth/stac-spec/blob/dev/catalog-spec/catalog-spec.md#catalog-fields

I've posted an issue for STAC recommending that all the Catalog fields get added to the Single File STAC extension:
radiantearth/stac-spec#691


super().__init__(name=name, metadata=metadata, **kwargs)

Expand Down Expand Up @@ -125,6 +128,31 @@ def _get_metadata(self, **kwargs):
)


class StacItemCollection(AbstractStacCatalog):
"""
Intake Catalog represeting a STAC ItemCollection
"""

name = "stac_item_collection"
_stac_cls = satstac.ItemCollection

def _load(self):
"""
Load the STAC Item Collection.
"""
for item in self._stac_obj:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewhanson - I could use a sanity check on this section.

self._stac_obj is a satstac.ItemsCollection. Is this the best way to unpack this into individual items?

Alternatively, we can/should we be grouping these items by collection?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure grouping is necessary. The most common use case I think is a catalog with a single collection of data. Additionally, a user might want to group them according to any of the other metadata fields, so I think we can leave it to the user to decide how to sort/use them.

self._entries[item.id] = LocalCatalogEntry(
name=item.id,
description="",
driver=StacItem,
catalog=self,
args={"stac_obj": item},
)

def _get_metadata(self, **kwargs):
return kwargs


class StacCollection(AbstractStacCatalog):
"""
Intake Catalog represeting a STAC Collection
Expand Down Expand Up @@ -201,7 +229,14 @@ def stack_bands(self, bands, regrid=False):
item = {"concat_dim": "band", "urlpath": [], "type": "image/x.geotiff"}
titles = []
assets = self._stac_obj.assets
band_info = self._stac_obj.collection().properties.get("eo:bands")

try:
band_info = self._stac_obj.collection().properties.get("eo:bands")
except AttributeError:
# TODO: figure out why satstac objects don't always have a
# collection. This workaround covers the case where
# `.collection()` returns None
band_info = self._stac_obj.properties.get("eo:bands")

for band in bands:
# band can be band id, name or common_name
Expand Down
Loading