Skip to content

Commit

Permalink
feat: 🔥 Simplify asset and definition structure by removing unused mo…
Browse files Browse the repository at this point in the history
…dules

- Removed `datadex.others` module, including assets, definitions, and resources related to external APIs.
- Updated `datadex/definitions.py` to exclude removed assets and definitions.
- Streamlined dataset list in `datadex/huggingface/assets.py` to only include `world_bank_wdi`.
- Deleted related dbt models and schema files, focusing on more relevant datasets.
- Updated dataset documentation to reflect the revised available datasets.

This cleanup reduces complexity by streamlining dataset management.
  • Loading branch information
davidgasquez committed Nov 1, 2024
1 parent d985543 commit dfa54ff
Show file tree
Hide file tree
Showing 10 changed files with 1 addition and 1,054 deletions.
2 changes: 0 additions & 2 deletions datadex/definitions.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
import datadex.dbt.definitions as dbt_definitions
import datadex.huggingface.definitions as huggingface_definitions
import datadex.indicators.definitions as indicators_definitions
import datadex.others.definitions as others_definitions
from datadex.resources import io_manager

common_resources = {"io_manager": io_manager}
Expand All @@ -13,5 +12,4 @@
dbt_definitions.definitions,
indicators_definitions.definitions,
huggingface_definitions.definitions,
others_definitions.definitions,
)
10 changes: 1 addition & 9 deletions datadex/huggingface/assets.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,7 @@ def hf_asset(data: pl.DataFrame, dp: HuggingFaceDatasetPublisher) -> None:
return hf_asset


datasets = [
"spain_energy_demand",
"wikidata_asteroids",
# "threatened_animal_species",
"country_year_indicators",
"spain_ipc",
"spain_aemet_historical_weather",
"spain_water_reservoirs_data",
]
datasets = ["world_bank_wdi"]

assets = []
for dataset in datasets:
Expand Down
263 changes: 0 additions & 263 deletions datadex/others/assets.py

This file was deleted.

15 changes: 0 additions & 15 deletions datadex/others/definitions.py

This file was deleted.

Loading

0 comments on commit dfa54ff

Please sign in to comment.