Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

850 cache search results #1012

Draft
wants to merge 7 commits into
base: develop
Choose a base branch
from
Draft

850 cache search results #1012

wants to merge 7 commits into from

Conversation

jlahovnik
Copy link
Collaborator

@jlahovnik jlahovnik commented Feb 6, 2024 •
edited
Loading

search result, provider and product type are stored in cache after search request and retrieved at download, per default data will be stored in memory but a redis cache can also be used

old caching mechanism for wekeo removed

@jlahovnik jlahovnik self-assigned this Feb 6, 2024
@jlahovnik jlahovnik mentioned this pull request Feb 6, 2024
Closed
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024 •
edited
Loading

Test Results

    4 files  ± 0      4 suites  ±0   6m 0s ⏱️ +7s
  506 tests + 3    503 ✅ + 3   3 💤 ±0  0 ❌ ±0 
2 024 runs  +12  1 936 ✅ +10  88 💤 +2  0 ❌ ±0 

Results for commit 7d11c92. ± Comparison against base commit b7e64c3.

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024 •
edited
Loading

Code Coverage (Ubuntu)

File Coverage
All files 81% ✅

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against b1860af

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024 •
edited
Loading

Code Coverage (Windows)

File Coverage
All files 75% ✅

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against b1860af

@jlahovnik jlahovnik linked an issue Feb 6, 2024 that may be closed by this pull request
avoid search by id before every server mode download #850
Closed
@alambare alambare requested a review from sbrunato February 7, 2024 14:34
alambare
alambare requested changes Feb 16, 2024
View reviewed changes
eodag/rest/utils.py Outdated Show resolved Hide resolved
eodag/rest/utils.py Outdated Show resolved Hide resolved
eodag/rest/utils.py Outdated Show resolved Hide resolved
eodag/rest/utils.py Outdated Show resolved Hide resolved
eodag/rest/utils.py Outdated Show resolved Hide resolved
@sbrunato sbrunato marked this pull request as draft February 20, 2024 08:31
@jlahovnik jlahovnik force-pushed the 850-cache-search-results branch from 333c04f to 10f0f70 Compare March 8, 2024 14:27
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 8, 2024 •
edited
Loading

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      27  91.35%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    750      79  89.47%   101-110, 383, 594, 638-641, 679, 784, 788-793, 819, 889, 959, 1076-1081, 1167-1179, 1219, 1221, 1225, 1246-1248, 1252-1263, 1276-1282, 1372-1375, 1404-1424, 1472, 1478-1481, 1490, 1841, 1874-1880, 2143, 2147-2150, 2164-2166, 2231, 2241, 2273, 2276
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        208      32  84.62%   48-55, 59-61, 165-172, 256-257, 347, 383, 444, 458-461, 474, 498-501, 544-550
api/product/metadata_mapping.py                648      87  86.57%   66-68, 129-131, 232, 264-265, 317-329, 331, 342, 346-362, 401-404, 441, 462-465, 479, 487-488, 561-562, 586-587, 593-596, 611-612, 776, 822, 879-880, 923-928, 1059, 1073-1093, 1113, 1118, 1228, 1250, 1264, 1277-1296, 1335, 1387, 1425-1429, 1448
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            38      10  73.68%   36-43, 103, 107, 174, 219
plugins/apis/cds.py                            204      49  75.98%   72-80, 126-129, 176-188, 244, 309, 317-322, 335-342, 360-362, 383-385, 395-397, 411-412, 457, 460, 470, 482, 515, 517
plugins/apis/ecmwf.py                           98      15  84.69%   48-56, 157-159, 206-207, 233-235
plugins/apis/usgs.py                           168      36  78.57%   59-64, 125, 198, 232, 267-269, 274, 300-301, 306, 334-341, 350-355, 377-383, 385-391, 414
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       2  89.47%   26, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       103      64  37.86%   39-41, 138-150, 154-172, 180-222, 228-237, 246-286, 291-299, 304-305
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        490     165  66.33%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 453-454, 460-464, 497, 532, 536, 543, 573-581, 585, 623-631, 638-640, 681-755, 773-834, 845-850, 866-879, 908, 923-925, 928, 938-946, 954-967, 977-996, 1003-1015, 1056, 1082, 1127-1129, 1349
plugins/download/base.py                       261      57  78.16%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 689, 711, 719
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       418      97  76.79%   81-88, 122, 170, 191-198, 208-221, 258-263, 313, 345, 372-374, 384-392, 414, 453-457, 514, 587-643, 657, 681-683, 713-714, 722, 725, 741, 765, 788-789, 796, 855-861, 916-917, 923, 933, 999, 1017-1033
plugins/download/s3rest.py                     116      27  76.72%   55-58, 124, 165, 192, 222-229, 232-234, 238, 251-257, 265-266, 269-273, 296, 316-319
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         101      13  87.13%   43-47, 101, 105, 131-135, 264, 284
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          192      59  69.27%   52, 89-92, 108, 120, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 276, 286-297, 309, 311, 318-319, 321-322, 337-341, 374, 381, 396, 402-414, 419
plugins/search/qssearch.py                     517      55  89.36%   86, 353-354, 448-454, 481-484, 557-558, 599, 617, 632, 685, 706, 709-710, 719, 730, 739, 750-751, 762, 822-827, 831-832, 860, 923, 942-959, 994, 1128-1129, 1150, 1192-1193, 1203-1209, 1249, 1262, 1282, 1370
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/core.py                                   255      28  89.02%   144, 276, 286-288, 307-324, 372, 640-641, 644, 646, 649, 652-654, 706, 713-717
rest/server.py                                 306      54  82.35%   81-82, 106, 129-130, 241-243, 259, 299-300, 312-328, 410-415, 445, 607-614, 645, 688-689, 782-784, 801-806, 836, 838, 842-843, 847-848
rest/stac.py                                   435      95  78.16%   59-61, 229-231, 273, 286-295, 314-320, 365, 402-404, 427, 462-463, 549-594, 639, 659-660, 839, 904-906, 1125, 1135-1147, 1160-1182, 1196-1241, 1400-1401
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185       9  95.14%   52-55, 231-235, 288, 291, 359
rest/types/stac_queryables.py                   38       7  81.58%   28, 120-126, 129
rest/types/stac_search.py                      122      10  91.80%   48-51, 167, 182-184, 192, 196
rest/utils/__init__.py                         116      14  87.93%   53, 108-109, 128-130, 180, 190-204, 236
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                               76       6  92.11%   53, 87, 152, 172, 177, 185
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              487      43  91.17%   82, 87, 108-110, 189-190, 199-226, 229, 243, 325-329, 405-409, 430-432, 514, 519, 529, 567-568, 964-967, 975-976, 1017-1018, 1098, 1182, 1200, 1373
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8739    1582  81.90%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
plugins/search/data_request_search.py       -3      -1  +0.04%
rest/core.py                               +87     +15  -3.24%
utils/__init__.py                           +1       0  +0.02%
TOTAL                                      +85     +14  +0.02%

Results for commit: 7d11c92

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 8, 2024 •
edited
Loading

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      28  91.03%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 625, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    750      83  88.93%   101-110, 383, 594, 638-641, 679, 784, 788-793, 819, 889, 959, 1076-1081, 1167-1179, 1219, 1221, 1225, 1246-1248, 1252-1263, 1276-1282, 1372-1375, 1404-1424, 1472, 1478-1481, 1490, 1841, 1874-1880, 2143, 2147-2150, 2164-2166, 2208-2215, 2231, 2241, 2273, 2276
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        208      32  84.62%   48-55, 59-61, 165-172, 256-257, 347, 383, 444, 458-461, 474, 498-501, 544-550
api/product/metadata_mapping.py                648      88  86.42%   66-68, 129-131, 232, 264-265, 317-329, 331, 342, 346-362, 401-404, 441, 462-465, 479, 487-488, 561-562, 586-587, 593-596, 611-612, 776, 822, 879-880, 923-928, 1059, 1073-1093, 1113, 1118, 1228, 1250, 1264, 1277-1296, 1335, 1387, 1410, 1425-1429, 1448
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            38      11  71.05%   36-43, 103, 107, 120, 174, 219
plugins/apis/cds.py                            204      56  72.55%   72-80, 126-129, 176-188, 244, 309, 317-322, 335-342, 360-362, 383-385, 395-397, 411-412, 457, 460, 470, 482, 507-522
plugins/apis/ecmwf.py                           98      15  84.69%   48-56, 157-159, 206-207, 233-235
plugins/apis/usgs.py                           168      36  78.57%   59-64, 125, 198, 232, 267-269, 274, 300-301, 306, 334-341, 350-355, 377-383, 385-391, 414
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       103      64  37.86%   39-41, 138-150, 154-172, 180-222, 228-237, 246-286, 291-299, 304-305
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        490     165  66.33%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 453-454, 460-464, 497, 532, 536, 543, 573-581, 585, 623-631, 638-640, 681-755, 773-834, 845-850, 866-879, 908, 923-925, 928, 938-946, 954-967, 977-996, 1003-1015, 1056, 1082, 1127-1129, 1349
plugins/download/base.py                       261      59  77.39%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 689, 711, 719
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       418      97  76.79%   81-88, 122, 170, 191-198, 208-221, 258-263, 313, 345, 372-374, 384-392, 414, 453-457, 514, 587-643, 657, 681-683, 713-714, 722, 725, 741, 765, 788-789, 796, 855-861, 916-917, 923, 933, 999, 1017-1033
plugins/download/s3rest.py                     116      27  76.72%   55-58, 124, 165, 192, 222-229, 232-234, 238, 251-257, 265-266, 269-273, 296, 316-319
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         101      13  87.13%   43-47, 101, 105, 131-135, 264, 284
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          192      59  69.27%   52, 89-92, 108, 120, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 276, 286-297, 309, 311, 318-319, 321-322, 337-341, 374, 381, 396, 402-414, 419
plugins/search/qssearch.py                     517      79  84.72%   86, 353-354, 448-454, 481-484, 557-558, 599, 617, 632, 685, 706, 709-710, 719, 730, 739, 750-751, 762, 822-827, 831-832, 860, 923, 942-959, 994, 1128-1129, 1150, 1192-1193, 1203-1209, 1249, 1262, 1282, 1330-1399
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/core.py                                   255      94  63.14%   139, 144, 197, 199, 201, 207-208, 225-233, 240-246, 275-280, 286-288, 307-338, 361-393, 547-574, 592, 639-669, 706, 713-717
rest/server.py                                 306     306  0.00%    18-861
rest/stac.py                                   435     154  64.60%   59-61, 214, 229-231, 273, 286-295, 314-320, 365, 402-404, 427, 462-463, 549-594, 639, 647-648, 652-660, 781, 839, 904-906, 923-925, 933-935, 948-950, 964-981, 991-1012, 1022-1044, 1052-1069, 1092-1115, 1125, 1135-1147, 1160-1182, 1196-1241, 1394-1420
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185      18  90.27%   52-55, 231-235, 268-270, 288, 291, 297, 301, 359, 371-374
rest/types/stac_queryables.py                   38      13  65.79%   28, 53-58, 120-126, 129, 133-134
rest/types/stac_search.py                      122      12  90.16%   48-51, 167, 182-184, 192, 196, 240, 243
rest/utils/__init__.py                         116      30  74.14%   53, 79-85, 105, 108-109, 128-130, 173-181, 188-209, 236
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       5  83.87%   73-74, 78, 90, 92
types/__init__.py                               76      11  85.53%   53, 87, 125, 152, 162-164, 172, 177, 185, 195
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              487      43  91.17%   82, 87, 108-110, 189-190, 199-226, 229, 243, 325-329, 405-409, 430-432, 514, 519, 529, 567-568, 964-967, 975-976, 1017-1018, 1098, 1182, 1200, 1373
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8739    2066  76.36%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
plugins/search/data_request_search.py       -3      -1  +0.04%
rest/core.py                               +87     +34  -1.15%
rest/utils/__init__.py                       0      -1  +0.86%
utils/__init__.py                           +1       0  +0.02%
TOTAL                                      +85     +32  -0.14%

Results for commit: 7d11c92

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@jlahovnik jlahovnik force-pushed the 850-cache-search-results branch from ee13071 to bec8acb Compare March 11, 2024 10:40
@jlahovnik jlahovnik marked this pull request as ready for review March 11, 2024 11:23
@sbrunato sbrunato marked this pull request as draft October 10, 2024 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alambare alambare alambare requested changes

@sbrunato sbrunato Awaiting requested review from sbrunato

Requested changes must be addressed to merge this pull request.

Assignees

@jlahovnik jlahovnik

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

avoid search by id before every server mode download
2 participants
@jlahovnik @alambare