How to use ‘id_query’ ?
the ‘id_query’ is used in the Geo Pandas DataFrame as a key to distinguish different requests passed through the same GeoDataFrame.
Exact same query given twice with distinct ‘id_query’ will result in excact same results but stored twice in the output DataFrame.
The ‘id_query’ is also stored in the output DataFrame as ‘id_original_query’.
[1]:
import geopandas as gpd
import numpy as np
import cdsodatacli.query as qr
def example():
"""
define a query with 2 identical sets of parameters except 'id_query'
"""
gdf_multi_id = gpd.GeoDataFrame(
{
"start_datetime": [
np.datetime64("2022-05-03 00:00:00"),
np.datetime64("2022-05-03 00:00:00"),
],
"end_datetime": [
np.datetime64("2022-05-03 00:02:00"),
np.datetime64("2022-05-03 00:02:00"),
],
"geometry": [None, None],
"collection": ["SENTINEL-2", "SENTINEL-2"],
"name": [None, None],
"sensormode": [None, None],
"producttype": [None, None],
"Attributes": [None, None],
"id_query": ["test1", "test2"],
}
)
result = qr.fetch_data(gdf=gdf_multi_id, top=1000)
# check that a product is present twice with different id_original_query
counts = result["id_original_query"].value_counts()
return result,counts
[2]:
result,counts = example()
assert all(counts[counts > 1].index.isin(["test1", "test2"]))
result
[2]:
| @odata.mediaContentType | Id | Name | ContentType | ContentLength | OriginDate | PublicationDate | ModificationDate | Online | EvictionDate | S3Path | Checksum | ContentDate | Footprint | GeoFootprint | Attributes | id_original_query | geometry | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | application/octet-stream | 7b561bb5-bb5c-476a-ac51-d361cace788d | S2B_MSIL2A_20220503T000139_N0510_R016_T11XML_2... | application/octet-stream | 905950656 | 2024-10-22T03:57:01.163000Z | 2024-11-30T10:52:42.448541Z | 2024-11-30T10:52:58.857924Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L2A_N0500/2022/05/03/S2... | [{'Value': '8dc89dabd44eb3869de612042a6a427d',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-123.37811606469... | {'type': 'Polygon', 'coordinates': [[[-123.378... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-123.37812 81.90748, -122.69284 80.9... |
| 1 | application/octet-stream | f05036fd-66ee-42bb-a12f-92b319f4c7fb | S2B_MSIL2A_20220503T000139_N0510_R016_T08XNR_2... | application/octet-stream | 906648145 | 2024-10-22T03:56:59.617000Z | 2024-11-30T10:52:13.209895Z | 2024-11-30T10:52:24.656867Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L2A_N0500/2022/05/03/S2... | [{'Value': '8e855e16387253f7aca2fcf38b0d85fb',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-135.00128035426... | {'type': 'Polygon', 'coordinates': [[[-135.001... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-135.00128 81.95692, -135.00114 80.9... |
| 2 | application/octet-stream | b2f271b3-0217-4dcd-93b3-29ceb81b78f1 | S2B_MSIL1C_20220503T000139_N0510_R016_T07XEL_2... | application/octet-stream | 739061869 | 2024-10-22T03:56:56.046000Z | 2024-11-30T10:51:43.843079Z | 2024-11-30T10:51:54.844138Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L1C_N0500/2022/05/03/S2... | [{'Value': 'f514d8376518117f2e345263982d4ec1',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-140.96097336164... | {'type': 'Polygon', 'coordinates': [[[-140.960... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-140.96097 80.97292, -134.75703 80.9... |
| 3 | application/octet-stream | 9501c4d2-4315-4033-8fc0-6ffdb6ed0a8a | S2B_MSIL2A_20220503T000139_N0510_R016_T10XER_2... | application/octet-stream | 906168896 | 2024-10-22T03:57:00.756000Z | 2024-11-30T10:51:41.950143Z | 2024-11-30T10:51:53.955437Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L2A_N0500/2022/05/03/S2... | [{'Value': '6ee6c35e7a4fb8ad3913279ad053d32a',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-123.00128035426... | {'type': 'Polygon', 'coordinates': [[[-123.001... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-123.00128 81.95692, -123.00114 80.9... |
| 4 | application/octet-stream | 6b7817c3-a120-4c1b-9d00-672249307e85 | S2B_MSIL2A_20220503T000139_N0510_R016_T07XEL_2... | application/octet-stream | 899169465 | 2024-10-22T03:56:59.076000Z | 2024-11-30T10:51:11.639296Z | 2024-11-30T10:51:22.361116Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L2A_N0500/2022/05/03/S2... | [{'Value': 'a54d86f0a52c6588bcc1fd581a8d2b42',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-140.96097336164... | {'type': 'Polygon', 'coordinates': [[[-140.960... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-140.96097 80.97292, -134.75703 80.9... |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 151 | application/octet-stream | cc29e73b-56b3-4a48-bce6-268785ee7d34 | S2B_MSIL1C_20220503T000139_N0510_R016_T06XWR_2... | application/octet-stream | 201206381 | 2024-10-22T03:56:55.563000Z | 2024-11-30T10:45:08.042394Z | 2024-11-30T10:45:13.163135Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L1C_N0500/2022/05/03/S2... | [{'Value': '8faec36c74e72f561fb34c5dd91d7219',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-140.77883895807... | {'type': 'Polygon', 'coordinates': [[[-140.778... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test2 | POLYGON ((-140.77884 80.92048, -140.75703 80.9... |
| 152 | application/octet-stream | 7832c103-0838-49a2-bb80-6c932cab908d | S2B_MSIL1C_20220503T000139_N0510_R016_T11XMJ_2... | application/octet-stream | 65098736 | 2024-10-22T03:56:57.926000Z | 2024-11-30T10:45:07.858114Z | 2024-11-30T10:45:11.861222Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L1C_N0500/2022/05/03/S2... | [{'Value': 'beb4e8fd14b8bb124ef0109753c9c85d',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-116.48997853732... | {'type': 'Polygon', 'coordinates': [[[-116.489... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test2 | POLYGON ((-116.48998 80.12639, -116.48817 80.1... |
| 153 | application/octet-stream | ba0acd60-ae9a-4282-8a5f-a313fe8cfdcf | S2B_MSIL1C_20220503T000139_N0510_R016_T10XEP_2... | application/octet-stream | 76120489 | 2024-10-22T03:56:57.524000Z | 2024-11-30T10:45:07.824853Z | 2024-11-30T10:45:11.755815Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L1C_N0500/2022/05/03/S2... | [{'Value': 'bf216267943adf61f51b6f858ab6192e',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-117.28586000606... | {'type': 'Polygon', 'coordinates': [[[-117.285... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test2 | POLYGON ((-117.28586 80.11698, -123.00105 80.1... |
| 154 | application/octet-stream | 0714ebc9-4b3e-416d-a6c3-bb51ce28e5c4 | S2B_MSIL1C_20220503T000139_N0510_R016_T06XWS_2... | application/octet-stream | 16311928 | 2024-10-22T03:56:55.646000Z | 2024-11-30T10:45:07.313818Z | 2024-11-30T10:45:11.288750Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L1C_N0500/2022/05/03/S2... | [{'Value': 'c30810846f52a8cefd4512b63348be5f',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-141.78746379933... | {'type': 'Polygon', 'coordinates': [[[-141.787... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test2 | POLYGON ((-141.78746 81.82497, -140.08085 81.8... |
| 155 | application/octet-stream | 877b73c7-da3a-4f64-b46b-a8c8d8050fb2 | S2B_MSIL1C_20220503T000139_N0510_R016_T07XDK_2... | application/octet-stream | 17579643 | 2024-10-22T03:56:55.731000Z | 2024-11-30T10:45:07.375601Z | 2024-11-30T10:45:10.435212Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L1C_N0500/2022/05/03/S2... | [{'Value': '165d1a5730c67da3238fa28c9c7f1076',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-140.45053600549... | {'type': 'Polygon', 'coordinates': [[[-140.450... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test2 | POLYGON ((-140.45054 80.82657, -140.43739 81.0... |
156 rows × 18 columns
[3]:
counts
[3]:
id_original_query
test1 78
test2 78
Name: count, dtype: int64
we can check that product IDs are exatcly the same
[4]:
cpt_common = 0
cpt_uncommon = 0
total = 0
for uu in result[result['id_original_query']=='test1']['Id']:
total += 1
is_comon = False
for vv in result[result['id_original_query']=='test2']['Id']:
if uu==vv:
# print('bingo')
is_comon = True
if is_comon is True:
cpt_common += 1
else:
cpt_uncommon += 1
# print(uu)
print('cpt_common',cpt_common,'/',total)
print('cpt_uncommon',cpt_uncommon,'/',total)
cpt_common 78 / 78
cpt_uncommon 0 / 78
doing the same query with a single ‘id_query’ returns a DataFrame with 78 lines instead of 156
[5]:
gdf_unique_id = gpd.GeoDataFrame(
{
"start_datetime": [
np.datetime64("2022-05-03 00:00:00"),
],
"end_datetime": [
np.datetime64("2022-05-03 00:02:00"),
],
"geometry": [None],
"collection": ["SENTINEL-2"],
"name": [None],
"sensormode": [None],
"producttype": [None],
"Attributes": [None],
"id_query": ["test1"],
}
)
result_2 = qr.fetch_data(gdf=gdf_unique_id, top=1000)
result_2.head()
[5]:
| @odata.mediaContentType | Id | Name | ContentType | ContentLength | OriginDate | PublicationDate | ModificationDate | Online | EvictionDate | S3Path | Checksum | ContentDate | Footprint | GeoFootprint | Attributes | id_original_query | geometry | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 77 | application/octet-stream | 7b561bb5-bb5c-476a-ac51-d361cace788d | S2B_MSIL2A_20220503T000139_N0510_R016_T11XML_2... | application/octet-stream | 905950656 | 2024-10-22T03:57:01.163000Z | 2024-11-30T10:52:42.448541Z | 2024-11-30T10:52:58.857924Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L2A_N0500/2022/05/03/S2... | [{'Value': '8dc89dabd44eb3869de612042a6a427d',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-123.37811606469... | {'type': 'Polygon', 'coordinates': [[[-123.378... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-123.37812 81.90748, -122.69284 80.9... |
| 24 | application/octet-stream | f05036fd-66ee-42bb-a12f-92b319f4c7fb | S2B_MSIL2A_20220503T000139_N0510_R016_T08XNR_2... | application/octet-stream | 906648145 | 2024-10-22T03:56:59.617000Z | 2024-11-30T10:52:13.209895Z | 2024-11-30T10:52:24.656867Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L2A_N0500/2022/05/03/S2... | [{'Value': '8e855e16387253f7aca2fcf38b0d85fb',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-135.00128035426... | {'type': 'Polygon', 'coordinates': [[[-135.001... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-135.00128 81.95692, -135.00114 80.9... |
| 76 | application/octet-stream | b2f271b3-0217-4dcd-93b3-29ceb81b78f1 | S2B_MSIL1C_20220503T000139_N0510_R016_T07XEL_2... | application/octet-stream | 739061869 | 2024-10-22T03:56:56.046000Z | 2024-11-30T10:51:43.843079Z | 2024-11-30T10:51:54.844138Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L1C_N0500/2022/05/03/S2... | [{'Value': 'f514d8376518117f2e345263982d4ec1',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-140.96097336164... | {'type': 'Polygon', 'coordinates': [[[-140.960... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-140.96097 80.97292, -134.75703 80.9... |
| 75 | application/octet-stream | 9501c4d2-4315-4033-8fc0-6ffdb6ed0a8a | S2B_MSIL2A_20220503T000139_N0510_R016_T10XER_2... | application/octet-stream | 906168896 | 2024-10-22T03:57:00.756000Z | 2024-11-30T10:51:41.950143Z | 2024-11-30T10:51:53.955437Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L2A_N0500/2022/05/03/S2... | [{'Value': '6ee6c35e7a4fb8ad3913279ad053d32a',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-123.00128035426... | {'type': 'Polygon', 'coordinates': [[[-123.001... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-123.00128 81.95692, -123.00114 80.9... |
| 0 | application/octet-stream | 6b7817c3-a120-4c1b-9d00-672249307e85 | S2B_MSIL2A_20220503T000139_N0510_R016_T07XEL_2... | application/octet-stream | 899169465 | 2024-10-22T03:56:59.076000Z | 2024-11-30T10:51:11.639296Z | 2024-11-30T10:51:22.361116Z | True | 9999-12-31T23:59:59.999999Z | /eodata/Sentinel-2/MSI/L2A_N0500/2022/05/03/S2... | [{'Value': 'a54d86f0a52c6588bcc1fd581a8d2b42',... | {'Start': '2022-05-03T00:01:39.024000Z', 'End'... | geography'SRID=4326;POLYGON ((-140.96097336164... | {'type': 'Polygon', 'coordinates': [[[-140.960... | [{'@odata.type': '#OData.CSC.StringAttribute',... | test1 | POLYGON ((-140.96097 80.97292, -134.75703 80.9... |
When there is multiple “id_query”, the input GeoDataFrame is split/group-by into smaller GeoDataFrame with identical “id_query”.
Setting carefully the “id_query” column in the input GeoDataFrame is important because it impacts the number of queries and the output returned.
In most of the case the ‘id_query’ column can be filled with a unique value, but for some cases, for instance when co-locating a set of moored buoys with Sentinel products, the user needs the exhaustive list of Sentinel products per location and he also needs to be able to separate the results per location (i.e. buoys).