API Documentation

This section provides the API documentation for the main modules of cdsodatacli.

cdsodatacli

odata client for Copernicus Data Space catalog

cdsodatacli.query

Main module for querying the CDSE OData catalog.

Functions:

cdsodatacli.query.query_client()

Main entry point for the MetaData query command line interface.

Parses command line arguments, prepares the query GeoDataFrame, and executes the data fetching process.

Returns:: A DataFrame containing the query results (footprint, Name, Id, etc.).
Return type:: pd.DataFrame

cdsodatacli.query.fetch_data(gdf, timedelta_slice=None, min_sea_percent=None, top=None, cache_dir=None, querymode='seq', email=None, password=None, display_tqdm=False, checkpoint_dir=None)

Fetches meta-data of CDSE products based on provided parameters.

Splits the input GeoDataFrame by the id_query column and executes fetching for each unique query ID.

Parameters:

gdf (gpd.GeoDataFrame) –
Geospatial data for the query.
Mandatory columns:
- ’start_datetime’: Starting date for the query.
- ’end_datetime’: Ending date for the query.
- ’id_query’: Unique identifier of the query.
Optional columns:
- ’name’: SAFE name pattern.
- ’collection’: e.g., SENTINEL-1.
- ’sensormode’: e.g., IW.
- ’producttype’: e.g., GRD.
- ’geometry’: shapely geometry for area of interest.
- ’Attributes’: Extra OData filters.
timedelta_slice (datetime.timedelta, optional) – Time window size to split queries to avoid OData’s 1000 product limit.
min_sea_percent (float, optional) – Minimum sea percent to filter products.
top (int, optional) – Max rows per individual OData query.
cache_dir (str, optional) – Path to directory for storing/reusing results.
querymode (str) – ‘seq’ (sequential) or ‘multi’ (multithreaded). Defaults to ‘seq’.
email (str, optional) – CDSE account email for authentication.
password (str, optional) – CDSE account password.
display_tqdm (bool) – Whether to show a progress bar. Defaults to False.

Returns:

Concatenated meta-data results from all queries.

Return type:

pd.DataFrame

cdsodatacli.query.normalize_gdf(gdf, timedelta_slice=None)

Standardizes input column names and geometry for the fetching logic.

Ensures dates are in UTC, indices are unique, and geometries are valid. Missing geometries are filled with the WORLDPOLYGON.

Parameters:

gdf (gpd.GeoDataFrame) –
Input query data.
Mandatory columns:
- ’start_datetime’ or ‘startdate’: Starting date for the query.
- ’end_datetime’ or ‘stopdate’: Ending date for the query.
- ’id_query’: Unique identifier of the query.
Optional columns:
- ’name’ or ‘Name’: SAFE name pattern.
- ’collection’: e.g., SENTINEL-1.
- ’sensormode’: e.g., IW.
- ’producttype’: e.g., GRD.
- ’geometry’: shapely geometry for area of interest.
- ’Attributes’: Extra OData filters.
timedelta_slice (datetime.timedelta, optional) – Time slicing parameter.

Returns:

The normalized and optionally time-sliced GeoDataFrame.

Return type:

gpd.GeoDataFrame

Raises:

IndexError – If the input GeoDataFrame index is not unique.
ValueError – If invalid geometries are found.

cdsodatacli.query.create_urls(gdf, top=None, email=None, password=None, headers=None)

Constructs OData query URLs based on GeoDataFrame attributes.

Parameters:

gdf (gpd.GeoDataFrame) – Normalized query data.
top (int, optional) – The $top OData parameter. Defaults to 1000.
email (str, optional) – Account email for access token generation.
password (str, optional) – Account password.
headers (dict, optional) – Pre-obtained authentication headers. [optional]

Returns:

A dictionary containing:

’urls’: List of tuples (id_original_query, url_string).
’headers’: Auth headers dict or None.

Return type:

dict

cdsodatacli.query.remove_duplicates(safes_ori)

Removes duplicate SAFE products based on their Name.

Keeps the entry with the most recent ModificationDate.

Parameters:: safes_ori (pd.DataFrame) – Input DataFrame containing duplicate rows.
Returns:: Deduplicated DataFrame.
Return type:: pd.DataFrame

cdsodatacli.query.multy_to_poly(collected_data=None)

Converts WKT Footprints to Shapely geometries and unifies MultiPolygons.

Parameters:: collected_data (pd.DataFrame) – Input meta-data with a ‘Footprint’ column.
Returns:: GeoDataFrame with a clean ‘geometry’ column.
Return type:: gpd.GeoDataFrame

cdsodatacli.query.sea_percent(collected_data, min_sea_percent=None)

Computes the sea coverage percentage for each product footprint.

Filters the GeoDataFrame to keep only products meeting the minimum threshold.

Parameters:

collected_data (gpd.GeoDataFrame) – Resulting meta-data.
min_sea_percent (float, optional) – Threshold (0-100). Defaults to None.

Returns:

Filtered GeoDataFrame with an extra ‘sea_percent’ column.

Return type:

gpd.GeoDataFrame

cdsodatacli.get_products_metadata

Module for retrieving metadata from known SAFE names or Product IDs.

Functions:

GET PRODUCTS METADATA - CLI tool to retrieve metadata from SAFE names or Product IDs

Simplified version: only –input-listing as input, CSV output only.

cdsodatacli.get_products_metadata.validate_safe_name(safe_name: str) → Dict: Valide et parse un nom de produit SAFE en utilisant ExplodeSAFE.

cdsodatacli.get_products_metadata.create_urls_from_safe_names(safe_names: List[str], batch_size: int = 15, logger: Logger | None = None) → List[Dict]: Construit les URLs OData à partir d’une liste de noms SAFE.

cdsodatacli.get_products_metadata.create_urls_from_product_ids(product_ids: List[str], batch_size: int = 50, logger: Logger | None = None) → List[Dict]: Construit les URLs OData à partir d’une liste de Product IDs.

cdsodatacli.get_products_metadata.fetch_metadata_for_batch(batch_info: Dict, headers: Dict | None = None, cache_dir: str | None = None, timeout: int = 30, logger: Logger | None = None) → Tuple[Dict, DataFrame | None]: Récupère les métadonnées pour un batch.

cdsodatacli.get_products_metadata.post_process_data(df: DataFrame, logger: Logger | None = None) → GeoDataFrame

Applique tout le post-traitement aux données récupérées.

Étapes: 1. remove_duplicates() 2. multy_to_poly()

cdsodatacli.get_products_metadata.fetch_all_batches(batches: List[Dict], headers: Dict | None = None, cache_dir: str | None = None, max_workers: int = 4, timeout: int = 30, logger: Logger | None = None) → Tuple[DataFrame, Dict]: Exécute toutes les requêtes en parallèle avec barre de progression.

cdsodatacli.download

Module for downloading products from CDSE.

Functions:

cdsodatacli.download.cds_s3_download_one_product(s3_path, s3_credentials, output_filepath, conf)

Download a single SAFE product via the CDSE S3 endpoint using boto3.

The conf dict must contain:

pre_spool (str): temp directory for .tmp files
s3_access_key (str): CDSE S3 access key
s3_secret_key (str): CDSE S3 secret key
s3_endpoint (str): e.g. “https://eodata.dataspace.copernicus.eu”
s3_bucket (str): e.g. “eodata”
s3_path (str): prefix inside the bucket, e.g.
“Sentinel-1/SAR/GRD/2022/05/03/S1A_IW_GRDH_1SDV_20220503T000000.SAFE”
s3_region (str): ‘default’ CDSE requires “default”

Argument:

s3_path (str): e.g. “Sentinel-1/SAR/GRD/2022/05/03/S1A_IW_GRDH_1SDV_20220503T000000.SAFE” s3_credentials (dict): with keys ‘s3-access-key’ and ‘s3-secret’ output_filepath (str): output full path when download is finished (not the pre-spool but the spool) conf (dict): configuration see details above

Returns:

speed (float) (download speed in Mo/second)
elapsed_time (int) (number of seconds to download the product)
total_mb (int) (MegaBytes downloaded (zip))
status_meaning (str) (human-readable outcome)
safename_base (str) (basename without .zip)

cdsodatacli.download.filter_product_already_present(cpt, df, outputdir, cdsodatacli_conf, force_download=False, extension='.zip')

Based on a dataframe of products to download, filter those already present locally.

Parameters:

(collections.defaultdict(int)) (cpt)
(pd.DataFrame) (df)
(str) (extension)
(dict) (cdsodatacli_conf)
(bool) (force_download)
(str)

Returns:

df_todownload (pd.DataFrame) (dataframe of products to download)
cpt (collections.defaultdict(int)) (updated counter)

cdsodatacli.download.download_list_product_multithread_v4(inputdf, outputdir, account_group, hideprogressbar=False, check_on_disk=True, cdsodatacli_conf_file=None)

v4 is working as deamon like v3 (while loop) multi account round-robin: and token semaphore files but using S3 endpoint to download each product

In this method is working for a group of account with one or many account. Each account can run 4 parallel sessions. step 1: filter the dataframe containing the raw list of products to download -> remove duplicate and remove products already downloaded step 2: create multiple threads to download in parallel (depends on number of account and sessions per account) step 3: loop until all the products are treated step 3.1: get an account (i.e. S3 credentials) for which one session is free/available for download step 3.2: submit future downloads up to the current limit of available sessions step 3.3: wait for the first download thread/session to be finished step 3.4: clean lock on the session to free the session step 4: security lock cleaning (to avoid any orphan busy sessions at the end of the process) step 5: print out the download speed and elapsed times.

Parameters:

(pd.DataFrame) (inputdf)
(str) (cdsodatacli_conf_file)
(str)
(bool) (check_on_disk)
(bool)
(str)

Return type:

df2 (pd.DataFrame)

cdsodatacli.utils

Utility functions for configuration, file operations, and data conversion.

Functions:

cdsodatacli.utils.get_conf(path_config_file=None) → dict

Load configuration from localconfig.yml or config.yml in cdsodatacli package directory.

Parameters:: path_config_file (str, optional) – Full path to the configuration YAML file. Defaults to None.
Returns:: Configuration parameters loaded from the YAML file.
Return type:: dict

cdsodatacli.utils.check_safe_in_outputdir(outputdir, safename)

Parameters:: basename (safename (str))
Returns:: present_in_outputdir (bool)
Return type:: True -> the product is already in the spool dir

cdsodatacli.utils.check_safe_in_spool(safename, conf)

Parameters:

basename (safename (str))
package (conf (dict) configuration dictionary of cdsodatacli)

Returns:

present_in_spool (bool)

Return type:

True -> the product is already in the spool dir

cdsodatacli.utils.WhichArchiveDir(safe, conf, archive_type)

Determine the archive directory path for a given safe based on its naming convention.

Parameters:

safe (str) – safe base name
conf (dict)
archive_type (str) – type of archive directory to use from conf (e.g., ‘datawork’, ‘scale’)

Returns:

full path of the archive directory where the safe should be stored

Return type:

gooddir (str)

cdsodatacli.utils.check_safe_in_archive(safename, conf)

Check if a given safe is already present in the different archive directories.

Parameters:

(str) (safename)
package (conf (dict) configuration dictionary of cdsodatacli)

Returns:

present_in_archive (bool)

Return type:

True -> the product is already in the archive dir. False -> not present.

cdsodatacli.session

Session management for S3 downloads.

cdsodatacli.session.get_sessions_download_available_s3(conf, active_s3_sessions_status, subset_to_treat, blacklist, logins_group='logins')

This method should return the list of available sessions for a group of CDSE accounts contrarily to get_sessions_download_available() it use thread locked in memory variable to list active sessions

Parameters:

conf (dict)
active_s3_sessions_status (dict) – login:session_id(int):False->inactive True>-active (set to inactive at begining of a download)
subset_to_treat (pandas.DatFrame)
blacklist (list) – list of account not usable
logins_group (str) – name of the group of CDSE accounts to use (can contain multiple accounts, it depends on the localconfig.yml)

Returns:

with columns ‘s3_session’, ‘login’, ‘S3Path’, ‘output_path’, ‘safe’, ‘s3_access_key’, ‘s3_secret’ active_s3_sessions_status (dict): updated with the sessions that are now set to active

Return type:

df_products_ready_for_download (pandas.DataFrame)

cdsodatacli.s3_temporary_access_token

Functions for obtaining temporary S3 access tokens.

this script is now deprecated since S3 backend download is using long term S3 credentials from config file

cdsodatacli.s3_temporary_access_token._get_fresh_s3_client(conf, headers)

Create a S3 resources (boto3 client) and S3 temporary credentials.

Parameters:

conf (dict) – Configuration dictionary containing S3 endpoint, region, and bucket information.
headers (dict) – Headers containing the Bearer token for authentication.

Returns:

A tuple containing the S3 temporary credentials and the boto3 S3 resource object.

Return type:

tuple

cdsodatacli.rate_limiter

Rate limiting implementation for API calls.

Rate Limiter for CDSE OData API. Thread-safe token bucket implementation.

class cdsodatacli.rate_limiter.RateLimiter(max_requests_per_second: float = 30.0, max_burst: int = 50)

Rate limiter thread-safe basé sur un bucket de jetons.

wait_if_needed() → None: Attend si nécessaire pour respecter la limite de taux.

get_available_tokens() → float: Retourne le nombre de jetons disponibles.

reset() → None: Réinitialise le rate limiter à pleine capacité.

cdsodatacli.retry

Retry logic with exponential backoff.

cdsodatacli.retry.retry_with_backoff(max_retries: int = 5, base_delay: float = 1.0, max_delay: float = 60.0, exponential_base: float = 2.0, retry_on_exceptions: tuple = (<class 'Exception'>, )): Décorateur pour réessayer une fonction avec backoff exponentiel.

cdsodatacli.product_parser

Parser for SAFE product names (Sentinel-1, Sentinel-2, Sentinel-3).

Author: Antoine.Grouazel@ifremer.fr Purpose:separate information in SAFE name sentinel1 Creation: 2014-11-28 Arguments: basename SAFE directory note: valid also for Sentinel3 SRAL data

class cdsodatacli.product_parser.ExplodeSAFE(basename_safe): input basename_safe (str) SAFE name only (no parent directories before neitheir children files)

cdsodatacli.s3_path

Utilities for S3 path handling.

cdsodatacli.fetch_access_token

Authentication and token fetching.

cdsodatacli.fetch_access_token.get_access_token(email, password): Helper to retrieve OIDC token. one can generate as many access token as wanted per CDSE account

Example: Using fetch_data() for spatio-temporal queries

import cdsodatacli.query as query
import geopandas as gpd
from shapely import wkt
import datetime

# Create query parameters
gdf = gpd.GeoDataFrame(
    {
        "start_datetime": [datetime.datetime(2023, 1, 1, 0, 0, 0)],
        "end_datetime": [datetime.datetime(2023, 1, 5, 10, 10, 10)],
        "geometry": [wkt.loads("POINT (-5.02 48.4)")],
        "collection": ["SENTINEL-1"],
        "sensormode": ["IW"],
        "producttype": ["SLC"],
        "id_query": ["my_query_id"],
    }
)

# Fetch metadata
results = query.fetch_data(gdf=gdf, top=1000, cache_dir="./cache", querymode="seq")

Example: Using get-metadata-from-product-names

from cdsodatacli.get_products_metadata import main
import argparse

# Command line usage is preferred:
# get-metadata-from-product-names --input-listing safes.txt --output results.csv

Example: Downloading products

from cdsodatacli.download import download_list_product_multithread_v4

# Download products from a listing
results = download_list_product_multithread_v4(
    listing_path="products.csv", output_dir="./downloads", max_workers=4
)