Repository navigation
Conversation
Added MAAP_API_HOST to cwltool command for environment preservation.
- Add parse_cwl_type() to extract base type and optional flag from CWL types
- Add convert_value_to_cwl_type() for proper type conversion (int, bool, string)
- Remove str() conversion in get_inputs_from_context() to preserve value types
- Update map_inputs_from_context() to apply type conversion for primitive types
- Filter None values in create_input_yml() to omit optional parameters with no value
- Fix boolean representation ('False' -> false) and integer string conversion
- Handle optional type representations from cwl_utils (lists, ? suffix)
Fixes incorrect YAML output where booleans were strings and optional parameters
with missing values were written as "None" instead of being omitted.
Test with process.cwl now produces:
no_stac_incremental: false
stac_duplicate_policy: update
verbose: false
Instead of:
month: None
no_stac_incremental: 'False'
stac_duplicate_policy: update
verbose: 'False'
year: None
* feat(stac): add CWL output discovery and STAC catalog extraction
Add CWL output discovery and STAC catalog extraction infrastructure for
intelligent output handling and metadata management. Includes stage_out.py for
discovering and validating CWL-defined outputs and hysds_metadata.py for
centralizing HySDS metadata creation with STAC support.
* feat(outputs): integrate multi-output handling with STAC metadata generation
Integrate STAC generation into the execution pipeline with multi-output dataset
creation and centralized metadata handling. Refactors dps_wrapper.sh to call
stage_out.py for output discovery and process_outputs.py for dataset creation,
replacing the previous single-output logic with intelligent multi-output handling
that preserves STAC catalogs and generates proper HySDS metadata.
* feat: update catalog object
* chore: existing version
* feat: update assets URLs
* chore: refactor
* feat: update method to match what we have
* fix: update output
* fix: version check downgraded to 1.0
* chore: refactor
* chore: refactor - delete unused file
* feat: add testcase
* feat: refactor get_job_info + add write_metadata_files
* fix: wrong indentation
* refactor(hysds_metadata): simplify STAC catalog metadata structure
Simplified STAC catalog metadata from comprehensive item inventory to basic catalog information (id, description, stac_version). This reduces metadata bloat while preserving essential catalog references. Added creation_timestamp parameter with ISO format generation to track dataset creation times consistently.
* feat(process_outputs): add timestamp-based unique dataset IDs
Implement microsecond-precision timestamp naming to prevent dataset ID collisions
when multiple jobs run in parallel or the same algorithm is executed multiple times.
Changes:
- Add timestamp formatting (YYYY-MM-DDTHH:MM:SS.mmmmmm) to dataset names
- Use symlinks instead of moving/copying data for efficiency
- Symlinks save disk space and are instant to create
- Each CWL output gets unique timestamp (via microsecond precision)
Implementation details:
- Format matches HySDS pattern: {output_name}-{timestamp}
- Updated process_single_output() for fallback path
- Updated process_single_output_with_metadata() for main path
- Metadata files created in original directory, accessible via symlink
Benefits:
- Zero collisions: microsecond precision prevents any name conflicts
- No disk duplication: symlinks use negligible space
- Parallel safe: simultaneous jobs get different timestamps
- STAC compatible: relative paths work through symlinks
- Backward compatible: existing metadata and STAC systems continue to work
Example:
Before: outputs/ → HySDS dataset "outputs"
After: outputs-2026-02-03T16:41:32.281209/ → symlink to outputs/
* refactor(hysds_metadata): organize STAC items by collection and defer directory creation
Enhance STAC metadata handling in create_met_metadata() to group items by collection ID
in the .met.json output, enabling better organization and discoverability of STAC catalog
items. Move item directory creation from stage_out.py to process_outputs.py behind a
--create-item-metadata flag to separate concerns and give callers control over this behavior.
Changes:
- hysds_metadata.py: Group items by collection, sort alphabetically, handle uncategorized items
- stage_out.py: Remove unconditional os.makedirs() call for item directories
- .gitignore: Add test/ directory to exclusions
* fix(stage_out): deduplicate STAC items when referenced via multiple links
When a STAC catalog contains both direct item links and child/collection links
that reference the same items, the walk_stac_catalog_2 function was counting
each item multiple times. This fix adds deduplication using a set to track
seen item IDs, ensuring accurate item counts in HySDS metadata.
- Items referenced via both direct and collection links are now counted once
- Maintains correct counts for catalogs with legitimate multiple items
- Adds logging for skipped duplicate items
* refactor(stage_out): separate JSON output to stdout and logs to stderr
Configure logging to write to stderr with INFO level, replacing all
diagnostic print() statements with LOGGER calls. This ensures:
- JSON output goes to stdout only (via print(json.dumps()))
- All diagnostic/informational messages go to stderr
- Scripts can cleanly separate structured output from logs
Maintains backward compatibility for human-readable mode and allows
shell integration like: python stage_out.py file.cwl --json-output > output.json 2> logs.txt
---------
Co-authored-by: Sujen Shah <sujen1412@gmail.com>
- Add GeoJSON type detection to route Feature vs FeatureCollection inputs - Extract _download_single_item() helper to avoid code duplication - Add _build_collection() to derive pystac.Collection from FeatureCollection metadata - Add download_stac_collection_assets_from_json() for multi-item collections - Add download_stac_assets_from_json() dispatcher that inspects type field - Compute spatial extent from union of item bboxes - Compute temporal extent with robust ISO 8601 datetime parsing - Create Catalog → Collection → Items hierarchy for collections - Update create_inputs.py to use the new dispatcher function - All relative asset hrefs correctly configured and validated with pystac
Instead of re-parsing the CWL file and using glob patterns to discover outputs, use cwltool's --write-summary flag which provides exact output paths, types, and metadata. Falls back to CWL parsing if summary is unavailable.
Replace long cwltool temp dir basename with CWL output ID (e.g., 'outputs_result-20260311T173254575976' instead of 'job-mdps-...-490960Zmrz_evjc-2026-03-11T17:32:54.575976'). Also fix datetime.utcnow() deprecation warning.
Separate compact timestamp (for directory names) from ISO 8601 timestamp (for .met.json creation_timestamp field). ES strict_date_optional_time requires format like 2026-03-11T18:04:44.474949Z.
) - Add TestDispatcher for feature/collection type routing - Add TestExtentComputation for spatial/temporal extent logic - Add TestSingleItemDownload for asset downloading and item assembly - Add TestSingleItemCatalogAssembly for catalog creation - Add TestCollectionCatalogAssembly for nested collection structure - Add TestMainEntryPoint for entry point behavior All test items include required STAC fields (stac_version, links)
#4) * Preserve STAC extra_fields and extensions during asset staging stage_in.py was silently dropping stac_extensions and extra_fields from localized items, and dropping title, description, roles, and extra_fields from localized assets. This caused all per-asset metadata (e.g. eo:bands, custom extension fields) to be lost in the output catalog. https://claude.ai/code/session_01CqBXmoE7bujUKkwRbXnMaa * Use item.clone() instead of manually copying fields Simpler and future-proof: deep-copy the source item to preserve all metadata, clear remote links, then only overwrite the asset href with the local downloaded path. Failed asset downloads are removed from the cloned item rather than silently omitted. https://claude.ai/code/session_01CqBXmoE7bujUKkwRbXnMaa * test: Add comprehensive tests for STAC metadata preservation - Fix stage_in.py to explicitly preserve extra_fields after clone (pystac.clone() preserves stac_extensions but loses extra_fields) - Add 7 new tests in TestStacMetadataPreservation class: * test_extra_fields_preserved: verify custom fields in item.extra_fields * test_extra_fields_survive_serialization: verify extra_fields in to_dict() * test_stac_extensions_preserved: verify stac_extensions list preserved * test_stac_extensions_empty_list_preserved: edge case for empty list * test_original_item_dict_not_mutated: verify deep copy doesn't mutate input * test_links_cleared_and_self_href_reset: verify remote links replaced * test_self_href_set_to_local_path: verify local path set correctly All tests verify intended behavior from commits 71841b1 and cac56c4. All 36 tests pass (7 new + 29 existing). --------- Co-authored-by: Claude <noreply@anthropic.com>
Introduce _download_asset() helper that routes downloads based on URL type: - MAAP S3 buckets (s3://maap-*) use boto3 client directly - All other URLs (DAAC, non-MAAP S3) fall back to maap.downloadGranule() This optimizes downloads for MAAP-hosted assets while maintaining compatibility with external sources. Add comprehensive routing tests covering all three cases.
- Add _collect_files_from_listing() to recursively extract files from cwltool summary
- Modify parse_cwl_summary() to also build {basename → absolute_path} map from listing
- Modify walk_stac_catalog_2() to:
* Accept optional file_map from cwl_summary
* Use file_map to resolve stale cwltool temp paths (e.g. /fSjDeF/)
* Compute relative hrefs from item.json's own directory (not projected item_dir)
* Write corrected asset hrefs back to item.json on disk
- Modify extract_stac_metadata() to accept and pass file_map to walk_stac_catalog_2
- Modify stage_out_outputs() to unpack file_map and thread it through
This eliminates the need to glob or guess when resolving asset paths. The
cwl_summary already has exact paths for all files. Asset hrefs are now:
1. Resolved using the file_map (for cwltool temp paths)
2. Made relative to the item.json's actual directory
3. Written back to the item.json file so they're persisted
…g_metadata - item_dir → original_item_dir in success result dict - item.id → item_id (safe fallback) in except block, including append call - Initialize item = None before try so except block never references undefined var - Remove leftover debug = 1 assignment - Bare except: → except Exception: to avoid swallowing SystemExit/KeyboardInterrupt
Assets from a single STAC item were all downloaded flat into output_dir, causing silent overwrites when two assets shared the same basename (e.g. FULL-O3-B01 and FULL-O3-B02 both named AIRS_Aqua_L2-O3_*.nc). Assets now land in <item_id>/<asset_key>/ subdirectories. Forward slashes in asset keys are replaced with underscores to prevent path traversal.
…e relative to item directory pystac's make_asset_hrefs_relative() uses the parent of self_href as the base when resolving paths. Setting self_href to the directory (no filename) caused it to escape the item directory, producing ../asset_key/file.nc instead of asset_key/file.nc.
…ogging - Create dedicated cwltool_tmp directory instead of using PWD - Add --quiet flag to reduce verbose output - Add --timestamps flag for better timing visibility - Separates temporary files from working directory for cleaner output
…ument caching rationale
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
load_json_from_uri()now handless3://URIs viaboto3in addition tohttp/httpsdps_wrapper.sh: added--leave-tmpdirto cwltool invocation for easier debugging of temporary filesTest plan
load_json_from_urifetches correctly from ans3://URI (requires AWS credentials)load_json_from_uristill works forhttp/httpsURIs--leave-tmpdirdoesn't break the pipeline