Fix/pipeline component asset default runtime - #48513
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Adds asset-type defaults to Azure ML pipeline components and carries them into runtime job inputs.
Changes:
- Accepts and serializes defaults for data/model asset inputs.
- Materializes defaults when pipeline nodes are instantiated.
- Adds unit and end-to-end coverage plus a changelog entry.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
sdk/ml/azure-ai-ml/CHANGELOG.md |
Documents asset-default support. |
azure/ai/ml/_schema/job/input_output_entry.py |
Adds default fields to asset schemas. |
azure/ai/ml/constants/_component.py |
Permits defaults in asset input validation. |
azure/ai/ml/dsl/_component_func.py |
Supplies asset defaults to generated callables. |
azure/ai/ml/entities/_inputs_outputs/input.py |
Accepts string defaults for asset types. |
azure/ai/ml/entities/_job/pipeline/_io/mixin.py |
Materializes defaults as runtime inputs. |
azure/ai/ml/operations/_component_operations.py |
Restores defaults omitted from create responses. |
tests/component/unittests/test_component_operations.py |
Tests create-response restoration. |
tests/component/unittests/test_pipeline_component_entity.py |
Tests loading and serialization. |
tests/pipeline_job/e2etests/test_pipeline_job.py |
Exercises an omitted default end to end. |
tests/test_configs/components/pipeline_component_with_asset_defaults.yml |
Defines serialization test inputs. |
tests/test_configs/components/pipeline_component_with_uri_file_default.yml |
Defines the runtime test component. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
sdk/ml/azure-ai-ml/azure/ai/ml/_schema/job/input_output_entry.py:118
DataInputSchemais used for actual job inputs, wheredefaultis never promoted topath;_resolve_job_inputconsequently raisesInput path can't be empty for jobs.A pipeline component's input definition is parsed by the separateInputPortSchema, which already supports defaults, so this addition broadens job YAML without enabling valid runtime behavior.
default = generate_path_property(azureml_type=AzureMLResourceType.DATA)
sdk/ml/azure-ai-ml/azure/ai/ml/_schema/job/input_output_entry.py:136
- This also makes an MLTable job input with only
defaultpass schema loading, but job submission requirespathand does not consumedefault. Component interface defaults are already handled by_schema/component/input_output.py::InputPortSchema; remove this job-schema field to preserve early validation.
default = generate_path_property(azureml_type=AzureMLResourceType.DATA)
sdk/ml/azure-ai-ml/azure/ai/ml/entities/_inputs_outputs/input.py:377
- Local-path defaults are now accepted, but component registration does not resolve or upload top-level component input defaults:
_resolve_dependencies_for_componentonly resolves component code/environment and inputs of nested jobs. The registered component therefore persists a machine-local path, and invoking it elsewhere later fails when job submission tries to resolve that path relative to the caller. Either upload/replace local defaults during component registration or restrict defaults to portable AzureML/remote references.
# Asset-type inputs accept a string default value (e.g. "azureml:", "https://",
# local path) matching the public CLI v2 YAML schema.
…t-asset-default-runtime # Conflicts: # sdk/ml/azure-ai-ml/api.metadata.yml
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (1)
sdk/ml/azure-ai-ml/azure/ai/ml/entities/_inputs_outputs/input.py:381
- Accepting local paths here makes registered defaults non-portable. Component registration resolves dependencies in pipeline jobs but does not upload or rewrite top-level
component.inputs[*].default; the fallback metadata consequently stores the raw local path. Fetching and running that component from another working directory or machine then attempts to use a path that does not exist. Either restrict defaults to portable remote references or resolve/upload local defaults before registration and persist the resulting URI.
if not self._is_primitive_type and default_value is not None:
# Asset-type inputs accept a string default value (e.g. "azureml:", "https://",
# local path) matching the public CLI v2 YAML schema.
if not self._multiple_types and self.type in IOConstants.ASSET_INPUT_TYPES:
if isinstance(default_value, str):
self.default: Any = default_value
return
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (1)
sdk/ml/azure-ai-ml/azure/ai/ml/entities/_inputs_outputs/input.py:381
- This changes the shared public
Inputtype, not just pipeline-component ports. As a result, a top-level job can now be constructed withInput(type="uri_file", default="azureml:data:1"), but it has nopath; job submission still rejects that object withInput path can't be empty for jobsin_job_operations.py:1517-1525. The job YAML schemas were kept from acceptingdefaultfor this reason. Please scope asset defaults to component interfaces (or add context-specific job validation) so programmatic job inputs continue to reject this unsupported shape instead of accepting it and failing later during submission.
if not self._multiple_types and self.type in IOConstants.ASSET_INPUT_TYPES:
if isinstance(default_value, str):
self.default: Any = default_value
return
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (2)
sdk/ml/azure-ai-ml/azure/ai/ml/entities/_component/pipeline_component.py:532
- Local paths are explicitly accepted as asset defaults, but this persists the raw path without resolving/uploading it.
_resolve_dependencies_for_componentonly resolves code, environment, and inputs of pipeline jobs (_component_operations.py:871-906,1151-1169), notcomponent.inputs. After the component is registered and fetched, the original YAML base directory is unavailable, so an omitted local-path default is resolved relative to the caller (or fails) instead of referencing an uploaded asset. Resolve local interface defaults during registration and persist the resulting cloud URI.
asset_input_defaults = {
input_name: component_input.default
for input_name, component_input in self.inputs.items()
if component_input.type in IOConstants.ASSET_INPUT_TYPES and component_input.default is not None
}
sdk/ml/azure-ai-ml/azure/ai/ml/entities/_inputs_outputs/input.py:381
Inputis also the public type for top-level job inputs, so this now lets a programmatic job acceptInput(type="uri_file", default="..."). That job has nopath; submission reaches_job_operations.py:1517-1525, which ignoresdefaultand raises “Input path can't be empty for jobs.” The job YAML schemas rejectdefault, but direct construction bypasses those schemas. Please scope asset defaults to component-interface inputs, or add context-aware job validation so this unsupported configuration is not accepted by the shared API.
if not self._multiple_types and self.type in IOConstants.ASSET_INPUT_TYPES:
if isinstance(default_value, str):
self.default: Any = default_value
return
[Pilot] PR Pipeline Failure AnalysisA CI pipeline failed on this pull request. Here is an automated analysis of what went wrong and how to get the build green. What failedThe
Black wants to reformat a dictionary assignment (collapsing a multi-line subscript assignment into a parenthesized single-line form). This is a validation failure — no code logic is broken, only formatting. Recommended next steps
Raw pipeline analysis (azsdk ci analyze)
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (2)
sdk/ml/azure-ai-ml/azure/ai/ml/entities/_inputs_outputs/input.py:385
- The string-only check also accepts an empty asset reference. That value is serialized and later converted to
Input(path=""), where job submission always fails the non-empty path check. Reject empty defaults while loading the component so the error is reported at the invalid declaration.
if isinstance(default_value, str):
self.default: Any = default_value
return
msg = (
f"{msg_prefix}cannot be set: default for type '{self.type}' must be a "
f"string asset reference, got '{type(default_value)}'."
)
sdk/ml/azure-ai-ml/azure/ai/ml/entities/_inputs_outputs/input.py:381
Inputis also the public value type for top-level job inputs, so this now acceptsInput(type="uri_file", default="azureml:data:1")outside component interfaces even though itspathremainsNone. Job submission then deterministically fails in_job_operations.py:1517-1525with “Input path can't be empty for jobs.” Scope asset defaults to component-port construction/deserialization (or add context-aware validation) so the shared job-input API does not accept a configuration it cannot submit.
This issue also appears on line 379 of the same file.
if not self._multiple_types and self.type in IOConstants.ASSET_INPUT_TYPES:
if isinstance(default_value, str):
self.default: Any = default_value
return
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (1)
sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/pipeline/_io/mixin.py:122
- Returning only the default path drops the input's declared
modeon the non-dynamic node-loading path:NodeInput._build_datawraps this string asInput(type=..., path=...), so defaults configured withdownloadordirectare submitted without that mode. Return anInputhere, as the pipeline-job override below already does, so omitted asset defaults preserve their delivery mode.
if isinstance(val, Input) and val.type in IOConstants.ASSET_INPUT_TYPES and val.default is not None:
return val.default
Description
Fixes support for default values on asset-type inputs in Azure ML pipeline components.
Previously, loading a pipeline component with an asset input default failed with:
Default value of Input cannot be set: Non-primitive type Input has no default value.
Changes:
1.Allow string defaults for uri_file, uri_folder, mltable, mlflow_model, and custom_model inputs.
2.Apply asset defaults when a pipeline component is invoked without explicitly providing those inputs.
3.Preserve defaults when registered components are returned or retrieved from the service.
4.Prevent default [Input] objects from being shared between component invocations.
5.Add unit and end-to-end test coverage.
6.Update the changelog.
Validation
1.Verified that load_component() accepts a pipeline component with a uri_file default.
2.Verified that defaults survive serialization, registration, and retrieval.
3.Verified that an omitted uri_file input is populated when the pipeline runs.
4.All Azure SDK CI checks pass.
Testing
Samples validations: Azure/azureml-examples#4102
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines