fix: classify invalid function inputs as user errors instead of runtime crashes#1804
Conversation
…me crashes Input that fails Pydantic/dataclass conversion in the functions runtime previously surfaced as a generic FUNCTION_EXECUTION_ERROR with a full Python traceback embedded in the error detail, and the eval runtime unconditionally flagged it as a runtime exception. This made test-data problems indistinguishable from eval infrastructure crashes in App Insights and triggered SRE alerting on EvalRun.Failed events (SRE-618832). - functions runtime: catch input conversion failures at convert_to_class and raise a USER-category "Invalid input" error with a concise per-field message and no traceback - eval runtime: only set runtime_exception=True when the root error is not a user-category UiPathBaseRuntimeError - eval telemetry: emit ErrorCode and ErrorCategory properties so alerts can filter user-caused failures without org-ID allowlists Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
test:uipath-langchain
There was a problem hiding this comment.
Pull request overview
This PR improves error classification and telemetry for typed agent/function input validation so schema-mismatched inputs are reported as USER-category failures (without tracebacks), reducing false-positive “runtime crash” alerts during eval runs.
Changes:
- Functions runtime: convert typed-input validation failures into
UiPathRuntimeErrorwithUiPathErrorCategory.USER,Invalid inputtitle, concise per-field detail, and no traceback. - Eval runtime: introduce
_is_user_facing_error()and setruntime_exceptionbased on whether the root exception is a USER-categoryUiPathBaseRuntimeError. - Eval telemetry: include
ErrorCodeandErrorCategoryproperties forUiPathBaseRuntimeErrorinEvalRun.Failedevents; add targeted tests.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/uipath/src/uipath/functions/runtime.py | Converts input-conversion failures into USER-category runtime errors without tracebacks, with concise validation details. |
| packages/uipath/src/uipath/eval/runtime/runtime.py | Adds user-facing error classifier and adjusts runtime_exception marking for eval runs. |
| packages/uipath/src/uipath/_cli/_evals/_telemetry.py | Enriches eval failure telemetry with ErrorCode/ErrorCategory when available. |
| packages/uipath/tests/functions/test_input_validation.py | Adds tests ensuring invalid typed inputs become USER-category errors and avoid tracebacks. |
| packages/uipath/tests/cli/eval/test_runtime_error_classification.py | Adds unit tests for user-facing error classification helper. |
| packages/uipath/tests/cli/eval/test_eval_telemetry.py | Adds telemetry test ensuring user-category errors include classification fields and are not runtime exceptions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Chibionos
left a comment
Chibionos
left a comment
There was a problem hiding this comment.
approving, fix the copilot comment and run linting, the spacing seems off to me.
| UiPathRuntimeError, | ||
| ) | ||
|
|
||
|
|
There was a problem hiding this comment.
nit: spacing is wrong here lint should fail for this
There was a problem hiding this comment.
The spacing here is the standard two blank lines between the import block and the first top-level def (PEP 8 E303/E302 rules) — ruff check and ruff format --check both pass on this file, so lint is green. Happy to adjust if you meant something else!
2.13.8 is reserved for #1801, which merges first per its companion-PR sequence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lidation-errors # Conflicts: # packages/uipath/pyproject.toml # packages/uipath/uv.lock
Non-EvaluationRuntimeException errors previously fell through with the model default runtime_exception=False, hiding genuine infrastructure failures in telemetry. Unwrap the root exception once and classify every failure uniformly: runtime exception unless user-category. Addresses PR review feedback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rrors SonarCloud new-code coverage was 85.7% (<90% gate): the except-path classification in the eval runtime and the TypeError branch of the input validation message formatter were untested. Add an end-to-end failing-execution eval test asserting runtime_exception classification for user vs unexpected errors, and a dataclass missing-field test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
🚨 Heads up:
|
🚨 Heads up:
|
Summary
Fixes the root cause behind SRE-618832: eval runs feeding invalid inputs to typed agent functions (e.g. the SDK's own
crash-string-input-a/bCI evals) surfaced as unhandled runtime crashes withIsRuntimeException=Trueand a full Python traceback inErrorMessage, repeatedly firing theagents-evalrun-failed-urtApp Insights alert. A schema-mismatched input is a user/test-data problem, not an eval infrastructure failure, and should be reported as such.Changes
Functions runtime (
functions/runtime.py)ValidationError/TypeError/ValueErrorat theconvert_to_classinput-conversion boundary and raise aUiPathRuntimeErrorwithUiPathErrorCategory.USER, titleInvalid input, a concise per-field message (e.g.a: Input should be a valid integer), andinclude_traceback=False— no more CI workspace paths or tracebacks in telemetry.UiPathErrorCode.INPUT_INVALID_JSONas the closest existing code;uipath-runtime0.12.x has no dedicated input-validation code yet (follow-up: addINPUT_VALIDATION_ERRORthere).Eval runtime (
eval/runtime/runtime.py)runtime_exceptionis now only set toTruewhen the root error is not a user-categoryUiPathBaseRuntimeError. User-category errors are correctly-reported workload failures of the agent under evaluation.Eval telemetry (
_cli/_evals/_telemetry.py)EvalRun.Failedevents now includeErrorCodeandErrorCategoryproperties from the error contract, giving SRE a first-class KQL dimension (ErrorCategory == "User") to filter alert noise instead of substring-matching messages or hardcoding CI org IDs.Behavior notes
Test plan
tests/functions/test_input_validation.py: string/None/list/dict inputs to a Pydantic-typed function yield USER-categoryInvalid inputerrors with per-field details and no traceback; valid input still executes.tests/cli/eval/test_runtime_error_classification.py: user vs system/unknown/plain-exception classification.tests/cli/eval/test_eval_telemetry.py: user-category errors emitErrorCode/ErrorCategoryandIsRuntimeException=False.tests/functions/+tests/cli/eval/suites pass (492 tests); ruff check/format and mypy clean.🤖 Generated with Claude Code