Skip to content

Latest commit

 

History

History
186 lines (138 loc) · 5.7 KB

cli-reference.md

File metadata and controls

186 lines (138 loc) · 5.7 KB

CLI Reference

日本語概要

本書は、CSVの構造確認、規則候補の生成、整形、統制例の実行に使用するコマンドラインを定義します。引数、標準出力、生成ファイル、処理を継続・中止する条件、終了コードを確認できます。

完全なコマンド例と動作契約は以下の英語本文を参照してください。


English Summary

This document defines the complete command-line workflows and exit semantics. Start with the minimal example in the README, then use the reference below for inspection, schema suggestion, cleaning, and controlled examples.

Command Reference

Inspect

data-cleaning-toolkit inspect INPUT.csv [--output REPORT.json]

When --output is omitted, the JSON report is written to standard output. Inspection is structural: it does not infer whether a column is numeric, a date, an identifier, or a category.

The report contains:

  • Input row and column counts
  • Empty and non-empty cell counts per column
  • Distinct non-empty value counts per column
  • Exact duplicate row count
  • Rows whose field count differs from the header

Suggest Schema

data-cleaning-toolkit suggest-schema INPUT.csv \
  --output SUGGESTION.json \
  [--date-format FORMAT]

The command ignores empty cells while measuring parse coverage. A non-string type is suggested only when every non-empty value matches that type. Candidate precedence is boolean, integer, decimal, and date; otherwise the column remains a string. Textual forms such as true and false may be suggested as boolean, while 0 and 1 remain eligible for integer inference. Integer-shaped values with leading zeros are kept as strings to avoid destructive normalization of identifiers.

The JSON output includes a ready-to-review suggested_schema object plus per-column counts, whitespace observations, leading-zero counts, and parse coverage. Copy or extract the candidate only after reviewing it against the dataset's meaning and downstream requirements.

Clean

data-cleaning-toolkit clean INPUT.csv \
  --schema SCHEMA.json \
  --output CLEAN.csv \
  [--report AUDIT.json]

If --report is omitted, clean.csv produces clean.report.json. The input CSV and schema are protected sources: neither can also be used as the clean CSV or report path.

Controlled Workflows

Inspect a CSV before defining cleaning rules:

data-cleaning-toolkit inspect examples/demo_dirty.csv \
  --output output/inspection.json

Generate a reviewable schema candidate with per-column evidence:

data-cleaning-toolkit suggest-schema examples/schema_suggestion_demo.csv \
  --output output/schema_suggestion.json

Run the controlled explicit-value-mapping example:

data-cleaning-toolkit clean examples/value_mapping_demo.csv \
  --schema examples/value_mapping_schema.json \
  --output output/value_mapping_clean.csv \
  --report output/value_mapping_report.json

Run the controlled cross-column validation example:

data-cleaning-toolkit clean examples/cross_column_demo.csv \
  --schema examples/cross_column_schema.json \
  --output output/cross_column_clean.csv \
  --report output/cross_column_report.json

Run the controlled conditional-presence example:

data-cleaning-toolkit clean examples/conditional_presence_demo.csv \
  --schema examples/conditional_presence_schema.json \
  --output output/conditional_presence_clean.csv \
  --report output/conditional_presence_report.json

Run the controlled mapping-coverage example:

data-cleaning-toolkit clean examples/mapping_coverage_demo.csv \
  --schema examples/mapping_coverage_schema.json \
  --output output/mapping_coverage_clean.csv \
  --report output/mapping_coverage_report.json

Run the controlled unmatched-frequency example:

data-cleaning-toolkit clean examples/unmatched_frequency_demo.csv \
  --schema examples/unmatched_frequency_schema.json \
  --output output/unmatched_frequency_clean.csv \
  --report output/unmatched_frequency_report.json

Compare raw, redacted, and disabled frequency modes:

data-cleaning-toolkit clean examples/privacy_modes_demo.csv \
  --schema examples/privacy_modes_schema.json \
  --output output/privacy_modes_clean.csv \
  --report output/privacy_modes_report.json

The dirty-data demo is intentionally invalid, so inspect and clean write their outputs and return exit code 1. The value-mapping example also returns 1 because its final row contains deliberately unmapped values. The cross-column example returns 1 because it includes deliberate relationship violations. The conditional-presence example returns 1 because it includes deliberately missing dependent values. The mapping-coverage example returns 1 because it includes one deliberately unknown value. The unmatched-frequency example returns 1 because it includes four deliberately unknown values. The controlled privacy-mode and suggest-schema examples return 0. These semantics make the CLI usable as a data-quality gate in scripts and CI.

Regenerate the committed reference artifacts with:

python examples/run_demo.py

Run the tests with:

python -m pytest

Run the public Python API example:

python examples/public_api_demo.py

Verify the committed reference artifacts without regenerating them:

python examples/run_demo.py --verify-only

Exit Codes

Code Meaning
0 Command completed without validation errors
1 Output was produced, but invalid or malformed rows were detected
2 Input, schema, configuration, or output path prevented normal processing

Duplicate-key removal is reported as a warning and does not by itself cause exit code 1.