Add verbatim ord_text field to one-shot extraction schemas by rajeee · Pull Request #487 · NatLabRockies/COMPASS

rajeee · 2026-06-29T13:54:48Z

Adds a verbatim ord_text field to the one-shot extraction schemas (geothermal electricity, GHP, water rights demo), capturing the source sentence an ordinance value/requirement was extracted from, unparaphrased.

Threads ord_text through finalize.py so it routes into both the qualitative and quantitative output CSVs, and through the SchemaOrdinanceParser column list. Includes a regression test confirming the value round-trips into both output files.

codecov-commenter · 2026-06-29T14:02:04Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.94%. Comparing base (049f03f) to head (2c50fa8).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #487      +/-   ##
==========================================
+ Coverage   63.83%   63.94%   +0.11%     
==========================================
  Files          78       78              
  Lines        7446     7469      +23     
  Branches      755      757       +2     
==========================================
+ Hits         4753     4776      +23     
  Misses       2532     2532              
  Partials      161      161

Flag	Coverage Δ
unittests	`63.94% <100.00%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR adds an ord_text field to one-shot extraction schemas and threads it through the one-shot parsing and finalization pipeline so the verbatim source sentence can be emitted into final CSV outputs, with a regression test ensuring the field round-trips.

Changes:

Add required ord_text to one-shot schemas (geothermal electricity, GHP, water rights demo) with instructions to keep it verbatim.
Include ord_text in one-shot parser DataFrame construction and in finalized output column sets.
Extend unit test coverage to assert ord_text is written to both quantitative and qualitative output CSVs.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`compass/utilities/finalize.py`	Adds `ord_text` to parsed/output column definitions and qualitative output column selection.
`compass/plugin/one_shot/components.py`	Ensures one-shot parsing output includes `ord_text` when present in schema outputs.
`tests/python/unit/utilities/test_utilities_finalize.py`	Regression test for `ord_text` round-tripping into both output CSVs.
`examples/water_rights_demo/one-shot/water_rights_schema.json5`	Adds required `ord_text` field, updates examples/instructions for verbatim extraction.
`compass/extraction/ghp/geothermal_heat_pump_schema.json5`	Adds required `ord_text` field plus guidance/examples.
`compass/extraction/geothermal_electricity/geothermal_schema.json`	Adds required `ord_text` field plus guidance/examples.

ppinchuk · 2026-07-07T21:15:59Z

      "Extract only enacted district requirements, not proposed language or general background text.",
      "Use direct excerpts/quotes in summary whenever possible.",
+      "Whenever a feature row is emitted, ord_text must be the first full sentence from the source document corresponding to the requirement being extracted, copied verbatim: for quantitative features the full source sentence containing the extracted value, and for qualitative features the first full source sentence relating to the requirement. Unlike summary, ord_text must be a single contiguous sentence reproduced exactly as written, with no paraphrasing, normalization, ellipses, added context, or commentary. If a feature has no requirement, omit the feature row entirely rather than emitting a row with an empty ord_text.",
      "If a feature has no requirement, set value, units, section, and summary to null or omit the feature row.",


This might be worth adjusting?

ppinchuk

This is really nitpicky, but can we use ord_first_sentence instead of ord_text by chance? In my experience, a more explicit key name actually helps the model, and I am hping this would distinguish it enough from the summary output

ppinchuk · 2026-07-07T21:17:47Z

          "units": "HH:MM (24-hour)",
          "section": "Section 6.7 - Drilling Operations",
          "summary": "'Routine geothermal drilling activities may occur only between 7:00 a.m. and 7:00 p.m., Monday through Saturday, and shall not occur on Sundays or legal holidays except in an emergency.'",
+          "ord_text": "Routine geothermal drilling activities may occur only between 7:00 a.m. and 7:00 p.m., Monday through Saturday, and shall not occur on Sundays or legal holidays except in an emergency.",


Can we add one or two examples where the sentence for the ordinance text extraction is completely different than the summary?

We should remove the summary - and use value for both qualitative and quantitative features (like how we ask the interns to do)

That would break the data model we have established here and here. While we could do that, we should have a strong reason for it

ppinchuk · 2026-07-29T17:02:23Z

After meeting, this was decided:

Qualitative features should now be reported under "Value" column.
Qualitative features have "str" as a unit
LLM explanation now output in the CSV (can be "Explanation" or "LLM Reasoning" or something else)
Summary column is deprecated
New "ordinance_text" column captures the complete relevant text excerpt containing the ordinance information, copy-pasted from the document, potentially with ellipses (...) to remove irrelevant info. The first sentence should be the first part of the ordinance text pertaining to the extraction (and maybe even require it to be reported in full? Decision up to Rajendra for use in QA tool). Text may be manually trimmed down to a certain limit (5k chars?) after LLM output/
Two output CSV's combined into a single one

Resolve conflicts from main's plugin-driven output-column refactor. Main replaced the hardcoded output column lists with schema-derived OutputColumn objects, superseding the manual list edits on this branch: - utilities/finalize.py: _PARSED_COLS / QUANT_OUT_COLS / QUAL_OUT_COLS were removed in favor of columns passed in from the plugin's OUTPUT_COLUMNS. Took main's version; the "ord_text" entry added here is no longer needed. - plugin/one_shot/components.py: the literal possible_out_cols list was replaced by self.POSSIBLE_OUT_COLS. Took main's version for the same reason. - extraction/ghp/geothermal_heat_pump_schema.json5: combined both sides by keeping main's expanded summary guidance alongside the ord_text guidance from this branch. The ord_text column still reaches both CSV outputs because _out_cols_from_config() derives output columns from each schema's "required" array, where ord_text is already listed. Verified for the GHP, geothermal electricity, and water rights schemas. Also dropped the ord_text assertions from test_save_db_writes_csvs. That test auto-merged without a conflict but was left broken: it exercises COMPASSWindExtractor, which has no ord_text column, so save_db() would never emit it.

Per the decisions from the extraction output meeting: - Rename the column to "ordinance_text" in all one-shot schemas (GHP, geothermal electricity, water rights demo). - Change the semantics from "a single verbatim sentence, no ellipses" to "the complete relevant excerpt". The first sentence must still be the start of the ordinance text and must be reproduced in full, but an ellipsis may now be used after it to drop irrelevant passages. Everything outside the ellipses stays verbatim. Also add shared ORDINANCE_TEXT_PROMPT and EXPLANATION_PROMPT templates and wire them into the legacy decision-graph output nodes for wind, solar, small wind, and the shared extra-restriction graphs, so those plugins produce the same two fields the one-shot schemas do. The legacy min_dist/max_dist nodes are deliberately left alone: their results are merged with dict.update() over the main setback result, so emitting ordinance_text there would overwrite the primary excerpt. System-size classification nodes are also unchanged since they do not produce ordinance rows. No plugin declares the new legacy columns yet, so wind/solar/small_wind CSV output is unchanged by this commit. Note that there is no eval suite for wind or solar (only date extraction), so the quality of ordinance_text under the legacy multi-turn prompts is not yet measured.

Implements the output-format decisions from the extraction meeting, uniformly across schema-driven and legacy plugins. - Qualitative rows now carry value and units. Output columns no longer vary by row kind, so include_in_qual_output / include_in_quant_output are no longer consulted when writing. - The LLM explanation is now written to the CSV. _out_cols_from_config previously filtered it out. - summary is deprecated and no longer written. It stays in the one-shot schemas (so the model still reasons through it) but is dropped from the output via _DEPRECATED_OUT_COLS, and is removed outright from the legacy plugin column lists. - ordinance_text is added to the legacy plugin column lists, so the fields the graph prompts already request now reach the output. - Both CSVs are replaced by a single ordinances.csv. The quantitative flag is retained as a column so the two row kinds stay separable, and the run manifest now exposes one DATA_FILE key. This also fixes a pre-existing bug where QUAL_DATA_FILE pointed at the quantitative CSV. - ordinance_text is trimmed to 5000 characters on the way to disk, cut at a word boundary and marked with a trailing ellipsis. The RMP schema packed a summary and a justification into one field; that is split into ordinance_text (verbatim excerpt) and explanation (justification), and all 16 few-shot examples are rewritten so they demonstrate verbatim quoting rather than paraphrase. The water rights demo schema gains the explanation field it was missing. Breaking change for anyone reading qualitative_ordinances.csv or quantitative_ordinances.csv, including the validation notebooks under support/validation. There is no eval suite for wind, solar, or small wind, so the quality of ordinance_text and explanation under the legacy multi-turn prompts is still unmeasured.

Now that both kinds of row share one CSV, qualitative features get the literal string "str" in the units column. Selecting one kind of feature is then a plain column comparison, without needing the boolean quantitative flag. The sentinel is derived from the quantitative flag at write time rather than requested from the LLM. That keeps the invariant (quantitative is False) == (units == "str") true by construction, and avoids a failure mode where the model writes "string", "text", or an invented unit for a qualitative feature. Any such invented unit is overwritten. Applied in both _formatted_db and save_db so it holds no matter which path assembled the frame; the operation is idempotent.

The units sentinel ("str" on qualitative rows) already distinguishes the two kinds of feature, so the boolean quantitative column is redundant in the published file. It is removed from OUTPUT_COLUMNS for the one-shot path and for wind, solar, and small wind. The flag is still needed internally to derive the sentinel, so _formatted_db now carries it alongside the output columns and save_db projects it away just before writing. _formatted_db also tolerates frames that arrive without the column, which previously could not happen because it was always part of parsed_cols. Also fixes a bug introduced when summary was deprecated: ordinances_bool_index decides whether a row is an ordinance by checking for a non-null value or summary. Qualitative rows have neither once summary is gone, so they were being silently dropped from the output. ordinance_text now counts as evidence too; summary stays in the list so previously written per-jurisdiction CSVs still load. Lint: trim_ordinance_text moves to compass.utilities.finalize (where it was already applied) and is re-exported from compass.plugin.post_processing for the step registry. That removes the function-level import that import-outside-top-level flagged, which existed only to dodge a circular import. Verified with `ruff check ./compass` and `ruff format --check ./compass` on ruff 0.16.2, matching the "latest" pin the CI lint job uses.

Reverses the summary deprecation. The four content columns now have clearly separated jobs: value the extracted answer; for a qualitative feature this is the requirement itself, stated in full summary prose restatement of the rule, carrying caveats and conditions that value and units cannot hold ordinance_text exact quotes copied from the source document explanation the model's reasoning about its interpretation summary no longer asks for quotes, since verbatim text is what ordinance_text is for. It is also never blank: on qualitative rows the value is copied into it at write time, in _normalize_qualitative_rows alongside the units sentinel. Doing it there rather than in the prompt keeps the two columns identical and saves the model writing the same text twice. Backward compatibility removed, since this is a single application that never reads previous-generation output: - "summary" is out of _ORD_CHECK_COLS; ordinance_text is the evidence column that decides whether a row is an ordinance. - The "column not in db.columns" guards in _formatted_db and the qualitative labeling are gone. Every parser sets "quantitative" on each row it emits, and every plugin that reaches this code declares units, so the guards only masked bugs. The invariant is documented on _formatted_db instead. - _DEPRECATED_OUT_COLS is deleted. Test fixtures that used summary as a row's only ordinance evidence now use ordinance_text, matching what the parsers actually produce.

The previous commit had the copy backwards. The schemas ask the LLM for a null value on qualitative rows and put the requirement in summary, so copying value into summary blanked the one field that had the text. _normalize_qualitative_rows now copies summary into value instead, which matches what the schemas already request and keeps the model writing the requirement only once. Both columns end up populated on every row. The summary description is updated to say plainly that it is the primary output field for qualitative features, and that value is filled in from it when the output is written. Also widens the value column to object dtype before the copy. A numeric-only value column is float-typed, and writing qualitative text into it raised a pandas FutureWarning that becomes an error in a later release. Numeric values are unaffected.

Two items from Paul's review on #487. Examples now show summary and ordinance_text diverging (his request for "one or two examples where the sentence for the ordinance text extraction is completely different than the summary"). Every example summary used to be the ordinance_text wrapped in quotes, which demonstrated exactly the redundancy the two-field split is meant to remove. All 13 examples across the GHP, geothermal electricity, and water rights schemas are rewritten so summary reads as prose while ordinance_text stays verbatim, and several now show a multi-sentence excerpt joined with an ellipsis so the elision convention is demonstrated rather than only described. Examples that were missing explanation now have one. The water rights instruction block he and Copilot both flagged is fixed. It carried two contradictions: it still described ordinance_text as a single contiguous sentence with no ellipses, which is the semantics that was replaced everywhere else, and it told the model it could emit a row with null fields when a feature has no requirement even though ordinance_text is required and non-empty. The rule now matches the current definition and says plainly to omit the row.

rajeee · 2026-08-11T17:45:21Z

          "summary": {
            "type": "string",
-            "description": "A short summary with direct ordinance excerpts or quotes whenever possible. For qualitative features such as permitting, fencing, lighting, seismic monitoring, decommissioning, and prohibitions, this is the primary output field and should contain direct ordinance language. For numeric features, summary must support the same requirement used to extract value and units. Must be a non-null, non-empty string. Do not output absence placeholders such as 'No explicit requirement found'; omit the feature instead when no requirement is present."
+            "description": "A natural-language restatement of the requirement in your own words. For qualitative features this is the primary output field: state the requirement in full here, since the value column is left null and is filled in from this field when the output is written. For quantitative features, use it to capture caveats, conditions, exceptions, and alternative thresholds that the value and units columns cannot hold on their own (for example tiered limits, conditional reductions, or requirements that vary by district or system size). Do not quote the document here: verbatim text belongs in ordinance_text, and this field should read as prose rather than an excerpt. Must be a non-null, non-empty string."


Maybe include the "such as permitting, fencing, lighting, seismic monitoring, decommissioning, and prohibitions"

rajeee · 2026-08-11T17:48:06Z

+          },
+          "ordinance_text": {
+            "type": "string",
+            "description": "The complete relevant text excerpt containing the ordinance information, copy-pasted verbatim from the source document. The first sentence must be the beginning of the ordinance text pertaining to this extraction, reproduced in full without elision. After that first sentence, you may use an ellipsis ('...') to omit passages that are not relevant to this ordinance, keeping the excerpt focused while preserving all text that supports the extracted value, units, or qualitative requirement. Every character outside of the ellipses must appear exactly as written in the source: no paraphrasing, summarizing, normalization, added context, or commentary. Must be a non-null, non-empty string."


Refine the prompt to allow going both up and down the document to include other relevant excerpts while maintaining that the first sentence should correspond to the extracted value / summary.

rajeee · 2026-08-11T19:40:07Z

    "title of the section (including numerical labels), if it's given, "
    "and `null` otherwise."
 )
 _SUMMARY_PROMPT = (


This needs to be rewritten. We should no longer require summary to have direct text excerpts since that is going to be available in the ordinance text column.

rajeee · 2026-08-11T19:43:13Z

          "summary": {
            "type": ["string", "null"],
-            "description": "A structured text field with two labeled sections separated by a newline. Format exactly as: 'Summary: [summary text]\\nJustification: [justification text]'. The Summary section (2–4 sentences) covers: (1) the area name and acreage if stated, (2) the exact language used in the document describing what is restricted (quoted where possible), (3) the stated reason or resource being protected, and (4) any conditions, exceptions, or relevant context. Use direct text excerpts and quotes from the document where possible. The Justification section (1–2 sentences) explains why this restriction was included — specifically, how it relates to or could affect geothermal leasing or development (e.g., 'Included because mineral leasing closures under the Mineral Leasing Act encompass geothermal leasing.', 'Included because this NSO stipulation applies to all surface-disturbing activities associated with mineral leasing, which includes geothermal drilling.'). Null only if no relevant information is found."
+            "description": "A natural-language restatement of the requirement in your own words. For qualitative features this is the primary output field: state the requirement in full here, since the value column is left null and is filled in from this field when the output is written. For quantitative features, use it to capture caveats, conditions, exceptions, and alternative thresholds that the value and units columns cannot hold on their own (for example tiered limits, conditional reductions, or requirements that vary by district or system size). Do not quote the document here: verbatim text belongs in ordinance_text, and this field should read as prose rather than an excerpt. Must be a non-null, non-empty string."


I don't think this needed to change. We should keep existing description of summary.

rajeee · 2026-08-11T19:44:26Z

      "units": null,
      "section": "Minerals and Energy: Areas Closed to Oil, Gas and Geothermal Leasing",
-      "summary": "Walker Lake is closed to oil, gas, and geothermal leasing as part of 'Key Scenic, Wildlife, Recreation, and Historic Areas' totaling 45,392 acres. Document language: 'Areas Closed to Oil, Gas and Geothermal Leasing.' Individual acreage not specified; group total is 45,392 acres.",
+      "ordinance_text": "Areas Closed to Oil, Gas and Geothermal Leasing. ... Key Scenic, Wildlife, Recreation, and Historic Areas ... 45,392 acres.",


Summary should still be present

rajeee · 2026-08-11T19:44:49Z

      "units": "acres",
      "section": "Minerals and Energy: Areas Closed to Mineral Entry (22,672 Acres)",
-      "summary": "Grimes Point Archaeological Area (400 acres) is closed to mineral entry under the General Mining Law. Stated reason: cultural resource protection. Note: a separate geothermal-specific closure also exists for this area (640 acres) under section 9.",
+      "ordinance_text": "Areas Closed to Mineral Entry (22,672 Acres) ... Grimes Point Archaeological Area (400 acres).",


summary should remain unchanged - only two new columns should be added.

rajeee · 2026-08-11T19:54:25Z

            {
                "feature": "prohibitions",
                "value": None,
-                "summary": "older prohibition",


Does this need to change?

Scopes the PR back to adding two columns, combining the outputs, and the qualitative value/units handling. rmp_schema.json (:85, :150, :164): summary is restored byte-for-byte, description and all 16 example values, so the RMP diff is now purely additive. Its two-section Summary/Justification format is left alone as the deliberate exception; ordinance_text and explanation are simply added beside it. common/base.py (:20) and the geothermal, GHP, and water rights schemas: the original summary wording is restored, including the per-schema lists of qualitative features, with only the "use direct excerpts/quotes" clause replaced. Summary now asks for the gist of the requirement with all of its specifics and details, and points at ordinance_text for verbatim wording. geothermal_schema.json (:88): the qualitative feature list (permitting, fencing, lighting, seismic monitoring, decommissioning, prohibitions) comes back with the restored description. geothermal_schema.json (:92): ordinance_text may now gather passages from anywhere in the document, before or after the primary sentence, joined with an ellipsis in document order. The first sentence must still be the one corresponding to the extracted value and summary. Applied to all four schemas and the legacy prompt. test_plugin_ordinances.py (:435): reverted. "summary" goes back into _ORD_CHECK_COLS, so a row carrying only a summary counts as an ordinance again and the fixtures no longer needed to change. That file and test_utilities_parsing.py are now identical to their pre-PR state.

rajeee added 2 commits June 23, 2026 14:13

Add verbatim ord_text field to one-shot extraction schemas

2b33052

Add tests

e30f347

Copilot AI review requested due to automatic review settings June 29, 2026 13:54

rajeee requested review from castelao and ppinchuk as code owners June 29, 2026 13:54

Copilot started reviewing on behalf of rajeee June 29, 2026 13:59 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

ppinchuk requested changes Jul 7, 2026

View reviewed changes

rajeee added 8 commits August 10, 2026 10:54

rajeee commented Aug 11, 2026

View reviewed changes

Uh oh!

Conversation

rajeee commented Jun 29, 2026

Uh oh!

codecov-commenter commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ppinchuk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ppinchuk commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Jun 29, 2026 •

edited

Loading

ppinchuk commented Jul 29, 2026 •

edited

Loading