[BUG]: Ontology generation using AI is failing if there are lot of tables(100+) and columns

[manikiran0]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Ontology generation using AI is failing if there are lot of tables(100+) and columns.

Expected Behavior

AI Agent should intelligently process like iteratively or process related tables together that way it will be able to generate ontology for a big data set.

Steps To Reproduce

Just create a data set with 100 tables and each table having 15-20 columns and add it to data source and click on generate ontology.Agent will go in a loop saying truncated output and eventually it fails after some time.

Cloud

Azure

Browser

Chrome

OntoBricks Version

0.8

Relevant log output

Additional Context

No response

[benoitcayladbx]

Thanks for the report — this is a real limitation of the current generator, not a flaky run.

Conclusion

Ontology generation still produces one OWL/Turtle document in a single agent pass. Databricks Foundation Model endpoints cap output at ~8k tokens. A warehouse of 100+ tables × 15–20 columns cannot fit in that completion (especially with exhaustive attributes).

When the model hits that cap (finish_reason=length), the agent rejects the truncated Turtle and asks for a more concise re-emit. That is the loop you see (“truncated output”), until the iteration budget is exhausted and the run fails. Retrying the same full schema will keep failing: shorter comments do not shrink the ontology enough.

Input is also bounded: schema context is summarized (first tables / columns only). The agent is not designed to iteratively cluster related tables and merge modules today.

Workarounds (until the 0.9 flow lands)

There is no workaround that makes “select all 100 tables → Generate” succeed. Slice the schema and merge.

1. Generate in clusters
   In Ontology → Generate, check only ~8–15 related tables at a time (same subject: customer, meter, billing, …). Name the core entities in the guidelines (e.g. Customer, Contract, Invoice) instead of “model every table”. Stay well under ~40 classes per run.

2. Apply once, then merge
   Apply on the wizard replaces the current ontology.

   • First cluster: Generate → Apply.
   • Later clusters: Generate → copy the Turtle → Ontology → Import → append / merge (resolve name conflicts). Do not Apply again.

3. If a cluster still truncates
   Shrink the table set further. Optionally uncheck datatype properties for a first pass (classes + relationships only), then add attributes on a later pass. Skip large documents on the first generation.

What will not help: picking a “larger” serving endpoint (output cap is still ~8k), or waiting out the truncated-output retries.

What changes in 0.9

Ontology generation will be reshaped in 0.9 into a two-step flow:

1. The agent detects candidate entities from the schema (and documents).
2. You confirm or edit that entity list (add / remove / rename).
3. The agent then adds attributes and relationships only for the entities you accepted.

That splits the work that currently overflows a single completion, and matches the “process related tables iteratively” behaviour you described. Tracking this issue against that 0.9 redesign.

[benoitcayladbx]

I forgot ... in 0.9 by generation we'll limit the number of tables you'll be able to add, however you'll be able to work by appending the existing ontology, which will be a big add and should solve your issue by having better accuracy in the generation.

[amit10may]

The problem is not just with number of tables but even with single table having 400+ columns. I tried and got errors "ontology output is truncated. Retrying..." it retried for more than 5 times and I ran out of patience. The error gives no clue whats wrong. I was using claude opus 5 on databricks. could not find anyway to selectively add attributes. and even if it was possible, hard to imagine if it makes sense to iteratively add tables or attributes. wouldnt that eventually run out of context length?

[awolf-ed]

We're hitting the same problem on a client use case, but it doesn't seem to be driven by the number of tables alone; column count and input document size look like they matter just as much:

• Selecting a single table with 360 columns reproduces the same truncated-output loop and eventual failure.
• Same behaviour with another table of 164 columns.
• It also happens with 0 tables selected, when the attached document describing the business is very comprehensive.

Thanks for the workaround in the meantime.

[benoitcayladbx]

Thanks — that matches what @amit10may reported, and it updates the diagnosis.

The failure is output size in a single OWL/Turtle pass, not “too many tables”. Anything that produces a large Turtle hits the Foundation Model ~8k completion cap (finish_reason=length → “ontology output is truncated. Retrying…” until the run dies). Same loop for:

• many tables
• one wide table (164 / 360 / 400+ columns)
• no tables, if the attached document is large enough that the model tries to emit a full ontology from it

A table-count limit in 0.9 would not fix your cases. Picking Claude Opus (or any larger endpoint) does not raise that output cap either. The retry loop is also unhelpful: shortening the same full ontology never fits.

Workaround for wide tables / fat docs

The “8–15 tables” slice still helps warehouses. For one fat table or document-only:

1. Do not generate all attributes in one shot. First pass: classes + relationships only (uncheck datatype properties if the UI allows it, or ask in the guidelines: “classes and object properties only, no datatype properties”).
2. Split columns by theme (identity, address, metering, billing, …) and generate / import each slice, then append/merge — do not Apply again (Apply replaces).
3. Documents: attach a short extract (entity glossary, 1–2 pages), not the full spec. Or generate from a small table set first, then fold the document in later passes.
4. If you still see truncated output, stop the run — more retries will not succeed.

There is still no “generate this 360-column table in one click” path.

What we will actually change in 0.9

Besides a table cap and append to the existing ontology:

• Two-step generation: detect candidate entities → you confirm/edit → then attributes/relationships only for accepted entities.
• Bound each attribute pass (chunks of properties, not “all 360 columns at once”).
• Same bound for documents (chunk / summarize; never one-shot the full spec into Turtle).
• Fail fast on truncation with a clear message instead of silent retries.
• Append must send a delta / summary of the current ontology, not the whole Turtle every time — otherwise we just move the wall from output length to context length (Amit’s point).

Tracking this issue against that work. If you can share a rough column count + whether a document was attached, that helps us set the per-pass limits.

[sanjaysuresha]

Hi Team,
This is the real limitation of the ontobricks product version 0.8 v . Kindly fix this bug atleast in the next version or fixes .