Repository navigation
Raw citation control markers such as citeturn1view0 leak into Codex TUI output #3150
Description
Activity
Automated translation bookkeeping — detected language: English.
Current
devconfirms one important part of this report: there is no handler for the exact U+E200 / U+E202 / U+E201 citation grammar, and the GitHub Copilot Responses repair currently relays ordinary assistantresponse.output_textunchanged. If Copilot sends this syntax as literal text, OpenCodex will expose it to Codex exactly as reported.I am keeping this open as a real compatibility candidate, but I am not adding a global private-use-character scrubber from the report alone. The reproduction uses OpenCodex 2.10.1-preview while current
devhas a newer provider-scoped Copilot stream repair, and the first component that creates the marker is still unproven. A global or bridge-wide strip could also corrupt ordinary assistant output that is intentionally discussing this syntax.Please attach one sanitized reproduction from the current release or current
devcontaining:- the upstream Copilot SSE blocks for the affected assistant text;
- the corresponding downstream OpenCodex SSE blocks;
response.output_text.delta,.done,content_part.done,output_item.done, andresponse.completedwhen present;- the non-streaming response for the same prompt if it also reproduces.
Remove Authorization, account ids, request prompts, and unrelated content; keep only event types and the citation-bearing text fields. Do not post credentials.
If the marker is already literal upstream, the scoped fix will live in the GitHub Copilot Responses repair: an exact-grammar, stateful per-output normalizer for split deltas plus the authoritative done/completed snapshots, with raw user/tool content untouched and existing structured
url_citationannotations preserved. If it first appears downstream, the fix belongs at that exact translation step instead. This evidence decides the correct boundary rather than only the visible symptom.리뷰 · 우선순위 56 / 80
설명
이 이슈는 Codex CLI가 OpenCodex를 통해 GitHub Copilot 제공자를 쓸 때, 웹 인용이 OpenAI/ChatGPT 스타일 citation control marker 그대로 TUI와 트랜스크립트에 남는다는 호환 버그입니다. 예시는
citeturn1view0turn1view1이고, 유니코드 private-useU+E200/U+E202/U+E201로 감싼cite토큰입니다. turn-scoped opaque id라 사용자에게 보여줄 값이 아닙니다. 보고 환경은 오래된 preview 패키지@bitkyc08/opencodex@2.10.1-preview...와 Codex CLI 0.151.0 입니다.지금 HEAD를 보면
src/web-search과src/bridge.ts는url_citation어노테이션을 Sources 칩으로 붙이는 경로가 있습니다. 하지만 Copilot/업스트림이 assistant 텍스트 안에 private-use cite 마커를 인라인으로 넣는 경우, TUI는 그것을 렌더하지 않고 글자 그대로 보여 줍니다. OpenCodex가 응답 텍스트를 클라이언트로 전달하기 전에 마커를 제거하거나url_citation으로 정규화하는 계층이 있는지는 이 이슈만으로 단정할 수 없습니다.rg기준U+E200/stripCit전용 처리는 src에 보이지 않았습니다.버전 괴리가 큽니다. 보고는 2.10.1-preview 이고 현재 패키지는 2.40.0, HEAD는 remote hub 이후입니다. 먼저 2.39+/2.40 에서 Copilot+web citation 재현이 되는지 확인해야 합니다. 재현되면 Responses/Chat 정규화 한곳(예: adapter 또는 bridge 출구)에서 private-use cite 블록을 strip 또는 변환하는 패치가 맞습니다. types/config 분할 무관합니다. 점수는 56입니다. 보기 흉한 UX이지만 라우팅/인증 사고는 아닙니다.
경로 유니코드 U+E200–E202 cite 블록 - TUI가 모르는 presentation syntax입니다. 링크 URL이 마커 안에 없고 turn id만 있으면 strip이 안전합니다
경로 src/bridge.ts url_citation - desktop Sources 칩 경로와는 별개일 수 있습니다. Copilot 인라인 마커는 다른 입구입니다
경로 보고 버전 2.10.1-preview - 현재 2.40.0과 너무 멉니다. 재현 요청이 먼저입니다
경로 GitHub Copilot provider - #3156/#3163 context window와 같은 제공자 축이지만 증상이 다릅니다. 한 PR로 묶지 마십시오메인테이너의 판단이 필요한 지점
- strip-only vs url_citation 변환 중 무엇인지. URL이 없으면 strip이 맞습니다
- Codex 쪽 렌더 수정에 맡길지, 프록시가 항상 정규화할지
- 최신 버전 재현 전까지 needs-repro 로 둘지
너의 추천
최신dev/2.40.0 에서 Copilot 인용 재현을 요청하는 댓글을 남기고, 재현되면 private-use cite 블록 strip 패치를 adapter/응답 출구에 둡니다. #3163과 묶지 않습니다. 라벨은 바꾸지 않습니다.이 댓글은 grok-bot이 작성했습니다
Translated Message
Original language: Korean
Review · Priority 56 / 80
Description
This issue is a compatibility bug where, when Codex CLI uses the GitHub Copilot provider through OpenCodex, web citations remain as OpenAI/ChatGPT-style citation control markers in the TUI and transcripts. An example is
citeturn1view0turn1view1, which is acitetoken wrapped in Unicode private-use charactersU+E200/U+E202/U+E201. It's a turn-scoped opaque ID and should not be shown to users. The reported environment uses an older preview package@bitkyc08/opencodex@2.10.1-preview...and Codex CLI 0.151.0.Looking at HEAD now,
src/web-searchandsrc/bridge.tshave paths that attachurl_citationannotations as Sources chips. However, when Copilot/upstream inlines private-use cite markers inside assistant text, the TUI doesn't render them and displays them as literal characters. It cannot be determined from this issue alone whether there is a layer in OpenCodex that removes markers or normalizes them tourl_citationbefore passing response text to the client. Based onrgsearches, no dedicated handling forU+E200/stripCitis visible in src.There is a large version gap. The report uses 2.10.1-preview, the current package is 2.40.0, and HEAD is post-remote hub. First, we need to confirm whether Copilot+web citation can be reproduced in 2.39+/2.40. If reproduced, a patch that strips or converts private-use cite blocks in one place for Responses/Chat normalization (e.g., adapter or bridge exit) would be appropriate. Separation of types/config is irrelevant. The score is 56. It's an ugly UX but not a routing/authentication concern.
Paths:
- Unicode U+E200–E202 cite blocks - presentation syntax unknown to TUI. If there's no link URL inside the marker and only a turn ID, stripping is safe
src/bridge.tsurl_citation - may be separate from the desktop Sources chip path. Copilot inline markers are a different entry point- Report version 2.10.1-preview - too far behind current 2.40.0. Repro request comes first
- GitHub Copilot provider - same provider axis as Opencodex doesnt recognise context window from the github-copilot provider #3156/fix(catalog): read GitHub Copilot context window metadata #3163 context window but different symptoms. Do not bundle into one PR
Points requiring maintainer judgment:
- Whether to do strip-only or url_citation conversion. If there's no URL, stripping is correct
- Whether to leave render fixes to the Codex side or have the proxy always normalize
- Whether to keep it as needs-repro until reproduction with the latest version
Recommendation:
Leave a comment requesting reproduction of Copilot citations in the latest
dev/2.40.0, and if reproduced, place a private-use cite block stripping patch at the adapter/response exit. Do not bundle with #3163. Do not change labels.This comment was written by grok-bot
- added a commit that references this issue
on Sep 1, 2026 Fixed by #3195, landed on dev as f3bcc67.
Your diagnosis was right on the first hypothesis. Searching the source for U+E200/U+E201/U+E202 and citeturn returns nothing: OpenCodex neither emits nor recognizes that grammar, and its citation support is entirely the structured url_citation path you found. The markers are already literal text in the upstream response, because the Copilot backend is ChatGPT-derived. The proxy is simply the last place that can see them before a client that cannot render them, so that is where the strip now happens.
Two things about the implementation worth knowing, since you clearly read the bridge:
A span can straddle a delta boundary — U+E200cite in one chunk and the rest in the next — so the streaming path uses a stateful filter rather than a per-delta strip. It withholds an unterminated tail and releases it at close, so a stream dying mid-marker still delivers the words the model produced instead of swallowing them.
The accumulated text is stripped separately, because closeCurrentMessage re-sends it in output_text.done, content_part.done and output_item.done. Filtering only the deltas would have produced a clean-looking stream and a transcript that still contained the markers — which is the half of your report about the saved transcript.
On your three expected behaviors: this implements option 3. Options 1 and 2 are not reachable here — turnNviewN ids are turn-scoped and opaque, and the response carries no mapping from them to a URL, so there is nothing to convert them into. The structured url_citation path is untouched, so desktop Sources chips are unaffected.
One caveat: I could not obtain a live Copilot citation response, so this is verified against synthesized text matching the exact grammar you documented plus the real bridge end-to-end. If you see a variant shape after upgrading, please reopen with the raw codepoints.
Thank you for the report — naming the exact codepoints and proposing the three candidate origins turned this from an investigation into a confirmation.
- added a commit that references this issue
on Sep 14, 2026 - added a commit that references this issue
on Sep 17, 2026
Summary
When Codex CLI is routed through OpenCodex to the GitHub Copilot provider, assistant messages containing web citations can display OpenAI/ChatGPT-style citation control markers literally:
The same raw markers are retained in the Codex transcript. They are not rendered as source links, converted to a readable citation, or removed.
The delimiters are Unicode private-use characters. The example above is:
U+E200begins the citation marker.U+E202separatesciteand each source reference.U+E201ends the marker.turn1view0are opaque, turn-scoped source identifiers and are not useful when displayed directly to the user.This appears to be a response-compatibility issue: citation presentation syntax reaches the TUI as ordinary assistant text, but the TUI does not understand or render it.
Environment
Observed on September 1, 2026 with:
@bitkyc08/opencodex@2.10.1-preview.202608052.10.1-preview.20260805-4f47f002f0b85fef0.151.0github-copilotgithub-copilot/gpt-5.6-sol26.6.2, Apple Silicon (arm64)TERM=xterm-256color,COLORTERM=truecolorMinimal relevant Codex configuration:
Reproduction
Start OpenCodex:
Configure Codex CLI to use the local OpenCodex Responses endpoint and a GitHub Copilot-backed model:
Start Codex CLI.
Ask a question that causes a web lookup and requires source citations. For example:
Observe a citation-bearing commentary or final assistant message.
Open the Codex transcript with
Ctrl+T.Actual behavior
The TUI and transcript contain literal private-use citation syntax:
Consequences:
The issue occurs in both intermediate commentary and the final answer, so it does not appear limited to one Codex message phase.
Expected behavior
Raw citation control markers should never be visible in user-facing assistant text.
Depending on available source metadata and client capability, OpenCodex should do one of the following:
url_citationannotations that the client can render.Sourceslist.The desktop citation/Sources-chip behavior should remain intact.
Relevant OpenCodex implementation details
The installed source already has structured web-source support:
src/types.tsOcxUrlCitationrepresents a web source.url_citationannotations while the TUI ignores annotations.src/web-search/loop.tsweb_search_call_end.src/bridge.tspendingWebSources.takeWebAnnotations()converts them intourl_citationannotations.closeCurrentMessage()emitsresponse.output_text.done,response.content_part.done, andresponse.output_item.done.flushText()path also attaches annotations.src/web-search/parse.tsurl_citationannotations.Sources:block.I could not find handling for the private-use marker grammar (
U+E200,U+E202, andU+E201) in the installed OpenCodex source.The observed
turnNviewNreferences may originate from a Codex-hosted web tool rather than OpenCodex's synthetic web-search sidecar. Therefore, the first diagnostic step should be to determine whether the markers are:This report does not assume which component originally creates the marker. The actionable compatibility problem is that the OpenCodex-routed path allows it to reach user-visible TUI text without a usable fallback.
Suggested investigation
Capture one sanitized failing streaming turn at both sides of the proxy:
Inspect these downstream event fields in particular:
Questions to answer:
url_citationannotations present at the same time as literal markers?Possible fix direction
If OpenCodex receives these markers as literal provider text, add a narrowly scoped citation normalizer at the assistant-output bridge boundary.
Important implementation constraints:
response.output_text.deltaboundaries..delta,.done, content-part, output-item, and completed snapshots remain consistent.url_citation/Sources-chip behavior.A stateful stream normalizer may need to retain a small suffix beginning at
U+E200until it seesU+E201or can determine that the sequence is not a valid citation marker.Suggested tests
Add unit/integration coverage for both streaming and non-streaming response paths:
One citation with one source:
One citation with multiple sources.
Multiple citations in one assistant message.
A citation adjacent to punctuation and at end-of-message.
Every possible application-level split between Unicode scalar values within the marker. If normalization happens below SSE decoding, also test transport-byte splits within a multibyte UTF-8 delimiter.
A marker split across the final delta and
.doneevent.Existing structured
url_citationannotations without marker text.Marker text plus structured annotations, ensuring sources are not duplicated.
A malformed/unclosed marker, ensuring output is deterministic and no stream data is lost.
Ordinary Unicode and unrelated private-use characters, ensuring they remain unchanged.
Commentary-phase and final-answer messages.
Transcript/snapshot output, ensuring raw markers are not persisted.
Acceptance criteria
U+E200 cite U+E202 ... U+E201sequence is displayed in Codex TUI output.