Skip to content

Check plugin output and post markup, fix post errors - #89

Merged
bits-bytes-nn merged 2 commits into
mainfrom
review-r1
Oct 13, 2026
Merged

bits-bytes-nn merged 2 commits into
mainfrom
review-r1

Conversation

@bits-bytes-nn

Copy link
Copy Markdown
Owner

Adversarial review rounds 1-16 plus a final round under the agreed criterion (defects in current posts or output, false positives on existing posts, defects from patterns existing posts use). Final round: 0 MAJOR.

_data/post_conventions.yml (scholar-lens copies this file)

  • New key languages: [ko, en]: the allowed lang values; a translation is one post in each.
  • Comments reworded (header; forbidden_keys note on last_modified_at).

New lint rules (script/lint-posts.rb), with pass / fail examples

Rule Fails Passes
$$ math needs use_math: true 식 $$x$$입니다. without the flag the same with use_math: true; echo $$ in a fence
| inside $$ in a table row (kramdown splits the cell; | becomes |) | $$p(y|x)$$ |, | $$|\theta|$$ | | $$p(y \mid x)$$ |, | $$\lVert w \rVert$$ |
translation pair: one ko and one en post share translation_id; an -en post sets it Korean original without translation_id both posts with it
description: is plain text $$, *emphasis*, backticks, ](, <tag>, {{ Mixtral 8*7B와 C++ [1]
Liquid outside {% raw %} ```text\nQuestion: {{ question }}\n``` the same inside {% raw %}…{% endraw %}; render_with_liquid: false
tags read as HTML <think>, <s>, <input>, bare <img> in prose `<think>`, \<think\>, <br>, <img src=…>, i<j
code fence kramdown does not read ```python title="a.py"; a fence never closed ```python
file name YYYY-MM-DD-; date: not after it (UTC) date: 2025-03-01 in 2025-02-27-*.md a paper's earlier date
tag spelled two ways across posts Sliding-window-Attention next to Sliding-Window-Attention one spelling
only .md files under _posts/; front matter on line 1 x.mkd; a blank line before --- —

Other gates

  • script/check-plugin-output.rb (run by validate-site.sh): reads the posts through Jekyll, checks each hook's mark (legacy redirect stub, modified time, prev/next, image loading/size, table container), the feed cap, and block HTML kramdown could not parse (closing tag shipped as text).
  • validate-site.sh: every Ruby check fails when it crashes; listing check scans every non-post page; hreflang-gone guard; site author: required; external redirect_to and directory/bare permalinks handled.
  • _config.yml: liquid.strict_filters: true; future: true (a same-day post dated 12:00 UTC no longer drops out of a morning deploy).
  • CI htmlproofer no longer passes --no-enforce-https; all arXiv links are https.

Hand fixes to generated posts (source value vs. what the post said)

  • Llama 3 405B: MMLU 87.3, GSM8K 96.8 (Table 2); post said 85.2 / 89.0 (fixed in Add descriptions to paper posts and fix factual errors #88, listed for completeness).
  • Gemini 2.5: SFT data "loosely inspired by Constitutional AI" plus human revision; RL reward from a reward model and a rubric Critic; post said the model critiques itself via Constitutional AI. Video and benchmark figures attributed to 2.5 Pro.
  • DeepSeek-V3: MMLU-Pro 75.9 / GPQA 59.1 best among open models; MATH-500 90.2 best among compared models; post said best overall.
  • LightRAG: GraphRAG retrieval needs "hundreds of" API calls, 610 × 1,000 / C_max; post said 610.
  • Gemma 2: 0.00026% is the share of memorized outputs with low-severity personal data (post: "emission rate"); the 15T-token point cites Llama 3, not Chinchilla (two places).
  • Gemma 3: Gemma 3 27B IT tied 9th (Elo 1338); Llama 3.1 405B 1269 (post said 1257, Qwen2.5-72B's score); GPT-4.5-Preview tied 1st.
  • Qwen3: the 1/3 total, 2/3 active comparison is Qwen3-235B-A22B-Base vs DeepSeek-V3-Base, ahead on 14 of 15.
  • DeepSeek-V2: AlpacaEval 38.9 / MT-Bench 8.97 are DeepSeek-V2 Chat (RL).
  • Zep: LongMemEval 15.2% / 18.5% are relative gains (55.4→63.8, 60.2→71.2); post said percentage points.
  • LLaMA, DeepSeek-Coder, Mixtral, Mistral 7B: "most benchmarks" / "almost all" where the post's own tables show losses.
  • RAPTOR: +20 percentage points on QuALITY. GraphRAG: "over 97%" fewer tokens.
  • DeepSeek-R1: <think>/<answer> in the prompt-template table were swallowed as HTML; now code spans.

Not changed (pending a decision)

Paper posts publish date: = paper date as datePublished / feed published.

Gates: tests (169 runs), post lint, strict build, htmlproofer (CI flags), validate-site, shellcheck, actionlint; MathJax 0 errors on all 30 math posts.

- script/check-plugin-output.rb reads the posts through Jekyll and checks
  what each hook adds (redirect stubs, modified time, prev/next, image
  loading and size, table containers), the feed cap, and block HTML that
  kramdown could not parse. validate-site.sh runs it and fails when any
  of its Ruby checks crash.
- lint-posts.rb: use_math, translation pairs, a | inside $$ in tables,
  plain-text descriptions, Liquid outside raw, non-HTML tags such as
  <think>, fences kramdown does not read, date and file-name rules, tags
  spelled two ways. Allowed languages move to post_conventions.yml.
- _config.yml: strict_filters, and future: true so a same-day post is
  not dropped from a morning deploy.
- CI enforces https links; unused Sass and Liquid branches removed.
Paper numbers checked against the papers, arXiv links over https,
<think> tokens shown as code, English captions on the English post.
@bits-bytes-nn
bits-bytes-nn merged commit 36ccff2 into main Oct 13, 2026
3 checks passed
@bits-bytes-nn
bits-bytes-nn deleted the review-r1 branch October 13, 2026 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant