Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions checks/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,16 @@ <h2>Numbers pinned to code</h2>
<td><code>pyproject.toml</code> extras</td>
<td><code>tests/test_packaging.py</code></td>
</tr>
<tr>
<td>Every Python package here type-checks</td>
<td>the organisation profile — "open source, typed, and tested"</td>
<td><code>mypy</code> in each repository's CI</td>
</tr>
<tr>
<td>No workflow runs an action it took from a moving tag</td>
<td>implied by every badge above, which those workflows produce</td>
<td><code>scripts/org_triage.py</code>, weekly</td>
</tr>
<tr>
<td>This site makes no third-party request of any kind</td>
<td>the legal page</td>
Expand Down Expand Up @@ -220,6 +230,11 @@ <h2>What runs on a schedule</h2>
<td>weekly</td>
<td>one standing issue names the page, what it says, and what the source says</td>
</tr>
<tr>
<td>Actions that have gone back to a moving tag</td>
<td>weekly, across every repository</td>
<td>the triage names the repository and the references — and says <em>unknown</em> rather than <em>fine</em> when it could not read the workflows</td>
</tr>
<tr>
<td>Organisation-wide maintenance triage</td>
<td>weekly</td>
Expand Down
15 changes: 15 additions & 0 deletions ru/checks/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,16 @@ <h2>Числа, прибитые к коду</h2>
<td>extras в <code>pyproject.toml</code></td>
<td><code>tests/test_packaging.py</code></td>
</tr>
<tr>
<td>Каждый Python-пакет здесь проходит проверку типов</td>
<td>профиль организации — «open source, typed, and tested»</td>
<td><code>mypy</code> в CI каждого репозитория</td>
</tr>
<tr>
<td>Ни один воркфлоу не запускает action, взятый с подвижного тега</td>
<td>подразумевается каждым бейджем выше — их делают эти воркфлоу</td>
<td><code>scripts/org_triage.py</code>, еженедельно</td>
</tr>
<tr>
<td>Этот сайт не делает ни одного запроса к сторонним доменам</td>
<td>юридическая страница</td>
Expand Down Expand Up @@ -221,6 +231,11 @@ <h2>Что работает по расписанию</h2>
<td>еженедельно</td>
<td>одна стоячая issue называет страницу, её версию и версию источника</td>
</tr>
<tr>
<td>Actions, вернувшиеся на подвижный тег</td>
<td>еженедельно, по всем репозиториям</td>
<td>триаж называет репозиторий и ссылки — и говорит <em>неизвестно</em>, а не <em>в порядке</em>, если не смог прочитать воркфлоу</td>
</tr>
<tr>
<td>Триаж обслуживания по всей организации</td>
<td>еженедельно</td>
Expand Down
50 changes: 48 additions & 2 deletions scripts/check_figures.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@
Each figure here names its source and is checked against it:

algorithms / environments decisionrl's CITATION.cff, which decisionrl's own
test suite pins to the package it ships
test suite pins to the package it ships -- checked
on this site and in the organisation profile, which
spells the same figures out in words and so escaped
every check that looked for digits
required dependencies glia's pyproject.toml, read directly
line coverage Codecov, the run that produced it

Expand All @@ -23,6 +26,11 @@
Standard library only. Needs network, which is why this runs in its own weekly
workflow rather than in the required CI check.

Sources are read through raw.githubusercontent.com, which serves from a cache
that can lag a commit by a few minutes. That is invisible to a weekly run and
confusing to anyone running this by hand right after a merge: if it reports a
figure you have just corrected, check the commit landed before believing it.

Usage:
python scripts/check_figures.py [site_root]
"""
Expand All @@ -41,6 +49,16 @@

PAGES = ("index.html", "ru/index.html")

# The organisation profile states the same counts in words. That is how "thirty-one
# algorithms and twenty-two environments" survived every correction made to the
# digits elsewhere.
PROFILE = f"{RAW}/.github/master/profile/README.md"
WORDS = {
22: "twenty-two", 23: "twenty-three", 24: "twenty-four", 25: "twenty-five",
29: "twenty-nine", 30: "thirty", 31: "thirty-one", 32: "thirty-two",
33: "thirty-three", 34: "thirty-four", 35: "thirty-five",
}


def fetch(url: str) -> str | None:
try:
Expand Down Expand Up @@ -147,7 +165,35 @@ def main(root: Path) -> int:
f"{name}: coverage says {stated}, Codecov measures {coverage:.1f}%"
)

unchecked.append("recall@5 — praxis states it in prose, with no eval run pinning it")
if counts is not None:
algorithms, environments, _ = counts
profile = fetch(PROFILE)
if profile is None:
unchecked.append("the organisation profile could not be read")
else:
lowered = profile.lower()
for value, label in ((algorithms, "algorithms"), (environments, "environments")):
word = WORDS.get(value)
if word is None:
unchecked.append(f"no word for {value} {label} in the profile check")
continue
# Read the word actually in front of the noun, rather than asking
# whether the right word appears anywhere: "thirty-two algorithms
# and twenty-four environments" contains both, and a check that
# only looks for presence cannot tell which belongs to which.
stated = re.search(rf"([a-z]+(?:-[a-z]+)?) {label}\b", lowered)
if stated is None:
problems.append(f"organisation profile: no longer states a count of {label}")
elif stated.group(1) != word:
problems.append(
f"organisation profile: says {stated.group(1)} {label}, "
f"decisionrl ships {word} ({value})"
)

unchecked.append(
"praxis on real models — the offline column of its quality table is pinned by a "
"test; recall@5 0.92 and MRR 0.94 need a GPU CI does not have"
)

for problem in problems:
print(f" DRIFTED {problem}")
Expand Down
Loading