Skip to content

feat: use site rank in front and read - #5

Merged
voska merged 6 commits into
voska:masterfrom
shbernal:feat/site-rank
Nov 19, 2026
Merged

voska merged 6 commits into
voska:masterfrom
shbernal:feat/site-rank

Conversation

@shbernal

@shbernal shbernal commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Built on #4 for the Firebase client, so this diff shows that commit until it merges.

Algolia has no site ranking. Its front_page tag returns the right stories in a different order, and threads come back oldest first. When I compared against the live site, 5 of 30 stories in hn front were in the right position, and its #1 was really #19. hn read opened one large thread with a comment reading "oof". The skill calls that "top comments".

hn front takes the order from Firebase and the details from one Algolia request. Job posts have no story_<id> tag on Algolia, so anything it lacks comes from single Firebase lookups. Position now matches the site, 30 of 30. --feed reaches new, best, ask, show and job.

hn read orders comments by site rank, plus the replies of the comments it shows. Deeper replies stay oldest first, because ranking them needs a request per comment.

It also caps its output now, which changes the default. On that thread, plain hn read printed 141 KB, about 35k tokens for one call. The default is now 18.6 KB in about 1.3s:

  • 20 comments, tunable with -n
  • 3 replies each, tunable with --replies
  • 1 level of replies, tunable with --depth
  • 800 characters per comment, tunable with --max-chars

Each cut says what it dropped: [12 more replies], [546 more comments], [+116 chars]. Comments now show #ID, and hn read COMMENT_ID opens a collapsed thread. --expand still prints everything, and --json returns the whole tree in rank order.

If Firebase fails during read, it warns on stderr and prints oldest first.

The README example .comments | length never matched the JSON. It's now .children.

Checked with make test, golangci-lint, and live runs of all six feeds and a comment drill-down.

Network failures, rate limits and server errors now exit 8. Previously
`read` and `user` reported every failure as not found (5), and `search`
and `front` exited 1, so callers could not tell when to retry.

`hn user` moves to the Firebase API because Algolia answers 500 for
unknown users. An empty `hn front` exits 3, and `hn status --json` exits
8 when the API is down.
Algolia has no ranking: its front_page tag returns the right stories in
another order, and threads come back oldest first. Take the order from
the Firebase API and the details from Algolia, so `hn front` matches the
site and `hn read` opens with the comments readers actually see.

`hn front --feed` reaches the other lists. Job posts have no story tag on
Algolia, so anything it lacks comes from single Firebase lookups.

`hn read` now caps its output, because a large thread ran to 140 KB by
default. It reports what it cuts, and shows comment IDs so a collapsed
thread can be opened directly.
@voska

voska commented Oct 30, 2026

Copy link
Copy Markdown
Owner

The ranking bug is real. Algolia's front_page order and oldest-first threads both misrepresent the site, and 5 of 30 positions is bad. Order from Firebase plus details from one Algolia request is a good split, and the tests cover the ordering, the job backfill and dead items.

I'm on board with the new hn read default too. 141 KB for one thread defeats the point of an agent-friendly CLI, and every cut says what it dropped. One mismatch: --json only ranks the top-level comments and the direct replies of the first -n, so the rest of the tree is still oldest first. The help text says the whole tree is in rank order, so that line needs tightening.

This carries #4's commit, so it needs #4 first. I'll handle the CHANGELOG and README overlap across the four PRs, no need to rebase for that.

@voska

voska commented Oct 31, 2026

Copy link
Copy Markdown
Owner

One more fix to fold in with the help text. fillMissing reads found without holding mu, while the lookups it has already started write to it under the lock. In TestFeed, the lookup for 20 can write while the loop is still checking 40, and go test -race can report that whenever 20's lookup takes the lock before 40's does. This head passed CI, so it comes down to timing. Building the list of missing IDs before starting any goroutines fixes it.

@martiant

Copy link
Copy Markdown

I'd like this merged: agents reading threads through hn read currently get the oldest comments first, and anyone piping into head misses the discussion. I checked three threads; Algolia's /items children matched id-ascending exactly, never Firebase's ranked kids.

If it helps, I'm happy to take the two requested changes (the --json help text and building the missing-ID list before the goroutines in fillMissing) in a follow-up PR on top of this branch. Just say which you'd prefer.

@voska
voska merged commit d2f8743 into voska:master Nov 19, 2026
2 checks passed
@voska

voska commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Thanks for checking those threads. No follow-up PR needed: I brought this branch up to date with master, added both fixes on top of @shbernal's commits, and merged it. fillMissing now builds the missing-ID list before starting any goroutine, and the --json help says only the top-level comments and the direct replies of the first -n come back in rank order.

@voska

voska commented Nov 20, 2026

Copy link
Copy Markdown
Owner

This shipped in v0.4.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants