Skip to content

feat(content-scraper): optional johnvc/linkedin-posts-api source for LinkedIn profiles - #565

Open
johnisanerd wants to merge 1 commit into
decocms:mainfrom
johnisanerd:feat/content-scraper-johnvc-linkedin-posts
Open

johnisanerd wants to merge 1 commit into
decocms:mainfrom
johnisanerd:feat/content-scraper-johnvc-linkedin-posts

Conversation

@johnisanerd

@johnisanerd johnisanerd commented Oct 22, 2026 •

Copy link
Copy Markdown

Summary

Adds an optional linkedinActor setting to content-scraper. The default
stays harvestapi~linkedin-profile-posts. Setting it to johnvc fetches
the same weekly profile posts from the johnvc/linkedin-posts-api Apify
Actor and maps them into the existing LinkedInRawPost shape, so scoring,
dedup and storage are unchanged.

Disclosure: I build and maintain the johnvc Actor (Alpha OSINT).

Why this Actor

content-scraper already pulls a small number of recent posts per tracked
profile each week. This Actor returns public profile posts with no LinkedIn
login, mapped into the shape scoring and storage already use. A signed-out
person profile shows about 3 to 8 recent posts, which covers the current
maxPosts = 5.

Changes

  • server/types/env.ts: optional linkedinActor ("harvestapi" | "johnvc"),
    default "harvestapi".
  • server/lib/linkedin-scraper.ts: new fetch path calling the Actor with
    profileUrls: [profileUrl] and maxPostsPerProfile: maxPosts, plus a
    mapper to LinkedInRawPost (id, linkedinUrl, content, author name, URL and
    headline, likes, comments, postedAt.date).
  • server/tools/scraping.ts and cli.ts: pass the setting through so it
    takes effect.
  • README config note. CLI: LINKEDIN_ACTOR=johnvc.

Differences to know

  • Share counts are not available, so num_reposts is stored as 0.
  • No author avatar; author_profile_image is null.
  • A person profile returns the recent posts LinkedIn shows a signed-out
    visitor (usually 3 to 8). That covers the current maxPosts = 5.
  • Reposts on person profiles come back as posts with postType: "repost",
    not with a separate repostedBy, so they are attributed to the profile
    owner.

Testing

  • Offline mapper fixture test (bun test content-scraper/server/lib/linkedin-scraper.test.ts)
  • Pre-existing tsc errors in @decocms/runtime node_modules still present on a clean install; no new errors introduced in content-scraper sources
  • Live two-profile run not claimed in this PR

Made with Cursor


Summary by cubic

Adds an optional linkedinActor setting to content-scraper that selects which Apify Actor fetches LinkedIn profile posts. The default stays harvestapi~linkedin-profile-posts; setting it to johnvc uses the johnvc/linkedin-posts-api Actor and maps results into the existing LinkedInRawPost shape, leaving scoring, dedup, and storage unchanged. Disclosure: I build and maintain the johnvc Actor.

Notes

  • Configure via LINKEDIN_ACTOR=johnvc in the CLI or linkedinActor: "johnvc" in MCP state.
  • Share counts aren't available, so num_reposts is stored as 0.
  • Author avatars aren't returned, so author_profile_image is null.
  • Signed-out person profiles typically return 3 to 8 recent posts, which covers maxPosts = 5.
  • Reposts come back as postType: "repost" attributed to the profile owner rather than a separate repostedBy.

Written for commit 6c1d84f. Summary will update on new commits.

View guided diff

Add linkedinActor setting (harvestapi default, johnvc opt-in) and map
johnvc/linkedin-posts-api rows into the existing LinkedInRawPost shape.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant