Repository navigation
feat(content-scraper): optional johnvc/linkedin-posts-api source for LinkedIn profiles - #565
Open
johnisanerd wants to merge 1 commit into
Open
johnisanerd wants to merge 1 commit into
johnisanerd wants to merge 1 commit into
Conversation
Add linkedinActor setting (harvestapi default, johnvc opt-in) and map johnvc/linkedin-posts-api rows into the existing LinkedInRawPost shape. Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional
linkedinActorsetting to content-scraper. The defaultstays
harvestapi~linkedin-profile-posts. Setting it tojohnvcfetchesthe same weekly profile posts from the
johnvc/linkedin-posts-apiApifyActor and maps them into the existing
LinkedInRawPostshape, so scoring,dedup and storage are unchanged.
Disclosure: I build and maintain the johnvc Actor (Alpha OSINT).
Why this Actor
content-scraper already pulls a small number of recent posts per tracked
profile each week. This Actor returns public profile posts with no LinkedIn
login, mapped into the shape scoring and storage already use. A signed-out
person profile shows about 3 to 8 recent posts, which covers the current
maxPosts = 5.Changes
server/types/env.ts: optionallinkedinActor("harvestapi" | "johnvc"),default "harvestapi".
server/lib/linkedin-scraper.ts: new fetch path calling the Actor withprofileUrls: [profileUrl]andmaxPostsPerProfile: maxPosts, plus amapper to
LinkedInRawPost(id, linkedinUrl, content, author name, URL andheadline, likes, comments, postedAt.date).
server/tools/scraping.tsandcli.ts: pass the setting through so ittakes effect.
LINKEDIN_ACTOR=johnvc.Differences to know
num_repostsis stored as 0.author_profile_imageis null.visitor (usually 3 to 8). That covers the current
maxPosts = 5.postType: "repost",not with a separate
repostedBy, so they are attributed to the profileowner.
Testing
bun test content-scraper/server/lib/linkedin-scraper.test.ts)tscerrors in@decocms/runtimenode_modules still present on a clean install; no new errors introduced in content-scraper sourcesMade with Cursor
Summary by cubic
Adds an optional
linkedinActorsetting to content-scraper that selects which Apify Actor fetches LinkedIn profile posts. The default staysharvestapi~linkedin-profile-posts; setting it tojohnvcuses thejohnvc/linkedin-posts-apiActor and maps results into the existingLinkedInRawPostshape, leaving scoring, dedup, and storage unchanged. Disclosure: I build and maintain the johnvc Actor.Notes
LINKEDIN_ACTOR=johnvcin the CLI orlinkedinActor: "johnvc"in MCP state.num_repostsis stored as 0.author_profile_imageis null.maxPosts = 5.postType: "repost"attributed to the profile owner rather than a separaterepostedBy.Written for commit 6c1d84f. Summary will update on new commits.