Get automatic copilot code reviews happening in our orgs

[BigLep]

Done Criteria

When a PR targets the default branch, I want a copilot code review to happen automatically, regardless of who the author was.
I would like this to be the case for FOC-related repos in FilOzone and filecoin-project

Why Important

This is a good objective way for the author to get feedback before taking human reviewer time.
In addition, I think most people aren't using the copilot agent for their development, this copiliot serves as a different set of eyes.

Notes

We have various rulesets about this already in place. I think this used to work, but something has changed on the GitHub side. Now copilot reviews only happen for those who have a copilot plan.

Example ruleset that is configured properly: https://github.com/FilOzone/dealbot/settings/rules/10467327

• But see this:

It also looks like open source developers can no longer apply for a free individual plan: https://github.blog/news-insights/company-news/changes-to-github-copilot-individual-plans/

[BigLep]

When I attempted to open a support ticket I got the following.

It looks like we need copilot business?

Enable Copilot code review for all PRs without individual licenses
Automatic Copilot reviews for all PRs require organization or enterprise Copilot policies to be enabled, not an individual Copilot Pro plan.

What you’re seeing is expected behavior with the current Copilot model.

Why your existing rulesets no longer trigger Copilot
Automatic Copilot code review for all pull requests (including PRs opened by authors without a Copilot license) is not covered by Copilot Pro. Copilot Pro only enables features for the individual subscriber.

For organization-wide automatic reviews, GitHub requires a Copilot Business or Copilot Enterprise plan, plus specific policies enabled by an org or enterprise owner.

How “Premium request paid usage” actually works
The Premium request paid usage setting you found is:

Only visible to organization or enterprise owners
Only available on Copilot Business or Copilot Enterprise plans
Not available to individual Copilot Pro subscribers
Once an organization is on one of those plans, two policies must be enabled:

Premium request paid usage – allows the org or enterprise to pay for Copilot code review usage
Allow members without a Copilot license to use Copilot code review in GitHub.com – enables automatic reviews for PRs from unlicensed contributors
GitHub documentation confirms this behavior in About GitHub Copilot code review: https://docs.github.com/en/enterprise-cloud@latest/copilot/concepts/agents/code-review (opens in a new tab)

What will work today
You have three supported options:

Option 1: Manual or licensed-only reviews (current state)
Automatic reviews only run when the PR author has a Copilot license, or
You manually request a Copilot review
This works with Copilot Pro, but does not scale org-wide.

Option 2: Upgrade the organization to Copilot Business or Enterprise
This is the only way to restore fully automatic reviews for all PRs targeting the default branch.

High-level steps (org owner required):

Enroll the organization in Copilot Business or Copilot Enterprise
Enable Premium request paid usage
Enable Allow members without a Copilot license to use Copilot code review in GitHub.com
Re-enable your existing rulesets with Automatically request Copilot code review
Once configured, Copilot will automatically review every PR targeting the default branch, regardless of the author’s license.

Option 3: Enterprise-managed setup (if applicable)
If FilOzone or filecoin-project are owned by a GitHub Enterprise account, the policies must be enabled at the enterprise level first. Organization-level admins can’t override this.

Documentation: https://docs.github.com/en/enterprise-cloud@latest/copilot/how-tos/administer/organizations/managing-policies-for-copilot-in-your-organization (opens in a new tab)

Bottom line
✅ Your rulesets are correct
✅ Your understanding of premium paid usage is correct
❌ Copilot Pro cannot enable this behavior
✅ You do need an organizational Copilot plan (Business or Enterprise)
If you want GitHub to confirm eligibility, pricing, or help enable this on your orgs, proceed with submitting this ticket so Support can review your organization setup directly.

[BigLep]

I submitted a sales request in case that helps:

[BigLep]

Nothing additional here, but these were the most useful docs pages:
https://docs.github.com/en/copilot/concepts/agents/code-review
https://docs.github.com/en/copilot/how-tos/copilot-on-github/set-up-copilot/configure-automatic-review

[BigLep]

There are two angle of attack I see such that we get copilot reviews happening on our PRs automatically:

1. Get sufficient AI credits/plan on our orgs (FilOzone/filecoin-project) so that copilot reviews happen automatically for orgs we care about. I have an outstanding thread to GitHub sales about "how I can pay for reviews for certain repos within the FilOzone or filecoin-project orgs". I'll report back if I hear anything. I didn't in my first ping in May.

2. We get it so at least all the FOC Maintainers have copilot PR so at least their PRs will have copilot run. Some of us have github copilot

• I got whitelisted into copilot pro for free 6+ months ago as an "open source maintainer". @rvagg tipped me to this.) I don't know if this is still accurate or working, but here are the directions for this: https://docs.github.com/en/copilot/how-tos/copilot-on-github/set-up-copilot/enable-copilot/set-up-for-teachers-and-os-maintainers#accessing-copilot-pro-for-free
  • Note: github has recently done a lot of changing around with github accounts and signups. See https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing , https://github.com/orgs/community/discussions/192948, and https://github.com/orgs/community/discussions/197089. For awhile there you couldn't even sign up any more even to pay.
• individuals could just sign up for copilot for $10 a month and expense it: https://github.com/features/copilot/plans . Per https://github.com/orgs/community/discussions/197089 , it looks like they're taking signups again.

[Kubuxu]

We could also look for an external LLM Review bot/solution. My bet is that someone is doing it better and cheaper (and with less hassle) than GitHub.

[BigLep]

We could also look for an external LLM Review bot/solution. My bet is that someone is doing it better and cheaper (and with less hassle) than GitHub.

Agreed this is an option. I have been pursuing the GitHub copilot options because:

1. It has the potential to be low hassle if all our developers need to do is sign up for free GitHub Copilot free account. Then copilot reviews just work.
2. I assume copilot has the ability to have better integration with github.
3. I assume most aren't using copilot during their development, so it gives an alternate agent review then what was seen during development (although behind the scenes it's just using OpenAI, Anthropic models)

But yes, point noted that we shouldn't ram a bunch of effort to make copilot happen. The larger goal is to have meaningful, low-friction, automated reviews.

[rvagg]

As long as GitHub is making this harder and expensiver I think we should explore alternatives. Claude could be worth a go again although I was not happy with the friction last time. https://www.coderabbit.ai/ is another one, I'm sure there are plenty more that are just wrapping Claude & GPT. Maybe the friction is inherent in having an app to install and Copilot has an edge here? I doubt it, but that and cost would be the main reason for persisting. Also I guess we don't want to spend too much time yak shaving on this but we could experiment with a couple of alternatives on some repos?