Eval, usage

[wolfgangihloff]

@cedricwider do you have any details on performance, quality and use? Which model did you use with it? Did you run some eval?

[cedricwider]

@wolfgangihloff Only subjectively, not any generic evals.

Because each PR can be so vastly different from one another, I did not run any evals at all. Instead I let other engineers test it on their reviews and report any findings. Subjective conclusion was: It's useful and does produce findings that could easily be overlooked by human reviewers.

[wolfgangihloff]

Any repo you can point me to, where I can see its output?

[cedricwider]

Sorry, only private repos for clients whose source we mustn't share.

I often use it locally before submitting a PR and instruct it to report findings back in chat instead of commenting on any PR.

You also asked for the model: Best results I had with Sonnet 4.6 for the lenses and Opus for the orchestrator. I know that others have used the setting with gpt-based models. I can't tell you about their exact setup, though.