evalship is a GitHub App that performs quality control for LLM evaluation suites by commenting on every pull request with an interpretation of eval results, distinguishing real regressions from flaky noise and flagging prompt or LLM-call changes that no eval tests. It runs 24+ quality checks against eval code and data, identifying flaws such as judges that always pass or scoring weights too small to ever block a bad reply. evalship reads results from popular frameworks including pytest, promptfoo, Langfuse, Braintrust, and LangSmith, requiring no SDK changes. It can also auto-optimize prompts and models on autopilot, and its check can block merges that introduce genuine regressions.