AI tool evaluation rubric — "do we need this?"
Context
A fast, consistent test for any "do we need <external AI tool>?" question, so the answer is grounded in Aspire's locked invariants instead of hype. Derived from four back-to-back evaluations on 2026-06-22 (see external-ai-tool-evaluations).
Detail
Run a candidate through five checks. Default to PASS (don't adopt) unless it clears redundancy AND every invariant.
- Redundant? Does it duplicate a layer we already own — the LLM Gateway, the Aspire Hub (MCP registry + vault), Knowledge OS, the 4 browser MCPs, or OpenClaw / AIOS orchestration? If yes, pass.
- Single-egress? Will its LLM traffic route through the Aspire LLM Gateway? A tool that runs its own LLM loop or isn't OpenAI-compatible creates a second egress, a strong negative (breaks aspire-llm-gateway-only-egress).
- Economics? Is its core value per-token cost reduction? Our marginal LLM cost is ~$0 (flat-rate Claude Max / ChatGPT Pro + on-prem Qwen), so per-token savings ≈ $0. Metered tools that bill per token fight this.
- Locked rules? Respects API-first (browser/computer-use only when no API exists), Hub-vault for secrets, and no direct vendor SDKs?
- Security? Official source (not a lookalike org)? Any impersonation signal (wrong org name, tiny footprint, claims affiliation)? Does it ship binaries to download and run? Untrusted binaries are a hard stop.
The real question is usually "build-vs-redundant", not "build-vs-buy"
Because the stack already centralizes compute (Gateway), secrets (Hub), and knowledge (KO), most external AI tools land inside a layer we own. The rare keeper offers a capability we genuinely lack and clears all five — and even then, integrate it behind the gateway, not beside it.
Actions
- Apply to every "do we need X?" — record the outcome in external-ai-tool-evaluations.
- Genuine gaps (clears all 5): trial behind the gateway on a non-critical workload, measure, then decide.
Provenance
Authored 2026-06-22 from the patterns across four tool evaluations + the locked architecture decisions.
🔗 Relationships
graph LR
ai_tool_evaluation_rubric["ai-tool-evaluation-rubric"]:::self
ai_tool_evaluation_rubric --> external_ai_tool_evaluations["external-ai-tool-evaluations"]
ai_tool_evaluation_rubric --> aspire_llm_gateway_only_egress["aspire-llm-gateway-only-egress"]
classDef self fill:#715EE3,color:#fff,stroke:#291F50;