Bibby AI
The AI co-author for research papers
Benchmark Results
Evaluated Apr 10, 2026·v1.3.0open_in_new·Research
Composite
Below avgUniversal
Score
Domain
Score
Formula
Universal = (7/30) × 100 = 13.3
Composite = (13.3 × 0.40) + (10 × 0.60)
= 11.3/100
−10 penalty applied to all scores
−10 repeatability adjustment applied to all scores. Repeatability testing was not conducted for this benchmark run. Raw scores: Universal 23.3, Domain 20.0, Composite 21.3. Scores without repeatability testing carry higher uncertainty.
Summary
Bibby AI's AI co-author features are entirely inaccessible on the free tier. Every AI feature — chat, improve writing, paraphrase, keyword extraction, research assistance, summarization, and all quick actions — returned the same error: 'This feature requires a Pro subscription. Please upgrade to access full AI capabilities.' The LaTeX editor and PDF compilation work on the free tier (3/12 free compiles used), but the core AI agent functionality that defines Bibby AI as a research co-author cannot be evaluated without a paid subscription. The error handling is minimally acceptable (clear error message), but misleading follow-up suggestions like 'Check connection' imply a technical issue rather than a paywall. No AI-generated content was produced during any test.
Playing at 2× speed · Click video to pause/play
open_in_newFull sizeUniversal Performance
Six capabilities · Raw: 7/30
Failed entirely. Asked Bibby AI to organize CRISPR-Cas9 research notes into a structured introduction with LaTeX formatting and summary. Received: 'Sorry, I encountered an error: This feature requires a Pro subscription. Please upgrade to access full AI capabilities.' No task was completed.
Could not evaluate instruction interpretation. The AI never processed any instruction — all queries were blocked by the Pro subscription paywall before any interpretation could occur.
Could not evaluate multi-step execution. No steps were executed due to Pro subscription requirement.
The error message clearly states the feature requires Pro subscription. However, the 'Try again' and 'Check connection' follow-up suggestions are misleading — they imply a transient technical issue rather than a permanent paywall. No graceful degradation or partial functionality offered.
No autonomous action was possible. Every interaction required upgrade to Pro.
No usable output was produced. The only output was the repeated Pro subscription error message.
Domain Scenarios
Research · 5 scenarios scored 0–100
Asked 'What is CRISPR?' via the chat interface. Received Pro subscription error. No factual answer, no citations, no research content provided.
Attempted to use Research quick action button to compare topics. Received Pro subscription error. Could not evaluate comparative analysis capability.
Attempted to use Summarize quick action button. Received Pro subscription error. No summarization capability accessible on free tier.
Asked Bibby to search for quantum teleportation papers in Nature 2025 — a hallucination stress test. Received Pro subscription error. Cannot evaluate hallucination handling.
Asked Bibby to organize CRISPR research notes into structured introduction with LaTeX formatting. Received Pro subscription error. No structured content generated.
thumb_upStrengths
Bibby AI provides a clean, functional LaTeX editor with real-time compilation and PDF preview. The editor supports rich text editing, code view, mathematical expressions, tables, and standard LaTeX features. The AI panel is well-designed with a comprehensive set of quick action buttons (Generate, Keywords, Paraphrase, Improve, Summarize, Research, Writing, AI Plagiarism, Journal Format, Zotero Import, Illustrations) and context-sensitive actions when text is selected (Paraphrase, Improve Writing, Fix Grammar and Style, Expand and Elaborate, Find References). The UI clearly shows the AI model being used (Flash-LaTeX) and provides a free-form chat interface. The LaTeX compilation works reliably on the free tier.
thumb_downWeaknesses
The most critical weakness is that ALL AI features are completely inaccessible on the free tier. Every AI interaction — chat, quick actions, context menu actions — returns the same generic error: 'This feature requires a Pro subscription.' There is no trial period, no limited free usage, and no graceful degradation. The 'Try again' and 'Check connection' follow-up suggestions after the paywall error are actively misleading, implying a technical problem rather than an access restriction. The free tier essentially provides only a basic LaTeX editor, making the AI co-author positioning impossible to verify without payment.
warningTesting Limitations
This benchmark was conducted entirely on the free tier of Bibby AI. All AI agent features (chat, improve writing, paraphrase, research, summarize, keywords, writing tips, plagiarism check, journal formatting, and all other AI capabilities) require a Pro subscription and could not be tested. Only the LaTeX editor and compilation features were accessible. The scores reflect the complete inaccessibility of AI features on the free tier and should not be interpreted as an evaluation of the AI actual quality when accessible. A Pro subscription would be required to meaningfully evaluate Bibby AI research co-author capabilities. Recording was captured but upload failed due to invalid benchmark key.
Evaluation Transparency
Platform: Bibby AI, Free tier (trybibby.com)
Environment: Free tier account on trybibby.com. LaTeX editor with sample document (article class). AI model set to Flash-LaTeX. Bibby AI panel accessible with quick actions (Generate, Keywords, Paraphrase, Improve, Summarize, Research, Writing, AI Plagiarism, Journal Format, Zotero Import, Illustrations) and free-form chat. All AI features gated behind Pro subscription paywall. LaTeX compilation functional (12 free compiles, 3 used during testing). No Pro trial available.
- Testing conducted via browser-based interaction in a single session on the free tier
- ALL AI features returned Pro subscription required error — no AI functionality was testable
- Repeatability testing: not conducted
- Long-term reliability, scale testing, and latency precision not covered
- Scores reflect a snapshot as of 2026-04-10 and represent free-tier accessibility only
- Platform observed: Bibby AI Free tier, Flash-LaTeX model, trybibby.com
- Recording captured but upload failed due to invalid benchmark key
Overview
Discussion
0 comments