Bridging PubMed and AI Conversations
Medical professionals, researchers, and students spend hours sifting through PubMed for relevant literature. What if your AI assistant could do that for you — search PubMed, extract the important details, store everything in a growing knowledge base, and never fetch the same article twice?
That’s exactly what the PubMed Deep Research Tool does. Built as a native OpenWebUI tool, it integrates directly with PubMed’s E-utilities API and OpenWebUI’s knowledge base system to deliver intelligent, deduplicated medical literature research right inside your chat conversations.
What It Does
At its core, the PubMed Deep Research Tool turns a simple natural-language query into a structured research workflow:
- Searches PubMed for articles matching your query
- Deduplicates against your existing knowledge base so you never store the same article twice
- Processes articles with NLP — extracting medical entities, keywords, and structured metadata using spaCy
- Archives each article individually into your OpenWebUI knowledge base for future RAG retrieval
- Returns concise results to the LLM with truncated abstracts, keeping token usage low
The result is a knowledge base that grows over time as you research different topics, and OpenWebUI’s RAG system automatically surfaces relevant prior research in future conversations.
Key Features
Intelligent Deduplication
One of the biggest pain points with research tools is getting the same results over and over. The PubMed Deep Research Tool solves this with dual-layer deduplication:
- File metadata scanning — Each archived article stores its PMID in file metadata, enabling fast and reliable duplicate detection
- RAG text search fallback — As a secondary check, the tool scans existing knowledge base text for PMIDs
When you search for “diabetes treatment” today and again next week, you’ll only get articles published since your last search — no duplicates, no wasted tokens.
Smart Query Optimization
Not every query hits on the first try. The tool includes an automatic query variation engine that kicks in when no new results are found:
- Spell check — Uses PubMed’s built-in spell checker to fix typos (e.g., “diabeetes” → “diabetes”)
- Query broadening — Strips restrictive terms like date ranges, field tags, and Boolean operators
- Entity extraction — Falls back to searching with just the key medical and scientific terms
Each variation attempt is shown transparently in the chat, so you always know what’s happening behind the scenes.
Per-Article Knowledge Base Storage
Every article is stored as an individual file in your knowledge base with a structured filename (PMID_{id}_{title}.txt) and rich metadata. This approach enables:
- Precise deduplication by PMID across searches
- Clean knowledge base management — browse, delete, or review individual articles
- Resilient archival — if one article fails to upload, the rest still succeed
NLP-Powered Entity Extraction
Each article is processed through spaCy’s NLP pipeline to extract:
- Diseases and medical conditions
- Chemicals and drug names
- Genes and biological entities
- Key medical terminology and MeSH-aligned keywords
These extracted entities enrich the stored articles, improving RAG retrieval accuracy for follow-up questions.
Token-Efficient Output
The tool is designed to minimize token usage without sacrificing information:
- Abstracts are truncated to 1,500 characters in LLM output, with full text stored in the knowledge base
- No context duplication — previously stored articles are NOT echoed back to the LLM; OpenWebUI’s RAG injects them automatically
- Compact archive summaries — new articles are listed as brief PMID + title entries
- Reranker support — optionally limit LLM output to the top-N most relevant articles while still archiving everything
Hybrid Search Integration
For users with hybrid search enabled in OpenWebUI, the tool supports:
- Combined semantic + keyword search against your knowledge base
- Configurable BM25 weighting to balance between meaning-based and exact-term matching
- Enriched hybrid search that adds document metadata to improve keyword matching accuracy
How It Works in Practice
First Research Session
Ask your AI assistant something like:
“Search PubMed for recent studies on SGLT2 inhibitors in heart failure”
The tool will:
- Search PubMed and find matching articles
- Process each article with NLP
- Store them individually in your knowledge base
- Return structured results with abstracts, authors, DOIs, keywords, and extracted entities
Follow-Up Sessions
A week later, ask the same question:
“Any new research on SGLT2 inhibitors in heart failure?”
This time the tool will:
- Fetch articles from PubMed using a fetch multiplier (e.g., 2.5×) to cast a wider net
- Filter out articles already in your knowledge base
- Only process and store genuinely new articles
- Show you what’s new while OpenWebUI’s RAG automatically provides context from your existing articles
Handling Typos and Difficult Queries
Even imperfect queries work:
“diabeetes complications treatment options”
The tool recognizes the typo, applies PubMed’s spell correction, and searches for “diabetes complications treatment options” — all automatically.
Configuration at a Glance
The tool is fully configurable through OpenWebUI’s valve system. Here are the most impactful settings:
| Setting | Default | What It Controls |
|---|---|---|
| max_results | 10 | Maximum NEW articles to retrieve per search |
| fetch_multiplier | 2.5 | How many extra articles to fetch to account for duplicates |
| enable_query_variation | true | Auto-retry with corrected/broadened queries |
| max_query_attempts | 3 | How many query variations to try |
| reranker_results | 0 | Limit LLM output to top-N after reranking (0 = all) |
| relevance_threshold | 0.0 | Minimum relevance score to include results |
| enable_hybrid_search | false | Enable semantic + keyword hybrid search |
Recommended Configurations
Quick research update:
Set max_results: 5, fetch_multiplier: 2.0, and max_query_attempts: 2 for fast, focused updates.
Deep comprehensive research:
Set max_results: 50, fetch_multiplier: 3.0, max_query_attempts: 5, and enable hybrid search for thorough literature reviews.
High-precision filtering:
Set reranker_results: 5 and relevance_threshold: 0.7 to surface only the most relevant articles while still archiving everything.
Built for Physicians and Researchers
The tool ships with an optional physician-focused system prompt that transforms your AI assistant into an evidence-based clinical research companion. When configured, the assistant will:
- Prioritize high-quality evidence (systematic reviews, meta-analyses, RCTs)
- Cite PMIDs for every study mentioned
- Report effect sizes, confidence intervals, NNT/NNH, and study design details
- Flag study limitations and separate statistical from clinical significance
- Reference conflict of interest statements when clinically relevant
- Structure responses with direct answers, key evidence, clinical takeaways, and limitations
This makes the tool particularly valuable for clinical decision support, literature reviews, journal clubs, and continuing medical education.
Technical Details
The PubMed Deep Research Tool is built with:
- PubMed E-utilities API for article search and retrieval
- spaCy (
en_core_web_sm) for NLP entity extraction - NLTK for tokenization and stopword handling
- pandas for structured data processing
- OpenWebUI’s native APIs for knowledge base management, file upload, and RAG retrieval
It requires no external databases or services beyond PubMed — everything is stored directly in OpenWebUI’s knowledge base system.
Requirements
pandas
spacy
nltkAn optional NCBI API key increases rate limits from 3 to 10 requests per second for heavier research workloads.
Get Started
- Install the tool in your OpenWebUI instance
- Configure the
default_knowledge_basevalve (or leave the default) - Start asking research questions — the knowledge base is created automatically on first use
- Watch your medical literature library grow with every search
The PubMed Deep Research Tool is open source and available on GitHub. Contributions, feedback, and feature requests are welcome.
If your team wants PubMed-backed research workflows inside a governed private workspace, see HIPAA-aware private AI for healthcare admin & research.
What’s Next
We’re continuing to improve the tool with planned enhancements including expanded journal metadata, citation network analysis, and tighter integration with OpenWebUI’s evolving RAG capabilities. Stay tuned for updates.