Readers/Information Retrieval/Phase 0/ja

Overview

Hybrid Search Phase 0

Hybrid Search Phase 1

Hybrid Search Phase 2

Related Ideas

Phase 0: Idea Validation

The Foundation is conducting an early stage exploration to better understand why readers find it easier to locate Wikipedia information elsewhere than on Wikipedia itself, and how to improve information discovery while staying true to community-driven, verifiable knowledge.

At this stage, the focus is on identifying whether the problem is real, clarifying assumptions through data and community perspectives, and understanding our technical capabilities and limitations.

In particular, this exploration aims to answer:

• Why do readers find it easier to locate Wikipedia information elsewhere than on Wikipedia itself?
• What expectations do today’s readers have when searching for information?
• How might answer-seeking be improved within Wikipedia while preserving verifiability and community norms?

目的

• Determine whether and to what extent readers experience real friction with finding information on Wikipedia.
• Evaluate community perspectives and determine concepts worth testing in production.
• Determine timely but impactful methods to better meet reader needs.
• Verify feasibility of lightweight prototypes that align with Wikimedia’s infrastructure and privacy standards.

Possible Hypotheses to Evaluate

• Readers want to type queries and receive results that go beyond keyword matching.
• Surfacing excerpts and section-level snippets may be more satisfying than linking only to full articles.
• Presenting article-derived questions may improve engagement by guiding readers toward relevant sections and encourage further exploration of related topics more intuitively.

Actions Being Taken

As of October 2025, we are currently taking the following steps towards understanding and sharing insights in this problems space:

• Reviewing prior reader research on frustrations with Wikipedia’s search experience.
• Analyzing how often natural-language questions appear in search logs.
• Comparing how semantic and keyword results are perceived through conceptual mockups.
• Collecting qualitative feedback from new readers of displaying excerpts vs. full article links when returning results to queries through conceptual mockups.
• Collecting qualitative feedback on information discoverability
• Investigating infrastructure feasibility for modifications to search retrieval
• Spinning up internal small-scale prototypes to understand a variety of technical approaches and constraints
  • Note: These prototypes are exploratory only and not in a state to be put in production in front of real users.
• Collaborate with the community to refine problem space

Purpose of This Phase

The concepts and hypotheses shared here are meant to spark discussion and inform future decisions. The goal of Phase 0 is to collaboratively identify and test opportunities that could strengthen Wikipedia’s role as both a destination and a discovery space for reliable information.

Report: Improving How Readers Find Information on Wikipedia

Synthesis of Findings

Overall Signal

The following synthesis summarizes key learnings around Wikipedia’s search system across disciplines. Based on this research, evidence supports that Wikipedia’s current search system underperforms against modern reader expectations when using semantic-style queries and natural language queries.

In this report, we define semantic retrieval as retrieving information based on the meaning, concepts, or contextual relationships of a search, and lexical retrieval as retrieving information based on exact words, such as keyword matching.

Semantic-assisted retrieval that is anchored in human-authored content can improve reader satisfaction and task success. While each team approached the problem through a different lens, the collective findings point to clear reader value, community openness, and technical feasibility of a small scale experiment with a hybrid search (ie, a mix of both semantic and lexical search) minimum viable product (MVP), provided that implementation remains transparent, human-centered, iterative, and design-led.

Depending on the results of the MVP and outcomes from community discussions, we will determine the advisability of building a permanent feature. If we decide to proceed, we will work in collaboration with the community to determine what such a feature should look like.

Key Findings from Across Research Projects

Note that preliminary research was conducted in English.

Theme	Evidence Across Disciplines
Readers leave when discovery feels inefficient.	Literature review and design studies confirm most sessions are single-page visits; users habitually re-search externally.
Hybrid retrieval offers the best balance of user experience and coverage.	Data proof of concepts and search investigations show that semantic retrieval is not sufficient as a standalone experience; lexical retrieval is still needed (and can be further optimized) for certain types of queries.
UX improvements are as critical as model improvements.	Design proof of concepts show that how results are presented (anchored snippets, clear provenance) remains most important, despite what results are surfaced.
Community is cautiously optimistic.	WikiCon North America and ENWP Discord discussions reveal interest in search modernization, especially framed as combating declining readership and prioritizing discovery of human generated content. Discussions with a wider and more representative group of volunteers is needed.
Trust depends on transparency, not disclaimers.	Both Q&A and semantic user interview results showed users were confused when the feature attempted to signal AI, either by a disclosure label or icon.
Evaluation gaps remain.	All teams note the absence of a unified, multilingual benchmark or detailed success metrics for a potential final product.

Strategic Takeaway

Phase 0 establishes that internal search must evolve from keyword retrieval alone to also supporting intent-aware discovery; ie, a search model that can understand the deeper meaning of a query to direct readers to the answers they’re looking for.

We also want to use the opportunity to explore both the benefits and risks of different search models, including those that leverage LLMs.

The fastest way to understand how to evolve on-wiki search would be a small-scale experiment with an MVP to validate user experience hypotheses, collect benchmark data, and shape the infrastructure decisions required for sustainable, trustworthy hybrid search across the Wikimedia ecosystem.

Research Breakdown by Discipline

Each subsection summarizes methods, key findings, and representative evidence.

Research Team Literature Review

Research conducted by Martin Gerlach, Senior Research Scientist on the Research Team .

Hypothesis: If we review existing research on how readers interact with search and navigation tools on Wikipedia, and how they use external search to find knowledge on Wikipedia, we will be able to provide the Reader teams with ≥3 actionable recommendations and findings that help them scope a search and discovery MVP to address gaps in reader expectations and needs.

Status: Supported.

Methods & Inputs:

• Synthesis of Wikimedia and external studies on: internal search usage & queries, external search referrals, semantic search experiments, and known UX barriers.
• Analysis of full‑text vs autocomplete behavior, zero‑results and abandonment, multilingual and non‑main‑namespace use cases, and Commons media search.

知見:

• Internal search is under‑used and editor‑skewed.
  • Only ~0.8 – 2% of pageviews originate from internal search; 4 – 12% of on‑site actors use search, with editors over‑represented among search users. Many sessions are single‑page visits.
• Autocomplete dominates behavior.
  • ~80 – 95% of search sessions rely on autocomplete; only 5 – 8% are full‑text only. Satisfaction and CTR (clickthrough rate) are far higher for autocomplete than for full‑text.
• Natural‑language queries are uncommon.
  • Estimated at ~4 – 7% of full‑text queries on‑wiki; external referrals containing explicit question words appear to be a very small share as well. Some users expect longer, natural phrasing to work.
• Users face discoverability & comprehension barriers.
  • Many readers don’t realize Special:Search performs full‑text; absence of autocomplete suggestions is misread as lack of coverage, causing premature exits. Full‑text entry points and affordances are buried/unclear.
• External search is habitual—and used for navigation.
  • Readers frequently bounce to external engines (esp. Google) to move between Wikipedia pages, even when blue links exist on‑wiki. Convenience and habit play major roles.
• Commons and non‑main namespaces are pain points.
  • Commons search precision is low (esp. non‑English); editors request better search for templates, help/policy, and discussion threads.
• No single retrieval method is globally superior.
  • Lexical and semantic models both have strengths; what matters is matching the method to the query and user intent. Hybrid models that use lexical recall with semantic re‑ranking are likely to deliver the best balance of relevance and efficiency.
• Evaluation gaps exist.
  • We lack a trusted, representative benchmark of queries/passages to compare models and quantify relevance improvements.

Near-Term Recommendations:

• R1 – Build a benchmark & eval loop.
  • Create a representative, multilingual query set with annotated relevant passages; cover reader and editor use cases. Establish ongoing offline/online evaluation (A/B, interleaving) to compare lexical, hybrid, and semantic variants. T406207
• R2 – Improve search UI & affordances now.
  • Clarify the relationship between autocomplete vs full‑text; avoid the “no suggestion = no coverage” trap; add clearer entry points to section‑level and cross‑namespace results.
• R3 – Apply hybrid retrieval.
  • Use semantic components selectively – e.g., for re‑ranking top lexical results, within‑article passage retrieval, or domains lacking text metadata (e.g., Commons via vision‑language models). This balances capability and cost without over‑indexing on infrastructure concerns at this stage.

Medium-Term Recommendations:

• R4 – Enhance queries within the current stack.
  • Use query reformulation/enhancement (entity detection, alias expansion, spelling/lemmatization) to improve long/natural phrasing without full semantic infra.
• R5 – Strengthen multilingual discovery.
  • Improve cross‑language surfacing via Wikidata labels/aliases; consider pairing with MinT for readers (opt‑in), especially for monolingual audiences.

Design Concepts

Research conducted by Tais Lessa, Lead UX Designer on the Apps Team , and Justin Scherer, Lead UX Designer on the Reader Growth team.

Hypothesis: If we show high-fidelity design concepts for content discovery through semantic search to 10–20 casual Wikipedia readers in a qualitative study, we will see positive sentiment for the feature and gain the confidence needed to proceed with a search and discovery MVP that relies on short-form human-written excerpts to search queries.

Status: Supported.

目的: Two complementary concept explorations were conducted to understand how Wikipedia might evolve its on-platform information finding experience: one focused on user-initiated semantic search (helping readers find answers more efficiently through on-wiki search) and one on question-and-answer (Q&A) interfaces (helping readers recognize what a page can answer through lean-back methods of information discovery).

These were parallel explorations, not a comparative test. Each concept informs a different layer of the reader's journey and will likely be pursued in sequence rather than in competition.

Concept A – Semantic Search

Hypothesis: Surfacing section-level and anchored snippets will be more satisfying than full-article links.

Methodology: Four mobile studies (n = 20; mix of casual and frequent readers) compared the current keyword-based experience to two prototypes (please view on mobile browser).

• Concept 1: full-question queries, stacked suggestions, immediate results. Github link to prototype here.

Gif of the semantic search prototype, featuring full question, stacked search suggestions and in-article results.

• Concept 2: partial-sentence queries, segmented controls, animated results. Github link to prototype here.

知見:

• Main hypothesis validated.
  • 17 of 20 (85%) testers preferred section-level excerpts over full-page links.
• Expectation gap with current search.
  • Current experience is precise but slow for fact‑finding; readers expect quick, direct answers with optional depth.
• Concept 1 outperformed Concept 2 for clarity, speed, and trust.
  • Readers appreciated direct anchoring and highlight in-article; 11 of 20 praised this affordance.
• Sense-making requires clarity, not just ranking.
  • Users skim snippets to orient themselves.
• Some participants misattributed snippets as AI-generated, underscoring the need for transparent provenance labeling, but many participants reported that they would welcome more AI tools integrated into Wikipedia.

Concept B — Q&A Interfaces

Hypothesis: Adding “Answers in this article” modules and extended page previews will help readers discover specific facts faster, while clarifying what a page can answer.

Methodology: Three unmoderated studies (n = 30; “inactive/former” readers) tested:

• Accordions (embedded questions + answers)
• Deep links (jump to anchored sections)
• Extended page previews (larger hover cards)

One example of a potential accordion style question and answer design.

知見:

• Overall positive reception.
  • All concepts increased perceived content efficiency (“100% helpful to anyone who doesn’t want to read the whole article”).
• Accordions + deep links made Wikipedia feel more active and conversational (“hone in,” “quick,” “snappy”).
  • Accordions risk becoming perceived cul‑de‑sacs without explicit “View in article” affordances.
• Disclosures confused users.
  • The robot icon and AI label failed to clarify what was machine- vs human-generated and instead increased confusion.
• Question relevance is the critical dependency.
  • When questions matched users’ needs, satisfaction was high; otherwise, usefulness decreased.
• Extended page previews reduced tab overload but risked distraction when many are present.
• Although the questions were AI-generated, participants often assumed questions were crowdsourced or editor-curated.
  • Participants correctly understood that the answers were not AI-generated.
• “Casual” readers in this cohort tended to identify as former readers who now rely on GenAI tools, suggesting Q&A could re-engage this group.

Combined Takeaways:

• User needs:
  • Semantic Search – Faster, clearer answers within search flow
  • Q&A Interfaces – Contextual cues for “what this page can answer”
• Primary audiences:
  • Semantic Search: Active readers / searchers
  • Q&A Interface: Lapsed or casual readers
• Core interaction:
  • Semantic Search: Query → ranked excerpts
  • Q&A: Interface: Browsing → surfaced questions
• Dependency:
  • Semantic Search: Reliable retrieval (hybrid model)
  • Q&A Interface: Reliable question generation
• AI transparency risk
  • Semantic Search: Users may misread human-generated snippets as AI-generated
  • Q&A Interface: Users misinterpret disclosure icons and believe human-generated content is AI
• Build Feasibility
  • Semantic Search: Lower-risk near-term
  • Q&A Interface: Requires model maturity + design guardrails

Community Feedback

Research conducted by Eliza Blackorby, Movement Communications Specialist on the Reader Growth and Reader Experience teams.

Early on-wiki engagement:

A project page was created to outline background, reasoning, and early research findings for the initiative. This serves as a transparent space for editors and readers to follow progress, discuss open questions, and share perspectives on information discovery challenges.

Early in-person engagement:

At WikiCon North America (October 2025), we presented a lightning talk discussing:

• How current Wikipedia search functions and its gaps relative to modern expectations
• Early semantic search explorations
• Opportunities for community involvement

In a December community call with members of the English Wikipedia Discord, Jazmin Tanner and Tais Lessa presented some of the research detailed in this report, as well as early stage design ideas.

These sessions encouraged questions, feedback, and conversation on the future of on‑wiki discovery.

知見:

• Community members are aware of and concerned by declining on‑wiki readership.
• Editors expressed that many readers, including friends, family, and colleagues, now turn to AI chatbots for quick answers, raising concerns about Wikipedia’s visibility.
• Over and over, editors shared stories of frustration with Wikipedia content being used without attribution and search engines pulling from Wikipedia, so that readers no longer visit the article directly.
• Participants were energized by the idea of semantic search as a potential solution to re‑engage readers and potentially encourage new editors joining in the future.
• Community members understood that semantic search will use AI/ML tools to make search more modern and relevant.
• Community members expressed concern over creating incentives for editors to “optimize” articles for search visibility, as well as potential worries about generative AI tools.
• Community members see potential in Q&A to help redirect readers who arrive with non-encyclopedic queries, similar to how disambiguation pages redirect topic ambiguity.
• There’s a need to develop a Wikipedia-appropriate way to redirect readers who ask questions Wikipedia cannot answer, ie “let them down gently.”
• Article contributors could provide metadata or annotations that improve semantic search relevance.
• Other ideas included:
  • Use of an internal confidence score for how well a question maps to content, and only show answers above a defined threshold.
  • Exploration of in-article search powered by ML (searching within a page) rather than placing Q&A boxes at the top of articles, Google-style.
  • Allow for the ability to mix exact-match and semantic search in one query.

Takeaways:

• We will continue to share our thinking around semantic search early and often, both in conversations onwiki and at in-person community events.
• Communities globally will help us shape our learnings into a search feature that works well, matches reader needs, and feels true to Wikipedia.
• We will work with wiki communities globally to ensure that semantic search solutions on Wikipedia are multilingually and multiculturally appropriate and effective.
• Semantic search and related features will not be deployed without community involvement and consensus.

Machine Learning Data Prototype

Research was conducted by the 機械学習 Team with contributions from Fabian Kaelin, Research Engineer, and Santhosh Thottingal, Software Engineer, and led by Sucheta Salgaonkar, Lead AI Product Manager.

Hypothesis: If we produce an internal prototype for semantic search and in-article Q&A, delivered as a demo interface that contrasts the current approach with new exploratory approaches, then the Reader teams will be able to qualitatively evaluate how each approach performs across different user journeys and surface gaps or opportunities for further iteration.

Status: Supported.

方法論: The ML team explored multiple prototype implementations: embedding-based similarity search, question-to-question matching and retrieval, and LLM-generation and scoring for article questions.

Prototypes: These prototypes were built for the sake of speed and not based on the “best” approach:

• User Initiated Search Prototypes: https://semantic-search.wmcloud.org/ および https://ask.toolforge.org/
• Q&A Prototype: https://semantic-search-qa.toolforge.org/

Surfaced Considerations:

• Hybrid retrieval is essential.
  • Semantic retrieval alone underperforms for most lexical queries; blending lexical recall with semantic reranking is required to deliver high satisfaction.
• Granularity tradeoffs
  • There’s a cost–precision balance between article-, section-, and paragraph-level embeddings. Further research into chunking and alternative architectures (e.g., semantic highlighting models) is needed to improve efficiency.
• Infrastructure readiness
  • A production-ready MVP requires scalable pipelines for embedding generation, updating, and storage. Current capacity supports limited-scale tests but not ongoing refresh cycles.
• Model selection matters.
  • Embedding models vary in performance; a wider range of models should be explored and OpenSearch-native and open-source options should be benchmarked. Multi-pass systems with re-rankers can mitigate dimensionality and performance issues.
• Bias and hallucination risks
  • For Q&A use cases, AI-generated questions can potentially introduce factual or representational bias. Such AI generation should occur offline with human validation before deployment.
• New success metrics needed.
  • Traditional clickthrough metrics don’t capture when users find answers without clicking. Introduce “good abandonment” and human evaluation to better measure search quality.

What Would Be Needed for Scaled Production:

To move forward from an internal prototype to a large scale production-ready state, the following components are required:

• Embedding infrastructure: Scalable pipeline for generating, refreshing, and storing text embeddings across multiple wikis.
• Hybrid search service: Better pathways and handling for queries of various types, ensuring that each query is served by the most appropriate retrieval method.
• Evaluation framework: Human-annotated benchmark dataset and A/B testing to measure relevance improvements.
• Monitoring and observability: Real-time telemetry for model service performance (latency, errors, etc.) and tooling for monitoring model prediction quality over time.
• Ethical and quality controls: Bias audit, content filtering, and correctness rubrics for any AI-generated outputs.
• Design Collaboration: Ensure that design dictates what data is needed in a given UX, and ML systems provide that data at an expected level of quality and technical performance.

Relaxing Keyword Matching in our Existing Search

Research conducted by Peter Fischer, Senior Software Engineer on the Data Platform Engineering team.

Hypothesis: If we relax the matching of keywords in Search, then we can better support natural language queries, and enable Product to evaluate this capability, including it in how they design, prioritize and deliver work in the Semantic Search space.

Status: Not supported.

Key Questions:

• Do we actually need to invest in new retrieval methods to provide the experience we have in mind?
• Is our current retrieval method flexible enough to support different types of queries?

方法論: Prototype work was conducted under phab:T405869, where engineers tested variations of query matching strictness within CirrusSearch/OpenSearch.

The approach involved:

• Relaxing keyword matching constraints to allow more flexible retrieval (e.g., partial term matching, stemming, lemmatization).
• Running comparative relevance evaluations on a representative set of natural-language-style queries.
• Measuring effects on recall and perceived relevance against baseline search and embedding-based retrieval samples produced by the ML team.

知見:

• Relaxing keyword matching alone is insufficient.
  • Looser lexical matching increases recall but not relevance; the quality of matches remains below those achievable with embeddings.
• Incremental value in preprocessing.
  • Entity extraction, alias expansion, and query reformulation can improve recall for natural-language-style queries at relatively low cost. These optimizations are worth pursuing, but they cannot fully replicate semantic relevance.
• Current system limitations.
  • CirrusSearch/OpenSearch can support modest lexical improvements and re-ranking extensions, but lacks the architecture to natively handle vector embeddings or meaning-based retrieval at scale.

Implications:

• Short-term:
  • Search engineering can enhance existing lexical retrieval to improve baseline quality and user experience, regardless of semantic implementation.
  • An MVP would have to rely on the hybrid retrieval approach recommended by the ML and Design teams. Existing search enhancements could feed into this pipeline but not replace it.
• Long-term:
  • As hybrid infrastructure matures, Search can integrate preprocessing improvements to reduce computational cost and expand multi-language support.
  • Search optimizations should complement, not substitute semantic or lexical capabilities ensuring Wikipedia can handle both traditional keyword and meaning-based queries within a unified experience remains our aim.

Design Research of Data Prototype

Research conducted by Mike Raish, Lead Design Researcher on the Design Research Team .

Hypothesis: If we evaluate two semantic search prototypes (natural language search and Q&A) with external participants, we can learn whether users see value in improved search tools and provide the Readers teams with a recommendation on how to move forward with a search and discovery MVP.

Status: Somewhat supported.

方法論: A randomized survey of 52 English-speaking Wikipedia readers (recruited via Userlytics) was conducted using Qualtrics. Participants rated the relevance of search results for 14 real-Wikipedia queries—10 without typos, 4 containing typos—comparing Semantic Search results to existing Wikipedia keyword search results.

Each participant viewed randomized result sets and reported:

• Which result set they found more relevant (Semantic, Wikipedia, Both, or Neither).
• Their familiarity with each topic, tech-savviness, and English reading proficiency.

Participants saw actual search result pages drawn from the Semantic Search data prototype (October 2025).

知見:

• Overall preference for Semantic Search.
  • Across 10 non-typo queries, participants preferred Semantic Search for 8, Wikipedia Search for 1, and rated both as irrelevant for 1.
• Handling typos
  • For 2 of 4 typo queries, participants preferred Semantic Search; for the remaining 2, neither set was considered relevant.
• High agreement with internal “eyeball test.”
  • Participant preferences matched the WMF team’s internal relevance assessments for 7/10 non-typo and all 4 typo queries.
• Variable performance by topic.
  • Semantic Search struggled with highly specific named entities (e.g., “General Grievous”) where Wikipedia’s keyword precision outperformed.
• Behavioral insight
  • Pilot testers (who were screen recorded) clicked through to Wikipedia articles before completing evaluations—suggesting that readers may perceive search primarily as a navigation and exploration tool, not an endpoint.
• Demographic factors
  • No significant differences by tech-savviness, education, or Wikipedia familiarity. Slight indication of a relationship between high English proficiency and preference for Semantic Search.

Recommendations Conduct a larger-scale study within a “Wikified” interface to inform recommendations for:

• Result text length (sentence vs paragraph)
• Query type (factual vs exploratory)
• Presentation order and labeling
• Evaluation metrics for meaningful reader outcomes
• Human rated relevance

Potential Next Steps

Hybrid Search Experiment

Purpose: Translate the validated insights from Phase 0 into a production experiment that functions as a scaled research environment.

Core Areas

• Hybrid Retrieval Implementation – Implement an initial, lightweight approach to utilizing both semantic and lexical retrieval methods based on the intent of the query.
• User Experience Validation – Test how excerpt length, snippet style, and provenance labeling affect satisfaction, click-through, and “good abandonment.”
• Search Service SLOs – Ensure that our MVP infrastructure allows us to monitor key SLOs like latency, cost, error rate, etc. under real traffic conditions.
• Community Collaboration – Partner with target wikis to pilot a scaled experiment wikis with open talk-page discussion, discord discussions and periodic newsletters.
• Evaluation Framework – Begin continuous offline/online benchmarking with human-rated query sets to guide model tuning.

Parallel Work

• Lexical Search Enhancements – Continue improving query preprocessing (e.g., entity extraction, lemmatization) and ranking logic in the current stack. These improvements will eventually be incorporated into the hybrid search approach, potentially after the delivery of the MVP.
• Evaluation Infrastructure – Build shared tooling for logging, annotation, and telemetry so that lexical and semantic searches can be measured the same way.
• ML Model Exploration – Test additional embedding and re-ranking models to improve efficiency and accuracy without delaying MVP rollout.
• Community & Comms – Maintain outreach, feedback loops, and documentation in parallel with technical development to inform short-term work and explore long-term possibilities or pivots.

Q&A Experiment

Purpose: Build on the infrastructure and behavioral insights to explore lean-back discovery experiences that help readers quickly understand what a page can answer.

Prospective Focus Areas:

• Explore editor collaboration models for question generation or approval.
• Continue UX iteration on deep-link affordances, disclosure clarity, and number of surfaced questions.
• Introduce Q&A modules only once question generation achieves sufficient precision and human validation.

Dependencies:

• Reliable evaluation dataset and metrics pipeline
• Proven latency of hybrid retrieval.
• Established community norms and feedback mechanisms.

Parallel Work Streams:

• Detailed product analytics using MVP telemetry to improve question generation and semantic matching.
• Design research cycles exploring curiosity-based navigation and user comprehension of lean-back features.

Questions for communities

We would like to collaborate with communities to understand:

• What are your overall reactions to the report? Is there something that especially stands out to you?
• Would you like us to approach your language community about the possibility of piloting small-scale experiments in the near-term future ?
• What signals/data would be useful to determine if a more refined hybrid search and/or Q&A modules would be worth making available for your language wiki post-experimentation?