WISE Search Engine

Manual on MediaWiki Tools

WISE Image Search Engine Release status: stable
Implementation	Database
Description	A content‑based multimodal search engine that uses vision‑language and other models to find images, video and audio by natural language, by example, or by a set of exemplars.
Author(s)	Prasanna Sridhar, Horace Lee, David M. S. Pinto, Andrew Zisserman, Abhishek Dutta
License	Apache License v2.0
Download	GitLab repo README.md changelog
Example	https://wise.wmcloud.org/

Screenshot of the WISE Image Search Engine user interface.

WISE Image Search Engine (wise) is an open‑source, content‑based multimodal search engine that leverages recent advances in vision‑language and other neural models.

Unlike traditional metadata‑only search, WISE understands the content of media and allows users to search using flexible natural-language descriptions, a single example, or a set of examples that define a category. It works across images, video, and audio.

WISE is designed for large media collections such as Wikimedia Commons, where missing or incomplete metadata often makes relevant media impossible to retrieve with a conventional keyword search.

Try it out: https://wise.wmcloud.org/

Implementation

Diagram illustrating the OpenCLIP vector pipeline used to map text and images to a common feature space.

WISE integrates a range of multimodal retrieval capabilities into a single tool accessible to users without machine learning expertise. Its modular architecture is not tied to any single model, and supports a range of feature extractors (with additional feature extractors under active development). Currently supported search modes are:

• Vision‑language model - OpenCLIP (Radford et al., 2021; Ilharco et al., 2021) for natural-language and reverse-image search at the scene level.
• Object detection - OWLv2 for object-level search.
• Face recognition - ArcFace (via the InsightFace library) for face-based search of specific individuals.
• Audio - OpenCLAP for retrieval of acoustic events from text or an audio file.
• Speech - search over automatically transcribed speech.
• Metadata - filtering by user-provided metadata, which can be combined with the content-based queries above.

WISE supports a wide range of formats, including images (e.g., JPEG, PNG), audio (e.g., MP3) and video (e.g., WebM, MP4).

Once the features from the different media files are extracted, it employs a FAISS vector index. This index is used for fast search, with search results ranked by decreasing cosine similarity between the query vector and media vectors.

Supported query modes are:

• Text – the user provides a natural language description (e.g., “a cat angry at another cat”).
• Example media – the user supplies one example (e.g., an image); WISE finds similar media.
• Set of examples (classifier) – the user provides multiple positive examples of a category (e.g., “Gothic cathedrals”); WISE trains a linear binary classifier on the fly to retrieve media of that category.
• Combined – queries across different modalities can be combined (e.g., the object query "train" with the metadata query "Germany").

• 
  Natural language search.
• 
  Visual similarity search.
• 
  Multimodal search combining different inputs.

Demos

WISE has been demonstrated on Wikimedia Commons media:

• https://wise.wmcloud.org/ (running on Wikimedia's Cloud VPS) indexes around 55 million Wikimedia Commons images (a bit less than half of Commons) presented at Wiki Workshop 2023.
• https://meru.robots.ox.ac.uk/motd indexes Wikimedia Commons "media of the day" videos – roughly 5,000 videos, corresponding to about 5 million frames.

In addition to the demos with Wikimedia Commons media, WISE lists multiple public demos on other collections on their website.

The software can be adapted to any large media collection by changing the media source and re‑running the offline indexing steps. No metadata is required during content-based search: retrieval is based on the content of the media itself, though metadata filtering is available as an additional query mode.

Administration

The WISE software is self‑hosted. To deploy it on a new media collection:

1. Obtain the media (WISE does not prescribe how this is done; you can acquire the files in whatever way suits your collection).
2. Run the offline feature extraction (a GPU speeds this up, but a CPU is possible).
3. Build the FAISS index (supports both flat and compressed indexes for memory/accuracy trade‑offs).
4. Start the query server (a lightweight FastAPI service).

System requirements depend heavily on the number of media items and the model(s) used; the compressed FAISS index keeps memory requirements modest for large collections.

Source

The source code of WISE is hosted on GitLab and released under the Apache License 2.0. The repository includes:

• Feature extraction scripts for the supported models.
• FAISS indexing and querying modules.
• A web demo frontend (optional).
• Examples for Wikimedia Commons and custom collections.

Bugs and feature requests should be filed in the GitLab issue tracker.

See also

• Sridhar, Prasanna; Lee, Horace; Pinto, David M. S.; Zisserman, Andrew; Dutta, Abhishek (2026). "WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata". To appear in Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. arXiv:2602.12819
• Visual Geometry Group (VGG) – the research group developing WISE
• Wikidata free image search