Structured Data Across Wikimedia/Image Suggestions/Data Pipeline
 | This page in a nutshell: The image suggestions data pipeline is a wide effort that spans several teams: we describe here the Structured Data side of the road. |
General Architecture
Who Does What
| Team | Pipeline components |
|---|---|
| Data Engineering | Data Lake |
| Data Platform | pipeline infrastructure, Cassandra |
| Platform Operations and Dumps | Image Suggestions API |
| Search Platform | Search indices update |
| Structured Data | Airflow job |
The Structured Data Side
Our main duties are:
- extract relevant content from Commons, Wikidata and Wikipedias;
- transform it into datasets suitable for Commons and Wikipedias search indices;
- load image suggestions into the database that will serve the API.
Speaking with the above architecture diagram terms, we:
- own the Airflow job;
- feed specific Hive tables in the Data Lake;
- provide content for Data Persistence, namely input for Elasticsearch updates and Cassandra.
The Airflow job breaks down into the following steps:
- a set of sensors that give green lights as soon as fresh data is available in the Lake;
- a Python Spark task that gathers Elasticsearch weighted tags for the Commons search index;
- a Python Spark task that gathers actual image suggestions, filtering out those rejected by the user community;
- a Python Spark task that gathers suggestion flags for Wikipedias search indices;
- a set of Scala Spark tasks that feed Cassandra with suggestions.
How it Works
Get Good Suggestions from Commons
First, we aim at retrieving relevant image candidates for unillustrated Wikipedia articles. To achieve so, we build sets of (property, value, score) triples that serve as tags to weight images in the Commons search index. We leverage three properties:
- two from Wikidata, namely image (P18) and Commons category (P373);
- one from Wikipedias, namely article lead images.
For each property, we retrieve the corresponding Wikidata item and compute a confidence score.
Scores Computation
When available, we consider image (P18) as a crucial property, thus setting a constant maximum score of 1,000 to its values.
For Commons category (P373), we implement the following simple intuition: a category with few members is more important than one with many members. As a result, given a Wikidata category item, its score is inversely proportional to the logarithm of the total images it holds.
For article lead images, the score is based on the number of main namespace pages that link to articles with the given lead image, grouped by the Wikidata item of the article.
For instance, let Commons_File_A be a lead image of xxwiki/Article_X and yywiki/Article_Y, which map to the Wikidata item Q123.
The Q123 score is proportional to the sum of incoming links for Article_X and Article_Y.
Based on empirical evidence, we pick a scaling factor of 0.2 and a threshold of 5,000 for incoming links.
Hence, if the sum of incoming links is:
- less than the threshold, then we set the score to incoming links * scaling factor;
- equal to or more than the threshold, then we set the score to its maximum value, i.e., 1,000.
Find Suggestions for Wikipedia Articles
The second major step involves mining images that serve as suitable candidates for unillustrated Wikipedia articles. We consider an article to be unillustrated if:
- it has no images at all;
- its images are used so widely across Wiki projects that they are likely to be icons or placeholders;
- the corresponding Wikidata item is an instance of a list, a year, a number, or a name.
Therefore, we implement the following set of heuristics to filter irrelevant candidates:
- the file name contains substrings that indicate icons or placeholders, such as
flagorreplace_this_image; - the image is in the placeholder category.