Talk:Wikibase/Indexing/RDF Dump Format

Triple representation of labels

The RDF dumps have

Entity labels - the main name of the entity. Labels are defined as schema:name, rdfs:label and skos:prefLabel predicates with objects being language-tagged string literals.

Why say the same thing thrice, particularly as there are lots of labels for many items? Pfps (talk) 20:59, 21 July 2016 (UTC)

The idea was to be compatible with all 3 ontologies, so that things work if you support one of them. Jan Zerebecki 15:43, 25 July 2016 (UTC)

The problem is that this triple representation adds a *lot* of redundant information. Similarly for the multiple representation of values.

This would not be a problem if it was easy to ignore the "other" versions of the information one wants, but this dump has everything in one very large file. As the file is in Turtle format simple textual methods cannot be trusted to find and remove the pieces that are not neeeded. Pfps (talk) 19:00, 11 November 2016 (UTC)

Units representation and conversion

Planned units representation and conversion for full values:

1. There will be a configuration on wiki (not specified here) describing conversion of units to standard units.
2. Values in standard units will be expressed as "normalized" values (which later can be expanded to also represent other kinds of normalized values, e.g. for times, external IDs, etc.)
3. Each of  psv/pqv/prv gets a duplicate called psn/pqn/prn linking to a normalized value node in parallel to a regular value. Normalized value can be the same value (if the value is already in standard units) or a different one. If the value has no units or the units can not be converted to a standard unit, no normalized value is produced and no psn/pqn/prn predicate is generated.
4. Original value gets wikibase:quantityNormalized predicate linking it to the normalized value. This predicate is generated even if the value is already normalized - in this case, it links the value to itself. Normalized value also has wikibase:quantityNormalized linking to itself, since normalized value is its own normalization.

Unit representation & conversion for simple values is still TBD. Smalyshev (WMF) (talk) 17:43, 25 August 2016 (UTC)

Example:

wds:Q3-24bf3704-4c5d-083a-9b59-1881f82b6b37 a wikibase:Statement, wikibase:BestRank ;
        psv:P2 wdv:cb213eea7a0b90d1d7f65c6eabfab9da ;
        psn:P2 wdv:3efb2709bd74285cfc7e72b6a599125b ;

wdv:cb213eea7a0b90d1d7f65c6eabfab9da a wikibase:QuantityValue ;
    wikibase:quantityAmount "+123"^^xsd:decimal ;
    wikibase:quantityUpperBound "+124"^^xsd:decimal ;
    wikibase:quantityLowerBound "+122"^^xsd:decimal ;
    wikibase:quantityNormalized wdv:3efb2709bd74285cfc7e72b6a599125b ;
    wikibase:quantityUnit <http://www.wikimedia.org/entity/Q828224> .

wdv:3efb2709bd74285cfc7e72b6a599125b a wikibase:QuantityValue ;
    wikibase:quantityAmount "+123000"^^xsd:decimal ;
    wikibase:quantityUpperBound "+124000"^^xsd:decimal ;
    wikibase:quantityLowerBound "+122000"^^xsd:decimal ;
    wikibase:quantityNormalized wdv:3efb2709bd74285cfc7e72b6a599125b ;
    wikibase:quantityUnit <http://www.wikimedia.org/entity/Q11573>

. Smalyshev (WMF) (talk) 20:50, 25 August 2016 (UTC)

It appears that this is already underway, at least for geospatial information. Is this the case? Pfps (talk) 19:01, 11 November 2016 (UTC)

Yes, we are starting to introduce unit conversions. Smalyshev (WMF) (talk) 17:20, 16 November 2016 (UTC)

Time revision

My first question is how to update the this document. A "sister" document, Wikibase/DataModel/JSON says that the document should not be edited in the normal way, but rather "NOTE: The canonical copy of this document can be found in the Wikibase source code and should be edited there. Changes can be requested by filing a ticket on Phabricator"

Does a similar process apply to this document, or is this document edited directly?

The change I think should be made is as follows, with bold showing material to add:

The full value includes the simple value above under wikibase:timeValue, precision and timezone as integers and calendar model as IRI. The timezone parameter has never been implemented and should be ignored; all times in the database are local times; that is, the timezone is not recorded as part of the time and must be deduced from other clues, such as the place an event occurred.

I consider this important because editors may be unwilling to contribute to Wikidata if it forces them to make false statements, such as the time zone being UT when it is really United States Eastern Daylight Time.

I will be making a parallel request for revision of Wikibase/DataModel/JSON.

[Text above added September 2016. Text below added 31 December 2016.]

At d:Wikidata:Project chat/Archive/2016/12#Adding a source @Pasleim: stated

If a date is given with day precision, one has to ignore all information which make more precise claims including the time zone. The time zone parameter is only needed for dates/times with at least hour precision. So Wikidata doesn't say anything if a date is in UTC or in local time. Basically a specific day is a time period of 50 hours, from 12:00a.m. in UTC+14:00 (Q7130) to 11:59p.m. in UTC−12:00:

While this may have been a defensible interpretation of earlier versions of this document, and maybe even a defensible interpretation of the JSON dump spec, this spec says

The simple value of the time value is either datetime value of type xsd:dateTime, if the value can be converted to Gregorian date in ISO format, or a string as represented in the database, if not. The xsd:dateTime dates follow XSD 1.1 standard...

Considerable effort has recently been expended to respect the XSD standard to always use the Gregorian calendar, by creating code to convert Julian dates to Gregorian dates. If this effort has been put into respect the Gregorian calendar aspect of the XSD spec, I infer the meaning of the "Z" an the end of the representations, which means Universal Time, would be equally respected. Jc3s5h (talk) 15:58, 31 December 2016 (UTC)

No this one is maintained here :) Lydia Pintscher (WMDE) (talk) 12:20, 30 September 2016 (UTC)

In view if a Wikidata ambiguity about the date of Isaac Newton's death, I believe the section should also be revised to state the year is always deemed to begin on January 1, even though historically some countries have observed other dates to increment year numbers. Jc3s5h (talk) 14:19, 24 July 2017 (UTC)

wrong example for Duck?

A query for the schema:about value of <https://en.wikipedia.org/wiki/Duck> shows that it's wd:Q3736439, but the Sitelinks section of this page says that it's wd:Q3. Am I misunderstanding something or does this example need to be corrected? Bobdc (talk) 16:27, 14 February 2017 (UTC)

The example doesn't have correct values. It just has values to show how it's formatted. Examples usually have just random values as does this example. Mbch331 (talk) 16:58, 14 February 2017 (UTC)

Sitelinks with spaces

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

When storing sitelinks, shouldn't it be normalized URL ('_' instead of spaces)? Otherwise they differ from the canonical wiki representation. CC: @Smalyshev (WMF): Yurik (talk) 05:37, 26 May 2017 (UTC)

See https://phabricator.wikimedia.org/T131960 Smalyshev (WMF) (talk) 02:57, 10 June 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Disputed interpretation

In an edit at wikidata:Help:Dates Jarekt changed a statements to read "Wikibase software interprets years 1801-1900 with precision 7 as 19th century" and "Wikibase software interprets years 1001-2000 with precision 6 as second millenium". There is discussion on the associated talk page.

I believe Jarekt is referring to the interactive user interface, but I consider it wrong to refer to that interface as "Wikibase software". I believe the RDF API is just as much a part of Wikibase software as the interactive interface. The RDF data model documentation indicates RDF follows ISO 8601 and XSD 1.1 standard. Both of those indicate precisions by truncating the unneeded information. So, for example, for precision 100 years and a year of 1900, it would be truncated to 19 and understood to include any year from 1900 to and including 1999. Jc3s5h (talk) 22:30, 17 July 2018 (UTC)

RDF Dump Format deals with how data is stored not what it means. Current Wikidata standard of interpreting concept of 1st century as years 1-100, second century as years 101-200, etc. and 1st century BC as years 100 BC-1 BC, is perfectly consistent with international understanding of those terms for hundreds of years.See 1st century, etc. If you like to redefine those term than discussion on ''RDF Dump Format'' talk page is not the right place. Wikidata is not consistent with ISO 8601 and XSD 1.1 standards, which is unfortunate but as wikidata:Help:Dates mentioned Wikidata dates are "resembling ISO 8601" but do not follow it. Other difference is how we store BC dates and section d:Help:Dates#Years_BC explains conversions which are done to from the format used by Wikidata to RDF Dump format.

I do not understand your point about of Wikidata GUI not being part of "Wikibase software". I never tried this but I believe that if you create a different instance of wikibase using wikibase software than it comes with the GUI. So I am lost... Jarekt (talk) 15:59, 19 July 2018 (UTC)

When I wrote "I consider it wrong to refer to that interface as "Wikibase software" I meant that the interactive user interface is not the only Wikibase-provide method to read and write data, so that writing as if the meanings implicit in that interface were followed by all intefaces is incorrect. I completely reject the notion that "RDF Dump Format deals with how data is stored not what it means." Explaining what the RDF Dump Format means is the purpose of this page. Jc3s5h (talk) 11:01, 20 July 2018 (UTC)

a wikibase:Item doesn't work

According to the site this query should result true:

ASK { wd:Q2 a wikibase:Item }

But it results in false. It this a bug? Is there any other way to check is a entity is an item? MichaelSchoenitzer (talk) 23:58, 1 September 2019 (UTC)

Wikibase/Indexing/RDF Dump Format#WDQS data differences Matěj Suchánek (talk) 06:16, 2 September 2019 (UTC)

Thanks, I oversaw this section. Is there any other way to check if an entity is an item (not an property, lemma, etc.) in a query? MichaelSchoenitzer (talk) 09:06, 3 September 2019 (UTC)

It may be unrealiable but I would do ASK { wd:Q2 wikibase:sitelinks [] }. Matěj Suchánek (talk) 11:35, 3 September 2019 (UTC)

osmm osmt prefixes

What is prefixed by osmm: osmt: and other osm_: ?

These prefixes can be found everywhere in examples, but they are not listed here. 88.156.143.3 (talk) 17:07, 11 May 2020 (UTC)

They are the prefixes from the OpenStreetMap (OSM) Data Model which are necessary for querying the OSM SPARQL Endpoint at https://sophox.org

 "OSM Data":{
      "osmnode":"https://www.openstreetmap.org/node/",
      "osmway":"https://www.openstreetmap.org/way/",
      "osmrel":"https://www.openstreetmap.org/relation/",
      "osmt":"https://wiki.openstreetmap.org/wiki/Key:",
      "osmm":"https://www.openstreetmap.org/meta/",
      "pageviews":"https://dumps.wikimedia.org/other/pageviews/"
 }

All other osm* prefixes are the Sophox prefixes for its Wikibase instance, corresponding to the same concepts of Wikidata that start with 'w' , e.g.

 PREFIX osmd: <http://wiki.openstreetmap.org/entity/>

analog to

PREFIX wd: <http://www.wikidata.org/entity/>"

Same for all of the other prefixes like

PREFIX osmdt: <http://wiki.openstreetmap.org/prop/direct/>

etc. I hope this helps.

Cheers Frog23 Frog23 (talk) 11:14, 21 May 2020 (UTC)

Mapping Images from Wikidata to Commons with federated SPARQL

based on the existing RDF Dump Format model it isn't possible to match the Media-Info RDF-Dump based on a Wikidata-SPARQL-Query for images using ?item wdt:P18 ?img. The reason is, that the commons media file value stored in the Wikidata Wikibase like http://commons.wikimedia.org/wiki/Special:FilePath/%FileName% but in the media-info rdf-dump this values isn't represented, only the full qualified upload url, which is hard to guess without any other information. Mfchris84 (talk) 13:44, 23 July 2020 (UTC)

wrong Wikidata element in the example

Hallo,

the second example box is wrong - Q3 is "life", while the "Universe" is Q1.

I would edit it by myself, but i have no knowledge about the long identifiers starting with `wbs:` Giacomo Lanza (talk) 06:49, 23 September 2020 (UTC)

For en example the data doesn't have to be correct, just correctly formatted. Besides if we were to correct it, we would need more things to be fixed. It still contains a reference to P7, which property no longer exists. Mbch331 (talk) 07:33, 23 September 2020 (UTC)

exact meaning of wikibase:geoPrecision?

What does the value of wikibase:geoPrecision mean exactly?

A couple years ago I did some research and came up with this:

• "measured in 1/111000 meters (eg street address or building -> 30.8/111000)"

But I can't remember how I came to that conclusion. Is it correct?

cc @Smalyshev (WMF) Vladimir Alexiev (talk) 09:10, 17 September 2021 (UTC)

Does Wikibase/DataModel#Geographic_locations help? Lydia Pintscher (WMDE) (talk) 09:19, 17 September 2021 (UTC)

• Wikibase/DataModel#Geographic locations "representing degrees of distance"
• https://en.wikibooks.org/wiki/SPARQL/WIKIDATA_Precision,_Units_and_Coordinates#Coordinates "measured in degrees. Multiply by 111000 to convert to meters"
• Google "A degree of longitude is about 111 kilometers (69 miles) at its widest. The widest areas of longitude are near the Equator, where the Earth bulges out. Because of the Earth's curvature, the actual distance of a degree depends on its distance from the Equator."

Using degrees is not the best way to measure precision since it depends on location and direction. But it is what we got.

I'll add this to Wikibase/Indexing/RDF Dump Format Vladimir Alexiev (talk) 09:33, 17 September 2021 (UTC)

Is wds: now just s:?

The docs here say "Full statement is represented as separate node, with prefix wds:" however a TTL dump shows no "wds:" but only "s:" - So should this documentation be changed? Thanks. Fuzheado (talk) 09:13, 21 April 2022 (UTC)

In the query service it seems only the wds: is defined.

https://query.wikidata.org/#select%20%23%28str%28s%3A%29%20as%20%3Fs%29%20%0A%20%20%20%20%20%20%20%28str%28wds%3A%29%20as%20%3Fwds%29%20%0A%7B%7D%20

works fine, but decomenting the first line in the query leads to an error because it’s undefined :

select (str(s:) as ?s) 
       (str(wds:) as ?wds) 
{ }

Try it! TomT0m (talk) 10:35, 21 April 2022 (UTC)

V: in quantity example

The example for quantity contains the prefix v: which I don't see anywhere else, does anyone know what this is supposed to be? M.alten.tue (talk) 15:53, 25 March 2024 (UTC)

I think this is supposed to be "wdv" for <http://www.wikidata.org/value/>. See for example this query. I have updated the examples accordingly. Frog23 (talk) 21:52, 25 March 2024 (UTC)

wdtn prefix

I do not see the wdtn prefix used in the Wikidata Query Service. For instance, This gives zero results:

SELECT
  ?s ?o
WHERE {
  ?s wdtn:P5822 ?o .
} 
LIMIT 10

Try it!

Is there something that I am missing? Fnielsen (talk) 18:24, 4 April 2024 (UTC)

Looking which properties have a "DirectClaimNormalized" predicate in the query service : https://w.wiki/9fQA it seems only identifier properties have this normalization. TomT0m (talk) 18:52, 4 April 2024 (UTC)

Thanks, I see now.

SELECT 
  ?s ?o
WHERE {
  ?s wdtn:P212 ?o .
}
LIMIT 10

Try it! Gives a result. I thought that amount properties would use wdtn for SI conversion Fnielsen (talk) 19:22, 4 April 2024 (UTC)

Can SPARQL return the hash of an item-valued qualifier?

I see that pqv: links statements to qualifier value nodes, but it doesn't return anything for item-valued qualifiers [e.g.]. I am using qualifier hashes in Wikibase-cli commands, and I'm having to get the JSON for each statement from the database and extract the qualifier hash from there. In thousands-long batches, those database calls take a lot of extra time; it would help a lot if I could just get the hashes in my sparql query. Thanks! Swpb (talk) 18:28, 21 November 2024 (UTC)

Indeed, in the RDF representation complex value nodes are meaningful only for values that describe something where extra information is useful (units, calendars). Because other wikidata items can be described unequivocally with their ID they do not need a complex value node attached to them. We do not store them because they don't add any value to the graph and would probably increase the number of triples quite significantly.

My understanding is that qualifier hashes are very much internal to the wikibase entity representation and you should not assume that the RDF model is using them, in fact it does not seem to use them for values (it does for references). So I would strongly suggest not using WDQS to track such low-level information. DCausse (WMF) (talk) 21:46, 21 November 2024 (UTC)

Ok, thanks. Swpb (talk) 14:41, 22 November 2024 (UTC)

version information in the dump

Is there any easy way to get version information out of a dump?

I have loaded a dump (both regular and lexemes) into several SPARQL engines. Is there a SPARQL query that will retrieve the versions of the dumps that went into the result? I want to find this out to check whether the engines actually loaded everything in the dumps. Peter F. Patel-Schneider (talk) 17:54, 22 January 2025 (UTC)

The doc has a "wikibase:Dump" predicate. Is it what you're looking for ? TomT0m (talk) 18:24, 22 January 2025 (UTC)

Not possible to identify novalue qualifier with the same property as the statement itself (double novalue)?

The statement Q1-P2-novalue has the same RDF dump as another statement Q1-P2-novalue-P2(qualifier)-novalue. The RDF dump is as follows:

 wds:Q1-XXXX wikibase:rank wikibase:NormalRank;
 	rdf:type wikibase:BestRank;
 	rdf:type wdno:P2.

How to fix it? Midleading (talk) 12:22, 24 February 2025 (UTC)

new entity types?

The page says: "Entity node describes the actual entity data and has type wikibase:Item or wikibase:Property depending on the kind of entity. Other entity types can be introduced in the future."

Shouldn't this be updated to include at least entity schemas? Also any other entity types should show up as well. Peter F. Patel-Schneider (talk) 14:23, 4 August 2025 (UTC)

incorrect information

The Turtle files use different prefixes from what are mentioned here. Is this a violation of the stable interface policy? Should I just change this page? Peter F. Patel-Schneider (talk) 22:53, 30 September 2025 (UTC)