I asked how the English Wikipedia word count is done and which words are considered (for instance, words inside image templates and tables) towards the count at en.wikipedia.org/wiki/Wikipedia:Help_desk and someone directed me here. Can anyone confirm that the counting is done using CirrusSearch? I can see from the Help page that it can count the number of pages, but it doesn't isn't immediately obvious to me how you would use it to count all words in all content pages like in en.wikipedia.org/wiki/Special:Statistics. Thank you in advance.
Topic on Help talk:CirrusSearch/Flow
Hi,
en.wikipedia.org/wiki/Special:Statistics does indeed rely on CirrusSearch to count the number of words in the content pages.
CirrusSearch relies on elasticsearch to search content on the wikis and we use the feature named token-count to count words. The field used is the text field which is a plain text version of the HTML output of the wiki page and does indeed contains table values and template content, it is mostly everything that is visible when viewing the page from a web browser. The way the text is tokenized into words is pretty standard.
If you are curious to know what is inside this text field please append &action=cirrusDump to any wiki page URL, this will display a json document in which you will find the field named text (you might want to install a browser extension to display json properly or copy/paste it into a tool that presents it to you nicely).
Some correction here: table content are considered auxiliary elements and not part of the text content used for counting words.
Words in the content matching the selector defined here will be part of the auxiliary text and thus not counted.