Help talk:CirrusSearch/2021

Search Function

Hey guys,

i have installed CirrusSearch and Elastica on my Wiki (1.35), but the search functionality does not get better/ or i do not see any improvements? Do I have to activat or add a code to my wiki page?

thanks Onmiy (talk) 09:56, 12 February 2021 (UTC)

Hi,

Did you follow the instructions present in the README file after installing the extensions?

CirrusSearch relies on elasticsearch and if you did not install it I doubt that CirrusSearch is functioning properly. DCausse (WMF) (talk) 10:35, 12 February 2021 (UTC)

That should be the problem. I will try :-) thank you Onmiy (talk) 11:06, 12 February 2021 (UTC)

CirrusSearch - Search button

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Hello,

after installing the extension and define the settings, is it necessary to put a CirrusSearch search bar on a page or is the CirrusSearch function integrated n the MediaWiki search funtion? Thanks Onmiy (talk) 11:00, 17 February 2021 (UTC)

The latter —TheDJ (Not WMF) (talk • contribs) 09:20, 19 February 2021 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

How to use Sort paramater syntax

Hello,

If I'm not mistaken, the section under "Explicit sort orders" lists a number of possible parameters and how to use them in a URL, but it's not clear to me how to use these by entering them in the normal search bar. Thanks. 2600:1700:55C0:6B50:5CF4:CABD:3472:B30A (talk) 19:24, 20 February 2021 (UTC)

It is (at least in the moment) not possible to influence the sort order by adding some parameter/-s into the search bar. As clearly stated in the English help text (may perhaps not be translated into some other languages yet): “Sorting options can be manually added to a search URL by adding &sort=order” (emphasis is from me). This means you have to alter the URL in the address bar. Speravir (talk) 01:42, 21 February 2021 (UTC)

Or install the AdvancedSearch extension, which provides UI for this and many other options. —TheDJ (Not WMF) (talk • contribs) 09:42, 23 February 2021 (UTC)

It should be mentioned, though, that this has less capability regarding sort options. So, for some cases it may not be sufficient. Speravir (talk) 22:44, 2 March 2021 (UTC)

my search suggestions are case sensitive

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Issues with CirrusSearch case sensitivity

I am running MW 1.35, CirrusSearch 6.5.4, Elastica 6.1.3

When I begin a search entry the search suggestions work fine:

https://jfwiki.org/csjpg/1.jpg

But after several letters it becomes case sensitive.

Here I search for "the d" and it shows no suggestions; they disappear.

https://jfwiki.org/csjpg/2.jpg

but searching "the D" shows me good suggestions:

https://jfwiki.org/csjpg/3.jpg

I would like lowercase search entries to find Title Cased suggestions. I have been trying to determine where/how this is configured but am having no luck. Thank you for any guidance!

JM Jackson Mornard (talk) 17:08, 22 March 2021 (UTC)

Checking your wiki, CirrusSearch is installed (per Special:Version) but does not appear to be the configured search engine. The most telling that this is the builtin search, and not CirrusSearch, is that Special:Search returns separate result sets for page title and page text matches.

It seems installation is incomplete, I would probably go back to the install instructions and ensure no steps were missed. EBernhardson (WMF) (talk) 21:34, 22 March 2021 (UTC)

Thank you so much for your time, that's very helpful. I thought that if it was appearing in Special Pages Version I was good.

I will investigate.

Thanks again. Jackson Mornard (talk) 21:49, 22 March 2021 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Search by uploader

Hi! I'd like to generate a category with the pictures I uploaded, but where I'm not the photographer. For my 10.000 pictures, I would need a search like

uploadedby:"Derbrauni" -incategory:"Photographs by User:Derbrauni"

Is something like this possible? Thanks! Derbrauni (talk) 11:06, 10 April 2021 (UTC)

I think the visible file upload log is not stored in the index, and also not in the wiki source, so you cannot search for it with CirrusSearch.

This said another search query: search results for file: insource:derbrauni -incategory:"photographs by user:derbrauni" -insource:/\| *[Aa]uthor *= *(\[\[[Uu]ser:)?[Dd]erbrauni/. Speravir (talk) 22:31, 10 April 2021 (UTC)

You might try to utilize insource:Derbrauni which will yield all pages containing your nick, or just Derbrauni -incategory:Photographs_by_User:Derbrauni and choose File: namespace.

• w:de:Benutzer:Derbrauni might consider w:de:H:Cirrus. PerfektesChaos (talk) 15:56, 10 April 2021 (UTC)

Wohl diese hier. Zur beliebigen Verfeinerung. PerfektesChaos (talk) 16:00, 10 April 2021 (UTC)

Hallo zusammen. Danke für die Hilfe. Die letzte Idee hatte ich auch schon. Die bringt aber letztlich nur Dateien, wo ich die "Fotos von mir"-Kategorie vergessen habe. Auch nicht schlecht, so konnte ich Einiges nachkategorisieren.

Aber es bringt leider nicht Bilder wie dieses zum Vorschein. Derbrauni (talk) 06:18, 11 April 2021 (UTC)

Verwirrung mit Suchergebnissen in Preview und Such-Seite

Hallo,

Ich hätte ein paar kleine Probleme mit Cirrussearch und Elaisticsearch

1. Ist es normal, dass im Suchergebis-Prieview (das, was unterhalb der Eingabefeldes der Suche rechts oben aufklappt, wenn man lostippt) nur Seiten aufgelistet werden, deren Namen auch mit dem eingetippten String anfangen aufgelistet werden? Wenn Ja, kann man das ändern, dass alles gefunden wird?
2. Wenn eine Seite, die schon länger (einige Tage) erzeigt wurde nicht in der Suche auftaucht. Was kann ich machen um herauszufinden, an was das liegt?

Danke! Und Grüße

Matthias Mattes1000 (talk) 10:02, 10 May 2021 (UTC)

1. Ja, das ist normal. Diese Ergebnisse sind (meiner Meinung nach) nur, um man nicht weiter tippen zu sollen; ich weiß nicht, ob das geändert werden kann. (Aber ich denke, dass eher nicht.) Tacsipacsi (talk) 13:57, 10 May 2021 (UTC)

How can I search for all the work of a particular artist? Search box is useless.

How can I search for all the work of a particular artist?  Search box is useless. 2601:643:8880:160:9181:5380:4E4D:62BB (talk) 19:49, 28 May 2021 (UTC)

Do you speak of media in Commons? If so:

• First you could search for a template {{Creator}} for the artist you search for. Type in search: creator: artist name or, if this lists too many results, creator: "artist name".
• Then with the known creator template search for file: hastemplate:"creator:artist name" (strangely, this is apparently case sensitive for "artist name", which differs from the default behaviour of this filter).
• Also, for every artist there should be a category which could be explored.

If want to search for media that is from an artist, but does not has the Creator template search for file: insource:"artist name" -hastemplate:"creator:artist name". If someone does not have a creator template then leave out the last part. Speravir (talk) 21:41, 28 May 2021 (UTC)

Wikipedia word count

I asked how the English Wikipedia word count is done and which words are considered (for instance, words inside image templates and tables) towards the count at en.wikipedia.org/wiki/Wikipedia:Help_desk and someone directed me here. Can anyone confirm that the counting is done using CirrusSearch? I can see from the Help page that it can count the number of pages, but it doesn't isn't immediately obvious to me how you would use it to count all words in all content pages like in en.wikipedia.org/wiki/Special:Statistics. Thank you in advance. Jasper Norbert (talk) 02:03, 5 June 2021 (UTC)

Hi,

en.wikipedia.org/wiki/Special:Statistics does indeed rely on CirrusSearch to count the number of words in the content pages.

CirrusSearch relies on elasticsearch to search content on the wikis and we use the feature named token-count to count words. The field used is the text field which is a plain text version of the HTML output of the wiki page and does indeed contains table values and template content, it is mostly everything that is visible when viewing the page from a web browser. The way the text is tokenized into words is pretty standard.

If you are curious to know what is inside this text field please append &action=cirrusDump to any wiki page URL, this will display a json document in which you will find the field named text (you might want to install a browser extension to display json properly or copy/paste it into a tool that presents it to you nicely). DCausse (WMF) (talk) 06:54, 7 June 2021 (UTC)

Some correction here: table content are considered auxiliary elements and not part of the text content used for counting words.

Words in the content matching the selector defined here will be part of the auxiliary text and thus not counted. DCausse (WMF) (talk) 12:42, 8 June 2021 (UTC)

¿Cómo puedo cambiar título de un artículo ya creado?

Hola, espero se encuentren bien la leer esta nota.

Hace unos dos años he creado un artículo sobre la Orquesta Failde, agrupación cubana nominada al Latin Grammy y de importante relieve nacional y en el campo danzonero mexicano. Necesito saber si me pueden ayudar a cambiar el título del artículo pues puse en aquel entonces el nombre original Orquesta Miguel Failde, pero ahora se llaman solo Orquesta Failde, cómo puedo cambiarlo?

Otra pregunta: cómo puedo hacer para que se verifique el artículo, pues creo tienes bastas referencias para lograr la verificabilidad del mismo.

Gracias de antemano

Les dejo el link del artículo es.wikipedia.org/wiki/Orquesta_Miguel_Failde Pedro Pablo Cruz (talk) 03:55, 12 June 2021 (UTC)

ES: Esta página es para discutir la función de búsqueda. Puede obtener ayuda para mover o cambiar el nombre de una página en Wikipedia: Café. ¡Buena suerte!

EN: This page is for discussing the search function. You can get help with moving or renaming a page at the Wikipedia:Café. Good luck! TJones (WMF) (talk) 04:14, 12 June 2021 (UTC)

Cirrus Search settings with elastic search

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I have installed Elastic Server on my mediawiki project.

Extension elastica and Cirrus Search also insatlled by using

wfLoadExtension( 'Elastica' );

wfLoadExtension( 'CirrusSearch' );

Iam able to see both extensions and elastic search in special:version.

But Now unable to configure it with my wiki existing search engine.

pls suggest so that i am able to use search engine as elastic search engine.

thanks! 143.97.2.35 (talk) 08:33, 15 June 2021 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Elasticsearch v7.13.1 in mediawiki 1.35.1

Hi , i have installed elasticsearch version 7.13.1 , into my mediawiki app 1.35.1 and Php 7.4.15 . Elastic search is installed properly and running .

now i have added cirrus Search and Elastica extensions.

Currently i am getting search engine result as MW default Search engine . No changes into my search engine.

should i downgrade the elastic search version or it will work fine.???

Please suggest me , 143.97.2.35 (talk) 11:14, 15 June 2021 (UTC)

Hi,

If you don't see any errors it is perhaps because CirrusSearch is not yet activated, please follow the steps in the README files:

• run the maintainance script to create and populate the indices UpdateSearchIndexConfig.php and ForceSearchIndex.php
• activate CirrusSearch setting $wgSearchType = 'CirrusSearch';

If you see errors please share them so that we can better you. DCausse (WMF) (talk) 07:07, 18 June 2021 (UTC)

Thanks for your reply, Pooja2425 (talk) 14:08, 22 June 2021 (UTC)

Hi,

One more thing to add - Cirrussearch isn't compatible with Elasticsearch 7.* - I recommend 6.* version, I personally use 6.5.4.

Hope that helps. ZPapierski (WMF) (talk) 07:20, 18 June 2021 (UTC)

Sorry (I misread your question) as ZPapierski pointed out CirrusSearch with MediaWiki 1.35 is not compatible with elasticsearch 7.13.1 please check the README file for information about compatible versions. DCausse (WMF) (talk) 07:27, 18 June 2021 (UTC)

Thanks for your rely!!

I think Cirrus Search is not Starting, I am getting MW default search engine results.

MediaWiki	1.35.1
PHP	7.4.15 (apache2handler)
MySQL	8.0.25
Elasticsearch	6.5.4

Done these steps

php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

php /extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip

php /extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse

Now $wgSearchType = 'CirrusSearch';

Not getting further what to do , to start my search engine as elastic search engine.

Please suggest me. Pooja2425 (talk) 07:09, 23 June 2021 (UTC)

If all the previous steps worked without error after setting $wgSearchType = 'CirrusSearch'; CirrusSearch should be active.

You can verify by searching on your wiki and adding &cirrusDumpQuery at the end of the search results URL, it should display a JSON document reprensenting the query that CirrusSearch is sending to elasticsearch. If not it means that $wgSearchType = 'CirrusSearch'; is not taken into account or that the CirrusSearch extension is not loaded properly (make sure that it is loaded using wfLoadExtension). DCausse (WMF) (talk) 07:29, 23 June 2021 (UTC)

Thanks for repling !!

Yes i am getting Json document after adding &cirrusDumpQuery .

But getting this into freshly installed test wiki.

In my actual wiki application, when i was trying ,,

php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

php /extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip

php /extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse

Now $wgSearchType = 'CirrusSearch';

indexing namespaces...

PHP Fatal error:  Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getParamWhitelist(): array in /data/www/html/wiki135/extensions/Elastica/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Indices/Exists.php on line 45

Since elastic search have installed, on root so test wiki working fine But my actual mediawiki application showing error.

pls suggest... Pooja2425 (talk) 10:51, 23 June 2021 (UTC)

Hi,

Similar problem was reported in T267106. DCausse (WMF) (talk) 08:22, 24 June 2021 (UTC)

Thanks for your help!!

Now this problem is resolved ,But Further when I have used these.

1)php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

2)php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip

3) php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse

then Indexed 10 pages ending at 66507 at 185/second

But when i try to search anything in my search engine its not giving any result.

Since i have used yet above 3 steps only. Should i have to use Bootstrapping large wikis methods??? 143.97.2.35 (talk) 13:04, 24 June 2021 (UTC)

Hi,

CirrusSearch relies on the MediaWiki JobQueue for processing its updates, please be sure it's setup properly and that no jobs are still in the queue:

php maintenance/showJobs.php --group

If it's empty please check mediawiki logs and elasticsearch logs there might be error messages there.

If it's not empty please see Manual:Job_queue. DCausse (WMF) (talk) 14:27, 24 June 2021 (UTC)

Thanks for reply,

Please suggest, what are the setting need to do with my DataBase.

Pls suggest!! Bcs not getting search result . 169.149.213.185 (talk) 13:42, 25 June 2021 (UTC)

Hi Since i have Run jobs , Now they is no job in the queue.

PoolCounter and Bootstrapping large wikis methods are not tried yet.

Should i have to implement all also . Bcs ima not getting results form search engine

Pls suggest !!! Pooja2425 (talk) 14:02, 28 June 2021 (UTC)

Issue with CirrusSearch

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Iam using these

MediaWiki	1.35.1
PHP	7.4.15 (apache2handler)
MySQL	8.0.25
Elasticsearch	6.5.4

But when trying to use php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php , facing below issue.

I havent changes anything in my Localsettings.php just used and not using elasticsearch Php Client yet..

wfLoadExtension( 'Elastica' );

wfLoadExtension( 'CirrusSearch' );

$wgDisableSearchUpdate = true;

indexing namespaces...

PHP Fatal error:  Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getParamWhitelist(): array in /data/www/html/wiki135/extensions/Elastica/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Indices/Exists.php on line 45

please suggest. Pooja2425 (talk) 14:10, 22 June 2021 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Searching Wikipedia on Nintendo Switch

I could search Wikipedia on Nintendo Switch, but it takes very long (2+ min) to appear search by type dialog. For 2 mins, it appears three dots (●●●) below advanced search and above search results. 86.141.193.112 (talk) 15:25, 27 June 2021 (UTC)

Cirrus Search SearchEngine not giving results

Iam using these

MediaWiki	1.35.1
PHP	7.4.15 (apache2handler)
MySQL	8.0.25
Elasticsearch	6.5.4

Done these steps

php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

php /extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip

php /extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse

Now $wgSearchType = 'CirrusSearch';

then Indexed 10 pages ending at 66507 at 185/second

after i run php maintenance/runJobs.php the jobs which are into php maintenance/showJobs.php --group.

Some i unable to run , then i directly deleted them from job table.

Now since there are no job in the queue.

Should i have to do connect with db, if yes , pls suggest steps.

Bcs i unable to get any result from my wiki search engine. Pooja2425 (talk) 05:58, 29 June 2021 (UTC)

When Checking Response in console getting error.

{"error":{"code":"toomanyvalues","info":"Too many values supplied for parameter \"namespace\".

The limit is 50.","limit":50,"lowlimit":50,"highlimit":500,"docref":"See http://wikipoc.equinor.com/wiki135/api.php for API usage.

Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce&gt;

for notice of API deprecations and breaking changes."}} Pooja2425 (talk) 06:19, 29 June 2021 (UTC)

Should i have to use all these for getting results

• Extension:Wikibase Repository
• Wikibase Extension
• WikibaseClient Pooja2425 (talk) 06:52, 29 June 2021 (UTC)

cirrus search not giving suggestion while typing in search box

Hi Iam using these

MediaWiki	1.35.1
PHP	7.4.15 (apache2handler)
MySQL	8.0.25
Elasticsearch	6.5.4

Done these steps

php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

php /extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip

php /extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse

Now $wgSearchType = 'CirrusSearch';

After that i run all the jobs. Now iam able to search pages into my search engine But, its not showing auto suggestion while i am typing like default mediawiki search engine, iam getting result only when i hit enter in search box,

pls suggest , if i missed any of the step, kindly check all above steps and let me know further steps to enhance my search engine.. Pooja2425 (talk) 18:16, 29 June 2021 (UTC)

showing error in console

{"error":{"code":"toomanyvalues","info":"Too many values supplied for parameter \"namespace\".

The limit is 50.","limit":50,"lowlimit":50,"highlimit":500, 169.149.246.113 (talk) 11:39, 1 July 2021 (UTC)

Is Redis really required?

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I'm currently working on testing CirrusSearch with AWS Elasticsearch in my dev environment, but first had to implement (AWS Elasticache) Redis for the job queue. However, I was recently told by someone on the Search team (I apologize, I forget who) that this may not be necessary unless there are perhaps hundreds of thousands to millions of jobs as WMF has. I've seen at most maybe 100-120k (aggregate for my five wikis) but mine are usually 100s, occasionally 1000s, and even more rarely 10s of thousands, which were otherwise handled fine in MySQL. So is it really necessary for CirrusSearch to have the job queues in Redis at that level? Justin C Lloyd (talk) 12:32, 5 August 2021 (UTC)

Extension:CirrusSearch#Dependencies has a note about Redis. It's not required, though. Ciencia Al Poder (talk) 09:51, 17 April 2023 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Can't find words that include ʿ or ʾ

Hello everybody,

In our MediaWiki we have titles that use diacritics for transliterations. Since CirrusSearch does accent/diacritic folding out of the box, we are able to search for pages without using diacritics. This is important especially for mobile users. However, if a word includes either ʿ or ʾ then it is not possible to find a page by searching for that word without including either ʿ,ʾ, or a \? (wildcard). I was wondering if it was possible to add ʿ and ʾ to the filter to have words be searchable without including them. For example:

Search for ʿilm by typing ilm, or 'ilm (normal quote).

This is our setup:

MediaWiki	1.35.3
PHP	7.4.22 (apache2handler)
MariaDB	10.4.20-MariaDB
ICU	66.1
Semantic MediaWiki	3.2.3
Elasticsearch	6.5.4
CirrusSearch	6.5.4 (ad4210f)
Elastica	6.1.1

Regards,

Pooya Pooya72 (talk) 13:59, 6 August 2021 (UTC)

Different languages have different analysis chains configured, besides the language-specific components. We have a long term plan (see T219550) to make them more consistent. I mention that because your description of the situation doesn't match the behavior I'm seeing on English Wikitionary. (If I search for ʿayin on English Wiktionary, then ʿayin is the first result and ayin is second. If I search for ayin, then ayin is first and ʿayin is third. If I search for 'ayin, then ayin is second and ʿayin is third.) What language config are you using?

You should be able to add something to your analysis chain to deal with this. I looked in detail at the English analysis and ʿ & ʾ are both removed by icu_folding. Ahh... that could be it! If you don't have analysis-icu installed, then you get ascii_folding, not icu_folding from the default English config, and ascii_folding does not remove ʿ & ʾ. That could be it!

If you don't want to or can't install analysis-icu or don't want all the aggressive folding of icu_folding, then you could have a much more targeted solution by adding a character filter to remove ʿ & ʾ. (You could map them to ' if ' gets used elsewhere in similar contexts, but it may lead to unwanted behavior. The standard tokenizer strips ' at the edges of words, and aggressive_splitting splits on ', so a'ilm'b gets tokenized as a, ilm, b, while aʿilmʾb gets tokenized as ailmb—at least when icu_folding is enabled.)

I hope that helps. If that doesn't address all your problems, let us know what plugins you have installed, please!

—Trey TJones (WMF) (talk) 16:26, 6 August 2021 (UTC)

Thanks @TJones (WMF)! Looks like installing anaysis-icu is the way to go. Tokenizing aʿilmʾb as ailmb is what we want. If you could point me towards the relevant documentation that would be great.

Edit: I installed the plugin as per the instructions here, and saw the plug-in come up when I restarted elasticsearch. I also added $wgCirrusSearchUseIcuFolding = 'yes'; to LocalSettings.php but the search functionality is still the same. My current language code is:

$wgLanguageCode = "en-gb"; Pooya72 (talk) 16:43, 6 August 2021 (UTC)

Glad to help. Looks like you found the documentation on your own!

Did you also reindex with UpdateSearchIndexConfig.php? It will rebuild the analysis chain with the ICU upgrades and reindex.

You can search for "text" and "plain" (with quotes) on the English Wikipedia config to see what the analysis config should look like, and see where yours differs (if it does). TJones (WMF) (talk) 17:17, 6 August 2021 (UTC)

Thanks @TJones (WMF), will look into this on Monday. Pooya72 (talk) 17:40, 6 August 2021 (UTC)

Sounds good, @Pooya72! Monday is a holiday for us, so we'll check in on Tuesday. TJones (WMF) (talk) 18:34, 6 August 2021 (UTC)

@TJones (WMF) ran php ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php but no change. This is our config: https://pastebin.com/mG3Jv4tk Pooya72 (talk) 10:13, 9 August 2021 (UTC)

@Pooya72, check out the docs on in-place reindexing for UpdateSearchIndexConfig.php usage. You need to call it with mwscript . Internally, we use the reindex() function, or some variation of it, to reindex our wikis. In addition to calling UpdateSearchIndexConfig.php, it keeps track of the time the reindex started (REINDEX_START) and calls ForceSearchIndex.php to catch up on activity that happened while the reindex was running—reindexing English Wikipedia, for example, can take hours, so a lot can happen in the meantime. TJones (WMF) (talk) 17:11, 10 August 2021 (UTC)

Just wanted to call out that the params --reindexAndRemoveOk --indexIdentifier now are the really necessary ones to make the reindex happen. You need to specify the wiki and cluster (if you have multiple clusters, and then you need to reindex each cluster, too). If things don't seem to work, please share the command you used and its output. TJones (WMF) (talk) 17:35, 10 August 2021 (UTC)

Thanks again @TJones (WMF). I ran php ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --reindexAndRemoveOk --indexIdentifier=now and this was the output: pastebin. We only have one instance, and it's at the beginning of the project so there are only a handful of entries, and no users. Pooya72 (talk) 10:44, 11 August 2021 (UTC)

That looks like a successful run, but I see that you only have the analysis-icu plugin. You also need the Wikimedia extra plugin to enable icu_folding. I'm sorry that I'm not familiar with the base MediaWiki install, so I didn't realize this sooner; I had assumed extra and experimental-highlighter and maybe ltr would be installed by default.. You can install extra like so: elasticsearch-plugin install org.wikimedia.search:extra:6.5.4

After that, reindex again and let's see where we are. TJones (WMF) (talk) 14:51, 11 August 2021 (UTC)

Worked like a charm. Many thaks @TJones (WMF)! Pooya72 (talk) 15:35, 11 August 2021 (UTC)

Woo-hoo! Glad to help. TJones (WMF) (talk) 15:43, 11 August 2021 (UTC)

Problem indexing pdf documents that include cyrilic characters

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Hello guys,

Lately i've been up to a task of deploying local MediaWiki. Everything went smooth until it came to indexing inside of pdf files that contain characters other that US ascii. Doing '?action=cirrusDump' and looking at 'file_text' field shows that all cyrillic characters are getting dropped while latin characters are preserved. Folks at ru.wikipedia.org somehow managed to do it but i couldn't find solution online. I would be very thankful if somebody could point out why that happens and how i could potentially solve this problem.

My configuration is:

MediaWiki - 1.36.1

PHP - 7.4.22 (apache2handler)

PostgreSQL - 13.3

ICU - 66.1

Elasticsearch - 6.5.4

PDF Handler - c9705a8

AdvancedSearch - c8a42b8

CirrusSearch - 6.5.4 (ab802b7)

Elastica - 6.1.3 (9f6e66a)

My elasticsearch configuration:

analysis-icu

extra MediaWiki plugin

ingest-attachment LveFunc (talk) 03:19, 19 August 2021 (UTC)

Hi,

CirrusSearch does not manipulate the text it receives from Extension:PdfHandler. I would check if this extension is working properly especially that the tooling it depends on (set via $wgPdftoText, likely to be pdftotext) is properly extracting the text you expect. DCausse (WMF) (talk) 09:46, 23 August 2021 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

2 questions

Hello!

Some of the filenames of these files aren't complete for some reason. Also you would expect that this search would say "File:PDP-CH - Philadelphia Orchestra - -Wikipedia-Leopold Stokowski-Leopold Stokowski - Brandenburg Concerto No. 2 in F major, BWV 1047 - 1st Movement- Allegro - Johann Sebastian Bach - Hmv-d1708-42-606.flac (redirect from File:PDP-CH - Philadelphia Orchestra - -Wikipedia-Leopold Stokowski-Leopold Stokowski - Brandenburg Concerto No. 2 in F major, BWV 1047 - 1st Movement- Allegro - Johann Sebastian Bach - Hmv-d1708-42-606.flac.flac)" however it isn't. Might anyone here know why? Jonteemil (talk) 02:29, 6 September 2021 (UTC)

Hi,

I'm not sure to understand what is not complete in the query results for intitle:/Philadelphia Orchestra/ filemime:audio/x-flac do you have a specific page missing.

The second query you pasted contains error and after fixing it (I assumed you wanted to search for the redirect with the doubled .flac extension) finds the page you mention. Here is how I fixed the query:

intitle:/Philadelphia Orchestra/ filemime:audio/x-flac intitle:/\.flac\.flac/.

When searching for a redirect the search engine will always display the redirected page, sometimes you may see a hint that you matched a redirect when the mention (redirect from: page_name) appears after the page title, see for instance the results for: intitle:/\.flac\.flac/ filemime:audio/x-flac incategory:"Swiss Foundation Public Domain". DCausse (WMF) (talk) 07:38, 6 September 2021 (UTC)

DCausse, the filenames are only partially displayed, from the first search Jonteemil provided it seems there is a maximum length for display, some limit for characters, and then the second search condition intitle:/\.flac\.flac/ only narrows down the result(s) without adjusting the displayed lines.

But with an altered search I get the full display, it links to the redirected file with only one file extension, though, as already pointed out by DCausse: file: filemime:audio/x-flac intitle:/Philadelphia Orchestra.+\.flac\.flac/. Note that I merged both regex searches, as the one with two of them is really bad in terms of server loading (I also added the namespace as search domain; this should be always added if possible). Speravir (talk) 17:54, 6 September 2021 (UTC)

Set up CirrusSearch

Hi,

I believe it is working. If I change the config to an invalid search it fails and correct it , it works

$wgCirrusSearchServers = [ [ 'host' => "192.168.1.blahblah", 'port' => 31130 ]  ];

My expectation, which may be incorrect, was that I could type in a subheading e.g. "Cors" and the page containing ==Cors== would return the page in the search

Here is my set up

Thanks

LocalSettings

# This is the elastic search config

# Dont forget kubebernetes

wfLoadExtension( 'Elastica' );

wfLoadExtension('CirrusSearch');

#$wgDisableSearchUpdate = true;

$wgCirrusSearchServers = [ [ 'host' => "192.168.1.70", 'port' => 31130 ]  ];

$wgSearchType = 'CirrusSearch';

Add this to LocalSettings.php:

wfLoadExtension( 'Elastica' );

wfLoadExtension( 'CirrusSearch' );

$wgDisableSearchUpdate = true;

Add this elastic search endpoint or points

$wgCirrusSearchServers = [ 'elasticsearch0', 'elasticsearch1' ];

your elasticsearch index:

php $MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

Now remove $wgDisableSearchUpdate = true from LocalSettings.php.  Updates should start heading to Elasticsearch.

Next bootstrap the search index by running:

php $MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip

php $MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse

I am using elasticsearch 6.5.4 Bwwiseman (talk) 08:08, 18 September 2021 (UTC)

Hi,

It seems to me that the Manual:Job_queue might not be properly configured. CirrusSearch relies on it to ship the documents to elasticsearch and if it's not properly configured the indices might remain empty. DCausse (WMF) (talk) 07:51, 21 September 2021 (UTC)

Couldn't connect to host, Elasticsearch down?", Elastica\\Exception\\Connection\\HttpException

Hi, I am using these, I installed elastic server But not showing below.

Product	Version
MediaWiki	1.35.3
PHP	7.4.24 (apache2handler)
MySQL	8.0.26
ICU	65.1
Lua	5.1.5

also installed required extensions and elasticsearch/elasticsearch": "6.5 client.

Elastica

6.1.3 (f3c9459) 01:29, 3 September 2021

CirrusSearch

6.5.4 (95b958b) 19:07, 20 August 2021

wfLoadExtension( 'Elastica' );

wfLoadExtension( 'CirrusSearch' );

$wgDisableSearchUpdate = true;

$wgCirrusSearchIndexBaseName =  ''; //DataBase Name

php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

Now remove wgDisableSearchUpdate = true;

php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip

php extensions/CirrusSearch/maintenance/ForceSearchIndex.php –-skipParse

$wgSearchType = 'CirrusSearch';

# php /data/www/html/wiki/maintenance/runJobs.php

1)when trying to search anything in search engine :

An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later.

2) http://wiki/api.php?action=cirrus-settings-dump

"code": "internal_api_error_Elastica\\Exception\\Connection\\HttpException", "info": "[YUsbMuAOtsHM4e8mbGG6NwAAAAs] Exception caught: Couldn't connect to host, Elasticsearch down?", "errorclass": "Elastica\\Exception\\Connection\\HttpException", "*": "Elastica\\Exception\\Connection\\HttpException at /data/www/html/wiki/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Transport/Http.php(190)\n#0 /data/www/html/wiki/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Request.php(194): Elastica\\Transport\\Http->exec()\n#1 /data/www/html/wiki/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Client.php(689): Elastica\\Request->send()\n#2 /data/www/html/wiki/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Index.php(571): Elastica\\Client->request()\n#3 /data/www/html/wiki/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Index/Settings.php(383): Elastica\\Index->request()\n#4 /data/www/html/wiki/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Index/Settings.php(75): Elastica\\Index\\Settings->request()\n#5 /data/www/html/wiki/extensions/CirrusSearch/includes/Api/SettingsDump.php(36): Elastica\\Index\\Settings->get()\n#6 /data/www/html/wiki/includes/api/ApiMain.php(1593): CirrusSearch\\Api\\SettingsDump->execute()\n#7 /data/www/html/wiki/includes/api/ApiMain.php(529): ApiMain->executeAction()\n#8 /data/www/html/wiki/includes/api/ApiMain.php(500): ApiMain->executeActionWithErrorHandling()\n#9 /data/www/html/wiki/api.php(90): ApiMain->execute()\n#10 /data/www/html/wiki/api.php(45): wfApiMain()\n#11 {main}" }

pls suggest which step i am missing, Pooja2425 (talk) 12:20, 22 September 2021 (UTC)

Exclude edits by users

How to exclude edits by users (let from some certain vector of users) from search view via cirrus?

It'd be useful for patrolling recent changes when you don't want see 2 most active users right now, for example. DonSimon (talk) 18:17, 9 October 2021 (UTC)

Unsatisfying answer: Search can analyze content only, but not metadata.

• The adjoined metadata of a page would be timestamp, user, summary etc.
• The content is the visible text of the rendered page, or source text of the page itself without transclusions.

For recent changes a search within all articles or pages within the wiki is the wrong tool.

• There might be some gadgets already existing, which will filter the recent change list by a vector of trusted users, to focus on less well known people. At least a gadget programmer could easily remove those by screen grabbing. This one could do that, among many other things, but is not focussing on such list of trusted users.
• By MediaWiki software there are options on watchlist and recent changes (which are based on the identical software) to suppress bots, registered users, or minor edits (unsafe). PerfektesChaos (talk) 08:38, 10 October 2021 (UTC)

Safe sandbox

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Is there a sandbox to test in, so I don't affect performance ~ Wakelamp (talk) 12:02, 1 November 2021 (UTC)

What are you planning to test? Abuluntu (talk) 12:46, 1 November 2021 (UTC)

No So I am not doing a performance load test, :-) I just saw the performance issue affecting others. Have many people screwed up? I assume you have some sort of timeout - Is that a switch I can set lower? Anway I am trying to work out ways of getting what I am after some other way. Wakelamp (talk) 14:02, 1 November 2021 (UTC)

I’m not sure if I’m the right one to ask. If there is a performance issue it’s probably discussed at phabricator. Wish you the best of luck! Wkee4ager (talk) 14:37, 1 November 2021 (UTC)

If you are referring to the WMF wikis, when the search system cuts off a query for performance reasons (typically it took too long to execute) that's normal and expected behaviour that shouldn't negatively impact others (assuming you aren't a bot making many parallel queries). Overall you shouldn't need to worry about it, beyond pondering how to construct a query that doesn't timeout

If you are trying to construct a query and it keeps timing out, could you could post some details of the information you are trying to retrieve? Someone might be able to point out a more efficient way to get the same information. EBernhardson (WMF) (talk) 18:12, 1 November 2021 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Need to upgrade elastic search Library log4j-1.2-api-2.11.1.jar

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Hi Team,

Hi Team,

we are using below,

MediaWiki	1.35.3
PHP	7.4.23 (apache2handler)
MySQL	8.0.26
Lua	5.1.5
Elasticsearch	6.5.4

/usr/share/elasticsearch/lib/log4j-1.2-api-2.11.1.jar

log4j-api-2.11.1.jar

log4j-core-2.11.1.jar

x-pack-security/log4j-slf4j-impl-2.11.1.jar

please provide us any patch which is higher then log4j>2.15.0 Pooja2425 (talk) 12:15, 14 December 2021 (UTC)

We do not provide elasticsearch. Mediawiki only uses it. Please contact elastic.co itself. Or just restart elasticsearch with the variable which disables the affected functionality. This is widely documented but looks something like -Dlog4j2.formatMsgNoLookups=true —TheDJ (Not WMF) (talk • contribs) 14:03, 14 December 2021 (UTC)

Thanks @TheDJ for help,

please let me know only i need to set -Dlog4j2.formatMsgNoLookups=true into etc/elasticsearch/jvm.options or anything else also need to do.

As we are using Elasticsearch v 6.5.4 and java 1.8.0

Or pls suggest me link where i can confirm all this.

thanks Pooja2425 (talk) 06:30, 15 December 2021 (UTC)

Also should i remove the JndiLookup class ??

pls suggest. Pooja2425 (talk) 08:30, 15 December 2021 (UTC)

Please contact elastic.co —TheDJ (Not WMF) (talk • contribs) 09:15, 15 December 2021 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.