Extension talk:SphinxSearch/2012

Usage with Sphinx 2.0.4

Has anyone been able to successfully use this extension with Sphinx version 2.0.4? I receive the following error when "testing out sphinx" (step 4):

 index 'wiki_main': search error: .

The only explanation I can think of is that perhaps the way the config file sets up the index is no longer compatible with the newer version of Sphinx. Has anyone been able to successfully run Sphinx 2.0.4 with their wiki, or should I downgrade my installation to Sphinx 2.0.3? I appreciate any assistance.

Update: The SphinxSearch extension itself appears to be functional even though queries from the command line are not. Feel free to close/delete this support thread. Jake8 (talk) 13:48, 6 May 2012 (UTC)

Actually we are using 2.0.4 in production with the latests SphinxSearch 0.8+ and we are using the exact same setup as with. (we haven't changed the configuration neither in case of 2.0.3/2.0.4). I'm not sure from where you get the message testing out sphinx, we have never seen this kind of message coming from Sphinx nor SphinxSearch. --MWJames (talk) 00:43, 6 May 2012 (UTC)

Oh no, I meant that I received the error in the code block in my original post when following the instructions under the "test out sphinx" step on the SphinxSearch article. The string "testing out sphinx" was never returned from either system.

As I wrote in my update though, SphinxSearch seems to work perfectly on the wiki itself (through the search daemon) even though it's broken from the command line. I have no idea what's causing this, but I guess it must be unique to my system configuration.

Thanks a lot for your help, and for your work on this excellent extension! :) Jake8 (talk) 02:38, 19 May 2012 (UTC)

Sphinx Extension Doesnt work for me

I am new to media wiki and for that matter to any web application development. I was exploring Mediawiki and have installed "MediaWiki 1.18.2" and have attempted to integrate Sphinx search. I have used "Sphinx 2.0.4-release".

The problem I am facing is that I am unable to search through the contents of the pages or the documents embedded in the MediWiki. It is only able to search the Title of the Pages. This is the same irrespective of the "GO" button or the "SEARCH" button. Whereas Through the command in the Sphinx Search, I am able to find all the text. Can someone please help me in debugging this. Sashidhar (talk) 08:54, 11 May 2012 (UTC)

The documentation is quite extensive (compared with other extensions), and it might be that you need to do the setup more than ones to fully understand the implication in how searchd, indexer and SphinxSearch is working together. If you follow the steps outlined in the documentation you should be able get the extension working but it will be hard for anyone to give any guidance as long their are no particular problems to solve.

For example did you do "Step 4 - Test Out Sphinx" to ensure that the infrastructure is setup correctly otherwise if this step fails something is wrong in your configuration and searching within MediaWiki will definitely not work as advertised.

In order to use Sphinx in MediaWiki you have register the extension, see "Step 9 - Extension Installation - Local Settings"

You should also consult the Extension:SphinxSearch/Troubleshooting page. MWJames (talk) 10:32, 11 May 2012 (UTC)

SphinxSearch and MediaWiki 1.19.0 -- fixed

After migraging to MediaWiki 1.19.0 SphinxSearch 0.85 does not work on the Spechial:Search page anymore (no results). The sidebar Search works.

After logging the searchd with --console, you found a message "query error: no such filter attribute 'page_is_redirect'" in the console. I searched this text and found it in 'SphinxMWSearch.php'. I change the line

$cl->SetFilter( 'page_is_redirect', array( 0 ) );

to

// $cl->SetFilter( 'page_is_redirect', array( 0 ) );

and everything works (for me).

I also change the method 'userHighlightPrefs' to

	public static function userHighlightPrefs() {return array( 2, 75 ); }

because the the definition changed (and I got warning when I activated "$wgShowExceptionDetails" in LocalSettings.php).

I hope this helps others fixing the same problems. 212.23.156.114 08:21, 16 May 2012 (UTC)

Perhaps page_is_redirect is missing in the sphinx.conf? It did not use to be there by default, but in more recent versions you should see it in at least three places in your sphinx.conf, in both sql_query settings, and in one sql_attr_uint setting. After you change the config file, you need to reindex the wiki. Svemir Brkic (talk) 14:23, 16 May 2012 (UTC)

Hello Svemir,

thanx, that was the solution. 212.23.156.114 14:20, 21 May 2012 (UTC)

During our tests with MW 1.19 / SphinxSearch 0.8.5 we haven't seen any problems and therefore are able to run SphinxSearch 0.8.5 without any changes on MW 1.19. MWJames (talk) 15:35, 16 May 2012 (UTC)

Trouble to find words in pages that are inside a custom namespace

I am having trouble to find words in pages that are inside a custom namespace.

I have installed Sphinx and SphinxSearch and it seems to be working, the queries are definitely going from the MediaWiki UI to the sphinx daemon, I executed "searchd.exe" in the console, and it is receiving the words that I type in the mediawiki search box.

If I type a word that exists in the main page, then it finds the main page, but if I type a word that is in a page with a custom namespace, then nothing... mediawiki just gives the option to create a new page, and nothing more.

I am sure that the word I am searching is there, and that sphinx can find it. If I execute "search.exe" and try to find the very same words, it says that it exists in some documents with a lot of hits.

I am executing this:

   search -c C:\Sphinx\sphinx.conf -i wiki_main -e "equip"

and it returns this:

   displaying matches:
   1. document=5, weight=2620, page_namespace=3000, page_is_redirect=0, old_id=11,
   category=()
           page_title=Equipe
           page_namespace=3000
   2. document=4, weight=1620, page_namespace=3000, page_is_redirect=0, old_id=12,
   category=()
           page_title=Principal
           page_namespace=3000

   words:
   1. 'equip': 2 documents, 4 hits

Notice that it knows about my custom namespace that have the id=3000.

If I try to search for the same thing using MediaWiki search box, the search daemon shows this:

   [Fri May 18 11:20:44.101 2012] 0.004 sec [ext2/1/rel 0 (0,20)] [*] equip

That means that it is receiving the "equip" word from the search box. But MediaWiki just says that there is no results, and suggests that I create the page.

What is wrong?

Why I cannot find the same results from "search.exe", by using MediaWiki search box?

Thanks! 200.251.63.3 14:27, 18 May 2012 (UTC)

Are you sure your custom namespace is selected? By default, MW searches only a limited set of namespaces. You need to change your search preferences, or use "Everything" or "Advanced" tabs to search additional namespaces. Svemir Brkic (talk) 15:00, 18 May 2012 (UTC)

This might be of interest default namespaces setting. MWJames (talk) 08:06, 19 May 2012 (UTC)

$wgMaxShellMemory default value too low for SphinxSearch_setup.php (MediaWiki 1.19, Sphinx 0.8.5/r115479, Sphinx 2.0.4)

I'm running MediaWiki in a Fedora 16 virtual machine (MediaWiki 1.19, Sphinx 0.8.5/r115479, Sphinx 2.0.4). Thanks for the great extension and documentation. The documentation was clear except for two issues. First, it wasn't clear that php enchant support had to be separately installed (php-enchant package on Fedora). Second, when running php SphinxSearch_setup.php to build the "Did you mean?" suggestion dictionary, the sphinx indexer kept crashing with an odd error: "libgcc_s.so.1 must be installed for pthread_cancel to work". But running the appropriate command indexer --buildstops ... directly worked fine. I saw that ulimit was being used to set resource limits on the shell used to run the indexer in the case of SphinxSearch_setup.php. It turned out that the default value of 102400 KB for the memory limit was too low for an almost-empty (4 documents, 4531 bytes - did I mention it was almost empty) instance of MediaWiki. I don't understand how this could be, so if anyone has ideas please let me know. Setting wgMaxShellMemory to 204800 KB in LocalSettings.php worked. 198.176.188.201 22:20, 5 June 2012 (UTC)

On my development box (47 docs, 17983 bytes) it works just fine with the default memory settings. Currently I have Sphinx 2.0.2 here, but I do not think that would be a reson for a huge difference in memory usage. Perhaps something related to PHP version you use or some settings it has? Svemir Brkic (talk) 13:57, 7 June 2012 (UTC)

Minor typo fix

I got tripped up by the use of 'spinxconf' instead of 'sphinxconf' as an option to SphinxSearch_setup.php. The following patch may avoid minor confusion for new users:

--- a/SphinxSearch/SphinxSearch_setup.php   2012-06-04 14:08:10.000000000 -0700
+++ b/SphinxSearch/SphinxSearch_setup.php   2012-06-05 12:59:57.289298038 -0700
@@ -24,7 +24,7 @@
        
        /* Override parameters setup becuase we do not need some of the default ones */
        protected function addDefaultParams() {
-               $this->addOption( 'spinxconf', 'Location of Sphinx configuration file', true, true );
+               $this->addOption( 'sphinxconf', 'Location of Sphinx configuration file', true, true );
                $this->addOption( 'indexer', 'Full path to Sphinx indexer if not in the path', false, true );
                $this->addOption( 'useindex', 'Sphinx index to use (defaults to wiki_main)', false, true );
                $this->addOption( 'maxwords', 'Maximum number of words to extract (defaults to 10000)', false, true 
@@ -36,7 +36,7 @@
                $max_words = intval( $this->getOption( 'maxwords', 10000 ) );
                $indexer = wfEscapeShellArg( $this->getOption( 'indexer', 'indexer' ) );
                $index = wfEscapeShellArg( $this->getOption( 'useindex', 'wiki_main' ) );
-               $conf = wfEscapeShellArg( $this->getOption( 'spinxconf' ) );
+               $conf = wfEscapeShellArg( $this->getOption( 'sphinxconf' ) );
 
                $cmd = "$indexer  --config $conf $index --buildstops sphinx.dic $max_words";
                $this->output( wfShellExec( $cmd, $retval ) );

198.176.188.201 22:27, 5 June 2012 (UTC)

Thank you for the fix. MW and extensions are moving from SVN to Git - I will apply and commit your change once all that is done. Svemir Brkic (talk) 00:54, 6 June 2012 (UTC)

It seems like its going it be a while before this extension is moved to Git. I made the change in SVN now (revision 115486) Svemir Brkic (talk) 12:18, 7 June 2012 (UTC)

Using Sphinx with Multi-Wiki Sites

We use 5 different wiki sites by introducing an HTTP switch statement in our LocalSettings.php which selects other LocalSettings_wiki1.php, LocalSettings_wiki2.php files. Additionally, each wiki has its own DB prefix. IE: wiki1_, wiki2_, etc...

When I changed the sphinx.conf file to include the wiki1_ prefix, wiki1 works great, but wiki2 and 3 don't return good searches.

How can I fix this?

Thank you so much! 97.78.241.155 17:58, 8 June 2012 (UTC)

Please provide a link to your sphinx.conf file (with passwords etc. removed of course.) Also include your "HTTP switch statement" Svemir Brkic (talk) 18:22, 8 June 2012 (UTC)

http://pastebin.com/MDuRMszc = LocalSettings.php

http://pastebin.com/ZQZfZVyJ = sphinx.conf

Note: The DB prefix in sphinx.conf is IES_ 97.78.241.155 20:20, 8 June 2012 (UTC)

Maybe I am missing something, but I see only one index definition in the sphinx file. Do you have one instance of sphinx daemon running, or one per database? If you have only one, you are correct that you need different index name for each one, but those indexes need to be define in the sphinx.conf file (they can each extend the main one, so that you do not need to repeat everything.) Svemir Brkic (talk) 20:31, 8 June 2012 (UTC)

So I can run more daemons with separate .conf files, and then reference them in each LocalSettings_wikiX.php file? 97.78.241.155 21:14, 8 June 2012 (UTC)

You can have sphinx instance with one sphinx.conf file, with multiple sphinx indexes defined in that file. If you do not have too much traffic, you probably want to do it this way.

Or you can have multiple sphinx instances, each with their own sphinx.conf file, each running on their own port or IP address. You probably only want to do this if you have a lot of traffic and can run each instance on a different box. Svemir Brkic (talk) 22:12, 8 June 2012 (UTC)

Do you have an example of how to define multiple indices? Thanks! 97.78.241.155 22:52, 8 June 2012 (UTC)

Enable search box suggestions-as-you-type that match words or phrases anywhere in the page title

I've slightly modified the extension to enable suggestions-as-you-type for the search box that match a word or phrase anywhere in the page title, not just at the beginning. For instance, typing "baby strol" should bring up pages with titles "Plik-o-matic baby stroller 2003", "Swift Baby Stroller", "Zippy one hand fold baby stroller by inglesina - new 2005", etc. as seen in this example. Use the patch below and add the following lines to LocalSettings.php:

$wgEnableMWSuggest = true;
$wgEnableSphinxInfixSearch = true;

Note that this overrides $wgEnableSphinxPrefixSearch = true; (search suggestions matching the beginning of the page title only).

--- a/SphinxSearch/SphinxSearch.php
+++ b/SphinxSearch/SphinxSearch.php
@@ -103,10 +103,17 @@
 # $wgEnableMWSuggest needs to be set to true as well
 $wgEnableSphinxPrefixSearch = false;
 
+# Same as $wgEnableSphinxPrefixSearch except that search suggestions include
+# matches against words in the middle of the page title. Takes precedence 
+# over wgEnableSphinxPrefixSearch if both are true.
+$wgEnableSphinxInfixSearch = false;
+
 function efSphinxSearchPrefixSetup() {
-       global $wgHooks, $wgEnableSphinxPrefixSearch;
+       global $wgHooks, $wgEnableSphinxPrefixSearch, $wgEnableSphinxInfixSearch;
 
-       if ( $wgEnableSphinxPrefixSearch ) {
+       if ( $wgEnableSphinxInfixSearch ) {
+               $wgHooks[ 'PrefixSearchBackend' ][ ] = 'SphinxMWSearch::infixSearch';
+       } elseif ( $wgEnableSphinxPrefixSearch ) {
                $wgHooks[ 'PrefixSearchBackend' ][ ] = 'SphinxMWSearch::prefixSearch';
        }
 }
--- a/SphinxSearch/SphinxMWSearch.php
+++ b/SphinxSearch/SphinxMWSearch.php
@@ -56,6 +56,23 @@
        }
 
        /**
+        *  PrefixSearchBackend override for OpenSearch results
+        */
+       static function infixSearch( $namespaces, $term, $limit, &$results ) {
+               $search_engine = new SphinxMWSearch( wfGetDB( DB_SLAVE ) );
+               $search_engine->namespaces = $namespaces;
+               $search_engine->setLimitOffset( $limit, 0 );
+               $result_set = $search_engine->searchText( '@page_title: ' . $term . '*' );
+               $results = array();
+               if ( $result_set ) {
+                       while ( $res = $result_set->next() ) {
+                               $results[ ] = $res->getTitle()->getPrefixedText();
+                       }
+               }
+               return false;
+       }
+
+       /**
         * Perform a full text search query and return a result set.
         *
         * @param string $term - Raw search term

This works for me using MediaWiki 1.19 and SphinxSearch 0.8.5 (SVN rev 115479). 198.176.189.201 14:29, 19 June 2012 (UTC)