User:TJones (WMF)/Notes/N.O.R.M. MVP Design Notes/Hindi Transliteration

(April 2026) See also T297761.

Background

Lots of Latin

Hindi Wikipedia has a very high zero-results rate. During the analyzer unpacking project, I noted that it was over 60%. In my sample at the time, more than 80% of zero-results queries were in the Latin script, almost 70% of those looked to be Romanized Hindi, and 40% of those got results when transliterated back to Devanagari using a quick-n-dirty online transliteration method.

Input Options

I'm not sure why so many Hindi Wikipedia users search in Romanized Hindi. The desktop website offers a number of input tools. See the screenshot and translation below, "Wikipedia: How to type in Devanagari?" (in Hindi), and "Universal Language Selector Input methods—Hindi Transliteration" (in English). Mobile doesn't seem to to support those input tools, and I don't know how well commonly used mobile devices support Devanagari input—though my 8-year-old American iPhone does.

In a sample of over 30,000 Hindi Wikipedia queries, I don't see any evidence of Latin queries using the mixed-case transliteration scheme in the Universal Language Selector documentation linked to above. (Searchers may be using the transliteration scheme and the input tool to generate queries in Devanagari, of course.)

So, for whatever reasons, a lot of searchers on Hindi Wikipedia are using Romanized Hindi, which generally isn't very effective... so how can we meet them where they are?

Second-Try Searching

Now that we have Russian and Hebrew DWIM wrong-keyboard mappings and Georgian Latin/Cyrillic transliteration mappings all running for autocomplete on their respective wikis, it's time to look at implementing Hindi Transliteration.

There are a number of open-source Hindi transliteration projects, but they all seem to use some form or machine learning, which doesn't seem lightweight enough to use as part of our autocomplete lookups (which occur as people type their query in the search box, so they need to be very fast). I understand why Hindi transliteration projects use machine learning: the English/Latin alphabet isn't "big" enough to transliterate Hindi properly—there are too many sounds and not enough (easy to type) letters. (The Latin alphabet isn't really big enough to spell English properly, but that's a different story.)

Google's Input Tools are fast, but only because they can throw a lot of hardware at the problem—and they aren't open source.

My hope is that with a more lightweight approach we can still get a high degree of accuracy, and, for this first project, we can also benefit from the the fuzziness of autocomplete.

A set of lightweight rewrite rules, plus some hard-coded exceptions, plus autocomplete fuzziness could get us a long way toward better results for Hindi searchers.

A possible failure mode is mapping English or other languages to Hindi when we shouldn't. (Though with names this can be a benefit as the Romanized Hindi → Hindi mapping can work as an English → Hindi mapping at least part of the time, and there are lots of transliterated—rather than translated—titles of films, books, etc.)

Another failure mode is just ending up with a poor mapping. In both cases, we have two possible ways of being rescued in the context of autocomplete. If the result is truly terrible, it won't get any results and the searcher will never know we tried. If the result is only off by one or two characters, and not terribly ambiguous, good results will be shown anyway, and the searcher will never know we messed up. But there's a sweet spot (sour spot?) where the mapping will fuzzily match only unwanted results. Part of the goal is to make sure that doesn't happen too often.

Data

I started with 50,000 fulltext queries (i.e., those that did not get routed to an article by autocomplete) pulled from Hindi Wikipedia from a 6½ month period (July 2025 through Mid-January 2026). Around 5,000 queries were removed as duplicates. (This is an anti-spam, anti-bot measure, though it can remove genuine repeated queries.) Around 4,000 queries were filtered, mostly obvious porn, but also all numbers, URLs, and junk queries.

This collection is probably a little pessimistic for autocomplete, since not all fulltext queries are necessarily failed autocomplete queries—some wildly long queries, for example, are clearly not reasonable article titles. However, looking at these will also give insight into how future did-you-mean suggestions for fulltext queries might perform.

Categorizing Queries

I did a cursory review of the queries and categorized the remaining 41,187 queries by writing system. "Latin" means queries that are only Latin and possibly non-letters (punctuation, numbers, etc.).

About 80% of the remaining queries have Latin characters, and about 78% are just Latin. Only about 16% are just Devanagari, and about 2% have Devanagari and Latin characters. The next biggest category is Cyrillic, with 0.6% of queries, and the rest are smaller than that.

#queries	Script / category
32,010	Latin
843	Devanagari & Latin
4	Bengali & Latin
3	Gujarati & Latin
2	Gurmukhi & Latin
2	Kannada & Latin

6,549	Devanagari
626	.js/.css/etc filenames
240	Cyrillic
137	Georgian
133	Greek
133	Armenian
114	Korean
107	Japanese
61	Arabic
53	Hebrew
46	Chinese
46	Gujarati
14	Telugu
13	Tamil
31	wrong encodings (21 \x..)
9	namespace/keywords (:)
3	symbols
2	Ol Chiki
2	Oriya
2	Thai
1	𝕄𝕒𝕥𝕙 𝓛𝓪𝓽𝓲𝓷
1	Malayalam

A quick review of the Latin queries reveals lots of what looks like Romanized Hindi to my very non-expert eye. There's also plenty of English and obvious names. If I go looking, I can find French, Polish, German, Afrikaans, Vietnamese, and others.

Latin Word Frequency

Taking all the "words" (strings of extended Latin letters or numbers delineated by whitespace or punctuation) from these Latin queries that are only made up of letters, gives 108,529 total Latin words. Lowercasing and grouping by frequency (note that these groups overlap, and each larger group contains the smaller groups):

unique words	frequencies
31,033	1+ freq (i.e., all unique words)
8,883	2+ freq
5,428	3+ freq
3,080	5+ freq
1,480	10+ freq
508	25+ freq

So, there are only about 31K unique Latin words in this sample, and only 1480 words occur 10 times or more.

Looking at the list from the most frequent end, the top ten most frequent words, below, account for about 13% of all words. A quick look with Google Translate's Hindi input UI indicates that all of these could be Romanized Hindi words... though looking at the actual queries they come from, of is in fact usually English and in is frequently English. On the third hand, plenty of words get borrowed or just transliterated (as in titles) from English to Hindi.

freq	word
2,518	ka
2,152	ke
1,945	ki
1,802	hai
1,184	me
1,119	in
999	hindi
924	kya
800	of
660	mein

14,103	[total]

The number of total words accounted for by the top 10, 50, 100, 508 (i.e., all those with frequency of 25+), 1000 and 1480 (10+) are listed below.

total words	%	top n
14,103	13%	top 10
24,759	23%	top 50
30,349	28%	top 100
47,049	43%	top 508 (25+)
55,901	52%	top 1000
61,290	56%	top 1480 (10+)

For the purposes of making my own data set, I dropped the words that occurred less than 3 times, leaving around 5,400 unique word types representing around 77,400 word tokens.

Not that we ever doubted it, but Zipf's law is real!

Early Thoughts

On the one hand, if the most common 508 or 1480 transliterated Latin words could each be perfectly mapped to a Hindi word in a lookup table, we'd already be about halfway to success! Unfortunately, I don't think that's the case, because I worry that really short words (ka, ke, ki, me) are going to be very ambiguous.

For example, the Google Translate input UI suggests both क and का for "ka". If the split is 90/10 in favor of one or the other, we might do okay. If it's 50/50, we're going to have a lot more errors. And, of course, if a single query has several ambiguous words in it, we're more likely to get too many wrong for autocomplete to be able to compensate.

Also, skimming the top 100 most frequent words in the list I see that at least 20 look very much like English. Out of context it can be hard to be sure—very short words show up in many languages, and words can be borrowed into other languages. But movie, list, history, and English look like English.

We'll have to figure out what to do in this situation that is efficient and effective. Maybe transliterating them will succeed (because the word has been borrowed) or totally fail (because the result doesn't look like a Hindi word at all and gets no matches). Or maybe we'll put the most common ones in the hard-coded mapping table and map them to themselves so they don't get transliterated. Or maybe we'll learn that having a certain number of such words (maybe just 1, maybe several) means we shouldn't bother to try to transliterate.

More Data

I was fortunately able to find an online transliteration data set with a sufficiently permissive license: The Dakshina Dataset (CC BY-SA 4.0). This data set has data for 12 South Asian languages, but I'm only looking at the Hindi data for this project.

The Dakshina data set provides Hindi words and Latin transliterations, broken up reasonably into training, test, and dev sets, with over 50K entries total. The transliterations are many-to-many, so not only can a given Hindi word have multiple transliterations, a given transliteration can represent multiple (usually pretty similar) Hindi words.

It includes frequency data, but the distribution is a little wonky, in that they max out at 10, and only 48 words (out of over 44K, or 0.11%) have a frequency of 10.

It also includes a few errors, which is to be expected, considering the size of the data set and the amount of human annotation involved. Some errors include duplicated diacritics, like े, ै, or ं; or using आ + ॅ instead of ऑ, or ा + ॅ instead of ॉ. There are some suspect transliterations, like जायंट is transliterated as giant (the likely original English word), jayant (a reasonable transliteration of the Hindi), and joint, which feels like a mistake. जॉइंट is also transliterated as joint, which seems more reasonable. The Dakshina data set is super useful even with little foibles like this, and the joint situation may be an accurate reflections of what people actually do.

I used the Dakshina data set to determine "probabilistically preferred transliterations" for my list of most frequent on-wiki query words, taking the highest frequency term (and arbitrarily breaking ties). I did break with typical machine learning protocol (but this isn't a machine learning exercise) and reached into the test and dev sets to get transcriptions for my query words data set.

I also found transliterations for all of the remaining words that occurred 25 times or more in my sample and verified that they exist and seem to mean the right thing by searching Hindi Wikipedia.

Assessment Framework

I built a tool for assessing the quality of automatic transliteration compared to the assigned human transliterations, using both the Dakshina training data and my query words sample. The Dakshina data is more realistic in its ambiguity, as a single fixed transliteration from a given Latin word to a specific Hindi word will often not be correct 100% of the time because multiple transliterations occur. My query words sample is more realistic in its frequencies, though, because a lot of words occur more than 10x more frequently than other words. The query words sample also better reflects the actual words used by Hindi searchers, obviously.

In addition to counting correct transliterations as types (i.e., each entry in the data set counts as 1) and as tokens (i.e., each entry in the data set is weighted by its frequency), I also rigged up an edit distance calculation to track how many types/tokens were off by 1, 2, or 3 characters. This is useful for roughly estimating what the completion suggester autocomplete can compensate for (though I don't filter first-letter errors, which autocomplete doesn't handle), and rarely as a secondary metric for whether a change to the transliteration algorithm is useful (e.g., a change that results no difference in fully correct transliterations, but increases the number of transliterations that are off by only 1 letter might be worth keeping).

I also later built another tool analyzing the results of running queries en masse against the Hindi Wikipedia autocomplete and fulltext to compare changes in zero-results rate, number of results, and impact of second-try searches. We will see the results later.

Transliteration Algorithm

As I mentioned before, there aren't enough letters in the (English plain ASCII) Latin alphabet to map to all the letters and sounds of Hindi. There are several formal, official, and academic transliteration schemes that cover everything, but they are also clearly not used by Hindi searchers. There are some obvious digraphs—like sh, bh, dh, aa, ii—that extend the number of letters and sounds that can be transliterated, though there are common ambiguous digraphs, too—like th, ee, oo.

I started with some basic mappings of lowercase ASCII letters common to most or all of the formal mappings and worked up from there. My initial transliteration accuracy (for types, with no errors) was around 4%. I admit I was not feeling very optimistic at that point.

I generally just kept looking for patterns of errors between my transliteration and the provided transliteration and slowly ratchet up the accuracy. A few of my initial transliteration values were "wrong", in that in practice the split between options is 55/45 in favor of the other choice. hat is, there's no good answer, but it's better to get it wrong 45% of the time instead of 55%.

I found contextual patterns both based on location in the word (particularly initial or final), and the environment of other letters. I'd done something similar for Georgian transliterated in Cyrillic, which doesn't use upper- and lowercase to distinguish ambiguous sounds.

I also looked for patterns of errors in the near-final Hindi transliteration. Because there are several ways to transliterate the same Hindi letters, a pattern of co-occurring Hindi letters is less obvious while still in Latin but much more obvious once in Hindi.

There was a recurring pattern throughout the project of thinking I was mostly done and then noticing a particular word, looking for a pattern, finding more things to handle, and improving the algorithm. Each step was kind of small, but in the end, I got that 4% (types with no errors in the Dakshina training data) up over 31%—which translates to over 75% of tokens with no errors in the query word sample!

Hard-Coded Transliterations

Of course, an easy way to get good results is to memorize the right answer. I originally added a look-up table for ~500 of the most common words in both the Dakshina training data and my query words sample, including all the ones with frequency counts over 25. I also added a few related words, for example, the rest of the English and Hindi numbers up to 12.

I validated the hard-coded transliterations by looking up both the Latin transliterations and Hindi versions in Hindi Wikipedia. I reviewed the ones with a very low number of Hindi results, and replaced a few transliterations with better versions, based on usage in Hindi Wikipedia.

Hard-coding some words gives a nice baseline for accuracy (though I worry about ambiguous words being hard no matter what), insulates the most common words from changes to the algorithm during development, and gives good answers for some of the most egregious ~~stupidities~~ "idiosyncrasies" of English spelling (since English words are also commonly transliterated to Devanagari)—like one, two, and women. It also probably speeds up word-by-word processing if 40% or more of words are looked up in a hash table rather than being processed through all the regex-based rewrite rules.

I had originally planned to hard-code all words with frequency counts over 10 (~1400 words, rather than ~500), but in addition to roughly tripling the size of the look-up table (not a huge concern at this scale) a fair number of them don't have transliterations in the Dakshina data set, so I'd have to spend even more time looking them up. Eventually I decided to take a middle road and hard-code all the words with frequency counts over 10 that had transliterations, which was most of them.

Training Results

The performance of the type-level Dakshina training data (i.e., how many unique words transliterated correctly) was my original benchmark for improvements during development, but I would verify improvements with token-level performance and types and tokens in the Hindi Wiki query word data. Due to the greater skew in the frequency distribution in the query word data, it generally has better numbers in the token measures. The most common frequency values in the Dakshina data are 1–3, so the type and token measures are fairly similar (with the token counts being roughly double the type counts).

With ~1400 hard-coded words the algorithm correctly transliterates about 32% of the unique words in the Dakshina data set, for about 35% of the token count. In the query word data, we get almost 38% of the unique words correct, but that translates to about 76% of the token count.

Including words with 1 or 2 errors get the query word data set to about 65% of unique words and 87% of tokens. Of course, two errors per word will quickly overwhelm the ability of the autocomplete to compensate. We will look more at query length later.

		~1400 hard-coded
		tokens			types
	#err	#	%	cumul %	#	%	cumul %
Hindi Wiki	0	58,674	75.80%	75.80%	2,033	37.68%	37.68%
	1	5,813	7.51%	83.31%	860	15.94%	53.62%
	2	3,129	4.04%	87.35%	592	10.97%	64.59%
	3	2,801	3.62%	90.97%	507	9.40%	73.99%

Dakshina train	0	29,494	34.72%	34.72%	14,050	31.78%	31.78%
	1	27,479	32.35%	67.07%	14,731	33.33%	65.11%
	2	15,367	18.09%	85.16%	8,663	19.60%	84.71%
	3	7,304	8.60%	93.76%	4,100	9.28%	93.99%

As a point of verification, I ran the analysis against the smaller Dakshina test data set, and the performance was very similar (actually, a little better), so we have some indication that we didn't over-fit the Dakshina training data.

		~1400 hard-coded
		tokens			types
	#err	#	%	cumul %	#	%	cumul %
Dakshina test	0	2,874	34.64%	34.64%	1,445	32.10%	32.10%
	1	2,863	34.51%	69.15%	1,597	35.47%	67.57%
	2	1,548	18.66%	87.81%	895	19.88%	87.45%
	3	645	7.77%	95.58%	375	8.33%	95.78%

For reference, here are the stats for the query word data and the Dakshina training data with only ~500 hard-coded words.

		~500 hard-coded
		tokens			types
	#err	#	%	cumul %	#	%	cumul %
Hindi Wiki	0	50,793	65.62%	65.62%	1,503	27.86%	27.86%
	1	9,894	12.78%	78.40%	1,130	20.95%	48.81%
	2	5,401	6.98%	85.38%	747	13.85%	62.66%
	3	3,741	4.83%	90.21%	572	10.60%	73.26%

Dakshina train	0	27,791	32.72%	32.72%	13,581	30.72%	30.72%
	1	28,198	33.20%	65.92%	14,915	33.74%	64.46%
	2	15,939	18.76%	84.68%	8,824	19.96%	84.42%
	3	7,549	8.89%	93.57%	4,170	9.43%	93.85%

The Dakshina training data shows a 1% difference for correct types that slowly fades to very small cumulative difference as the number of errors increases to 3. The token data has roughly double the difference—so not much effect.

The impact on the query word data is much bigger—about 10% more of both types and tokens are correct, though again the differences fade as we get up to three errors.

The count differences indicate that by adding 900 lower frequency words (10–25), we got between 400 and 500 more fully correct (the other 400 to 500 were already correct), but that the vast majority of newly correct words were pulled from words that only had 1 or 2 errors.

While we could certainly handle a lot more words in our look-up table, right now it doesn't seem to be worth it, though we could revisit that in the future. Looking at the 173 words with a frequency of 9, only 43 have transliterations. Assuming we also already get half of those correct, we'd add about 22 new types (0.4%) and about 200 new tokens (<0.3%) to the correct piles. The returns would likely be even less in terms of words with transliterations as we go down the frequency list. And, of course, at some point we'd have so much of the list hard-coded that we it wouldn't be a useful test set at all.

Of course, those are surmountable obstacles, finding a lot more transliterations and generating new query word test sets, but I think we have a pretty good baseline for now.

Autocomplete and Fulltext Search Results

Warning: Big Piles of Numbers Incoming! Now with pictures!

It's an imperfect metric, but I think there is insight to be gained from comparing the number of results for a query before and after transliteration, much like looking at fulltext zero-results rate in general. There are false positives, where you search for the wrong thing and get a lot more results (searching for the instead of the intended tse); and there are false negatives, where the query doesn't really match anything in the index (nothing is going to turn research into strategies for the procurement of fluffernutters by bulbous-eyed crocodilians to facilitate interdimensional trade alliances into a useful Wikipedia query).

To limit the impact of fairly useless queries, and to leverage the value of the Dakshina data set, I set out to query individual transliterated Latin words, their human-provided Hindi version, and the second-try transliterated Hindi version. I queried both the Hindi Wikipedia completion suggester autocomplete, where we plan to initially deploy the second-try algorithm, and the Hindi Wikipedia fulltext results, where we may deploy second-try algorithms for "did you mean" suggestions.

Fulltext querying is more forgiving in some ways, since a word just has to appear in an article somewhere, but the completion suggester autocomplete also allows for one or two letters to be different when matching a title.

For the autocomplete results, I asked for up to 50 suggestions, rather than the usual 10, just to get more spread in the results. The graphs below have categories both for "10+ results" to mirror the real-world usage and for "50+ results" to show more granular results.

In addition to using individual words as queries, I ran a sample of 1,000 queries with Latin words in them. These can be pretty awful as title matches, but the results are interesting. There is no Hindi version of these queries, though, since they are raw user queries.

I've also defined a new category of "Combo" for zero-results metrics, which is the combination of the original Latin and the second-try Transliterated Hindi. A combo query only counts as "zero results" if both the Latin and the transliteration get zero results. The combined results are what we would see in the autocomplete suggestions, and for fulltext we could automatically roll over to the transliterated "did you mean" suggestion if the Latin query got zero results.

Dakshina Training Weighted Word Autocomplete

Let's start with words from the Dakshina Training data set as title-match/autocomplete queries.

When weighted by their reported frequency (which, you will recall, seems a little wonky), they represent 84,932 searches. 100% of these words have human Hindi transliterations. 25.46% of the second-try transliterations match the human Hindi transliterations (100% is impossible, since the same Latin words map to different Hindi words in the data set).

Total ZRR

Latin: 64.25%
Hindi: 5.69%
Transliterated: 11.26%
Combo: 10.23%

The Latin words do pretty awfully, with over 60% getting zero results. The Hindi does very well at <6%, but the Transliterated does much better than the Latin at 11%. The combo doesn't improve much because there isn't much room for improvement there.

Transliteration vs Latin ZRR

ZRR↓: 54.03%
ZRR↑: 1.03%

More than half of the time, the Transliteration took a zero-results query Latin word to some results. Only 1% of the time did a Latin word get results and the Transliteration get no results.

Transliteration vs Latin Result Counts

Res↑: 87.26%
Res↓: 1.65%

Similarly, more than 87% of the time, the Transliteration got more results than the Latin word, and vice versa less than 2% of the time.

Autocomplete Second-Try

Shown: 85.32%

Taking into account when the Latin words get more than or fewer than 10 results (including zero results), Transliterated results would be appended to the Latin results more than 85% of the time.

The charts below show the spread of number of autocomplete suggestions across all searches. Latin words mostly get 0 autocomplete suggestions. Hindi and Transliterated words mostly get 10+ (and in fact, 50+) suggestions!

Query Sample Weighted Word Autocomplete

Now let's look at words from actual Hindi Wikipedia queries as title-match/autocomplete queries.

When weighted by their frequency in the sample, they represent 77,405 searches. 89.14% have human Hindi transliterations (mostly from the Dakshina data set, though I sourced more from Hindi Wikipedia as above). 83.79% of the second-try transliterations match the human Hindi transliterations (100% is theoretically possible since every Latin word has only one Hindi transliteration, but the long tail is loooooong, and overfitting is overfitting.)

Total ZRR

Latin: 28.37%
Hindi: 0.68%
Transliterated: 2.17%
Combo: 1.69%

With more realistic frequency information, the word-level zero-results rates are lower all around, but the pattern is the same. Hindi from humans does the best, Latin is pretty bad, the Transliteration is quite good, and the Combo is a tad beter.

Human Hindi Only ZRR

Latin: 26.12%
Hindi: 0.68%
Transliterated: 1.24%
Combo: 0.96%

No Human Hindi ZRR

Latin: 46.80%
Transliterated: 9.78%
Combo: 7.70%

I also pulled out the 89.14% of queries with human Hindi transliterations available, vs the 10.86% without. Rarer words do worse (47% vs 26% ZRR) and their Transliterations do worse, too, but the Transliterations are still much better!

Transliteration vs Latin ZRR

ZRR↓: 26.68%
ZRR↑: 0.48%

Transliteration vs Latin Result Counts

Res↑: 96.71%
Res↓: 1.09%

As before, the Transliterations generally increase results more often than not.

Autocomplete Second-Try

Shown: 75.03%

Second-try searches would be shown a little less often, but still in the large majority of cases, but that's not a surprise. The much lower zero-results rate for these queries should correspond to generally more results, especially since these queries include proper names and other things that are likely to appear in titles or redirects.

The charts below show the spread of number of autocomplete suggestions across all searches. Latin words—including proper names and more terms in English and other Latin-based languages—get 0 and 10+ suggestions in roughly similar proportions, though rarely 50+. Hindi and Transliterated words mostly get 10+ (and in fact, 50+) suggestions, though Transliterated words get no suggestions more often.

Full Query Autocomplete

For our last round of autocomplete suggestions, let's look at the full queries sampled from Hindi Wikipedia. Note that here "Latin" really means "some Latin in the query" and includes a number of mixed-script queries. As mentioned before, the full queries also have a lot more "poor" queries that aren't going to get results no matter what you do for them, let alone match article titles.

There are exactly 1,000 randomly selected queries with Latin characters in them in this sample.

Since we aren't looking at individual words anymore, I also sliced things by number of whitespace-delimited words. In terms of query count:

Word Count Frequency

#words	1	2	3	4	5	6	7	8	9	10+
freq	20.20%	26.60%	17.40%	11.50%	8.80%	5.80%	3.90%	2.00%	1.40%	2.40%

Total ZRR

Latin: 92.40%

#words	1	2	3+
ZRR	68.81%	95.11%	100.00%

Transliterated: 71.60%

#words	1	2	3	4	5+
ZRR	23.76%	56.02%	93.10%	99.13%	100.00%

Combo: 70.90%

#words	1	2	3	4	5+
ZRR	20.79%	55.64%	93.10%	99.13%	100.00%

None of these results are great, but they are interesting. Long fulltext queries generally don't do particularly well as title matches.. no big surprise. But the transliteration helps get within two characters of a match in >20% of cases, which is pretty huge! And the 1-word transliterated zero-results rate (24%) actually looks good!

Transliteration vs Latin ZRR

ZRR↓: 21.50%

#words	1	2	3	4	5+
ZRR↓	48.02%	39.47%	6.90%	0.87%	0.00%

ZRR↑: 0.70%

#words	1	2	3+
ZRR↑	2.97%	0.38%	0.00%

This is similar to before, though the impact on 1- and 2-word queries is much bigger than the overall rate.

Transliteration vs Latin Result Counts

Res↑: 27.70%

#words	1	2	3	4	5+
Res↓	74.75%	42.48%	6.90%	0.87%	0.00%

Res↓: 0.80%

#words	1	2	3+
ZRR↑	3.47%	0.38%	0.00%

A now-expected pattern of increasing results counts, with bigger impacts for 1- and 2-word queries... especially the one-word queries!

Autocomplete Second-Try

Shown: 27.30%

#words	1	2	3	4	5+
Shown	70.79%	43.98%	6.90%	0.87%	0.00%

Second-try results would be shown less often overall because there are no results to show for longer queries (rather than there being too many first-try results for the second-try results to fit). But lots of second-try results for 1- and 2-word queries, which implies that for longer queries it's possible that we'd show a decent result before the searcher typed out 3+ words.

The charts below show the spread of number of autocomplete suggestions across all searches. Notice that the 0 column is chopped off at the top (with actual values shown above) in both charts—otherwise it would be a lot harder to see the differences in the other columns. Zero-results still dominate for both.

Intermission

That was fun! Let's do the whole exercise again, but with fulltext results! These should go a little quicker now that we are familiar with the metrics.

Note that while autocomplete can be more forgiving in terms of getting the right match, fulltext search has some wiggle room, too. The words don't have to be next to each other or in the right order. For Hindi, there is some stemming and other analysis that can possibly make some errors irrelevant. And there is always the possibility that mis-transliterating a word as a very common word works out almost the same as deleting that word. (In English, for example, improperly "correcting" tse to the will get you lots of results, and if there are other search terms it is effectively ignoring tse.)

On with the show....

Dakshina Training Weighted Word Fulltext

As before, with the wonky frequency data, this represents 84,932 searches, all of which have human Hindi transliterations available. 25.46% of the second-try transliterations match the human Hindi transliterations.

Total ZRR

Latin: 46.54%
Hindi: 0.04%
Transliterated: 29.33%
Combo: 15.85%

The baseline ZRR for Latin is lower since you don't have to have an article title or redirect that starts with a given word to get a result. Same for human Hindi transliterations.

The second-try transliteration doesn't do as well here, presumably because the transliterations are off a little. The Combo ZRR is a decent improvement over the Transliterated ZRR. Overall, still a good improvement.

Transliteration vs Latin ZRR

ZRR↓: 30.69%
ZRR↑: 13.48%

Transliteration vs Latin Result Counts

Res↑: 61.04%
Res↓: 22.55%

A much less lopsided improvement to the total number of results, which tells me that the fuzziness of the autocomplete was helping shore up the weaknesses of the transliteration, but still a clear improvement.

Since we would most likely implement the second-try algorithm as "did-you-mean" suggestions for fulltext, the fact that the transliterations are sometimes worse is okay—I've been advocating for more strict filtering of suggestions anyway.

The charts below show the spread of number of fulltext results across all searches. It's not clear that the differences between 500 results and 50,000 results is meaningful to a searcher, but the decrease in 0–10 results (especially 0–3) is, to me, a mild indicator of quality of the transliteration. We clearly aren't generating complete gibberish, or the ZRR would not improve.

Query Sample Weighted Word Fulltext

As before, with more realistic frequency data, this represents 77,405 searches, 89.14% of which have human Hindi transliterations available. 83.79% of the second-try transliterations match the human Hindi transliterations.

Total ZRR

Latin: 3.39%
Hindi: 0.19%
Transliterated: 6.08%
Combo: 0.72%

Overall, the ZRR numbers are much, much lower, and the Transliterated ZRR is actually higher than for the original Latin. However, the Combo number tells the real story—together, the Latin and Transliterated queries have amazingly low ZRR at the word-level.

Human Hindi Only ZRR

Latin: 2.01%
Hindi: 0.19%
Transliterated: 3.32%
Combo: 0.20%

No Human Hindi ZRR

Latin: 14.74%
Transliterated: 28.68%
Combo: 5.00%

Words with human-provided Hindi transliterations have very low ZRR, vs the long tail of less common and non-transliterated words. The Combo numbers are really good in both cases, indicating that the Transliterations are shoring up the Latin shortcomings, and vice versa.

Transliteration vs Latin ZRR

ZRR↓: 2.67%
ZRR↑: 5.36%

This is related to the low Combo ZRR score above. Sometimes the Latin is pretty good when the Transliteration is not, and vice versa...

Transliteration vs Latin Result Counts

Res↑: 74.14%
Res↓: 25.01%

... But overall the transliteration does get more results.

The charts below show the spread of number of fulltext results across all searches. Transliterated searches have a higher ZRR, but also have more queries with a lot of results, and a distribution more similar to Hindi.

Full Query Fulltext

Now that we get to full queries (~80% of which are 2+ words), the ZRR goes up, and longer queries do worse....

Total ZRR

Latin: 70.40%

#words	1	2	3	4	5	6	7+
ZRR	50.50%	58.27%	64.94%	83.48%	96.59%	96.55%	100.00%

Transliterated: 65.10%

#words	1	2	3	4	5	6	7	8	9	10+
ZRR	50.99%	56.77%	63.22%	72.17%	76.14%	82.76%	84.62%	95.00%	100.00%	95.83%

Combo: 52.80%

#words	1	2	3	4	5	6	7	8	9	10+
ZRR	36.63%	39.85%	44.25%	60.87%	73.86%	81.03%	84.62%	95.00%	100.00%	95.83%

... But the Combo ZRR really shines, as it is more than 12% lower than the Transliterated ZRR, so, again, the Latin and Transliterated queries complement each other.

Transliteration vs Latin ZRR

ZRR↓: 17.60%

#words	1	2	3	4	5	6	7	8	9	10+
ZRR↓	13.86%	18.42%	20.69%	22.61%	22.73%	15.52%	15.38%	5.00%	0.00%	4.17%

ZRR↑: 12.30%

#words	1	2	3	4	5	6	7+
ZRR↓	14.36%	16.92%	18.97%	11.30%	2.27%	1.72%	0.00%

Transliteration decreases ZRR overall, but increases it sometimes, too. Rates are surprisingly similar for 1-to-3-word queries, but a net decrease for 4+ word queries!

Transliteration vs Latin Result Counts

Res↑: 29.70%

#words	1	2	3	4	5	6	7	8	9	10+
Res↓	40.59%	33.83%	33.33%	26.09%	22.73%	15.52%	15.38%	5.00%	0.00%	4.17%

Res↓: 16.90%

#words	1	2	3	4	5	6	7+
Res↓	21.78%	25.19%	22.41%	12.17%	3.41%	3.45%	0.00%

Transliteration generally gets more results for these full queries, regardless of query length.

The charts below show the spread of number of fulltext results across all searches. Notice that the 0 column is chopped off at the top (with actual values shown above) in both charts—otherwise it would be a lot harder to see the differences in the other columns. Zero-results still dominate for both.

As above, Transliterated queries have more results in general.

PHP Implementation & Speed Check

The initial implementation of the second-try transliteration algorithm was all regex substitutions done individually, to minimize complexity and maximize flexibility for me in terms of re-ordering, grouping, and modifying elements of the algorithm.

Working on the final PHP implementation required some algorithmic refactoring, since individual substitutions using the Full Power of Regular Expressions™ is not the fastest code. I regrouped elements that were mostly independent into simple substitutions and context-dependent regex substitutions. The simple substitutions are done greedily (longest first) in batches and cannot overlap. The regex substitutions are done in order, but also in batches that are presumably faster than individual substitutions. A few simple substitutions remained mixed in with the regexes when ordering was important, and some groups of simple substitution were broken into multiple batches to maintain ordering and allow needed overlaps. ("Baseline")

I also tried doing everything as one big regex batch, meaning everything is done in order. ("Easy")

I also tried breaking out the simple substitutions embedded in the regex lists, which meant more function calls in the innermost loop, but also less regexness overall. ("Complex")

Overall, the Complex approach was 1.5–2% faster than the Baseline approach, so breaking things apart did speed them up a little, but the decreased readability of the code (which is already not great) wasn't worth it for now.

The Easy approach was 35–40% slower than the Baseline, which is too high of a price to pay!

I settled on the Baseline approach, with the hard-coded list of ~1400 words and five "phases" of substitutions. Whew!

Conclusion

It's far from perfect, but I'm happy with how well the transliteration has turned out, even with (especially with?) the hack for hard-coding the most common words.

While I wouldn't want to try to extrapolate too precisely from the data here to the expected quality of transliteration in production, it's clearly going to help improve the zero-results rate for autocomplete suggestions—and I think the fuzziness of the completion suggester will compensate for many of its failings. (Similarly for a results count filter on "did you mean" suggestions if/when we extend second-try to fulltext.)