View source for Extension talk:External Data

{{External Data navigation}}
{{Archive box|
* [[Extension talk:External Data/Archive 2009|2009]]
* [[Extension talk:External Data/Archive 2010|2010]]
* [[Extension talk:External Data/Archive 2011|2011]]
* [[Extension talk:External Data/Archive 2012|2012]]
* [[Extension talk:External Data/Archive 2013|2013]]
* [[Extension talk:External Data/Archive 2014|2014]]
* [[Extension talk:External Data/Archive 2015|2015]]
* [[Extension talk:External Data/Archive 2016|2016]]
* [[Extension talk:External Data/Archive 2017 to 2018|2017 to 2018]]
* [[Extension talk:External Data/Archive 2019 to 2020|2019 to 2020]]
* [[Extension talk:External Data/Archive 2021 to 2022|2021 to 2022]]
* [[Extension talk:External Data/Archive 2023|2023]]
}}


== Iterating over one variable while keeping another one in the same state ==
I retrieve data from JSON files for a certain Host. Among that data is a list (1 item or more) mac addresses and the host name (1 item for sure).

I then try to iterate over the list of mac addresses to call a template that uses the mac address to look up the ip addresses on that interface.

However i need to supply the host name to the template too, so it knows what json file to look up the values in. I can’t get this to work properly.

This is an annotated example of the calls I’ve tried:

<pre><nowiki>
{{#for_external_table:|
 ### this produces and empty hostname, if macaddresses has more than one item
 {{Interface|hostname={{{hostname}}}|hwaddress={{{macaddresses}}}}}

 ### this also produces an empty hostname, if macadresses has more than one item but will also fail if macaddresses has only one item, because external value will be evaluated to an empty string on the second iteration (bug?)
 {{Interface|hostname={{#external_value:hostname}}|hwaddress={{{macaddresses}}}}}

 ### this also doesn’t work, because the external value will be empty the same moment that name is empty, because they’re both iterated over
 {{Interface|hostname={{{name|{{#external_value:name}}}}}|hwaddress={{{macaddresses|}}}}}
}}
</nowiki></pre>

Is there a way to achieve this or will this have to be implemented first? 

I could imagine something like <nowiki>{{{hostname.keep}}}</nowiki> to mark those variables that don’t get iterated or just returning the same value over and over if the iteration variable isn’t an iterable

:That's a tricky one. You may have to use the [[Extension:Variables|Variables]] extension, to set some variable to the value of #external_value, and then use that variable within #for_external_table. Although maybe the best solution is to move all this code into a Lua module, using [[Extension:Scribunto|Scribunto]] - it will give you much more flexibility. [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 12:59, 1 May 2024 (UTC)

:How to deal with the data columns of different height could be an interesting topic for discussion some time ago. But now, the obvious solution for this and some other issues is Lua. ''Variables'' should not now be used at all, for the parsing order is now uncertain.<br /> [[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 08:17, 13 May 2024 (UTC)

== Using multiple JOIN ON parameters in ED ==
Hi, I am having trouble getting a query to work

I have translated my problem to [https://docs.rfam.org/en/latest/database.html the public RFAM-Database] so you can reproduce the problem.

This is the query that i'm aiming for:

<syntaxhighlight lang="sql">
SELECT family.description,clan.description FROM clan
JOIN clan_membership on clan.clan_acc = clan_membership.clan_acc
JOIN family ON family.rfam_acc = clan_membership.rfam_acc;
</syntaxhighlight>

It gives me 544 rows ( i copied the Rfam-DB a while ago, so there might be more rows now)

This is the <code>get_db_data</code> line that i have on a page:

<pre><nowiki>
{{#get_db_data: db = Rfam
 |join on=clan_membership.clan_acc = clan.clan_acc
 |join on=family.rfam_acc = clan_membership.rfam_acc
 |from=clan
 |data=clan_description=clan.description,family_description=family.description
}}
</nowiki></pre>

i have also tried this as data line: <code>|data=clan_description=clan.description AS cdesc,family_description=family.description AS fdesc</code>
the <code>AS</code> makes it into the query, i checked. but it doesn't help

I then call this to list the results: 

<pre><nowiki>
{{#for_external_table:|
;{{{clan_description}}}
 : {{{family_description}}}
}}
</nowiki></pre>

The list is empty.

The SQL that is being produced by ED looks like this:

<syntaxhighlight lang="sql">
SELECT clan.description,family.description FROM `clan`,family`
JOIN `clan_membership` ON ((family.rfam_acc = clan_membership.rfam_acc))
</syntaxhighlight>

So this query only has one <code>JOIN</code> and the "((" "))" which i have never seen before (i'm not a sql pro though, so that might not mean anything)
This query selects clan and family which, again, i'm not sure if it’s right to explicitly state that in a join statement.
The query returns two columns called "description" unless i add an alias for the columns in the {{para|data}} param, but that  doesn’t make it work.
running this query on the DB directly gives 66430 rows ( with a lot of duplicates) but limiting to a few rows with <code>LIMIT=25</code> doesn’t make it work either.

So, how do i make this work?

Edit: This is the current Code i have, after discussing with Yaron Koren

<pre><nowiki>
{{#get_db_data: db = Rfam
 |join on=clan.clan_acc=clan_membership.clan_acc,join on=family.rfam_acc = clan_membership.rfam_acc
 |from=clan
 |data=clan_description=clan.description,family_description=family.description <!-- empty result -->
}}
</nowiki></pre>

:I think it would just need to be <code>join on=clan_membership.clan_acc = clan.clan_acc, family.rfam_acc = clan_membership.rfam_acc</code>. [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 19:10, 29 April 2024 (UTC)

:Alright, that gives me the correct query in the logs, but no results appear in for_external_table [[Special:Contributions/24.134.95.253|24.134.95.253]] 20:50, 29 April 2024 (UTC)
::some values appear, when i remove the table names in the data:-line, but then i will get duplicates, because there is a clan_acc row in each table [[Special:Contributions/24.134.95.253|24.134.95.253]] 21:00, 29 April 2024 (UTC)
:::If you run that correct query directly in the database, does it return results? [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 01:44, 30 April 2024 (UTC)
::::yes, running the query in the DB directly returns two columns called "description" and the corresponding values, so the joins do work, but now i cannot get the data into mediawiki because of the identical column names. [[Special:Contributions/77.22.6.114|77.22.6.114]] 09:14, 30 April 2024 (UTC)
:::::Can't you do "clan_description=clan.description, family_description=family.description"? [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 13:42, 30 April 2024 (UTC)
::::::i mean yes, that’s what i had first, but it doesn’t show any data. [[Special:Contributions/77.22.6.114|77.22.6.114]] 19:39, 30 April 2024 (UTC)
:::::::Okay, good news! We just checked in a fix, [https://phabricator.wikimedia.org/rEXDAc5dd614027c76bbffb176162292debddb9e6a262 here], so that now the "AS" keyword is handled correctly, which is what was needed here. So now if you do "|data=clan_description=clan.description AS clandesc, family_description=family.description AS familydesc", it should (hopefully) work. Note that the "AS" aliases don't matter, as long as they exist and they're different from one another. [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 16:20, 3 May 2024 (UTC)

== Deprecated : strtolower(): Passing null to parameter #1 ($string) of type string is deprecated ==

; Setup
* MediaWiki 1.39.6 (0e03068) 2024-03-11T16:26:30
* PHP 8.1.2-1ubuntu2.14 (apache2handler)
* MariaDB 10.6.16-MariaDB-0ubuntu0.22.04.1
* External Data 3.4-alpha (20a6b7f) 2024-03-15T09:43:16

; Issue
Deprecated : strtolower(): Passing null to parameter #1 ($string) of type string is deprecated in /../w/extensions/ExternalData/includes/EDParsesParams.php on line 117

-- [[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 17:49, 15 March 2024 (UTC)

:I need the wikicode of the parser function you call and the relevant <syntaxhighlight lang="php" inline>$wgExternalDataSource[…]</syntaxhighlight> (with sensitive information censored out). Also, when did the warning appear: after upgrading MediaWiki, External Data, or adding a new data source or parser function call?<br /> [[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 03:42, 16 March 2024 (UTC)
::The extensions <code>$wgExternalDataSource</code> configuration parameter is at its default. On pages with this issue we are using the extensions with calls like this one: <syntaxhighlight lang=wikitext>
{{#display_external_table:
   source=https://example.org/w/images/9/9c/Export_slice_123.csv
  |format=CSV with header
  |header lines=1
  |start line=3
  |end line=10
  |data= mynr=recordnumber, priref=orig_performance_ref, pc=productcode
  |template=Test Template
}}
|}
</syntaxhighlight>
::I cannot tell it this was an issue before the upgrade since I did not enable logging for the wiki before.
::;Backtrace
::<syntaxhighlight lang=text>
[4b34db6f537fa5e035bcd426] /wiki/Test_Page   PHP Deprecated: strtolower(): Passing null to parameter #1 ($string) of type string is deprecated
#0 [internal function]: MWExceptionHandler::handleError()
#1 /../w/extensions/ExternalData/includes/EDParsesParams.php(117): strtolower()
#2 /../w/extensions/ExternalData/includes/EDParsesParams.php(55): EDConnectorBase::paramsFit()
#3 /../w/extensions/ExternalData/includes/connectors/EDConnectorBase.php(234): EDConnectorBase::getMatch()
#4 /../w/extensions/ExternalData/includes/connectors/EDConnectorBase.php(248): EDConnectorBase::getConnectorClass()
#5 /../w/extensions/ExternalData/includes/EDParserFunctions.php(88): EDConnectorBase::getConnector()
#6 /../w/extensions/ExternalData/includes/EDParserFunctions.php(115): EDParserFunctions::get()
#7 /../w/extensions/ExternalData/includes/EDParserFunctions.php(204): EDParserFunctions::fetch()
#8 /../w/extensions/ExternalData/includes/EDParserFunctions.php(428): EDParserFunctions::emulateGetExternalData()
#9 /../w/extensions/ExternalData/includes/EDParserFunctions.php(487): EDParserFunctions::actuallyDisplayExternalTable()
#10 /../w/includes/parser/Parser.php(3439): EDParserFunctions::doDisplayExternalTable()
#11 /../w/includes/parser/Parser.php(3124): Parser->callParserFunction()
#12 /../w/includes/parser/PPFrame_Hash.php(275): Parser->braceSubstitution()
#13 /../w/includes/parser/Parser.php(2953): PPFrame_Hash->expand()
#14 /../w/includes/parser/Parser.php(1609): Parser->replaceVariables()
#15 /../w/includes/parser/Parser.php(723): Parser->internalParse()
#16 /../w/includes/content/WikitextContentHandler.php(301): Parser->parse()
#17 /../w/includes/content/ContentHandler.php(1721): WikitextContentHandler->fillParserOutput()
#18 /../w/includes/content/Renderer/ContentRenderer.php(47): ContentHandler->getParserOutput()
#19 /../w/includes/Revision/RenderedRevision.php(266): MediaWiki\Content\Renderer\ContentRenderer->getParserOutput()
#20 /../w/includes/Revision/RenderedRevision.php(237): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
#21 /../w/includes/Revision/RevisionRenderer.php(221): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#22 /../w/includes/Revision/RevisionRenderer.php(158): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#23 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}()
#24 /../w/includes/Revision/RenderedRevision.php(199): call_user_func()
#25 /../w/includes/poolcounter/PoolWorkArticleView.php(91): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#26 /../w/includes/poolcounter/PoolWorkArticleViewCurrent.php(97): PoolWorkArticleView->renderRevision()
#27 /../w/includes/poolcounter/PoolCounterWork.php(162): PoolWorkArticleViewCurrent->doWork()
#28 /../w/includes/page/ParserOutputAccess.php(299): PoolCounterWork->execute()
#29 /../w/includes/page/Article.php(714): MediaWiki\Page\ParserOutputAccess->getParserOutput()
#30 /../w/includes/page/Article.php(528): Article->generateContentOutput()
#31 /../w/includes/actions/ViewAction.php(78): Article->view()
#32 /../w/includes/MediaWiki.php(542): ViewAction->show()
#33 /../w/includes/MediaWiki.php(322): MediaWiki->performAction()
#34 /../w/includes/MediaWiki.php(904): MediaWiki->performRequest()
#35 /../w/includes/MediaWiki.php(562): MediaWiki->main()
#36 /../w/index.php(50): MediaWiki->run()
#37 /../w/index.php(46): wfIndexMain()
#38 {main}
</syntaxhighlight>
:: [[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 17:08, 18 March 2024 (UTC)

::* I have submitted a [https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ExternalData/+/1012461 patch]. Please test it, for I was unable to reproduce the issue fully.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 02:44, 19 March 2024 (UTC)
::*:Looks like you patch fixed the issue. Thanks a lot. [[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 17:17, 19 March 2024 (UTC)

== Warning : Cannot modify header information - headers already sent by  ==

; Setup
* MediaWiki 1.39.6 (0e03068) 2024-03-11T16:26:30
* PHP 8.1.2-1ubuntu2.14 (apache2handler)
* MariaDB 10.6.16-MariaDB-0ubuntu0.22.04.1
* External Data 3.4-alpha (20a6b7f) 2024-03-15T09:43:16

; Issue
Warning : Cannot modify header information - headers already sent by (output started at /var/www/html/w/extensions/ExternalData/includes/EDParsesParams.php:117) in
/../w/includes/WebResponse.php on line 75

-- [[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 17:51, 15 March 2024 (UTC)

:I assume this error message is just due to the one you reported above. [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 18:08, 15 March 2024 (UTC)
::I cannot tell. The wiki is using Tweeki and the respective pages look really messy. [[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 18:11, 15 March 2024 (UTC)
::It could be connected though the issue does not appear everytime I get the above reported deprecation issue. Here is a backtrace:
::<syntaxhighlight lang=text>
#0 [internal function]: MWExceptionHandler::handleError()
#1 /../w/includes/WebResponse.php(75): header()
#2 /../w/includes/OutputPage.php(2732): WebResponse->header()
#3 /../w/includes/OutputPage.php(2891): OutputPage->sendCacheControl()
#4 /../w/includes/MediaWiki.php(922): OutputPage->output()
#5 /../w/includes/MediaWiki.php(562): MediaWiki->main()
#6 /../w/index.php(50): MediaWiki->run()
#7 /../w/index.php(46): wfIndexMain()
#8 {main}
</syntaxhighlight>
::Cheers --[[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 17:17, 18 March 2024 (UTC)

== Accessing identically named subkeys ==

Hi! I'm trying to use External Data to get data from an API that returns(part of) it's data as <code>{"midweekMeetingTime":{"weekday":2,"time":"18:30:00"},"weekendMeetingTime":{"weekday":7,"time":"12:00:00"}}</code>, where the subkey names are the same in different arrays. Is there a way to specify which value to set a variable to (ie <code>weekend_day=weekendMeetingTime.time,midweek_day=midweekMeetingTime.time</code> without using a Lua module, as while I believe it would work in Lua, I'd prefer to keep it in wikitext, as most other parameters work and making a module for just this seems like a waste. Thanks all in advanced! [[User:PixDeVl|PixDeVl]] ([[User talk:PixDeVl|talk]]) 20:37, 1 May 2024 (UTC)

:It looks like the "use jsonpath" parameter would be helpful for this case; see [[Extension:External_Data/Parsing_data#JSON_and_YAML|here]]. [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 21:03, 1 May 2024 (UTC)
::Oh, thanks! Would you happen to know any existing instances of the extension using this to reference? There seems to be some weirdness going on with the parser showing the expression working as intended(ie <code>$..midweekMeetingTime.time</code>, the double dot is there since the API returns the array in a list, <code>[{...}]</code>) and the extension giving an undefined error, so I want to refer it to the syntax used by others, although I'll be the first to admit I may be simply be misunderstanding or writing something. Thanks again for the tip and making such a useful extension! [[User:PixDeVl|PixDeVl]] ([[User talk:PixDeVl|talk]]) 22:18, 1 May 2024 (UTC)
:::Sure - all three example queries [https://discoursedb.org/wiki/Wikidata_querying_demo here] use JSONPath. I hope this helps uncover the problem... [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 13:06, 2 May 2024 (UTC)

== #store_external_table rendering raw ==

; Setup
* MediaWiki 1.39.7 (2ba7e95) 2024-05-13T11:45:16
* PHP 8.1.2-1ubuntu2.17 (apache2handler)
* MariaDB 10.6.16-MariaDB-0ubuntu0.22.04.1
* External Data 3.4-alpha (c23dc0d) 2024-05-18T10:11:41

; Issue
The <code>#store_external_table</code> parser function is rendering raw, i.e., instead of not visibly being shown on a page the user sees <code><nowiki>{{#store_external_table:Is fruit in |Has name={{{name}}} |Has color={{{color}}} |Has shape={{{shape}}} }}</nowiki></code>. Despite this the parser function still does it's job, i.e., stores the subobjects holding the annotations. [[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 18:25, 5 June 2024 (UTC)

:Yes, the #store_external_table parser function was removed from External Data - I need to release version 3.4, to make it an official removal. I don't think it's storing data - my guess is that that SMW data you're seeing was already there beforehand. [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 19:04, 5 June 2024 (UTC)
::Thanks for the info. How is information stored for SMW in new releases, i.e., what replaced this parser function? Is <code>#get_web_data</code> doing this now or do I need to create a template that maps the data to the properties? [[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 19:09, 5 June 2024 (UTC)
:::You now need to do it via a template, yes. [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 19:18, 5 June 2024 (UTC)
::::Ah, ok. Thank you for confirming. It looks like one needs to use <code>#display_external_table</code> for this. [[User:Kghbln|&#91;&#91;kgh&#93;&#93;]] ([[User talk:Kghbln|talk]]) 19:20, 5 June 2024 (UTC)
::::* Lua will be faster and can be made more robust. Both External Data nad SMW have Lua APIs.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 14:06, 16 June 2024 (UTC)

== Suggestion: add cookies to getWebData in Lua ==

Hi. I have played around with '''getWebData''', especially to see the potential of calling my private wiki's API. I already use '''getDbData''', but I think both can be complimentary.

The problem is that anonymous users cannot read my private wiki (they have to create an account) and '''getWebData''' only allows for anonymous connections. I have tried to login to my wiki using '''getWebData''' the same way my custom bot does, but it doesn't work. I get an error ''"Unable to continue login. Your session most likely timed out."'' and I reckon this is because the session ID is passed as a cookie which is ignored here.

Furthermore, once you are logged in, you have a cookie to avoid logging in every time. That could also work with other sites.

So my suggestion/request is a way to handle cookies with '''getWebData''' in Lua (I don't think it would be useful in the template version). I propose the following:
* The Lua '''getWebData''' would return three values instead of two: ''result'', ''error'' and ''cookies'' (so that it doesn't break existing code which take only ''result'' and ''error'')
* The Lua '''getWebData''' would accept a new field in its table argument named "cookies" as a table of cookies which would be passed along with the HTTP request.

That way, we could get cookies from an HTTP request and send them to another request, if necessary, without breaking any existing code.

Thanks. [[User:Steff-X|Steff-X]] ([[User talk:Steff-X|talk]]) 13:21, 16 June 2024 (UTC)
* You could try to declare a [[Extension:External Data/Web pages#ExternalDataBeforeWebCall hook|ExternalDataBeforeWebCall hook]] to get somehow your cookie and put it into <syntaxhighlight lang="php" inline>$options['headers']['Cookie'] .= ";your_cookie=$cookie";</syntaxhighlight>.<br />Note to self: add or remember a less global way to add callbacks to External Data sources.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 14:18, 16 June 2024 (UTC)
*:Thanks Alex, that worked.
*:For other people interested, here's how to do it:
*:* In Firefox or Chrome, connect to the wiki you're interested in
*:* Display the cookies. The method depends on your browser.
*:* Copy the name and value of the cookies ending with UserID, UserName and Token
*:* In ''LocalSettings.php'', add the following code, substituting the cookies' name and value by yours (don't miss the '''.=''' (dot-equal) sign and the final semi-colon):
*:<syntaxhighlight lang="php">
*:$wgHooks['ExternalDataBeforeWebCall'][] =
*:function ( string $method, string &$url, array &$options, array &$errors ): bool {
*:	if ( $url === 'https://your.wiki/api.php' ) ) {
*:        $options['headers']['Cookie'] .= ";mediawikiUserID=5;mediawikiUserName=myUserName;mediawikiToken=(censored)";
*:		return true;
*:	}
*:};</syntaxhighlight>
*:Unfortunately, anything more complicated than that is beyond my knowledge in PHP. That is why I still think that an easy way to handle cookies would be beneficial.
*:Thanks again for the tip. [[User:Steff-X|Steff-X]] ([[User talk:Steff-X|talk]]) 12:11, 18 June 2024 (UTC)
** If you get your cookie outside of MediaWiki, your method is overcomplicated. Just set <syntaxhighlight lang="php" inline>$wgExternalDataSources['https://your.wiki/api.php']['options']['headers']['Cookie'] = 'mediawikiUserID=5;mediawikiUserName=myUserName;mediawikiToken=(censored)';</syntaxhighlight><br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 12:56, 18 June 2024 (UTC)
**:Well I can't get it to work.
**:I always get error: ''Error sending API request: SyntaxError: JSON Parse error: Unrecognized token '<'''
**:And I don't know where it comes from because the error shows up even if I disable "format" = "json" in the request and everything related to json in my code. [[User:Steff-X|Steff-X]] ([[User talk:Steff-X|talk]]) 14:32, 18 June 2024 (UTC)

== This extension documentation really lacks some examples in Lua ==

Hi everyone. This extension is great in its capabilities but I think it really lacks some code examples in Lua/Scribunto.

The usage notice says that ''"there is one-to-one correspondence between parser functions retrieving data and Lua functions evident from their names"''. OK cool, but the syntax is entirely different between the two, especially when you just start using Scribunto/Lua because your wiki template keeps failing!

I think the documentation should have some advanced examples in Lua/Scribunto (to show the power of Lua vs Templates) and I'm ready to participate. [[User:Steff-X|Steff-X]] ([[User talk:Steff-X|talk]]) 13:47, 16 June 2024 (UTC)
* A couple of examples, perhaps, too advanced:
** [https://nkvd.memo.ru/index.php/Module:Chrono Module:Chrono] used to show [https://nkvd.memo.ru/index.php/%D0%9D%D0%9A%D0%92%D0%94:%D0%A5%D1%80%D0%BE%D0%BD%D0%BE%D0%BB%D0%BE%D0%B3%D0%B8%D1%8F this]. Funnily, it drills into Cargo tables;
** [https://traditio.wiki/Module:Tzdata Module:Tzdata], which accesses <code>timedatectl</code> and is used to parse data / time strings;
** [https://strategywiki.org/wiki/Module:WikiList Module:WikiList] that parses a [https://gamewikis.github.io/WikiLookup/WikiLookup.json JSON] from GitHub;
** [https://phabricator.wikimedia.org/T336457#8843530 My own example] from Phabricator;
** [https://traditio.wiki/Module:External_data/new Module:External_data/new] -- this module assembles external data split over several web pages with links to each oher, e.g. a long list, from only one URL, linking the first page (or any page, from which there is a path to other pages).
: [[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 14:34, 16 June 2024 (UTC)

== Issue with caching of data ==

Hi there, 

Great extension, we use it a lot. We have somewhat recently adopted LUA to fetch data from another private wiki onto a public wiki, via a whitelisted Special:Ask page and the use of the <code><nowiki>mw.ext.externalData.getExternalData</nowiki></code> function. It's been working well until recently. We realized that any changes on the private wiki were taking quite a lot of time to refresh on the public wiki (many hours, to a few days). We've tried everything we could think of to reduce any kind of caching to 0, but no luck. And it seems to be compounded. ExtData has some weird invisible level of caching and then LUA does too, it seems. We tried:
* <code><nowiki>wgExternalDataSources['*']['min cache seconds'] = 0</nowiki></code>
* We've tried the standalone, non-lua function: <code>#external_value</code> with <code><nowiki>cache seconds=0</nowiki></code>
* In LUA we first had <code>mw.LoadData()</code> with a big call to <code><nowiki>mw.ext.externalData.getExternalData</nowiki></code> to bring the data once to the page and then parse into a large template.
* Then in LUA we tried individual little calls to <code><nowiki>mw.ext.externalData.getExternalData</nowiki></code> without using <code><nowiki>mw.LoadData()</nowiki></code>.
* We've turned off all kinds of cache everywhere we could think of, including client side, etc.

No luck! Any ideas?

[[User:JeremiPlazas|Jeremi Plazas]] ([[User talk:JeremiPlazas|talk]]) 16:03, 22 July 2024 (UTC)
*# Setting cache period to zero would be a bad idea, because it will make both wikis vulnerable to DOS attacks. Set, at least, to one minute, or more if it takes Special:Ask more time to run;
*# Whether or not you use Lua or <syntaxhighlight lang="lua" inline>mw.LoadData()</syntaxhighlight> is not relevant.
*# I suggest that you set <syntaxhighlight lang="php" inline>wgExternalDataSources['(Full URL of Special:Ask with query; or that wiki's hostname, e.g., example.org)']['min cache seconds'] = 60;</syntaxhighlight>. That should be enough.
*# The only additional caching level that External Data introduces is the table <code>ed_url_cache</code>. It can contain cache entries with expitation time in the future, set when the cache expiration period was longer. You may want to have them removed.
: [[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 02:01, 23 July 2024 (UTC)
::Thanks for the reply! We tried your suggestions but our problem persists. We set  wgExternalDataSources as per your example and also truncated the ed_url_cache table. Still we have pages with data that is stale of a couple of days. We can't quite figure out at what level this is happening. DB? [[User:JeremiPlazas|Jeremi Plazas]] ([[User talk:JeremiPlazas|talk]]) 09:39, 23 July 2024 (UTC)
::Our current config is:
::wfLoadExtension( 'ExternalData' );
::$wgExternalDataSources['https://research.tsadra.org/index.php']['min cache seconds'] = 5;
::$wgExternalDataSources['*']['always allow stale cache'] = false;
::But even with 5 seconds, the entry in ed_url_cache does not get updated.
::[[User:JeremiPlazas|Jeremi Plazas]] ([[User talk:JeremiPlazas|talk]]) 10:29, 23 July 2024 (UTC)
:::This may be an issue of MediaWiki cache. To make sure, open <code>https://your-wiki.org/wiki/Page_with_ED?action=purge</code>.<br /> [[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 11:50, 23 July 2024 (UTC)
::Also, the key to <code>$wgExternalDataSources</code> should be either the full precise url of the data source, including what goes after <code>?</code>, or the domain name (e.g., <code>research.tsadra.org</code>) or the second-level domain (<code>tsadra.org</code>) or asterisk.<br /> [[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 11:53, 23 July 2024 (UTC)
::Thanks, yes we tried purging the page many times. Also we tried all the different URL types, full, just domain, nothing does it. It looks to be a deeper lever of caching going on possibly the ed_url_cache table not updating...
::So we have a little test page that we whitelisted on this wiki site we're working on to illustrate:
::https://bca.tsadra.org/index.php/Test_external_data
::If it helps.
::[[User:JeremiPlazas|Jeremi Plazas]] ([[User talk:JeremiPlazas|talk]]) 14:18, 23 July 2024 (UTC)
:::@[[User:Alex Mashin|Alex Mashin]] - Any new thoughts on this? Thanks! [[User:JeremiPlazas|Jeremi Plazas]] ([[User talk:JeremiPlazas|talk]]) 15:35, 1 August 2024 (UTC)
:::* None, so far. You could try debugging.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 15:53, 1 August 2024 (UTC)
:::*:Ok thanks. [[User:JeremiPlazas|Jeremi Plazas]] ([[User talk:JeremiPlazas|talk]]) 10:20, 2 August 2024 (UTC)

== working with self signed certificates ==

is there any way to allow self signed certificates while querying webpages?

I get the error<pre>
Fehler beim Abruf der URL: SSL certificate problem: self-signed certificate in certificate chain
</pre>[[Special:Contributions/130.149.196.182|130.149.196.182]] 15:01, 30 July 2024 (UTC)
* Have you tried explicitly adding that self-signed certificate to trusted on OS level?<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 07:53, 6 August 2024 (UTC)
* You could also try adding <syntaxhighlight lang="php" inline>$wgExternalDataSources['URL or domain of the web page with a self-signed certificate']['options']['sslVerifyCert'] = false;</syntaxhighlight>.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 02:36, 21 August 2024 (UTC)

== Bug when data contains "umlauts"? ==

mediawiki 1.39, newest source of Extension. 

I'm using<pre>
{{#get_db_data:
   db=db
   |from=table
   |where=wiki_pageid=43
   |data=Titel=title
}}
</pre>title contains <pre>
AA - some Text überprüfen 
</pre>error is:<pre>
[a718c169286b5857d6612120] /index.php/Test_ExternalData_error ValueError: mb_convert_encoding(): Argument #3 ($from_encoding) must specify at least one encoding

Backtrace:

from /var/www/html/extensions/ExternalData/includes/connectors/EDConnectorDb.php(159)
#0 /var/www/html/extensions/ExternalData/includes/connectors/EDConnectorDb.php(159): mb_convert_encoding(string, string, string)
#1 /var/www/html/extensions/ExternalData/includes/connectors/EDConnectorDb.php(140): EDConnectorDb::processField(string)
#2 /var/www/html/extensions/ExternalData/includes/connectors/EDConnectorDb.php(102): EDConnectorDb->processRows(Wikimedia\Rdbms\MysqliResultWrapper, array)
#3 /var/www/html/extensions/ExternalData/includes/EDParserFunctions.php(90): EDConnectorDb->run()
</pre>probably a problem with the "umlauts"? [[Special:Contributions/91.64.64.183|91.64.64.183]] 09:25, 1 August 2024 (UTC)

:I have a feeling that your External Data is not up to date.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 15:57, 1 August 2024 (UTC)
::a
<pre>
git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/ExternalData
</pre>
::gives me the exact same error. [[Special:Contributions/91.64.64.183|91.64.64.183]] 11:10, 2 August 2024 (UTC)
::* [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ExternalData/+/af8809651b16b72010574fc2d01694394b2e8563%5E%21/#F0 Corrected] on July 27th, 2023. Checkout to <code>master</code>.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 08:53, 8 August 2024 (UTC)
::*:now I get a different error:
::*:<code>preg_match_all error 4: Malformed UTF-8 characters, possibly incorrectly encoded</code> [[Special:Contributions/91.64.64.183|91.64.64.183]] 10:07, 21 October 2024 (UTC)
::*::editing EDConnectorDb.php
::*::<code>// Convert the encoding to UTF-8 if necessary.</code>
::*::<code>//              $encoding = mb_detect_encoding( $value, 'UTF-8', true ) ?: 'UTF-8';</code>
::*::<code>//              return $encoding === 'UTF-8' ? $value : mb_convert_encoding( $value, 'UTF-8', $encoding );</code>
::*::<code>return mb_convert_encoding($value, 'UTF-8', 'ISO-8859-15');</code>
::*::fixed this for me (bloody hack). [[Special:Contributions/130.149.196.182|130.149.196.182]] 09:41, 23 October 2024 (UTC)
::*::* There has been a patch to ExternalData to address this issue; you could try to upgrade ED.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 02:13, 3 April 2025 (UTC)

== Getting faulty data with #get_web_data: / #get_file_data: ==

Hello dear people. I have been having issues with this extension for a few days now and my head is soon to be bursting ))

I have the following problem. I need to get external .json files (or .csv) and use those to fill up pages on my wiki.

However the problem lies in the consistency of the data I gather. To explain this issue, I best show you an example.

First I show you how the json I gather seems to be faulty when trying to filter it.
<pre>
{{#get_web_data:
    source=https://uralez.de/foaop/planner/ammo.json
    |format=json
    |data=Name,ProtoId,Weight,PicInv,Ammo_Caliber,Ammo_Caliber_EXT,Ammo_DmgMult,Ammo_DmgDiv,Tier,Poison,Ammo_AcMod,Ammo_DRMod,Ammo_DTMod,Ammo_SubShots,Weapon_DmgType_0,Weapon_DmgMin_0,NoCrit,Weapon_DmgType_1,Weapon_DmgMin_1,Weapon_BleedStr,Ammo_Radius,Special,Fuse,Weapon_Extra_0,Weapon_Extra_1,Weapon_Spread_0
    |filters=ProtoId={{{id|14017}}}
}}
</pre>

In this example, the id 14017 does not have a lot of data. This is what is written inside that .json for that particular id:

<pre>
{
"Name":"Paper Cartridges",
"ProtoId":14017,
"PicInv":"art/ammo/papercartridges.png",
"Weight":35,
"Ammo_Caliber":1,
"Ammo_Caliber_EXT":44,
"Ammo_DRMod":20,
"Ammo_DTMod":0,
"Weapon_DmgType_0":1,
"Weapon_DmgMin_0":250,
"Weapon_BleedStr":150,
"Tier":1
},
</pre>

However, running the script on my mediawiki (<pre>https://uralez.de/foaop/wiki/index.php/Template:Ammo_Infobox</pre> - in case this is of any importance) will provide vastly different results for variables that ID does not have, but others do.
For example it also gives me 'Subshots=5', even though that particular ID has no Subshots variable to begin with! In other words, getting json and filtering it out for an ID fills all the "unused" variables with values from other IDs. This should not happen, but it does, and I do not know why.

Trying to fix this issue, I hoped that converting the file to a .csv and instead gathering information from there could be a solution. It only partially was. Trying to get the same data from the .csv has the issue, that the first row of the .csv is skipped and not read out for any of the variables.

So having this for example:
<pre>
{{#get_web_data:
    source=https://uralez.de/foaop/planner/ammo.csv
    |format=csv with headers
    |data=Name,ProtoId,Weight,PicInv,Ammo_Caliber,Ammo_Caliber_EXT,Ammo_DmgMult,Ammo_DmgDiv,Tier,Poison,Ammo_AcMod,Ammo_DRMod,Ammo_DTMod,Ammo_SubShots,Weapon_DmgType_0,Weapon_DmgMin_0,NoCrit,Weapon_DmgType_1,Weapon_DmgMin_1,Weapon_BleedStr,Ammo_Radius,Special,Fuse,Weapon_Extra_0,Weapon_Extra_1,Weapon_Spread_0
    |filters=ProtoId={{{id|14017}}}
}}
</pre>
Will fill every variable BUT the "Name" variable. If I modify the .csv to put another variable to the front, it will ignore that variable. So in this case, no matter what id I will filter for - the "Name" variable will NOT be populated.

Both issues plague me - I do not know what I am doing wrong and this extension seems to be such a core part if one wants to read out external .json / .csv files that I do not even know of a different approach. If anyone could provide me with some help - or perhaps tell me what might be wrong, I would be very grateful. Thanks to anyone reading this. [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 23:00, 20 February 2025 (UTC)

* First of all, the recommended syntax now is like this (I left only the first few columns and skipped the filter). Note <code>:|</code>; that there are only one parser function, and no <code>data</code> parametre:
<pre>{|
! Name !! ProtoId !! Weight !! PicInv !! Ammo_Caliber !! Ammo_Caliber_EXT !! Ammo_DmgMult !! Ammo_DmgDiv
{{#for_external_table:|
{{!}}-
{{!}} {{{Name}}} {{!}}{{!}} {{{ProtoId}}} {{!}}{{!}} {{{Weight}}} {{!}}{{!}} {{{PicInv}}} {{!}}  {{!}} {{{Ammo_Caliber}}} {{!}}{{!}} {{{Ammo_Caliber_EXT}}} {{!}}{{!}} {{{Ammo_DmgMult}}} {{!}}{{!}} {{{Ammo_DmgDiv}}}
     |source = https://uralez.de/foaop/planner/ammo.json
     |format = json
}}
|}</pre>
* From the output of this wikicode the problem will be obvious. On heterogenous records (that is, with optional fields) the extension always worked counter-untuitively, since it emulates a sort of column-based storage, in which columns are not dependent on each other. Emply fields are filled with values from next records.<br />Since you want only one record, a possible solution is to use JSONpath. None the default em dashes for missing properties:
<pre>{|
! Name !! ProtoId !! Weight !! PicInv !! Ammo_Caliber !! Ammo_Caliber_EXT !! Ammo_DmgMult !! Ammo_DmgDiv
{{#for_external_table:|
{{!}}-
{{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].Name|—}}} {{!}}{{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].ProtoId|—}}} {{!}}{{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].Weight|—}}} {{!}}{{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].PicInv|—}}} {{!}}{{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].Ammo_Caliber|—}}} {{!}}{{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].Ammo_Caliber_EXT|—}}} {{!}}{{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].Ammo_DmgMult|—}}} {{!}}{{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].Ammo_DmgDiv|—}}}
    |source = https://uralez.de/foaop/planner/ammo.json
    |format = json with jsonpath
}}
|}</pre>
* Also, you could use Lua:
<syntaxhighlight lang="lua">local id = 14017
local record = mw.ext.externalData.getExternalData {
	source = 'https://uralez.de/foaop/planner/ammo.json',
	format = 'json with jsonpath',
	data = { record = '$[?(@.ProtoId == ' .. tostring (id) .. ')]' }
}.record</syntaxhighlight>
* [[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 08:26, 21 February 2025 (UTC)
*:Hello! Thank you for your input! This definitely seems to be the solution I was looking for. There is only 1 question I have remaining - Using <nowiki>{{{id}}}</nowiki> seems to be not supported inside the #for_external_table: function.
*:<pre>$[?(@.ProtoId == {{{id (Illegal JSONpath $1)</pre>
*:What are the options to make the variable useable inside #for_external_table ?
*:Using the ID like this works of course: 
*:<pre>{{{$[?(@.ProtoId == 14017)].Name|—}}}</pre>
*:But then I can not have it generated (Unless I take the Lua approach of course) [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 09:52, 21 February 2025 (UTC)
*::Small addendum. I have managed to resolve the issue with a small hack.
<pre>{|
! Name !! ProtoId
{{#for_external_table:|
{{!}}-
{{!}} {{{$[?(@.ProtoId == {{ProtoId|{{{id|14017}}}}})].Name|—}}} {{!}}{{!}} {{{$[?(@.ProtoId == {{ProtoId|{{{id|14017}}}}})].ProtoId|—}}}
    |source = http://localhost/foaop/planner/ammo.json
    |format = json with jsonpath
}}
|}</pre>
*::Creating a ProtoId Template which just shows the id I give to him.
<pre>
{{{1}}}
</pre>
*::This basically.
*::It looks extremely wrong and I am sure I might be able to fix the syntax, but for now this works. I hope. I will do some tests now.
*::I thank you for your time and effort for going through my issue and providing me with help and support. Thank you again.
*::I would of course not mind a solution to the syntaxing problem, but, at least I can work with the hacky solution I have right now. <3 [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 11:03, 21 February 2025 (UTC)
*:::I have managed to fix all the issues! I am very thankful for your input once again! I am very happy :)
*:::In case anyone has a similar problem like I do, this is basically what I did:
<pre>
{{#get_file_data:
    source=planner
    |file name=ammo.json
    |format=json with jsonpath
    |data=Name=$[?(@.ProtoId == {{{id|14017}}})].Name,ProtoId=$[?(@.ProtoId == {{{id|14017}}})].ProtoId,Weight=$[?(@.ProtoId == {{{id|14017}}})].Weight,PicInv=$[?(@.ProtoId == {{{id|14017}}})].PicInv
}}
</pre>
*:::Of course you can also do the same with get_web_data
*:::Basically the "filtering" happens in "data" and it works flawlessly - remeding the issues I had beforehand which Alexander Mashin explained (Values in optional fields are filled with other .json entries) [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 12:32, 21 February 2025 (UTC)

::::* First of all, you should no longer use the <code><nowiki>{{#get_…_data:}}</nowiki></code> functions. They are deprecated, because in new versions of MediaWiki these functions can be called ''after'' <code><nowiki>{{#external_value:}}</nowiki></code>, etc.<br />I have tested the following syntax for a template, using the otherwise also deprecated parametre <code>data</code>:
<pre>{|
! Name !! ProtoId !! Weight !! PicInv !! Ammo_Caliber !! Ammo_Caliber_EXT !! Ammo_DmgMult !! Ammo_DmgDiv
{{#for_external_table:|
{{!}}-
{{!}} {{{Name|—}}} {{!}}{{!}} {{{ProtoId|—}}} {{!}}{{!}} {{{Weight|—}}} {{!}}{{!}} {{{PicInv|—}}} {{!}}{{!}} {{{Ammo_Caliber|—}}} {{!}}{{!}} {{{Ammo_Caliber_EXT|—}}} {{!}}{{!}} {{{Ammo_DmgMult|—}}} {{!}}{{!}} {{{Ammo_DmgDiv|—}}}
    |source = https://uralez.de/foaop/planner/ammo.json
    |format = json with jsonpath
    |data =
         Name = $[?(@.ProtoId == {{{id|14017}}})].Name,
         ProtoId = $[?(@.ProtoId == {{{id|14017}}})].ProtoId,
         Weight = $[?(@.ProtoId == {{{id|14017}}})].Weight,
         PicInv = $[?(@.ProtoId == {{{id|14017}}})].PicInv,
         Ammo_Caliber = $[?(@.ProtoId == {{{id|14017}}})].Ammo_Caliber,
         Ammo_Caliber_EXT = $[?(@.ProtoId == {{{id|14017}}})].Ammo_Caliber_EXT,
         Ammo_DmgMult = $[?(@.ProtoId == {{{id|14017}}})].Ammo_DmgMult,
         Ammo_DmgDiv = $[?(@.ProtoId == {{{id|14017}}})].Ammo_DmgDiv
}}
|}</pre>

::::* The root of the issue is that the <code>{{{…}}}</code> macros in the now recommended <code>data</code>-less syntax are not processed correctly when they contain template parametres. I will try to fix it.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 05:29, 22 February 2025 (UTC)
::::*:Thank you very much! I have used your approach to do my templates and it works '''wonderful'''! Just as expected. I do not use the deprecated function, (Apart from the data parametre due to the obvious issue you have outlined) and get exactly the results I want. As seen here:
<pre>
https://uralez.de/foaop/wiki/index.php?title=Template:Weapon_Infobox
https://uralez.de/foaop/wiki/index.php?title=Template:Armor_Infobox
https://uralez.de/foaop/wiki/index.php?title=Template:Ammo_Infobox
</pre>
For this once again a very big thank you!
::::*:While my initial question have been answered, I would like to ask you, if it is possible to get a #for_external_table function to make a list of the .json entries without the afromentioned issue of values going into different entries. (Basically when trying to get all the "weapons" (in my case), it once again has the same issue of behaving unintuively.
To show with an example, I mean the following:
<pre>
{{{!}} class="wikitable sortable"
! Tier
! Type
! Name
! Picture
! Strength
! Damage Type
! Max Distance<br>(Attack 1)
! Max Distance<br>(Attack 2)
{{#for_external_table:|
{{!}}-
{{!}} {{{Tier}}}
{{!}} {{{Weapon_Skill_0}}}
{{!}} {{{Name}}}
{{!}} https://uralez.de/foaop/planner/{{{PicInv}}}
{{!}} {{{Weapon_MinStrength}}}
{{!}} {{{Weapon_DmgType_0}}}
{{!}} {{{Weapon_MaxDist_0}}}
{{!}} {{{Weapon_MaxDist_1}}}
    |source=https://uralez.de/foaop/planner/weapons.json
    |format = json with jsonpath
    |data =
Name=$[?(@)].Name,
PicInv=$[?(@)].PicInv,
Weapon_MinStrength=$[?(@)].Weapon_MinStrength,
Tier=$[?(@)].Tier,
Weapon_Skill_0=$[?(@)].Weapon_Skill_0,
Weapon_DmgType_0=$[?(@)].Weapon_DmgType_0,
Weapon_MaxDist_0=$[?(@)].Weapon_MaxDist_0,
Weapon_MaxDist_1=$[?(@)].Weapon_MaxDist_1
}}
{{!}}}
</pre>
(For clarity - Weapon_MaxDist_1 is a value not every entry has. And if it does not have it it takes one from another entry.)
Is there a way to just use for_external_table and still mitigate the issue of having values jump into other entries due to the way the extension stores them? Or would I here once again have to resort to a hack. (Either converting the .json to a .csv specifically for the whole table or... Do some other shenanigan like pulling a template for each entry which however would result in a MASSIVE amount of calls on the target webserver) [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 13:16, 24 February 2025 (UTC)
* For a set of records not of all of which have the same attributes set, the safest would be to use Lua: <syntaxhighlight lang="lua" inline>mw.ext.externalData.getExternalData 'https://uralez.de/foaop/planner/weapons.json'.__json</syntaxhighlight>. Note the syntactic sugar.<br />Also, you definitely need to set up [[Extension:External Data/Caching data|this extension's cache]]: it not only reduces the load on your and remote servers, but also helps if the latter goes offline.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 15:30, 24 February 2025 (UTC)
*:Understood! I had the extension cache already set up so I should be good in that regard.
*:I tried to avoid Lua, not because I am not proficient with it, quite the contrary, but moreso because I thought it would add unnecesarry risks/data handling/stress. But I guess the usage of Lua is quite well-spread in the mediawiki community and shouldn't pose any grave issues :)
*:I can't stress this enough but give you a warm thanks once again. You have helped me out with all the issues I had and allowed me to make a solution that works out very well for me!
*:o7 [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 15:37, 24 February 2025 (UTC)
*:: If you upgrade the extension, you can now use the consize syntax for <code><nowiki>{{#for_external_table:}}</nowiki></code>, i.e., without <code>data</code> even with template parametres in external variables' names and default values.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 06:09, 26 February 2025 (UTC)
*:::Thank you very much! )) [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 09:28, 28 February 2025 (UTC)
*:::I noticed a small bug with using Template Parameters in external variables' names with default values.
*:::For example I have this code:
<pre>
{{#for_external_table:|
{{{!}} class="wikitable" style="width:30%; float:right"
! colspan="1" {{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].Name|—}}}<br>Tier {{{$[?(@.ProtoId == {{{id|14017}}})].Tier|—}}}
{{!}}}
    |source = https://uralez.de/foaop/planner/ammo.json
    |format = json with jsonpath
}}
</pre>
*:::The last `Tier` will result in a dash-line (-) even though that value exists. If I will then ask for a third variable (Or even the same) it will work. It's like the last variable call is not handled properly.
*:::This for example would now have a working Tier, but now ProtoId will be a dash line because it is the last query.
<pre>
{{#for_external_table:|
{{{!}} class="wikitable" style="width:30%; float:right"
! colspan="1" {{!}} {{{$[?(@.ProtoId == {{{id|14017}}})].Name|—}}}<br>Tier {{{$[?(@.ProtoId == {{{id|14017}}})].Tier|—}}}<br>ProtoId {{{$[?(@.ProtoId == {{{id|14017}}})].ProtoId|—}}}
{{!}}}
    |source = https://uralez.de/foaop/planner/ammo.json
    |format = json with jsonpath
}}
</pre> [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 13:33, 7 March 2025 (UTC)

:Small Addendum, having if requests inside the function will not work outright:
:<pre>{{#if: {{{$[?(@.ProtoId == {{{id|14017}}})].PicInv|}}} | https://uralez.de/foaop/planner/{{{$[?(@.ProtoId == {{{id|14017}}})].PicInv}}} |No Image}}</pre>
:even if after the if statement you ask for different kind of data. I suspect it has something to do with the order the statement is processed. (First the if, the the actual variable call, maybe?) [[User:Zmejaa|Zmejaa]] ([[User talk:Zmejaa|talk]]) 14:04, 7 March 2025 (UTC)
:* Thank you for reporting this; there has been a fix; try to upgrade.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 09:19, 15 March 2025 (UTC)

: I'm afraid I'm having the same kind of issue here. I'm querying JSON data with JSONPath and use <code>#for_external_table</code> to output them to the page. But at the end of the table, it appears that the final rows are wanting data that happen to be wrongly assigned to some previous rows (two rows off). I updated ED to the latest master, but that did not solve anything. [[User:Rand(1,2022)|Rand(1,2022)]] ([[User talk:Rand(1,2022)|talk]]) 08:03, 3 April 2025 (UTC)
:* You might want to provide a reproducible publically available example, in case there's another regression. But the chances are small. There is a fault in underlying legacy architecture of the extension: when the data records' structure is not the same, as is the case with JSON and XML, as opposed to CSV, the data columns which are build for each variable independently, will be of different height. It is not likely that it is going to be fixed soon. You should use Lua and the <code>__json</code> variable that will preserve the structure of the JSON file.<br />'''P.S.''' Now that I think of it more, I realise that it's not even a bug. When we have several Xpaths, JSONPaths or plain field names, there is no reason to assume that they describe fields of the same data structure. The implied data record is a phantom; for none has ever been queried for. External Data has no mechanisms to query data records and then, fields for each record, -- other than Lua, where either <code>__json</code> table can be indexed, or, indeed, the required rows can be queried as a whole, and each returned as a table.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 12:01, 3 April 2025 (UTC)

: It happens when at least one of the items in the array lacks a value or key-value pair. If that's not covered by ED, then so be it. I now solved it using a temporary extension that converts JSON to a PHP array internally and some custom code to fetch the relevant data. [[User:Rand(1,2022)|Rand(1,2022)]] ([[User talk:Rand(1,2022)|talk]]) 11:02, 4 April 2025 (UTC)
::OK, but if you have Scribunto, it can already do it.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 11:11, 4 April 2025 (UTC)
::: That's true, of course 👍 [[User:Rand(1,2022)|Rand(1,2022)]] ([[User talk:Rand(1,2022)|talk]]) 11:49, 4 April 2025 (UTC)

== Nesting #for_external_table  ==

Hello, I have a table in a db that I'm trying to render in a wiki as an indented list. Below is an example of my approach with a nested #for_external_table, and how it's failing.

Is there a better way to approach this, and is it possible within External_Data?

{| class="wikitable"
|-
! Underlying Data !! Goal
|-
|
{| class="wikitable"
|-
! topic_id !! topic_name !! topic_parentname
|-
| 1 || topic1 || 
|-
| 2 || topic2 || topic1
|-
| 3 || topic3 ||
|-
| 4 || topic4 || topic3
|-
| 5 || topic5 || topic3
|}
||
* [[topic1]]
** [[topic2]]
* [[topic3]]
** [[topic4]]
** [[topic5]]
|}

I have a query defined in LocalSettings.php:

 <nowiki>
'childrenTopics' => [
 'query' => "SELECT * FROM topics WHERE topics.topic_parentname = ?",
 'types' => 's'
]</nowiki>

A call to the query to show all top-level (i.e. parent-less) topics works as desired:

{| class="wikitable"
|-
! Input !! Output
|-
|
 <nowiki><ul>
{{#for_external_table:
 |<li>[[{{{topicName}}}]]</li>
 |source = CaseData
 |query = childrenTopics
 |data = topicName=topic_name
 |parameters = ""
}}
</ul> </nowiki>
||
* [[topic1]]
* [[topic3]]
|}

As does a check to output the children of topic3:

{| class="wikitable"
|-
! Input !! Output
|-
|
 <nowiki><ul>
{{#for_external_table:
 |<li>[[{{{topicName}}}]]</li>
 |source = CaseData
 |query = childrenTopics
 |data = topicName=topic_name
 |parameters = "topic3"
}}
</ul> </nowiki>
||
* [[topic4]]
* [[topic5]]
|}

And I'm able to nest identical #for_external_table calls:

{| class="wikitable"
|-
! Input !! Output
|-
|
 <nowiki><ul>
{{#for_external_table:
 |<li>[[{{{topicName}}}]]</li>
  <ul>
  {{#for_external_table:
   |<li>[[{{{topicName}}}]]</li>
   |source = CaseData
   |query = childrenTopics
   |data = topicName=topic_name
   |parameters = ""
  }}
  </ul>
 |source = CaseData
 |query = childrenTopics
 |data = topicName=topic_name
 |parameters = ""
}}
</ul></nowiki>
||
* [[topic1]]
** [[topic1]]
** [[topic3]]
* [[topic3]]
** [[topic1]]
** [[topic3]]
|}

But when I try to assign a different parameter name in the data of the nested query (topicName -> topicName2), then the outer query fails to return a second entry. Can you help point to why?

{| class="wikitable"
|-
! Input !! Output
|-
|
 <nowiki><ul>
{{#for_external_table:
 |<li>[[{{{topicName}}}]]</li>
  <ul>
  {{#for_external_table:
   |<li>[[{{{topicName2}}}]]</li>
   |source = CaseData
   |query = childrenTopics
   |data = topicName2=topic_name
   |parameters = ""
  }}
  </ul>
 |source = CaseData
 |query = childrenTopics
 |data = topicName=topic_name
 |parameters = ""
}}
</ul></nowiki>
||
* [[topic1]]
** [[topic1]]
** [[topic3]]
* [[]]
** [[topic1]]
** [[topic3]]
|}

Based on that, it makes sense then that my full attempt fails:

{| class="wikitable"
|-
! Input !! Output
|-
|
 <nowiki><ul>
{{#for_external_table:
 |<li>[[{{{topicName}}}]]</li>
  <ul>
  {{#for_external_table:
   |<li>[[{{{topicName2}}}]]</li>
   |source = CaseData
   |query = childrenTopics
   |data = topicName2=topic_name
   |parameters = {{{topicName}}}
  }}
  </ul>
 |source = CaseData
 |query = childrenTopics
 |data = topicName=topic_name
 |parameters = ""
}}
</ul></nowiki>
||
* [[topic1]]
** [[topic2]]
* [[]]
Error: No return values specified.
|}

Is there a different way to do this that you'd recommend? Thanks much! [[User:BrokenUrn|BrokenUrn]] ([[User talk:BrokenUrn|talk]]) 18:44, 6 June 2025 (UTC)

:It seems that it is an issue of the extension's architecture, inherited from the times of the ''legacy mode'', wih separate <code>{{#get_…_data:}}</code> calls, that includes a global context consisting of several data columns. This context is cleared, including variables from the outer loop, when even the&nbsp;inner loop is complete.
:I filed a bug ([[phab:T396267|T396267]]).
:I suggest that you build a hierarchy, using Lua.
:<br /> [[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 07:09, 7 June 2025 (UTC)
::Thanks for the context - will do that instead. [[User:BrokenUrn|BrokenUrn]] ([[User talk:BrokenUrn|talk]]) 11:41, 7 June 2025 (UTC)
:::Here's the recursive Lua that I substituted (with apologies for the distorted formatting):
:::<br/>
:::<nowiki>{{#invoke:TestModule|recurseLineage|parentTopic=""}}</nowiki>
:::In Module:TestModule
<syntaxhighlight lang="lua">
local p = {}
function funcRecurseLineage(parentTopic)
 --Check for children of the parent topic
 local output = ""
 local topics, errors = mw.ext.externaldata.getExternalData {
   source = "CaseData",
   query = "childrenTopics",
   data = "topicName=topic_name",
   parameters = parentTopic
  }
 --If there are children topics, start a new layer of the list
 if topics ~= nil then
  output = output .. "<ul>"
  --For each child topic, add them as a list item, and check for their children
  for i, topic in ipairs(topics) do
   output = output .. " <li>" .. "[[" .. topic.topicName .. "]]"
   output = output .. funcRecurseLineage(topic.topicName)
  end
  output = output .. "</ul>"
 end
 return output
end
function p.recurseLineage( frame )
 return funcRecurseLineage(frame.args.parentTopic)
end
return p
</syntaxhighlight> [[User:BrokenUrn|BrokenUrn]] ([[User talk:BrokenUrn|talk]]) 17:21, 10 June 2025 (UTC)

* <code><nowiki>{{#for_external_table:}}</nowiki></code> in standalone mode now allows nesting.[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 03:00, 30 January 2026 (UTC)
*:Thank you! I'm returning to this project after hiatus, and your fix has been a big help. [[User:BrokenUrn|BrokenUrn]] ([[User talk:BrokenUrn|talk]]) 15:53, 10 June 2026 (UTC)

== Retrieving JSON responses from pages responding with 404 Not Found ==

How can I retrieve the response from a page that returns a JSON response along with a 404 Not Found?

And strangely, <code>$wgExternalDataVerbose = false;</code> doesn't seem to work. [[User:Indeedfore|Indeedfore]] ([[User talk:Indeedfore|talk]]) 07:35, 1 September 2025 (UTC)
* Try <code>| suppress error</code>. See https://traditio.wiki/Traditio:Test/ED/web/404.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 09:29, 1 September 2025 (UTC)
*:Thanks, I missed that parameter. But even with it, I still can’t retrieve a JSON response, right? [[User:Indeedfore|Indeedfore]] ([[User talk:Indeedfore|talk]]) 13:12, 1 September 2025 (UTC)
*** If you want to parse a 404 page to get something useful — I don't think so. You might try to parse the error message with Lua.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 15:17, 1 September 2025 (UTC)
****Even Lua was ineffective. data is <code>nil</code> for 404 pages.
****Actually, there's a more significant issue than data retrieval; even when suppressing errors, it still appears to attempt three retries. The loading speed of pages with multiple 404 URL requests is extremely degraded. Is there a way to prevent retries? [[User:Indeedfore|Indeedfore]] ([[User talk:Indeedfore|talk]]) 13:30, 6 September 2025 (UTC)
***** Try <code>| max tries = 1</code> in parser function call or the Lua equivalent.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 05:33, 15 September 2025 (UTC)

== Limiting cache size ==

I'm using ExternalData for a wiki with 100000 pages, and my ISP has warned me my database has exceeded their 3Gb, limit, and checking I see that ed_url_cache was 1.5Gb, not that surprising as all the page content is read through ExternalData. Is there any way of keeping the cache size down to a fixed 1Gb? I know I can change $wgExternalDataSources['*']['min cache seconds'] from 30 days to some smaller value, but will that ensure the stale cache is purged, and I worry it leaves me vulnerable to a surge in activity within the time window. I also cache the wikidata locally, and its 2.3Gb of files, so as it gets read into the wiki it will overwhelm it. [[User:Vicarage|Vicarage]] ([[User talk:Vicarage|talk]]) 16:15, 2 September 2025 (UTC)

:When using this script I find that mysql under-reports the size of the table, as when rows are cleared they are kept in a (hidden) reserve to be used by the table later, and a 1.2 Gb table can have 500Mb of hidden space, and I need to run the OPTIMIZE command. It would be so useful to have ExternalData do this itself to keep to a fixed cache size, perhaps using a maintenance script I could run with cron [[User:Vicarage|Vicarage]] ([[User talk:Vicarage|talk]]) 17:12, 24 October 2025 (UTC)

Gemini suggested this script I could run from cron.

<syntaxhighlight lang="python">
#!/usr/bin/python3
import mysql.connector
from mysql.connector import errorcode
import sys

# --- Configuration ---
# You need to fill in your database connection details here.
DB_HOST = 'localhost'
DB_USER = 'USER'
DB_PASSWORD = 'PASSWORD'
DB_NAME = 'DATABASE'
TABLE_NAME = 'ed_url_cache'

# Set the size threshold in bytes (1.5 GB).
# 1.5 * 1024 * 1024 * 1024
SIZE_THRESHOLD_BYTES = 1610612736 

# Number of rows to delete if the threshold is exceeded.
ROWS_TO_DELETE = 5000

def monitor_and_clean_cache():
    """
    Monitors the size of a specified MySQL table and deletes
    the oldest rows if it exceeds a predefined threshold.
    """
    db_connection = None
    try:
        # Establish the database connection
        db_connection = mysql.connector.connect(
            host=DB_HOST,
            user=DB_USER,
            password=DB_PASSWORD,
            database=DB_NAME
        )
        cursor = db_connection.cursor()

        # Get the size of the table in bytes
        print(f"Checking size of table '{TABLE_NAME}'...")
        size_query = f"""
            SELECT (data_length + index_length)
            FROM information_schema.TABLES
            WHERE table_schema = '{DB_NAME}'
            AND table_name = '{TABLE_NAME}';
        """
        cursor.execute(size_query)
        table_size = cursor.fetchone()

        if table_size and table_size[0]:
            current_size_bytes = table_size[0]
            current_size_gb = current_size_bytes / (1024 ** 3)
            print(f"Current table size: {current_size_gb:.2f} GB")

            # Check if the table size exceeds the threshold
            if current_size_bytes > SIZE_THRESHOLD_BYTES:
                print(f"Table size exceeds the {SIZE_THRESHOLD_BYTES / (1024 ** 3):.2f} GB threshold.")
                print(f"Deleting the oldest {ROWS_TO_DELETE} rows...")

                # Construct the DELETE query to remove the oldest rows
                # The 'req_time' field is used to identify the oldest entries.
                delete_query = f"""
                    DELETE FROM {TABLE_NAME}
                    ORDER BY req_time ASC
                    LIMIT {ROWS_TO_DELETE};
                """
                print (f"query ${delete_query}")
                cursor.execute(delete_query)
                db_connection.commit()
                
                print(f"Successfully deleted {cursor.rowcount} rows.")
            else:
                print("Table size is within the acceptable limit. No action needed.")
        else:
            print(f"Could not retrieve the size of table '{TABLE_NAME}'.")

    except mysql.connector.Error as err:
        if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
            print("Something is wrong with your user name or password")
        elif err.errno == errorcode.ER_BAD_DB_ERROR:
            print("Database does not exist")
        else:
            print(f"An error occurred: {err}")
    finally:
        if db_connection and db_connection.is_connected():
            cursor.close()
            db_connection.close()
            print("Database connection closed.")

if __name__ == "__main__":
    monitor_and_clean_cache()</syntaxhighlight>

* A new configuration variable has been introduced: <code>$wgExternalDataCacheSize</code> (in bytes). The cache is freed before writing to it.<br />[[User:Alex Mashin|Alexander Mashin]]<sup>&nbsp;''[[:User talk:Alex Mashin|talk]]''</sup> 02:58, 30 January 2026 (UTC)

== #external_value with a default/fallback value broken in 3.6.1 ==

Since 3.6.1, any <code>#external_value</code> call that passes a default/fallback value returns the error "Error: No suitable data source is defined for this external data query" instead of the variable's value (or the default). The plain form without a default still works, so the regression is limited to calls that use the documented fallback parameter.

'''Minimal reproduction''' (variable not set):
* <code><nowiki>{{#external_value:myvar}}</nowiki></code> &rarr; <code>Error: no local variable "myvar" has been set.</code> (correct)
* <code><nowiki>{{#external_value:myvar|}}</nowiki></code> &rarr; <code>Error: No suitable data source is defined for this external data query</code> (should be empty)
* <code><nowiki>{{#external_value:myvar|fallback}}</nowiki></code> &rarr; <code>Error: No suitable data source is defined for this external data query</code> (should be "fallback")

It also fails when the variable IS set by a preceding <code>#get_web_data</code>: <code><nowiki>{{#external_value:myvar|}}</nowiki></code> returns the error instead of the fetched value, because the standalone-mode check runs before the stored value is read.

'''Affected versions:''' introduced in 3.6.1; 3.6 and earlier are fine. Tested on MediaWiki 1.43, PHP 8.3.

'''Cause:''' the "allowed for nesting standalone parser functions" work in 3.6.1 ([https://phabricator.wikimedia.org/T396267 T396267]) added a standalone-mode branch to <code>EDParserFunctions::doExternalValue()</code>:
<pre>
if ( count( $args ) > 0 ) {
    $args['data'] ??= "$variable=$variable";
    $fetched = self::emulateGetExternalData( $args, $title );
    if ( is_string( $fetched ) ) {
        return $fetched; // error
    }
}
</pre>
A default/fallback is a value-only positional argument. <code>parseParams()</code> records it under both a null-valued string key and a numeric index; <code>array_shift()</code> drops the (first) string-keyed entry but leaves the numeric one, so <code>count( $args ) > 0</code> stays true. The call is then treated as a standalone fetch with no data source, which yields "No suitable data source is defined". For comparison, the sibling display functions (<code>#for_external_table</code>, <code>#display_external_table</code>) guard the same branch with <code>&& EDConnectorBase::sourceSet( $args )</code>, whereas <code>doExternalValue()</code> does not. [[User:Morne Alberts|Morne Alberts]] ([[User talk:Morne Alberts|talk]]) 10:59, 26 June 2026 (UTC)

:Tracked in https://phabricator.wikimedia.org/T431818 [[User:Morne Alberts|Morne Alberts]] ([[User talk:Morne Alberts|talk]]) 12:31, 10 July 2026 (UTC)
:Fixed in latest master. [[User:Morne Alberts|Morne Alberts]] ([[User talk:Morne Alberts|talk]]) 12:24, 22 July 2026 (UTC)