MediaWiki - User contributions [en]

Topic:Y90tdydridd4fu7o

2024-07-22T13:38:28Z

93.153.250.62 (talk) commented on "the link to the extension in gerrit is not active" (the link to the extension in gerrit is not active)

Extension talk:External Data

2023-07-21T07:33:23Z

93.153.250.62: /* Read BSON array from MongoDB */ Reply

{{External Data navigation}}
{{Archive box|
* [[Extension talk:External Data/Archive 2009|2009]]
* [[Extension talk:External Data/Archive 2010|2010]]
* [[Extension talk:External Data/Archive 2011|2011]]
* [[Extension talk:External Data/Archive 2012|2012]]
* [[Extension talk:External Data/Archive 2013|2013]]
* [[Extension talk:External Data/Archive 2014|2014]]
* [[Extension talk:External Data/Archive 2015|2015]]
* [[Extension talk:External Data/Archive 2016|2016]]
* [[Extension talk:External Data/Archive 2017 to 2018|2017 to 2018]]
* [[Extension talk:External Data/Archive 2019 to 2020|2019 to 2020]]
* [[Extension talk:External Data/Archive 2021 to 2022|2021 to 2022]]
}}

== accessing the internal variable (__json) ==

I need some help. Background: I'm working on an extension that uses a webcall to retrieve a JSON object and converts it into concise html code. This extension now is working properly, but lacks caching. This JSON object is complex: containing multiple nested arrays with varying number of elements and it took me a while to program it in php.

My current line of thought is to transfer the entire json object (that is stored in the internal variable __json) to my extension.

Question 1) how does one access the internal variables?? The code below doesn't work.

Question 2) do you have any comments / suggestions on my approach?

<nowiki>{{#get_web_data:url=https://jisho.org/api/v1/search/words?keyword="先生"|format=json}}</nowiki>

* first bullit: {{#external_value:__json}}
* second bullit:{{#external_value:meta}}

{{#myextension:{{#external_value:__json}}|param1|param2}} [[User:Harro Kremer|Harro Kremer]] ([[User talk:Harro Kremer|talk]]) 21:20, 3 January 2023 (UTC)
* Since <code>__json</code> is not a string but a Lua table, it is only accessible from Lua ([[Extension:Scribunto|Scribunto]] is needed) and not from wikitext:
<syntaxhighlight lang="lua">local parsed, errors = mw.ext.externalData.getExternalData{ url = 'https://jisho.org/api/v1/search/words?keyword="先生"' }
if parsed then
local json = parsed.__json
end</syntaxhighlight>
[[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 03:46, 9 January 2023 (UTC)

== Using External Data in a Template ==
I want to fill a custom Infobox Template with data from ExternalData. The data will reside in a CSV file matching the non-namespace part of the page importing.

<code>mynamespace:mypage content: <nowiki>{{template:infobox myinfobox}}</nowiki>" 
template:myinfobox content: <nowiki>{{infobox | header1 = header | data1 = {{#external_value:value_name|source=data|format=csv with header|delimiter=;|file name={{PAGENAME}}.csv}}</nowiki>
</code>
However, this tries to fetch data from a CSV file named "myinfobox" instead
What is a better way to achieve this?

:Is the template called "infobox myinfobox", or just "myinfobox"? And do you see the problem on the "mypage" page, or right on the template page? [[User:Yaron Koren|Yaron Koren]] ([[User talk:Yaron Koren|talk]]) 00:34, 10 January 2023 (UTC)
::my template page is called "Vorlage:Infobox_VM". The problem occurs on "mypage" as the template page doesn’t really have any visible content nor has it the CSV providing information. "mypage" tries to read from a csv called Vorlage:Infobox_VM.csv instead if mypage.csv.
* This looks like a strange way to invoke a template: <code><nowiki>{{template:infobox myinfobox}}</nowiki></code>. Why not just <code><nowiki>{{myinfobox}}</nowiki></code>? And, as said above, fetching data from <code>myinfobox.csv</code> is what to be expected on the template page itself, unless the code is wrapped with <code><nowiki><includeonly>...</includeonly></nowiki></code>. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 02:11, 10 January 2023 (UTC)

::yes you’re right, i tried being verbose for clarity. in my page i have only <nowiki> {{Infobox VM}}</nowiki> and that seems to work well enough. I'm not really concerned what csv is being read on the template page itself, as that page isn’t really supposed to be looked at. I want "mypage" to look at mypage.csv through the template. that’s what doesn’t work.
* What are the settings for the data source <code>data</code> in <code>LocalSettings.php</code> (<syntaxhighlight lang="php" inline>$wgExternalDataSources['data']</syntaxhighlight>)?
: <syntaxhighlight lang="php">$wgExternalDataSources['data']['path'] = '/opt/infradb-data/vm/';
</syntaxhighlight>
: that path and all the files in it are world readable. that stuff works in principle. i am able to pull info from csv. just not via the page name in the template.
* Could not reproduce the issue at my ''MediaWiki'' installation. Looks like there you have some sophisticated wiki code that substituted <code><nowiki>{{PAGENAME}}</nowiki></code> too early on. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 10:59, 14 January 2023 (UTC)
*:I solved the problem this way: on "mypage" i call <nowiki><code><nowiki>{{Infobox Host | filename={{PAGENAME}} }} </nowiki></nowiki><nowiki></code></nowiki> and on the template page i use that variable to find the right file. [[Special:Contributions/24.134.95.253|24.134.95.253]] 15:51, 17 January 2023 (UTC)

== Slightly different csv data files, only one works ==

I have 2 48 line csv files, < 10k in size with a minor difference, but only one can be read in ExternalData in 1.39

Simplified case https://johnbray.org.uk/expounder/Extdataproblem1 uses

get_web_data: url=https://files.johnbray.org.uk/Documents/Expounder/Q/532/9928/datagood and then
get_web_data: url=https://files.johnbray.org.uk/Documents/Expounder/Q/532/9928/databad

checking that {{#external_value:ddescription}} has something from a line of the csv file. datagood works, but databad does not. The difference between the files is a few characters on one line, and the good file is actually longer than the bad

<nowiki>
< "item",+1996-04-05T00:00:00Z,+1996-04-08T00:00:00Z,"{{link|Q111529509|Evolution}}","in Heathrow wit
h {{link|Q312405|Vernor Vinge}}, {{link|Q472872|Jack Cohen}}, {{link|Q2927188|Bryan Talbot}}, {{link|
Q7151782|Paul Kincaid}}, {{link|Q742918|Colin Greenland}}, {{link|Q62625219|Maureen Kincaid Speller}}
,","","","","","","",51.4673,-0.4529
</nowiki>

<nowiki>
> "item",+1996-04-05T00:00:00Z,+1996-04-08T00:00:00Z,"{{link|Q111529509|Evolution}}","in Heathrow wit
h {{link|Q312405|Vernor Vinge}}, {{link|Q472872|Jack Cohen}}, {{link|Q2927188|Bryan Talbot}}, {{link|
Q7151782|Paul Kincaid}}, {{link|Q742918|Colin Greenland}}AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBA,","","","","","","",51.4673,-0.4529
</nowiki>

Both files are < 10k in size, 48 lines, and the good file is actually longer than the bad

wc datagood databad
48 338 9245 datagood
48 341 9180 databad [[User:Vicarage|Vicarage]] ([[User talk:Vicarage|talk]]) 14:05, 13 January 2023 (UTC)
* Neither of the CSV files is formed perfectly from PHP's point of view. The problems start at the line 7 (one-based). For <code>databad</code> this causes the automatically recognised delimiter to be <code>|</code> rather than <code>,</code>. You can overcome this by adding <code>delimiter=,</code>. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 10:45, 14 January 2023 (UTC)
*:Thanks for the quick response. That cured the problem, both for my trivial and full cases. [[User:Vicarage|Vicarage]] ([[User talk:Vicarage|talk]]) 11:00, 16 January 2023 (UTC)

== What is the best practice to fetch many values (>300) from the same place? ==

Is it better to use the legacy method, like #get_web_data, and fetch all the values at once, then display using #external_value; or is it better to use the new method, #external_value with source parameter, 300 times? What performance consideration might there be?

[[User:JeremiPlazas|Jeremi Plazas]] ([[User talk:JeremiPlazas|talk]]) 17:51, 17 January 2023 (UTC)
* If you use caching, the difference is not that big. Using <code><nowiki>{{#get_web_data:}}</nowiki></code> will save cache lookups, but the legacy mode will stop working once MediaWiki is upgraded to use Parsoid, since it does not guarantee parsing order. The optimal solution is to handle data fetching and display with one Lua function, where you can save the fetched data into a variable and later display it. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 03:22, 18 January 2023 (UTC)
*:Thanks, we'll look into Lua. We do have caching setup so the standalone method might be fine, now that you've helped us iron out the kinks. Thanks again for the help! [[User:JeremiPlazas|Jeremi Plazas]] ([[User talk:JeremiPlazas|talk]]) 17:54, 19 January 2023 (UTC)

== Some JSONPATH doesn’t retrieve results ==
I have a JSON file with Information about network interfaces in it.
I am retrieving this information with get_file_data and two jsonpath instructions. However, only one of them seems to be executed/filled with data.

This is my json: <code><nowiki>[{"ip-addresses": [{"prefix": 24, "ip-address": "1.1.1.1", "ip-address-type": "ipv4"}, {"ip-address-type": "ipv6", "ip-address": "fe80::aaaa:aaaa:aaaa:aaaa", "prefix": 64}], "hardware-address": "ab:ab:ab:ab:ab:ab", "name": "eth0"}, {"ip-addresses": [{"ip-address": "1.1.1.2", "ip-address-type": "ipv4", "prefix": 24}, {"ip-address-type": "ipv6", "ip-address": "fe80::ccff:ceff:feff:ceff", "prefix": 64}], "hardware-address": "ce:ff:ce:ff:ce:ff", "name": "eth1"}]</nowiki></code> 
and this is the template code i use to retrieve the data: <code><nowiki>{{#get_external_data: source=host_data
|file name={{{{{ucfirst:{{{hostname}}}}}|dummy}}}_net.json
|data=mymacaddress=$[?(@.name == '{{{interfacename}}}')].hardware-address,
adressen=$[?(@.name == '{{{interfacename}}}')][ip-addresses][*].ip-address

|use jsonpath
|format=json
}}
{{{hostname}}}
{{{interfacename}}} ({{#external_value:mymacaddress|invalid}})
{{#display_external_table: template=bulleted list|data=1=adressen}}
</nowiki></code>
 
i call this template in the following fashion: <code><nowiki>{{#display_external_table: template=Interface
|data=interfacename={{{netnames}}},hostname={{{filename}}}
}}</nowiki></code> 
where netnames is just an array, usually only one entry like <code><nowiki>["ens18"]</nowiki></code> and filename points to the correct file

* If <code><nowiki>{{{netnames}}}</nowiki></code> is an array, I don't know how transcluding it within single quotes in a JsonPath query could work. Neither <code>$[?(@.name == '["ens18"]')].hardware-address</code>, nor <code>$[?(@.name == '["eth1"]')].hardware-address</code> is a working JsonPath. <code>mymacaddress=$[?(@.name == 'eth1')].hardware-address</code> would be. I would suggest replacing <code>==</code> with <code>in</code>, but this does not seem to be implemented. You could try a regular expression: <code>$[?(@.name =~ 'eth0|eth1')].ip-addresses.[*].ip-address</code>. Also, you can get the bulleted list without a template:
<nowiki>{{#for_external_table:|
* {{{adressen}}}}}</nowiki>
[[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 06:23, 19 January 2023 (UTC)

== Strange bug parsing CSV with pipe character ==

I have a field called <code>title</code> which is called in <code>#for_external_table</code>. All of the values have at least one pipe character, and the page cuts off everything before and including the first pipe character. But the strange thing is that it only happens if {{{title}}} is located in a specific place.

The page can be seen at https://comprehensibleinputwiki.org/wiki/Mandarin_Chinese/Videos and the external data is at https://comprehensibleinputwiki.org/wiki/Data:Mandarin_Chinese/Videos. If you look at the wiki source of the first link, I have {{{title}}} twice, one of them in a hidden div. If you view the source of the page, the first one is missing part of the value, while the hidden one is complete. [[User:Dimpizzy|Dimpizzy]] ([[User talk:Dimpizzy|talk]]) 20:08, 22 January 2023 (UTC)
* Add <code>delimiter=,</code> to <code>{{#get_web_data:}}</code>. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 02:01, 23 January 2023 (UTC)
*:It didn't seem to change anything. I changed it to:
*:<code><nowiki>{{#get_web_data:url={{fullurl:Data:Mandarin_Chinese/Videos|action=raw}}|format=csv with header|data=language=Language, title=Title, videoId=Video ID, service=Service, level=Level, channel=Channel, index=Index|delimiter=,}}</nowiki></code> [[User:Dimpizzy|Dimpizzy]] ([[User talk:Dimpizzy|talk]]) 03:00, 23 January 2023 (UTC)
*** At least, the videos are displayed now. If the current problem is the trimmed titles in the "Title" column, this is not related directly to the extension. The beginning of the title is treated as attributes to the <code><td></code> tag by ''MediaWiki'' parser. Wrap <code>{{{title}}}</code> with <code>nowiki</code>, like this: <code><nowiki>{{#tag:nowiki|{{{title}}}}}</nowiki></code>, to see the first chunk of the title. '''UPD:''' Or, you can add a second <code><nowiki>{{!}}</nowiki></code> before <code>{{{title}}}</code>. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 07:21, 23 January 2023 (UTC)
***:That worked, thanks! I didn't notice any issues on my end with the videos not displaying before, but good to know! [[User:Dimpizzy|Dimpizzy]] ([[User talk:Dimpizzy|talk]]) 09:36, 23 January 2023 (UTC)

==Parameter parsing problems==

{| class="wikitable"
|-
| PHP || 8.2.1
|-
| MediaWiki || 1.38.4
|-
| PostgreSQL || 14.6
|-
| External Data || 3.2 (5d30e60) 08:38, 2. Nov. 2022
|}

Hi. I use External Data to retrieve data from a PostgreSQL database. In most cases I use prepared statements and I noticed that the passing of parameters seem not to work correctly. Here a self contained test to visualize what I mean.

In the database I have a table with a single column of type text:

<pre>
SELECT * FROM public.test;
txt
----------------------------
a simple text
another simple text
a text with, a comma in it
</pre>

Notice that the lines contain spaces and in one case a comma.

Then I have a search function that receives a parameter of type text and returns a set of text:

<pre>
SETOF TEXT public.mw_test(p_search TEXT)
</pre>

in the configuration LocalSettings.php for this looks like this:

<pre>
$wgExternalDataSources['wikidoc'] = [
'server' => 'xxx',
'type' => 'postgres',
'name' => 'xxx',
'user' => 'xxx',
'password' => 'xxx',
'prepared' => [
'test' => 'SELECT mw_test
FROM public.mw_test($1);'
]
];
</pre>

In the wikipage the snippet is as follow:

<pre>
{{#get_db_data:
db = wikidoc
| query = test
| parameters = with
| data = documentation=mw_test
}}
{{#for_external_table:<nowiki />
{{{documentation}}}
}}
{{#clear_external_data:}}
</pre>

It simply displays what it finds on a line.

What happens is the the list of parameters cannot contain a comma. The snippet as is above works fine and returns:

<pre>
a text with, a comma in it
</pre>

But something like this not:

<pre>
{{#get_db_data:
db = wikidoc
| query = test
| parameters = with, a
| data = documentation=mw_test
}}
{{#for_external_table:<nowiki />
{{{documentation}}}
}}
{{#clear_external_data:}}
</pre>

The error is "Fehler: Es wurden keine Rückgabewerte festgelegt."

It is clear that a comma is used to separate parameters and that is the reason why this does not work. My question is how can I pass the whole string "with, a" as a single parameter.

I tried enclosing it in single and double quotes, but this did not help. It leads to this exception:

<pre>
[6882cc0e426ebdb6cf6911bc] /w/index.php?title=IT/IT_Infrastructure/KOFDB_Uebersicht&action=submit TypeError: EDParserFunctions::formatErrorMessages(): Argument #1 ($errors) must be of type array, null given, called in /home/wiki/application/w/extensions/ExternalData/includes/EDParserFunctions.php on line 98
</pre>

Any idea what I could do to solve this? Help is very appreciated. Thanks

It looks like I found a way to solve this. I can enclose the whole string in round parenthesis and it works.

<pre>
{{#get_db_data:
db = wikidoc
| query = test
| parameters = (with, a)
| data = documentation=mw_test
}}
{{#for_external_table:<nowiki />
{{{documentation}}}
}}
{{#clear_external_data:}}
</pre>

* An intresting workaround; however, upgrade to be able to use double quotes. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 06:10, 11 February 2023 (UTC)

== composer.json ==

When trying to install dependencies via composer, the current requirement says <code>"composer/installers": "~2.1"</code> on REL1_39 branch. The latest version is currently 2.5.4 which doesn't meet this requirement. Could this be updated to a more permissive requirement? [[User:Prod|Prod]] ([[User talk:Prod|talk]]) 18:29, 24 February 2023 (UTC)

:This is being addressed in [[phab:T330485]]. [[User:Prod|Prod]] ([[User talk:Prod|talk]]) 19:39, 13 March 2023 (UTC)

== Not pulling the correct result from the json ==

I have a json file that validates and have used the following code in the page: 

{{#get_web_data:url=https://blah.json 
| data = id=id,type=type,description=description 
| use jsonpath $['Generic']['Defensive']['Wanted']: 
| format=json 
}} 

I have tried all sorts of iterations code to get External data to bring back the result for ['Wanted'] but it will only bring back the first entry in the json file, not the ['Wanted'] entry. The patch checks out and I have checked it on a validator so I am not sure what I am doing wrong.

Any help would be appreciated. [[User:SyrinxCat|SyrinxCat]] ([[User talk:SyrinxCat|talk]]) 23:22, 25 April 2023 (UTC)

:The correct syntax seems to be:
: {{#get_web_data:
: url=https://blah.json
: | data = id=$.Generic.Defensive.Wanted.id,type=$.Generic.Defensive.Wanted.type,description=$.Generic.Defensive.Wanted.description
: | use jsonpath
: | format=json
: }}
:A simpler variant may also work, depending on the JSON contents:
: {{#get_web_data:
: url=https://blah.json
: | data = id=id,type=type,description=description
: | format=json
: }} [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 12:47, 26 April 2023 (UTC)

:

:Thanks for those. The first example returns the error
:ID: Error: no local variable "id" has been set.
:Type: Error: no local variable "type" has been set.
:Description(s): Error: no local variable "description" has been set.
:I was calling the results with
:* ID: {{#external_value:id}}
:* Type: {{#external_value:type}}
:* Description(s): {{#external_value:description}}
:The second example I had already tried and it has the same problem. [[User:SyrinxCat|SyrinxCat]] ([[User talk:SyrinxCat|talk]]) 16:52, 26 April 2023 (UTC)
::Then an example of the JSON would be helpful. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 08:13, 28 April 2023 (UTC)

== Read BSON array from MongoDB ==

submit TypeError EDConnectorMongodb::getValueFromJSONArray(): Argument #1 ($origArray) must be of type array, MongoDB\Model\BSONDocument given

<nowiki>When I put data to {{get_external_data: source=Mongo|from collection|find query ={"email.domain":"some_domain"}</nowiki>

<nowiki>|data=email=email.domain}} recieve error above</nowiki> [[Special:Contributions/93.153.250.62|93.153.250.62]] 09:47, 18 July 2023 (UTC)
* You can now upgrade and check if the error is still there. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 15:27, 18 July 2023 (UTC)
*:yes, the same error [[Special:Contributions/93.153.250.62|93.153.250.62]] 07:13, 19 July 2023 (UTC)
*** What is the Special:Version line for External Data? [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 07:20, 19 July 2023 (UTC)
***:3.3-alpha [[Special:Contributions/93.153.250.62|93.153.250.62]] 12:30, 19 July 2023 (UTC)
**** Yes, but what is after "3.3-alpha"? [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 13:02, 19 July 2023 (UTC)
****:3.3-alpha (67b5813) 11:15, 3 июля 2023 [[Special:Contributions/93.153.250.62|93.153.250.62]] 13:42, 19 July 2023 (UTC)
***** And the bug was fixed on the eighteenth of July. [[User:Alex Mashin|Alexander Mashin]] ''[[:User talk:Alex Mashin|talk]]'' 14:06, 19 July 2023 (UTC)
*****:Do I need update extension? [[Special:Contributions/93.153.250.62|93.153.250.62]] 07:33, 21 July 2023 (UTC)

Extension talk:External Data

2023-07-19T13:42:21Z

93.153.250.62: /* Read BSON array from MongoDB */ Reply

Extension talk:External Data

2023-07-19T12:30:49Z

93.153.250.62: /* Read BSON array from MongoDB */ Reply

Extension talk:External Data

2023-07-19T07:13:43Z

93.153.250.62: /* Read BSON array from MongoDB */ Reply

Extension talk:External Data

2023-07-18T09:47:42Z

93.153.250.62: /* Read BSON array from MongoDB */ new section