Abstract Wikipedia team/Architecture
Wikifunctions Architecture
[edit]Last updated: 2025-03-06 by Jdforrester (WMF)
Overview
[edit]This document describes the system architecture for Wikifunctions, highlighting its core components, interactions and dependencies.
The following diagram reflects:
- The system components and their dependencies.
- The services, their API gateways, and their interactions.
- The interaction between wikifunctions.org and a client production Wikipedia with the capability of embedding functions in its pages.
System Components
[edit]WikiLambda
[edit]Git repository: https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/WikiLambda
MediaWiki extension which enables Wikifunctions features for two types of production system:
- Server mode: for wikifunctions.org:
- Sets the main content type to ZObject.
- Exposes public and internal APIs.
- Implements the resource module ext.wikilambda.app to provide a new interface for pages of content type ZObject.
- Keeps secondary tables with indexed and searchable ZObject content.
- Provides an authentication layer to restrict the creation/update of ZObject content type objects.
- Provides a persistence layer to validate the syntax of created/updated ZObject content objects.
- Client mode: for other Wikipedia deployments that want to use embedded functions in their content pages
- Exposes {{ #function|ZID|...args }} parser function to embed Wikifunctions calls in wikitext pages.
Function orchestrator
[edit]Git repository: https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator
Node service built using @wikimedia/service-utils.
The orchestrator service manages the execution of Function Calls. It is the point of interoperation between MediaWiki+WikiLambda and the function evaluator, which executes native code in various programming languages.
The main responsibilities of the orchestrator are the following:
- Exposes the v1/evaluate endpoint for WikiLambda to access via POST.
- Resolves references by fetching objects from WikiLambda, via wikilambda_fetch ActionAPI.
- Requests Wikidata objects via EntityData linked data interface.
- Keeps a process-level cache with the objects fetched from Wikifunctions and Wikidata.
- Builds the necessary scope to execute the requested function call.
- Requests the evaluator to execute native code in a supported programming language.
- Builds the response object with the status, the return value, and the related metrics.
Function evaluator
[edit]Git repository: https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator
Node service built using @wikimedia/service-utils responsible for validating the incoming Function Call and redirecting it to the right executor, depending on the programming language. The evaluator in its particular language variant maintains a pool of wasmedge subprocesses that run the execution of that programming language native code. Currently, there are two different executors: python3 and JavaScript. We're planning to move the outer wrapper of the service from Node to Rust eventually.
The executors do three things:
- communicate with the main process via standard I/O streams;
- type-convert ZObjects to appropriate native types (and perform the inverse operation, converting native types back to ZObjects); and
- execute code (via constructions like, e.g., exec or eval).
Function schemata
[edit]Git repository: https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata
Git submodule shared through all projects with:
- A library of JavaScript utility functions.
- The JSON definitions of all the built-in ZObject content pages that are necessary for a functioning blank instance of Wikifunctions (languages, native types, built-in functions, etc.)
| Repo | Components | Technologies | Testing tools | Testing approaches | Status | Size | Coverage |
|---|---|---|---|---|---|---|---|
| WikiLambda |
|
|
run instructions
|
Unit: Exhaustive unit testing of components, store modules and mixins. (Front-end-to-middle) End-to-end with reservations: Test most important happy paths for create/edit/run functions.. (Middle-to-back-end) Integration: This particular test suite (ApiFunctionCallTest.php) can be run with different orchestration services by changing the value of the wgWikiLambdaOrchestratorLocation config var in LocalSettings.php. Through the WikiLambda "wikilambda_function_call" entrypoint, it makes given requests to the orchestrator and checks that it's returning the right value (success execution or failure state). This manages to test the whole back-end system (orchestrator and evaluator) as well as the caller entrypoint of the "middleware". Integration: Majority of orchestrator functionality. Tests set timeout, environment, mock services (MW, Wikidata and evaluator) and call endpoint. Builtin implementations are also tested with integration tests. |
|
|
>90%
50-60% ~11 scenarios
* traceability matrix
|
| function-orchestrator | Function orchestrator service | Node, Express | Mocha, Chai |
Unit: Very few unit tests, mostly used for testing ZWrapper class, which contains a representation of a ZObject and its scope. |
✅ passing and running in CI |
Test suites: 20 Tests: 336 |
~85%
|
| function-evaluator |
|
Node, Express, WASM, WASI, Python, QuickJS |
Mocha, Chai, unittest (python3 executor) run instructions |
Unit:
Executor: Good and comprehensive unit testing coverage for executors, testing very well defined pieces of functionality: type conversion, "magic functions", communication over IO channel with the evaluator, etc. |
✅ passing and running in CI |
Tests: 23 Tests: 32 |
~59%
~61%
|
| function-schemata |
|
JavaScript | QUnit |
Unit: Exhaustive unit testing for all the JavaScript utility functions. Integration: Format and validation testing to check the integrity and validity of Builtin ZObject JSON files. |
✅ passing and running in CI | Tests: 5839 | >90%
|
Testing Status (Details)
[edit]WikiLambda
[edit]Jest tests
[edit]- Component: Front-end modules (ext.wikilambda.app and ext.wikilambda.languageselector)
- Technologies: Vue+Pinia, Codex
- Testing tools: Jest, Vue Testing Library - run instructions
- Testing approaches:
- Unit: Exhaustive unit testing of components, store modules and mixins.
- Integration: Integration testing of a limited number of happy paths and the most common possible errors, with mocked API responses.
- Status: ✅ passing and running in CI
- Size: 100 test suites, 1487 tests
- Coverage: >90% (published)
- Statements: 98.52%
- Branches: 92.38%
- Functions: 96.75%
- Lines 98.52%
- Observations:
- No snapshot testing, unit or integration.
- Tests run with the local dev version of @wikimedia/codex, which might not be updated to the current codex version being served by MW. This places a risk of having a dissonance between how the components testing output and their behavior in production. Keeping this up to sync relies on the engineering team keeping track and manually updating the version.
- It would be great to tweak how coverage reports are shown to link this as well.
Phpunit tests
[edit]- Components: WikiLambda extension (APIs, Authentication layer, Persistence layer and Deferred updates, Special pages, Hooks, ZObject validation utils, Maintenance scripts)
- Technologies: PHP, MediaWiki
- Testing tools: Phpunit - run instructions
- Testing approaches:
- Unit: Quite well covered unit testing for ZObject validation classes and utils
- Integration: Everything else is tested through integration tests, mostly by setting a wanted environment (cache, database, secondary tables) before testing complex features.
- Status: ✅ passing and running in CI
- Size: Test suites: 52, Tests: 988, Assertions: 2990
- Coverage: 50-60% (see published report)
- Observations:
- Phpunit tests directory structure is quite chaotic, while there are subdirectories for integration/unit currently the unit dir contains barely anything and the integration dir contains both unit and integration tests. Could do with some housekeeping.
- Some components have zero testing: e.g. Maintenance scripts are fully untested, although they are often run in the production environment. E.g. Special Pages are barely tested.
End-to-middle: WikiLambda Selenium tests
[edit]- Components: WikiLambda Front-end + WikiLambda extension
- Technologies: Vue, Pinia, PHP, Mediawiki
- Testing tools: Webdriverio (Selenium) - run instructions
- Testing approaches:
- End-to-end with reservations: Test most important happy paths for create/edit/run functions.
- Status: ⚠️ work in progress, disabled
- Size: 5 tests
- Coverage: ~11 scenarios (see Wikifunctions e2e Test Traceability Matrix )
- Observations:
- These tests use Beta cluster backend services for function evaluation, but they use them only for a very basic test of the interface (a simple function can be run). While they use end-to-end functionality, they don't really test it.
- They depend on the Beta cluster services. Work is in progress to remove this and replace it with Catalyst.
- Very flaky with codex updates
Middle-to-end: WikiLambda Orchestrator tests
[edit]- Components: WikiLambda Function Call API + Function orchestrator + Function evaluator (python3 and JavaScript)
- Technologies: PHP, Mediawiki ActionAPI, Node, Express
- Testing tools: Phpunit - run instructions
- Testing approaches:
- Integration: This particular test suite (ApiFunctionCallTest.php) can be run with different orchestration services by changing the value of the wgWikiLambdaOrchestratorLocation config var in LocalSettings.php. Through the WikiLambda "wikilambda_function_call" entrypoint, it makes given requests to the orchestrator and checks that it's returning the right value (success execution or failure state). This manages to test the whole back-end system (orchestrator and evaluator) as well as the caller entrypoint of the "middleware".
- Status: ❌ mostly broken and disabled
- Size: 10 tests
- Coverage: ?
- Observations:
Function orchestrator
[edit]- Component: Function orchestrator service
- Technologies: Node, Express
- Testing tools: Mocha, Chai - run instructions
- Testing approaches:
- Unit: Very few unit tests, mostly used for testing ZWrapper class, which contains a representation of a ZObject and its scope.
- Integration: Majority of orchestrator functionality. Tests set timeout, environment, mock services (MW, Wikidata and evaluator) and call endpoint. Builtin implementations also tested with integration tests. Three major groups::
- features/v1/orchestrateTest: hermetic tests which don't need external services. Some tests for builtin implementations (those which don't require mocked services)
- features/v1/mswOrchestrateTest: test functionality that need external services (e.g. user-defined types, wikidata, evaluator, etc.). Uses mockServiceWorker to mock MW, Wikidata, and Localhost. Some more tests for builtin implementations (those which require mocked services).
- features/v1/mockedServicesOrchestrateTest: targeted tests for separate functionalities, all using mockServiceWorker.
- Status: ✅ passing and running in CI
- Size: 20 test suites, 336 tests
- Coverage: ~85% (coverage report)
- Statements : 88.07% ( 1868/2121 )
- Branches : 76.02% ( 704/926 )
- Functions : 91.9% ( 261/284 )
- Lines : 88.15% ( 1860/2110 )
- Observations:
- Builtin implementations could benefit from having unit tests (T277914)
- All builtin implementations have at least one integration test, either in orchestrateTest.js or mswOrchestrateTest.js
- We should move more tests to the mockServicesOrchestrateTest suite
Function evaluator
[edit]- Components: Function evaluator service and executor
- published report of the evaluator layer
- Function evaluator service and JavaScript executor
- Size: 23 tests
- Coverage: ~59%
- Statements: 64.32% ( 1147/1783 )
- Branches: 46.28% ( 380/821 )
- Functions: 64.35% ( 195/303 )
- Lines: 64.41% ( 1142/1773 )
- Function evaluator service and python3 executor
- Size: 32 tests
- Coverage: ~ 61%
- Statements: 66.51% ( 1190/1789 )
- Branches: 48.96% ( 402/821 )
- Functions: 65.13% ( 198/304 )
- Lines: 66.55% ( 1184/1779 )
- Technologies: Node, Express, WASM, WASI, Python, QuickJS
- Testing tools: Mocha, Chai, unittest (python3 executor) - run instructions
- Testing approaches:
- Unit:
- Executor: Good and comprehensive unit testing coverage for executors, testing very well defined pieces of functionality: type conversion, "magic functions", communication over IO channel with the evaluator, etc.
- Integration:
- Executor-level tests for main functionality and primary data flow:; happy paths and a few most common edge cases.
- Evaluator-level tests for each flavor of executor: start at the endpoint with a service similar to production and then test the evaluator like a black box: happy paths and the most interesting error cases, but not all. Another set of integration tests which are similar to the integration tests in the orchestrator, which dependency-inject the environment.
- Service runtime characteristics: load tests and benchmarks. Load tests make sure that the executor selection strategy works as expected (executor pool replenishes itself, etc.)
- Unit:
- Status: ✅ passing and running in CI
- Observations:
- Benchmarks: we run benchmarks but we don't do anything with this data.
- Load tests are very flaky and don't add a lot of value
Function schemata
[edit]- Components: ZObject JavaScript utility functions, Builtin ZObject JSON files, and ZObject validation schemata
- Technologies: JavaScript
- Testing tools: QUnit - "npm run test:nolint"
- Testing approaches:
- Unit: Exhaustive unit testing for all the JavaScript utility functions
- Integration: Format and validation testing to check the integrity and validity of Builtin ZObject JSON files.
- Status: : ✅ passing and running in CI (published here)
- Size: 5839 tests
- Coverage: >90%
- Statements: 98.4% ( 1169/1188 )
- Branches: 93.55% ( 595/636 )
- Functions: 100% ( 202/202 )
- Lines: 98.38% ( 1159/1178 )
- Observations:
- Note that all other repositories depend on function-schemata as a git submodule, so the output of their tests can depend on the schemata version being used.