JADE
 | This page in a nutshell: introducting a new software initiative to provide human feedback about AI judgments. |
The Wikimedia Foundation Scoring Platform team is proposing the Judgement And Dialog Engine (JADE) as a way to attach human judgments and dialogue to any wiki artifact (e.g. a revision, a whole page, a user, a log event, etc.), similar to how the Talk namespace relates to articles, but with associated structured data such as an article quality rating to challenge ratings made by automated scoring. It will serve as a robust false-positive and feedback gathering system, to tune the AI and encourage democratic oversight. This lets users refute automated scoring, or independently review a wiki entity.
The first iteration will support judgments targeting articles and revisions, in order to refute ORES scores, but can be expanded later to target more artifact types.
These discussions can be curated and vandalism suppressed through the usual mechanisms, and may integrate with Extension:AbuseFilter. We'll be asking the volunteer community to take on a new workload in patrolling these artifacts, so we'll make accessible in all the usual dashboards.
T168993: A UI mockup would go a long way towards explaining what's going on here.
Why is JADE important?
JADE will facilitate a community of people overseeing the AI, perhaps even in "partnership" with the AI. JADE is needed so that editors can effectively challenge the AIs' automated judgments. Currently this work is done ad-hoc on wiki pages. E.g. it:Progetto:Patrolling/ORES. JADE would bring structure to this process and make the human judgements queryable.
Open discussions
See JADE/Schema and JADE/Implementations.
See also
- "Best practices for AI in the social spaces: Integrated refutations"
- Technical work: T148700 and its subtasks.
- Meta-ORES/Open questions
- Meta-ORES/Implementation
- Past examples of (manual, wiki-based) ORES auditing: