Projects and method
Robustness and impact: what we evaluate, and how.
We test the robustness of AI systems, measure their effects on people and on society, and propose desirable ways of integrating them.
Why now
AI is entering education, work, health and private life faster than its effects are being measured. Every design choice sets habits that are hard to undo.
Where we start from
First-hand technical knowledge. Knowing how a model is trained, evaluated and deployed shows where to act.
Four areas of work
From how a system fails to what it changes in those who use it.
-
01
Robustness
How well a system resists misuse and circumvention, measured together with what that resistance costs in production: false positives, latency, price. This is the ground covered by BELLS-O, our first reference evaluation.
Typical questionWhich supervisor best detects attacks without blocking legitimate requests, and at what cost?
-
02
Impact on the person
Effects on cognition, learning, health and productivity, studied empirically. Our preferences, attention and abilities are shaped by the tools we use: these effects are real, measurable and cumulative.
Typical questionDoes an assistant lighten the effort of learning, or the learning itself?
-
03
Impact on society
Transformations already under way, less spectacular than a catastrophe scenario but very much present: what AI changes in work, in culture and in how knowledge is produced.
Typical questionWhat becomes of a profession, or a scientific practice, when part of the work is delegated to a model?
-
04
Desirable integration
Measuring is not enough: we propose and evaluate concrete solutions, so that each design choice is made knowingly rather than by default.
Typical questionHow much friction should an interface keep so that users remain the authors of their decisions?
Approach
Three steps, from concept to solution.
-
Step 1
Map
Lay out the concepts at the intersection of AI science, cognitive neuroscience, philosophy of technology and the social sciences.
-
Step 2
Measure
Define targeted evaluation protocols, for robustness and for impact alike, and apply them under real conditions of use.
-
Step 3
Propose
Put forward concrete solutions, then evaluate them to the same standard as the systems they correct.
Principles
The science of measurement, the conscience of its consequences.
Independence
No financial ties to the developers we evaluate. Our conclusions are not negotiable.
Reproducibility
Documented, versioned, re-runnable protocols. Every score can be traced back to its source.
Transparency
Public methodologies, explicit limitations. What we don't measure is stated as clearly as what we do.
European roots
Data hosted in Europe, a framework designed with the AI Act and its General-Purpose AI Code of Practice in mind.
Two requirements
There is no single desirable way of living with AI.
Plurality
There are as many good ways of integrating AI as there are cultures and languages. An integration has to adapt, not be exported unchanged.
Democratic debate
Which uses to encourage, which limits to set: that is for the societies concerned to decide. Our role is to inform those choices.
A project in one of these areas?
A system to evaluate, a public decision to inform, a research project: let's talk.