AI in administrative procedure
German administrative procedure law draws tighter limits around automation than the AI Act does. Where discretion exists, the best technology does not help.
When administration deploys AI, two sets of rules meet that ask different questions. The AI Act asks about the risk of the system. Administrative procedure law asks whether a decision came about properly. Both answers have to hold.
Two regimes, two questions
AI Act
- Is the application prohibited?
- Is this a high-risk system?
- Are we deployer or provider?
- Do transparency, oversight and training duties apply?
Administrative procedure law
- Does a legal provision allow a fully automated decision?
- Is there discretion or a margin of appraisal?
- Were the facts actually investigated?
- Do the reasons reflect the considerations that carried the decision?
The sensitive points
- Investigation of the factsThe authority establishes the facts of its own motion. A system deriving a proposal from existing data does not replace that.
- The right to be heardAdverse decisions require a hearing. Whoever hears must also be able to process what was said, and that happens outside the model.
- ReasonsThe reasons must show why the decision went this way and not another. A reference to a system output does not do that.
- The file and the recordWhat the system proposed and what became of it belongs in the file. Without that, nothing can be reconstructed in an appeal.
To be settled before deployment
- Does the system fall into a high-risk category, for instance access to public services?
- Is a fundamental rights impact assessment required, and who prepares it?
- Is there a duty to register in the European database?
- Is human oversight designed so that it can actually be exercised?
- Do staff know what to check the output against?
- Is it settled when a case leaves the system and is handled by hand?
More on classification as a high-risk system and the duties attached to it is set out under high-risk AI.
Deploying AI in a procedure?
We check the classification and say what procedural law demands on top.
Have the deployment reviewedFrequently asked questions
May a decision be issued fully automatically?
Only where a legal provision expressly allows it and there is neither discretion nor a margin of appraisal. Federal administrative procedure law provides for this, and the state statutes largely follow. An AI system that fills in margins is precisely what the provision does not cover.
What if the AI only assists?
Then responsibility stays with the case worker, and a good deal follows from that. The investigation of the facts has to happen, the reasons have to reflect the considerations that carried the decision, and a system proposal adopted unchecked is not an exercise of discretion. Assistance that decides in practice is no longer assistance.
What is the fundamental rights impact assessment?
Deployers of high-risk systems that are public bodies must assess the impact on fundamental rights before use. It is not the same as the data protection impact assessment, although the two overlap and are sensibly worked on together.