Abstract  2

Acknowledgements  3

Executive summary  6

1 Introduction  8

General AI governance frameworks  9

Need for operational level guidance in experimenting with generative AI  12

2 Overview of existing guidance for experimenting with GenAI  14

Existing guidelines for experimenting with generative AI: Common themes and divergence  14

3 Prioritisation, monitoring and evaluation of generative AI experimentation  25

Prioritisation  25

Monitoring and evaluation  26

Developing criteria for prioritisation, monitoring and evaluation  26

4 Factors and conditions for successful experimentation and evaluation  37

(Non-)Material resources and assets  37

Lessons from the experimental practice  43

5 The road towards more systemic experimentation  51

1) Establish public trust in GenAI experiments  52

2) Provide AI literacy training and upskilling opportunities to civil servants  53

3) Prevent fragmentation of projects and AI systems  53

4) Prioritise investments in shared tools, data and secure testing mechanisms  53

5) Limit and mitigate risks  54

6) Invest in experimental AI research  54

7) Make AI experimentation frameworks future-fit  55

8) Ensure efficiency through prioritisation  55

9) Plan experiments and their evaluation early  55

10) Create environments that encourage experimentation  56

References  57

Annex A. List of AI guidelines 67

Notes 68

Table 1.1. OECD AI principles 11

Table 2.1. Criteria covered in countries' AI guidelines 16

Figure 5.1. Policy initiatives to improve the use of AI in public administration 53

Boxes 10

Box 1.1. Definition of experimentation 10

Box 1.2. The European Union (EU) AI Act and its implications for the public sector 11

Box 1.3. The OECD Framework for Trustworthy AI in Government 12

Box 2.1. AI guidance in the government: The United Kingdom 17

Box 2.2. AI guidance in government: Australia 19

Box 2.3. AI guidance in government: Canada 20

Box 2.4. Public Sector AI Playbook: Singapore 22

Box 2.5. Accelerating AI Innovation: the US directive to federal agencies 24

Box 3.1. Impact assessment for AI (Italy) 30

Box 3.2. Impact evaluation of AI interventions: the case of the UK 30

Box 3.3. Summary overview of AI risks 36

Box 3.4. Monitoring and evaluation of AI: the case of Italy 36

Box 4.1. The use of AI across the innovation process in government 42

Box 4.2. Guidelines for AI procurement: The UK 43

Box 4.3. "Humphrey" - leveraging AI for the streamlining of public services (the UK) 44

Box 4.4. A Co-Pilot for public servants (Australia and New Zealand) 46

Box 4.5. Efficiency gains abound (Singapore) 47

Box 4.6. Experimental approach in Estonia 48

Box 4.7. Piloting the Fundamental Rights and Algorithms Impact Assessment (FRAIA) in the Netherlands 50