Title page 1

Contents 12

Report Highlights 13

CHAPTER 1: Research and Development 25

Overview 27

Chapter Highlights 28

1.1. Publications 30

Overview 30

Total Number of AI Publications 30

By Venue 32

By National Affiliation 33

By Sector 37

By Topic 39

Top 100 Publications 40

By National Affiliation 40

By Sector 41

By Organization 42

1.2. Patents 43

Overview 43

By National Affiliation 44

1.3. Notable AI Models 47

By National Affiliation 47

By Sector 48

By Organization 50

Model Release 51

Parameter Trends 53

Compute Trends 57

Highlight: Will Models Run Out of Data? 60

Inference Cost 65

Training Cost 66

1.4. Hardware 69

Overview 69

Highlight: Energy Efficiency and Environmental Impact 72

1.5. AI Conferences 76

Conference Attendance 76

1.6. Open-Source AI Software 78

Projects 78

Stars 80

CHAPTER 2: Technical Performance 83

Overview 86

Chapter Highlights 87

2.1. Overview of AI in 2024 89

Timeline: Significant Model and Dataset Releases 89

State of AI Performance 95

Overall Review 95

Closed vs. Open-Weight Models 96

US vs. China Technical Performance 98

Improved Performance From Smaller Models 100

Model Performance Converges at the Frontier 101

Benchmarking AI 102

2.2. Language 105

Understanding 106

MMLU: Massive Multitask Language Understanding 106

Generation 107

Chatbot Arena Leaderboard 107

Arena-Hard-Auto 109

WildBench 110

Highlight: o1, o3, and Inference-Time Compute 112

MixEval 114

RAG: Retrieval Augment Generation (RAG) 115

Berkeley Function Calling Leaderboard 115

MTEB: Massive Text Embedding Benchmark 117

Highlight: Evaluating Retrieval Across Long Contexts 119

2.3. Image and Video 121

Understanding 121

VCR: Visual Commonsense Reasoning 121

MVBench 122

Generation 124

Chatbot Arena: Vision 125

Highlight: The Rise of Video Generation 126

2.4. Speech 128

Speech Recognition 128

LSR2: Lip Reading Sentences 2 128

2.5. Coding 130

HumanEva 130

SWE-bench 131

BigCodeBench 132

Chatbot Arena: Coding 133

2.6. Mathematics 134

GSM8K 134

MATH 135

Chatbot Arena: Math 136

FrontierMath 136

Highlight: Learning and Theorem Proving 138

2.7. Reasoning 139

General Reasoning 139

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI 139

GPQA: A Graduate-Level Google-Proof Q&A Benchmark 140

ARC-AGI 141

Humanity's Last Exam 143

Planning 145

PlanBench 145

2.8. AI Agents 146

VisualAgentBench 146

RE-Bench 147

GAIA 149

2.9. Robotics and Autonomous Motion 150

Robotics 150

RLBench 150

Highlight: Humanoid Robotics 152

Highlight: DeepMind's Developments 153

Highlight: Foundation Models for Robotics 156

Self-Driving Cars 157

Deployment 157

Technical Innovations and New Benchmarks 158

Safety Standards 159

CHAPTER 3: Responsible AI 162

Overview 164

Chapter Highlights 165

3.1. Background 167

Definitions 167

3.2. Assessing Responsible AI 168

AI Incidents 168

Examples 169

Limited Adoption of RAI Benchmarks 171

Factuality and Truthfulness 172

Hughes Hallucination Evaluation Model (HHEM) Leaderboard 172

Highlight: FACTS, SimpleQA, and the Launch of Harder Factuality Benchmarks 173

3.3. RAI in Organizations and Businesses 175

Highlight: Longitudinal Perspective 182

3.4. RAI in Academia 186

Aggregate Trends 186

Topic Area 189

3.5. RAI Policymaking 193

3.6. Privacy and Data Governance 194

Featured Research 194

Large-Scale Audit of Dataset Licensing and Attribution in AI 194

Data Consent in Crisis 195

3.7. Fairness and Bias 197

Featured Research 197

Racial Classification in Multimodal Models 197

Measuring Implicit Bias in Explicitly Unbiased LLMs 199

3.8. Transparency and Explainability 201

Featured Research 201

Foundation Model Transparency Index v1.1 201

3.9. Security and Safety 203

Benchmarks 203

HELM Safety 203

AIR-Bench 204

Featured Research 206

Beyond Shallow Safety Alignment 206

Improving the Robustness to Persistently Harmful Behaviors in LLMs 207

3.10. Special Topics on RAI 209

AI Agents 209

Identifying the Risks of LM Agents With LM-Simulated Sandboxes 209

Jailbreaking Multimodal Agents With a Single Image 209

Election Misinformation 211

AI Misinformation in the US Elections 211

Rest of World 2024 AI-Generated Election Content 212

CHAPTER 4: Economy 216

Overview 218

Chapter Highlights 219

4.1. What's New in 2024: A Timeline 221

4.2. Jobs 225

AI Labor Demand 225

Global AI Labor Demand 225

US AI Labor Demand by Skill Cluster and Specialized Skill 227

US AI Labor Demand by Sector 230

US AI Labor Demand by State 231

AI Hiring 234

AI Skill Penetration 236

AI Talent 238

Highlight: Measuring AI's Current Economic Integration 244

4.3. Investment 248

Corporate Investment 248

Startup Activity 249

Global Trends 249

Regional Comparison by Funding Amount 253

Regional Comparison by Newly Funded AI Companies 257

Focus Area Analysis 260

4.4. Corporate Activity 262

Industry Usage 262

Use of AI Capabilities 262

Deployment of AI Capabilities 266

AI's Labor Impact 269

4.5. Robot Deployments 274

Aggregate Trends 274

Industrial Robots: Traditional vs. Collaborative Robots 276

By Geographic Area 277

Country-Level Data on Service Robotics 281

CHAPTER 5: Science and Medicine 282

Overview 284

Chapter Highlights 285

5.1. Notable Medical and Biological AI Milestones 287

Protein Sequence Optimization 287

Aviary 288

AlphaProteo 289

Human Brain Mapping 289

Virtual AI Lab 290

GluFormer 291

Evolutionary Scale Modeling v3 (ESM3) 291

AlphaFold 3 292

5.2. The Central Dogma 293

Protein Sequence Analysis 293

AI-Driven Protein Sequence Models 293

Public Databases for Protein Science 295

Research and Publication Trends 296

AI-Driven Protein Science Publications 296

Image and Multimodal AI for Scientific Discovery 297

5.3. Clinical Care, Imaging 298

Data: Sources, Types, and Needs 298

Advanced Modeling Approaches 300

5.4. Clinical Care, Non-Imaging 302

Clinical Knowledge 302

MedQA 302

Highlight: AI Doctors and Cost-Efficiency Considerations 303

Evaluation of LLMs for Healthcare Performance 304

Overview 304

Diagnostic Reasoning With LLMs 306

Highlight: LLMs Influence Diagnostic Reasoning 306

Management Reasoning and Patient Care Decisions 306

Highlight: GPT-4 Assistance on Patient Care Tasks 307

Ambient AI Scribes 308

Deployment, Implementation, Deimplementation 310

FDA Authorization of AI-Enabled Medical Devices 310

Successful Use Cases: Stanford Health Care 310

Screening for Peripheral Arterial Disease 311

Social Determinants of Health 312

Extracting SDoH From EHR and Clinical Notes 312

AI Adoption Across Medical Fields and the Integration of SDoH 313

Synthetic Data 313

Clinical Risk Prediction 313

Drug Discovery 314

Data Generation Platforms 314

Electronic Health Record System 315

Clinical Decision Support 317

5.5. Ethical Considerations 319

Meta Review 319

5.6. AI Foundation Models in Science 322

Highlight: Notable Model Releases 322

CHAPTER 6: Policy and Governance 325

Overview 327

Chapter Highlights 328

6.1. Major Global AI Policy News in 2024 329

6.2. AI and Policymaking 338

Global Legislative Records on AI 338

Overview 338

By Geographic Area 339

Highlight: A Closer Look at Global AI Legislation 340

US Legislative Records 341

Federal Level 341

State Level 342

Highlight: A Closer Look at State-Level AI Legislation 344

Highlight: Anti-deepfake Policymaking 345

Global AI Mentions 347

Overview 347

US Committee Mentions 350

US Regulations 351

Overview 351

By Agency 351

Highlight: A Closer Look at US Federal Regulations 353

6.3. Public Investment in AI 354

Total AI Public Investments 355

Spending Across Agencies and Sectors 362

Highlight: AI Grant Spending in the US 364

CHAPTER 7: Education 366

Overview 368

Chapter Highlights 369

7.1. Background 370

7.2. K-12 CS and AI Education 371

United States 371

Foundational Computer Science 371

Advanced Computer Science 375

Education Standards and Guidance 378

Teacher Perspectives 379

Global 381

Access 381

Guidance 382

7.3. Postsecondary CS and AI Education 384

Degree Graduates 384

United States 384

Global 390

Guidance 394

7.4. Looking Ahead 395

CHAPTER 8: Public Opinion 396

Overview 398

Chapter Highlights 399

8.1. Public Opinion 401

Global Public Opinion 401

AI Products and Services 401

AI and Jobs 407

AI and Livelihood 409

Highlight: Self-Driving Cars 411

8.2. US Policymaker Opinion 412

APPENDIX 416

Chapter 1: Research and Development 418

Chapter 2: Technical Performance 422

Chapter 3: Responsible AI 429

Chapter 4: Economy 433

Chapter 5: Science and Medicine 443

Chapter 6: Policy and Governance 453

Chapter 7: Education 456

Chapter 8: Public Opinion 457

Figures 30

Figure 1.1.1. Number of AI publications in CS worldwide, 2013-23 30

Figure 1.1.2. AI publications in CS (% of total) worldwide, 2013-23 31

Figure 1.1.3. Number of AI publications in CS by venue type, 2013-23 32

Figure 1.1.4. AI publications in CS (% of total) by region, 2013-23 33

Figure 1.1.5. AI publication citations in CS (% of total) by region, 2013-23 34

Figure 1.1.6. AI publications in CS (% of total) by select geographic areas, 2013-23 35

Figure 1.1.7. AI publication citations in CS (% of total) by select geographic areas, 2013-23 36

Figure 1.1.8. AI publications in CS (% of total) by sector, 2013-23 37

Figure 1.1.9. AI publications in CS (% of total) by sector and select geographic areas, 2023 38

Figure 1.1.10. Number of AI publications by select top topics, 2013-23 39

Figure 1.1.11. Number of highly cited publications in top 100 by select geographic areas, 2021-23 40

Figure 1.1.12. Number of highly cited publications in top 100 by sector, 2021-23 41

Figure 1.1.13. Number of highly cited publications in top 100 by organization, 2021-23 42

Figure 1.2.1. Number of AI patents granted worldwide, 2010-23 43

Figure 1.2.2. Granted AI patents (% of world total) by region, 2010-23 44

Figure 1.2.3. Granted AI patents (% of world total) by select geographic areas, 2010-23 45

Figure 1.2.4. Granted AI patents per 100,000 inhabitants by country, 2023 46

Figure 1.2.5. Percentage change of granted AI patents per 100,000 inhabitants by country, 2013 vs. 2023 46

Figure 1.3.1. Number of notable AI models by select geographic areas, 2024 47

Figure 1.3.2. Number of notable AI models by select geographic areas, 2003-24 47

Figure 1.3.3. Number of notable AI models by geographic area, 2003-24 (sum) 48

Figure 1.3.4. Number of notable AI models by sector, 2003-24 49

Figure 1.3.5. Notable AI models (% of total) by sector, 2003-24 49

Figure 1.3.6. Number of notable AI models by organization, 2024 50

Figure 1.3.7. Number of notable AI models by organization, 2014-24 (sum) 50

Figure 1.3.8. Number of notable AI models by access type, 2014-24 51

Figure 1.3.9. Notable AI models (% of total) by access type, 2014-24 52

Figure 1.3.10. Number of notable AI models by training code access type, 2014-24 52

Figure 1.3.11. Number of parameters of notable AI models by sector, 2003-24 53

Figure 1.3.12. Number of parameters of select notable AI models by sector, 2012-24 54

Figure 1.3.13. Training dataset size of notable AI models, 2010-24 55

Figure 1.3.14. Training length of notable AI models, 2010-24 56

Figure 1.3.15. Training compute of notable AI models by sector, 2003-24 57

Figure 1.3.16. Training compute of notable AI models by domain, 2012-24 58

Figure 1.3.17. Training compute of select notable AI models in the United States and China, 2018-24 59

Figure 1.3.18. Estimated median data stocks 60

Figure 1.3.19. Projections of the stock of public text and data usage 61

Figure 1.3.20. Effect of data accumulation on language models pretrained on TinyStories 62

Figure 1.3.21. Factual accuracy: percentage of correct answers in biographies 64

Figure 1.3.22. Inference price across select benchmarks, 2022-24 65

Figure 1.3.23. Output price per million tokens for select models 66

Figure 1.3.24. Estimated training cost of select AI models, 2019-24 67

Figure 1.3.25. Estimated training cost of select AI models, 2016-24 68

Figure 1.3.26. Estimated training cost and compute of select AI models 68

Figure 1.4.1. Peak computational performance of ML hardware for different precisions, 2008-24 69

Figure 1.4.2. Performance of leading Nvidia data center GPUs for machine learning 70

Figure 1.4.3. Price-performance of leading Nvidia data center GPUs for machine learning 71

Figure 1.4.4. Cumulative number of notable AI models trained by accelerator, 2017-24 71

Figure 1.4.5. Energy efficiency of leading machine learning hardward, 2016-24 72

Figure 1.4.6. Total power draw required to train frontier models, 2011-24 73

Figure 1.4.7. Estimated carbon emissions from training select AI models and real-life activities, 2012-24 74

Figure 1.4.8. Estimated carbon emissions and number of parameters by select AI models 75

Figure 1.5.1. Attendance at select AI conferences, 2010-24 76

Figure 1.5.2. Attendance at large conferences, 2010-24 77

Figure 1.5.3. Attendance at small conferences, 2010-24 77

Figure 1.6.1. Number of GitHub AI projects, 2011-24 78

Figure 1.6.2. GitHub AI projects (% of total) by geographic area, 2011-24 79

Figure 1.6.3. Number of GitHub stars in AI projects, 2011-24 80

Figure 1.6.4. Number of GitHub stars by geographic area, 2011-24 81

Figure 2.1.1. (Omit) 89

Figure 2.1.2. (Omit) 89

Figure 2.1.3. (Omit) 89

Figure 2.1.4. (Omit) 89

Figure 2.1.5. (Omit) 89

Figure 2.1.6. (Omit) 90

Figure 2.1.7. (Omit) 90

Figure 2.1.8. (Omit) 90

Figure 2.1.9. (Omit) 90

Figure 2.1.10. (Omit) 90

Figure 2.1.11. (Omit) 90

Figure 2.1.12. (Omit) 91

Figure 2.1.13. (Omit) 91

Figure 2.1.14. (Omit) 91

Figure 2.1.15. (Omit) 91

Figure 2.1.16. (Omit) 91

Figure 2.1.17. (Omit) 92

Figure 2.1.18. (Omit) 92

Figure 2.1.19. (Omit) 92

Figure 2.1.20. (Omit) 92

Figure 2.1.21. (Omit) 92

Figure 2.1.22. (Omit) 92

Figure 2.1.23. (Omit) 93

Figure 2.1.24. (Omit) 93

Figure 5.1.25. (Omit) 93

Figure 2.1.26. (Omit) 93

Figure 2.1.27. (Omit) 93

Figure 2.1.28. (Omit) 93

Figure 2.1.29. (Omit) 94

Figure 2.1.30. (Omit) 94

Figure 2.1.31. (Omit) 94

Figure 2.1.32. (Omit) 94

Figure 2.1.33. Select AI Index technical performance benchmarks vs. human performance 95

Figure 2.1.34. Performance of top closed vs. open models on LMSYS Chatbot Arena 97

Figure 2.1.35. Performance of top closed vs. open models on select benchmarks 97

Figure 2.1.36. Performance of top United States vs. Chinese models on LMSYS Chatbot Arena 98

Figure 2.1.37. Performance of top United States vs. Chinese models on select benchmarks 99

Figure 2.1.38. Smallest AI models scoring above 60% on MMLU, 2022-24 100

Figure 2.1.39. Performance of top models on LMSYS Chatbot Arena by select providers 101

Figure 2.1.40. Five stages of the benchmark lifecycle 103

Figure 2.1.41. Design vs. usability scores across select benchmarks 104

Figure 2.2.1. A sample output from GPT-4o 105

Figure 2.2.2. Gemini 2.0 in an agentic workflow 105

Figure 2.2.3. A sample question from MMLU 106

Figure 2.2.4. MMLU: average accuracy 106

Figure 2.2.5. MMLU-Pro: overall accuracy 107

Figure 2.2.6. A sample model response on the Chatbot Arena Leaderboard 108

Figure 2.2.7. LMSYS Chatbot Arena for LLMs: Elo rating (overall) 108

Figure 2.2.8. Arena-Hard-Auto vs. other benchmarks 109

Figure 2.2.9. Arena-Hard-Auto with no modification 109

Figure 2.2.10. Arena-Hard-Auto with style control 109

Figure 2.2.11. Evaluation framework for WildBench 110

Figure 2.2.12. WildBench: WB-Elo (length controlled) 111

Figure 2.2.13. Chain-of-thought thinking in o1 112

Figure 2.2.14. GPT-4o vs. o1-preview vs. o1 on select benchmarks 113

Figure 2.2.15. Evaluation framework for MixEval 114

Figure 2.2.16. MixEval-Hard on chat models: score 114

Figure 2.2.17. Data composition on the Berkeley Function Calling Leaderboard 115

Figure 2.2.18. Berkeley Function-Calling: overall accuracy 116

Figure 2.2.19. Tasks in the MTEB benchmark 117

Figure 2.2.20. MTEB on English subsets across 56 datasets: average score 118

Figure 2.2.21. RULER: weighted average score (increasing) 119

Figure 2.2.22. RULER: claimed vs. effective context length 119

Figure 2.2.23. Comparing long-context benchmarks 120

Figure 2.2.24. HELMET: average score 120

Figure 2.3.1. Sample question from Visual Commonsense Reasoning (VCR) challenge 121

Figure 2.3.2. Visual Commonsense Reasoning (VCR) task: Q→AR score 122

Figure 2.3.3. Sample tasks on MVBench 122

Figure 2.3.4. MVBench: average accuracy 123

Figure 2.3.5. Which face is real? 124

Figure 2.3.6. Midjourney generations over time: "a hyper-realistic image of Harry Potter" 124

Figure 2.3.7. Sample from the Chatbot Vision Arena 125

Figure 2.3.8. LMSYS Chatbot Arena for LLMs: Elo rating (vision) 125

Figure 2.3.9. Still generations from Stable Video Diffusion 126

Figure 2.3.10. Still generation from Sora 126

Figure 2.3.11. Veo 2: overall preference 127

Figure 2.3.12. Will Smith eating spaghetti, 2023 vs. 2025 127

Figure 2.4.1. Still images from the BBC lip reading sentences 2 dataset 128

Figure 2.4.2. LRS2: word error rate (WER) 129

Figure 2.5.1. Sample HumanEval problem 130

Figure 2.5.2. HumanEval: Pass@1 130

Figure 2.5.3. A sample model input from SWE-bench 131

Figure 2.5.4. SWE-bench: percent solved 131

Figure 2.5.5. Programming tasks in BigCodeBench 132

Figure 2.5.6. BigCodeBench on the hard set: Pass@1 (average) 132

Figure 2.5.7. BigCodeBench on the full set: Pass@1 (average) 132

Figure 2.5.8. LMSYS Chatbot Arena for LLMs: Elo rating (coding) 133

Figure 2.6.1. Sample problems from GSM8K 134

Figure 2.6.2. GSM8K: accuracy 134

Figure 2.6.3. Sample problem from MATH dataset 135

Figure 2.6.4. MATH word problem-solving: accuracy 135

Figure 2.6.5. LMSYS Chatbot Arena for LLMs: Elo rating (Math) 136

Figure 2.6.6. Sample problems from FrontierMath 137

Figure 2.6.7. FrontierMath: percent solved 137

Figure 2.6.8. Number of solved geometry problems in IMO-AG-30 138

Figure 2.7.1. Sample MMMU questions 139

Figure 2.7.2. MMMU on validation set: overall accuracy 139

Figure 2.7.3. Sample chemistry question from GPQA 140

Figure 2.7.4. GPQA on the diamond set: accuracy 140

Figure 2.7.5. Sample ARC-AGI task 141

Figure 2.7.6. ARC-AGI-1 on private evaluation set: high score 142

Figure 2.7.7. Same questions on HLE 143

Figure 2.7.8. Humanity's Last Exam (HLE): accuracy 144

Figure 2.7.9. PlanBench: instances correct 145

Figure 2.8.1. Tasks on VisualAgentBench 146

Figure 2.8.2. VisualAgentBench on the test set: success rate 147

Figure 2.8.3. RE-Bench Process and Flow 147

Figure 2.8.4. RE-Bench: average normalized score@k 148

Figure 2.8.5. Sample questions on GAIA 149

Figure 2.8.6. GAIA: average score 149

Figure 2.9.1. Tasks on VisualAgentBench 150

Figure 2.9.2. RLBench: success rate (18 tasks, 100 demo/task) 151

Figure 2.9.3. Figure robot making coffee 152

Figure 2.9.4. Figure robot assisting in automotive assembly 152

Figure 2.9.5. AutoRT workflow 153

Figure 2.9.6. Speedtests for SARA vs. non-SARA enhanced models 153

Figure 2.9.7. ALOHA-trained robot attempting complex tasks 154

Figure 2.9.8. ALOHA: success rate 154

Figure 2.9.9. Robots playing amateur-level table tennis 155

Figure 2.9.10. GROOT blueprint for synthetic motion generation 156

Figure 2.9.11. Waymo rider-only miles driven without a human driver 157

Figure 2.9.12. Baidu's RT-6 158

Figure 2.9.13. An overview of Bench2Drive 158

Figure 2.9.14. Bench2Drive: driving score 159

Figure 2.9.15. Waymo driver vs. human benchmarks in Phoenix and San Francisco 160

Figure 2.9.16. Waymo driver percent difference to human benchmarks in Phoenix and San Francisco 160

Figure 2.9.17. Comparison of liability insurance claims by type: Waymo driver vs. human-driven vehicles 161

Figure 3.1.1. Responsible AI dimensions, denitions, and examples 167

Figure 3.2.1. Number of reported AI incidents, 2012-24 168

Figure 3.2.2. (Omit) 169

Figure 3.2.3. (Omit) 169

Figure 3.2.4. (Omit) 170

Figure 3.2.5. (Omit) 170

Figure 3.2.6. Reported general capability benchmarks for popular foundation models 171

Figure 3.2.7. Reported safety and responsible AI benchmarks for popular foundation models 171

Figure 3.2.8. HHEM: hallucination rate 172

Figure 3.2.9. Still generations from Stable Video Diffusion 173

Figure 3.2.10. FACTS: factuality score 173

Figure 3.2.11. Sample questions from SimpleQA 174

Figure 3.2.12. SimpleQA: percent of questions 174

Figure 3.3.1. Business functions assigned primary responsibility for AI governance, 2024 175

Figure 3.3.2. Investment in responsible AI by company revenue, 2024 176

Figure 3.3.3. AI risks: considered relevant vs. actively mitigated, 2024 177

Figure 3.3.4. Percentage of organizations that have experienced AI incidents, 2024 178

Figure 3.3.5. Number of AI incidents reported by organizations, 2024 178

Figure 3.3.6. Impact of responsible AI policies in organizations, 2024 179

Figure 3.3.7. Main obstacles to the implementation of responsible AI measures, 2024 180

Figure 3.3.8. Percentage of organizations influenced by AI regulations in responsible AI decision making 181

Figure 3.3.9. AI-related types of incidents reported by organizations in the past two years 182

Figure 3.3.10. Relevance of selected responsible AI risks for organizations, 2024 vs. 2025 183

Figure 3.3.11. Organizational and operational maturity model 184

Figure 3.3.12. Organizational responsible AI maturity distribution, 2024 vs. 2025 184

Figure 3.3.13. Operational responsible AI maturity distribution, 2024 vs. 2025 184

Figure 3.3.14. Organizational attitudes and philosophies surrounding responsible AI 185

Figure 3.4.1. Number of responsible AI papers accepted at select AI conferences, 2019-24 186

Figure 3.4.2. Responsible AI papers accepted (% of total) at select AI conferences by conference, 2019-24 187

Figure 3.4.3. Number of responsible AI papers accepted at select AI conferences by geographic area, 2024 188

Figure 3.4.4. Number of responsible AI papers accepted at select AI conferences by select geographic area, 2019-24 188

Figure 3.4.5. Number of responsible AI papers accepted at select AI conferences by geographic area, 2019-24 (sum) 188

Figure 3.4.6. AI privacy and data governance papers accepted at select AI conferences, 2019-24 189

Figure 3.4.7. AI fairness and bias papers accepted at select AI conferences, 2019-24 190

Figure 3.4.8. AI transparency and explainability papers accepted at select AI conferences, 2019-24 191

Figure 3.4.9. AI security and safety papers accepted at select AI conferences, 2019-24 192

Figure 3.5.1. Notable RAI policymaking milestones 193

Figure 3.6.1. Accuracy of dataset license classifications by select aggregators 194

Figure 3.6.2. Percentage of tokens in the top web domains of C4 by robots.txt restriction category, 2016-24 196

Figure 3.6.3. Percentage of tokens in the top web domains of C4 by terms of service restriction category, 2016-24 196

Figure 3.7.1. Faces and their likelihood of being classified as "criminal" by model and dataset sizes 197

Figure 3.7.2. Effect of dataset scaling on model predictions across demographic groups 198

Figure 3.7.3. Example of implicit bias in LLMs 199

Figure 3.7.4. LLMs implicit bias across stereotypes in four social categories 200

Figure 3.8.1. Foundation Model Transparency Index Scores by Domain, May 2024 201

Figure 3.8.2. Foundation Model Transparency Index Scores by Major Dimensions of Transparency, May 2024 202

Figure 3.9.1. HELM Safety: mean score 203

Figure 3.9.2. AIR-Bench: refusal rate 204

Figure 3.9.3. AIR-Bench: refusal rate across select risk categories 205

Figure 3.9.4. Attack success rate vs. number of prefilled harmful tokens in LLMs 206

Figure 3.9.5. Targeted latent adversarial training in LLMs 207

Figure 3.9.6. General performance on nonadversarial data 207

Figure 3.9.7. Model resistance to jailbreaking attacks 208

Figure 3.10.1. Overview of ToolEmu 209

Figure 3.10.2. Failure incidence of LM agents 210

Figure 3.10.3. Infection ratio by chat round 210

Figure 3.10.4. Conceptualization of ethical concerns around AI and information manipulation 211

Figure 3.10.5. Rest of World 2024 AI elections: summary statistics 212

Figure 3.10.6. (Omit) 213

Figure 3.10.7. (Omit) 213

Figure 3.10.8. (Omit) 214

Figure 3.10.9. (Omit) 214

Figure 3.10.10. (Omit) 215

Figure 3.10.11. (Omit) 215

Figure 4.1.1. (Omit) 221

Figure 4.1.2. (Omit) 221

Figure 4.1.3. (Omit) 221

Figure 4.1.4. (Omit) 221

Figure 4.1.5. (Omit) 221

Figure 4.1.6. (Omit) 222

Figure 4.1.7. (Omit) 222

Figure 4.1.8. (Omit) 222

Figure 4.1.9. (Omit) 222

Figure 4.1.10. (Omit) 222

Figure 4.1.11. (Omit) 222

Figure 4.1.12. (Omit) 223

Figure 4.1.13. (Omit) 223

Figure 4.1.14. (Omit) 223

Figure 4.1.15. (Omit) 223

Figure 4.1.16. (Omit) 223

Figure 4.1.17. (Omit) 223

Figure 4.1.18. (Omit) 223

Figure 4.1.19. (Omit) 224

Figure 4.1.20. (Omit) 224

Figure 4.1.21. (Omit) 224

Figure 4.1.22. (Omit) 224

Figure 4.1.23. (Omit) 224

Figure 4.1.24. (Omit) 224

Figure 4.2.1. AI job postings (% of all job postings) by select geographic areas, 2014-24 (part 1) 225

Figure 4.2.2. AI job postings (% of all job postings) by select geographic areas, 2014-24 (part 2) 226

Figure 4.2.3. AI job postings (% of all job postings) in the United States by skill cluster, 2010-24 227

Figure 4.2.4. Top 10 specialized skills in 2024 AI job postings in the United States, 2012-14 vs. 2024 228

Figure 4.2.5. Generative AI skills in AI job postings in the United States, 2023 vs. 2024 229

Figure 4.2.6. Share of generative AI skills in AI job postings in the United States, 2023 vs. 2024 229

Figure 4.2.7. AI job postings (% of all job postings) in the United States by sector, 2023 vs. 2024 230

Figure 4.2.8. Number of AI job postings in the United States by state, 2024 231

Figure 4.2.9. Percentage of US states job postings in AI, 2024 231

Figure 4.2.10. Percentage of US AI job postings by state, 2024 232

Figure 4.2.11. Percentage of US states' job postings in AI by select US state, 2010-24 232

Figure 4.2.12. Percentage of US AI job postings by select US state, 2010-24 233

Figure 4.2.13. Relative AI hiring rate year-over-year ratio by geographic area, 2024 234

Figure 4.2.14. Relative AI hiring rate year-over-year ratio by geographic area, 2018-24 235

Figure 4.2.15. Relative AI skill penetration rate by geographic area, 2015-24 236

Figure 4.2.16. Relative AI skill penetration rate across gender, 2015-24 237

Figure 4.2.17. AI talent concentration by geographic area, 2024 238

Figure 4.2.18. Percentage change in AI talent concentration by geographic area, 2016 vs. 2024 238

Figure 4.2.19. AI talent concentration by gender and geographic area, 2016-24 239

Figure 4.2.20. Global AI talent representation, 2016-24 240

Figure 4.2.21. AI talent representation by gender and geographic area, 2016-24 241

Figure 4.2.22. Net AI talent migration per 10,000 LinkedIn members by geographic area, 2024 242

Figure 4.2.23. Net AI talent migration per 10,000 LinkedIn members by geographic area, 2019-24 243

Figure 4.2.24/Figure 4.2.23. Occupational representation in Claude usage data vs. US workforce distribution 244

Figure 4.2.25. Occupational usage of Claude by median annual wage 245

Figure 4.2.26. Depth of AI usage across organizations 246

Figure 4.2.27. Percentage of Claude conversations by type of task execution 247

Figure 4.2.28. Distribution of occupational skills exhibited by Claude in conversations 247

Figure 4.3.1. Global corporate investment in AI by investment activity, 2013-24 248

Figure 4.3.2. Global private investment in AI, 2013-24 249

Figure 4.3.3. Global private investment in generative AI, 2019-24 250

Figure 4.3.4. Number of newly funded AI companies in the world, 2013-24 251

Figure 4.3.5. Number of newly funded generative AI companies in the world, 2019-24 251

Figure 4.3.6. Average size of global AI private investment events, 2013-24 252

Figure 4.3.7. Global AI private investment events by funding size, 2023 vs. 2024 252

Figure 4.3.8. Global private investment in AI by geographic area, 2024 253

Figure 4.3.9. Global private investment in AI by geographic area, 2013-24 (sum) 254

Figure 4.3.10. Global private investment in AI by geographic area, 2013-24 255

Figure 4.3.11. Global private investment in generative AI by geographic area, 2019-24 256

Figure 4.3.12. Number of newly funded AI companies by geographic area, 2024 257

Figure 4.3.13. Number of newly funded AI companies by geographic area, 2013-24 (sum) 258

Figure 4.3.14. Number of newly funded AI companies by geographic area, 2013-24 259

Figure 4.3.15. Global private investment in AI by focus area, 2023 vs. 2024 260

Figure 4.3.16. Global private investment in AI by focus area, 2018-24 261

Figure 4.4.1. Share of respondents who say their organization uses AI in at least one function, 2017-24 262

Figure 4.4.2. AI use by industry and function, 2024 263

Figure 4.4.3. Cost decrease and revenue increase from analytical AI use by function, 2024 264

Figure 4.4.4. AI use by organizations in the world, 2023 vs. 2024 265

Figure 4.4.5. Most common generative AI use cases by function, 2024 266

Figure 4.4.6. Cost decrease and revenue increase from generative AI use by function, 2024 267

Figure 4.4.7. Generative AI use by organizations in the world, 2023 vs. 2024 268

Figure 4.4.8. Impact of AI on customer support agents 269

Figure 4.4.9. Impact of AI on scientific innovation 269

Figure 4.4.10. AI's productivity equalizing effects 270

Figure 4.4.11. Distribution of productivity gains from AI use 271

Figure 4.4.12. Expectations about the impact of generative AI on organizations' workforces in the next 3 years, 2024 272

Figure 4.4.13. Expectations about the impact of AI on organizations' workforces in the next 3 years, 2023 vs. 2024 273

Figure 4.5.1. Number of industrial robots installed in the world, 2012-23 274

Figure 4.5.2. Operational stock of industrial robots in the world, 2012-23 275

Figure 4.5.3. Number of industrial robots installed in the world by type, 2017-23 276

Figure 4.5.4. Number of industrial robots installed by geographic area, 2023 277

Figure 4.5.5. Number of new industrial robots installed in top 5 countries, 2011-23 278

Figure 4.5.6. Number of industrial robots installed (China vs. rest of the world), 2016-23 279

Figure 4.5.7. Annual growth rate of industrial robots installed by geographic area, 2022 vs. 2023 280

Figure 4.5.8. Number of service robots installed in the world by application area, 2022 vs. 2023 281

Figure 5.1.1. Single-objective optimization results for fitness optimization 287

Figure 5.1.2. Performance of LLMs and language agents to solve tasks using Aviary environments 288

Figure 5.1.3. AlphaProteo generating successful binders 289

Figure 5.1.4. 3D brain map images 289

Figure 5.1.5. Workflow in AI-based lab 290

Figure 5.1.6. GluFormer versus glucose management indicator 291

Figure 5.1.7. ESM3 models evaluated on protein generation from atomic coordination prompts 291

Figure 5.1.8. AlphaFold 3 vs. baselines for protein-ligand docking 292

Figure 5.2.1. Emergent structure prediction success, CASP15 293

Figure 5.2.2. Size of protein sequencing models, 2020-24 294

Figure 5.2.3. Key protein science databases 295

Figure 5.2.4. Growth of public protein science databases, 2019-25 295

Figure 5.2.5. Proportion of AI-driven protein research in the biological sciences, 2024 296

Figure 5.2.6. Number of foundation models per microscopy techniques, 2023-24 297

Figure 5.3.1. US patient cohorts used to train clinical machine learning algorithms by state, 2015-19 298

Figure 5.3.2. Training dataset token volumes: medical vs. nonmedical language and imaging models 299

Figure 5.3.3. Imaging modeling approaches and notable AI models 300

Figure 5.3.4. Medical disciplines and notable AI models 301

Figure 5.4.1. MedQA: test accuracy 302

Figure 5.4.2. Performance of select LLMs on medical datasets 303

Figure 5.4.3. Enhanced pareto frontier: accuracy vs. cost 303

Figure 5.4.4. Number of publications on large language models in PubMed, 2019-24 304

Figure 5.4.5. Healthcare tasks, NLP and NLU tasks, and dimensions of evaluation acrss 519 studies 305

Figure 5.4.6. LLM performance in clinical diagnosis 306

Figure 5.4.7. Impact of LLM assistance on clinical management 307

Figure 5.4.8. Cumulative Use of the Ambient Artificial Intelligence (AI) Scribe Tool, October 16-December 24, 2023 308

Figure 5.4.9. Impact of AI Scribe on physician EHR usage 309

Figure 5.4.10. Number of AI medical devices approved by the FDA, 1995-2023 310

Figure 5.4.11. Proposed model and workflow for integrating PAD screening into clinical practice 311

Figure 5.4.12. Model performance on in-domain RT test dataset (any SDoH) 312

Figure 5.4.13. (Omit) 313

Figure 5.4.14. Principal component analysis 313

Figure 5.4.15. Percolation threshold prediction and validation based on AI-generated synthetic structures 314

Figure 5.4.16. Areas under the curve for evaluating synthetic heart disease datasets 314

Figure 5.4.17. Predictive model use across primary inpatient EHR vendor 315

Figure 5.4.18. Developer of predictive models across EHR vendor 316

Figure 5.4.19. Number of clinical trials that have included mentions of AI, 2014-24 317

Figure 5.4.20. Number of clinical trials that have included mentions of AI by select geographic areas, 2021-24 318

Figure 5.5.1. (Omit) 319

Figure 5.5.2. Number of medical AI ethics publications, 2020-24 319

Figure 5.5.3. Top 10 ethical concerns discussed in medical AI ethics publications, 2020-24 320

Figure 5.5.4. AI tools discussed in medical AI ethics publications, 2020-24 320

Figure 5.5.5. Number of NIH grants for medical AI ethics by fiscal year, 2020-24 321

Figure 5.5.6. NIH grant funding for medical AI ethics by fiscal year, 2020-24 321

Figure 5.6.1. (Omit) 322

Figure 5.6.2. (Omit) 322

Figure 5.6.3. (Omit) 323

Figure 5.6.4. (Omit) 323

Figure 5.6.5. (Omit) 323

Figure 5.6.6. (Omit) 324

Figure 5.6.7. (Omit) 324

Figure 5.6.8. (Omit) 324

Figure 5.6.9. (Omit) 324

Figure 6.1.1. Singapore plans to invest $1B in AI over 5 years 329

Figure 6.1.2. Abu Dhabi launches $100B AI investment firm 329

Figure 6.1.3. Artificial Intelligence Act is passed by European Parliament 329

Figure 6.1.4. India drops plan to require government approval for launch of new AI models 330

Figure 6.1.5. India launches IndiaAI Mission with $1.25B investment 330

Figure 6.1.6. French government fines Google 250 million euros over use of copyrighted information 330

Figure 6.1.7. U.N. General Assembly adopts resolution promoting "safe, secure, and trustworthy" AI 331

Figure 6.1.8. Canada pledges CA$2.4B investment to ensure country's AI advantage 331

Figure 6.1.9. U.K. AI Safety Institute launches open-source tool for assessing AI model safety 331

Figure 6.1.10. U.K. and South Korea cohost AI safety summit in Seoul 332

Figure 6.1.11. China creates country's largest-ever state-backed investment fund to back its semiconductor industry 332

Figure 6.1.12. European Commission establishes AI Office 332

Figure 6.1.13. U.S. NIST unveils framework to help organizations identify and mitigate GenAI risks 333

Figure 6.1.14. U.S. State Department releases AI Risk Management Profile for Human Rights 333

Figure 6.1.15. U.K. withdraws £1.3B promised for technology and AI infrastructure 333

Figure 6.1.16. U.S. White House launches task force on AI data center infrastructure 334

Figure 6.1.17. California governor signs three bills on AI and elections communications 334

Figure 6.1.18. United Nations adopts Global Digital Compact to ensure an inclusive and secure digital future 334

Figure 6.1.19. California governor vetoes expansive AI legislation 335

Figure 6.1.20. U.S. judge blocks new California AI law over Kamala Harris deepfake 335

Figure 6.1.21. Saudi Arabia announces "Project Transcendence" 335

Figure 6.1.22. European Commission AI Office releases first draft of Code of Practice for General-Purpose AI 336

Figure 6.1.23. U.S. launches international AI safety network with global partners 336

Figure 6.1.24. U.S. increases export controls of semiconductor manufacturing equipment and software to China 336

Figure 6.1.25. U.N. Security Council debates uses of AI in conflicts and calls for global framework 337

Figure 6.2.1. Number of AI-related bills passed into law by country, 2016-24 338

Figure 6.2.2. Number of AI-related bills passed into law in 116 select geographic areas, 2016-24 339

Figure 6.2.3. Number of AI-related bills passed into law in select geographic areas, 2024 339

Figure 6.2.4. Number of AI-related bills passed into law in select geographic areas, 2016-24 (sum) 339

Figure 6.2.5. (Omit) 340

Figure 6.2.6. Number of congressional AI-related proposed bills and passed laws in the United States, 2016-24 341

Figure 6.2.7. Number of AI-related bills passed into law in select US states, 2024 342

Figure 6.2.8. Number of state-level AI-related bills passed into law in the United States by state, 2016-24 (sum) 342

Figure 6.2.9. Number of AI-related bills passed into law by all US states, 2016-24 343

Figure 6.2.10. (Omit) 344

Figure 6.2.11. Number of state-level laws enacted on AI-generated deepfakes in intimate imagery and elections in the United States, 2019-24 345

Figure 6.2.12. State-level laws regulating AI-generated deepfakes in elections in the US by state and status as of 2024 346

Figure 6.2.13. State-level laws regulating AI-generated deepfakes in intimate imagery in the US by state and status as of 2024 346

Figure 6.2.14. Number of mentions of AI in legislative proceedings in 75 select geographic areas, 2016-24 347

Figure 6.2.15. Number of mentions of AI in legislative proceedings by country, 2024 348

Figure 6.2.16. Number of mentions of AI in legislative proceedings by country, 2016-24 (sum) 348

Figure 6.2.17. Mentions of AI in legislative proceedings vs. AI-related bills passed into law in select countries, 2016-24 349

Figure 6.2.18. Mentions of AI in US committee reports by legislative session, 2001-24 350

Figure 6.2.19. Number of AI-related regulations in the United States, 2016-24 351

Figure 6.2.20. Number of AI-related regulations in the United Stated by agency, 2016-24 352

Figure 6.2.21. (Omit) 353

Figure 6.3.1. Public spending on AI-related contracts in select countries, 2013-23 (sum) 355

Figure 6.3.2. Number of AI-related contracts in select countries, 2013-23 (sum) 356

Figure 6.3.3. Median value of public AI-related contracts in select countries, 2013-23 356

Figure 6.3.4. Public spending on AI-related contracts per 100,000 inhabitants in select countries, 2013-23 (sum) 357

Figure 6.3.5. Public spending on AI-related contracts in select countries, 2023 358

Figure 6.3.6. Public spending on AI-related contracts in the United States and Europe, 2013-23 359

Figure 6.3.7. Difference in public spending on AI-related contracts between the United States and Europe, 2013-23 360

Figure 6.3.8. Public spending on AI-related contracts in top 5 European countries, 2013-23 361

Figure 6.3.9. Public spending on AI-related contracts (% of total) in the United States by funding agency, 2013-23 362

Figure 6.3.10. Public spending on AI-related contracts (% of total) in Europe by funding agency activity, 2013-23 363

Figure 6.3.11. US AI-related grants, 2013-23 364

Figure 6.3.12. Public spending on AI-related grants in the United States, 2013-23 364

Figure 6.3.13. Public spending on AI-related grants (% of total) by funding agency, 2013-23 365

Figure 7.1.1. (Omit) 370

Figure 7.2.1. Public high schools teaching foundational CS (% of total in state), 2024 371

Figure 7.2.2. Schools offering foundational CS courses by size, 2024 372

Figure 7.2.3. Schools offering foundational CS courses by geographic area, 2024 372

Figure 7.2.4. Schools offering foundational CS courses by free and reduced lunch student population, 2024 372

Figure 7.2.5. Access to foundational CS courses by race/ethnicity, 2024 373

Figure 7.2.6. Public high school enrollment in CS (% of students), 2024 373

Figure 7.2.7. Public high school enrollment in CS vs. national demographics by race/ethnicity, 2024 374

Figure 7.2.8. Public high school enrollment in CS vs. national demographics by subgroup, 2024 375

Figure 7.2.9. Number of AP computer science exams taken, 2007-23 376

Figure 7.2.10. AP computer science exams taken by race/ethnicity, 2007-23 376

Figure 7.2.11. AP computer science exams taken (% of total responding students) by race/ethnicity, 2007-23 377

Figure 7.2.12. AP computer science exam participation vs. national demographics by race/ethnicity, 2023 377

Figure 7.2.13. Adoption of AI-specific K-12 computer science standards by US state 378

Figure 7.2.14. Percentage of teachers who feel equipped to teach AI by grade level 379

Figure 7.2.15. AI concepts taught in CS classrooms by grade level 379

Figure 7.2.16. Time spent learning AI in CS classrooms by grade level 380

Figure 7.2.17. Availability of CS education by country, 2024 381

Figure 7.2.18. Change in access to CS education by continent, 2019 vs. 2024 382

Figure 7.2.19. AI4K12 guidelines organized around 5 Big Ideas in AI 383

Figure 7.3.1. New CS postsecondary graduates in the United States, 2013-23 385

Figure 7.3.2. CS postsecondary graduates in the United States by gender, 2023 385

Figure 7.3.3. CS vs. all postsecondary graduates in the United States by race/ethnicity (US residents only), 2023 386

Figure 7.3.4. Number of international CS master's students enrolled in US universities, 2022 387

Figure 7.3.5. Number of international CS PhD students enrolled in US universities, 2022 387

Figure 7.3.6. Number of institutions offering AI bachelor's and master's degrees in the US, 2013-23 388

Figure 7.3.7. New AI bachelor's and master's graduates in the United States, 2013-23 388

Figure 7.3.8. Top postsecondary institutions graduating students in AI in 2023 by degree type 389

Figure 7.3.9. New ICT short-cycle tertiary graduates by country, 2022 390

Figure 7.3.10. New ICT bachelor's graduates by country, 2022 391

Figure 7.3.11. New ICT master's graduates by country, 2022 391

Figure 7.3.12. New ICT PhD graduates by country, 2022 392

Figure 7.3.13. Percentage of new ICT postsecondary graduates who are female by country, 2022 393

Figure 8.1.1. Global opinions on products and services using AI (% of total), 2022-24 401

Figure 8.1.2. 'Products and wervices using AI have more benefits than drawbacks,' by country (% of tatal), 2022-24 402

Figure 8.1.3. Opinions about AI by country (% agreeing with statement), 2024 403

Figure 8.1.4. Global opinions about products and services using AI by country, 2024 404

Figure 8.1.5. Percentage point change in opinions about AI by country (% agreeing with statement), 2023-24 405

Figure 8.1.6. Percentage point change in opinions about AI by country (% agreeing with statement), 2022 vs. 2024 406

Figure 8.1.7. Global opinions on the perceived impact of AI on current jobs, 2024 407

Figure 8.1.8. Global opinions on whether AI will change how current jobs are done in the next five years (% agreeing with statement), 2023 vs. 2024 408

Figure 8.1.9. Global opinions on the potential of AI to improve life by country, 2024 409

Figure 8.1.10. Global opinion on the potential of AI to improve the job market vs. individual jobs, 2024 410

Figure 8.1.11. Global opinion on the potential of AI to improve time to get things done vs. individual jobs, 2024 410

Figure 8.1.12. US driver attitude toward self-driving vehicles, 2021-25 411

Figure 8.2.1. Local US officials' support for government regulation of AI by party and year 412

Figure 8.2.2. Local US officials' views on what AI policies would be beneficial for 2025-50 413

Figure 8.2.3. Local US officials' likelihood of making AI policy decisions by party and year 414

Figure 8.2.4. Local US officials' feeling adequately informed to make decisions about AI by party and year 415