Artificial intelligence index report 2023

Title page 1

Contents 10

Report Highlights 11

CHAPTER 1: Research and Development 20

Overview 22

Chapter Highlights 23

1.1. Publications 24

Overview 24

Total Number of AI Publications 24

By Type of Publication 25

By Field of Study 26

By Sector 27

Cross-Country Collaboration 29

Cross-Sector Collaboration 31

AI Journal Publications 32

Overview 32

By Region 33

By Geographic Area 34

Citations 35

AI Conference Publications 36

Overview 36

By Region 37

By Geographic Area 38

Citations 39

AI Repositories 40

Overview 40

By Region 41

By Geographic Area 42

Citations 43

Narrative Highlight: Top Publishing Institutions 44

All Fields 44

Computer Vision 46

Natural Language Processing 47

Speech Recognition 48

1.2. Trends in Significant Machine Learning Systems 49

General Machine Learning Systems 49

System Types 49

Sector Analysis 50

National Affiliation 51

Systems 51

Authorship 53

Parameter Trends 54

Compute Trends 56

Large Language and Multimodal Models 58

National Affiliation 58

Parameter Count 60

Training Compute 61

Training Cost 62

1.3. AI Conferences 64

Conference Attendance 64

1.4. Open-Source AI Software 66

Projects 66

Stars 68

CHAPTER 2: Technical Performance 69

Overview 72

Chapter Highlights 73

2.1. What's New in 2022: A Timeline 74

2.2. Computer Vision-Image 81

Image Classification 81

ImageNet 81

Face Detection and Recognition 82

National Institute of Standards and Technology Face Recognition Vendor Test (FRVT) 83

Deepfake Detection 84

Celeb-DF 84

Human Pose Estimation 85

MPII 85

Semantic Segmentation 86

Cityscapes Challenge, Pixel-Level Semantic Labeling Task 86

Medical Image Segmentation 87

Kvasir-SEG 87

Object Detection 88

Common Objects in Context (COCO) 88

Image Generation 89

CIFAR-10 and STL-10 89

Narrative Highlight: A Closer Look at Progress in Image Generation 90

Visual Reasoning 92

Visual Question Answering (VQA) Challenge 92

Narrative Highlight: The Rise of Capable Multimodal Reasoning Systems 93

Visual Commonsense Reasoning (VCR) 95

2.3. Computer Vision-Video 96

Activity Recognition 96

Kinetics-400, Kinetics-600, Kinetics-700 96

Narrative Highlight: A Closer Look at the Progress of Video Generation 98

2.4. Language 99

English Language Understanding 99

SuperGLUE 99

Reading Comprehension Dataset Requiring Logical Reasoning (ReClor) 100

Narrative Highlight: Just How Much Better Have Language Models Become? 102

Narrative Highlight: Planning and Reasoning in Large Language Models 103

Text Summarization 104

arXiv and PubMed 104

Natural Language Inference 105

Abductive Natural Language Inference (aNLI) 105

Sentiment Analysis 106

SST-5 Fine-Grained Classification 106

Multitask Language Understanding 107

Massive Multitask Language Understanding (MMLU) 107

Machine Translation (MT) 108

Number of Commercially Available MT Systems 108

2.5. Speech 109

Speech Recognition 109

VoxCeleb 109

Narrative Highlight: Whisper 110

2.6. Reinforcement Learning 112

Reinforcement Learning Environments 112

Procgen 112

Narrative Highlight: Benchmark Saturation 114

2.7. Hardware 115

MLPerf Training 115

MLPerf Inference 117

Trends in GPUs: Performance and Price 118

2.8. Environment 120

Environmental Impact of Select Large Language Models 120

Narrative Highlight: Using AI to Optimize Energy Usage 122

2.9. AI for Science 123

Accelerating Fusion Science Through Learned Plasma Control 123

Discovering Novel Algorithms for Matrix Manipulation With AlphaTensor 123

Designing Arithmetic Circuits With Deep Reinforcement Learning 124

Unlocking de Novo Antibody Design With Generative AI 124

CHAPTER 3: Technical AI Ethics 125

Overview 128

Chapter Highlights 129

3.1. Meta-analysis of Fairness and Bias Metrics 130

Number of AI Fairness and Bias Metrics 130

Number of AI Fairness and Bias Metrics (Diagnostic Metrics Vs. Benchmarks) 131

3.2. AI Incidents 133

AI, Algorithmic, and Automation Incidents and Controversies (AIAAIC) Repository: Trends Over Time 133

AIAAIC: Examples of Reported Incidents 134

3.3. Natural Language Processing Bias Metrics 137

Number of Research Papers Using Perspective API 137

Winogender Task From the SuperGLUE Benchmark 138

Model Performance on the Winogender Task From the SuperGLUE Benchmark 138

Performance of Instruction-Tuned Models on Winogender 139

BBQ: The Bias Benchmark for Question Answering 140

Fairness and Bias Trade-Offs in NLP: HELM 142

Fairness in Machine Translation 143

RealToxicityPrompts 144

3.4. Conversational AI Ethical Issues 145

Gender Representation in Chatbots 145

Anthropomorphization in Chatbots 146

Narrative Highlight: Tricking ChatGPT 147

3.5. Fairness and Bias in Text-to-Image Models 148

Fairness in Text-to-Image Models (ImageNet Vs. Instagram) 148

VLStereoSet: StereoSet for Text-to-Image Models 150

Examples of Bias in Text-to-Image Models 152

Stable Diffusion 152

DALL-E 2 153

Midjourney 154

3.6. AI Ethics in China 155

Topics of Concern 155

Strategies for Harm Mitigation 156

Principles Referenced by Chinese Scholars in AI Ethics 157

3.7. AI Ethics Trends at FAccT and NeurIPS 158

ACM FAccT 158

Accepted Submissions by Professional Affiliation 158

Accepted Submissions by Geographic Region 159

NeurIPS 160

Real-World Impact 160

Interpretability and Explainability 161

Causal Effect and Counterfactual Reasoning 162

Privacy 163

Fairness and Bias 164

3.8. Factuality and Truthfulness 165

Automated Fact-Checking Benchmarks: Number of Citations 165

Missing Counterevidence and NLP Fact-Checking 166

TruthfulQA 167

CHAPTER 4: The Economy 168

Overview 170

Chapter Highlights 171

4.1. Jobs 173

AI Labor Demand 173

Global AI Labor Demand 173

U.S. AI Labor Demand by Skill Cluster and Specialized Skill 174

U.S. AI Labor Demand by Sector 176

U.S. AI Labor Demand by State 177

AI Hiring 180

AI Skill Penetration 182

Global Comparison: Aggregate 182

Global Comparison: By Gender 183

4.2. Investment 184

Corporate Investment 184

Startup Activity 187

Global Trend 187

Regional Comparison by Funding Amount 189

Regional Comparison by Newly Funded AI Companies 193

Focus Area Analysis 195

4.3. Corporate Activity 198

Industry Adoption 198

Adoption of AI Capabilities 198

Consideration and Mitigation of Risks From Adopting AI 206

Narrative Highlight: The Effects of GitHub's Copilot on Developer Productivity and Happiness 208

Industry Motivation 210

Perceived Importance of AI 210

AI Investments and Implementation Outcomes 211

Challenges in Starting and Scaling AI Projects 213

Earnings Calls 215

Aggregate Trends 215

Specific Themes 216

Narrative Highlight: What Are Business Leaders Actually Saying About AI? 217

Sentiment Analysis 219

4.4. Robot Installations 220

Aggregate Trends 220

Industrial Robots: Traditional Vs. Collaborative Robots 222

By Geographic Area 223

Narrative Highlight: Country-Level Data on Service Robotics 227

Sectors and Application Types 230

China Vs. United States 232

CHAPTER 5: Education 234

Overview 236

Chapter Highlights 237

5.1. Postsecondary AI Education 238

CS Bachelor's Graduates 238

CS Master's Graduates 240

CS PhD Graduates 242

CS, CE, and Information Faculty 246

Narrative Highlight: Who Funds CS Departments in the U.S.? 255

5.2. K-12 AI Education 257

United States 257

State-Level Trends 257

AP Computer Science 258

Narrative Highlight: The State of International K-12 Education 260

CHAPTER 6: Policy and Governance 263

Overview 265

Chapter Highlights 266

6.1. AI and Policymaking 267

Global Legislative Records on AI 267

By Geographic Area 269

Narrative Highlight: A Closer Look at Global AI Legislation 270

United States Federal AI Legislation 271

United States State-Level AI Legislation 272

Narrative Highlight: A Closer Look at State-Level AI Legislation 275

Global AI Mentions 276

By Geographic Area 277

Narrative Highlight: A Closer Look at Global AI Mentions 279

United States Committee Mentions 280

United States AI Policy Papers 283

By Topic 284

6.2. National AI Strategies 285

Aggregate Trends 285

By Geographic Area 285

6.3. U.S. Public Investment in AI 286

Federal Budget for Nondefense AI R&D 286

U.S. Department of Defense Budget Requests 287

U.S. Government AI-Related Contract Spending 288

Total Contract Spending 288

6.4. U.S. AI-Related Legal Cases 291

Total Cases 291

Geographic Distribution 292

Sector 293

Type of Law 294

Narrative Highlight: Three Significant AI-Related Legal Cases 295

CHAPTER 7: Diversity 296

Overview 298

Chapter Highlights 299

7.1. AI Conferences 300

Women in Machine Learning (WiML) NeurIPS Workshop 300

Workshop Participants 300

Demographic Breakdown 301

7.2. AI Postsecondary Education 305

CS Bachelor's Graduates 305

CS Master's Graduates 307

CS PhD Graduates 309

Narrative Highlight: Disability Status of CS, CE, and Information Students 311

New AI PhDs 312

CS, CE, and Information Faculty 313

7.3. K-12 Education 316

AP Computer Science: Gender 316

AP Computer Science: Ethnicity 318

CHAPTER 8:Public Opinion 319

Overview 321

Chapter Highlights 322

8.1. Survey Data 323

Global Insights 323

AI Products and Services 323

AI: Harm or Help? 327

United States 329

Narrative Highlight: How Does the Natural Language Processing (NLP) Research Community Feel About AI? 334

8.2. Social Media Data 340

Dominant Models 340

Appendix 344

Chapter 1: Research and Development 346

Chapter 2: Technical Performance 352

Chapter 3: Technical AI Ethics 363

Chapter 4: The Economy 366

Chapter 5: Education 375

Chapter 6: Policy and Governance 377

Chapter 7: Diversity 384

Chapter 8: Public Opinion 385

Figures 24

Figure 1.1.1. Number of AI Publications in the World, 2010-21 24

Figure 1.1.2. Number of AI Publications by Type, 2010-21 25

Figure 1.1.3. Number of AI Publications by Field of Study (Excluding Other AI), 2010-21 26

Figure 1.1.4. AI Publications (% of Total) by Sector, 2010-21 27

Figure 1.1.5. AI Publications (% of Total) by Sector and Geographic Area, 2021 28

Figure 1.1.6. United States and China Collaborations in AI Publications, 2010-21 29

Figure 1.1.7. Cross-Country Collaborations in AI Publications (Excluding U.S. and China), 2010-21 30

Figure 1.1.8. Cross-Sector Collaborations in AI Publications, 2010-21 31

Figure 1.1.9. Number of AI Journal Publications, 2010-21 32

Figure 1.1.10. AI Journal Publications (% of World Total) by Region, 2010-21 33

Figure 1.1.11. AI Journal Publications (% of World Total) by Geographic Area, 2010-21 34

Figure 1.1.12. AI Journal Citations (% of World Total) by Geographic Area, 2010-21 35

Figure 1.1.13. Number of AI Conference Publications, 2010-21 36

Figure 1.1.14. AI Conference Publications (% of World Total) by Region, 2010-21 37

Figure 1.1.15. AI Conference Publications (% of World Total) by Geographic Area, 2010-21 38

Figure 1.1.16. AI Conference Citations (% of World Total) by Geographic Area, 2010-21 39

Figure 1.1.17. Number of AI Repository Publications, 2010-21 40

Figure 1.1.18. AI Repository Publications (% of World Total) by Region, 2010-21 41

Figure 1.1.19. AI Repository Publications (% of World Total) by Geographic Area, 2010-21 42

Figure 1.1.20. AI Repository Citations (% of World Total) by Geographic Area, 2010-21 43

Figure 1.1.21. Top Ten Institutions in the World in 2021 Ranked by Number of AI Publications in All Fields, 2010-21 44

Figure 1.1.22. Top Ten Institutions in the World by Number of AI Publications in All Fields, 2021 45

Figure 1.1.23. Top Ten Institutions in the World by Number of AI Publications in Computer Vision, 2021 46

Figure 1.1.24. Top Ten Institutions in the World by Number of AI Publications in Natural Language Processing, 2021 47

Figure 1.1.25. Top Ten Institutions in the World by Number of AI Publications in Speech Recognition, 2021 48

Figure 1.2.1. Number of Significant Machine Learning Systems by Domain, 2022 49

Figure 1.2.2. Number of Significant Machine Learning Systems by Sector, 2002-22 50

Figure 1.2.3. Number of Significant Machine Learning Systems by Country, 2022 51

Figure 1.2.4. Number of Significant Machine Learning Systems by Select Geographic Area, 2002-22 51

Figure 1.2.5. Number of Significant Machine Learning Systems by Country, 2002-22 (Sum) 52

Figure 1.2.6. Number of Authors of Significant Machine Learning Systems by Country, 2022 53

Figure 1.2.7. Number of Authors of Significant Machine Learning Systems by Select Geographic Area, 2002-22 53

Figure 1.2.8. Number of Authors of Significant Machine Learning Systems by Country, 2002-22 (Sum) 53

Figure 1.2.9. Number of Parameters of Significant Machine Learning Systems by Sector, 1950-2022 54

Figure 1.2.10. Number of Parameters of Significant Machine Learning Systems by Domain, 1950-2022 55

Figure 1.2.11. Training Compute (FLOP) of Significant Machine Learning Systems by Sector, 1950-2022 56

Figure 1.2.12. Training Compute (FLOP) of Significant Machine Learning Systems by Domain, 1950-2022 57

Figure 1.2.13. Authors of Select Large Language and Multimodal Models (% of Total) by Country, 2019-22 58

Figure 1.2.14. Timeline and National Affiliation of Select Large Language and Multimodal Model Releases 59

Figure 1.2.15. Number of Parameters of Select Large Language and Multimodal Models, 2019-22 60

Figure 1.2.16. Training Compute (FLOP) of Select Large Language and Multimodal Models, 2019-22 61

Figure 1.2.17. Estimated Training Cost of Select Large Language and Multimodal Models 62

Figure 1.2.18. Estimated Training Cost of Select Large Language and Multimodal Models and Number of Parameters 63

Figure 1.2.19. Estimated Training Cost of Select Large Language and Multimodal Models and Training Compute (FLOP) 63

Figure 1.3.1. Number of Attendees at Select AI Conferences, 2010-22 64

Figure 1.3.2. Attendance at Large Conferences, 2010-22 65

Figure 1.3.3. Attendance at Small Conferences, 2010-22 65

Figure 1.4.1. Number of GitHub AI Projects, 2011-22 66

Figure 1.4.2. GitHub AI Projects (% Total) by Geographic Area, 2011-22 67

Figure 1.4.3. Number of GitHub Stars by Geographic Area, 2011-22 68

Figure 2.1.1. DeepMind Releases AlphaCode 74

Figure 2.1.2. DeepMind Trains Reinforcement Learning Agent to Control Nuclear Fusion Plasma in a Tokamak 74

Figure 2.1.3. IndicNLG Benchmarks Natural Language Generation for Indic Languages 74

Figure 2.1.4. Meta AI Releases Make-A-Scene 75

Figure 2.1.5. Google Releases PaLM 75

Figure 2.1.6. OpenAI Releases DALL-E 2 75

Figure 2.1.7. DeepMind Launches Gato 75

Figure 2.1.8. Google Releases Imagen 76

Figure 2.1.9. 442 Authors Across 132 Institutions Team Up to Launch BIG-bench 76

Figure 2.1.10. GitHub Makes Copilot Available as a Subscription-Based Service for Individual Developers 76

Figure 2.1.11. Nvidia Uses Reinforcement Learning to Design Better-Performing GPUs 77

Figure 2.1.12. Meta Announces 'No Language Left Behind' 77

Figure 2.1.13. Tsinghua Researchers Launch GLM-130B 77

Figure 2.1.14. Stability AI Releases Stable Diffusion 77

Figure 2.1.15. OpenAI Launches Whisper 78

Figure 2.1.16. Meta Releases Make-A-Video 78

Figure 2.1.17. DeepMind Launches AlphaTensor 78

Figure 2.1.18. Google Uses PaLM to Improve the Reasoning of PaLM 79

Figure 2.1.19. International Research Group Releases BLOOM 79

Figure 2.1.20. Stanford Researchers Release HELM 79

Figure 2.1.21. Meta Releases CICERO 80

Figure 2.1.22. OpenAI Launches ChatGPT 80

Figure 2.2.1. A Demonstration of Image Classification 81

Figure 2.2.2. ImageNet Challenge: Top-1 Accuracy 82

Figure 2.2.3. A Demonstration of Face Detection and Recognition 82

Figure 2.2.4. National Institute of Standards and Technology (NIST) Face Recognition Vendor Test (FRVT) : Verification Accuracy by Dataset 83

Figure 2.2.5. Real-Life Deepfake: President Zelenskyy Calling for the Surrender of Ukrainian Soldiers 84

Figure 2.2.6. Celeb-DF: Area Under Curve Score (AUC) 84

Figure 2.2.7. A Demonstration of Human Pose Estimation 85

Figure 2.2.8. MPII: Percentage of Correct Keypoints (PCK) 85

Figure 2.2.9. A Demonstration of Semantic Segmentation 86

Figure 2.2.10. Cityscapes Challenge, Pixel-Level Semantic Labeling Task: Mean Intersection-Over-Union (mIoU) 86

Figure 2.2.11. A Demonstration of Medical Imaging Segmentation 87

Figure 2.2.12. Kvasir-SEG: Mean Dice 87

Figure 2.2.13. A Demonstration of Object Detection 88

Figure 2.2.14. COCO: Mean Average Precision (mAP50) 88

Figure 2.2.15. Which Face Is Real? 89

Figure 2.2.16. CIFAR-10 and STL-10: Fréchet Inception Distance (FID) Score 89

Figure 2.2.17. GAN Progress on Face Generation 90

Figure 2.2.18. Images Generated by DALL-E 2, Stable Diffusion and Midjourney 90

Figure 2.2.19. Notable Text-to-Image Models on MS-COCO 256 × 256 FID-30K: Fréchet Inception Distance (FID) Score 91

Figure 2.2.20. A Collection of Visual Reasoning Tasks 92

Figure 2.2.21. Visual Question Answering (VQA) V2 Test-Dev: Accuracy 92

Figure 2.2.22. BEiT-3 Vs. Previous State-of-the-Art Models 93

Figure 2.2.23. A Collection of Vision-Language Tasks 94

Figure 2.2.24. A Sample Question from the Visual Commonsense Reasoning (VCR) Challenge 95

Figure 2.2.25. Visual Commonsense Reasoning (VCR) Task: Q→AR Score 95

Figure 2.3.1. Example Classes From the Kinetics Dataset 96

Figure 2.3.2. Kinetics-400, Kinetics-600, Kinetics-700: Top-1 Accuracy 97

Figure 2.2.3. Notable Text-to-Video Models on UCF-101: Inception Score (IS) 98

Figure 2.4.1. A Set of SuperGLUE Tasks 99

Figure 2.4.2. SuperGLUE: Score 100

Figure 2.4.3. A Sample Question from the Reading Comprehension Dataset Requiring Logical Reasoning (ReClor) 100

Figure 2.4.4. Reading Comprehension Dataset Requiring Logical Reasoning (ReClor): Accuracy 101

Figure 2.4.5. Select Large Language Models on the Blocksworld Domain: Instances Correct 103

Figure 2.4.6. ArXiv and PubMed: ROUGE-1 104

Figure 2.4.7. Sample Question From the Abductive Natural Language Inference Benchmark (aNLI) 105

Figure 2.4.8. Abductive Natural Language Inference (aNLI): Accuracy 105

Figure 2.4.9. A Sample Sentence from SST 106

Figure 2.4.10. SST-5 Fine-Grained: Accuracy 106

Figure 2.4.11. Sample Questions From MMLU 107

Figure 2.4.12. MMLU: Average Weighted Accuracy 107

Figure 2.4.13. Number of Independent Machine Translation Services 108

Figure 2.5.1. VoxCeleb: Equal Error Rate (EER) 109

Figure 2.5.2. wav2vec 2.0 Large (No LM) Vs. Whisper Large V2 Across Datasets 110

Figure 2.5.3. Notable Models on X→EN Subset of CoVoST 2 110

Figure 2.5.4. Notable Speech Transcription Services on Kincaid46 111

Figure 2.5.5. Notable Models on FLEURS: Language Identification Accuracy 111

Figure 2.6.1. The Different Environments in Procgen 112

Figure 2.6.2. Procgen: Mean of Min-Max Normalized Score 113

Figure 2.6.3. Improvement Over Time on Select AI Index Technical Performance Benchmarks 114

Figure 2.7.1. MLPerf Training Time of Top Systems by Task: Minutes 115

Figure 2.7.2. MLPerf Hardware: Accelerators 116

Figure 2.7.3. MLPerf Best-Performing Hardware for Image Classification: Offline and Server Scenario 117

Figure 2.7.4. MLPerf Best-Performing Hardware for Language Processing: Offline and Server Scenario 117

Figure 2.7.5. MLPerf Best-Performing Hardware for Recommendation: Offline and Server Scenario 117

Figure 2.7.6. MLPerf Best-Performing Hardware for Speech Recognition: Offline and Server Scenario 117

Figure 2.7.7. FP32 (Single Precision) Performance (FLOP/s) by Hardware Release Date, 2003-22 118

Figure 2.7.8. Median FP32 (Single Precision) Performance (FLOP/s), 2003-22 118

Figure 2.7.9. FP32 (Single Precision) Performance (FLOP/s) per U.S. Dollar by Hardware Release Date, 2003-22 119

Figure 2.7.10. Median FP32 (Single Precision) Performance (FLOP/s) per U.S. Dollar, 2003-22 119

Figure 2.8.1. Environmental Impact of Select Machine Learning Models, 2022 120

Figure 2.8.2. CO₂ Equivalent Emissions (Tonnes) by Selected Machine Learning Models and Real Life Examples, 2022 121

Figure 2.8.3. Energy Savings Results Over Time for Select BCOOLER Experiment 122

Figure 2.9.1. Photos of the Variable Configuration Tokamak (TCV) at EPFL 123

Figure 2.9.2. A Demonstration of AlphaTensor's Matrix Manipulation Process 123

Figure 2.9.3. A Juxtaposition of Nvidia Circuits Designed by PrefixRL Vs. EDA Tools 124

Figure 2.9.4. Zero-Shot Generative AI for de Novo Antibody Design 124

Figure 3.1.1. Number of AI Fairness and Bias Metrics, 2016-22 130

Figure 3.1.2. Number of New AI Fairness and Bias Metrics (Diagnostic Metrics Vs. Benchmarks), 2016-22 132

Figure 3.2.1. Number of AI Incidents and Controversies, 2012-21 133

Figure 3.2.2. (Omit) 134

Figure 3.2.3. (Omit) 135

Figure 3.2.4. (Omit) 135

Figure 3.2.5. (Omit) 136

Figure 3.2.6. (Omit) 136

Figure 3.3.1. Number of Research Papers Using Perspective API, 2018-22 137

Figure 3.3.2. Model Performance on the Winogender Task From the SuperGLUE Benchmark 138

Figure 3.3.3. Winogender: Zero Shot Evaluation in the Generative Setting 139

Figure 3.3.4. Bias in Question Answering on BBQ by Identity Characteristic: Ambiguous Contexts 141

Figure 3.3.5. Bias in Question Answering on BBQ by Identity Characteristic: Disambiguated Contexts 141

Figure 3.3.6. Fairness and Bias Tradeoff in NLP by Scenario 142

Figure 3.3.7. Translation Misgendering Performance: Overall, "He," and "She" 143

Figure 3.3.8. RealToxicityPrompts by Model 144

Figure 3.4.1. Gender Representation in Chatbots, 2022 145

Figure 3.4.2. Characterizing Anthropomorphization in Chatbots: Results by Dataset 146

Figure 3.4.3. Tricking ChatGPT Into Building a Dirty Bomb, Part 1 147

Figure 3.4.4. Tricking ChatGPT Into Building a Dirty Bomb, Part 2 147

Figure 3.5.1. Fairness Across Age Groups for Text-to-Image Models: ImageNet Vs. Instagram 149

Figure 3.5.2. Fairness Across Gender/Skin Tone Groups for Text-to-Image Models: ImageNet Vs. Instagram 149

Figure 3.5.3. An Example From VLStereoSet 150

Figure 3.5.4. Stereotypical Bias in Text-to-Image Models on VLStereoSet by Category: Vision-Language Relevance (vlrs) Vs. Bias (vlbs) Score 151

Figure 3.5.5. Bias in Stable Diffusion 152

Figure 3.5.6. Bias in DALL-E 2 153

Figure 3.5.7. Bias in Midjourney, Part 1 154

Figure 3.5.8. Bias in Midjourney, Part 2 154

Figure 3.5.9. Bias in Midjourney, Part 3 154

Figure 3.6.1. Topics of Concern Raised in Chinese AI Ethics Papers 155

Figure 3.6.2. AI Ethics in China: Strategies for Harm Mitigation Related to AI 156

Figure 3.6.3. AI Principles Referenced by Chinese Scholars in AI Ethics 157

Figure 3.7.1. Number of Accepted FAccT Conference Submissions by Affiliation, 2018-22 158

Figure 3.7.2. Number of Accepted FAccT Conference Submissions by Region, 2018-22 159

Figure 3.7.3. NeurIPS Workshop Research Topics: Number of Accepted Papers on Real-World Impacts, 2015-22 160

Figure 3.7.4. NeurIPS Research Topics: Number of Accepted Papers on Interpretability and Explainability, 2015-22 161

Figure 3.7.5. NeurIPS Research Topics: Number of Accepted Papers on Causal Effect and Counterfactual Reasoning, 2015-22 162

Figure 3.7.6. NeurIPS Research Topics: Number of Accepted Papers on Privacy in AI, 2015-22 163

Figure 3.7.7. NeurIPS Research Topics: Number of Accepted Papers on Fairness and Bias in AI, 2015-22 164

Figure 3.8.1. Automated Fact-Checking Benchmarks: Number of Citations, 2017-22 165

Figure 3.8.2. Missing Counterevidence Renders NLP Fact-Checking Unrealistic for Misinformation 166

Figure 3.8.3. Multiple-Choice Task on TruthfulQA by Model: Accuracy 167

Figure 4.1.1. AI Job Postings (% of All Job Postings) by Geographic Area, 2014-22 173

Figure 4.1.2. AI Job Postings (% of All Job Postings) in the United States by Skill Cluster, 2010-22 174

Figure 4.1.3. Top Ten Specialized Skills in 2022 AI Job Postings in the United States, 2010-12 Vs. 2022 175

Figure 4.1.4. Top Ten Specialized Skills in 2022 AI Job Postings in the United States by Skill Share, 2010-12 Vs. 2022 175

Figure 4.1.5. AI Job Postings (% of All Job Postings) in the United States by Sector, 2021 Vs. 2022 176

Figure 4.1.6. Number of AI Job Postings in the United States by State, 2022 177

Figure 4.1.7. Percentage of U.S. States' Job Postings in AI, 2022 177

Figure 4.1.8. Percentage of United States AI Job postings by State, 2022 178

Figure 4.1.9. Percentage of U.S. States' Job Postings in AI by Select U.S. State, 2010-22 178

Figure 4.1.10. Percentage of United States AI Job Postings by Select U.S. State, 2010-22 179

Figure 4.1.11. Relative AI Hiring Index by Geographic Area, 2022 180

Figure 4.1.12. Relative AI Hiring Index by Geographic Area, 2016-22 181

Figure 4.1.13. Relative AI Skill Penetration Rate by Geographic Area, 2015-22 182

Figure 4.1.14. Relative AI Skill Penetration Rate Across Gender, 2015-22 183

Figure 4.2.1. Global Corporate Investment in AI by Investment Activity, 2013-22 184

Figure 4.2.2. Top Five AI Merger/Acquisition Investment Activities, 2022 185

Figure 4.2.3. Top Five AI Minority Stake Investment Activities, 2022 185

Figure 4.2.4. Top Five AI Private Investment Activities, 2022 186

Figure 4.2.5. Top Five AI Public Oering Investment Activities, 2022 186

Figure 4.2.6. Private Investment in AI, 2013-22 187

Figure 4.2.7. Number of Private Investment Events in AI, 2013-22 188

Figure 4.2.8. Number of Newly Funded AI Companies in the World, 2013-22 188

Figure 4.2.9. AI Private Investment Events by Funding Size, 2021 Vs. 2022 189

Figure 4.2.10. Private Investment in AI by Geographic Area, 2022 189

Figure 4.2.11. Private Investment in AI by Geographic Area, 2013-22 (Sum) 190

Figure 4.2.12. Private Investment in AI by Geographic Area, 2013-22 191

Figure 4.2.13. Top AI Private Investment Events in the United States, 2022 192

Figure 4.2.14. Top AI Private Investment Events in the European Union and United Kingdom, 2022 192

Figure 4.2.15. Top AI Private Investment Events in China, 2022 192

Figure 4.2.16. Number of Newly Funded AI Companies by Geographic Area, 2022 193

Figure 4.2.17. Number of Newly Funded AI Companies by Geographic Area, 2013-22 (Sum) 194

Figure 4.2.18. Number of Newly Funded AI Companies by Geographic Area, 2013-22 194

Figure 4.2.19. Private Investment in AI by Focus Area, 2021 Vs. 2022 195

Figure 4.2.20. Private Investment in AI by Focus Area, 2017-22 196

Figure 4.2.21. Private Investment in AI by Focus Area and Geographic Area, 2017-22 197

Figure 4.3.1. Share of Respondents Who Say Their Organizations Have Adopted AI in at Least One Function, 2017-22 198

Figure 4.3.2. Average Number of AI Capabilities That Respondents' Organizations Have Embedded Within at Least One Function or Business Unit, 2018-22 199

Figure 4.3.3. Most Commonly Adopted AI Use Cases by Function, 2022 200

Figure 4.3.4. AI Capabilities Embedded in at Least One Function or Business Unit, 2022 201

Figure 4.3.5. AI Adoption by Industry and Function, 2022 202

Figure 4.3.6. Percentage Point Change in Responses of AI Adoption by Industry and Function 2021 Vs. 2022 203

Figure 4.3.7. Cost Decrease and Revenue Increase From AI Adoption by Function, 2021 204

Figure 4.3.8. AI Adoption by Organizations in the World, 2021 Vs. 2022 205

Figure 4.3.9. Risks From Adopting AI That Organizations Consider Relevant, 2019-22 206

Figure 4.3.10. Risks From Adopting AI That Organizations Take Steps to Mitigate, 2019-22 207

Figure 4.3.11. Measuring Dimensions of Developer Productivity When Using Copilot: Survey Responses, 2022 209

Figure 4.3.12. Summary of the Experiment Process and Results 209

Figure 4.3.13. Importance of AI Solutions for Organizations' Overall Success 210

Figure 4.3.14. Believe AI Enhances Performance and Job Satisfaction, 2022 210

Figure 4.3.15. Expected AI Investment Increase in the Next Fiscal Year 211

Figure 4.3.16. Main Outcomes of AI Implementation, 2022 212

Figure 4.3.17. Top Three Challenges in Starting AI Projects, 2022 213

Figure 4.3.18. Main Barriers in Scaling AI Initiatives, 2022 214

Figure 4.3.19. Number of Fortune 500 Earnings Calls Mentioning AI, 2018-22 215

Figure 4.3.20. Themes for AI Mentions in Fortune 500 Earnings Calls, 2018 Vs. 2022 216

Figure 4.3.21. Sentiment Summary Distribution for AI Mentions in Fortune 500 Earnings Calls by Publication Date, 2018-22 219

Figure 4.4.1. Number of Industrial Robots Installed in the World, 2011-21 220

Figure 4.4.2. Operational Stock of Industrial Robots in the World, 2011-21 221

Figure 4.4.3. Number of Industrial Robots Installed in the World by Type, 2017-21 222

Figure 4.4.4. Number of Industrial Robots Installed by Country, 2021 223

Figure 4.4.5. Number of New Industrial Robots Installed in Top Five Countries, 2011-21 224

Figure 4.4.6. Number of Industrial Robots Installed (China Vs. Rest of the World), 2016-21 225

Figure 4.4.7. Annual Growth Rate of Industrial Robots Installed by Country, 2020 Vs. 2021 226

Figure 4.4.8. Service Robots in Medicine 227

Figure 4.4.9. Service Robots in Professional Cleaning 227

Figure 4.4.10. Service Robots in Maintenance and Inspection 227

Figure 4.4.11. Number of Professional Service Robots Installed in the World by Application Area, 2020 Vs. 2021 228

Figure 4.4.12. Number of Professional Service Robot Manufacturers in Top Countries by Type of Company, 2022 229

Figure 4.4.13. Number of Industrial Robots Installed in the World by Sector, 2019-21 230

Figure 4.4.14. Number of Industrial Robots Installed in the World by Application, 2019-21 231

Figure 4.4.15. Number of Industrial Robots Installed in China by Sector, 2019-21 232

Figure 4.4.16. Number of Industrial Robots Installed in the United States by Sector, 2019-21 233

Figure 5.1.1. New CS Bachelor's Graduates in North America, 2010-21 238

Figure 5.1.2. New International CS Bachelor's Graduates (% of Total) in North America, 2010-21 239

Figure 5.1.3. New CS Master's Graduates in North America, 2010-21 240

Figure 5.1.4. New International CS Master's Graduates (% of Total) in North America, 2010-21 241

Figure 5.1.5. New CS PhD Graduates in North America, 2010-21 242

Figure 5.1.6. New International CS PhD Graduates (% of Total) in North America, 2010-21 243

Figure 5.1.7. New CS PhD Students (% of Total) Specializing in AI, 2010-21 244

Figure 5.1.8. Employment of New AI PhDs in North America by Sector, 2010-21 245

Figure 5.1.9. Employment of New AI PhDs (% of Total) in North America by Sector, 2010-21 245

Figure 5.1.10. Number of CS, CE, and Information Faculty in North America, 2011-21 246

Figure 5.1.11. Number of CS Faculty in the United States, 2011-21 247

Figure 5.1.12. New CS, CE, and Information Faculty Hires in North America, 2011-21 248

Figure 5.1.13. Source of New Faculty in North American CS, CE, and Information Departments, 2011-21 249

Figure 5.1.14. Share of Filled New CS, CE, and Information Faculty Positions in North America, 2011-21 250

Figure 5.1.15. Reason Why New CS, CE, and Information Faculty Positions Remained Uufilled (% OF Total), 2011-21 251

Figure 5.1.16. Median Nine-Month Salary of CS Faculty in United States, 2015-21 252

Figure 5.1.17. New International CS, CE, and Information Tenure-Track Faculty Hires (% of Total) in North America, 2010-21 253

Figure 5.1.18. Faculty Losses in North American CS, CE, and Information Departments, 2011-21 254

Figure 5.1.19. External Funding Sources (% of Total) of CS Departments in United States, 2003-21 255

Figure 5.1.20. Median Total Expenditure From External Sources for Computing Research of U.S. CS Departments, 2011-21 256

Figure 5.2.1. States Requiring That All High Schools Offer a Computer Science Course, 2022 257

Figure 5.2.2. Public High Schools Teaching Computer Science (% of Total in State), 2022 257

Figure 5.2.3. Number of AP Computer Science Exams Taken, 2007-21 258

Figure 5.2.4. Number of AP Computer Science Exams Taken, 2021 259

Figure 5.2.5. Number of AP Computer Science Exams Taken per 100,000 Inhabitants, 2021 259

Figure 5.2.6. Government Implementation of AI Curricula by Country, Status, and Education Level 260

Figure 5.2.7. Time Allocated (% of Total) in K-12 AI Curricula by Topic, 2022 261

Figure 6.1.1. Number of AI-Related Bills Passed Into Law by Country, 2016-22 267

Figure 6.1.2. Number of AI-Related Bills Passed Into Law in 127 Select Countries, 2016-22 268

Figure 6.1.3. Number of AI-Related Bills Passed Into Law in Select Countries, 2022 269

Figure 6.1.4. Number of AI-Related Bills Passed Into Law in Select Countries, 2016-22 (Sum) 269

Figure 6.1.5. AI-Related Legislation From Select Countries, 2022 270

Figure 6.1.6. Number of AI-Related Bills in the United States, 2015-22 (Proposed Vs. Passed) 271

Figure 6.1.7. Number of AI-Related Bills Passed Into Law in Select U.S. States, 2022 272

Figure 6.1.8. Number of AI-Related Bills Passed Into Law in Select U.S. States, 2016-22 (Sum) 273

Figure 6.1.9. Number of State-Level AI-Related Bills Passed Into Law in the United States by State, 2016-22 (Sum) 273

Figure 6.1.10. Number of State-Level AI-Related Bills in the United States, 2015-22 (Proposed Vs. Passed) 274

Figure 6.1.11. AI-Related Legislation From Select States, 2022 275

Figure 6.1.12. Number of Mentions of AI in Legislative Proceedings in 81 Select Countries, 2016-22 276

Figure 6.1.13. Number of Mentions of AI in Legislative Proceedings by Country, 2022 277

Figure 6.1.14. Number of Mentions of AI in Legislative Proceedings by Country, 2016-22 (Sum) 278

Figure 6.1.15. AI-Related Parliamentary Men tions From Select Countries, 2022 279

Figure 6.1.16. Mentions of AI in U.S. Committee Reports by Legislative Session, 2001-22 280

Figure 6.1.17. Mentions of AI in Committee Reports of the U.S. House of Representatives for the 117th Congressional Session, 2021-22 281

Figure 6.1.18. Mentions of AI in Committee Reports of the U.S. Senate for the 117th Congressional Session, 2021-22 281

Figure 6.1.19. Mentions of AI in Committee Reports of the U.S. Senate, 2001-22 (Sum) 282

Figure 6.1.20. Mentions of AI in Committee Reports of the U.S. House of Representatives, 2001-22 (Sum) 282

Figure 6.1.21. Number of AI-Related Policy Papers by U.S.-Based Organizations, 2018-22 283

Figure 6.1.22. Number of AI-Related Policy Papers by U.S.-Based Organization by Topic, 2022 284

Figure 6.2.1. Yearly Release of AI National Strategies by Country 285

Figure 6.2.2. Countries With a National Strategy on AI, 2022 285

Figure 6.2.3. AI National Strategies in Development by Country and Year 285

Figure 6.3.1. U.S. Federal Budget for Nondefense AI R&D, FY 2018-23 286

Figure 6.3.2. U.S. DoD Budget Request for AI-Specific Research, Development, Test, and Evaluation (RDT&E), FY 2020-23 287

Figure 6.3.3. U.S. Government Spending by Segment, FY 2017-22 288

Figure 6.3.4. U.S. Government Spending by Segment, FY 2021 Vs. 2022 289

Figure 6.3.5. Total Value of Contracts, Grants, and OTAs Awarded by the U.S. Government for AI/ML and Autonomy, FY 2017-22 290

Figure 6.4.1. Number of AI-Related Legal Cases in the United States, 2000-22 291

Figure 6.4.2. Number of AI-Related Legal Cases in the United States by State, 2022 292

Figure 6.4.3. Number of AI-Related Legal Cases in the United States by State, 2000-22 (Sum) 293

Figure 6.4.4. Sector at Issue in AI-Related Legal Cases in the United States, 2022 293

Figure 6.4.5. Area of Law of AI-Related Legal Cases in the United States, 2022 294

Figure 7.1.1. Attendance at NeurIPS Women in Machine Learning Workshop, 2010-22 300

Figure 7.1.2. Continent of Residence of Participants at NeurIPS Women in Machine Learning Workshop, 2022 301

Figure 7.1.3. Gender Breakdown of Participants at NeurIPS Women in Machine Learning Workshop, 2022 302

Figure 7.1.4. Professional Positions of Participants at NeurIPS Women in Machine Learning Workshop, 2022 303

Figure 7.1.5. Primary Subject Area of Submissions at NeurIPS Women in Machine Learning Workshop, 2022 304

Figure 7.2.1. Gender of New CS Bachelor's Graduates (% of Total) in North America, 2010-21 305

Figure 7.2.2. Ethnicity of New Resident CS Bachelor's Graduates (% of Total) in North America, 2011-21 306

Figure 7.2.3. Gender of New CS Master's Graduates (% of Total) in North America, 2011-21 307

Figure 7.2.4. Ethnicity of New Resident CS Master's Graduates (% of Total) in North America, 2011-21 308

Figure 7.2.5. Gender of New CS PhD Graduates (% of Total) in North America, 2010-21 309

Figure 7.2.6. Ethnicity of New Resident CS PhD Graduates (% of Total) in North America, 2011-21 310

Figure 7.2.7. CS, CE, and Information Students (% of Total) With Disability Accomodations in North America, 2021 311

Figure 7.2.8. Gender of New AI PhD Graduates (% of Total) in North America, 2010-21 312

Figure 7.2.9. Gender of CS, CE, and Information Faculty (% of Total) in North America, 2011-21 313

Figure 7.2.10. Gender of New CS, CE, and Information Faculty Hires (% of Total) in North America, 2011-21 314

Figure 7.2.11. Ethnicity of Resident CS, CE, and Information Faculty (% of Total) in North America, 2010-21 315

Figure 7.3.1. AP Computer Science Exams Taken (% of Total) by Gender, 2007-21 316

Figure 7.3.2. AP Computer Science Exams Taken by Female Students (% of Total), 2021 317

Figure 7.3.3. AP Computer Science Exams Taken (% of Total Responding Students) by Race/Ethnicity, 2007-21 318

Figure 8.1.1. Global Opinions on Products and Services Using AI (% of Total), 2022 323

Figure 8.1.2. 'Products and services using AI have more benefits than drawbacks,' by Country (% of Total), 2022 324

Figure 8.1.3. Opinions About AI by Country (% Agreeing With Statement), 2022 325

Figure 8.1.4. Opinions About AI by Demographic Group (% Agreeing With Statement), 2022 326

Figure 8.1.5. Views on Whether AI Will 'Mostly Help' or 'Mostly Harm' People in the Next 20 Years Overall and by Gender (% of Total), 2021 327

Figure 8.1.6. Views on Whether AI Will 'Mostly Help' or 'Mostly Harm' People in the Next 20 Years by Region: Ratio of 'Mostly Help'/'Mostly Harm', 2021 328

Figure 8.1.7. Perceptions of the Safety of Self-Driving Cars (% of Total), 2021 328

Figure 8.1.8. Americans' Feelings Toward Increased Use of AI Programs in Daily Life (% of Total), 2022 329

Figure 8.1.9. Americans' Feelings on Potential AI Applications (% of Total), 2022 329

Figure 8.1.10. Americans' Perceptions of Specific AI Use Cases (% of Total), 2022 330

Figure 8.1.11. Main Reason Americans Are Concerned About AI (% of Total), 2022 331

Figure 8.1.12. Main Reason Americans Are Excited About AI (% of Total), 2022 332

Figure 8.1.13. People Whose Experiences and Views Are Considered in the Design of AI Systems (% of Total), 2022 333

Figure 8.1.14. State of the Field According to the NLP Community, 2022 334

Figure 8.1.15. Language Understanding According to the NLP Community, 2022 335

Figure 8.1.16. Ethics According to the NLP Community, 2022 336

Figure 8.1.17. Artifiial General Intelligence (AGI) and Major Risks According to the NLP Community, 2022 337

Figure 8.1.18. Promising Research Programs According to the NLP Community, 2022 338

Figure 8.1.19. Scale, Inductive Bias, and Adjacent Fields According to the NLP Community, 2022 339

Figure 8.2.1. Net Sentiment Score of AI Models by Quarter, 2022 341

Figure 8.2.2. Select Models' Share of AI Social Media Attention by Quarter, 2022 343