Title page

Contents

REPORT HIGHLIGHTS 10

CHAPTER 1: RESEARCH & DEVELOPMENT 13

OVERVIEW 15

CHAPTER HIGHLIGHTS 16

1.1. PUBLICATIONS 17

OVERVIEW 17

Total Number of AI Publications 17

By Type of Publication 18

By Field of Study 19

By Sector 20

Cross-Country Collaboration 22

Cross-Sector Collaboration 23

AI JOURNAL PUBLICATIONS 24

Overview 24

By Region 25

By Geographic Area 26

Citations 27

AI CONFERENCE PUBLICATIONS 28

Overview 28

By Region 29

By Geographic Area 30

Citations 31

AI REPOSITORIES 32

Overview 32

By Region 33

By Geographic Area 34

Citations 35

AI PATENTS 36

Overview 36

By Region and Application Status 37

By Geographic Area and Application Status 39

1.2. CONFERENCES 41

CONFERENCE ATTENDANCE 41

WOMEN IN MACHINE LEARNING (WIML) NEURIPS WORKSHOP 43

Workshop Participants 43

Demographics Breakdown 44

1.3. AI OPEN-SOURCE SOFTWARE LIBRARIES 45

GITHUB STARS 45

CHAPTER 2: TECHNICAL PERFORMANCE 47

OVERVIEW 50

CHAPTER HIGHLIGHTS 51

2.1. COMPUTER VISION-IMAGE 52

IMAGE CLASSIFICATION 52

ImageNet 52

ImageNet: Top-1 Accuracy 52

ImageNet: Top-5 Accuracy 52

IMAGE GENERATION 54

STL-10: Frechet Inception Distance (FID) Score 54

CIFAR-10: Frechet Inception Distance (FID) Score 55

DEEPFAKE DETECTION 56

FaceForensics++ 56

Celeb-DF 57

HUMAN POSE ESTIMATION 57

Leeds Sports Poses: Percentage of Correct Keypoints (PCK) 58

Human3.6M: Average Mean Per Joint Position Error (MPJPE) 59

SEMANTIC SEGMENTATION 60

Cityscapes 60

MEDICAL IMAGE SEGMENTATION 61

CVC-ClinicDB and Kvasir-SEG 61

FACE DETECTION AND RECOGNITION 62

National Institute of Standards and Technology (NIST) Face Recognition Vendor Test (FRVT) 62

FACE DETECTION: EFFECTS OF MASK-WEARING 63

Face Recognition Vendor Test (FRVT): Face-Mask Effects 63

Masked Labeled Faces in the Wild (MLFW) 64

VISUAL REASONING 65

Visual Question Answering (VQA) Challenge 65

2.2. COMPUTER VISION-VIDEO 67

ACTIVITY RECOGNITION 67

Kinetics-400, Kinetics-600, Kinetics-700 67

ActivityNet: Temporal Action Localization Task 69

OBJECT DETECTION 70

Common Object in Context (COCO) 71

You Only Look Once (YOLO) 72

Visual Commonsense Reasoning (VCR) 73

2.3. LANGUAGE 74

ENGLISH LANGUAGE UNDERSTANDING 74

SuperGLUE 74

Stanford Question Answering Dataset (SQuAD) 75

Reading Comprehension Dataset Requiring Logical Reasoning (ReClor) 76

TEXT SUMMARIZATION 78

arXiv 78

PubMed 79

NATURAL LANGUAGE INFERENCE 80

Stanford Natural Language Inference (SNLI) 80

Abductive Natural Language Inference (aNLI) 81

SENTIMENT ANALYSIS 82

SemEval 2014 Task 4 Sub Task 2 82

MACHINE TRANSLATION (MT) 83

WMT 2014, English-German and English-French 84

Number of Commercially Available MT Systems 85

2.4. SPEECH 86

SPEECH RECOGNITION 86

Transcribe Speech: LibriSpeech (Test-Clean and Other Datasets) 86

VoxCeleb 87

2.5. RECOMMENDATION 88

Commercial Recommendation: MovieLens 20M 88

Click-Through Rate Prediction: Criteo 89

2.6. REINFORCEMENT LEARNING 90

REINFORCEMENT LEARNING ENVIRONMENTS 90

Arcade Learning Environment: Atari-57 90

Procgen 91

Human Games: Chess 93

2.7. HARDWARE 94

MLPerf: Training Time 94

MLPerf: Number of Accelerators 96

IMAGENET: Training Cost 97

2.8. ROBOTICS 98

Price Trends in Robotic Arms 98

AI Skills Employed by Robotics Professors 99

CHAPTER 3: TECHNICAL AI ETHICS 100

OVERVIEW 102

ACKNOWLEDGMENT 103

CHAPTER HIGHLIGHTS 105

3.1. META-ANALYSIS OF FAIRNESS AND BIAS METRICS 106

AI ETHICS DIAGNOSTIC METRICS AND BENCHMARKS 107

3.2. NATURAL LANGUAGE PROCESSING BIAS METRICS 109

TOXICITY: REALTOXICITYPROMPTS AND THE PERSPECTIVE API 109

LARGE LANGUAGE MODELS AND TOXICITY 111

DETOXIFICATION OF MODELS CAN NEGATIVELY INFLUENCE PERFORMANCE 113

STEREOSET 114

CROWS-PAIRS 115

WINOGENDER AND WINOBIAS 117

WINOMT: GENDER BIAS IN MACHINE TRANSLATION SYSTEMS 119

WORD AND IMAGE EMBEDDING ASSOCIATION TESTS 120

MULTILINGUAL WORD EMBEDDINGS 122

Mitigating Bias in Word Embeddings With Intrinsic Bias Metrics 122

3.3. AI ETHICS TRENDS AT FACCT AND NEURIPS 123

ACM CONFERENCE ON FAIRNESS, ACCOUNTABILITY, AND TRANSPARENCY (FACCT) 123

NEURIPS WORKSHOPS 125

Interpretability, Explainability, and Causal Reasoning 126

Privacy and Data Collection 127

Fairness and Bias 129

3.4. FACTUALITY AND TRUTHFULNESS 130

FACT-CHECKING WITH AI 130

Measuring Fact-Checking Accuracy With FEVER Benchmark 133

TOWARD TRUTHFUL LANGUAGE MODELS 134

Model Size and Truthfulness 134

MULTIMODAL BIASES IN CONTRASTIVE LANGUAGE-IMAGE PRETRAINING (CLIP) 136

Denigration Harm 136

Gender Bias 136

Propagating Learned Bias Downstream 138

Underperformance on Non-English Languages 138

CHAPTER 4: THE ECONOMY AND EDUCATION 139

OVERVIEW 141

CHAPTER HIGHLIGHTS 142

4.1. JOBS 143

AI HIRING 143

AI LABOR DEMAND 145

Global AI Labor Demand 145

U.S. AI Labor Demand: By Skill Cluster 146

U.S. Labor Demand: By Sector 147

U.S. Labor Demand: By State 147

AI SKILL PENETRATION 149

Global Comparison 149

Global Comparison: By Industry 149

Global Comparison: By Gender 150

4.2. INVESTMENT 151

CORPORATE INVESTMENT 151

STARTUP ACTIVITY 152

Global Trend 152

Regional Comparison by Funding Amount 154

Regional Comparison by Newly Funded AI Companies 156

Focus Area Analysis 158

4.3. CORPORATE ACTIVITY 160

INDUSTRY ADOPTION 160

Global Adoption of AI 160

AI Adoption by Industry and Function 161

Type of AI Capabilities Adopted 162

Consideration and Mitigation of Risks From Adopting AI 163

4.4. AI EDUCATION 165

CS UNDERGRADUATE GRADUATES IN NORTH AMERICA 165

NEW CS PHDS IN NORTH AMERICA 166

New CS PhDs by Specialty 166

New CS PhDs with AI/ML and Robotics/Vision Specialties 167

NEW AI PHDS EMPLOYMENT IN NORTH AMERICA 168

Academia vs. Industry vs. Government 168

DIVERSITY OF NEW AI PHDS IN NORTH AMERICA 169

By Gender 169

By Race/Ethnicity 170

NEW INTERNATIONAL AI PHDS IN NORTH AMERICA 171

CHAPTER 5: AI POLICY AND GOVERNANCE 172

OVERVIEW 174

CHAPTER HIGHLIGHTS 175

5.1. AI AND POLICYMAKING 176

GLOBAL LEGISLATION RECORDS ON AI 176

By Geographic Area 177

Federal AI Legislation in the United States 178

A Closer Look at the Legislation 179

STATE-LEVEL AI LEGISLATION IN THE UNITED STATES 180

By State 181

Sponsorship by Political Party 182

MENTIONS OF AI IN LEGISLATIVE RECORDS 183

AI Mentions in U.S. Congressional Records 183

AI Mentions in Global Legislative Proceedings 184

By Geographic Area 185

U.S. AI POLICY PAPERS 186

By Topic 187

5.2. U.S. PUBLIC INVESTMENT IN AI 188

FEDERAL BUDGET FOR NONDEFENSE AI R&D 188

U.S. DEPARTMENT OF DEFENSE BUDGET REQUEST 189

DOD Top Five Highest-Funded Programs 190

DOD AI R&D Spending by Department 191

U.S. GOVERNMENT AI-RELATED CONTRACT SPENDING 192

Total Contract Spending 192

Contract Spending by Department and Agency 193

Largest Contract for Five Top-Spending Departments in 2021 195

APPENDIX 196

CHAPTER 1: RESEARCH & DEVELOPMENT 198

CHAPTER 2: TECHNICAL PERFORMANCE 200

CHAPTER 3: TECHNICAL AI ETHICS 212

CHAPTER 4: THE ECONOMY AND EDUCATION 216

CHAPTER 5: AI POLICY AND GOVERNANCE 222

Table 4.2.1. (Omit) 153

Table 5.2.1. DOD Top Five Highest-Funded Programs 190

Table 5.2.2. Largest Contract for Five Top-Spending Departments in 2021 195

FIGURE 1.1.1. NUMBER OF AI PUBLICATIONS IN THE WORLD, 2010-21 17

FIGURE 1.1.2. NUMBER OF AI PUBLICATIONS BY TYPE, 2010-21 18

FIGURE 1.1.3. NUMBER OF AI PUBLICATIONS BY FIELD OF STUDY (EXCLUDING OTHER AI), 2010-21 19

FIGURE 1.1.4A. AI PUBLICATIONS (% OF TOTAL) BY SECTOR, 2010-21 20

FIGURE 1.1.4B. AI PUBLICATIONS IN UNITED STATES (% OF TOTAL) BY SECTOR, 2010-21 20

FIGURE 1.1.4C. AI PUBLICATIONS IN CHINA (% OF TOTAL) BY SECTOR, 2010-21 21

FIGURE 1.1.4D. AI PUBLICATIONS IN EUROPEAN UNION AND UNITED KINGDOM (% OF TOTAL) BY SECTOR, 2010-21 21

FIGURE 1.1.5A. UNITED STATES AND CHINA COLLABORATIONS IN AI PUBLICATIONS, 2010-21 22

FIGURE 1.1.5B. CROSS-COUNTRY COLLABORATIONS IN AI PUBLICATIONS (EXCLUDING U.S. AND CHINA), 2010-21 23

FIGURE 1.1.6. CROSS-SECTOR COLLABORATIONS IN AI PUBLICATIONS, 2010-21 23

FIGURE 1.1.7. NUMBER OF AI JOURNAL PUBLICATIONS, 2010-21 24

FIGURE 1.1.8. AI JOURNAL PUBLICATIONS (% OF TOTAL JOURNAL PUBLICATIONS), 2010-21 24

FIGURE 1.1.9. AI JOURNAL PUBLICATIONS (% OF WORLD TOTAL) BY REGION, 2010-21 25

FIGURE 1.1.10. AI JOURNAL PUBLICATIONS (% OF WORLD TOTAL) BY GEOGRAPHIC AREA, 2010-21 26

FIGURE 1.1.11. AI JOURNAL CITATIONS (% OF WORLD TOTAL) BY GEOGRAPHIC AREA, 2010-21 27

FIGURE 1.1.12. NUMBER OF AI CONFERENCE PUBLICATIONS, 2010-21 28

FIGURE 1.1.13. AI CONFERENCE PUBLICATIONS (% OF TOTAL CONFERENCE PUBLICATIONS), 2010-21 28

FIGURE 1.1.14. AI CONFERENCE PUBLICATIONS (% OF WORLD TOTAL) BY REGION, 2010-21 29

FIGURE 1.1.15. AI CONFERENCE PUBLICATIONS (% OF WORLD TOTAL) BY GEOGRAPHIC AREA, 2010-21 30

FIGURE 1.1.16. AI CONFERENCE CITATIONS (% OF WORLD TOTAL) BY GEOGRAPHIC AREA, 2010-21 31

FIGURE 1.1.17. NUMBER OF AI REPOSITORY PUBLICATIONS, 2010-21 32

FIGURE 1.1.18. AI REPOSITORY PUBLICATIONS (% OF TOTAL REPOSITORY PUBLICATIONS), 2010-21 32

FIGURE 1.1.19. AI REPOSITORY PUBLICATIONS (% OF WORLD TOTAL) BY REGION, 2010-21 33

FIGURE 1.1.20. AI REPOSITORY PUBLICATIONS (% OF WORLD TOTAL) BY GEOGRAPHIC AREA, 2010-21 34

FIGURE 1.1.21. AI REPOSITORY CITATIONS (% OF WORLD TOTAL) BY GEOGRAPHIC AREA, 2010-21 35

FIGURE 1.1.22. NUMBER OF AI PATENT FILINGS, 2010-21 36

FIGURE 1.1.23A. AI PATENT FILINGS (% OF WORLD TOTAL) BY REGION, 2010-21 37

FIGURE 1.1.23B. GRANTED AI PATENTS (% OF WORLD TOTAL) BY REGION, 2010-21 (PART 1) 38

FIGURE 1.1.23C. GRANTED AI PATENTS (% OF WORLD TOTAL) BY REGION, 2010-21 (PART 2) 38

FIGURE 1.1.24A. AI PATENT FILINGS (% OF WORLD TOTAL) BY GEOGRAPHIC AREA, 2010-21 39

FIGURE 1.1.24B. GRANTED AI PATENTS (% OF WORLD TOTAL) BY GEOGRAPHIC AREA, 2010-21 39

FIGURE 1.1.24C. AI PATENTS BY APPLICATION STATUS BY GEOGRAPHIC AREA, 2010-21 40

FIGURE 1.2.1. NUMBER OF ATTENDEES AT SELECT AI CONFERENCES, 2010-21 41

FIGURE 1.2.2. ATTENDANCE AT LARGE AI CONFERENCES, 2010-21 42

FIGURE 1.2.3. ATTENDANCE AT SMALL AI CONFERENCES, 2010-21 42

FIGURE 1.2.4. ATTENDANCE AT NEURIPS WOMEN IN MACHINE LEARNING WORKSHOP, 2010-21 43

FIGURE 1.2.5. CONTINENT OF RESIDENCE OF PARTICIPANTS AT NEURIPS WOMEN IN MACHINE LEARNING WORKSHOP, 2021 44

FIGURE 1.2.6. PROFESSIONAL POSITIONS OF PARTICIPANTS AT NEURIPS WOMEN IN MACHINE LEARNING WORKSHOP, 2021 44

FIGURE 1.3.1. NUMBER OF GITHUB STARS BY AI LIBRARY (OVER 40K STARS), 2014-21 45

FIGURE 1.3.2. NUMBER OF GITHUB STARS BY AI LIBRARY (UNDER 40K STARS), 2014-21 46

FIGURE 2.1.1. A DEMONSTRATION OF IMAGE CLASSIFICATION 52

FIGURE 2.1.2. IMAGENET CHALLENGE: TOP-1 ACCURACY 53

FIGURE 2.1.3. IMAGENET CHALLENGE: TOP-5 ACCURACY 53

FIGURE 2.1.4. GAN PROGRESS ON FACE GENERATION 54

FIGURE 2.1.5. STL-10: FRÉCHET INCEPTION DISTANCE (FID) SCORE 54

FIGURE 2.1.6. CIFAR-10: FRÉCHET INCEPTION DISTANCE (FID) SCORE 55

FIGURE 2.1.7. FACEFORENSICS++: ACCURACY 56

FIGURE 2.1.8. CELEB-DF: AREA UNDER CURVE SCORE (AUC) 57

FIGURE 2.1.9. A DEMONSTRATION OF HUMAN POSE ESTIMATION 57

FIGURE 2.1.10. LEEDS SPORTS POSES: PERCENTAGE OF CORRECT KEYPOINTS (PCK) 58

FIGURE 2.1.11. HUMAN3.6M: AVERAGE MEAN PER JOINT POSITION ERROR (MPJPE) 59

FIGURE 2.1.12. A DEMONSTRATION OF SEMANTIC SEGMENTATION 60

FIGURE 2.1.13. CITYSCAPES CHALLENGE, PIXEL-LEVEL SEMANTIC LABELING TASK: MEAN INTERSECTION-OVER-UNION (IOU) 60

FIGURE 2.1.14. A DEMONSTRATION OF KIDNEY SEGMENTATION 61

FIGURE 2.1.15A. CVC-CLINICDB: MEAN DICE 61

FIGURE 2.1.15B. KVASIR-SEG: MEAN DICE 61

FIGURE 2.1.16. NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY (NIST) FACE RECOGNITION VENDOR TEST (FRVT): VERIFICATION ACCURACY... 62

FIGURE 2.1.17. NIST FRVT FACE MASK EFFECTS: FALSE-NON MATCH RATE 63

FIGURE 2.1.18. EXAMPLES OF MASKED FACES IN THE MASKED LABELED FACES IN THE WILD (MLFW) DATABASE 64

FIGURE 2.1.19. STATE-OF-THE-ART FACE DETECTION METHODS ON MASKED LABELED FACES IN THE WILD (MLFW): ACCURACY 64

FIGURE 2.1.20. AN EXAMPLE OF A VISUAL REASONING TASK 65

FIGURE 2.1.21. SAMPLE QUESTIONS IN THE VISUAL QUESTION ANSWERING (VQA) CHALLENGE 65

FIGURE 2.1.22. VISUAL QUESTION ANSWERING (VQA) CHALLENGE: ACCURACY 66

FIGURE 2.2.1. EXAMPLE CLASSES FROM THE KINETICS DATASET 67

FIGURE 2.2.2. KINETICS-400, KINETICS-600, KINETICS-700: TOP-1 ACCURACY 68

FIGURE 2.2.3. ACTIVITYNET, TEMPORAL ACTION LOCALIZATION TASK: MEAN AVERAGE PRECISION (mAP) 69

FIGURE 2.2.4. A DEMONSTRATION OF HOW OBJECT DETECTION APPEARS TO AI SYSTEMS 70

FIGURE 2.2.5. COCO-TEST-DEV: MEAN AVERAGE PRECISION (mAP50) 71

FIGURE 2.2.6. STATE OF THE ART (SOTA) VS. YOU ONLY LOOK ONCE (YOLO): MEAN AVERAGE PRECISION (mAP50) 72

FIGURE 2.2.7. A SAMPLE QUESTION OF THE VISUAL COMMONSENSE REASONING (VCR) CHALLENGE 73

FIGURE 2.2.8. VISUAL COMMONSENSE REASONING (VCR) TASK: Q-〉AR SCORE 73

FIGURE 2.3.1. A SET OF SUPERGLUE TASKS 74

FIGURE 2.3.2. SUPERGLUE: SCORE 75

FIGURE 2.3.3. HARDER QUESTIONS ADDED TO STANFORD QUESTION ANSWERING DATASET (SQUAD) 2.0 75

FIGURE 2.3.4. SQUAD 1.1 AND SQUAD 2.0: F1 SCORE 76

FIGURE 2.3.5. A SAMPLE QUESTION IN READING COMPREHENSION DATASET REQUIRING LOGICAL REASONING (RECLOR) 76

FIGURE 2.3.6. READING COMPREHENSION DATASET REQUIRING LOGICAL REASONING (RECLOR): ACCURACY 77

FIGURE 2.3.7. ARXIV: ROUGE-1 78

FIGURE 2.3.8. PUBMED: ROUGE-1 79

FIGURE 2.3.9. QUESTIONS AND LABELS IN STANFORD NATURAL LANGUAGE INFERENCE (SNLI) 80

FIGURE 2.3.10. STANFORD NATURAL LANGUAGE INFERENCE (SNLI): ACCURACY 81

FIGURE 2.3.11. EXAMPLE QUESTIONS IN ABDUCTIVE NATURAL LANGUAGE INFERENCE (ANLI) 81

FIGURE 2.3.12. ABDUCTIVE NATURAL LANGUAGE INFERENCE (ANLI): ACCURACY 82

FIGURE 2.3.13. A SAMPLE SEMEVAL TASK 82

FIGURE 2.3.14. SEMEVAL 2014 TASK 4 SUB TASK 2: ACCURACY 83

FIGURE 2.3.15. WMT2014, ENGLISH-FRENCH: BLEU SCORE, WMT2014, ENGLISH-GERMAN: BLEU SCORE 84

FIGURE 2.3.16. NUMBER OF INDEPENDENT MACHINE TRANSLATION SERVICES 85

FIGURE 2.4.1. LIBRISPEECH, TEST CLEAN: WORD ERROR RATE (WER), LIBRISPEECH, TEST OTHER: WORD ERROR RATE (WER) 86

FIGURE 2.4.2. VOXCELEB: EQUAL ERROR RATE (EER) 87

FIGURE 2.5.1. MOVIELENS 20M: NORMALIZED DISCOUNTED CUMULATIVE GAIN@100 (NDCG@100) 88

FIGURE 2.5.2. CRITEO: AREA UNDER CURVE SCORE (AUC) 89

FIGURE 2.6.1. ATARI-57: MEAN HUMAN-NORMALIZED SCORE 91

FIGURE 2.6.2. A SCREENSHOT OF THE 16 GAME ENVIRONMENTS IN PROCGEN 91

FIGURE 2.6.3. PROCGEN: MEAN-NORMALIZED SCORE 92

FIGURE 2.6.4. CHESS SOFTWARE ENGINES: ELO SCORE 93

FIGURE 2.7.1. MLPERF TRAINING TIME OF TOP SYSTEMS BY TASK: MINUTES 94

FIGURE 2.7.2. MLPERF: SCALE OF IMPROVEMENT ACROSS TASK 95

FIGURE 2.7.3. MLPERF HARDWARE: ACCELERATORS 96

FIGURE 2.7.4. IMAGENET: TRAINING COST (TO 93% ACCURACY) 97

FIGURE 2.8.1. MEDIAN PRICE OF ROBOTIC ARMS, 2017-21 98

FIGURE 2.8.2. DISTRIBUTION OF ROBOTIC ARM PRICES, 2017-21 99

FIGURE 2.8.3. AI SKILLS EMPLOYED BY ROBOTICS PROFESSORS 99

FIGURE 3.1.1. NUMBER OF AI FAIRNESS AND BIAS METRICS, 2016-21 106

FIGURE 3.1.2. NUMBER OF AI FAIRNESS AND BIAS METRICS (DIAGNOSTIC METRICS VS. BENCHMARKS), 2016-21 108

FIGURE 3.2.1. TOXICITY: REALTOXICITYPROMPTS AND THE PERSPECTIVE API 109

FIGURE 3.2.2. TOXICITY IN LANGUAGE MODELS BY TRAINING DATASET 110

FIGURE 3.2.3A. GOPHER: PROBABILITY OF TOXIC CONTINUATIONS BASED ON PROMPT TOXICITY BY MODEL SIZE 111

FIGURE 3.2.3B. GOPHER: FEW-SHOT TOXICITY CLASSIFICATION ON THE CIVILCOMMENTS DATASET 112

FIGURE 3.2.4. PERPLEXITY: LANGUAGE MODELING PERFORMANCE BY MINORITY GROUPS ON ENGLISH POST-DETOXIFICATION 113

FIGURE 3.2.5. STEREOSET: STEREOTYPE SCORE BY MODEL SIZE 114

FIGURE 3.2.6. CROWS-PAIRS: LANGUAGE MODEL PERFORMANCE ACROSS BIAS ATTRIBUTES 115

FIGURE 3.2.7. BOOKCORPUS AND SMASHWORDS21: SHARE OF BOOKS ABOUT RELIGION IN PRETRAINING DATASETS 116

FIGURE 3.2.8. MODEL PERFORMANCE ON THE WINOGENDER TASK FROM THE SUPERGLUE BENCHMARK 117

FIGURE 3.2.9. WINOBIAS AND WINOGENDER: NUMBER OF CITATIONS, 2018-21 118

FIGURE 3.2.10. WINOMT: GENDER BIAS IN GOOGLE TRANSLATE ACROSS LANGUAGES 119

FIGURE 3.2.11. SENTENCE EMBEDDING ASSOCIATION TEST (SEAT): MEASURING STEREOTYPICAL ASSOCIATIONS WITH EFFECT SIZE 120

FIGURE 3.2.12. GENDER AND RACIAL BIAS IN WORD EMBEDDINGS TRAINED ON 100 YEARS OF TEXT DATA 121

FIGURE 3.2.13. GENDER BIAS IN SPANISH WORD EMBEDDINGS: EMBEDDING SIMILARITY DISTANCE 122

FIGURE 3.3.1. NUMBER OF ACCEPTED FACCT CONFERENCE SUBMISSIONS BY AFFILIATION, 2018-21 123

FIGURE 3.3.2. NUMBER OF ACCEPTED FACCT CONFERENCE SUBMISSIONS BY REGION, 2018-21 124

FIGURE 3.3.3. NEURIPS WORKSHOP RESEARCH TOPICS: NUMBER OF ACCEPTED PAPERS ON REAL-WORLD IMPACTS, 2015-21 125

FIGURE 3.3.4. NEURIPS RESEARCH TOPICS: NUMBER OF ACCEPTED PAPERS ON CAUSAL EFFECT AND COUNTERFACTUAL REASONING, 2015-2021 126

FIGURE 3.3.5. NEURIPS RESEARCH TOPICS: NUMBER OF ACCEPTED PAPERS ON INTERPRETABILITY AND EXPLAINABILITY, 2015-21 127

FIGURE 3.3.6. NEURIPS RESEARCH TOPICS: NUMBER OF ACCEPTED PAPERS ON PRIVACY IN AI, 2015-21 128

FIGURE 3.3.7. NEURIPS RESEARCH TOPICS: NUMBER OF ACCEPTED PAPERS ON FAIRNESS AND BIAS IN AI, 2015-21 129

FIGURE 3.4.1. DATASETS FOR AUTOMATED FACT-CHECKING: GRANULARITY OF LABELS 130

FIGURE 3.4.2. AUTOMATED FACT-CHECKING BENCHMARKS: NUMBER OF CITATIONS, 2017-21 131

FIGURE 3.4.3. NUMBER OF AUTOMATED FACT-CHECKING BENCHMARKS FOR ENGLISH, 2010-21 132

FIGURE 3.4.4. NUMBER OF AUTOMATED FACT-CHECKING BENCHMARKS BY LANGUAGE 132

FIGURE 3.4.5. FACT EXTRACTION AND VERIFICATION (FEVER) BENCHMARK: ACCURACY AND FEVER SCORE, 2018-21 133

FIGURE 3.4.6. TRUTHFULQA MULTIPLE-CHOICE TASK: TRUTHFUL AND INFORMATIVE ANSWERS BY MODEL 134

FIGURE 3.4.7. TRUTHFULQA GENERATION TASK: TRUTHFUL AND INFORMATIVE ANSWERS BY MODEL 135

FIGURE 3.4.8. BIAS IN CLIP: FREQUENCY OF IMAGE LABELS BY GENDER 137

FIGURE 3.4.9. RESULTS OF THE CLIP-EXPERIMENTS PERFORMED WITH THE COLOR IMAGE OF THE ASTRONAUT EILEEN 138

FIGURE 4.1.1. RELATIVE AI HIRING INDEX BY GEOGRAPHIC AREA, 2021 143

FIGURE 4.1.2. RELATIVE AI HIRING INDEX BY GEOGRAPHIC AREA, 2016-21 144

FIGURE 4.1.3. AI JOB POSTINGS (% OF ALL JOB POSTINGS) BY GEOGRAPHIC AREA, 2013-21 145

FIGURE 4.1.4. AI JOB POSTINGS (% OF ALL JOB POSTINGS) IN THE UNITED STATES BY SKILL CLUSTER, 2010-21 146

FIGURE 4.1.5. AI JOB POSTINGS (% OF ALL JOB POSTINGS) IN THE UNITED STATES BY SECTOR, 2021 147

FIGURE 4.1.6. NUMBER OF AI JOB POSTINGS IN THE UNITED STATES BY STATE, 2021 147

FIGURE 4.1.7. AI JOB POSTINGS (TOTAL AND % OF ALL JOB POSTINGS) BY U.S. STATE AND DISTRICT, 2021 148

FIGURE 4.1.8. RELATIVE AI SKILL PENETRATION RATE BY GEOGRAPHIC AREA, 2015-21 149

FIGURE 4.1.9. RELATIVE AI SKILL PENETRATION RATE BY INDUSTRY ACROSS GEOGRAPHIC AREA, 2015-21 150

FIGURE 4.1.10. RELATIVE AI SKILL PENETRATION RATE BY GENDER, 2015-21 150

FIGURE 4.2.1. GLOBAL CORPORATE INVESTMENT IN AI BY INVESTMENT ACTIVITY, 2013-21 151

FIGURE 4.2.2. PRIVATE INVESTMENT IN AI, 2013-21 152

FIGURE 4.2.3. NUMBER OF NEWLY FUNDED AI COMPANIES IN THE WORLD, 2013-21 153

FIGURE 4.2.4. PRIVATE INVESTMENT IN AI BY GEOGRAPHIC AREA, 2021 154

FIGURE 4.2.5. PRIVATE INVESTMENT IN AI BY GEOGRAPHIC AREA, 2013-21 155

FIGURE 4.2.6. PRIVATE INVESTMENT IN AI BY GEOGRAPHIC AREA, 2013-21 155

FIGURE 4.2.7. NUMBER OF NEWLY FUNDED AI COMPANIES BY GEOGRAPHIC AREA, 2021 156

FIGURE 4.2.8. NUMBER OF NEWLY FUNDED AI COMPANIES BY GEOGRAPHIC AREA, 2013-21 (SUM) 157

FIGURE 4.2.9. NUMBER OF NEWLY FUNDED AI COMPANIES BY GEOGRAPHIC AREA, 2013-21 157

FIGURE 4.2.10. PRIVATE INVESTMENT IN AI BY FOCUS AREA, 2020 VS. 2021 158

FIGURE 4.2.11. PRIVATE INVESTMENT IN AI BY FOCUS AREA, 2017-21 (SUM) 158

FIGURE 4.2.12. PRIVATE INVESTMENT IN AI BY FOCUS AREA, 2017-21 159

FIGURE 4.3.1. AI ADOPTION BY ORGANIZATIONS IN THE WORLD, 2020-21 160

FIGURE 4.3.2. AI ADOPTION BY INDUSTRY AND FUNCTION, 2021 161

FIGURE 4.3.3. AI CAPABILITIES EMBEDDED IN STANDARD BUSINESS PROCESSES, 2021 162

FIGURE 4.3.4. RISKS FROM ADOPTING AI THAT ORGANIZATIONS CONSIDER RELEVANT, 2019-21 163

FIGURE 4.3.5. RISKS FROM ADOPTING AI THAT ORGANIZATIONS TAKE STEPS TO MITIGATE, 2019-21 164

FIGURE 4.4.1. NUMBER OF NEW CS UNDERGRADUATE GRADUATES AT DOCTORAL INSTITUTIONS IN NORTH AMERICA, 2010-20 165

FIGURE 4.4.2. NEW CS PHDS (% OF TOTAL) IN THE UNITED STATES BY SPECIALITY, 2020 166

FIGURE 4.4.3. PERCENTAGE POINT CHANGE IN NEW CS PHDS IN THE UNITED STATES BY SPECIALTY, 2010-20 167

FIGURE 4.4.4A. NEW CS PHDS WITH AI/ML AND ROBOTICS/VISION SPECIALTY IN THE UNITED STATES, 2010-20 167

FIGURE 4.4.4B. NEW CS PHDS (% OF TOTAL) WITH AI/ML AND ROBOTICS/VISION SPECIALTY IN THE UNITED STATES, 2010-20 167

FIGURE 4.4.5A. EMPLOYMENT OF NEW AI PHDS TO ACADEMIA, GOVERNMENT, OR INDUSTRY IN NORTH AMERICA, 2010-20 168

FIGURE 4.4.5B. EMPLOYMENT OF NEW AI PHDS (% OF TOTAL) TO ACADEMIA, GOVERNMENT, OR INDUSTRY IN NORTH AMERICA, 2010-20 168

FIGURE 4.4.6. FEMALE NEW AI AND CS PHDS (% OF TOTAL NEW AI AND CS PHDS) IN NORTH AMERICA, 2010-20 169

FIGURE 4.4.7. NEW U.S. AI RESIDENT PHDS (% OF TOTAL) BY RACE/ETHNICITY, 2010-20 170

FIGURE 4.4.8. NEW COMPUTING PHDS, U.S. RESIDENT (% OF TOTAL) BY RACE/ETHNICITY, 2010-20 170

FIGURE 4.4.9. NEW INTERNATIONAL AI PHDS (% OF TOTAL NEW AI PHDS) IN NORTH AMERICA, 2010-20 171

FIGURE 4.4.10. INTERNATIONAL NEW AI PHDS (% OF TOTAL) IN THE UNITED STATES BY LOCATION OF EMPLOYMENT, 2020 171

FIGURE 5.1.1. NUMBER OF AI-RELATED BILLS PASSED INTO LAW IN 25 SELECT COUNTRIES, 2016-21 176

FIGURE 5.1.2A. NUMBER OF AI-RELATED BILLS PASSED INTO LAW IN SELECT COUNTRIES, 2021 177

FIGURE 5.1.2B. NUMBER OF AI-RELATED BILLS PASSED INTO LAW IN SELECT COUNTRIES, 2016-21 (SUM) 178

FIGURE 5.1.3. NUMBER OF AI-RELATED BILLS IN THE UNITED STATES, 2015-21 (PROPOSED VS. PASSED) 178

FIGURE 5.1.4. STATE-LEVEL AI LEGISLATION IN THE UNITED STATES 180

FIGURE 5.1.5. NUMBER OF STATE-LEVEL PROPOSED AI-RELATED BILLS IN THE UNITED STATES BY STATE, 2012-21 (SUM) 181

FIGURE 5.1.6. NUMBER OF STATE-LEVEL PROPOSED AI-RELATED BILLS IN THE UNITED STATES BY STATE, 2021 181

FIGURE 5.1.7. NUMBER OF STATE-LEVEL PROPOSED AI-RELATED BILLS IN THE UNITED STATES BY SPONSOR PARTY, 2012-21 182

FIGURE 5.1.8. MENTIONS OF AI IN THE U.S. CONGRESSIONAL RECORD BY LEGISLATIVE SESSION, 2001-21 183

FIGURE 5.1.9. NUMBER OF MENTIONS OF AI IN LEGISLATIVE PROCEEDINGS IN 25 SELECT COUNTRIES, 2016-21 184

FIGURE 5.1.10A. NUMBER OF MENTIONS OF AI IN LEGISLATIVE PROCEEDINGS IN SELECT COUNTRIES, 2021 185

FIGURE 5.1.10B. NUMBER OF MENTIONS OF AI IN LEGISLATIVE PROCEEDINGS IN SELECT COUNTRIES, 2016-2021 (SUM) 185

FIGURE 5.1.11. NUMBER OF AI-RELATED POLICY PAPERS BY U.S.-BASED ORGANIZATIONS, 2018-21 186

FIGURE 5.1.12. NUMBER OF AI-RELATED POLICY PAPERS BY U.S.-BASED ORGANIZATIONS BY TOPIC, 2021 187

FIGURE 5.2.1. U.S. FEDERAL BUDGET FOR NONDEFENSE AI R&D, FY 2018-22 188

FIGURE 5.2.2. U.S. DOD BUDGET FOR AI-SPECIFIC RESEARCH, DEVELOPMENT, TEST AND EVALUATION (RDT&E), FY 2020-22 189

FIGURE 5.2.3. U.S. DOD BUDGET FOR AI-SPECIFIC RESEARCH, DEVELOPMENT, TEST AND EVALUATION (RDT&E) BY DEPARTMENT, FY 2020-22 191

FIGURE 5.2.4. U.S. GOVERNMENT TOTAL CONTRACT SPENDING ON AI, FY 2000-21 192

FIGURE 5.2.5. TOP CONTRACT SPENDING ON AI BY U.S. GOVERNMENT DEPARTMENT AND AGENCY, 2021 193

FIGURE 5.2.6. TOP CONTRACT SPENDING ON AI BY U.S. GOVERNMENT DEPARTMENT AND AGENCY, 2000-21 (SUM) 194