-
Notifications
You must be signed in to change notification settings - Fork 5
/
deck.json
1506 lines (1506 loc) · 129 KB
/
deck.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[
{
"category": "Technique & Processes",
"id": 1,
"colour": "83b3db",
"cards": [
{
"question": "Is the task or assignment completely clear?",
"threatif": "No",
"explanation": "* Is the problem you want to solve well defined?\n* Are the possible benefits clear?",
"recommendation": "* Clearly define the problem and outcome you are optimizing for.\n* Assess if your AI system will be well-suited for this purpose.\n* Always discuss if there are alternative ways to solve the problem.\n* Define success! Working with individuals who may be directly affected can help you identify an appropriate way to measure success.\n* Make sure there is a stakeholder involved (product owner for instance) with enough knowledge of the business and a clear vision about what the model needs to do.\n* Did you try analytics first? In this context analytics could also offer inspiring views that can help you decide on the next steps. They can be a good source of information and are sometimes enough to solve the problem without the need of AI/ML.",
"sources": "",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Can we assure that the data that we need is complete and trustworthy?",
"threatif": "No",
"explanation": "Can you avoid the known principle of “garbage in, garbage out”? Your AI system is only as reliable as the data it works with.",
"recommendation": "* Verify the data sources:\n * Is there information missing within the dataset?\n * Are all the necessary classes represented?\n * Does the data belong to the correct time frame and geographical coverage?\n * Evaluate which extra data you need to collect/receive.\n* Carefully consider representation schemes, especially in cases of text, video, APIs, and sensors. Text representation schemes are not all the same. If your system is counting on ASCII and it gets Unicode, will your system recognize the incorrect encoding? Source: [BerryVilleiML](https://berryvilleiml.com/interactive/)",
"sources": "",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input"
]
},
{
"question": "Can the data be representative of the different groups/populations?",
"threatif": "No",
"explanation": "* It is important to reduce the risk of bias and different types of discrimination. Did you consider diversity and representativeness of users/individuals in the data?\n* When applying statistical generalisation, the risk exists of making inferences due to misrepresentation, for instance: a postal code where mostly young families live can discriminate the few old families living there because they are not properly represented in the group.",
"recommendation": "* Who is covered and who is underrepresented?\n* Prevent disparate impact: when the output of a member of a minority group is disparate compared to representation of the group. Consider measuring the accuracy from minority classes too instead of measuring only the total accuracy. Adjusting the weighting factors to avoid disparate impact can result in positive discrimination which has also its own issues: disparate treatment.\n* One approach to addressing the problem of class imbalance is to randomly resample the training dataset. This technique can help to rebalance the class distribution when classes are under or over represented:\n - random oversampling (i.e. duplicating samples from the minority class)\n - random undersampling (i.e. deleting samples from the majority class)\n* There are trade-offs when determining an AI system’s metrics for success. It is important to balance performance metrics against the risk of negatively impacting vulnerable populations.\n* When using techniques like statistical generalisation is important to know your data well, and get familiarised with who is and who is not represented in the samples. Check the samples for expectations that can be easily verified. For example, if half the population is known to be female, then you can check if approximately half the sample is female.",
"sources": "**Related to disparate impact**\n[AI Fairness - Explanation of Disparate Impact Remover](https://towardsdatascience.com/ai-fairness-explanation-of-disparate-impact-remover-ce0da59451f1)\n[Mitigating Bias in AI/ML Models with Disparate Impact Analysis](https://medium.com/@kguruswamy_37814/mitigating-bias-in-ai-ml-models-with-disparate-impact-analysis-9920212ee01c)\n[Certifying and removing disparate impact](https://arxiv.org/abs/1412.3756)\n[Avoiding Disparate Impact with Counterfactual Distributions](https://oconnell.fas.harvard.edu/files/hao/files/wesgai.pdf)\n\n**Related to random resampling**\n[Oversampling and Undersampling](https://towardsdatascience.com/oversampling-and-undersampling-5e2bbaf56dcf)\n[Random Oversampling and Undersampling for Imbalanced Classification](https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/)\n\n**Related to Statistical Generalization**\n[Generalization in quantitative and qualitative research: Myths and strategies](https://core.ac.uk/download/pdf/49282746.pdf)\n[Generalizing Statistical Results to the Entire Population](https://www.dummies.com/article/academics-the-arts/math/statistics/generalizing-statistical-results-to-the-entire-population-201267/)",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Have we identified all the important stakeholders needed in this phase of the project?",
"threatif": "No",
"explanation": "* Do you have all the necessary stakeholders on board? Not having the right people that can give the necessary input can put the design of the AI system in danger.\n* Think for instance when attributes or variables need to be selected, or when you need to understand the different data contexts.\n* Data scientists should not be the only ones making assumptions about variables, it should really be a team effort.",
"recommendation": "* Identify and involve on time the people that you need during the whole life cycle of the AI system. This will avoid unnecessary rework and frustrations.\n* Identifying who’s responsible for making the decisions and how much control they have over the decision-making process allows for a more evident tracking of responsibility in the AI’s development process.",
"sources": "",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Does the model need to be explainable for the users or affected persons?",
"threatif": "Yes",
"explanation": "Do you need to be able to give a clear explanation to the user about the logic that the AI system used to reach a certain decision? And can that decision have a big impact on the user?",
"recommendation": "* Evaluate the type of models that you could use to solve the problem as specified in your task.\n* Consider what the impact is if certain black box models cannot be used and interpretability tools do not offer sufficient results. You might need to evaluate a possible change in strategy.\n* Data scientists can evaluate the impact from a technical perspective and discuss this with the rest of stakeholders. The decision keeps being a team effort.",
"sources": "[Explainable Artificial Intelligence (XAI)](https://www.darpa.mil/program/explainable-artificial-intelligence)\n[LIME](https://github.com/marcotcr/lime)\n[Why Should I Trust You? Explaining the Predictions of Any Classifier](https://arxiv.org/pdf/1602.04938.pdf)\n[SHAP and LIME: An Evaluation of Discriminative Power in Credit Risk](https://www.frontiersin.org/articles/10.3389/frai.2021.752558/full)\n[Explainable AI](https://www.ibm.com/watson/explainable-ai)\n[Explainable AI - The TAILOR Handbook of Trustworthy AI](http://tailor.isti.cnr.it/handbookTAI/T3.1/T3.1.html)",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we preventing Data Leakage?",
"threatif": "No",
"explanation": " Data Leakage is present when your features contain information that your model should not legitimately be allowed to use, leading to overestimation of the model's performance.",
"recommendation": "* Avoid using proxies for the outcome variable as a feature.\n* Do not use the entire data set for imputations, data-based transformations or feature selection.\n* Avoid doing standard k-fold cross-validation when you have temporal data.\n* Avoid using data that happened before model training time but is not available until later. This is common where there is delay in data collection.\n* Do not use data in the training set based on information from the future: if X happened after Y, you shouldn’t build a model that uses X to predict Y.",
"sources": "[Leakage in data mining: formulation, detection, and avoidance](https://www.cs.umb.edu/~ding/history/470_670_fall_2011/papers/cs670_Tran_PreferredPaper_LeakingInDataMining.pdf)\n[The Treachery of Leakage](https://medium.com/@colin.fraser/the-treachery-of-leakage-56a2d7c4e931)\n[Top 10 ways your Machine Learning models may have leakage](http://www.rayidghani.com/2020/01/24/top-10-ways-your-machine-learning-models-may-have-leakage/)\n[Leakage and the Reproducibility Crisis in ML-based Science](https://reproducible.cs.princeton.edu/)",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we preventing Concept and Data Drift?",
"threatif": "No",
"explanation": "* Data Drift weakens performance because the model receives data on which it hasn’t been trained.\n* With Concept Drift the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways causing accuracy issues.",
"recommendation": "* Select an appropriate drift detection algorithm and apply it separately to labels, model’s predictions and data features.\n* Incorporate monitoring mechanisms to detect potential errors early.",
"sources": "[Data Drift vs. Concept Drift](https://deepchecks.com/data-drift-vs-concept-drift-what-are-the-main-differences/)\n[Characterizing Concept Drift](https://www.researchgate.net/publication/283761478_Characterizing_Concept_Drift)\n[Inferring Concept Drift Without Labeled Data](https://concept-drift.fastforwardlabs.com/)\n[Automatic Learning to Detect Concept Drift](https://arxiv.org/abs/2105.01419)\n[From concept drift to model degradation: An overview on performance-aware drift detectors](https://www.sciencedirect.com/science/article/pii/S0950705122002854)\n[Learning under Concept Drift: A Review](https://arxiv.org/abs/2004.05785)\n[Detect data drift (preview) on datasets](https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-monitor-datasets?tabs=python)",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Once our model is running, can we keep feeding it data?",
"threatif": "No",
"explanation": "* Will you use the output from other models to feed the model again (looping)? or will you use other sources ?\n* Are you sure this data will be continuously available?",
"recommendation": "* Consider how the model will keep learning. Design a strategy to prevent issues with the next steps.\n* Imagine you planned to feed your model with input obtained by mining surveys and it appears these surveys contain a lot of free text fields. To prepare that data and avoid issues (bias, inaccuracies, etc) you might need extra time. Consider these type of scenarios that could impact the whole life cycle of your product!",
"sources": "[Text Mining in Survey Data](https://www.surveypractice.org/article/6384-text-mining-in-survey-data)",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Output"
]
},
{
"question": "Is human intervention necessary to oversee the automatic decision making (ADM) process of the AI system?",
"threatif": "Yes",
"explanation": "* Do humans need to review the process and the decisions of the AI system? Consider the impact that this could have for the organisation.\n* Do you have enough capacitated employees available for this role?",
"recommendation": "It is important that people are available for this role and that they receive specific training on how to exercise oversight. The training should teach them how to perform the oversight without being biased by the decision of the AI system (automation bias).",
"sources": "[Automation Bias](https://en.wikipedia.org/wiki/Automation_bias)\n[The Flaws of Policies Requiring Human Oversight of Government Algorithms](https://arxiv.org/ftp/arxiv/papers/2109/2109.05067.pdf)\n[The False Comfort of Human Oversight as an Antidote to AI Harm](https://slate.com/technology/2021/06/human-oversight-artificial-intelligence-laws.html)",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Output"
]
},
{
"question": "Could the channels that we will use to collect real-time input data fail?",
"threatif": "Yes",
"explanation": "* Are these channels trustworthy?\n* What will happen in case of failure?\n* Think for instance about IoT devices used as sensors.",
"recommendation": "* If you are collecting/receiving data from sensors, consider estimating the impact it could have on your model if any of the sensors fail and your input data gets interrupted or corrupted.\n* Sensor blinding attacks are one example of a risk faced by poorly designed input gathering systems. Note that consistent feature identification related to sensors is likely to require human calibration. Source: [BerryVilleiML](https://berryvilleiml.com/)",
"sources": "",
"qr": "",
"categories": [
"Technique & Processes",
"Security"
],
"phases": [
"Design",
"Input",
"Output"
]
},
{
"question": "When datasets from external sources are updated, can we receive and process the new data on time?",
"threatif": "No",
"explanation": "* This could be especially risky in health and finance environments. How much change are you expecting in the data you receive?\n* How can you make sure that you receive the updates on time?",
"recommendation": "Not only do you need to be able to trust the sources but you also need to design a process in which data is prepared on time to be used in the model and where you can timely consider the impact it could have in the output of the model, especially when this could have a negative impact on the users. This process can be designed once you know how often changes in the data can be expected and how big the changes are.",
"sources": "",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Can we confirm the legitimacy of the data sources that we need?",
"threatif": "No",
"explanation": "* Data lineage can be necessary to demonstrate trust as part of your information transparency policy, but it can also be very important when it comes to assessing impact on the data flow. If sources are not verified and legitimised you could run risks such as data being wrongly labelled for instance.\n* Do you know where you need to get the data from? Who is responsible for the collection, maintenance and dissemination? Are the sources verified? Do you have the right agreements in place? Are you allowed to receive or collect that data? Also keep ethical considerations in mind!",
"recommendation": "* Develop a robust understanding of your relevant data feeds, flows and structures such that if any changes occur to the model data inputs, you can assess any potential impact on model performance. In case of third party AI systems contact your vendor to ask for this information.\n* If you are using synthetic data you should know how it was created and the properties it has. Also keep in mind that synthetic data might not be the answer to all your privacy related problems; synthetic data does not always provide a better trade-off between privacy and utility than traditional anonymisation techniques.\n* Do you need to share models and combine them? The usage of Model Cards and Datasheets can help providing the source information.",
"sources": "[Providing Assurance and Scrutability on Shared Data and Machine Learning Models with Verifiable Credentials](https://arxiv.org/pdf/2105.06370.pdf)\n[Synthetic Data – Anonymisation Groundhog Day](https://arxiv.org/pdf/2011.07018.pdf)\n[Model Cards](https://modelcards.withgoogle.com/about)\n[Model Cards for Model Reporting](https://arxiv.org/pdf/1810.03993.pdf)\n[Datasheets for Datasets](https://arxiv.org/pdf/1803.09010.pdf)",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input"
]
},
{
"question": "Do we have enough dedicated resources to monitor the algorithm?",
"threatif": "No",
"explanation": "Do you already have a process in place to monitor the quality of the output and system errors? Do you have resources to do this? Not having the right process and resources in place could have an impact on the project deadline, the organisation and the users.",
"recommendation": "* Put a well-defined process in place to monitor if the AI system is meeting the intended goals.\n* Define failsafe fallback plans to address AI system errors of whatever origin and put governance procedures in place to trigger them.\n* Put measure in places to continuously assess the quality of the output data: e.g. check that predictions scores are within expected ranges; anomaly detection in output and reassign input data leading to the detected anomaly.\n* Does the data measure what you need to measure? You could get measurement errors if data is not correctly labelled.",
"sources": "",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Can we collect all the data that we need for the purpose of the algorithm?",
"threatif": "No",
"explanation": "Could you face difficulties obtaining certain type of data? This could be due to different reasons such as legal, proprietary, financial, physical, technical, etc. This could put the whole project in danger.",
"recommendation": "In the early phases of the project (as soon as the task becomes more clear), start considering which raw data and types of datasets you might need. You might not have the definitive answer until you have tested the model, but it will already help to avoid extra delays and surprises. You might have to involve your legal and financial department. Remember that this is a team effort.",
"sources": "",
"qr": "",
"categories": [
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Output"
]
}
]
},
{
"category": "Accessibility",
"id": 2,
"colour": "7fccdc",
"cards": [
{
"question": "Can our system's user interface be used by those with special needs or disabilities?",
"threatif": "No",
"explanation": "* Does your AI system need to be accessible and usable for users of assistive technologies (such as screen readers)?\n* Is it possible to provide text alternatives for instance?",
"recommendation": "* Implement Universal Design principles during every step of the planning and development process. This is not only important for web interfaces but also when AI systems/robots assist individuals.\n* Test the accessibility of your design with different users (also with disabilities).",
"sources": "[A Proposal of Accessibility Guidelines for Human-Robot Interaction](https://www.mdpi.com/2079-9292/10/5/561/htm)\n[ISO/IEC 40500:2012 Information technology — W3C Web Content Accessibility Guidelines (WCAG) 2.0](https://www.iso.org/standard/58625.html)\n[ISO/IEC GUIDE 71:2001 Guidelines for standards developers to address the needs of older persons and persons with disabilities](https://www.iso.org/standard/33987.html)\n[ISO 9241-171:2008(en) Ergonomics of human-system interaction](https://www.iso.org/obp/ui/#iso:std:iso:9241:-171:ed-1:v1:en)\n[Mandate 376 Standards EU](https://ec.europa.eu/growth/tools-databases/mandates/index.cfm?fuseaction=search.detail&id=333)",
"qr": "",
"categories": [
"Accessibility"
],
"phases": [
"Design",
"Output"
]
},
{
"question": "Do we need to offer a redress mechanism to the users?",
"threatif": "Yes",
"explanation": "* For applications that can adversely affect individuals, you might need to consider implementing a redress by design mechanism where affected individuals can request remedy or compensation.\n* Article 22(3) GDPR provides individuals with a right to obtain human intervention if a decision is made solely by an AI system and it also provides the right to contest the decision.",
"recommendation": "* Think about implementing mechanisms to effectively detect and rectify wrong decisions made by your system.\n* Provide a mechanism to ignore or dismiss undesirable features or services.\n* Wrong decisions could also have an impact on people that have not been the target of the data collection (data spillovers). Consider designing a way to offer all affected people the opportunity to contest the decisions of your system and request remedy or compensation. This mechanism should be easily accessible and it implies that you would need to have internally implemented a process where redress can be effectibily executed. This also has impact on the resources/skills needed to fulfil this process.\n* Consider this a necessary step to ensure responsibility and accountability.",
"sources": "[EU guidelines on ethics in artificial intelligence: Context and implementation](https://www.europarl.europa.eu/RegData/etudes/BRIE/2019/640163/EPRS_BRI(2019)640163_EN.pdf)",
"qr": "",
"categories": [
"Accessibility",
"Non-compliance"
],
"phases": [
"Design",
"Model",
"Output"
]
},
{
"question": "Do we need to implement an age gate to use our product?",
"threatif": "Yes",
"explanation": "* Is your product not meant to be used by children? You might need to implement an age verification mechanism to prevent children from accessing the product.\n* Age verification can also be important to provide the right diagnosis (health sector).",
"recommendation": "* Clearly specify in the user instructions for which age group the application is built. Labels or symbols can be very helpful.\n* Consider which design is more appropriate based on your use case, and consider the possible risks associated with your design choice, and the mitigating measures you can implement to reduce that risk. Document the rest risks that you want to accept.\n* Test the accessibility of your design with different age groups.",
"sources": "[Evolution in Age-Verification Applications](https://montrealethics.ai/evolution-in-age-verification-applications-can-ai-open-some-new-horizons/)",
"qr": "",
"categories": [
"Accessibility",
"Technique & Processes"
],
"phases": [
"Design",
"Output"
]
},
{
"question": "If users need to provide consent, can we make the required information easily available?",
"threatif": "No",
"explanation": "* Can the information be easily accessible and readable?\n* Do you need to build a special place for it (think of a robot where you might need to have a screen for showing the text)",
"recommendation": "* As part of privacy compliance you need to provide clear information about the processing and the logic of the algorithm. This information should be easily readable and accessible. During the design phase consider when and how you are going to provide this information. Especially in robots using AI this could be a challenge.\n* Comply with accessibility rules.",
"sources": "[A Proposal of Accessibility Guidelines for Human-Robot Interaction](https://www.mdpi.com/2079-9292/10/5/561/htm)\n[ISO/IEC 40500:2012 Information technology — W3C Web Content Accessibility Guidelines (WCAG) 2.0](https://www.iso.org/standard/58625.html)\n[ISO/IEC GUIDE 71:2001 Guidelines for standards developers to address the needs of older persons and persons with disabilities](https://www.iso.org/standard/33987.html)\n[ISO 9241-171:2008(en) Ergonomics of human-system interaction](https://www.iso.org/obp/ui/#iso:std:iso:9241:-171:ed-1:v1:en)\n[Mandate 376 Standards EU](https://ec.europa.eu/growth/tools-databases/mandates/index.cfm?fuseaction=search.detail&id=333)",
"qr": "",
"categories": [
"Accessibility",
"Non-compliance"
],
"phases": [
"Design",
"Model",
"Output"
]
},
{
"question": "Could the user perceive the message from the AI system in a different way than intended?",
"threatif": "Yes",
"explanation": "* Is the perception of the provided information the same as the one intended?\n* Explainability is critical for end-users in order to take informed and accountable actions.",
"recommendation": "* Understanding who is going to interact with the AI system can help to make the interaction more effective. Identify your different user groups.\n* Involve communication experts and do enough user testing to reduce the gap between the intended and the perceived meaning.",
"sources": "[The Who in Explainable AI: How AI Background Shapes Perceptions of AI Explanations](https://arxiv.org/pdf/2107.13509.pdf)",
"qr": "",
"categories": [
"Accessibility",
"Unawareness"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could the learning curve of the product be an issue?",
"threatif": "Yes",
"explanation": "* Does usage of the AI system require new (digital) skills?\n* How quickly are users expected to learn how to use the product?\n* Difficulties to learn how the system works could also bring the users in danger and have consequences for the reputation of the product or organisation.",
"recommendation": "* You can also provide assistance, appropriate training material and disclaimers to users on how to adequately use the system.\n* The words and language used in the interface, the complexity and lack of accessibility of some features could exclude people from using the application. Consider making changes in the design of the product where necessary.\n* Consider this also when children are future users.",
"sources": "",
"qr": "",
"categories": [
"Accessibility",
"Safety"
],
"phases": [
"Design",
"Model",
"Output"
]
}
]
},
{
"category": "Identifiability & Linkability",
"id": 3,
"colour": "94cfbd",
"cards": [
{
"question": "Can the data used to feed the model be linked to individuals?",
"threatif": "Yes",
"explanation": "Do you need to use unique identifiers in your training dataset? If personal data is not necessary for the model you would not really have a legal justification for using it.",
"recommendation": "* Unique identifiers might be included in the training set when you want to be able to link the results to individuals. Consider using pseudo-identifiers or other techniques that can help you protect personal data.\n* Document the measures you are taking to protect the data. Consider if your measures are necessary and proportional.",
"sources": "",
"qr": "",
"categories": [
"Identifiability & Linkability"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could actions be incorrectly attributed to an individual or group?",
"threatif": "Yes",
"explanation": "Your AI system could have an adverse impact on individuals by incorrectly attributing them facts or actions. For instance, a facial recognition system that identifies a person incorrectly, or an inaccurate risk prediction model that could negatively impact an individual.",
"recommendation": "* Evaluate the possible consequences of inaccuracies of your AI system and implement measures to prevent these errors from happening: avoiding bias and discrimination during the life cycle of the model, ensuring the quality of the input data, implementing a strict human oversight process, ways to double check the results with extra evidence, implementing safety and redress mechanisms, etc.\n* Assess the impact on the different human rights of the individual.\n* Consider not to implement such a system if you cannot mitigate the risks.",
"sources": "",
"qr": "",
"categories": [
"Identifiability & Linkability"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could we be revealing information that a person has not chosen to share?",
"threatif": "Yes",
"explanation": "* How can you make sure the product doesn’t inadvertently disclose sensitive or private information during use (e.g., indirectly inferring user locations or behaviour)?\n* Could movements or actions be revealed through data aggregation?",
"recommendation": "* Be careful when making data public that you think is anonymised. Location data and routes can sometimes be de-anonymised (e.g. users of a running app disclosing location by showing heatmap).\n* It is also important to offer privacy by default: offer the privacy settings by default at the maximum protection level. Let the users change the settings after having offered them clear information about the consequences of reducing the privacy levels.",
"sources": "",
"qr": "",
"categories": [
"Identifiability & Linkability"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
}
]
},
{
"category": "Security",
"id": 4,
"colour": "bdd895",
"cards": [
{
"question": "Do we need to red-team/pen test the AI system?",
"threatif": "Yes",
"explanation": "Do you need to test the security of your AI system before it goes live? This could have an impact on your project deadlines.",
"recommendation": "Include the time you might need for a pen test in your project planning. Sometimes this can take weeks: you might have to hire an external party, agree on the scope, sign the corresponding agreements and even plan a retest.",
"sources": "[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are our APIs securely implemented?",
"threatif": "No",
"explanation": "APIs connect computers or pieces of software to each other. APIs are common attack targets in security and are in some sense your public front door. They should not expose information about your system or model.\nSource: [BerryVilleiML](https://berryvilleiml.com/interactive/)",
"recommendation": "* Check how do you handle time and state and how is authentication implemented in your APIs.\n* Make sure that sensitive information such us API calls secrets are not sent in your commands.\n* Implement encryption at rest and in transit (TLS) and test often your APIs for vulnerabilities.",
"sources": "[OWASP API Security Project](https://owasp.org/www-project-api-security/)\n[BerryVilleiML](https://berryvilleiml.com/interactive/)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Is our data storage protected?",
"threatif": "No",
"explanation": "Is your data stored and managed in a secure way? Think about training data, tables, models, etc. Are you the only one with access to your data sources?\nSource: [BerryVilleiML](https://berryvilleiml.com/interactive/)",
"recommendation": "* Implement access control rules.\n* Verify the security of the authentication mechanism (and the system as a whole).\n* Consider the risk when utilizing public/external data sources.",
"sources": "[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "If our AI system uses randomness, is the source of randomness properly protected?",
"threatif": "No",
"explanation": "Randomness plays an important role in stochastic systems. “Random” generation of dataset partitions may be at risk if the source of randomness is easy to control by an attacker interested in data poisoning.\nSource: [BerryVilleiML](https://berryvilleiml.com/interactive/)",
"recommendation": "Use of cryptographic randomness sources is encouraged. When it comes to machine learning (ML), setting weights and thresholds “randomly” must be done with care. Many pseudo-random number generators (PRNG) are not suitable for use. PRNG loops can really damage system behaviour during learning. Cryptographic randomness directly intersects with ML when it comes to differential privacy. Using the wrong sort of random number generator can lead to subtle security problems.\nSource: [BerryVilleiML](https://berryvilleiml.com/interactive/)",
"sources": "[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Is our model suited for processing confidential information?",
"threatif": "No",
"explanation": "* There are certain kinds of machine learning (ML) models which actually contain parts of the training data in its raw form within them by design. For example, ‘support vector machines’ (SVMs) and ‘k-nearest neighbours’ (KNN) models contain some of the training data in the model itself.\n* Algorithmic leakage is an issue that should be considered carefully.\nSource: [BerryVilleiML](https://berryvilleiml.com/interactive/)",
"recommendation": "When selecting the algorithm perform analyses and test to rule out algorithmic leakage.",
"sources": "[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[ICO - How should we assess security and data minimisation in AI?](https://ico.org.uk/for-organisations/guide-to-data-protection/key-dp-themes/guidance-on-ai-and-data-protection/how-should-we-assess-security-and-data-minimisation-in-ai/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Model"
]
},
{
"question": "Can our AI system scale in performance from data input to data output?",
"threatif": "No",
"explanation": "Can your algorithm scale in performance from the data it learned on to real data? In online situations the rate at which data comes into the model may not align with the rate of anticipated data arrival. This can lead to both outright ML system failure and to a system that “chases\" its own tail.\nSource: [BerryVilleiML](https://berryvilleiml.com/interactive/)",
"recommendation": "* Find out what the rate would be of expected data arrival to your model and perform tests in a similar environment with similar amount of data input.\n* Implement measures to make your model scalable.",
"sources": "[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from insider threats?",
"threatif": "No",
"explanation": "AI designers and developers may deliberately expose data and models for a variety of reasons, e.g. revenge or extortion. Integrity, data confidentiality and trustworthiness are the main impacted security properties. Source: [ENISA](https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges)",
"recommendation": "* Implement on and off boarding procedures to help guarantee the trustworthiness of your internal and external employees.\n* Enforce separation of duties and least privilege principle.\n* Enforce the usage of managed devices with appropriate policies and protective software.\n* Awareness training.\n* Implement strict access control and audit trail mechanisms.",
"sources": "[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected against model sabotage?",
"threatif": "No",
"explanation": "Sabotaging the model is a nefarious threat that refers to exploitation or physical damage of libraries and machine learning platforms that host or supply AI/ML services and systems. Sources: [ENISA](https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges)",
"recommendation": "* Implement security measures to protect your models against sabotage.\n* Assess the security profile of third party tooling and providers.\n* Consider implementing a disaster recovery plan with mitigation measures for this type of attack.",
"sources": "[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could there be possible malicious use, misuse or inappropriate use of our AI system?",
"threatif": "Yes",
"explanation": "An example of abusability:\nA product that is used to spread misinformation; for example, a chatbot being misused to spread fake news.",
"recommendation": "* Threat model your system: anticipate vulnerabilities and look for ways to hijack and weaponize your system for malicious activity.\n* Conduct *red team* exercises.",
"sources": "[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security",
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could environmental phenomena or natural disasters have a negative impact on our AI system?",
"threatif": "Yes",
"explanation": "* Examples of environmental phenomena are heating, cooling and climate change.\n* Examples of natural disasters to take into account are earthquakes, floods and fires.\nEnvironmental phenomena may adversely influence the operation of IT infrastructure and hardware systems that support AI systems.\nNatural disasters may lead to unavailability or destruction of the IT infrastructures and hardware that enables the operation, deployment and maintenance of AI systems.\nSuch outages may lead to delays in decision-making, delays in the processing of data streams and entire AI systems being placed offline. Sources: [ENISA](https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges)",
"recommendation": "Implement a disaster discovery plan considering different scenarios, impact, Recovery Time Objective (RTO), Recovery Point Objective (RPO) and mitigation measures.",
"sources": "[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from perturbation attacks?",
"threatif": "No",
"explanation": "* In perturbation style attacks, the attacker stealthily modifies the query to get a desired response.\n* Examples: \n * Image: Noise is added to an X-ray image, which makes the predictions go from normal scan to abnormal.\n * Text translation: Specific characters are manipulated to result in incorrect translation. The attack can suppress a specific word or can even remove the word completely.\n Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.\n* Random perturbation of labels is also a possible attack, while additionally there is the case of adversarial label noise (intentional switching of classification labels leading to deterministic noise, an error that the model cannot capture due to its generalization bias). Source: [ENISA](https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges)",
"recommendation": "**Reactive/Defensive Detection Actions:**\n* Implement a minimum time threshold between calls to the API providing classification results. This slows down multi-step attack testing by increasing the overall amount of time required to find a success perturbation.\n\n**Proactive/Protective Actions:**\n* Develop a new network architecture that increases adversarial robustness by performing feature denoising.\n* Train with known adversarial samples to build resilience and robustness against malicious inputs.\n* Invest in developing monotonic classification with selection of monotonic features. This ensures that the adversary will not be able to evade the classifier by simply padding features from the negative class.\n* Feature squeezing can be used to harden DNN models by detecting adversarial examples.\n\n**Response Actions:**\n* Issue alerts on classification results with high variance between classifiers, especially when from a single user or small group of users.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Adversarially Robust Malware Detection Using Monotonic Classification](https://people.eecs.berkeley.edu/~daw/papers/monotonic-iwspa18.pdf)\n[Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training](https://arxiv.org/abs/1711.08001)\n[Attribution-driven Causal Analysis for Detection of Adversarial Examples](https://arxiv.org/abs/1903.05821)\n[Feature Denoising for Improving Adversarial Robustness](https://arxiv.org/abs/1812.03411)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from poisoning attacks?",
"threatif": "No",
"explanation": "* In a poisoning attack, the goal of the attacker is to contaminate the machine model generated in the training phase, so that predictions on new data will be modified in the testing phase. This attack could also be caused by insiders.\n* Example: in a medical dataset where the goal is to predict the dosage of a medicine using demographic information, researchers introduced malicious samples at 8% poisoning rate, which changed the dosage by 75.06% for half of the patients.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.\n\n**Other scenarios:**\n* Data tampering: Actors like AI/ML designers and engineers can deliberately or unintentionally manipulate and expose data. Data can also be manipulated during the storage procedure and by means of some processes like feature selection. Besides interfering with model inference, this type of threat can also bring severe discriminatory issues by introducing bias.\nSource: [ENISA](https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges)\n* An attacker who knows how a raw data filtration scheme is set up may be able to leverage that knowledge into malicious input later in system deployment.\nSource:[BerryVilleiML](https://berryvilleiml.com/interactive/)\n* Adversaries may fine-tune hyper-parameters and thus influence the AI system’s behaviour. Hyper-parameters can be a vector for accidental overfitting. In addition, hard to detect changes to hyper-parameters would make an ideal insider attack.\nSource: [ENISA](https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges)",
"recommendation": "* Define anomaly sensors to look at data distribution on day to day basis and alert on variations.\n* Measure training data variation on daily basis, telemetry for skew/drift.\n* Input validation, both sanitization and integrity checking.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.\n\n* Implement measures against insider threats.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[Robustness Techniques & Toolkits for Applied AI](https://www.borealisai.com/research-blogs/robustness-techniques-toolkits-applied-ai/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from model inversion attacks?",
"threatif": "No",
"explanation": "* In a model inversion attack, if attackers already have access to some personal data belonging to specific individuals included in the training data, they can infer further personal information about those same individuals by observing the inputs and outputs of the ML model.\n* In model Inversion the private features used in machine learning models can be recovered. This includes reconstructing private training data that the attacker should not have access to.\n* Example: an attacker recovers the secret features used in the model through careful queries.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"recommendation": "* Interfaces to models trained with sensitive data need strong access control.\n* Implement rate-limiting on the queries allowed by the model.\n* Implement gates between users/callers and the actual model by performing input validation on all proposed queries, rejecting anything not meeting the model’s definition of input correctness and returning only the minimum amount of information needed to be useful.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from membership inference attacks?",
"threatif": "No",
"explanation": "* In a membership inference attack, the attacker can determine whether a given data record was part of the model’s training dataset or not.\n* Example: researchers were able to predict a patient’s main procedure (e.g.: Surgery the patient went through) based on the attributes (e.g.: age, gender, hospital).\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"recommendation": "* Some research papers indicate Differential Privacy would be an effective mitigation. Check for more information [Threat Modeling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml).\n* The usage of neuron dropout and model stacking can be effective mitigations to an extent. Using neuron dropout not only increases resilience of a neural net to this attack, but also increases model performance.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from model stealing attacks?",
"threatif": "No",
"explanation": "* In model stealing, the attackers recreate the underlying model by legitimately querying the model. The functionality of the new model is the same as that of the underlying model.\n* Example: in the BigML case, researchers were able to recover the model used to predict if someone should have a good/bad credit risk using 1,150 queries and within 10 minutes.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"recommendation": "* Minimize or obfuscate the details returned in prediction APIs while still maintaining their usefulness to *honest* applications.\n* Define a well-formed query for your model inputs and only return results in response to completed, well-formed inputs matching that format.\n* Return rounded confidence values. Most legitimate callers do not need multiple decimal places of precision.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from reprogramming deep neural nets attacks?",
"threatif": "No",
"explanation": "* By means of a specially crafted query from an adversary, Machine Learning systems can be reprogrammed to a task that deviates from the creator’s original intent.\n* Example: ImageNet, a system used to classify one of several categories of images was repurposed to count squares.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"recommendation": "* Configure a strong client-server mutual authentication and access control to model interfaces.\n* Takedown of the offending accounts.\n* Identify and enforce a service-level agreement for your APIs. Determine the acceptable time-to-fix for an issue once reported and ensure the issue no longer repros once SLA expires.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from adversarial example?",
"threatif": "No",
"explanation": "* An adversarial example is an input/query from a malicious entity sent with the sole aim of misleading the machine learning system.\n* Example: researchers constructed sunglasses with a design that could fool image recognition systems, which could no longer recognize the faces correctly.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"recommendation": "These attacks manifest themselves because issues in the machine learning layer were not mitigated. As with any other software, the layer below the target can always be attacked through traditional vectors. Because of this, traditional security practices are more important than ever, especially with the layer of unmitigated vulnerabilities (the data/algo layer) being used between AI and traditional software.\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from malicious AI/ML providers who could recover training data?",
"threatif": "No",
"explanation": "* Malicious ML providers could query the model used by a customer and recover this customer’s training data. The training process is either fully or partially outsourced to a malicious third party who wants to provide the user with a trained model that contains a backdoor.\n* Example: researchers showed how a malicious provider presented a backdoored algorithm, wherein the private training data was recovered. They were able to reconstruct faces and texts, given the model alone.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"recommendation": "* Research papers demonstrating the viability of this attack indicate Homomorphic Encryption could be an effective mitigation. Check for more information [Threat Modeling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n* Train all sensitive models in-house.\n* Catalog training data or ensure it comes from a trusted third party with strong security practices.\n* Threat model the interaction between the MLaaS provider and your own systems.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from attacks to the AI/ML Supply Chain?",
"threatif": "No",
"explanation": "* Owing to large resources (data + computation) required to train algorithms, the current practice is to reuse models trained by large corporations, and modify them slightly for the task at hand. These models are curated in a Model Zoo.\n In this attack, the adversary attacks the models hosted in the Model Zoo, thereby poisoning the well for anyone else.\n* Example: researchers showed how it was possible for an attacker to insert malicious code into one of the popular models. An unsuspecting ML developer downloaded this model and used it as part of the image recognition system in their code.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"recommendation": "* Minimize 3rd-party dependencies for models and data where possible.\n* Incorporate these dependencies into your threat modeling process.\n* Leverage strong authentication, access control and encryption between 1st/3rd-party systems.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.\n* Perform integrity checks where possible to detect tampering.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Are we protected from exploits on software dependencies of our AI/ML systems?",
"threatif": "No",
"explanation": "* In this case, the attacker does NOT manipulate the algorithms, but instead exploits traditional software vulnerabilities such as buffer overflows or cross-site scripting.\n* Example: an adversary customer finds a vulnerability in a common OSS dependency that you use and uploads a specially crafted training data payload to compromise your service.\n\nSource: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"recommendation": "Work with your security team to follow applicable Security Development Lifecycle/Operational Security Assurance best practices. Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.",
"sources": "[Microsoft, Threat Modelling AI/ML Systems and Dependencies](https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml)\n[Securing Machine Learning Algorithms, ENISA](https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms)\n[STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets](https://github.com/LaraMauri/STRIDE-AI)\n[Stride-ML Threat Model]( https://csf.tools/reference/stride-lm/)\n[MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/)",
"qr": "",
"categories": [
"Security"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
}
]
},
{
"category": "Safety",
"id": 5,
"colour": "f7f09f",
"cards": [
{
"question": "In case of system failure, could users be adversely impacted?",
"threatif": "Yes",
"explanation": "* Do you have a mechanism implemented to stop the processing in case of harm?\n* Do you have a way to identify and contact affected individuals and mitigate the adverse impacts?\n* Imagine a scenario where your AI system, a care-robot, is taking care of an individual (the patient) by performing some specific tasks and that this individual depends on this care.",
"recommendation": "* Implement some kind of *stop button* or procedure to safely abort an operation when needed.\n* Establish a detection and response mechanism for undesirable adverse effects on individuals.\n* Define criticality levels of the possible consequences of faults/misuse of the AI system: what type of harm could be caused to the individuals, environment or organisations?",
"sources": "",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Model",
"Output"
]
},
{
"question": "Could our AI system have an adverse impact on the environment?",
"threatif": "Yes",
"explanation": "* Ideally only models are used that do not demand the consumption of energy or natural resources beyond what is sustainable.\n* Your product should be designed with the dimension of environmental protection and improvement in mind.",
"recommendation": "* Establish mechanisms to evaluate the environmental impact of your AI system; for example, the amount of energy used and carbon emissions.\n* Implement measures to reduce the environmental impact of the AI system throughout its lifecycle.",
"sources": "[UNSDGs United Nations Sustainable Development goals](https://www.un.org/development/desa/disabilities/about-us/sustainable-development-goals-sdgs-and-disability.html)",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could our model be deployed in a different context?",
"threatif": "Yes",
"explanation": "Are you testing the product in a real environment before releasing it?\nIf the model is tested with one set of data and then is deployed in a different environment receiving other types of inputs there is less guarantee that it is going to work as planned. This is also the case in reinforcement learning with the so called wrong objective function where slight changes in the environment often require a full retrain of the model.",
"recommendation": "* Use different data for testing and training. Make sure diversity is reflected in the data. Specify your training approach and statistical method. Explore the different environments and contexts and make sure your model is trained with the expected different data sources. This also applies to reinforcement learning.\n* Are you considering enough aspects in the environment? Did you forget any environmental variable that could be harmful? Could limited sampling due to high costs be an issue? Document this risk and look for support in your organisation. The organisation is accountable and responsible for the mitigation or acceptance of this risk. And hopefully you get extra budget assigned.\n* Consider applying techniques such as *cultural effective challenge*; this is a technique for creating an environment where technology developers can actively participate in questioning the AI process. This better translates the social context into the design process by involving more people and can prevent issues associated with *target leakage* where the AI system trains on data that prepares it for an alternative job other than the one it was initially intended to complete.",
"sources": "Information about *cultural effective challenge*: [A Proposal for Identifying and Managing Bias in Artificial Intelligence](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270-draft.pdf)",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could the AI system become persuasive causing harm to the individual?",
"threatif": "Yes",
"explanation": "* This is of special importance in Human Robot Interaction (HRI): If the robot can achieve reciprocity when interacting with humans, could there be a risk of manipulation and human compliance?\n* Reciprocity is a social norm of responding to a positive action with another positive action, rewarding kind actions. As a social construct, reciprocity means that in response to friendly actions, people are frequently much nicer and much more cooperative than predicted by the self-interest model; conversely, in response to hostile actions they are frequently much more nasty and even brutal. Source: Wikipedia",
"recommendation": "* Signals of susceptibility coming from a robot or computer could have an impact on the willingness of humans to cooperate or take advice from it.\n* It is important to consider and test this possible scenario when your AI system is interacting with humans and some type of collaboration/cooperation in expected.",
"sources": "[The role of reciprocity in human-robot social influence](https://www.sciencedirect.com/science/article/pii/S258900422101395X)\n[Reciprocity in Human-Robot Interaction](https://ir.canterbury.ac.nz/bitstream/handle/10092/100798/Reciprocity-human-condition.pdf?sequence=2&isAllowed=y)\n[Social robots and the risks to reciprocity](https://link.springer.com/article/10.1007/s00146-021-01207-y)",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could our RL agents develop strategies that could have undesired negative side effects on the environment?",
"threatif": "Yes",
"explanation": "* Reinforcement Learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Source: [Wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning)\n\n* To better understand the threat consider a case where a robot is built to move an object, without manually programming a separate penalty for each possible bad behaviour. If the objective function is not well defined, the AI’s ability to develop its own strategies can lead to unintended, harmful side effects. In this case, the objective of moving an object seems simple, yet there are a myriad of ways in which this could go wrong. For instance, if a vase is in the robot’s path, the robot may knock it down in order to complete the goal. Since the objective function does not mention anything about the vase, the robot wouldn’t know how to avoid it. Source: [OpenAI](https://openai.com/blog/concrete-ai-safety-problems/)",
"recommendation": "AI systems don’t share our understanding of the world. It is not sufficient to formulate the objective as “complete task X”; the designer also needs to specify the safety criteria under which the task is to be completed. A better strategy could be to define a *budget* for how much the AI system is allowed to impact the environment. This would help to minimize the unintended impact, without neutralizing the AI system.\n\nAnother approach would be training the agent to recognize harmful side effects so that it can avoid actions leading to such side effects. In that case, the agent would be trained for two tasks: the original task that is specified by the objective function and the task of recognizing side effects.\nThe AI system would still need to undergo extensive testing and critical evaluation before deployment in real life settings.\nSource: [OpenAI](https://openai.com/blog/concrete-ai-safety-problems/)",
"sources": "[Concrete Problems in AI Safety](https://arxiv.org/pdf/1606.06565.pdf)\n[Concrete AI Safety Problems](https://openai.com/blog/concrete-ai-safety-problems/)",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Could our RL agents “hack” their reward functions?",
"threatif": "Yes",
"explanation": "* Reinforcement Learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Source: [Wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning)\n\n* Consider potential negative consequences from the AI system\nlearning novel or unusual methods to score well on its objective function.\nSometimes the AI can come up with some kind of “hack” or loophole in the design of the system to receive unearned rewards. Since the AI is trained to maximize its rewards, looking for such loopholes and “shortcuts” is a perfectly fair and valid strategy for the AI. For example, suppose that the office cleaning robot earns rewards only if it does not see any garbage in the office. Instead of cleaning the place, the robot could simply shut off its visual sensors, and thus achieve its goal of not seeing garbage.\nSource: [OpenAI]( https://openai.com/blog/concrete-ai-safety-problems/)",
"recommendation": "One possible approach to mitigating this problem would be to have a “reward agent” whose only task is to mark if the rewards given to the learning agent are valid or not. The reward agent ensures that the learning agent (robot for instance) does not exploit the system, but rather, completes the desired objective. For example: a “reward agent” could be trained by the human designer to check if a room has been properly cleaned by the cleaning robot. If the cleaning robot shuts off its visual sensors to avoid seeing garbage and claims a high reward, the “reward agent” would mark the reward as invalid because the room is not clean. The designer can then look into the rewards marked as “invalid” and make necessary changes in the objective function to fix the loophole. Source: [OpenAI]( https://openai.com/blog/concrete-ai-safety-problems/)",
"sources": "[Concrete Problems in AI Safety](https://arxiv.org/pdf/1606.06565.pdf)\n[Concrete AI Safety Problems](https://openai.com/blog/concrete-ai-safety-problems/)",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Can we provide human resources to supervise and give feedback every time the RL agent performs an action?",
"threatif": "No",
"explanation": "* Reinforcement Learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Source: [Wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning)\n\n* When the agent is learning to perform a complex task, human oversight and feedback are more helpful than just rewards from the environment. Rewards are generally modelled such that they convey to what extent the task was completed, but they do not usually provide sufficient feedback about the safety implications of the agent’s actions. Even if the agent completes the task successfully, it may not be able to infer the side-effects of its actions from the rewards alone. In the ideal setting, a human would provide fine-grained supervision and feedback every time the agent performs an action (Scalable oversight). Though this would provide a much more informative view about the environment to the agent, such a strategy would require far too much time and effort from the human. Source: [OpenAI](https://openai.com/blog/concrete-ai-safety-problems/)",
"recommendation": "One promising research direction to tackle this problem is semi-supervised learning, where the agent is still evaluated on all the actions (or tasks), but receives rewards only for a small sample of those actions (or tasks).\n\nAnother promising research direction is hierarchical reinforcement learning, where a hierarchy is established between different learning agents. There could be a supervisor agent/robot whose task is to assign some work to another agent/robot and provide it with feedback and rewards.\nSource: [OpenAI](https://openai.com/blog/concrete-ai-safety-problems/)",
"sources": "[Concrete Problems in AI Safety](https://arxiv.org/pdf/1606.06565.pdf)\n[Concrete AI Safety Problems](https://openai.com/blog/concrete-ai-safety-problems/)",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Can our AI/ML system be robust to changes in the data distribution?",
"threatif": "No",
"explanation": "A complex challenge for deploying AI agents in real life settings is that the agent could end up in situations that it has never experienced before. Such situations are inherently more difficult to handle and could lead the agent to take harmful actions.\nSource: [OpenAI](https://openai.com/blog/concrete-ai-safety-problems/)",
"recommendation": "One promising research direction focuses on identifying when the agent has encountered a new scenario so that it recognizes that it is more likely to make mistakes. While this does not solve the underlying problem of preparing AI systems for unforeseen circumstances, it helps in detecting the problem before mistakes happen. Another direction of research emphasizes transferring knowledge from familiar scenarios to new scenarios safely. Source: [OpenAI](https://openai.com/blog/concrete-ai-safety-problems/)",
"sources": "[Concrete Problems in AI Safety](https://arxiv.org/pdf/1606.06565.pdf)\n[Concrete AI Safety Problems](https://openai.com/blog/concrete-ai-safety-problems/)",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Can our RL agents learn about their environment without causing harm or catastrophic actions?",
"threatif": "No",
"explanation": "* Reinforcement Learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Source: [Wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning)\n\n* **Safe exploration**: An important part of training an AI agent is to ensure that it explores and understands its environment. While exploring, the agent might also take some action that could damage itself or the environment. Source: [OpenAI](https://openai.com/blog/concrete-ai-safety-problems/)",
"recommendation": "One approach to reduce harm is to optimize the performance of the learning agent in the worst case scenario. When designing the objective function, the designer should not assume that the agent will always operate under optimal conditions. Some explicit reward signal may be added to ensure that the agent does not perform some catastrophic action, even if that leads to more limited actions in the optimal conditions. Source: [OpenAI](https://openai.com/blog/concrete-ai-safety-problems/)",
"sources": "[Concrete Problems in AI Safety](https://arxiv.org/pdf/1606.06565.pdf)\n[Concrete AI Safety Problems](https://openai.com/blog/concrete-ai-safety-problems/)",
"qr": "",
"categories": [
"Safety"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
}
]
},
{
"category": "Unawareness",
"id": 6,
"colour": "f8d18c",
"cards": [
{
"question": "Do we need to inform users that they are interacting with an AI system?",
"threatif": "Yes",
"explanation": "* Are users adequately made aware that a decision, content, advice or outcome is the result of an algorithmic decision?\n* Could the AI system generate confusion for some or all users on whether they are interacting with a human or AI system?",
"recommendation": "In cases of interactive AI systems (e.g., chatbots, robots) you should inform the users that they are interacting with an AI system instead of a human. This information should be received at the beginning of the interaction.",
"sources": "",
"qr": "",
"categories": [
"Unawareness",
"Accessibility"
],
"phases": [
"Design",
"Output"
]
},
{
"question": "Can we provide the necessary information to the users about possible impacts, benefits and potential risks?",
"threatif": "No",
"explanation": "* Did you establish mechanisms to inform users about the purpose, criteria and limitations of decisions generated by the AI system?\n* If an AI-assisted decision has been made about a person without any type of explanation or information then this may limit that person's autonomy, scope and self-determination. This is unlikely to be fair.",
"recommendation": "* Provide clear information about how and why an AI-assisted decision was made and which personal data was used to train and test the model.\n* The model you choose should be at the right level of interpretability for your use case and for the impact it will have on the decision recipient. If you use a black box model make sure the supplementary explanation techniques you use provide a reliable and accurate representation of the systems behaviour. Source: UK ICO\n* Communicate the benefits, the technical limitations and potential risks of the AI system to users, such as its level of accuracy and/or error rates.\n* Ask your users (with a survey for instance) if they understand the decisions that your product makes.",
"sources": "",
"qr": "",
"categories": [
"Unawareness",
"Accessibility"
],
"phases": [
"Design",
"Model",
"Output"
]
},
{
"question": "Can users anticipate the actions of the AI system?",
"threatif": "No",
"explanation": "Are users aware of the capabilities of the AI system? Users need to be informed about what to expect, not only for transparency reasons but in some products also for safety precautions.",
"recommendation": "* Consider this as part of the GDPR transparency principle.\n* Users should be aware of what the AI system can do.\n* Clear Information should be provided on time and made accessible following accessibility design principles.",
"sources": "[GDPR transparency principle](https://gdpr-info.eu/recitals/no-58/)",
"qr": "",
"categories": [
"Unawareness",
"Safety"
],
"phases": [
"Design",
"Model",
"Output"
]
}
]
},
{
"category": "Ethics & Human Rights",
"id": 7,
"colour": "f2bc9a",
"cards": [
{
"question": "Bias & Discrimination: could there be groups who might be disproportionately affected by the outcomes of the AI system?",
"threatif": "yes",
"explanation": "* Could the AI system potentially negatively discriminate against people on the basis of any of the following grounds: sex, race, colour, ethnic or social origin, genetic features, language, religion or belief, political or any other opinion, membership of a national minority, property, birth, disability, age, gender or sexual orientation?\n* If your model is learning from data specific to some cultural background then the output could be discriminating for members of other cultural backgrounds.",
"recommendation": "* Consider the different types of users and contexts where your product is going to be used.\n* Consider the impact of diversity of backgrounds, cultures, and other important different attributes when selecting your input data, features and when testing the output.\n* Assess the risk of possible unfairness towards individuals or communities to avoid discriminating minority groups.\n* The disadvantage to people depends on the kind of harm, severity of the harm and significance (how many people are put at a disadvantage compared to another group of people). Statistical assessments on group differences are an important tool to assess unfair and discriminatory uses of AI.\n* Design with empathy, diversity and respect in mind.",
"sources": "[Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3547922)\n[The Fairness Handbook](https://www.amsterdamintelligence.com/resources/the-fairness-handbook)",
"qr": "",
"categories": [
"Ethics & Human Rights",
"Technique & Processes"
],
"phases": [
"Design",
"Input",
"Model",
"Output"
]
},
{
"question": "Can we expect mostly positive reactions from the users or individuals?",
"threatif": "No",
"explanation": "* Do the users expect a product functioning like this?\n* Do the users or individuals expect this type of processing of personal data?\n* Can you roll back if people are not happy with the product?",
"recommendation": "* Consider the different types of users and contexts your product is going to be used.\n* Consider diversity of backgrounds, cultures, and many other important different attributes.\n* Do enough user testing, like FUPs - Friendly User Pilots.\n* Design with empathy, diversity and respect in mind.\n* Assess the risk of possible unfairness towards individuals or communities to avoid discriminating minority groups and also to prevent a bad reputation for your organisation.",
"sources": "",
"qr": "",
"categories": [
"Ethics & Human Rights"
],
"phases": [
"Design",