Checklist for Privacy Professionals while assessing AI systems

Guest Author: Karthikeyan M at Linkedin. The views in the article are authors own and based on research. Adoption is at the discretion of the users.

Checklist for conducting PIA / DPIA for AI systems

While capturing the description of the envisaged processing and the purposes of the processing as part of PIA/DPIA 

• Identify the purpose of the AI system and the results intended to be produced by the AI system.

• Gather instances in which the AI system is replacing the human decision involving or not involving personal data.

• Conduct an exclusive RoPA to clearly identify the lifecycle of personal data in the AI system. Identify the presence of different methodologies of data collection and processing including collection such as web scraping.

• List the level of sensitivity of personal data involved in AI systems.

• Identify whether the training data set is used for supervised, unsupervised, reinforcement learning, Semi supervised or Generative and discriminative learning (it shall be used to determine the detrimental effect on the rights of data subjects) and to what extent that the AI system rely on data from third parties.

• Understand to what extent of data protection assurance and transparency that the personal data collected from third party to train AI systems provides to the data subjects

• Understand if the data subjects are aware of the collection of personal data for training your AI systems and what is the verifiable means provided for showcasing their consent for using their personal data for training AI systems

• Verify if the AI system is compatible with processing data for the intended purposes

• Alternatively, ensure there is proper oversight in place to guarantee that the AI system will not process data for unintended purposes.

Legal basis of processing 

• Is there a scope for identifying and separating data collected via different legal basis?

• If you intend to use previously collected personal data for training your AI system, whether the same has been informed to the data subjects?

• If data collected by web scraping is used, whether it is possible for ensure the legality of such data collected? (Like the presence of violation of Data use policy, AI training license terms etc.).

• Whether ethics by design principle is safeguarded while training the AI dataset?

Assessment of the necessity and proportionality of the processing

• List the intended purposes and the intended effect on the individuals.

• Assure that there is no presence of other less invasive methods to achieve the intended goal.

• Prepare a reason why AI is preferred over the previously existing methodology.

• Whether the intended purpose include automated decision making by AI? , if so then the principles of lawfulness, fairness and transparency can be recognized.

• Whether the principle of purpose limitation is being justified at the each stage of AI lifecycle and periodical reassessment is conducted to reassure the intended purposes?

Measures already envisaged

• Whether the existing privacy program management is sufficient to manage the probable risk arising out of AI systems?

• Is the training data sets is still within the compliance of local laws?

• Is the frequency of periodic audits by privacy team is adequate?

• What kinds of security controls are in place to secure the AI systems and datasets from tampering?

Assessment of the risks to the rights and freedoms of data subjects

• Identify the presence of CIA (confidentiality, integrity, accountability ) triad framework in data sets for training AI

• Assess if the AI system has exposed personal data beyond the intended purpose and scope.

• Determine whether a process and data owner has been appointed to ensure accountability.

• Assess whether their appointment is being carried out or if their non-appointment is being justified.

• Assess the possibility of “adversarial machine learning attacks” in the input personal data.

• Assess the presence of adequate techniques for detecting “Adversarial ML attacks”

• Assess the possibility and the extent of results arising out of such adversarial attacks impact the integrity of the trained personal data. (Or) analyze the possibility to manipulate the ML model to expose personal data.

• Whether there is a presence of appropriate technologies to enforce right to erasure, update requests in a trained AI dataset (if the legal basis permits)

• Ensure that the AI system and its outcome is proportionally balancing towards the rights and freedom of individuals.

• Examine the scope of detrimental effect on the data subjects due to bias and inaccuracy of AI data sets deployed.

Measures envisaged to address the risks

• Indicate the presence of industry accredited certification mechanisms to indicate the quality, integrity of the collected data.

• Assure the presence of periodical assessment of how AI system risks are being addressed from information security and privacy perspective.

• To what extent Human in the loop kind of approaches are deployed for ensuring accuracy in AI generated content.

• Evaluate the possibility of self-rectification or implementing a human-controlled risk mitigation process for the same.

• Evaluate if mechanisms have been established to enable end users to provide feedback on the predatory practices of the AI system and assure that such Process is simple and jargon free, the act of acknowledging the feedback and the process of working on that feedback is ensured and provisions detailing the limitations regarding non-performance of feedback.