Towards a multi-stakeholder value-based assessment framework for algorithmic systems

  Means Value(s) and Manifestation(s) DT AT DE DS Application (model) Approach Visual elements Additional details
[H] What-if tool 1 ([29]) PERFORMANCE
- Accuracy
- False Positive and Negative rates
- False Discovery and omissions rates

FAIRNESS
- Accuracy across groups
- False Positive and Negative rates across groups
- False Discovery and omissions rates across groups

TRANSPARENCY
- Disclosure of origin and properties of data

NON-DISCRIMINATION
- Analysis of data for potential biases, data quality assessment
    Classification tasks, Regression tasks   - Confusion matrices
- (Two-dimensional) Histogram
- Scatterplots
- Summary statistics of datasets
- Partial dependence plots
Interactive modules include: list of feature values, inference values, and counterfactual controls




Red: descriptive documents specifying whether/how a value manifestation is fulfilled
Blue: strategy for fulfilling value manifestations
Green : complete tool for enabling the fulfillment of value manifestations

DT: Development Team
AT: Auditing Team
DE: Data domain Experts
DS: Decision Subject

Back to “Tailored communication of system-related information”

Back to main page

  1. https://github.com/pair-code/what-if-tool