The World Privacy Forum has released a new report, “The Scoring of America” (pdf), concerning new types of consumer scoring and how they can affect individuals. Here’s an excerpt from the introduction:
To score is human. Ranking individuals by grades and other performance numbers is as old as human society. Consumer scores — numbers given to individuals to describe or predict their characteristics, habits, or predilections — are a modern day numeric shorthand that ranks, separates, sifts, and otherwise categorizes individuals and also predicts their potential future actions.
Consumer scores abound today. Credit scores based on credit files receive much public attention, but many more types of consumer scores exist. They are used widely to predict behaviors like, spending, health, fraud, profitability, and much more. These scores rely on petabytes of information coming from newly available data streams. The information can be derived from many data sources and can contain financial, demographic, ethnic, racial, health, social, and other data. […]
Predictive scores bring varying benefits and drawbacks. Scores can be correct, or they can be wrong or misleading. Consumer scores – created by either the government or the private sector – threaten privacy, fairness, and due process because scores, particularly opaque scores with unknown ingredients or factors, can too easily evade the rules established to protect consumers.
The most salient feature of modern consumer scores is the scores are typically secret in some way. The existence of the score itself, its uses, the underlying factors, data sources, or even the score range may be hidden. Consumer scores with secret factors, secret sources, and secret algorithms can be obnoxious, unaccountable, untrustworthy, and unauditable. Secret scores can be wrong, but no one may be able to find out that they are wrong or what the truth is. Secret scores can hide discrimination, unfairness, and bias. Trade secrets have a place, but secrecy that hides racism, denies due process, undermines privacy rights, or prevents justice does not belong anywhere.
Broader transparency for consumer scores with limited secrecy may offer a middle ground. Knowing the elements but not necessarily the weights of a scoring system provides a partial degree of openness and reassurance. Knowing that there is a scoring system and how and when it is used helps. Knowing the source and reliability of the information used to make a score helps. Being able to challenge a score and correct the data on which it is based helps. Knowing that some types of information will not be used for scoring helps. Knowing that data collected for one purpose will not be used for another or in violation of law helps. Knowing that the person running the scoring system is accountable in a meaningful way helps. […]
This report discusses and explores consumer scores, what goes into them and how they are made, how they are used, the regulations in place that control some but not most new consumer scores, and how scores affect broader privacy and fairness issues. The discussion of findings and recommendations points toward solutions and reforms that are needed.