Algorithms' Dark Side: Embedding Bias into Code - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Data Management // Big Data Analytics
Commentary
2/22/2017
10:05 AM
Ariella Brown
Ariella Brown
Commentary
50%
50%

Algorithms' Dark Side: Embedding Bias into Code

Do algorithms and AI eliminate bias or do they encode the biases of models? New work on AI policies is designed to shine the light on the black box of model design and use.

(Image: Mmaxer:Shutterstock)

(Image: Mmaxer:Shutterstock)

Does the shift toward more data and algorithmic direction for our business decisions assure us that organizations and businesses are operating to everyone's advantage? There are a number of issues involved that some people feel need to be addressed going forward.

Numbers don't lie, or do they? Perhaps the fact that they are perceived to be absolutely objective is what makes us accept the determinations of algorithms without questioning what factors could have shaped the outcome.

That's the argument Cathy O'Neil makes in Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. While we tend to think of big data as a counterforce to biased, just decisions, O'Neil finds that in practice, they can reinforce biases even while claiming unassailable objectivity.

"The models being used today are opaque, unregulated, and incontestable, even when they're wrong. The math destruction posed by algorithms is the result of models that reinforce barriers, keeping particular demographic populations disadvantaged by identifying them as less worthy of credit, education, job opportunities, parole, etc.

Now the organizations and businesses that make those decisions can point to the authority of the algorithm and so shut down any possible discussion that question the decision. In that way, big data can be misused to increase inequality. As algorithms are not created in a vacuum but are born of minds operating in a human context that already has some set assumptions, they actually can extend the reach of human biases rather than counteract them.

"Even algorithms have parents, and those parents are computer programmers, with their values and assumptions," Alberto Ibargüen, president and CEO and of the John S. and James L. Knight Foundation wrote in a blog post. ". . . As computers learn and adapt from new data, those initial algorithms can shape what information we see, how much money we can borrow, what health care we receive, and more."

The foundation's VP of Technology Innovation, John Bracken, told me about the foundation's partnership with the MIT Media Lab and the Berkman Klein Center for Internet & Society as well as its work with other individuals and organizations to create a $27 million fund for research in this area. The idea is to open the way to "bridging" together "people across fields and nations" to pull together a range of experiences and perspectives on the social impact of the development of artificial intelligence. AI will impact every aspect of human life, so it is important to think about sharpening policies for the tools to be built and how they will be implemented.

The fund, which is to be open for applicants even outside the founding university partners, may be used for exploring a number of issues identified including these:

  • Ethical design: How do we build and design technologies that consider ethical frameworks and moral values as central features of technological innovation?
  • Advancing accountable and fair AI: What kinds of controls do we need to minimize AI's potential harm to society and maximize its benefits?
  • Innovation in the public interest: How do we maintain the ability of engineers and entrepreneurs to innovate, create and profit, while ensuring that society is informed and that the work integrates public interest perspectives?

Independently of the organizations involved in the fund, the Association for Computing Machinery US Public Policy Council (USACM), has been doing its own research into the issues that arise in a world in which crucial decisions may be determined by algorithms. It recently released its take on what businesses should use for guidance in its its Principles for Algorithmic Transparency and Accountability.

The seven principles listed include awareness of "the possible biases involved in their design, implementation, and use and the potential harm that biases can cause to individuals and society;" the possibility of an audit of the data, algorithms, and models that were involved in a decision that may have been harmful; the possibility for "redress for individuals and groups that are adversely affected by algorithmically informed decisions;" as well as the obligation for the organization to provide explanations of their processes and accountability for their validity.

As we go forward in incorporating even more algorithms into the daily functions of business and other organizations, we will have to be mindful about the potential impact of decisions that may not be as objective as we assumed them to be. Better data doesn't automatically translate into better results, and we have to be aware of potential problems if we are to address them.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Slideshows
9 Steps Toward Ethical AI
Cynthia Harvey, Freelance Journalist, InformationWeek,  5/15/2019
Commentary
How to Assess Digital Transformation Efforts
Lisa Morgan, Freelance Writer,  5/14/2019
Commentary
Is AutoML the Answer to the Data Science Skills Shortage?
Guest Commentary, Guest Commentary,  5/10/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
A New World of IT Management in 2019
This IT Trend Report highlights how several years of developments in technology and business strategies have led to a subsequent wave of changes in the role of an IT organization, how CIOs and other IT leaders approach management, in addition to the jobs of many IT professionals up and down the org chart.
Slideshows
Flash Poll