"Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy" by Cathy O’Neil  discusses how government and private institutions are relying more on mathematical models to increase efficiency but with unintended consequences.
The book was published in September. It’s on Amazon’s (AMZN, Financial) best-seller list in the Business & Statistics category.
O’Neil was a math professor at Barnard College in the mid-2000s. Math appealed to her because of its objectivity and elegance. In theory, mathematical equations are indisputable. Either the equations are right or they’re wrong.
In 2007, O’Neil left academia and began working at hedge fund D.E. Shaw. While working there, she got a front row seat to witness how math was used not for objectivity but to obscure risk in derivative securities. D.E. Shaw operated in a way that employees were only given access to information related to their core competencies. This structure was intended to minimize the impact of employee turnover. If people left, they wouldn’t be able to give away all of the company’s secrets.
However, the downside to this structure was that nobody could see the big picture and truly understand the overall risks involved with all of the firm’s trading strategies. This secretive operating structure was common in trading groups across the industry.
The 2008 financial crisis was caused in part by banks packaging loans into mortgage-backed securities in exotic ways and then having the ratings agencies use math to give them a credit rating that misrepresented the true risks involved. The financial crisis involved subprime mortgages defaults, major financial institutions failing, liquidity drying up and the subsequent effects rippling to the broader economy.
After she left D.E. Shaw, O’Neil went to work at a company called RiskMetrics Group. The company’s goal was to assess risk for the banks’ portfolios. Instead, O’Neil saw that bank executives ignored their findings and wanted RiskMetrics to rubber-stamp internal processes. Again, math was being used to obscure the truth. She left the firm by 2011.
O’Neil then went to work as a data scientist for a firm that built models to predict consumer behavior for travel websites. Her cumulative work experiences in the corporate world made her realize the unintended consequences of faulty models and sloppy statistics. She dubbed mathematical models with unintended consequences “weapons of math destruction (WMD).” WMDs have the following characteristics.
- They’re secret where people affected don’t know they’re being used.
- They’re opaque where the creators won’t divulge the specifics of the algorithms.
- They have subjective definitions of success.
- They create pernicious feedback loops.
Below are three examples of WMDs discussed in the book.
Teacher ranking system
O’Neil describes a sound mathematical model as one that has a clear objective, has transparency and is revised as new data comes in. The opposite is when a mathematical model is presumed to be infallible, is opaque and is not revised. An example is how many cities around the country are using mathematical models to rate public school teachers.
O’Neil recounts the story of a teacher who was given a low score calculated by an opaque software. He would have been fired if he received another low score. When the teacher asked to see the scoring methodology, he was rebuffed and told that the mathematical models were complicated. What made the scoring suspicious is how his score swung in the subsequent year. In the first year, he scored in the bottom 15th percentile but in year two he jumped to the 90th percentile.
The model was supposedly designed to measure the amount of improvement in his students. The students he inherited had low reading comprehension skills. However, when he investigated the students’ standardized test scores from the prior year, they scored unbelievably high. It’s been reported that some teachers become overwhelmed by the stress caused by standardized tests and resort to cheating. One wonders what happens to honest teachers? The honest teachers are at risk of being removed from the public school system and the underperforming ones who cheat remain thereby worsening the quality of education.
The problem with these models is not just that they’re opaque, but there is also no follow-up process. Once a teacher is fired no further investigation is conducted. It’s assumed that the software did its job. Consider the alternative. Amazon uses predictive modeling to show consumers relevant products. Amazon constantly gathers new data and updates its models to improve effectiveness.
Insurance companies
The book points to FICO scores as an example of a model with admirable qualities. FICO scores evaluate the risk that borrowers may default on loans. The model is transparent, and it is revised as more data becomes available. The credit scoring industry is regulated, and consumers can check what affected their scores. FICO’s website also tells consumers how to improve their scores (i.e., reduce debt, pay bills on time, limit the number of credit cards).
In contrast, the insurance industry uses FICO scores along with its own proprietary algorithms. For example, “Consumer Reports found that the [scores compiled by insurance companies], which include all sorts of demographic data, often count for more than the driver’s record. In other words, how you manage money can matter more than how you drive a car. In New York State, for example, a dip in a driver’s credit rating from 'excellent' to merely 'good' could jack up the annual cost of insurance by $255. And in Florida, adults with clean driving records and poor credit scores paid an average of $1,552 more than the same drivers with excellent credit and a drunk driving conviction.”
The book questions whether insurance companies are pricing policies by how safely people drive or whether they’re pricing policies for maximum profitability and justifying it with opaque and subjective mathematical models.
The book points out that “Allstate analyzes consumer and demographic data to determine the likelihood that customers will shop for lower prices. If they aren’t likely to, it makes sense to charge them more. And that’s just what Allstate does.” For consumers, the moral of the story is to shop around and compare policy prices.
University ranking system
If you’ve ever wondered why the cost of college has gone up, O’Neil points to U.S. News’ ranking of universities as a WMD. The magazine’s university rankings have become highly influential. In 1983, U.S. News was a struggling magazine. It started a project to help students determine which universities best suited their needs.
The first rankings were determined by opinion surveys sent to college administrators. This is an example of how the definition of success is subjective. College officials complained, and the magazine revisited its methodology. U.S. News eventually created an algorithm that accounted for 75% of the score while the last 25% was based on the subjective views of college officials across the nation.
The problem is that people expect reputable universities like the Ivy League schools to end up near the top. If they don’t, it undermines the credibility of the ranking system. That means the model is built with a predetermined outcome in mind. Factors like alumni contribution and SAT scores were included and tuition cost was omitted as a result.
O’Neil argues that the ranking system has led to a vicious feedback loop. If a school scores poorly, fewer students will apply and the school’s finances suffer which leads to less money for faculty and so on.
One of the biggest factors affecting a school’s reputation and alumni contribution is the success of its athletic programs. Texas Christian University is an example of a school that has gone up in the rankings quickly. It has focused on fundraising and poured those funds into athletic facilities and improving the football program. In the end, students are paying for better buildings and a better football team, but has the quality of instruction improved?
Final thoughts
Big Data will become increasingly ingrained in our lives. In addition to the three examples above, O’Neil discusses how WMDs are being used in criminal justice, human resources and politics. The ramifications are unsettling when we consider how real lives are affected. People’s jobs, finances and in some cases freedom are being determined by opaque algorithms.
From an investor perspective, it’s apparent when I review investor presentations and conference call transcripts that many companies are struggling to grow revenue and are increasingly looking to cut costs. Big Data has been hyped as a way to increase revenue and drive down costs (through focused customer targeting, supply chain optimization, inventory control, lowering transportation costs, predicting the weather, measuring employee performance, etc.). "Weapons of Math Destruction" is a good read to remind us that not all Big Data outcomes are positive and to keep a healthy skepticism of the hype.
Disclosure: The author owns no stocks mentioned in this article.
Start a free seven-day trial of Premium Membership to GuruFocus.
