Susan Athey Delivers Kenneth J. Arrow Lecture on AI and the Future of Work

Columbia SIPA’s Institute of Global Politics (IGP) held the 15th annual Kenneth J. Arrow Lecture on March 10, featuring Susan Athey, Economics of Technology Professor at the Stanford Graduate School of Business and senior fellow at the Stanford Institute for Economic Policy Research. Titled “Using Machine Learning and Digital Technology to Identify Challenges and Improve Outcomes for Labor Market Transitions,” she previewed her most recent research about the future of work and the labor market in an AI-driven world. 

Cohosted by Columbia World Projects and the Columbia Economics Program for Economic Research, the Arrow Lecture Series honors the late Kenneth J. Arrow, a Nobel Prize-winning economist known for his contributions to welfare economics and to general economic equilibrium theory, and highlights economists whose work builds on his scholarship. 

In her remarks, Athey explored how machine learning and AI can be used to understand and address labor market challenges and help workers navigate job transitions. Her research highlighted the importance of targeted interventions. By using machine learning to identify the most vulnerable workers, policymakers can design more effective support programs, she explained. Notably, she found that older workers in manufacturing and those with lower education levels are particularly susceptible to long-term earnings damage.

She began by reviewing several recent papers focused on labor market transitions, the first of which analyzed worker resilience in response to layoffs. Using administrative data from 180,000 laid-off workers in Sweden, Athey explained how she built and used machine learning techniques to examine the heterogeneous impacts of job losses and identify groups who are most and least harmed. Her study found significant variations in the effects of job displacement – while some workers experienced devastating long-term impacts and never fully recovered, with the most-harmed group suffering a 45 percent earnings loss one year post-layoff, others bounced back quickly, with the impact on their earnings diminishing over time. Of the nuanced analysis her machine learning approach allows for, Athey said, “What this says is that based on characteristics of people before they were laid off, we can very accurately predict, on held out data, people who are going to still be badly off much later.”

Athey presented two experimental interventions designed to help workers navigate career transitions. The first project, conducted in Poland, focused on helping women build portfolios demonstrating skills relevant to IT jobs. A second project with Coursera explored the value of making digital credentials more visible to employers. “We’re not studying the effect of having the credential, we’re studying the effect of the employers being aware of the credential,” she explained. By making it easier for job seekers to share verified online course certificates on LinkedIn, the research showed that simple technological interventions could improve employment prospects, particularly for individuals with lower initial employability.

Athey emphasized the potential AI has to transform our understanding of labor markets, as it allows economists to look at much more information than traditional economic research. She discussed how she’s utilizing AI in a current project, using a dataset of 23 million resumes to build a transformer model that can predict career transitions, analyze job histories and their influence on wages, and provide insight into the gender wage gap.

Following the lecture, Athey was joined by Joseph Stiglitz, the 2001 Nobel Laureate, SIPA professor and IGP Faculty Advisory Board member, Bentley MacLeod, Sami Mnaymneh Professor Emeritus of Economics at Columbia, and SIPA Professor Suresh Naidu, also an IGP Faculty Advisory Board Member, to discuss her findings and AI’s impacts on labor markets. 

MacLeod highlighted both the opportunities and dangers of AI, noting the technology's potential to "describe what we see in labor markets,” while also raising concerns about job displacement and changing workplace dynamics. Naidu drew a parallel with financial economics, suggesting that AI and digital technologies might "not just study labor markets as they are, but actually wind up constructing new forms of labor markets and new forms of labor contracts."

When an audience member asked about the potential for AI to completely replace human labor, Athey pointed out that many companies still have people doing jobs that could be automated. “There are a lot of economic considerations that affect the pace of adoption that aren't part of the technology,” she said. “We have, you know, so many economic frictions along the way to actually fully implement technology to replace all of us.”