ChatGPT’s AI powers make better writers, MIT study finds

ChatGPT and its AI powers could help writers and office workers improve their writing quality and decrease the time spent on tasks, cutting out busy work in favor of better, more productive work. However, the MIT study that suggested these conclusions also warned that employers could use AI to increase layoffs, too.

The paper, “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence” by Shakked Noy and Whitney Zhang of the economics department at MIT, is considered a working paper and has not been peer-reviewed. Still, the conclusions it found about ChatGPT’s AI chatbot technology are both fascinating—and troubling, especially when the study factored in how it affects workers.

The two doctoral students split 444 college-educated professionals into two groups, and assigned them to write press releases, email, short reports, and analysis plans—a normal workday for many people. One of the groups was allowed to use a text editor. The other was trained and allowed to use ChatGPT to do the work for them. The output was rated, assigned a value score, and those with the highest score were paid an actual (though nominal) amount of money for their high scores, pushing them to deliver their best work.

The study found that those who used ChatGPT cut the time needed to complete the tasks by almost half (30 minutes to about 17 minutes). The quality of the work also increased, with the grades assigned by the evaluators (on a 1-7 scale) rising from 4 to about 4.7. Those who used the text editor saw a slight decrease in time spent but also a decrease in the quality of the work.

“College-educated professionals performing mid-level professional writing tasks experience substantial increases in productivity when given access to ChatGPT,” the authors concluded. “The generative writing tool increases the output quality of low-ability workers while reducing their time spent, and it allows high-ability workers to maintain their quality standards while becoming significantly faster.”

Other study findings were interesting, too: About 68 percent of the workers simply copied and pasted ChatGPT’s output without editing it. Furthermore, even though some testers spent more time editing ChatGPT’s responses, the way the study was set up (monitoring output on a minute-by-minute basis) also showed that those tested did not see any benefit in quality from editing ChatGPT’s work—though they spent more time doing it. That was true even when the study authors gave incentives for improving the work.

“Post-treatment, the share of time spent writing a rough draft falls by more than half and the share of time spent editing more than doubles,” the study found.

The study participants also reported that using ChatGPT significantly increased their job satisfaction, eliminating the “busy work” that can plague some tasks, and were willing to pay about 0.5 percent of their monthly salary for a subscription to ChatGPT. (ChatGPT already charges $20 per month for paid access.) But those tested also indicated that ChatGPT didn’t always know as much as they did.

The other problem, the authors noted—remember, they’re economists—is that employers may start recognizing the value of ChatGPT, too. That may put worker jobs in jeopardy.

“At the aggregate level, ChatGPT substantially compresses the productivity distribution, reducing inequality,” Noy and Zhang wrote. “It is also already being used by many workers in their real jobs. The experimental evidence suggests that ChatGPT largely substitutes for worker effort rather than complementing workers’ skills, potentially causing a decrease in demand for workers, with adverse distributional effects as capital owners gain at the expense of workers.”

In other words, while ChatGPT may help writers, they may be competing against AI, too.

Personal Software, Professional Software