What happens when online job applicants start using LLMs? It ain't good. 1. Pre-LLM, cover letter quality predicts your work quality, and a good cover gets you a job 2. LLMs wipe out the signal, and employer demand falls 3. Model suggests high ability workers lose the most 1/n

In April 2023, made it possible for workers to use AI in their cover letters. Employers can't see if they used the tool. Time spent to submit an application goes down, with a big increase in apps that took <30 seconds 2/

A measure of cover letter quality — how much the freelancer's email is customized to the specific job post — goes way up. 3/

In the pre-LLM era, people with higher quality cover letters are much more likely to get the job. Post-LLM, cover letter quality is useless as signal, so doesnt affect hiring at all. Note the total hiring rate goes way down as well. 4/

I think what's happening is firms are barraged by AI applications, and can'y distinguish who is good—and decide its just not worth trying. I certainly feel this with regard to inquiries from prospective students / @devdatalab applicants! 5/

We can also see that cover letter quality no longer predicts worker quality—the outcome is a 5-star rating from the employer. Coef on signal goes to zero in the post-LLM era.

Another view: cover letter quality goes way up with LLMs, while P(hired | good letter) goes way down.

I don't tweet structural models on principle, but the authors' model finds that LLMs move jobs from high to low-quality workers. Unclear whether completed jobs are worse, but I think there are fewer completed jobs. 8/

Cursed graph for anyone reading job applications or grant proposals. 9/

Super interesting paper, read the whole thing here: Jesse Silbert is on the econ job market. Some additional thoughts. 10/

It's wild to me that decided to make it so easy for job candidates to do this. It seems obvious to me that it would be terrible for firms! (Though the authors' model doesn't really agree with this.) 11/

Maybe they will create a paid tier where firms can see who used LLMs. Heh, then they can charge applicants for LLM access, and then charge firms to be able to exclude those applicants from consideration. 12/

In equilibrium, it would be really high value for firms to find some other signal of candidate quality. Job history and rating seems to still work well. But Goodhart's Law will apply to any new screening mechanism, especially if LLMs can game it. 13/

It's hard for firms hiring college grads—grade inflation makes it hard to find the good students, and now cover letters are useless too. We may see more trial employment periods. Work with someone for a month, you can get an *excellent* signal of their quality. 14/

Read the paper! @SilbertJesse is on X.

Read the paper! @SilbertJesse is on X and Anais Galdin is on BlueSky

The authors are Jesse Silbert and Anais Galdin. Read the paper!