Indeed uses OpenAI to deliver contextual job matching to millions of job seekers

Indeed(opens in a new window), whose mission is to help people get jobs, is the world’s #1 job site¹. Over 350 million unique visitors² come to Indeed every month to connect with more than 3.5 million employers and over 32 million jobs. But what’s more is that every three seconds someone gets hired on Indeed³.

Since Indeed’s inception, AI has powered the millions of connections between job seekers and employers on the platform, through features such as ‘Invite to Apply’ which sends AI-based job recommendations to job seekers based on their resume, Indeed Profile, and other qualifications. Improvements in AI—specifically generative AI—are helping match job seekers to jobs in new and exciting ways. Using OpenAI's GPT models and fine-tuning capabilities, Indeed enhanced the personalized language in the ‘Invite to Apply’ feature to better explain why a candidate’s background or previous work experience makes a job a good fit.

Indeed's early AI models effectively matched job seekers with employers’ job postings and provided brief explanations for these matches. But Indeed’s product and engineering teams believed that OpenAI’s GPT models could do even more by ingesting natural language to offer better context as to why a candidate is receiving a specific job recommendation.  

Adding a more insightful “why” to job recommendations became the goal for the Indeed team to work with OpenAI. 

Indeed’s engineering team started working with OpenAI in 2023, following the OpenAI documentation for the Chat Completions API and iterating on its few-shot prompting. Throughout the development process, the team conducted rigorous A/B testing and continuously monitored and evaluated every change, both during the implementation phase and post-launch.

Indeed found that training the model to respond using few-shot-prompting was effective at increasing the relevant “why” in explanations, but at Indeed's massive scale, this was resulting in high token consumption. To increase efficiency, OpenAI and Indeed worked together to fine-tune a smaller GPT model that was able to deliver similarly performing results but with 60% fewer tokens.

As part of the fine-tuning process, they harnessed GPT-4 for data augmentation, and built out specific content guidelines. The team also built out an annotation operation, which involved labeling LLM-generated output, creating ground truths for facilitating automated evaluation, and adding context to help the models understand nuances.

Further testing validated the value of personalization in job recommendations, increasing the pool of qualified candidates. To scale personalized job recommendations, Indeed worked with OpenAI to adopt dedicated instances which were provisioned in January 2024. The Indeed engineering team successfully deployed the fine-tuned GPT model to these instances, allowing them to personalize employment  opportunities to millions more job seekers.

“We have invested in our own AI matching technology for decades to help connect job seekers and employers. Regardless of how good our matching may be, explainability is key to any successful recommendation system. Combining OpenAI's GPT explanations with Indeed's own proprietary AI and vast marketplace data allows us to connect more people to jobs, faster—a win for job seekers, employers, and society,” says Chris Hyams, CEO of Indeed.

The effectiveness of GPT-developed personalized messaging was proven through a multi-stage experiment that expanded to nearly 20 million messages per day over the course of several months. 

“We’re always testing ways to improve the Invite to Apply experience and find relevant opportunities for job seekers. Fine-tuning OpenAI’s GPT models has helped us deliver more personalized recommendations, even as we scaled from 1 million to 20 million messages a day,” says Horatio Lun, Indeed Platform PM.

The Indeed team tested a traditional job matching version of Invite to Apply against the GPT-powered version that had additional customized context for the applicant. The performance uplift was immediately noticeable:

• +20% increase in started job applications

• +13% uplift in downstream success indicating that not only were more candidates likely to apply, but that employers were finding those applicants to be a good fit, ultimately leading to more hires

By leveraging GPT-powered context for its 'Invite to Apply' feature, Indeed experienced an increase in both qualified candidates and hires, which positively impacted its revenue during testing.

“Importantly, we’re able to leverage OpenAI in an ROI-positive way, so we see a lot of opportunity to continue to invest in this new infrastructure in ways that will help us grow revenue,” says Chris Hyams, CEO of Indeed.

And Invite to Apply is just one example of how Indeed is finding success with OpenAI technology. Nearly a dozen products at Indeed are using OpenAI to deliver more personalized and compelling experiences to help job seekers discover new opportunities and employers hire faster.