This podcast explores novel research from Spotify on unified generative models for information retrieval, specifically integrating search and recommendation. Moving beyond traditional index-based systems, this approach leverages large language models (LLMs) to directly predict item IDs, centralizing tasks like search and recommendation.

The study investigates whether jointly training search and recommendation tasks in a single generative model improves effectiveness. Key hypotheses explored are [H1], regarding regularization of item popularity estimation, and [H2], focusing on regularization of item latent representations. Experiments using simulated and real-world data show the joint model is generally more effective than task-specific models, with an average increase of 16% in R@30 on real datasets, primarily due to latent representation regularization ([H2]).

Applications for this technology span platforms like Spotify, YouTube, and Netflix. However, generative retrieval still faces scalability challenges with large item sets. Furthermore, effectiveness gains depend on factors like popularity distribution alignment and item co-occurrence patterns across tasks. This research represents a significant stride towards developing unified LLMs for diverse IR functions.

Paper: https://arxiv.org/pdf/2410.16823