What I Learned in Eight Weeks Building Langflix's Shorts Recommendation System
Hello! Last summer, Theta One and Growth Hackers, a business data club at Seoul National University, worked together on an industry collaboration project to build a Shorts recommendation system for Langflix. As the project manager for Growth Hackers, I want to share what I learned during those eight weeks.
Can Shorts recommendations improve retention?
Our main goal was to improve retention. Shorts are an easy entry point into the Langflix app. We hoped personalized videos would spark interest, help users form learning habits, and ultimately bring them back to the app.
Rather than immediately researching which model to use, we spent time understanding Langflix users and defining the purpose of the project. We analyzed current retention, session length, and usage patterns so that we could establish clear goals and KPIs before development.
Our exploratory data analysis showed that, among users who had visited the Shorts tab, those who took learning actions, such as today's review, shadowing, listening, or saving expressions, had higher retention than those who only watched. We also found that the existing definition of a meaningful action counted watching alone, and most actions counted as meaningful were views without further learning. This led us to recommend content that could naturally prompt learning actions.
After the analysis, we redefined a meaningful action around behaviors associated with a notable increase in retention, taking into account Langflix's role as a language-learning app and Theta One's direction. We chose the learning-action rate per video as our KPI and decided to build a recommendation system that could improve it.
Without that analysis, the project might have taken a very different direction. Maximizing watch time alone would have conflicted with the app's identity and might not have served users who came to learn a foreign language.
Once the goal was clear, it became easier to see which new features the system needed. I learned how important it is to define the project's purpose precisely and keep the team aligned.
An eight-week roadmap: From data to A/B tests
The project lasted eight weeks. The Growth Hackers team devoted their entire summer break to thinking through and building it.
At kickoff, we received access to Theta One's GitHub repositories and database. We first studied the Shorts user flow, database schema, and event logs. After tracing how the existing recommendation system worked and using exploratory analysis to align on a direction, we prepared to develop the new system.
That analysis led us to consider separate strategies for cold and warm users. We refined Theta One's existing video embeddings and clustering, created features that combined onboarding survey answers and meaningful-action history to capture user preferences, and recalculated Shorts difficulty using vocabulary level and speaking speed. We also added a filter for videos with too little speech to be useful for learning.
With those ingredients ready, we built two recommendation models and ran A/B tests. The first used cohort-level topic preferences: it grouped users into cohorts, built candidate sets from videos generally preferred within each cohort, then personalized individual video choices using each user's topic preferences. The second was a hybrid that applied different systems to cold and warm users. Cold users received recommendations based on content-based collaborative filtering, while warm users received recommendations from a deep-learning Two-Tower model.
After designing the models, we also updated the serving infrastructure. The simpler cohort-based model worked on the existing TypeScript server, but the deep-learning Two-Tower system required Python. With access to AWS, we used S3 and DynamoDB to build an environment for the model and put it into service. We then A/B tested both models, monitoring the KPI and supporting metrics in PostHog. The Two-Tower model, in particular, showed a meaningful increase in the KPI.
Questions we faced while implementing the system
In class, learning about recommendation systems often means understanding how a model works. Building one for a real service raised many more questions: how to design user and item features, which principles should guide recommendation policy, and how to collaborate so Theta One could keep using the system after our project ended. I learned firsthand that, alongside understanding and improving model performance, we had to design for reliable operations.
When introducing a new system, asking "How will we evaluate what we built?" matters as much as asking "What should we build?" Viewing histories are shaped by the previous recommendation system and are therefore biased. In this project, standard metrics such as Recall and NDCG gave the highest scores to models that simply ranked videos by popularity or recency. We considered IPS and SNIPS analysis, but the old system did not have the propensity information needed for it. Instead, we defined offline metrics suited to our goal, including Cold Recall, which measures whether recommendations include items users liked despite those items having no exposure history, and Ordinal Accuracy, which measures how well the model reorders a user's viewing history by preference.
In an age of coding agents, planning matters even more
Theta One generously provided Codex, so we could freely use a coding agent during the project. It made it faster and easier to implement structures we had in mind and connect them to the server.
The help also made me realize that we could not leave every decision to an agent. When we delegated choices that required product judgment and simply said "make it," the results were incomplete. They often looked plausible on the surface, but on closer inspection their intent was hard to understand or they conflicted with human judgment.
An agent could not decide for us what counted as a successful recommendation, how to separate cold and warm users, or whether performance came at the expense of user experience. AI can implement remarkably well, but people first need to plan and design what it should implement. I found that a well-written product requirements document made AI far more effective.
As both the PM of this collaboration and a Langflix user, I spent last summer immersed in recommendation development. I learned that a recommendation system is not just a model that predicts the right answer: it is a way to understand users within a product and encourage outcomes that serve the product's goals. I sincerely hope the system we built helps Langflix users have a better learning experience. 🙂
