Improving Shorts Recommendations Without a Development or ML Background
I joined a project to improve Langflix's Shorts recommendations. Until then, my work had mostly involved analyzing data with SQL and Python; I had little experience with software development, machine learning, or deep learning. I took on everything from analysis to define our goal to pipeline work and A/B test design, so I had to learn and apply many unfamiliar concepts.
This is a look back at my work: deciding what the recommendations should improve, learning what I needed along the way, and reflecting on the experiment and what I would change.
What we had to decide before improving recommendations
At the start, my first question was why improving recommendations mattered. Would it be enough to make people watch more videos? I felt we needed to understand the role Shorts played in the app before setting a goal.
To me, the Shorts tab was a place where users could quickly discover Langflix's content and features. Shorts were easier to start than long-form videos, and users could save expressions or use shadowing and listening exercises while watching.
The analysis did not fundamentally change that view. At the time, Shorts were already serving that role. But as I considered how they could help users experience the product's value sooner, my focus shifted.
The learning features were already on screen. If we showed a video that interested a user, might they be more likely to save an expression or repeat a line aloud while watching?
I looked across the app for behaviors associated with retention. Our analysis focused on review, bookmarks, saved expressions, shadowing, and listening. Making these behaviors the goal of our recommendations let us discuss what to improve in concrete terms.
This direction combined analysis and judgment. We examined the relationship between learning actions and retention, then chose to try increasing those actions in Shorts. Whether a new recommender would actually change behavior was a question for the experiment.
How much can recommendations solve?
When I used the app, similar videos kept appearing. The same thing happened on other team members' accounts. At the time, recommendations were based mainly on popularity and when videos were added to the platform, and the content pool was limited.
Honestly, I thought content supply was more urgent. To choose the right video for someone, there first have to be enough videos to choose from. I also questioned how much learning behavior the available content could encourage.
Still, that was no reason to leave the existing recommendations as they were. We could better reflect each user's interests and responses while expanding the content supply.
A Theta One developer built a pipeline to automate content sourcing, allowing us to work on recommendations in parallel. I also helped with the pipeline that retrieved content from the video pool.
We considered different approaches for users with little history and those with enough history. We needed to decide what evidence to use for new users and which actions to consider for returning users. The team explored content similarity, popularity, matrix factorization, and Two-Tower models.
In those discussions, I tried not to lose sight of the problem we were solving. When the technical details got deep, I revisited the overall direction and asked how each proposed method related to our goal.
The difficulty feature engineering I took on was also meant to help us understand user responses. I calculated vocabulary level and word frequency in subtitles and planned to examine how those values related to watching, clicking expressions, and saving them.
Even choosing how to calculate difficulty required care. Different vocabulary sources could produce different scores for the same subtitles, and we needed a rule for words missing from a source. We could compare difficulty with behavioral data only if we understood what the calculated value meant.
Interpreting behavior also required caution. A video with more clickable expressions might naturally get more clicks. Without accounting for that opportunity, we could mistakenly attribute the extra clicks to difficulty. That is why I thought about validating a feature's usefulness while building it.
Working without a development or ML background
There was a wide range of things I did not know when I first encountered recommendation systems. I had to learn not only what individual models did, but how models interacted and how their outputs could be combined. Collaborating through Git and using a database were unfamiliar as well.
I found that AI could dramatically speed up that learning. I kept asking about concepts I did not understand, and I also asked my teammates. One moment I was asking about models and ensembles; the next I was asking how to use Git for the work at hand.
I could not study everything before starting. I learned the concepts I needed while doing my tasks, then returned to the work. Experience in data analysis did not automatically make discussions about development and models easy to follow.
My role was also hard to sum up in one phrase. Alongside helping think through the overall direction, I worked on the content retrieval pipeline, difficulty features, and a pipeline for A/B testing. Each required different knowledge, so each task brought something new to learn.
It helped to have ways to ask questions and understand unfamiliar material. With AI, I could ask about the same concept repeatedly or request a different explanation without hesitation. Discussing it with teammates was just as important for filling gaps in my understanding.
What would prove that the recommendations improved?
Even after we chose to increase learning actions, we had many decisions to make for the experiment. Our metric would change depending on what counted as one impression and which video received credit for a learning action.
Initially, we defined the metric by whether a view and a learning action occurred in the same session. Later, we changed it to link views and learning actions at the level of each video impression.
A user can watch several videos in a session. Knowing that a learning action happened during that session does not tell us which video prompted it. Evaluating individual impressions required data that connected the action to the right video exposure.
We used an impression_id to distinguish each exposure. It let us distinguish repeated views of the same video and attach viewing and learning actions to a specific impression.
We also had to distinguish a recommendation request from an actual exposure. The app might fetch the next video in advance even if the user never sees it. Counting every API response as an impression would inflate the denominator of the learning-action rate. We therefore designed the metric around videos actually shown on screen.
Assignment to the treatment group did not always mean the new recommender served the video. A model or data problem could cause the old recommender to be used as a fallback. We recorded both the assigned and the executed policy separately, and designed the experiment to compare users by their assigned group.
After deployment, we had to confirm that these fields were actually recorded. If an impression had an ID but a learning action did not, the events could not be linked. Duplicate actions would inflate the count. Early on, issues like these meant we could only examine results exploratorily.
Designing the A/B test pipeline showed me how closely a metric's definition depends on implementation. Two analyses can both be called a "learning-action rate" yet produce different values if they record and link exposures and actions differently.
In the A/B test run after the last-week-of-August release, the treatment group's learning-action rate was about 4–5 percentage points higher than the control group's. We could see a difference between the groups in the behavior we had targeted.
What I would clarify first next time
My biggest regret is the initial exploratory data analysis. At first, I wanted to set a direction quickly, split up the analysis, and work in parallel. But recommendation systems were new to all of us, so learning what to examine took time too.
Looking back, I think we could have written down the questions to analyze and the decisions each answer would inform much earlier. Separating analysis needed to choose a recommendation approach from analysis that could wait would have made it easier to divide responsibilities.
Next time, before starting the analysis, I would define the questions each person will investigate and set a time to bring the results together. A shared agreement at that level would have made the parallel work we wanted more effective.
