Introduction to Modern Recommender Systems

In the dynamic world of modern internet companies, particularly those managing apps with tens of millions of daily active users, personalized recommender and search systems are critical. These systems are engineered to offer customized content to their extensive user base. A key component in these systems is the development of effective and innovative features for large-scale personalized models. These models are massive, with the total number of parameters often reaching hundreds of gigabytes, and sometimes even exceeding 1 terabyte in size, stored in float32 format.

The Role of Machine Learning Engineers in Feature Development

Feature Development: This may involve adding event tracking to gather user behavioral data or processing historical data to extract new feature fields.
Offline Evaluation: To optimize model performance, features are added in a continuous or discretized manner. This step often involves exploring feature combinations to maximize utility. Evaluation is typically based on offline metrics like AUC. Efficiency dictates that this process should be batch-based rather than feature-by-feature, as single feature additions rarely yield substantial improvements, especially in later stages of system development.
Online A/B Testing: The effectiveness of features is verified through A/B testing against final business targets (like Order/GMV/StayTime). Successful features are then implemented online.

Challenges in Managing Feature Embeddings

However, this seemingly robust process is not without its challenges. Over time, a major issue arises: an enormous table of feature embeddings. In a recommendation model, over 90% of the parameters may be in these embeddings rather than the model structure itself. This leads many companies to adopt parameter server architectures for training, where each worker only accesses the necessary feature embeddings for their batch. They calculate gradients and send them back to the server for aggregation.

Evolving Business Needs and Feature Management

As business needs evolve, new features are constantly added. However, due to historical deployment practices, features are often added in batches, resulting in many inefficient or outdated features in the online system. These features, as per my experience, can consume substantial training and inference resources, sometimes accounting for over 20% of the total model parameters.

The Importance of Feature Selection

Identifying and addressing this issue is crucial. The solution is conceptually straightforward: identify the most important features and discard the useless ones.
After a period of system evolution and growth, it becomes essential to evaluate the significance of each feature, for which we use Permutation Importance. This technique entails loading an already trained model and then systematically altering the values of a specific feature across various records. The goal is to measure how these alterations affect the overall performance of the model. As for the finer details of Permutation Importance, they fall outside the scope of this discussion, but I encourage you to explore other resources for a more in-depth understanding.

Insights from CTR/CVR Model Feature Development:A Real Word Example

Shifting focus to the findings from our CTR/CVR model feature development, we observed certain phenomena that underscore general principles in funnel conversion modeling:
Price Features in CTR vs. CVR Models: While price plays a crucial role in CTR models, its significance is remarkably lower in CVR models. This might seem counterintuitive at first — isn’t price important for final transactions? The answer lies in the funnel process: by the time of clicking, users have already expressed their price preferences, filtering out uninteresting products. Thus, features critical in the upper funnel stages may become less relevant downstream.
Top Features in CTR and CVR Models: For CTR models, the most important features are user_id, item_id, and query_id. However, this isn’t the case for CVR models, where generalized statistical features take precedence. ID features don’t dominate the importance rankings. This is because click behavior is more transient. CTR models effectively learn and utilize these ID features, but CVR models struggle due to the sparsity of such data. Instead, features like recent user exposures, clicks, and transactions become more pivotal.

Conclusion

This exploration into feature importance not only enhances the efficiency of our models but also provides deeper insights into user behavior and system performance, guiding future developments in our e-commerce platform’s shopping search functionality.
In future posts, I’ll delve into more intriguing tales about feature development and the surprising discoveries we’ve made along the way. Stay tuned for more insights into the captivating world of e-commerce recommendation&&search systems.