2022 LG Uplus AI Ground (1)

Modeling

In this section, we detail the modeling approaches used during the competition, focusing on the development, experimentation, and optimization of two recommendation algorithms: Neural Collaborative Filtering (NCF) and LightGCN.

728x90

Neural Collaborative Filtering (NCF)

• NCF Paper
• NCF
• Matrix Factorization
• Optuna

https://recbole.io/docs/user_guide/model/general/neumf.html

The Netflix Prize established matrix factorization (MF) as a fundamental technique in latent-factor-based recommendation systems. However, MF alone struggles to capture complex user-item interactions. To overcome this limitation, we implemented NeuMF, which combines MF with a deep neural network (DNN) to incorporate additional user and item features.

1. Initial Setup:
   • We implemented MF to process user and item data.
   • This initial implementation achieved a score of 0.2143.
2. Feature Engineering:
   • Rather than focusing solely on hyperparameter tuning, we enriched the model with diverse features.
   • Adding significant features like age, gender, protagonist data, etc. improved the score to 0.2167.
3. Architectural Adjustments:
   • Based on the NCF paper, we optimized the DNN layer count and final hidden layer output size.
   • These modifications increased performance to 0.2180.
4. Hyperparameter Optimization:
   • Following feature and architecture refinements, we conducted systematic parameter tuning.
   • We employed grid search for resource-intensive parameters and Optuna for lighter configurations.
   • This optimization achieved our best score of 0.2230.

LightGCN

• Bayesian Negative Sampling (BNS) for Recommendation Paper
• LightGCN Paper

From the LightGCN paper

In the final phase of the competition, we explored alternative models and sampling techniques. We discovered the Bayesian Negative Sampling (BNS) method from a July 2022 paper and implemented it with LightGCN—a streamlined version of the Graph Convolutional Network (GCN) that represents the current state of the art in collaborative filtering.

1. Implementation of LightGCN with BNS:
   • LightGCN enhances efficiency by streamlining traditional GCN, retaining only the essential embedding propagation process.
   • We integrated BNS to enhance negative sampling through item-level uncertainty modeling.
2. Challenges and Results:
   • The LightGCN-BNS combination required substantially more computational resources and training time than NeuMF.
   • The model's performance matched but did not exceed our NeuMF implementation with Popularity-biased Negative Sampling (PNS).

Given these limitations, we ultimately decided to concentrate our efforts on optimizing NeuMF instead.

Results

Out of 216 participating teams, our team secured 26th place in the competition.

 

During this intensive three-and-a-half-week competition, our team of three members worked collaboratively, each contributing unique perspectives. Our frequent discussions about the dataset and various methodological approaches enhanced our analytical capabilities and expanded our problem-solving horizons.

 

Through this experience, we gained hands-on expertise with recommendation system theory, PyTorch, and data preprocessing, deepening our practical understanding of these technologies.

Areas for Improvement

Despite achieving a respectable ranking, our analysis revealed several opportunities to refine our approach in data preprocessing, modeling, and resource management.

Data Preprocessing

1. Sample Handling:
   • While we successfully removed unnecessary features, we kept all available samples—a decision that proved inefficient.
   • A more selective preprocessing strategy prioritizing data quality over quantity could have yielded better results, even if it meant discarding some samples during merging.
2. Feature Selection:
   • Using techniques like dimensionality reduction, feature selection, and feature extraction could have helped identify the most impactful features.
   • These methods would have provided a more systematic approach to feature elimination.
3. Popularity-biased Negative Sampling (PNS):
   • Rather than relying solely on global view counts, we could have enhanced sampling quality by incorporating time of viewing, age, gender, and click-through rates into PNS.
4. Exploring Additional Negative Sampling Algorithms:
   • Beyond our experiments with RNS and PNS, we could have tested other promising approaches.
   • Model-based methods like Dynamic Negative Sampling (DNS) and Simplify and Robustify Negative Sampling (SRNS) could have enhanced performance.

Modeling

1. Limitations of NeuMF:
   • We aimed to experiment with larger models (e.g., increasing layer count and dimension size) to enhance NeuMF's performance.
   • However, limited resources (e.g., Colab Free and Colab Pro) caused frequent kernel crashes, preventing further exploration.
2. Alternative Model Architectures:
   • NeuMF combines MF with DNN, but experimenting with alternative models in place of DNN could have been experimental.
   • For instance, substituting DNN with architectures like recurrent neural networks (RNNs) or transformer-based models might have provided new insights.
3. LightGCN Optimization:
   • Although we integrated BNS with LightGCN, the results were underwhelming, leading to its discontinuation.
   • A missed opportunity was applying LightGCN with simpler sampling methods, such as RNS or PNS, instead of directly combining it with BNS.
   • Using LightGCN as a direct alternative to NeuMF might also have yielded interesting results.

This reflection highlights valuable lessons from the competition and identifies clear pathways for future improvements.