New AI Research Proposes Effective Optimization Method of Adaptive Column-Wise Clipping (CowClip) Reducing CTR Prediction Model Training Time from 12 Hours to 10 Minutes on 1 GPU


Are you looking for a way to speed up your online commerce, video applications, and web adverts? It’s time to take advantage of Clickthrough Rate (CTR) prediction! In this blog post, we’ll explore how to increase the batch size of a CTR prediction model with a simple and effective scaling technique.

CTR prediction is a time-sensitive activity. It requires accurate and up-to-date models, which can be achieved by reducing the time needed for re-training on a large dataset. Luckily, GPU processing power has been growing rapidly, allowing for larger batch sizes to benefit from the parallel processing capacity.

However, when increasing the batch size, it is important to preserve accuracy. To do this, we must understand why the scaling rules used in CV and NLP tasks are not well-suited for CTR prediction. This is because the embedding layers dominate the parameters of the whole network in CTR prediction and the inputs are more sparse and frequency-unbalanced.

In this study, we provide a successful algorithm and scaling control for large batch training. We also provide an efficient optimization strategy called adaptable Column-wise Clipping (CowClip) to stabilize the training process of the CTR prediction task. We successfully scale up four models 128 times the batch size on two open datasets. On the Criteo dataset, we specifically train the DeepFM model with a 72-fold speedup and 0.1% AUC increase.

In conclusion, we have demonstrated that with careful mathematical analysis and an efficient optimization strategy, large batch training for CTR prediction is possible. To learn more, check out the paper and Github, and join our 13k+ ML SubReddit, Discord Channel, and Email Newsletter.

Published
Categorized as AI

Leave a comment

Your email address will not be published. Required fields are marked *