Distilled Neural Networks for Efficient Learning to Rank
| Autor: | Nardini F.M., Rulli C., Trani S., Venturini R. |
|---|---|
| Rok vydání: | 2022 |
| Předmět: |
Matrix multiplication
FOS: Computer and information sciences Computer Science - Machine Learning Computer Science - Performance Computer Science - Artificial Intelligence Learning-to-rank Efficiency Computer Science - Information Retrieval Computer Science Applications Pruning Machine Learning (cs.LG) Performance (cs.PF) ComputingMethodologies_PATTERNRECOGNITION Artificial Intelligence (cs.AI) Computational Theory and Mathematics Web search Neural networks Information Retrieval (cs.IR) Information Systems Distillation |
| Zdroj: | IEEE transactions on knowledge and data engineering (Online) 35 (2022): 4695–4712. doi:10.1109/TKDE.2022.3152585 IEEE transactions on knowledge and data engineering (Online) (2022): 103330. doi:10.1109/TKDE.2022.3152585 info:cnr-pdr/source/autori:Nardini F.M.; Rulli C.; Trani S.; Venturini R./titolo:Distilled neural networks for efficient learning to rank/doi:10.1109%2FTKDE.2022.3152585/rivista:IEEE transactions on knowledge and data engineering (Online)/anno:2022/pagina_da:/pagina_a:103330/intervallo_pagine:103330/volume |
| DOI: | 10.48550/arxiv.2202.10728 |
| Popis: | Recent studies in Learning to Rank have shown the possibility to effectively distill a neural network from an ensemble of regression trees. This result leads neural networks to become a natural competitor of tree-based ensembles on the ranking task. Nevertheless, ensembles of regression trees outperform neural models both in terms of efficiency and effectiveness, particularly when scoring on CPU. In this paper, we propose an approach for speeding up neural scoring time by applying a combination of Distillation, Pruning and Fast Matrix multiplication. We employ knowledge distillation to learn shallow neural networks from an ensemble of regression trees. Then, we exploit an efficiency-oriented pruning technique that performs a sparsification of the most computationally-intensive layers of the neural network that is then scored with optimized sparse matrix multiplication. Moreover, by studying both dense and sparse high performance matrix multiplication, we develop a scoring time prediction model which helps in devising neural network architectures that match the desired efficiency requirements. Comprehensive experiments on two public learning-to-rank datasets show that neural networks produced with our novel approach are competitive at any point of the effectiveness-efficiency trade-off when compared with tree-based ensembles, providing up to 4x scoring time speed-up without affecting the ranking quality. Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|