Knowledge Transfer via Pre-training for Recommendation: A Review and Prospect

Abstract

Recommender systems aim to provide item recommendations for users and are usually faced with data sparsity problems (e.g., cold start) in real-world scenarios. Recently pre-trained models have shown their effectiveness in knowledge transfer between domains and tasks, which can potentially alleviate the data sparsity problem in recommender systems. In this survey, we first provide a review of recommender systems with pre-training. In addition, we show the benefits of pre-training to recommender systems through experiments. Finally, we discuss several promising directions for future research of recommender systems with pre-training. The source code of our experiments will be available to facilitate future research.

1 Introduction

With the rapid development of the Internet, users are faced with information overload, where the large quantity of online items makes it hard for users to make decisions effectively. Recommender systems aim to provide recommendations by capturing user preferences for items (e.g., movies, books, songs, news, and websites) from explicit item ratings given by users or implicit user-item interactions (e.g., browsing and purchasing histories). The application of recommender systems has enabled personalized services in many scenarios, such as e-commerce and website browsing.

Recommender systems are usually faced with data sparsity in real-world scenarios. Recommender systems can suffer when providing recommendations for new items or users due to lack of information, which is known as the cold start problem (Gope and Jain, 2017). Without sufficient data, the model parameters cannot be well estimated and users’ preference cannot be well modeled. It has been shown that the data sparsity problem in recommender systems can be alleviated by transferring knowledge from other domains or tasks (Cantador et al., 2015) and integrating heterogeneous external knowledge (Guo et al., 2020).

In the field of natural language processing, pre-trained models have achieved great success recently on a broad variety of tasks by knowledge transfer (Qiu et al., 2020). Models are usually first pre-trained on large-scale unsupervised data to learn universal language representations and then fine-tuned on downstream tasks to achieve knowledge transfer. The models can be pre-trained to learn either shallow context-free word embeddings (Mikolov et al., 2013) or deep context-aware language representations (Devlin et al., 2019). The resulting language representations have proven to be useful not only for different tasks [e.g., natural language inference and question answering (Devlin et al., 2019)] but also for different scenarios, such as few-shot learning (Brown et al., 2020) and domain adaptation (Rietzler et al., 2020).

In the context of recommender systems, we can group works that utilize pre-training mechanisms to improve the precision of recommendation into two categories: feature-based models and fine-tuning models. The feature-based models generally use pre-trained models to obtain features from side-information (e.g., the content of items and knowledge graphs) for users and items (Guo et al., 2020). The fine-tuning models leverage the user-item interaction records to pre-train a deep transferable neural model, which is subsequently fine-tuned to downstream recommendation tasks (Chen et al., 2019c). Generally, the benefits of pre-training to recommender systems can be summarized as being twofold: 1) pre-training tasks can better exploit user-item interaction data to capture user interests, and 2) pre-training can help integrate knowledge from different tasks and sources into universal user/item representations, which can be further adapted to various scenarios in recommender systems, such as cold starts and cross-domain transfer.

The contributions of this survey can be summarized the following. 1) Systematic Review: We provide a systematic review of the pre-training methods for recommender systems with a clear taxonomy. 2) Empirical Results: We present empirical results to show the benefits of pre-training to recommender systems. We conduct experiments on the task of movie recommendation where different types of knowledge are integrated by pre-training for better recommendations, especially in the cold start and cross-domain transfer scenarios. 3) Future Directions: Based on the review and experiments, we discuss several promising directions for future research, including how to better leverage pre-training approaches to improve recommender systems and how recommender systems can motivate better pre-training models.

The rest of the survey is organized as follows: In Sections 2 and 3, we provide a review of existing methods of recommender systems with pre-training; in Section 4, we conduct experiments to empirically show the benefits of pre-training to recommender systems; and in Section 5, we discuss promising directions for future research.

2 Feature-Based Models

Feature-based models leverage side-information (e.g., contents of items, knowledge graphs, and social networks) using pre-trained models to directly enrich the representations of users or items. Different from collaborative filtering (CF) methods that learn the representations from user-item interaction records (Hu et al., 2008; Su and Khoshgoftaar, 2009; Wang et al., 2020a), feature-based models focus on extracting widely applicable features from external information sources with pre-trained models and then integrate these features into the recommendation framework. By combining rich side-information and user-item interaction data, feature-based models can potentially solve some challenges, such as the data sparsity problem.

The general idea can be illustrated as follows. Given the external information resource, the pre-trained models are applied to obtain the external feature vectors, and for user and item respectively. Denote and as the features learned from the user-item interaction records for user and item respectively. Then the final representations and for each user and item are obtained by aggregating external feature vectors and the vectors from the user-item interaction data:where and are aggregate functions. The preference score for user and item is calculated bywhere is a recommendation function, which can be factorization machines (Rendle, 2010) and deep neural networks (He et al., 2016; Fan et al., 2018) and the like.

According to the type of external information resources, feature-based pre-trained models can be roughly categorized into content-based recommendation, knowledge graph-based recommendation, and social recommendation models. Different types of meta-information require different pre-trained models. We will introduce how the pre-training mechanism is used in these three kinds of recommender systems in detail.

2.1 Content-Based Recommendation

Content-based recommendation assumes that users prefer items similar to those being historically interacted with. Therefore, it is important for content-based recommender systems to encode the content of items into expressive low-dimensional representations. The pre-trained models have proven to be powerful in extracting generally applicable representations from text, images, audio, etc. Hence, many works learn features from the content of items to serve recommendation models.

Liang et al. (2015) pre-train a multilayer neural network to extract audio features for music recommendation via a semantic tag prediction task. In terms of dealing with textual data for recommendation, such as reviews (Zheng et al., 2017), tweets (Gong and Zhang, 2016), and news (Cao et al., 2017), pre-trained word embeddings or pre-trained sentence encoders become indispensable. Some works simply use the average of word embeddings to represent the whole documents (Brochier, 2019; Cenikj and Gievska, 2020). Other works focus on the design of task-specific frameworks, where the input word embeddings will be fed into a complex document encoder to generate the document representation (Song et al., 2016; Tan et al., 2016; Xu et al., 2016; Nguyen et al., 2017; Zhang et al., 2017). Similarly, the pre-training mechanism is widely used in image feature extraction for recommender systems (Chu and Tsai, 2017; He and McAuley, 2016a, He and McAuley, 2016b).

2.2 Knowledge Graph-Based Recommendation

Knowledge graph-based recommendation introduces knowledge graphs (KGs) as side-information to better characterize users and items. A KG is a structured graph containing fruitful facts and connections between users, items, and other related entities. Amounts of side-information, such as the user profiles, the attributes of items, and relations between cross-domain items, can be integrated into KGs. Hence, KGs can help recommender systems to capture essential knowledge and provide explanations for the recommendation results.

Various KGs have been used in different works. For instance, some works construct knowledge graphs with items and their related attributes (Zhang et al., 2016; Wang et al., 2018b; Huang et al., 2018). Some other works add users to build user-item graphs, which contain information, including the item attributes, user behaviors (e.g., purchase, click), and user profiles (Wang et al., 2018a; Cao et al., 2019; Dadoun et al., 2019). With the informative heterogeneous user-item graphs, the potential relations between users and items can be modeled directly.

In order to exploit KGs, one line of KG-based methods seeks to encode the KG into low-dimensional pre-trained embeddings with the knowledge graph embedding (KGE) methods (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015; Ji et al., 2015; Trouillon et al., 2016; Shi and Weninger, 2017; Balazevic et al., 2019). Then as stated in Eq. 1, the knowledge graph embeddings are aggregated with user/item features obtained from interaction data. Experimental results show that KGs are powerful information resources and can improve the performance of recommendation significantly (Qin et al., 2020).

2.3 Social Recommendation

Social recommendation is a type of recommendation method that utilize online social relations as an additional input (Tang et al., 2013). Different from KGs, which integrate various information about users and items, social graphs focus on modeling the social relation between users. Homophily theory indicates that the preference of a user is similar to or influenced by their socially connected friends (McPherson et al., 2001). Similar to the KG-based recommendation, many social recommender systems seek to integrate the pre-trained social network embeddings, which indicates the degree that a user is influenced by his/her friends (Guo et al., 2018; Wen et al., 2018; Zhang et al., 2018; Sathish et al., 2019; Chen et al., 2019a, Chen et al., 2019b).

2.4 Summary

Feature-based models preprocess side-information with various pre-trained models to obtain the embedding of users or items, which are then integrated into the recommender systems. By utilizing side-information, feature-based approaches are able to construct expressive representations for users and items and can achieve significant improvement for recommendation. In addition to have side-information, exploiting large-scale interaction data is also crucial to recommender systems. Therefore, some recent efforts have been made to pre-train models with user-item interaction records, which are introduced in the following section.

3 Fine-Tuning Models

The fine-tuning models for recommendation first pre-train the parameters with large-scale interaction data. The models are then transferred to downstream tasks by simply fine-tuning the pre-trained parameters. The fine-tuning paradigm has shown the effectiveness in other areas, such as natural language processing (Devlin et al., 2019; Joshi et al., 2020). According to the model architecture, we can categorize existing works in recommender systems into two classes: shallow neural networks (Hu et al., 2018; Ni et al., 2018) and deep residual neural networks. Existing deep residual neural networks for recommendation can be further divided into BERT-based models (Chen et al., 2019c; Sun et al., 2019; Yang et al., 2019) and parameter-efficient pre-trained convolutional neural networks (Yuan et al., 2020).

3.1 Shallow Neural Networks

Early works attempt to achieve knowledge transfer with shallow neural networks as base models, such as shallow multilayer perceptrons (MLP), and recurrent neural networks. Hu et al. (2018) attempt to improve recommendation via cross-domain knowledge sharing. They conduct a baseline experiment where an MLP together with user and items embeddings is pre-trained on the source domain. The user embeddings are then transferred to the target domain as warm-up. Experimental results show that this simple method cannot achieve obvious improvement in recommendation performance. The results demonstrate that the model architecture and pre-training tasks need to be carefully designed to achieve effective knowledge transfer in fine-tuning models. Therefore, many efforts have been devoted to investigating efficient pre-training tasks and transferrable model architectures.

Ni et al. (2018) propose the DUPN model which is able to learn universal user representations across multiple recommendation tasks. DUPN takes interaction sequence as inputs and then applies LSTM and an attention layer to obtain the user representations. DUPN is pre-trained by multiple tasks objectives, including click-through rate prediction, shop preference prediction, and price preference prediction. Experimental results show that DUPN can achieve not only improvement on the tasks used for pre-training but also faster convergence and promising results in related new tasks. Though the user representations learned with DUPN are powerful, DUPN requires many extra information sources like user profiles to facilitate different pre-training tasks.

3.2 BERT-Based Models

In order to capture the dynamic user preference, many researchers attempt to exploit the user chronological interaction sequence, which is called the session-based recommendation. Similar to natural language processing (NLP) that targets on word sequence, session-based recommendation investigates the item sequence and aims to take sequential information into account. Inspired by the rapid progress in pre-trained language models in NLP (Devlin et al., 2019; Liu et al., 2019; Joshi et al., 2020), many efforts have been devoted to capturing information from the user behavior sequence with pre-trained models, especially BERT-based models. In this section, we introduce the pre-trained BERT-based models for recommendation, including the widely used masked item prediction task, the architecture of BERT, and advanced BERT-based models for recommendation.

3.2.1 Masked Item Prediction

Similar to the masked language modeling task in NLP, the masked item prediction task (MIP) is proposed and widely applied in many recommender systems. In the MIP task, given the interaction sequence, some of the items are randomly masked. The models are required to reconstruct the masked item. Formally, we denote the interaction sequence in chronological order for user u as , where is the item that u interacted with at time step i. For pre-training stage, some input items in the interaction sequence are randomly masked with special token [MASK]. Then the models are asked to predict the masked items:

The objective of this task is the negative likelihood of the masked targets. Unlike the left-to-right Next Item Prediction (NIP) task that is used in many session-based recommender systems (Yu et al., 2016; Wu et al., 2017; Hidasi and Karatzoglou, 2018), MIP enables the models to learn representations of user behavior sequences from the whole context. Moreover, the MIP task can overcome the limitation that a rigidly ordered sequence is not always practical in user behaviors (Sun et al., 2019). Therefore, models pre-trained with the MIP task can achieve promising results.

3.2.2 BERT for Recommender System

FIGURE 1

The sublayer consists of two linear transformations and a ReLU activation. It should be noted that though the transformations are the same across different positions, the parameters are different for different layers.

Following each sublayer, a residual connection (He et al., 2016) and a layer normalization operation (Ba et al., 2016) are employed for stabilizing and accelerating the network training. As shown in Figure 1B, after L layers of Transformer, the final hidden states of masked tokens are fed into a feedforward network to get the output distribution over target items.

BERT4RS is effective in modeling user preference from the historical behaviors. Sun et al. (2019) train a two-layer BERT with the MIP task, which achieves the state-of-the-art performance on the session-based next item recommendation task. They observe that both the BERT architecture and the MIP task can significantly improve the performance, and stacking multiple Transformer layers can further boost the performance on large-scale datasets, which provides fundamental support for the following models.

Chen et al. (2019c) proposed to fine-tune BERT4RS with a content-based click through prediction task. Specifically, the user representation is produced with pre-trained BERT from the historical behavior sequence, and the item representation is extracted from its content. Then the preference score is calculated with Eq. 2, where is an MLP layer.

Some downstream recommendation tasks, such as next basket recommendation (Rendle et al., 2010; Yu et al., 2016) and list-wise recommendation (Shi et al., 2010; Zhao et al., 2017), require the model to capture the relations between item sequences. Therefore, in addition to MIP, researchers propose some sequence-level pre-training tasks to pre-train the model. Yang et al. (2019) adopts the BERT4RS for next basket recommendation task, which is pre-trained with MIP and next basket prediction (NBP) tasks. In real-world scenarios, a user usually buys or browses a series of items (a basket) at a time. Given two baskets, NBP requires the model to predict whether the two baskets are adjacent in the purchase records:where and are items from different baskets and [CLS] and [SEP] are special tokens. The final hidden state of [CLS] is used to predict the NBP label.

3.3 Parameter-Efficient Pre-trained Model

The pre-training mechanism can enable models to capture user preference from behavior history via self-supervised learning. Experimental results show that it can achieve significant improvement for recommendation. However, fine-tuning models separately for different tasks is computationally expensive and memory intensive (Mudrakarta et al., 2018; Stickland and Murray, 2019), especially for resource-limited devices.

To address this issue, Yuan et al. (2020) proposed the Peterrec, which utilizes a grafting neural network in fine-tuning, termed as model patch (MP). By inserting MPs in the pre-trained models, the fine-tuning networks can keep all pre-trained parameters unchanged. As shown in Figure 2, Peterrec is a stack of dilated convolutional (DC) layers (Yu and Koltun, 2016), and there is a residual connection between every two DC layers. Similar to other pre-trained models, Peterrec employs the MIP task to pre-train. During the fine-tuning stage, the MP blocks, which are simple residual two-layer convolutional networks, are inserted around the original DC layers. The pre-trained parameters are shared across different tasks. Only parameters of the MP blocks will be fine-tuned. To accelerate fine-tuning and minimize the number of parameters, the MP blocks are designed in a bottleneck architecture. Specifically, the first convolutional layer projects the k dimensional channels into d () dimensional vectors, and the second layer projects it back to its original dimension. Thus, the number of inserted parameters can be less than parameters of original pre-trained models. Empirical results show that the pre-trained Peterrec is useful for various downstream tasks, including user profile prediction and top-k recommendation. Moreover, it can achieve promising results even when the user is cold in the new target domain, which proves the effectiveness of pre-trained models in knowledge transfer for recommendation.

FIGURE 2

3.4 Summary

The pre-training mechanism works well in many recommendation tasks. Early works explore pre-training with shallow neural networks. Inspired by the success of pre-trained language models in NLP, the deep residual models pre-trained with MIP task are widely used in many different recommender systems. Although it has been proven effective to employ pre-trained models for recommendation, there are many open challenges to be addressed, which will be discussed in Section 5.

4 Experiment of Recommender System With Pre-training

In this section, we conduct experiments to verify the benefits of pre-training to recommender systems. We take next-item and rating prediction recommender systems as examples and investigate the potential of pre-trained recommender systems in cold start problem and cross-domain knowledge transfer.

4.1 Dataset

We evaluate the models on MovieLens¹, which is a representative and popular real-world benchmark in recommendation field. We choose the well-established lightweight version MovieLens 1 m (ML-1m). Each user-item interaction contains a timestamp and an integer rating score from 1 to 5. We follow Tang and Wang (2018) for data preprocessing. Users and items with too few ratings () are filtered out. We leave the last 5 items of each sequence to form the validation set (2 items for each user) and the test set (3 items for each user). To investigate the effectiveness of pre-trained recommender systems in cross-domain context, we divide all the movies in ML-1m into two domains according to movie genres and obtain interaction sequences of the same users on two domains. The scale of the target domain is smaller than the source domain (ML-1m-src). To better evaluate the models under the cold start scenario, we further truncate the interaction sequences in the target domain such that the sequence length of each user is less than 10, which results in the ML-1m-tgt set.

Apart from user-item interactions, we also use the meta information from IMDB² (IMDB 5000 Movie Dataset) to provide side-information for the items. For each movie, we query the director and three main actors. The statistics of the processed datasets are summarized in Table 1.

4.2 Task Settings and Baselines

We regard user ratings as interaction sequence based on timestamps and evaluate the models on two representative recommendation tasks, including NIP and rating prediction. For both tasks, models are first pre-trained on ML-1m-src and then fine-tuned and evaluated on ML-1m-tgt.

Next Item Prediction: NIP aims to predict the next item that is likely to interest the user given the historical interaction sequence. For each sequence, we generate negative samples in which items are not seen in training data. Since movie rating behavior is not strictly sequential, we treat the three items in each test sequence as equivalent. We adopt Normalized Discounted Cumulative Gain (NDCG@K) and Recall@K as evaluation metrics. Models are provided with IMDB 5000 as side-information for items.

For baseline methods, we choose two representative models. Caser (Tang and Wang, 2018) is a shallow recommendation model which combines vertical and horizontal CNNs to extract user and item representations. BERT4Rec (Sun et al., 2019) utilizes BERT as architecture backbone for session-based recommendation. To investigate the potential of pre-training in deep recommendation models, we increase the number of layers in BERT4Rec.

Rating Prediction. We also evaluate the pre-trained recommender systems on the rating prediction task, where models are required to predict the rating of the items based on users’ historical feedback and profile. Following previous works in rating prediction (Zheng et al., 2016; Xing et al., 2019), we formalize the rating prediction task as a regression task and adopt Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) as evaluation metrics. In addition to BERT4Rec (Sun et al., 2019), we choose DACF (Kuchaiev and Ginsburg, 2017) which utilizes deep autoencoder for rating prediction as our baseline models.

4.3 Model Design Choices

We investigate the effect of different design choices on pre-trained recommendation systems, including knowledge transfer approach and pre-training tasks. We also explore the potential of pre-trained recommendation systems in integrating external side-information.

Knowledge Transfer Approach. Our main purpose is to verify the effectiveness of pre-training on recommender systems. Therefore, we first investigate pre-training session-based recommendation models on ML-1m-src before fine-tuning on ML-1m-tgt. We hope that the knowledge learned from pre-training can be transferred between domains. During fine-tuning, we investigate two approaches to knowledge transfer: 1) embedding transfer where we can transfer and fine-tune only the input embedding layer (i.e., user embeddings and external knowledge embeddings) and leave other layers randomly initialized and learned from scratch in the target domain and 2) model transfer where we also explore transferring and fine-tuning the whole model on the target domain. In this way, user-item interaction knowledge is also expected to be transferred³.

Pre-training Tasks: the design of pre-training tasks is crucial for pre-trained models to capture general knowledge from large-scale data. We compare two widely used pre-training tasks in recommender systems: 1) NIP recurrently predicts the next item in a left-to-right fashion, and 2) Masked Item Prediction (MIP), which randomly masks the items and predicts the masked item according to bidirectional context.

External Knowledge: external knowledge is shown to be effective in handling data sparsity problem in recommender systems. We investigate whether external knowledge can be combined with general knowledge learned from pre-training to achieve better results. Specifically, we constrain the item features by concatenating the external knowledge embeddings (i.e., director embeddings and actor embeddings) with the item embeddings to obtain external knowledge enhanced item embeddings.

4.4 Implementation Details

We find the optimal settings of hyperparameters via grid search on the validation set.

For the Caser model, we use implementation provided by the authors⁴. Batch size is searched from and tuned as 512. Hidden dimension size is searched from and tuned as 100. We employ Adam to optimize the model, and we set the learning rate as , weight decay as . For masked item prediction, we set the mask probability as 0.2. Following Devlin et al. (2019), we do not always replace the masked item with [MASK] token. If an item is chosen, we replace it with 1) [MASK] of the time, 2) a random item of the time, and 3) the unchanged item of the time.

For BERT4Rec (Sun et al., 2019) model, we choose a PyTorch implementation⁵. We keep most of the original hyperparameter and initialization strategy settings. Batch size is searched from and tuned as 256. Hidden dimension size is used to search and is tuned to 128. We employ Adam to optimize the model, and we set the learning rate as . We add user embedding and set the user embedding size as 32. We change the layer of BERT blocks to 6. For masked item prediction, we set the mask probability as 0.15 and use the same masking strategy as in Caser.

When incorporating external knowledge, the size of director and main actor embedding is 25 for Caser and 16 for BERT4Rec. The number of negative samples is set to 100. In evaluation, negative samples size is 100 for both models.

For rating prediction task, the rating scores are regarded as classification tags, and we simply concatenate rating embedding for each user-item interaction and set the embedding size to 128. The output layer of BERT4Rec is modified from classification to regression. The model is required to predict the rating scores for rating-masked items during training and the last three items of each user sequence during testing. For collaborative filtering model, we choose a deep learning implementation DACF⁶ and keep the default settings.

4.5 Experiment Results

In conclusion, recommender systems could benefit from pre-training, which effectively transfer knowledge between domains and tasks, and have potential in solving problems, including cold start. It is worth exploring design models to utilize diverse data for various downstream tasks.

5 Open Challenges and Future Directions

6 Conclusion

In this article, we investigate pre-trained models for recommendation and summarize the efforts devoted to this direction. We conduct a comprehensive overview of two types of pre-trained models for recommendation, including feature-based models and fine-tuning models. Then we conduct experiments to show the benefits of pre-training for recommender systems. Finally, open challenges and future direction are discussed, hoping to promote the progress of this domain.

References

• 1

  BaJ. L.KirosJ. R.HintonG. E. (2016). Layer normalization. Preprint: arXiv:1607.06450.

  • Google Scholar

• 2

  BalazevicI.AllenC.HospedalesT. (2019). “Tucker: tensor factorization for knowledge graph completion,” in Proceedings of EMNLP-IJCNLP, Hong Kong, China, November 2029.

  • Google Scholar

• 3

  BordesA.UsunierN.Garcia-DuranA.WestonJ.YakhnenkoO. (2013). “Translating embeddings for modeling multi-relational data,” in Proceedings of the NIPS, Red Hook, NY, December 5–8, 2013, 2787–2795.

  • Google Scholar

• 4

  BrochierR. (2019). “Representation learning for recommender systems with application to the scientific literature,” in Proceedings of the WWW, San Francisco, CA, May 2019, 12–16.

  • Google Scholar

• 5

  BrownT. B.MannB.RyderN.SubbiahM.KaplanJ.DhariwalP.et al (2020). Language models are few-shot learners. Preprint: arXiv:2005.14165.

  • Google Scholar

• 6

  CantadorI.Fernández-TobíasI.BerkovskyS.CremonesiP. (2015). “Cross-domain recommender systems,” in Recommender systems handbook (New York, NY: Springer), 919–959.

  • Google Scholar

• 7

  CaoS.YangN.LiuZ. (2017). “Online news recommender based on stacked auto-encoder,” in Proceedings of the ICIS, Wuhan, China, 24–26 May, 2017 (New York, NY: IEEE), 721–726.

  • Google Scholar

• 8

  CaoY.WangX.HeX.HuZ.ChuaT.-S. (2019). “Unifying knowledge graph learning and recommendation: towards a better understanding of user preferences,” in Proceedings of the WWW, San Francisco, CA, May 2019, 151–161.

  • Google Scholar

• 9

  CenikjG.GievskaS. (2020). “Boosting recommender systems with advanced embedding models,” in Proceedings of the Web Conference, Taipei, Taiwan, April 2020, 385–389.

  • Google Scholar

• 10

  ChenJ.ChenW.HuangJ.FangJ.LiZ.LiuA.et al (2019a). “Co-purchaser recommendation based on network embedding,” in Proceedings of the WISE, Hong Kong, China, November 26–30, 2019 (New York, NY: Springer), 197–211. 10.1007/978-3-030-34223-4_13

  • CrossRef
  • Google Scholar

• 11

  ChenJ.WuY.FanL.LinX.ZhengH.YuS.et al (2019b). N2vscdnnr: a local recommender system based on node2vec and rich information network. IEEE Trans. Comput. Soc. Syst.6, 456–466. 10.1109/tcss.2019.2906181

  • CrossRef
  • Google Scholar

• 12

  ChenX.LiuD.LeiC.LiR.ZhaZ.-J.XiongZ. (2019c). “Bert4sessrec: content-based video relevance prediction with bidirectional encoder representations from transformer,” in Proceedings of the MM, Nice, France, October 2019, 2597–2601.

  • Google Scholar

• 13

  ChuW.-T.TsaiY.-L. (2017). A hybrid recommendation system considering visual information for predicting favorite restaurants, World Wide Web20, 1313–1331. 10.1007/s11280-017-0437-1

  • CrossRef
  • Google Scholar

• 14

  DadounA.TroncyR.RatierO.PetittiR. (2019). “Location embeddings for next trip recommendation,” in Proceedings of the WWW, San Francisco, CA, May 2019, 896–903.

  • Google Scholar

• 15

  DevlinJ.ChangM.-W.LeeK.ToutanovaK. (2019). “Bert: pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the NAACL, Minneapolis, MN, USA, June 2–7, 2019, 4171–4186.

  • Google Scholar

• 16

  FanW.LiQ.ChengM. (2018). “Deep modeling of social relations for recommendation,” in Proceedings of the AAAI, Minneapolis, MN, USA, June 2–7, 2019. (New Orleans, Louisiana, USA: AAAI press), 8075–8076.

  • Google Scholar

• 17

  GongY.ZhangQ. (2016). “Hashtag recommendation using attention-based convolutional neural network,” in Proceedings of the IJCAI. 2782–2788.

  • Google Scholar

• 18

  GopeJ.JainS. K. (2017). “A survey on solving cold start problem in recommender systems,” in Proceedings of the ICCCA, Greater Noida, May 5–6, 2017, 133–138.

  • Google Scholar

• 19

  GordonM. A.DuhK.AndrewsN. (2020). Compressing bert: studying the effects of weight pruning on transfer learning. Preprint: arXiv:2002.08307.

  • Google Scholar

• 20

  GuoL.WenY.-F.WangX.-H. (2018). Exploiting pre-trained network embeddings for recommendations in social networks. J. Comput. Sci. Technol.33, 682–696. 10.1007/s11390-018-1849-9

  • CrossRef
  • Google Scholar

• 21

  GuoQ.ZhuangF.QinC.ZhuH.XieX.XiongH.et al (2020). A survey on knowledge graph-based recommender systems. Preprint: arXiv:2003.00911.

  • Google Scholar

• 22

  HeK.ZhangX.RenS.SunJ. (2016). “Deep residual learning for image recognition,” in Proceedings of the CVPR, Las Vegas, NV, June 27–30, 2016, 770–778.

  • Google Scholar

• 23

  HeR.McAuleyJ. (2016a). “Ups and downs: modeling the visual evolution of fashion trends with one-class collaborative filtering,” in Proceedings of the WWW, Montréal Québec, Canada, April 2016, 507–517.

  • Google Scholar

• 24

  HeR.McAuleyJ. (2016b). “Vbpr: visual bayesian personalized ranking from implicit feedback,” in Proceedings of the AAAI, Minneapolis, MN, USA, June 2-7, 2019,144–150.

  • Google Scholar

• 25

  HidasiB.KaratzoglouA. (2018). “Recurrent neural networks with top-k gains for session-based recommendations,” in Proceedings of the CIKM, Minneapolis, MN, USA, June 2-7, 2019, 843–852.

  • Google Scholar

• 26

  HuG.ZhangY.YangQ. (2018). “Conet: collaborative cross networks for cross-domain recommendation,” in Proceedings of the CIKM, Torino, Italy, October 2018, 667–676.

  • Google Scholar

• 27

  HuY.KorenY.VolinskyC. (2008). “Collaborative filtering for implicit feedback datasets,” in Proceedings of ICDM, Pisa, Italy, December 15–19, 2008, 263–272.

  • Google Scholar

• 28

  HuangJ.ZhaoW. X.DouH.WenJ.-R.ChangE. Y. (2018). “Improving sequential recommendation with knowledge-enhanced memory networks,” in Proceedings of the SIGIR, Ann Arbor, MI, July 2018, 505–514.

  • Google Scholar

• 29

  JiG.HeS.XuL.LiuK.ZhaoJ. (2015). “Knowledge graph embedding via dynamic mapping matrix,” in Proceedings of ACL-IJCNLP, Beijing, China, December 5-8, 2013687–696.

  • Google Scholar

• 30

  JoshiM.ChenD.LiuY.WeldD. S.ZettlemoyerL.LevyO. (2020). Spanbert: improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguistics8, 64–77. 10.1162/tacl_a_00300

  • CrossRef
  • Google Scholar

• 31

  KuchaievO.GinsburgB. (2017). Training deep autoencoders for collaborative filtering. preprint: arXiv:1708.01715.

  • Google Scholar

• 32

  LeeH.ImJ.JangS.ChoH.ChungS. (2019). “Melu: meta-learned user preference estimator for cold-start recommendation,” in Proceedings of the SIGKDD. Anchorage, AK, August 2019, 1073–1082.

  • Google Scholar

• 33

  LiJ.JingM.LuK.ZhuL.YangY.HuangZ. (2019). From zero-shot learning to cold-start recommendation, Aaai33, 4189–4196. 10.1609/aaai.v33i01.33014189

  • CrossRef
  • Google Scholar

• 34

  LiangD.ZhanM.EllisD. P. (2015). “Content-aware collaborative music recommendation using pre-trained neural networks,” in Proceedings of the ISMIR, Malaga, Spain, October 26–30, 295–301.

  • Google Scholar

• 35

  LinY.LiuZ.SunM.LiuY.ZhuX. (2015). “Learning entity and relation embeddings for knowledge graph completion,” in Proceedings of the AAAI, Austin, Texas, USA, January 25–30 (Austin, Texas, USA: AAAI Press), 2181–2187.

  • Google Scholar

• 36

  LiuW.ZhouP.ZhaoZ.WangZ.JuQ.DengH.et al (2020). K-bert: enabling language representation with knowledge graph, Aaai34, 2901–2908. 10.1609/aaai.v34i03.5681

  • CrossRef
  • Google Scholar

• 37

  LiuY.OttM.GoyalN.DuJ.JoshiM.ChenD.et al (2019). Roberta: a robustly optimized bert pre-training approach. Preprint: arXiv:1907.11692.

  • Google Scholar

• 38

  MaH.ZhouD.LiuC.LyuM. R.KingI. (2011). “Recommender systems with social regularization,” in Proceedings of the WSDM, Hong Kong, China, February 2011, 287–296.

  • Google Scholar

• 39

  ManotumruksaJ.MacdonaldC.OunisI. (2016). “Regularising factorised models for venue recommendation using friends and their comments,” in Proceedings of the CIKM, Indianapolis, IN, October 2016, 1981–1984.

  • Google Scholar

• 40

  McCarleyJ. S. (2019). Pruning a bert-based question answering model. Preprint: arXiv:1910.06360.

  • Google Scholar

• 41

  McPhersonM.Smith-LovinL.CookJ. M. (2001). Birds of a feather: homophily in social networks. Annu. Rev. Soc.27, 415–444. 10.1146/annurev.soc.27.1.415

  • CrossRef
  • Google Scholar

• 42

  MikolovT.SutskeverI.ChenK.CorradoG. S.DeanJ. (2013). “Distributed representations of words and phrases and their compositionality,” in Proceedings of the NIPS, Red Hook, NY, December 5-8, 2013, 3111–3119.

  • Google Scholar

• 43

  MudrakartaP. K.SandlerM.ZhmoginovA.HowardA. (2018). “K for the price of 1: parameter-efficient multi-task and transfer learning,” in Proceedings of the ICLR, New Orleans, LA, USA, May 6-9. 10.18653/v1/p18-1176

  • CrossRef
  • Google Scholar

• 44

  NguyenH. T. H.WistubaM.GrabockaJ.DrumondL. R.Schmidt-ThiemeL. (2017). “Personalized deep learning for tag recommendation,” in Proceedings of the PAKDD, Jeju, South Korea, May 23–26, (New York, NY: Springer), 186–197. 10.1007/978-3-319-57454-7_15

  • CrossRef
  • Google Scholar

• 45

  NiY.OuD.LiuS.LiX.OuW.ZengA.et al (2018). “Perceive your users in depth: learning universal user representations from multiple e-commerce tasks,” in Proceedings of the SIGKDD, London, United Kingdom, August 2018, 596–605.

  • Google Scholar

• 46

  QinC.ZhuH.ZhuangF.GuoQ.ZhangQ.ZhangL.et al (2020). A survey on knowledge graph-based recommender systems. Sci. Sin. Informationis50, 937–956. 10.1360/SSI-2019-0274

  • CrossRef
  • Google Scholar

• 47

  QiuX.TianXiangS.YigeX.YunfanS.NingD.XuanjingH. (2020). Pretrained models for natural language processing: a survey. Sci. China Technol. Sci.63, 1872–1897. 10.1007/s11431-020-1647-3

  • CrossRef
  • Google Scholar

• 48

  RendleS. (2010). “Factorization machines,” in Proceedings of the ICDM, Sydney, NSW, December 13–17, 2010 (New York, NY: IEEE), 995–1000.

  • Google Scholar

• 49

  RendleS.FreudenthalerC.Schmidt-ThiemeL. (2010). “Factorizing personalized Markov chains for next-basket recommendation,” in Proceedings of the WWW, Raleigh North Carolina USA, April 2010, 811–820.

  • Google Scholar

• 50

  RietzlerA.StabingerS.OpitzP.EnglS. (2020). “Adapt or get left behind: domain adaptation through bert language model finetuning for aspect-target sentiment classification,” in Proceedings of the LREC, Marseille, 11–16 May 2020, 4933–4941.

  • Google Scholar

• 51

  SathishV.MehrotraT.DhinwaS.DasB. (2019). Graph embedding based hybrid social recommendation system. Preprint: arXiv:1908.09454.

  • Google Scholar

• 52

  ShiB.WeningerT. (2017). “Proje: embedding projection for knowledge graph completion,” in Proceedings of AAAI, San Francisco, California, USA, February 4–9, 1236–1242.

  • Google Scholar

• 53

  ShiY.LarsonM.HanjalicA. (2010). “List-wise learning to rank with matrix factorization for collaborative filtering,” in Proceedings of the RecSys, Barcelona, Spain, September 2010, 269–272.

  • Google Scholar

• 54

  SongY.ElkahkyA. M.HeX. (2016). “Multi-rate deep learning for temporal recommendation,” in Proceedings of the SIGIR, Pisa, Italy, July 2016, 909–912.

  • Google Scholar

• 55

  SticklandA. C.MurrayI. (2019). “Bert and pals: projected attention layers for efficient adaptation in multi-task learning,” in Proceedings of the ICML, Long Beach, California, USA, June 9–15, 5986–5995.

  • Google Scholar

• 56

  SuX.KhoshgoftaarT. M. (2009). A survey of collaborative filtering techniques. Adv. Artif. Intell.2009, 421425:1–421425:19. 10.1155/2009/421425

  • CrossRef
  • Google Scholar

• 57

  SunF.LiuJ.WuJ.PeiC.LinX.OuW.et al (2019). “Bert4rec: sequential recommendation with bidirectional encoder representations from transformer,” in Proceedings of the CIKM, Beijing, China, November, 2019, 1441–1450.

  • Google Scholar

• 58

  TanJ.WanX.XiaoJ. (2016). “A neural network approach to quote recommendation in writings,” in Proceedings of the CIKM, Indianapolis, IN, USA, October 24–28, 65–74.

  • Google Scholar

• 59

  TangJ.HuX.LiuH. (2013). Social recommendation: a review. Soc. Netw. Anal. Min.3, 1113–1133. 10.1007/s13278-013-0141-9

  • CrossRef
  • Google Scholar

• 60

  TangJ.WangK. (2018). “Personalized top-n sequential recommendation via convolutional sequence embedding,” in Proceedings of the WSDM, Marina Del Rey, CA, USA, February 5–9, 565–573.

  • Google Scholar

• 61

  TangX.WangT.YangH.SongH. (2019). “Akupm: attention-enhanced knowledge-aware user preference model for recommendation,” in Proceedings of the SIGKDD, Anchorage, AK, USA, August 4–8, 1891–1899.

  • Google Scholar

• 62

  TrouillonT.WelblJ.RiedelS.GaussierÉ.BouchardG. (2016). “Complex embeddings for simple link prediction,” in Proceedings of ICML, New York, NY, June 19–June 24, 2016.

  • Google Scholar

• 63

  VaswaniA.ShazeerN.ParmarN.UszkoreitJ.JonesL.GomezA. N.et al (2017). “Attention is all you need,” in Proceedings of the NIPS, Long Beach, CA, USA, December 4–9, 5998–6008.

  • Google Scholar

• 64

  WangH.ZhangF.HouM.XieX.GuoM.LiuQ. (2018a). “Shine: signed heterogeneous information network embedding for sentiment link prediction,” in Proceedings of the WSDM, Marina Del Rey, CA, February 2018, 592–600.

  • Google Scholar

• 65

  WangH.ZhangF.XieX.GuoM. (2018b). “Dkn: deep knowledge-aware network for news recommendation,” in Proceedings of the WWW, Lyon, France, April, 2018, 1835–1844.

  • Google Scholar

• 66

  WangC.ZhuH.ZhuC.QinC.XiongH. (2020a). Setrank: a setwise bayesian approach for collaborative ranking from implicit feedback. Aaai34, 6127–6136. 10.1609/aaai.v34i04.6077

  • CrossRef
  • Google Scholar

• 67

  WangR.TangD.DuanN.WeiZ.HuangX.CaoC.et al (2020b). K-adapter: infusing knowledge into pre-trained models with adapters. Preprint: arXiv:2002.01808.

  • Google Scholar

• 68

  WangZ.ZhangJ.FengJ.ChenZ. (2014). “Knowledge graph embedding by translating on hyperplanes,” in Proceedings of the AAAI, Quebec City, Quebec, Canada, July 27–31, 1112–1119.

  • Google Scholar

• 69

  WenY.GuoL.ChenZ.MaJ. (2018). “Network embedding based recommendation method in social networks,” in Proceedings of the WWW, Lyon, France, April 23–27, 11–12.

  • Google Scholar

• 70

  WuC.-Y.AhmedA.BeutelA.SmolaA. J.JingH. (2017). “Recurrent recommender networks,” in Proceedings of the WSDM, Cambridge, United Kingdom, February, 6–10, 495–503.

  • Google Scholar

• 71

  XingS.LiuF. a.WangQ.ZhaoX.LiT. (2019). A hierarchical attention model for rating prediction by leveraging user and product reviews. Neurocomputing332, 417–427. 10.1016/j.neucom.2018.12.027

  • CrossRef
  • Google Scholar

• 72

  XuZ.ChenC.LukasiewiczT.MiaoY.MengX. (2016). “Tag-aware personalized recommendation using a deep-semantic similarity model with negative sampling,” in Proceedings of the CIKM, Indianapolis, IN, USA, October 24–28, 1921–1924.

  • Google Scholar

• 73

  YangJ.XuJ.TongJ.GaoS.GuoJ.WenJ. (2019). Pretraining of context-aware item representation for next basket recommendation. Preprint: arXiv:1904.12604.

  • Google Scholar

• 74

  YuF.KoltunV. (2016). “Multi-scale context aggregation by dilated convolutions,” in Proceedings of ICLRSan Juan, Puerto Rico, May 2–4, 2016.

  • Google Scholar

• 75

  YuF.LiuQ.WuS.WangL.TanT. (2016). “A dynamic recurrent model for next basket recommendation,” in Proceedings of the SIGIR, Pisa, Italy, July 2016, 729–732.

  • Google Scholar

• 76

  YuJ.GaoM.LiJ.YinH.LiuH. (2018). “Adaptive implicit friends identification over heterogeneous network for social recommendation,” in Proceedings of the CIKM, Torino, Italy, October 2018, 357–366.

  • Google Scholar

• 77

  YuanF.HeX.KaratzoglouA.ZhangL. (2020). “Parameter-efficient transfer from sequential behaviors for user modeling and recommendation,” in Proceedings of the SIGIR, Virtual Event, China, July 25-30Virtual Event, China, July 25–30, 1469–1478.

  • Google Scholar

• 78

  ZhangF.YuanN. J.LianD.XieX.MaW.-Y. (2016). “Collaborative knowledge base embedding for recommender systems,” in Proceedings of the SIGKDD, San Francisco, CA, August 2016, 353–362.

  • Google Scholar

• 79

  ZhangM.HuB.ShiC.WuB.WangB. (2018). “Matrix factorization meets social network embedding for rating prediction,” in Proceedings of the APWeb-WAIM, Macau, China, July 23–25 (New York, NY: Springer), 121–129. 10.1007/978-3-319-96890-2_10

  • CrossRef
  • Google Scholar

• 80

  ZhangY.AiQ.ChenX.CroftW. B. (2017). “Joint representation learning for top-n recommendation with heterogeneous information sources,” in Proceedings of the CIKM, Singapore, November 2017, 1449–1458.

  • Google Scholar

• 81

  ZhangZ.HanX.LiuZ.JiangX.SunM.LiuQ. (2019). “Ernie: enhanced language representation with informative entities,” in Proceedings of the ACL, Florence, Italy, July 28–August 2, 1441–1451.

  • Google Scholar

• 82

  ZhaoX.ZhangL.XiaL.DingZ.YinD.TangJ. (2017). Deep reinforcement learning for list-wise recommendations. Preprint: arXiv:1801.00209.

  • Google Scholar

• 83

  ZhengL.NorooziV.YuP. S. (2017). “Joint deep modeling of users and items using reviews for recommendation,” in Proceedings of the WSDM, Cambridge, United Kingdom, February 2017, 425–434.

  • Google Scholar

• 84

  ZhengY.TangB.DingW.ZhouH. (2016). A neural autoregressive approach to collaborative filtering. Preprint: arXiv:1605.09477.

  • Google Scholar