Problem

• existing approaches in NLP still require training from scratch thus utilizing large datasets and days to converge
• two types of transfer
  • inductive
  • transductive (more popular)

Contributions

• Universal Language Model Fine-tuning that reduces the error on a majority of datasets by addressing the issues with inductive transfer learning
• it is not the idea of LM fine-tuning but our lack of knowledge of how to train them effectively has been hindering wider adoption.
• As stated by the paper their contributions are
  • We propose Universal Language Model Fine-tuning (ULMFiT), a method that can be used to achieve CV-like transfer learning for any task for NLP.
  • We propose discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing, novel techniques to retain previous knowledge and avoid catastrophic forgetting during fine-tuning.
  • We significantly outperform the state-of-the-art on six representative text classification datasets, with an error reduction of 18-24% on the majority of datasets.
  • We show that our method enables extremely sample-efficient transfer learning and perform an extensive ablation analysis.
  • We make the pre trained models and our code available to enable wider adoption.

Experiments

• focus on text classification
• use six widely studies datasets
• use 3 common text classification tasks
  • sentiment analysis
  • question classification
  • topic classification
• use the same set of hyperparameters
• compared their results against CoVe a current transfer learning model

Results

• did outperform previous models

How this relates to our paper and work

• they believe that language model fine-tuning will be particularly useful in the following settings compared to existing transfer learning approaches:
  • NLP for non-English languages, where training data for supervised pretraining tasks is scarce
  • new NLP tasks where no state-of-the-art architecture exists;
  • tasks with limited amounts of labeled data (and some amounts of unlabeled data).