Refine
Has Fulltext
- yes (2)
Year of publication
- 2019 (2)
Document Type
- Master's Thesis (2)
Language
- English (2)
Is part of the Bibliography
- yes (2)
Keywords
- ULMFiT (2) (remove)
Institute
This thesis examines the effectiveness of the latest Transfer Learning techniques for Natural Language Processing applied to the classification of research methods used in scientific journals in the domain of Information Systems. The task of automated knowledge extraction from academic articles has seen ongoing progress in recent years. However, the combination of transfer and Deep Learning in order to assign research methods to scientific papers has not been addressed in the literature yet. The main contribution of this thesis is, therefore, an artifact that applies cutting-edge Transfer Learning techniques to a Deep Learning model by conducting several experiments and comparing their effectiveness. The prototype considers various ways of fine-tuning that are crucial to retain the knowledge transferred from pretrained models and avoid catastrophic forgetting. Additionally, this work discusses the literature with regard to the task-specific Theory Ontology Learning and the method-specific state of the art in Transfer Learning for Natural Language Processing. As a result, the artifact surpassed the performance of previously developed models for research method extraction, presented in the literature, without applying any custom feature engineering and only using around a thousand of labeled observations.
This research paper explores the application of the Universal Language Model Fine-tuning (ULMFiT) technique, a novel deep transfer learning approach in the Natural Language Processing (NLP) field, to the financial statements fraud detection task. Additionally, the artifact investigated a simpler model, represented by a one-dimensional Convolutional Neural Network (CNN). Both methods have been assessed with respect to the training time and predefined evaluation metrics. Overall, ULMFiT turned out to be considerably more computationally expensive to train and achieved an accuracy of 77% with F-measure of 13%, if used with a decision boundary of 0.5, and accuracy of 59% with F-measure of 42%, if calculated with a threshold of 0.2. In contrast, CNN model trained significantly faster and obtained the following metrics: accuracy of 82% with F-measure of 42% for threshold 0.5, and accuracy of 83% with F-measure of 58% for threshold 0.2. As a result, ULMFiT has been outperformed by one-dimensional CNN on all examined metrics. The results are reported by using two different values for decision boundary due to the precision-recall trade-off, depending on the use case. In addition, this thesis investigated the impact of data preprocessing. The findings have shown that removing all numbers and special symbols with supplementary text truncation, limiting the sequence length, had a positive effect on both models, mentioned above.