Refine
Has Fulltext
- yes (6)
Document Type
- Master's Thesis (6)
Language
- English (6)
Is part of the Bibliography
- yes (6)
Keywords
- Deep Learning (6) (remove)
This thesis focuses on the automatic classification of research methods used in scientific articles in the domain of Information Systems, and studies the effectiveness of a state-of-the-art long-document Transformer technique, Longformer, on the multi-label classification task.
In recent years, the task of automatically extracting knowledge from academic articles has become more and more popular. However, as far as the author knows, due to the limitations of the max input sequence length of Transformer models, there has not yet been a comprehensive study on the classification of research methods using Transformer models, which have made extraordinary achievements in various Natural Language Processing (NLP) fields. Therefore, this thesis establishes an artifact that uses a modified Transformer model, Lomgformer, which can proceed with long-sequence inputs, to discuss its effectiveness and possible limitations. Additionally, this thesis also discusses and evaluates the performance of other benchmark models, such as the traditional Transformer - BERT and RoBERTa, and a non-transfer-learning model based on the architecture of convolutional neural network (CNN), which has been proven to have good performance by previous researchers.
As a result, this thesis proves that the Longformer-based artifact can effectively improve the classification performance for the scientific articles, and surpasses all the other models, not only the traditional Transformer models but also the models presented in the literature from the previous researchers.
Hypothesis Extraction from Academic Papers Using Neural Networks for Ontology Theory Learning
(2019)
In this study, we investigated a new use case of deep learning. We applied deep learning to extract causes and effects from the hypotheses of the scientific papers. The research presents a variety of RNN models, including RNN models with CRF layer for labelling the sequences. We used such models as Bi-LSTM, LSTM, SimpleRNN and GRU. The experiments were conducted with GloVe vector representation and character level vector representation of words. Moreover, along with RNN models, we evaluated various hyperparameters and model setups to achieve the highest performance scores. In the end, we obtained promising results and shared our thoughts on the future prospects of the following studies.
This thesis examines the effectiveness of the latest Transfer Learning techniques for Natural Language Processing applied to the classification of research methods used in scientific journals in the domain of Information Systems. The task of automated knowledge extraction from academic articles has seen ongoing progress in recent years. However, the combination of transfer and Deep Learning in order to assign research methods to scientific papers has not been addressed in the literature yet. The main contribution of this thesis is, therefore, an artifact that applies cutting-edge Transfer Learning techniques to a Deep Learning model by conducting several experiments and comparing their effectiveness. The prototype considers various ways of fine-tuning that are crucial to retain the knowledge transferred from pretrained models and avoid catastrophic forgetting. Additionally, this work discusses the literature with regard to the task-specific Theory Ontology Learning and the method-specific state of the art in Transfer Learning for Natural Language Processing. As a result, the artifact surpassed the performance of previously developed models for research method extraction, presented in the literature, without applying any custom feature engineering and only using around a thousand of labeled observations.
This research paper explores the application of the Universal Language Model Fine-tuning (ULMFiT) technique, a novel deep transfer learning approach in the Natural Language Processing (NLP) field, to the financial statements fraud detection task. Additionally, the artifact investigated a simpler model, represented by a one-dimensional Convolutional Neural Network (CNN). Both methods have been assessed with respect to the training time and predefined evaluation metrics. Overall, ULMFiT turned out to be considerably more computationally expensive to train and achieved an accuracy of 77% with F-measure of 13%, if used with a decision boundary of 0.5, and accuracy of 59% with F-measure of 42%, if calculated with a threshold of 0.2. In contrast, CNN model trained significantly faster and obtained the following metrics: accuracy of 82% with F-measure of 42% for threshold 0.5, and accuracy of 83% with F-measure of 58% for threshold 0.2. As a result, ULMFiT has been outperformed by one-dimensional CNN on all examined metrics. The results are reported by using two different values for decision boundary due to the precision-recall trade-off, depending on the use case. In addition, this thesis investigated the impact of data preprocessing. The findings have shown that removing all numbers and special symbols with supplementary text truncation, limiting the sequence length, had a positive effect on both models, mentioned above.
Deep Learning, a topic of broad and current interest, has undergone rapid development in the last decade. The performance of the algorithms is already above human level. However, the area of Natural Language Processing is still a great challenge for the researchers. This study aims at exploring the novel ULMFiT method for text classification by applying it to a dataset of scientific articles with advanced rhetorical categories. This classification task is challenging even for humans and it requires a substantial analysis of the texts when conducted by machine learning algorithms. The objective of this attempt is to achieve text classification with minimal preparation. The ULMFiT method is the first successful effort to apply transfer learning to NLP tasks. Its performance will be evaluated on a task that requires a level of understanding beyond the semantic meaning of the text.
The goal of this research is to investigate the use of deep convolutional neural networks for racing bib number recognition in sport images. Several deep neural network architectures are studied. Three final architectures are trained on three different sets of data: 1) Street View House Numbers (SVHN) Dataset, 2) A private dataset from Flashframe.io from different running events, and 3) A combination of dataset 1 and 2. This thesis investigates the performance that can be obtained on racing bib numbers from a neural network that has been trained on solely images from street house numbers, on a mixture of SVHN and RBN images as well as only on RBN images. The motivation behind this is to see how well this problem can be solved by transfer learning, as labelled images of racing bib numbers are scarce.
The models are tested on the RBNR Dataset (Ben-Ami et al., 2012) and a subset of the private dataset from Flashframe.io. The study shows that the best recognition results were obtained by a model trained on the hybrid dataset of all SVHN images plus an additional 50.000 images. This model outperformed the models that had been trained solely on the SVHN Dataset or the private racing bib number dataset.
The best model resulted in Recall of 0,92, Precision of 0,93 and F-measure of 0,93 on the RBNR Dataset (using the same formulas as previously reported on the RBNR dataset), and 0,97, 0,97 and 0,97 on the private dataset, respectively. The reported recognition results on the RBNR dataset are much higher than previously used methods and proves that neural networks can effectively be used for racing bib number recognition.