Workload Prediction using Apache Spark and R in a Mainframe Operating System
- This master thesis presents a machine learning approach using the distributed data- processing framework Apache Spark and the programming language R. The aim is to predict the time series of the Appl Percentage (ApplPerc) from the workload manager of the z/OS mainframe system using SMF 72.3 and SMF 70.1 records as input. System Management Facility (SMF) data are binary log files that are used to collect system performance data and information about system behaviour. The SMFs show how Apache Spark can be used for the pre-processing. This includes data collection, data extraction, and data selection. The machine learning algorithms are implemented in R. The input SMF datasets will be split into training and test data, and thereby applied to different machine learning and deep learning models such as random forest regression, recurrent neural network, and k-nearest neighbor regression to predict the ApplPerc. The validation of the prediction models will be proved by using cross-validation techniques to evaluate the best applied parameters for each model and therefore to locate the model with the best performance. The process of data analysis is followed by a data-mining methodology called Cross-Industry Standard Process for Data Mining (CRISP-DM), which outlines the steps involved in performing the analysis.