Multi-objective evolutionary feature selection with deep learning applied to air quality spatio-temporal forecasting

Abstract

Objetives - The general objective is to develop efficient and effective feature selection techniques for deep learning through multi-objective evolutionary algorithms and application of the created methods for time series forecasting in different areas of interest. This general objective has been broken down into the following specific objectives: - Develop a comprehensive methodology and implement a multi-criteria decision-making process for the comparison and evaluation of predictive models for time series forecasting. - Study, design and develop a multi-objective evolutionary approach based on spatio-temporal characteristics within the Autonomous Region of Murcia. - Define multi-objective optimization problems for feature selection, with objectives of different nature, both filter and wrapper. - Solve the proposed optimization problems by identifying the best state-of-the-art multi-objective evolutionary algorithms and developing surrogate-assisted approaches to reduce the computational cost of the algorithms. - Identify metrics to quantify the variability between surrogate-assisted approaches and facilitate the establishment of qualitative analysis. - Evaluate, validate and compare the developed feature selection methods with time series data for air quality forecasting in the context of the Autonomous Region of Murcia, as well as in other geographic locations and in other time series forecasting problems for the sake of generalization verification. Methodology - In the course of this thesis, a generic methodology and new metrics have been developed for the comparison of machine learning and deep learning models, thus establishing the best model to use for a given time series problem. A multi-criteria decision-making process has been designed that takes into account the accuracy and robustness of the RMSE, MAE and CC of the models, and gathers these criteria into a single weighted metric called goodness. A new technique based on the spatio-temporal properties of the data has been proposed to infer information from areas for which no data are available. For this purpose, an air quality prediction problem has been formalized as a multi-objective optimization problem. The multi-objective evolutionary algorithms evaluated have been: NSGA-II, MOEA/D and SPEA2. The Pareto fronts resulting from the evolutionary algorithm are the input to build an ensemble learning model. The following learning algorithms have been used to train the ensemble model: RF, LR, SVM, QRNN, KNN and ZeroR. On the other hand, optimization problems with up to four objectives, based on filter, wrapper and hybrid methods, have been formalized to perform feature selection. Thanks to a surrogate model, this method allows the use of deep learning models as a learning algorithm of a wrapper method but without the drawback of the high computational cost involved. The multi-objective evolutionary algorithms NSGA-II, NSGA-III, MOEA/D, SPEA2, IBEA, ε-NSGA-II and ε-MOEA have been studied, and the prediction performance with a surrogate model based on a LSTM neural network. Additionally, a new multi-criteria performance metric, H, is proposed, which allows adjusting the importance of the metrics that form it. The performance of using multiple surrogate models to achieve better generalization capability of predictive models in both regression and classification problems has also been evaluated. A metric has been developed to establish a qualitative analysis of the developed methods. In the approaches described above, the surrogate model always maintains the same information as at the beginning of the method. This does not take into account the underlying information obtained during the course of the evolutionary algorithm. For this reason, two approaches have been proposed to update the surrogate model, one based on incremental learning and the other based on updating the database and building a new surrogate model. Conclusions - The main conclusions drawn from this thesis after the execution of all the experiments are as follows: - The adoption of a complete methodology for the evaluation and comparison of learning algorithms has allowed to obtain unified and adapted results in order to solve any prediction problem with time series. - Recurrent neural networks, such as LSTM and GRU, have been able to capture the complexity of time series and build accurate and reliable predictive models. Among the analyzed machine learning techniques, RF has presented a satisfactory performance when applied to time series forecasting. - A multi-criteria decision-making process has allowed to pool several performance metrics and to establish a more appropriate comparison between different learning algorithms in the context of time series forecasting problems. - For air quality forecasting with time series in an area for which no information is available, the prediction has been approximated with multi-objective evolutionary algorithms using forecasts from other geographically nearby areas. - Surrogate-assisted multi-objective evolutionary algorithms has allowed feature selection in expensive problems such as time series forecasting based on deep learning. - The use of a surrogate-assisted MOEAs with a deep learning algorithm for feature selection has managed to find a satisfactory subset of features in a shorter computational time compared to a conventional wrapper-type feature selection method. - Among all the MOEAs studied, NSGA-II is the one that has obtained the best results in terms of hypervolume, compared to other MOEAs of the state of the art. - Generation-based fixed evolution control approach allows information to be efficiently added to surrogate models within the feature selection process. - Prediction models have been identified in various real contexts (Poland, Murcia, Valencia) that potentially allow forecasting in the near future and that can help institutions to make decisions on environmental issues.