Artificial Neural Networks (ANNs) were used as prediction models to explore design solutions for the atrium design of a school building. To this end, a solution space of 165 design variants was generated via parametric modeling. This paper details the process of extracting and selecting the input features required for ANN training in order to predict the DA and sDA metric. The feature selection undertaken in this study mainly consisted of two steps: Firstly, a computationally less extensive machine learning model was used to rank the input features according to their relevance in predicting daylight levels. Secondly, ANNs were trained applying sequential forward selection. The proposed method is investigated in terms of achievable improvements to prediction accuracy, reduceable training time and the feasibility of the method.
In the field of daylight design, ANNs have been used as prediction models for luminous efficacy
Input features, also known as predictor variables, make up the training data which is passed to
ANN models in order to facilitate supervised, unsupervised and reinforcement type learning.
Input feature selection refers to the reduction of the dimensionality of training data by selecting
a subset of input features. An objective function is commonly used as selection criterium to
minimize the predictive error and thereby identify an optimal or suboptimal subset of input
features. Such input feature selection methods have been shown to improve prediction accuracy
and identify the minimum number of features needed
As exhaustive search methods are computationally expensive, two other common methods of
input feature selection are Sequential Forward Selection (SFS), and Sequential Backward
Selection (SBS). SFS is a bottom-up approach that starts with an empty set of features to which
features are iteratively added
The different steps to developing, optimizing and validating ANN models are illustrated in Figure 1. The ANNs were employed to predict daylight for a design solution space. As such, performance data was collected from selected samples of the design solution space. The performance data was extracted from daylight simulations and recorded features describing the design changes and the corresponding daylight results. While a majority of data was used for training the ANN models, a part of it was retained for validation of the models. Once a training set had been generated, bagged decision trees, a machine learning technique, were used to determine a sequence for the forward-sequential selection of features. Thus, the ANN models were first trained with the input features that showed the highest impact as predictors. The training data set was again subdivided into a training subset, a validation subset and a test subset at the ratio of 65:25:15. The validation subset was used for early stopping to avoid overfitting and the test subset was used to estimate prediction accuracies on new cases. Both subsets were also used to assist in the optimization of the network architecture. Each network architecture was trained ten times with the initial weight settings and distribution of samples across the subsets varying in each training run. The network architecture with the lowest error across ten training runs was used for predictions, and the output of all ten networks was averaged to improve generalization.
The predictions were validated against the simulations withheld from training. In the case of insufficient accuracy, another training feature was added to the training set and the optimization of network architecture, and validation was repeated. There are several options for terminating the training cycle: once a desired threshold of accuracy is reached, once the accuracy converges for a number of training cycles, once all input features have been tested, or a combination of the three. In this study we tested all input features to allow for a more wholistic evaluation of the method. However, if the aim is not to find an optimal subset of features but to identify the minimum number of input features required to reduce the training time, the first two options of termination better serve the purpose.
The exploration of design solutions was done for the central atrium of a school-building in
Hamburg (Figure 2). As a first design variable, the atrium base dimension was reduced from
225m2 to 56,25m2, thereby splaying the atrium well walls in obtuse angles between 90 and 79°.
With a second design variable, the atrium well was slanted, changing the orientation between
north and south. 9 possible solutions were specified for the slant of the atrium well, modifying
its splay angles between 58 and 104°. This resulted in a 9 by 6 matrix of 54 possible design
variants for the atrium well geometry. As the third design variable, 3 possibilities were specified
for the window-to-wall ratios (WWR) across floor levels of the 6-storey building. The WWR
was reduced from the ground to higher floor levels in order to increase the reflected light and
daylight levels on the ground floor
From the solution space of 162 variants, a reduced set of 36 variants were selected to provide
training data and another 21 variants were selected for validation of ANN accuracy. The
architectural models were built in Grasshopper, and daylight simulation on the selected variants
were run in Diva – a radiance-based and validated software
In order to rank the input feature categories according to their impact as predictors, the training data set was passed to machine learning (ML) models for fitness approximation. Although ANNs can be trained to identify significant features, this was not done due to their increased computation time. Among the tested ML techniques were linear regression models, fine, medium and coarse trees, boosted tree ensembles, liner, quadratic, cubic, fine Gaussian support vector machines and Gaussian Process Regression models. Bagged decision trees showed the lowest root mean squared error (RMSE) and superior performance compared to the other models. Additionally, the computation time was low, with approximation taking around four minutes. Hence, they were selected to identify a feature selection sequence.
From the data set that was passed to the bagged trees model, every input feature category was individually removed and the corresponding variance in RMSE was measured. The features were then ranked according to the variance they inflicted on the error, with those features causing the largest variance ranked highest. The RMSE during approximation on the training data set and resulting sequence of input feature categories is given in Table 1.
The ANN models were trained starting from a data set with one input feature category. Once trained, the ANN model was used to predict the DA metric for the 21 retained variants and predictions were compared to the simulated DA. Two measures were used: the mean absolute error (MAE) and the root mean square error (RMSE). The MAE, which gives the absolute difference between the simulated and predicted values, was chosen for its ease of interpretation. The RMSE was selected as it weighs larger errors more heavily and is a commonly used measure of accuracy. The RMSE was observed for each added input feature and its minimization was used as the objective function to optimise the selection. As long as the RMSE reached a new low, one additional input feature category was added to the training data set. If the RMSE worsened, the last added feature was removed before adding the next feature in the sequence. In this way, all input features were added to the training data set at some point in a forward-sequential manner. The results discuss the observed errors in accuracy for every added feature.
Back-propagation ANN models were employed in conjunction with the Lavenberg-Marquardt algorithm. The training parameters were set to an initial mu of 1, a mu decrease factor of .8 and mu increase factor of 1.5. The training was run for 200 epochs, during which the connection strengths between neurons were adjusted to minimise the mean squared error (MSE) on the training data set. The training data set was again divided into a subset of training, validation and test data in order to ensure robustness of the trained networks through cross-validation and early stopping. The maximum number of validation failures was set to 6. Ten network architectures with 38 to 40 neurons in the hidden layer were trained and tested with said settings and the network architecture with the lowest MSE on the training, validation and test subset was used for predictions. This was done to ensure that a network optimised architecture was used to evaluate the input features.
The MSE on the training data set (36/162 simulations with 150.706 sensor point data samples) and the RMSE and MAE on the validation set (21/162 sim with 87.850 sensor point data samples) were recorded at each training stage of the sequential search. Figure 4 shows the MSE during feature selection (for the sequence refers back to Table 1). Figure 5 shows the MAE obtained on the validation set and Figure 6 the RMSE. The minimum achieved error has been highlighted in red. The sequence at which an added input feature category was again removed from the training data has been highlighted in yellow.
0.004 0.0035 0.003 0.0025 SE0.002 0M.0015 0.001 0.0005 0 1 2 3 10 11 12 4 5 6 7 8 9 Sequence/ number of tested input features
. . . . . MSE with all input features 1.5 1.4 1.3 ) 1.2 (DA1.1 E 1 AM0.9 0.8 0.7 0.6 2.4 2.2
2 SE1.8 RM1.6 1.4 1.2 1 1 2 3 10 11 12 4
5 6 7 8 9 Sequence/ number of tested input features
. . . . . MAE with all input features The MSE dropped below 0.01 after including six input features (which include distance and direction to atrium closest point, distance to façade, distance and direction to atrium center point, WWRs, location of sensor points inside or outside atrium, and atrium well spay angles). The MAE and RMSE remained in the similar range after the 7th input feature category, the sensor point identifiers, were added as training data. The lowest MSE was reached after adding the 8th input feature describing glazing areas on the simulated floor level. The MAE and RMSE however increased, resulting in the feature being removed from the feature set. Consecutively, the MAE and RMSE reached their minimum at .78 MAE and 1.16 RMSE. In comparison, the ANNs trained with all input features showed marginally higher errors of .79 MAE and 1.17 RMSE. Although this shows that the input feature selection could not result in a significant improvement of accuracies, it highlights that not all input features are needed and overall training time can be reduced.
In fact, training time of the ‘optimal’ feature set with 8 added and one removed input feature category (19 individual features) including network optimization took 04:33 (hh:mm), whereas the training and optimization of the model that included all input features was 05:55. Taking a more minimal approach, the training with 7 added feature categories (18 individual features) was 03:01. Predictions on the validation set were made in less than 1 second. Inadvertently, due to nature of the selection process, it lacks in feasibility compared to a trial and error approach validating an empirically selected feature set. The above outlined time savings can therefore only truly be achieved if: a) the selected input feature subset is extracted and trained for a larger data set, b) the feature subset holds validity for a new or similar design scenario, or c) the same input features would be selected with a smaller and thereby less computationally expensive network architecture of fewer hidden neurons. This however would still need validation. The training times as measured here were taken on a 2.6 GHz Intel Core i9 processor.
The prediction accuracy converged around .8 DA MAE, meaning that this was, on average, the absolute difference between the simulated and predicted DA. The simulated DA range from 0 to 88%, with 1% referring to 1% of occupied hours in a year. A difference of .8 DA can therefore hardly be interpreted and constitutes a negligible error. The RMSE, which converged around 1.2%, supports this finding and shows a high accuracy of ANNs in predicting the DA metric.
In order to achieve above-mentioned accuracies, data was extracted from a total of 57/162 simulations for training and validation. As daylight simulations took approximately 3 hours per design variant, a total of 315 hours of simulations were bypassed through ANN predictions. Given that the process relied on a small sample size of 13% for validation, 16,4% when including the validation subset within the training data, the results may vary. However, increasing the validation set within the ANN-integrated workflow would require a larger number of simulations to be run, thereby reducing the achievable time-savings.
Overall, the following conclusions for ANN-based daylight predictions can be drawn: a) On average, prediction accuracies of around .8 DA mean absolute error (MAE) were achieved. These accuracies could be maintained as the number of input features increased. b) The MSE on the training data set did not directly correlate with the MAE and RMSE on the validation data set, as the prediction accuracies could improve (lower MAE and RMSE) even though the ability of the network to fit the data decreased (higher MSE). d) Having superfluous input features did not significantly lower accuracies (e.g. calculation grid size and atrium dimension at WP height). It did however increase training time. On the other hand, too few input features (e.g. before sequence 4), though requiring less training time, compromised prediction accuracies. e) Using the proposed method, accuracies barely improved compared to the empirically selected full set of input features. However, the training time for the ANN models could be reduced by around 50% without significantly compromising the accuracies.
The paper proposes a viable method for selecting input features useful for predicting daylight in buildings. Additionally, the study investigated in how far improvements could be achieved. Although, in terms of accuracy, those were marginal in this study, the selection method may prove more valuable in larger and more complex solution spaces with a larger number of variables. A downside of the proposed method is that it remains computationally expensive, as it requires multiple training runs with already computationally demanding ANN models. It would therefore be useful to evaluate the results of feature selection using smaller and more feasible network architectures, or completely rely on computationally less expensive ML models for feature selection. A comparison to alternative feature selection methods is recommended, as well as the application of the proposed method on more complex solution spaces. Janjai, S. and Plaon, P. (2011) ‘Estimation of sky luminance in the tropics using artificial neural networks: Modeling and performance comparison with the CIE model’, Applied Energy. Elsevier Ltd, 88(3), pp. 840–847. doi: 10.1016/j.apenergy.2010.09.004. Pattanasethanon, S., Lertsatitthanakorn, C. and Atthajariyakul, S. (2008) ‘An accuracy assessment of an empirical sine model , a novel sine model and an artificial neural network model for forecasting illuminance / irradiance on horizontal plane of all sky types at Mahasarakham , Thailand’, 49, pp. 1999–2005. doi: 10.1016/j.enconman.2008.02.014.
Pudil, P., Novovi, J. and Kittler, J. (1994) ‘Floating search methods in feature selection’, Pattern Recognition