Feature Selection (FS) has been applied to numerous domains, and shown to be effective in increasing the performance of machine learning algorithms. In the semiconductor industry, FS is part of various prediction tasks that aim at avoiding production stops and yield loss. For example, it can be used for: (i) diagnostics, wherein relevant features constitute potential root causes, with their identification being the initial step in a detailed investigation of process defects [3]; (ii) control, as the values of a small set of relevant features can be used to group objects and apply actions per group [1]; (iii) improving prediction performance and interpretability, by enforcing sparsity [4, 7]. Nevertheless, when analyzing manufacturing datasets, one faces two particular challenges, as in other real-world datasets: - High-dimensionality: typically, the number of features p to be evaluated in FS can reach hundreds of thousands and it is much larger than the number of instances n. - Collinearity: some features may be strongly correlated with one another.
For integrating expert feedback, we use the knowledge elicitation framework
proposed by Daee et al. [
RRaannddoommquqeureyry
EExpxp..InInfoformrmataiotinonGGaianin
posteriors are sequentially adapted every time an expert is queried about the
relevance of a feature. In order not to overwhelm the expert, we use smart query
strategies that reduce the number of required user feedback. We implement the
Expected Information Gain (EIG) strategy proposed in Daee et al. [
We evaluated the implementation of our FS scheme with an ASML expert. We used a dataset with 344 features and 100 instances from the logs of ASML machines. The results are shown in Fig. 1, for a simulated expert (Fig. 1, left) and for a knowledgeable domain expert (Fig. 1, right), with the former being less trustworthy than the latter. In these results, “All feedback” refers to the performance of the bayesian regression with s&s priors after all expert feedback is received; we note that the best achieved MSE in Fig. 1 left is higher than the best MSE in Fig. 1 right, but in both cases the EIG strategy reaches the best MSE after ∼ 250 feedbacks. Even with a simulated expert (Fig. 1, left) we see an improvement on the predictive performance with respect to not having an expert in the loop (i.e., when the number of feedback equals 0). This is crucial for the robustness of our application. 3
At present, we still face several challenges in the implementation of the proposed scheme. First, our datasets have typically thousands of features, such that asking feedback from a domain expert for all these features is unfeasible. Second, domain experts are often not sure of the feedback for some particular features, and their input might also be biased. To address these challenges, we are adopting the following steps: – group features that relate to the same effects/machines in a single category and ask the expert to provide feedback on the entire category at once. – visualize the data in an informative way to help the expert make a decision on the relevance of a particular feature category and avoid biases.
How to group features into categories and how to visualize the data in the most informative way remain an open question. In particular, the data visualization is a challenging task since (1) dimensionality reduction methods show artifacts, (2) assessing the quality of a visualization is not straightforward and (3) often only well-known effects and structures emerge in the visualization. One needs to get rid of the latter to discover the underlying patterns.
In this work, we sought to use smart query strategies to reduce the number of
required user feedback. While some state-of-the-art tools allow for user feedback,
none of them are directly applicable to our problem setting. RapidMiner, Weka, and
FEXUM [
The expected users of our framework are domain experts without Machine Learning knowledge but with scientific background e.g., domain experts, field engineers, etc. The experts can use our tool to easily find the relevant features to a selected target variable. The User Interface of the proposed FS scheme is presented in Fig. 2. The expert initializes a workflow by selecting one or more performance metrics to monitor (targets y) on the left panel in Fig. 2. The application, with non-linear embeddings (Fig. 2 middle plot), helps the user understand complex relations and interconnections between multiple features and the target. Particularly, users can select multiple features and observe interactively, via a t-sne embedding, their 2D visualization colored by the values of the target.
In each iteration a user is provided with a feature (left most panel) for which he/she is required to give feedback (relevant feature, irrelevant feature or do not know). As mentioned in the abstract, this feature is selected sequentially via knowledge elicitation with the EIG query strategy. The user interacts with the application to provide feedback on a certain number of features. To help the decision-making process, on Fig. 2 (right), three plots are shown: (i) the R2 value of the linear regression model after each expert feedback. This provides information on the predictive power of the model in each iteration; (ii) the values of the feature that the user is evaluating colored per cluster. This gives information on how the target value relates to the underlying structure of the data; (iii) a list with all the feedback that he/she has already provided. Once the decision has been made, the feedback is given on the left panel, the posterior distribution of the regression is updated and the algorithm suggests a new feature.
In this way, actively querying the expert for feedback results in better user experience as, the users are not expected to examine all the features to find those whose relevance they should assess.