When analyzing processes, organizations increasingly resort to process mining (PM) techniques and tools [1]. Several methodologies and case studies are available to guide and inspire the planning and implementation of such projects. [2]. A similarity of them is that they include an initial phase in which questions should be formulated to guide subsequent project phases, like data extraction, preparation, or analysis [3]. For example, a question such as "What are the most common paths taken by cases in the process?" indicates an interest in the process flow. It requires the analyst to compute variants based on the available process data, identify the most frequent ones, and describe their execution. Instead the question "Are there any interesting patterns in the process?" is much broader. Without a keyword like "variant" or "path", it is not restricted to process flow patterns and analysts might also investigate other perspectives, such as time or resources. Additionally, "interesting" might not be equivalent to "most common", as it could refer to edge cases that are less relevant in terms of frequency. While someone familiar with PM techniques can likely envision the analysis for the first question, the second question requires more exploration and may depend on the available data and the analyst's experience. A less experienced project stakeholder interested in control-flow patterns might be disappointed by the outcome if they formulate a question similar to the second example.
Arguably, designing questions is an essential step in setting up PM initiatives, as they elicit requirements and direct the analysis. Research confirms that the execution of this step is a relevant success factor for projects [4]. However, formulating questions is not trivial and analysts lack support from tools and methods [5]. An interview study conducted with 40 analysts confirmed that question formulation is a significant challenge in PM projects, often arising when analysts work with questions that are unclear, overly specific, or too broad (as in the second example above) [6].
In this work, we address this problem and introduce the Process Mining Question Forge (PMQF), a tool that implements guidance for the crucial task of question design in PM projects. We developed PMQF based on findings from [5], that highlight that (experienced) analysts might rely on domain knowledge or analysis templates when confronted with a project that starts without clear questions. In practice, especially less experienced analysts lack this knowledge and templates are not always available. Therefore PMQF leverages categorized example questions and a classification schema to help users in designing their own questions. PMQF can be set up with any custom set of questions and a respective categorization schema. On top of these resources, it guides users to design their questions in a structured way.
PMQF has been developed as a web application in Python using Flask1 . When it is set up, it expects a categorized set of analysis questions and the corresponding classification schema as input. The source code of PMQF can be found at https://github.com/promise-ics-hsg/ demoApplication-PMQF, and a deployed version can be accessed via http://130.82.168.60:5000/. In the deployed version, PMQF runs on an exemplary set of 405 categorized analysis questions that we gathered from diverse sources, such as the BPI Challenges2 or Case Studies, and a classification schema that classifies questions across six dimensions. We provide a video demonstrating how the tool can be used: https://drive.switch.ch/index.php/s/Y9cW0Nk3DEmdVOR. PMQF supports users in (i) retrieving an overview of questions and their classification, (ii) designing new questions, and (iii) clarifying and refining existing questions.
PMQF features an advanced keyword search that allows for the efficient location of relevant analysis questions. Users can enter keywords related to their areas of interest and the tool returns a list of questions that match this search criteria. To this end, we integrated the computation of synonyms based on wordnet3 (using the nltk library4 ). The keyword search is implemented for project teams, learners, or teachers who are interested in a specific concept of PM and aim to retrieve an overview of what kind of questions they could ask in this regard.
The question design feature supports users in formulating analysis questions through three phases. It is particularly suited for less experienced analysts or project teams without extensive expertise who will benefit from guidance on how to iteratively identify areas of interest. Category Selection: In the first phase, users are directed to select categories based on the dimensions of the classification schema (Fig. 1a). For each dimension, we suggest a guiding question that helps them to choose categories that are aligned with their project goals. Definitions for all categories are displayed when hovering over the buttons. Question Filtering: After selecting categories, PMQF filters the available questions accordingly and displays the resulting set. Users are asked to review the questions and select and save those they identify as relevant for further investigation. Question Customization: In the last phase, users conduct their final review by discarding or reformulating questions to fit their domain-specific terminology (Fig. 1b). We assume that the reformulation maintains the original question categorization. Additionally, PMQF generates a heatmap to visualize the range of selected questions across the classification schema.
After customizing the questions, users can either go back to the category selection (e.g., when they identified the need to cover further categories) or end the question design by exporting the identified and reformulated set of questions.
The question refinement feature supports users in formulating concrete and understandable questions based on broad ideas. Users begin by entering their initial questions into an input form. After saving, their input appears on a second screen for reflection, where they are prompted to categorize the questions according to the existing categorization schema. This helps users refine their ideas to fit the categories and formulate them as direct questions.
We find the question refinement especially beneficial for project teams, allowing discussions and consensus on question formulation and categorization. Refined questions can be exported and PMQF stores a copy of the same export, enabling administrators to review and potentially add novel questions to the set of exemplary questions.
The three core features of PMQF run without any known errors. Additionally, we evaluated the tool in two sessions with (1) a leading international commercial vehicle manufacturer from Germany with initial PM experience through the analysis of one of their core manufacturing processes, and (2) a public sector organization with no prior PM experience, exploring the value of integrating it into their new BPM initiative. During the sessions, two representatives from each organization used the tool, aiming to design new analysis questions for their ongoing or planned PM projects. In both cases, the users were able to navigate the tool and use the features as expected. As part of the evaluation, the participants filled out the Technology Acceptance Model (TAM) [7]. Results are provided in Tab. 1. On average, the four participants evaluated the usefulness with 2.46 and the perceived ease of use with 2.33 on a 7-point Likert scale. However, they also pointed out that the usefulness largely depends on the quality and scope of the provided set of questions and the classification schema. For the evaluation, we used the one that is also available in the deployed version of PMQF.
Both organizations were able to successfully derive a set of relevant analysis questions for their projects which they planned to use further. Qualitative feedback addressed smaller aspects such as the use of colors, options to store reformulations at once, and saving the selection of categories for better traceability of the results. The feedback is already implemented in the current version of PMQF. Additionally, participants suggested enhancing the tool by adding more questions for specific domains and including guidelines for analyzing the questions.
Average responses to the TAM on a 7-point Likert scale ranging from extremely likely (1), to extremely unlikely (7).
Based on our knowledge, PMQF is the first tool providing practical support for analysis question design and refinement in PM. As such, it contributes to research by suggesting a standardization for these two tasks and thus enables higher consistency and comprehensiveness across projects. We believe that in the future, this can lead to more comparable and reliable project outcomes. However, in its current version, PMQF is sensitive to the existence of a set of categorized example questions and the respective classification schema. The deployed version we provide is optimized for one specific schema and runs on top of a collection of 405 analysis questions.
Local installations can be adapted to custom input during setup.
In the future, we aim to provide a stable, universal categorization schema applicable for all PM domains. Additionally, PMQF could be further advanced in several directions:
1. Integration of large language models (LLMs) to enhance the question design and question refinement features. 2. Integration of analysis guidance by linking questions to relevant analysis techniques and provide hints for how to answer them (e.g., supported by GenAI [8] or integrated in PM tools). 3. Requesting community feedback in the form of ratings for questions or indications on whether questions were answerable and valuable to project teams in practice. Such information would help identify what constitutes a good analysis question and which types of questions are most frequently addressed in projects.
By integrating insights from the community and refining our approach with advanced technological capabilities, PMQF may be able to offer even more sophisticated and tailored support functions in the future.
This work was funded by the Swiss National Science Foundation as part of the ProMiSE project under Grant No.: 200021_197032. I express my gratitude to my colleagues and the participants of the evaluation for taking the time to test the tool and providing their ideas and feedback.