The number of publications regarding technology research and development is growing exponentially every year. This makes it increasingly difficult to perform manual literature search for the assessment of emerging technologies. At the same time, automatic text processing techniques that can process and evaluate large amounts of unstructured (text) data are becoming increasingly powerful and can take over parts of the research tasks. However, they are prone to errors and need manual curation. This inspired a departmental research assignment by the German Centre for Rail Traffic Research (DZSF) for a semi-automated technology scouting system (TSS) for the application domain of railways [3]. The project has been awarded to d-fine GmbH and a running prototype can be demonstrated. For a semi-automated TSS, voluntary user participation is an essential component. Without active participation, the primary goal and the added value of the system cannot be achieved to its fullest extent. The main goal of user participation is to review, extend, correct and evaluate the automatically extracted knowledge, thus creating a more accurate and complete knowledge base. At the same time, manual corrections are intended to improve automated information extraction and can enhance the training data by retraining to improve the extraction models. The retraining can run as online or batch learning. User involvement might also generate useful data for the calculation of relevance, prominence, and stage of development for certain technologies. One of the work packages in this project was to explore system designs that motivate voluntary and constructive user participation. The Forschungs-Informations-System (FIS, English: research information system) for mobility and transport has a similar purpose as the developed TSS for railway research [1]. FIS was conceived by the Federal Ministry for Digital and Transport (BMDV) in 2000 and has been developed, maintained, and curated since then. The contents of FIS are primarily created manually by authors, who are motivated by organizational commitment, without automatic text processing. In contrast, the semi-automated TSS is intended to motivate voluntary users to curate and extend the automatically produced content. The TSS is similar in this respect to portals like Wikipedia, on which the contributors are motivated by, e.g., their status within a community, reciprocation desire, and similar [4]. And finally, the TSS is also intended to create training data for text algorithms by volunteers. The forms in which users can participate are diverse. They can contribute to the system in a low-threshold manner by evaluating, commenting on, and ensuring the system's quality. More extensive intervention is enabled by the options to modify, add or delete content. Users can modify the knowledge base or the automatically generated texts accordingly. In addition, simple textual feedback, e.g. in the form of comments, is also possible. For all activities, evidentiality and objectivity are essential while destructive behavior like wiki-hounding is undesired. The participation has, thus, to be embedded in a framework of rules
and defined processes so that it is clear which user has which permissions, what quality assurance steps are required to publish user input, and how to proceed in the event of conflicting assessments or actions to build consensus.
Role management should be as transparent and easy to understand as possible to keep the barriers to participation in the system low. An example would be the distinction between viewers, editors, and moderators. Anonymity lowers the barriers to entry to the system and thus leads to higher participation. On the other hand, various studies have shown that the motivation and effectiveness of crowdsourcing teams can be positively influenced when anonymity is reduced (e.g.,[2],[5]). In conceptual development, it can also be useful to consider models that reduce anonymity only after the user has reached a certain level of participation, to achieve a high level of participation.
Potential users were consulted in workshops to determine their requirements for a TSS. For this purpose, design thinking principles were applied, e.g., to develop user stories. The workshops also involved digital surveys and open brainstorming in small groups with experts from industry, academia, and federal agencies.