R&D portfolio management is a critical task with which the majority of the large companies are confronted. Despite its wide implementation in companies there are no widely accepted and used methods to perform this task. Each company uses its own mix of various qualitative and quantitative methods to achieve its goal. The objective of this thesis is to explore the adequacy and the design issues to use a prediction market for supporting the R&D portfolio management process. We chose prediction markets to perform this task since their aggregation mechanisms and information discovery process seems to solve most of the current issues of the R&D portfolio management process.
R&D portfolio management is a critical process in large organizations despite the fact that there are no well established methodologies or tools to support it. As prediction markets are growing in importance, we propose to use such tools to support the R&D portfolio management process.
To support the whole process, we will have to solve new design issues, particular to this specific instantiation of a prediction market. The main issues are related to the market's capabilities like the integration of an ontology to support claim formulation; the integration of an IPO mechanism to support the innovation process; and the integration of an automatic market maker to increase the quality of the evaluation. Beside these capabilities, we will also have to increase the traders' incentive, given that they are not familiar with such electronic markets. This includes a specific interface design to occult the financial mechanisms and to allow the combination of group sessions with individual sessions.
This paper will describe the research question and the relevance of our work in the next section. Then, the methodology based on a design science framework will be reviewed in section 3. Next, related work will be exposed in section 4. Section 5 will present the designs and issues of our prototypes. Finally, the last section presents our contribution. 2
The research objective is to explore the adequacy to use a prediction market for supporting the R&D portfolio management process. During this research, we will design and evaluate an IT artefact composed of different prediction markets instantiations.
The R&D project portfolio selection is a periodic activity, which aims at
optimizing the research effort of the company, while enabling it to select a portfolio
which corresponds to its strategic objectives and without exceeding the resources
available. Several studies evaluated the practices in “Fortune 500” companies, finding
that there is neither single method nor a solution applicable to all companies. The
most recent investigations showed that to be effective, portfolio management must
apply a mix of various qualitative and quantitative methods. However, the use of
quantitative methods presents weaknesses, mainly for (1) selecting the right criteria,
(2) collecting the data, (3) and negotiating the portfolio between the different
stakeholders. Many authors proposed different frameworks for selecting R&D
projects portfolio [
Our research assumption is that a specifically designed prediction market could improve the R&D portfolio management process. In our situation, prediction markets are future electronics markets (e-markets) concerning the potential projects of the portfolio. Prediction markets collect information coming from different actors, who trade on the market, and aggregate this information in an automatically negotiated equilibrium price, corresponding to the valuation of the project. All actors directly or indirectly linked to the project, can trade (buy or sell) contracts concerning the projects, based on their own appreciation of the project. The traders are on one hand the leaders and the teams of the project, but also the senior management, people from marketing, finance, as well as from all the other businesses units concerned by R&D. Their narrowly expertise of a particular company activity, like research, but also marketing, sales, customer care or finance will enable them to build their own opinion about the project, under the particular lighting of their activity field. The result of all aggregated appreciations will de facto include a multitude of implicit criteria related to all company activities. Such market mechanism addresses the three weaknesses mentioned above: (1) no more criteria to be explicitly selected, (2) less data to be manually collected, and (3) fewer issues to be explicitly negotiated between actors. These three activities are implicitly replaced by the buy and sell trading of claims concerning the portfolio contents. In addition, the prediction markets are very powerful tools to discover and aggregate the information disseminated between many people. Thus, using prediction markets should not only make all the process more effective, but also increase the quality of the decisions, based on more complete information.
To support the whole process, we designed our IT artefact to support the three invariant steps included in most frameworks. Each step required specific prediction market design as presented in Figure 1. 2.1
The output of this first step is a dynamic ranking of all projects (new and running). The goal is to be able to discover the best projects between all propositions. To reach this goal, we propose to use a prediction market on which contracts are created for each project and put on the market by projects leaders via a simple IPO process. The contracts are then exchanged on a scale from 0 to 100, representing the probability that they achieve their goals. Each contract referring to a specific project proposal, gathering all useful information needed for the comprehension of the project goals. Contrary to the traditional methods, it is not necessary to collect specific indicators allowing a comparison between the projects. The comparison being done in an implicit way by the price equilibrium at a given time. The scale being the same for all projects, it becomes very easy to compare the various projects and to retain those of which the potential is the most promising, represented by a high price on the market. 2.2
The second stage consists in balancing the portfolio so that the mix of projects is the
most effective for the organization. Based on the principle that the sum of the best
projects does not necessary correspond to the best portfolio [
We propose to transfer the projects which were adopted during the first stage on a new prediction market. To reach a balanced portfolio, we propose to create a market joining all projects into a single portfolio. So that all projects are interdependent and the variations of a project's price will have effects on all projects in the portfolio. The balance between the projects will be done in a gradual way according to the price trends of each project. At the end of the process, we will retain the most valued projects. Thus, we obtain the best portfolio according to the available projects. 2.3
The last stage of the process takes the internal and external environment of the organization into account.
In this case, we again propose to use a prediction market to obtain and aggregate the required information. However, we propose that this last stage directly influence the 2nd stage. Consequently, we will use a prediction market tool to study the environment. This prediction market will gather individual contracts related to the interest centres of the organization as well as to the elements of its strategy. The contracts, relating to the evolution of a particular technology or research field, will be used as indicators to support the transactions made during the second stage. We consequently propose to make this third stage implicit. This market will be continuously run alongside the two others, so that information gathered from this market could be used by traders as indicators to support their trades on the two others one. Resulting in an implicit aggregation of strategic and environmental information during the establishment of a balanced portfolio.
In conclusion, we propose to use a new method, based on three prediction markets designs, and gathered in an IT artefact. This IT artefact supports the selection and balance of R&D projects portfolios process. Our approach is based on the intuition that the information needed to manage the portfolios is generally available within the company in a subjective form. On the other hand prediction markets are able to discover and incorporate this information without the need of quantifying these data through a resources intensive process. Consequently, supported by this artefact, managers should be able to make their decisions in a faster way, while being better informed. 3
As a matter of fact, a suitable research paradigm (design science) applied to
information systems has been recently formalized. According to Weber [
We adopted the IS Research Framework suggested by Hevner et al. [
We will now focus essentially on the heart of the framework: Develop/Build and
Justify/Evaluate. Hevner [
We already run three iterations to test our intuitions in different prediction markets
settings applied to the first stage of the R&D Portfolio Management Process [
We will now use these design propositions to iteratively build the final artefact. As presented in the last section, the artefact will be composed of three parts. Each part will consist of a particular prediction market instantiation, specifically designed to support the various portfolio management process stages. 3.1
The first part consists of a winner-takes-all market, supported by a market-maker algorithm and based on the continuous double auction mechanism (CDA). This market was already used and evaluated within small and large scale experiments. 3.2
The second part of the thesis will consist in developing a prediction market composed
of unit portfolios and droved by a double paris mutuel (DPM) algorithm [
This part will be tested with an expert group in the field of mobile payment. The results will then be compared with results collected with traditional MCDM approaches. They will then be completed by qualitative evaluations based on interviews with selected experts.
The third part of our work will be devoted to the environment study. It consists on a market constituted by proportional contracts, droved by a market-maker algorithm and based on the continuous double auction mechanism. It will be instantiated within a SNF project constituted by a population of 150 researchers in the mobile information and communication systems (MICS) field. The experiment will last six months and will be followed by a quantitative evaluation based on the researchers' feedbacks as well as on objective indicators (scientometrics).
Finally, following these three instantiations, we will be able to finalize the IT artefact and evaluate it within research centres. It will be used to support the entire process of R&D project portfolio selection in an academic environment on one hand and in a private R&D centre on the other hand. The IT artefact evaluation will be done by carrying out, in parallel, the process according to a traditional method, and by comparing the obtained results. This will enable us to evaluate the relevance of our research idea. This evaluation will then be completed by a qualitative one, based on interviews with selected people, to obtain information on the advantages and disadvantages of this new approach. This qualitative evaluation will permit us to test the relevance of our approach.
In conclusion, using Hevner's framework we will iteratively run IT artefacts in three different configurations. This will allow us, by successive adaptations, to design an IT artefact supporting the complete R&D portfolio management process. Finally, this IT artefact will be instantiated and evaluated in two different environments in order to test our research assumptions. The relevance of our research, already been presented at Section 2, and the rigor, being presented in the following Section, we still have to present our contributions in the environment as well as in the knowledge base. This will be the subject of the Section 5 The related work relies on three separated domains: R&D portfolios management, prediction markets, technology foresight and environment scanning.
Our research is at the crossroad of three research domains. We will first present the R&D portfolio management literature and raise the most frequent issues, before presenting the prediction markets and their possible contributions. 4.1
Chien [
Cooper [
Liyanage refers to more than 200 quantitative and qualitative methods for selecting
R&D projects in his study [
There are many definitions of prediction markets, idea futures (IF) markets, information markets, virtual stock markets (VSM), securities trading of concepts (STOC) markets. Hanson, one of the inventors of this concept wrote:
"Information markets can be used to elicit a collective estimate of the expected
value or probability of a random variable, reflecting information dispersed across an
entire population of traders. The market prediction is not usually an average or
median of individual opinions, but is a complex summarization reflecting the
gametheoretic interplay of traders as they obtain and leverage information, and as they
react to the actions of others obtaining and leveraging their own information." [
Prediction markets have been used in many different public contexts and used as
case studies in many scientific papers [
Researchers from different disciplines study prediction markets: politics, economies, law, finance, decision science, and computer science.
A considerable amount of recent research in this domain was conducted to evaluate
the potential use and different design possibilities of prediction markets [
No research really addresses the use of prediction markets to support the R&D portfolio management process. Moreover, the literature weakly addresses the following design issues: claim formulation, IPO process and interface design for traders without financial background. 4.3
As conveyed by the widely quoted definition [
Environment scanning is seen as a necessary first step in the strategic decision making process providing managers with the necessary information for crafting strategies that are aligned with the environment where they must be implemented [2932]. We identified four principal environmental domains that are considered of strategic importance for the organization: market, value proposition, actor and issue. 5
To design our prediction market named MarMix we adopted the recommended
buildand-evaluate loop from the design science framework [
To support our design, we reviewed all the open-source prediction markets available at the time of our first experiment. We decided to improve the work of Peter McCluskey on USIFEX1. His prediction market had the advantage of having been developed with a robust object-oriented programming methodology. In addition, 1 http://www.usifex.com/ USIFEX was also the most complete software, implementing the majority of our requirements.
We chose to iteratively test the interfaces and incentives, the market mechanism as well as the market-maker algorithm and finally the claim description and the incentives again in three separate experiments. This allowed us to focus the experiments on specific design issues.
The first run was a small scale market instantiated with the collaborators of the IS department, supporting claims on various technical and environmental topics. The goal of this first attempt was to test our main design as well as our interface. The second run was a large scale market with more than 100 students playing on technological, sociological and economical claims. We tested our market-maker algorithm implementation and various design improvements. Finally, we run our prototype in the MICS community to test the claims formulation, the various incentive mechanisms and the adoption by the researchers.
Our experiments supported with our artefact led us to formulate five propositions
to design a predictions market for R & D portfolio management [
Proposition 1: A prediction market for R&D should integrate a standard framework to support claim formulation.
Using an indicator incorporated in the form of a price makes the evaluations comprehensible by all actors. Consequently, it is advisable to make sure that independently of their activity the traders have the same claims comprehension. Moreover, a structured framework helps defining claims or structuring interviews with concerned peoples.
Proposition 2: A prediction market for R&D should integrate an easy IPO mechanism to support the innovation process.
The prediction market should allow researchers to test their research ideas among all actors, without needing a review process or preliminary validation. This direct access to the market removes internal barriers to innovation without requiring modifying usual selection processes.
Proposition 3: A prediction market for R&D should occult the financial mechanisms to reduce the trader's learning curve and increase his incentive
Observations resulting from our interviews showed that the researchers are by no means familiar with the underlying concepts such as limit orders necessary to play on a prediction market, which results in errors and discourages them from playing on the market. This implies specific usability requirements on the human-computer interface. Proposition 4: A prediction market for R&D should allow the combination of group sessions with individual sessions to increase the incentive of the traders.
Group sessions allow obtaining very quickly an evaluation of the portfolio, generating a specific dynamic. Siemens used this configuration during one hour meetings, but it could also be used in a distance synchronous trading way (e.g. by opening the market for one hour every month). Proposition 5: A prediction market for R&D should integrate an automatic negotiation agent (i.e. market maker to increase the quality of the evaluation).
The market maker makes the market more reactive and fluid, allowing the traders to buy or sell each time new information is available. Thus the evaluation will aggregate more information, compared to a double auction market were the traders must wait for a similar offer to make the deal. 6
Our contribution corresponds to a new and unified R&D portfolio management approach using prediction markets, based on a design science approach.
Our contributions span three dimensions: the IT artefact, the prediction market based R&D portfolio management approach, and a new application of prediction markets. Our IT artefact will be an original working prototype of prediction market designed to support the R&D portfolio management process in real situations. This IT artefact will help managers and teams to support their periodical portfolio review using continuously updated information about their projects. Finally, this IT artefact should be viewed as a proof of concept to build new tools or to incorporate this new approach in existing tools.
We also will contribute to the R&D portfolio management literature, developing a new management methodology. This prediction market based approach should partially solve the recurring problem: the data collection process. Using this innovative approach, the whole process should be more efficient and transparent for projects leaders. Active on the market, projects leaders are able to introduce new ideas or concepts on the market via IPO and to follow their valuation in real time through the equilibrium price. If all our assumptions are verified, this should be a new and innovative way to support the R&D portfolio management process, applicable to all research types. The main advances made with our new methodology are: highly distributed and participative process, continuous actualization of the portfolio value, efficient and cost effective way to discover and aggregate the information disseminated between all actors and easy to understand resulting indicators.
Finally, our research will support the continuous effort made by the prediction market community to study new and innovative applications for e-markets. Currently, the applications are concentrated around forecasting public events like political elections, new technology adoption or sport results. The utilization of prediction markets in organizations is still at an early stage, more as an opportunity to test the concept than as productive applications. We think that such new applications would help organizations to enter in a new dynamic concerning prediction markets. This would then help to democratize the utilization of this concept and leverage new application opportunities for supporting critical business process.