A system for scenario analysis and forecasting of mass migration is presented. The system consists of a family of multi-scale models to address the need of responding agencies for better situational awareness, short and medium-term forecasts of migration patterns, and assess impact of changes on the ground. Such insights allow for better planning and resource allocations to address migrant needs. The analytical framework consists of three separate models (a) a global push-pull model to estimate macro-patterns, (b) a time-series prediction model for estimating future boundary conditions of crisis regions, and (c) a detailed network ow model that models population di usion within the crisis region and allows for scenario modeling. The paper presents the framework using the European refugee crisis as a case study. In addition, overall system design, practical considerations, end-user applications, and limitations of the modeling approach are discussed.
65.3 million people in the world today are forcibly displaced6. There are a further 232 million migrants worldwide who live outside their country of origin7. With these growing numbers, governments, international organizations, NGO's and other stakeholders face an increasing challenge in responding to migration crisis, such as the one in Europe recently. If necessary data and tools were available to forecast displacement crisis, response actions can be better coordinated. In this paper, we present a data-driven approach to enable responders to manage mass migration events.
Migration is an inherently complex and uncertain process. Direct observations of migration patterns are typically partial and inaccurate. Paths and destinations for migration are in uenced by a range of human factors. Information ? Currently at Vodafone, Italy
sources for some factors may exist and come in di erent forms. For example, structured data from organizations such as registration counts at selected sites or refugee camps may be available. For locations without observers, one may need to rely on unstructured data such as news reports. Past data on migration movements and di erent data sources can be used to learn patterns of movements. Governments policies impacting migration patterns can change over the course of time. The types of changes and the response of migrants to such changes can be di cult to ascertain from past observations. For such cases, approaches that model the underlying processes are required. The need for hybrid approaches that merge purely data-driven (capture past patterns) and ones that model the physics (capture aggregate interactions and exogenous inputs) are necessary.
The current climate of global displacement has inspired a growing awareness that an innovation gap exists in addressing social issues within the humanitarian sector and that creative methods of partnership between the public and private sectors are needed. While corporate philanthropy programs and social entrepreneurship have undoubtedly made inroads in this regard, in order to more comprehensively design, develop, apply solutions, and leverage the vast skill sets in big data and analytics, we must better incentivize competition and thereby critical thinking among the private sector.
The challenges in doing so are surmountable, but not trivial. We must overcome obstacles on both sides that are cultural and come to a mutual understanding that a symbiotic relationship between the humanitarian sector and revenue-driven organizations can not only exist, but will have an exponentially greater impact on social issues than when such projects are treated exclusively as philanthropic.
This research addresses the following questions. What information signals on mass migration are available? What analytical framework and models help in developing forecasts of mass migration? What scenarios are likely to occur and what is their impact on migration ows? The subsequent sections present the models that are designed to (a) enhance situational awareness using multiple data sources, (b) provide short and medium-term forecasts on migration patterns to aid operational decision making, and (c) enable `what-if' scenario analysis for agencies looking to study the impact of exogenous factors.
The model development is presented using the European migration crisis as a case study. The choice is motivated by availability of di erent datasets, for example the large volume of news reports on the crisis and detailed registration data at various sites. The dynamics of the crisis were notably complex, as migrants were crossing several international boundaries. The patterns of ight shifted as conditions across borders changed.
The forecasting and analysis tool for mass migration addresses the gap that the humanitarian sector has related to forecasting and scenario analysis. The developed models are presented in Section 4 with the system that implements these models is described in Section 5. The tool is aimed at agencies, governments, and NGOs that respond to crisis through the di erent states of refugee and migration movements.
Models dealing with the spatial movement of people have sparked the interest
of researchers for more than a hundred years, since Ravenstein's \Laws of
Migration" [
Current trends in quantitative mobility models have been in uenced in part by (a) the availability of personal digital traces such as mobile phone data, and (b) new information signals, such as satellite imagery and crowd-sourced initiatives.
Prediction of human mobility has bene ted from the large amount of
personal digital traces. Various methods have been proposed in the literature, such
as entropy rate of sequence of locations [
Additional information signals that have been leveraged to understand
migration recently include satellite imagery and crowd-sourcing e orts. Image
classi cation routines on satellite images have been used to quantify size and growth
of refugee camps. Examples include techniques using feature detection and
extraction methodologies [
Crowd-sourced information has been critical in providing better situational awareness for humanitarian agencies and NGOs. OpenStreetMap (OSM) provides an ecosystem of tools, events and volunteers that enable crowd-sourced gathering of information before and during crisis with focused initiatives such as a Humanitarian OSM Team8. These e orts are vital for relief e orts, estimation of refugee statistics, and community involvement.
With the European refugee crisis in 2015, there have been e orts aiming to fuse di erent data sources and provide migration awareness. The British Red
Cross9 provide a dashboard to monitor and forecast migration ows using daily arrivals information, border statuses, and news reports.
To address needs to end-user agencies, this work builds on these previous efforts by taking a multi-scale (global and local) approach that allows for scenario analysis. The multi-scale approach allows inclusion of factors at the global scale, while preserving detailed factors within the crisis region. Modeling scenarios allows for analysts to account for exogenous factors (e.g. repercussions of potential international policies). 3
Complex processes like migration depend on a wide range of factors. Data sources that describe the socio-demographics, overall context, con ict conditions (in the case of forced displacements), and changing conditions over a period of time are all determinants of migrations and paths of ight. There is no single functional relationship between these variables, so its essential to understand key information signals that can add value to data-driven modeling. Using the European crisis as a case, some common information sources are described. For multi-scale approaches, the data sources related to both operational (e.g. daily) and strategic (e.g. annual) characteristics.
Migrant Registration Data Registration data on migrant arrivals at key sites, such as the Greek islands provides daily, weekly or monthly arrival rates at various points along the crisis region. The UNHCR, UN's Refugee Agency, provides open data that is compiled from several sources10. The time series of arrivals at each of the sites can be used for several features in forecasting (see Section 4.1). Additionally, computing the cross-correlation function between the registration, provides an estimate of transit times through the network (used in Section 4.2). Figure 1 shows examples of time series shift that give estimates of transit times between select countries.
Weather Data Information on weather conditions and seasonal factors in uence movements. Adverse conditions are likely to restrict possible migration paths. For sea faring refugees arriving in the Greek islands, factors such as wind speed were found to be negatively correlated with arrivals over the winter months. Figure 2 shows the Pearson correlation coe cient between arrivals in di erent countries and di erent weather-related variables. Weather data is sourced from the Weather Underground11 service.
News Data In order to capture information about policy changes (e.g. close of the Hungary border) and other external events, we used the news data provided by the GDELT Project 12. It allows users to monitor the web news around
(a) Arrivals in Greece (6 days ahead) and Austria (b) Arrivals in Greece (2 days ahead) and FYRoM the world and extract valuable information from text such as entities (e.g people locations, organizations), counts, emotions and events in over 100 di erent languages. Documents are annotated applying a combination of state-of-the-art natural language processing techniques. Moreover, it is worth to mention that for each article Global Knowledge Graph (GKG) provides a variable called locations in which they report the places mentioned in the article. This is an important information since it allow us to localize the potential country of the news. Bilateral similarity scores, as shown in Figure 3, are used to train an a nity function (Section 4.3).
Country-level data Aggregate statistics at the country level were collected from sources such as the World Bank. Socio-economic indicators such as GDP, population, con ict status were considered. Datasets from the gravity model literature13 where included for dyadic features such as common languages, historical ties, trade volumes, and geographic distances. 4
Consider a crisis region of interest where dynamics of migration need to be evaluated. In the European crisis, this includes the regions around the Mediterranean Sea where paths of ight are concentrated. In addition, the region of interest should include potential transit paths that may be considered by migrants. The region could also include sources of forced displacement or intended destinations, e.g. Syria or Germany respectively.
The analysis framework consists of three distinct models that jointly provide forecasts. The main crisis region is modeled as a network ow model (Section 4.2). The representation is detailed and the models the rate of movements from one node to another. The boundary conditions are handled by the other two 13 http://www.cepii.fr/ models. An arrivals model (Section 4.1) is a time series forecasting model that pIrBeMdiRcetsseadrcahil–yIraelrarnidval rates at nodes on the edge of the crisis region. To determine exit rates and consider intended destinations, a macro push-pull model (Section 4.P3)reidsicutsiendg tmoigersatitmionatuesfirnagctaiofanmoiflymoifgmraondteslfsor each likely destination. Figure 4 shows this framework.
Model 2: macro migration patterns for exit nodes Model 3: flow modeling arrivals to an “intermediate/end node” Exit forecast
3 Fig. 4: Analytical framework showing delineation of a crisis region of interest. Arrival forecasts at boundary nodes are estimated using arrival prediction models (Section 4.1). Exit nodes forecasts are estimated from macro-migration model (Section 4.3). A detailed network ow model (Section 4.2) models the population movement within the crisis region. 4.1
Arrival Prediction Model The aim of this model is to predict the number of arrivals at the entry-points of the crisis region. Future arrivals are dependent on external variables such as weather, as well trends due to (de-) escalation of the crisis in question. To capture these e ects we used data from weather reports and news reports and experimented with multiple machine learning models to predict day-ahead arrivals of refugees to the Greek Islands. Around 180 features were extracted from the following datasets: Autoregressive Features (AF) Previously observed value describe the trend of arrivals until the time of observation. In the model we use as a feature the number of arrivals one day before as well as rolling statistic such as mean, variance, maximum and minimum computed over the previous week.
Weather Features (WF) We used six di erent weather variables: wind speed, temperature, sea level visibility, dew point, sea level pressure, humidity, and precipitation. To generate features, we aggregated the data computing the min, max, sum and mean of each variable over the day.
News Features (NF) To obtain news features, we queried the GKG historical data using the GKG Exporter 14. We ltered the news selecting those in which the word refugee and either migration or border appear. Then, we selected only articles that match the location eld with one of the countries involved in the migration crisis. Starting from these articles, three types of features are computed: (i) number of occurrences of relevant countries name (e.g. Syria and Greece) in the news, (ii) number of occurrences of relevant words related to humanitarian crisis (e.g. refugee camp) and, (iii) number of articles about humanitarian themes (e.g. migration). The former has been computed by applying a keyword-based approach where an article is assigned to a theme considering the presence of prede ned words like border or humanitarian.
We applied four di erent machine learning models: LASSO, Linear Regression (LR), Ridge Regression (RR) and Gradient Boosting of Regression Trees (GBRT). In order to reduce the trend in the signals, we trained our model on the number of arrivals minus the rolling mean of the arrivals in the previous seven days. We added this value to the prediction before computing error metrics. The hyper-parameters of each of the models were determined through crossvalidation; training the models on data between October 2015 and January 2016 and testing it on February 2016. Moreover, we normalized the feature values subtracting the mean and dividing by the variance. In Table 1 the results of the tested models are reported. Results of a persistence model prediction are also shown as a baseline. This model predicts arrivals each day to be equal to arrivals the previous day.
Due to the high number of features, we experimented with a feature selection step. To select features we rst tted one of the regression models. Then we ranked the features by the magnitude of their weight in the model (i.e. the slope). We used this approach on the results of LASSO, Linear Regression, and Ridge Regression. We also tried recursive feature elimination (RFE), and simple correlation.
We evaluated the quality of the feature selection by again through cross validation, training with the reduced set of features on the training set. RFE produced the worst set of features. The other methods produced similar sets of features with similar performance.
The smaller set of features lead to better model performance. After feature selection, each of the models shows improved performance, but the di erence between the models is small. The best performance are obtained using the LASSO model on a subset of 10 features that included weather, autoregressive and news data. In particular, from the weather features we used the forecasting mean and max for wind speed, the forecasting gust speed and the forecasting mean for hu14 http://analysis.gdeltproject.org/module-gkg-exporter.html midity, temperature and sea level pressure. Regarding autoregressive features, we used the di erence in number of arrivals between one and two days before, the mean and the minimum in number of arrivals for last three days, the number of arrivals one day before and the sign of the derivative in the last two days. Finally, the model also includes the number of news (for the arrival country) about international organization for migration. Table 1 shows the results.
Models
No Feature Selection RMSE Error
Reduction(%)
Feature Selection RMSE Error
Reduction(%)
For all of the models, the weather features are the most present, followed by the historical data. This could be due to the fact that we tested our model on arrivals in the Greek Islands where weather conditions in uence arrivals. Figure 5 shows a sample of forecasts for the Greek islands. There is not a signi cant di erence in terms of results between the linear models after the feature selection step. 4.2
Network Flow Model The challenge in modeling the paths of refugees through the crisis region is twofold. First, measured arrivals at di erent location along a path through the crisis region are known to be inaccurate. Second, the movement patterns can change due to changing conditions on the ground, such as border closures. In this section we present a network ow model that can address both issues.
To mitigate the e ect of inaccurate measurements, we developed a model that can impose some common sense boundary conditions on the prediction. Speci cally, arrivals in each country must correspond to departures in a di erent country. Departures from any country cannot exceed the total number of refugees present in that country.
Our model represents locations along the path of movement as nodes on a graph. The edges that connect the nodes i and j represent the likelihood that refugees will travel from node i to node j. As an example we will model travel of refugees from Greece to Austria through FYROM, Slovenia, Hungary, Croatia, and Serbia.
The number of people traveling from node i to node j at time t, denoted by Fij (t), is given by Arrivals in Greek Islands Lasso Regression Best Model (2) (3) (4) (5) Fij (t) = Pi(t)fij (1) where Pi(t) is the number of people present at node i, and fij is a constant that we will determine from historical arrivals data. The fij can be interpreted as split-fractions for the departures of each node. The net ow from node i to node j is then given by To simplify the problem, we consider only these net ows. Arrivals (Ai(t)), and departures (Di(t)) at node i are given by
Nij (t) = Pi(t)fij
Pj (t)fji j j Ai(t) = X max(0; Nji(t))
Di(t) = X max(0; Nij (t)) Feb 02 2016
The populations of the next time step can then be calculated as Pi(t + 1) = Pi(t) + Ai(t)
Di(t) + Ei(t) where Ei are exogenous arrivals, to be speci ed independently. Equations 1-5 can then be used iteratively to compute future ows.
In the present case we use Ei to specify arrivals to Greece from Turkey and departures from Austria to the rest of Europe. For arrivals to Greece, we use the outputs of the arrival prediction model discussed in the previous section. For departures from Austria, we assume all people present in the country, depart each day.
To determine the values fij , we optimized the following loss function.
L =
X(Ai(t) it
Mi(t))2 +
X ij jfij j (6) where Mi are the measured arrivals at node i, and is a regularization parameter. The second term in this equation is an L1 regularization. This regularization imposes sparsity in the possible paths.
The model is trained at the beginning of each week on the preceding 30 days. Forecasts are then generated for the subsequent 2 weeks. This means that for each day, there are two forecast values; the mean of those two values is shown in Figures 6 and 7.
Changing conditions on the ground can be de ned through adjustment of the exogenous arrivals. For example, starting mid February there were several policy changes that reduced the number of arrivals in Greece. This impacted arrivals in the other nodes of the network as well. We modeled this change by manually setting (exogenous) arrivals in Greece to 0 after February 16. Figures 6 and 7 show both scenarios compared to measured arrivals. Scenario 1 assumes no change to the arrivals in Greece, Scenario 2 assumes no more arrivals after February 16. This method can be used to make more accurate predictions if policy changes are known. In addition, this methods could be used to do counterfactual scenario analysis. 4.3
A push-pull macro migration model To derive destination preferences, a global model of migration that seeks to determine bilateral ows between countries is considered. The model seeks to estimate the fraction of refugees from each country that are likely to migrate to any other country. Such an approach is necessary to estimate macro-level movement patterns and serves to project intended destinations of migrants within the detailed network ow model.
The model assumes there are push (\repulsion") factors at a home country along with pull (\enticing") factors at the destination. Push factors at origin and pull factors at destination cause migration. Movement is also a function of distance/a nity between two countries. Countries with higher a nity are likely have more migration.
A nity metrics can be de ned in several ways. Classical approaches are using spatial properties such as geographic distance. In this work, we have considered an a nity function trained on past migration data and considering distance metrics, exogenous variables such as Gross Domestic Product (GDP), colonial relationships, commonality of language, contiguity, and con ict status of origin countries. One endogenous variable in bilateral migration in previous years (if available) was included to model `social pull' factors. News articles were also considered to include more current reports of migration.
FYROM arrivals (thousands)
FYROM measured FYROM predicted, Scenario 1
FYROM predicted, Scenario 2 Jan 2016
Feb
Mar Fig. 6: Output from network ow model - Predicted arrivals in the Former Yugoslav Republic of Macedonia (FYROM). The gure shows the measured arrivals as well as predicted arrivals in two di erent scenarios. Scenario 1 assumes the \status quo", that is, past data is representative of future movements. Scenario 2 assumes no more arrivals to Greece after February 16. Scenario 2 is more representative of the changed circumstances after several border closures. Forecasts are generated at the beginning of each week, for the subsequent two weeks.
Croatia arrivals (thousands)
Croatia measured Croatia predicted, Scenario 1
Croatia predicted, Scenario 2 Fig. 7: Output from network ow model - Predicted arrivals in Croatia. The gure shows the measured arrivals as well as predicted arrivals in two di erent scenarios. Scenario 1 assumes the \status quo", that is, past data is representative of future movements. Scenario 2 assumes no more arrivals to Greece after February 16. Scenario 2 is more representative of the changed circumstances after several border closures. Forecasts are generated at the beginning of each week, for the subsequent two weeks.
Since the true a nity measure is not known, a log-transformed linear model
with these features is tted on past bilateral migration and the set of features
described above [
More formally, we use the formulation presented in [
i2N j2N X Mij = Oi j2N X Mij = Ij 8i 2 N 8j 2 N i2N
The model computes Mij , the migration ows between sites i and j. The numerical push-pull estimates can be ascertained by solving a linear system of equations, once all out ows and in ows are known.
X j2S;j6=i
X i2S;i6=j
Mij = Ri
X j2S;j6=i dij 1
+ Mij =
X i2S;i6=j dij
Ri + Ej
X
X j2S;j6=i dij i2S;i6=j dij
Ej = Oi 8i 1 = Ij 8j; where Ri is the 'repulsion' (push) factor and Ej are estimates of the 'enticing' (pull) factor. For distance-based a nity functions, the units associated with the push-pull quantities can be interpreted as `person-kilometers'. However, for more complex a nity functions, the values are relative and cannot be interpreted directly.
We have tested 4 cases for the a nity function. The rst case considers geographic distances only. Case 2 additionally considers exogenous factors such as GDPs, common languages, contiguity, con ict status. Case 3 considers additionally \social pull" proxies, such as historical migration. Case 4 includes similarity measures based on news sources (see Section 3). Forecast errors were measured for years when the bilateral ows were known (currently 2013). The resulting split fractions serve as input to the ow model. 5
The models and tools are currently being deployed in an instance of IBM Bluemix15. A limited release is planned with a suitable agency for testing and validation. 15 http://www.ibm.com/cloud-computing/bluemix/ (7) (8) (9) (10) (11)
Case Features RMSE Case 1 Distance 11882.58 Case 2 Distance + exogenous variables 9494.30 Case 3 Distance + exo. vars + past migration 814.21 Case 4 Distance + exo. vars + past migration + news 814.21 Table 2: RMSE for di erent cases in persons per year (2013)
The prototype follows a client-server architecture in a scalable and e cient manner. The backend is implemented as a REST service and exposes the functionality of the models described in Section 4. The frontend application serves to allow for user-speci ed scenario modeling and displaying model results. The functionality exposed by the REST services is modular, so the general purpose functions can be consumed and by multiple applications. The system can therefore integrate easily with existing applications used by NGOs across the globe.
Methods associated with the push-pull model (Section 4.3) exposes only a training functionality. The computed model represents the actual migration ows between the observed countries, is presented to the user via the User Interface (UI). The predicted a nity function can be periodically updated to re ect the political, social and economic changes.
Methods associated with the network ow and arrivals prediction models (Sections 4.1 and 4.2 respectively) are for training and forecasting. The training methods returns a serialization of the computed model which can be used, by the prediction method, to produce a forecast of arrivals or arrivals distribution in the network, respectively.
Figure 8 shows the current version of the user interface for the prototype. It allows the user to travel across di erent spatial and temporal scales, as well as to uidly integrate additional external resources. Using this interface an operator can visualize the ows of migrants at di erent granularity. Similarly, the operator can interact with the models to alter assumptions and change model parameters.
The UI visualizes the predicted arrivals of migrants in the various nodes of the network. An NGO is then able to visually assess the criticality level for each area of a given crisis region, and to better orchestrate on-the-ground operations. 6
Data-aware tools, processes and methods have the potential to improve operations in the humanitarian sector. Forecasting and scenario modeling tools, such as those presented here, aid agencies to move to more proactive operations. There are several challenges to be addressed for wider uptake. The classical philanthropic engagement model with the private sector needs to shift to a more collaborative approach, where varied expertise across organizations can be tapped and models and methods re ned.
The work presented has the following limitations. Since each migration crisis is unique, impact of some features used to model the process will be di erent across di erent contexts. Past observed factors may have little or no role in future crises. For example, wind speeds used to forecast migration arrivals in Greece will not yield useful information for land-based migration.
Partly relying on physical models and enabling scenario analysis helps mitigate some of these limitations. However, such models are based on assumptions that may also be speci c to a particular context. Ensuring that the right assumptions are considered and appropriated incorporated within the models is key.
Precise measurements, and in turn forecasts, remain a challenge on the ground and for models. While relative measures provide indications on how resources could be potentially deployed, the absolute numbers may be critical for some applications.