Designing with citizens: Challenges and evaluation methods for crowd-sourced urban layouts Johannes Mueller ETH Zurich, Future Cities Laboratories, Singapore-ETH Centre, 1 CREATE Way, CREATE Tower, 138602 Singapore johannes.mueller@arch.ethz.ch Hangxin Lu ETH Zurich, Chair of Information Architecture Wolfgang-Pauli-Str. 27, CH-8093 Zurich, Switzerland lu@arch.ethz.ch Abstract must be as low as possible. Any discussion that requires citizens be physically present are exclusionary, and do not This paper presents analysis tools for evaluating leverage on contemporary communications technologies to crowdsourced geometry-based design proposals for include as many citizens as possible for more representative urban planning. With the Quick Urban Analysis feedback. Secondly, tools must be provided to facilitate the Kit, an online platform, citizens are able to creation of creative solutions to problems, and in particular manipulate objects and create a preferred layout design proposals. Surveys and voting systems are over a case study area. Given that our case study is inadequate for this purpose. on a meso-cale, our analysis is focused on the layout and plot configuration. The proposed American designer and social scientist Elisabeth Sanders analysis tools range from simple counting of object wrote about how people can contribute as co-designers types and a buffer analysis to clustering and spatial [Sanders, 2002]. In order for this contribution to happen, autocorrelation tools. Besides these form-based however, a designer must faciliate access to the experience criteria, perception-based criteria are also proposed of the user. For Sanders, people express these experiences to link the participating subject’s assessment of the by talking, thinking, doing, using, knowing, feeling and designs with the layout. Techniques deployed dreaming. While the first four activities are explicit and include supervised machine learning methods, observable, the latter are more tacit and latent. To access statistical spatial tests, and simple calculations of these levels of experience, she proposed applying “make the area size and frequency of objects. tools”. “Make tools” enable people to express themselves in many ways. For instance, cognitive toolkits that help people Keywords: citizen design science, crowd-creative evaluation, create maps and 3D models can show how they perceive geometry-based evaluation, creative participatory planning and understand a place, as such tools force people to think and express themselves in novel ways. We elaborate on 1 Introduction and Related Work Sanders’ concept and use make tools in the participatory urban design process for the layperson. This combination of citien science and design is what we name ‘Citizen Design In recent decades, the contribution of citizen participation Science’ [Mueller et al., 2017]. has led to the improvement of democratic governance and other adjacent fields. These contributions are thought to Crowdsourced participatory urban design may be regarded establish a sense of citizenship, increase positive attitudes as a specific case in collective intelligence. Previous work and strengthen responsive and accountable states [Pateman, on collective intelligence is divided into two parts, 1970; Mansbridge, 1997; Gaventa & Barrett, 2010]. participatory design and crowdsourcing, with the former Conversely, scholars are also skeptical about the perceived practiced in design communities and the latter researched in autonomy of citizen participation, citing external influences urban computing [Peffers et al., 2007]. Yu and Nickerson such as elite capture, lack of civic capacities, or other [Yu & Nickerson, 2013] integrate the two domains, namely factors [Bonfiglioli, 2003; Golooba-Mutebi, 2004; Banerjee human intelligence with machine processes, and postulate a et al., 2010]. We propose new forms for citizen participation crowdsourced idea generation process that facilitates the in the urban planning process. Two main factors are combination of ideas. Our work also combines the two important in our vision. Firstly, the hurdles for participation domains but we focus instead on design evaluation, since our platform already enables effective idea generation from such that participants can solve it in a few minutes. It is the crowd. possible to see it as a tool that gamifies design problem- solving. This paper describes some options for evaluating crowd- creative design proposals for redeveloping urban areas. We One of the tool’s drawbacks is that objects cannot be take a township in South Africa as a case study area. Using directly edited. This reduces participant creativity, but also a 3D geometry viewer and editor we explain how a Citizen ensures that they only focus solely on the configuration of Design Science project on meso-scale can be conducted and objects. analysed. 2.2 Case study area: Empower Shack 2 Tool and case study description Figure 2 shows a neighbourhood in Khayelitsha, an 2.1 Qua-kit: A 3D object viewer and editor informal settlement 22km southeast of Cape Town. The project Empower Shack developed prototypes of new shacks Quick Urban Analysis Kit (qua-kit) was developed by which can be extended to two storey accommodations, Artem Chirkin at the Chair of Information Architecture at thereby using the space more efficiently. The residents of the neighbourhood were involved in the rearrangement of the ETH Zurich [Chirkin & Koenig, 2016]. The software shacks, although they did not use the qua-kit. interface is an online viewer retrievable via http://qua- kit.ethz.ch/viewer. This viewer can show 3D objects which The site was prepared for the MOOC lectures Smart Cities are either movable or static. The main function of this (https://www.edx.org/course/smart-cities-ethx-ethx-fc-03x- platform is the manipulation of object positions in two 0) and approximately 500 students submitted their proposals dimensions, including rotation. It is not possible to place via qua-kit. This paper does not focus on the results of the blocks on top of one another. The user can make modifications with the left mouse button, right-click to change the point of view and use the scroll wheel to zoom in and out (Figure 1). A mouse is more intuitive than a touchscreen because the latter would require an additional key for further object modification. Figure 2: The informal settlement of Khayelitsha, South Africa. [Lloyd & Bolnick, 2015] students’ work as research is ongoing. Instead, it presents design criteria and techniques to make the mass of designs useful for designers and decision-makers. The data from the participants are not pictures, but geo-data. This allows for a Figure 1: Screenshot of the qua-kit viewer. The object are wide variety of evaluation options for the data which movable in x- and y-direction as well as rotatable. By right- improves precision, in comparison to pictures which must clicking and scrolling, the user can change the view be pre-analysed with image recognition methods. perspective and zoom in or out. This simple web application enables non-expert designers to 3 Evaluation tools modify given geometry layouts according to their individual preferences. The focus is on configuring geometries, not on Our data analysis distinguishes between form-based and the infrastructure or creating new items. The final layout can perception-based criteria. Form-based criteria quantify the be saved and submitted with optional comments on the layout of buildings and the appearance of objects. user’s design motives or further explanations. Participants Perception-based criteria formulate conclusions on the can also vote and comment on other particpant-proposals participants’ perception of the area that can be made by and reflect on their own ideas and preferences. analysing the geometry. To keep the different analysis tools clear and The ease of use is a key factor for citizen science studies. commensurable, we present them in form of profiles. We Qua-kit offers the opportunity for design without any explain the method and purpose of each analysis and instructions by designers. Design tasks can be formulated elaborate on the pros and cons. 3.2 Form-based criteria Frequency analysis Method: The objects are counted by object type. Purpose: To find out preferences for object types. The more often an object is used by the participant in the proposal, the more it is prefered. Pro: The comparisons between different building types (high-rise, mid-rise, low-rise) can be useful for decision makers. Several other standard design criteria can be deduced: given the area, plot size and number of floors, quantities like the Gross Plot Ratio, Gross Floor Ratio, the number of units, and the density of objects are easily Figure 3: Space and street detection. Streets are marked as computed. The advantage of the frequency analysis is that it black lines, public spaces as black polygons. The shaded areas is simple and thus easy to understand for designers and are private spaces (no street access). decision makers. Pro: This additional layer helps researchers understand the organisation of the area. Contra: Geometric information is ignored. Contra: Distance-based assignments of streets and space Buffer analysis demand some general interpretation in advance. If the scale of the site is clear, the participants’ interpretation of space Method: The objects in the circuit (buffer) of a particular will match the automatic detection. If, for instance, the area object type are counted. of our use case would not be introduced as an informal settlement in South Africa, one could interpret the site as a Purpose: This analysis considers the position of objects and suburban area in North America instead, for example. Also, assesses the mutual appearance of objects, and shows the algorithm could misidentify some public squares. interrelations between objects. Geometry pattern analysis Pro: This analysis is the perfect for association rule mining. The results can reveal insights into the citizen’s Method: According to Ching [1979], the arrangement of subconscious decision-making processes e.g. what to build buildings usually follows some typical ordering principles. along a river, or which building typologies should be built Axes and symmetries are quite obvious and easily next to each other. detectable. While axes can be identified by distance rules since they can be considered in our example as streets, the Contra: A carefully considered interpretation is essential to recognition of symmetries requires a specific algorithm. prevent far-fetched outcomes as not every result from the assocaition rule mining algorithm is inherently meaningful. Purpose: Axes and symmetries are strong indicators of how a study area is organised. Space and streets detection Pro: The two form criteria are well studied in architecture Method: Streets, and public and private spaces are and urban planning and already implemented in algorithms automatically added as an additional layer based on the [Chen, et al., 2007]. distance between buildings. A street is detected if the distance between buildings are within the boundaries of a Contra: While the axis may be detected using the distance minimum distance x and a maximum distance y. For our between buildings, strict symmetries are supposed to be case study, these boundaries could be set to ! = 1$ and rarely identifiable in human-made designs, especially in the % = 4$. Distances greater than y are interpreted as open qua-kit tool which allows for flexible rotation and spaces. If an open space is accessible via a street, it is placement of buildings with no “magnetic features”. public; otherwise, it is private (Figure 3). Heat maps Purpose: Researchers can draw indirect conclusions about the street network and the placement of public and private Method: The 2D plots from different participants (optional: spaces. all) are superimposed. The merging of data can be done visually and geometry-based, too. Is is optional to deploy a Pro: The autocorrelation test is an objective measurement hot spot analysis like the Gets Ord GI* [Mitchell, 2005; for a very subjective aspect for the perception of space. Getis & Ord, 1992; Ord & Getis, 1995] in advance. Contra: The dispersion of buildings can have many reasons, Purpose: The heat map shows preferred areas for particular making simple interpretations hard. objects and also directly reveals the spatial distribution of object types. 3.3 Perception-based criteria Pro: This methods allows the visual merging of proposals from different participants. All other proposed analyses are Creativity analysis applied for separate designs. Method: Human intelligence is required to label design Contra: The plots of the buildings are decontexualised. proposals regarding creativity. Crowdsourcing internet platforms like Amazon Turk or Crowdfower are used to present workers screenshots of the participants’ designs. One option is to show two designs and ask which is perceived to be more creative. By using the Microsoft TrueSkill algorithm [Herbrich et al., 2007], the designs can be ranked. Kanzjon et al. have evaluated the design creativity of mobile devices using factors like novelty, value and surprise [Grace et al., 2015]. Purpose: Creativity is hitherto a very subjective impression and there are no rules or algorithms for making a decision. Creativity is an important factor for the uniqueness and individuality of a design and needs to be assessed by Figure 4: Heat map exemplarily shown with five different humans. design proposals. Clustering Pro: The analysis can be extended for supervised machine learning. The 2D layout of the plot is labeled with the Method: The 2D centroids of buildings are calculated and creativity index according to the workers’ result. used for spatial clustering, e.g. with the DBSCAN algorithm [Ester et al., 1996]. Contra: A large number of workers need to be employed to ensure unbiased labeling and to compensate for unreliable Purpose: On meso-scale, the space is subdived into smaller works. neighbourhoods like blocks. The preferred number and block sizes can be accessed through clustering. Meta information analysis Pro: The clustering approach allows segmentation to happen Method: Similar to the creativity analysis, we want objective objectively. criteria that can be adhered to when human feedback is given for design proposals. The concern of the meta Contra: A cluster is not neccesarily meant to be a block. information analysis is not how other humans perceive the The examples in Figure 5 show that parameters in the participant’s design but what the participant thinks of their cluster algorithm need to be carefully adjusted for the case own proposal. By proposing categories for the main purpose study area, but even this does not guarantee a satisfactory or idea of the design (e.g. safety, dominance of greenery, result. accessibility,...), the designs can be labeled, and used for the application of supervised machine learning to the geometry Autocorrelation test and purpose. Method: A spatial autocorrelation, e.g. Global Moran's I Purpose: The idea is to identify characteristics in the [Moran, 1950] is applied for different objects types. geometry in order to infer the main purpose of the design. Purpose: The test reveals if object typologies appear Pro: This analysis allows very subjective characteristics of dispersed or clustered in the area. If they are clustered, the designs to be quantified. buffer analysis can give some indication of the interrelation between objects. Contra: The success of this method is not predictable. Participants may not be clear about what the main idea behind their designs is. 4 Discussion National Research Foundation (FI 370074016) under its Campus for Research Excellence and Technological The qua-kit tool is designed for online participation with Enterprise programme. Moreover, we wish to thank Artem citizens. The methods in this paper provide options to Chirkin who has developed the Quick Urban Analysis Kit, precisely evaluate a large number of designs which are for giving us access to the toolkit and geometric data from based on geo-information. Questions may be asked about the edX MOOC Smart Cities to conduct our research. whether these presented online design tools offer enough options for the participants to express their ideas, or if designers can gain any valueable inputs from these tools. References The form-based criteria are simple to understand, but how [Banerjee et al., 2010] Abhijit V. Banerjee, Rukmini can the results be interpreted? Especially the opportunity of Banerji, Esther Duflo, Rachel Glennerster, and Stuti letting non-experts design provides the chance to access Khemani. "Pitfalls of Participatory Programs: Evidence knowledge that cannot be expressed semantically (e.g. by from a randomized evaluation in education in India." surveys). The evaluation should therefore concentrate on American Economic Journal: Economic Policy 2.1: 1- identifying the design’s semantic meaning how to make 30. 2010 conclusions from the layout to the design idea beyond the [Bhatt et al., 2012] Mehul Bhatt, Carl Schultz, and Minqian proposal. This question is about to be answered by findings Huang. "The shape of empty space: Human-centred about how space is perceived [Bhatt et al. 2012, 2013, cognitive foundations in computing for spatial design." 2014]. Assuming that the design process is guided by the In Visual Languages and Human-Centric Computing participant’s experience of space (i.e. the perception of (VL/HCC), 2012 IEEE Symposium on, pp. 33-40. IEEE, space), it makes sense to analyse the design proposals by 2012. criteria that are found out to have a significant influence on the space organisation [Ching, 1979]. Axes and symmetries [Bhatt, et al., 2013] Mehul Bhatt, Carl Schultz, and are already mentioned and it is for discussion, how and to Christian Freksa. "The ‘Space’ in spatial assistance which extent the other perception-based criteria should be systems: conception, formalisation and computation." applied. Representing space in cognition: Interrelations of behavior, language, and formal models. Series: 5 Conclusion and future work Explorations in Language and Space. 2013: 978-0. This paper presents a brief summary of evaluation methods [Bhatt, et al., 2014] Mehul Bhatt, Carl Schultz, and for crowdsourced geometric data on the meso-scale. The Madhura Thosar. "Computing narratives of cognitive techniques are to be used in participatory urban planning user experience for building design analysis: KR for studies. The introduced design tool qua-kit allows non- industry scale computer-aided architecture design." 14th experts to view 3D geometries and make simple edits. We International Conference on the Principles of use a case study from a neighbourhood near Cape Town to Knowledge Representation and Reasoning. 2014. show options to evaluate the data on meso-scale. The most [Bonfiglioli, 2003] Angelo Maliki Bonfiglioli. Empowering comprehensible instruments for analysis are the frequency the poor: Local governance for poverty reduction. and buffer analysis. Streets and public or private space can United Nations Publications, 2003. be determined by appropriate distance rules while cluster [Chen, et al., 2007] Po-Chun Chen, J. H. Hays, Seungkyu algorithms make small neighbourhoods automatically Lee, Minwoo Park, and Yanxi Liu. "A quantitative identifyable. The overlays of plots from different evaluation of symmetry detection algorithms." participants can be regarded as a heat map and visualise Pennsylvania State University, Department of Computer favoured spots for particular objects. Besides these form- Science and Engineering. 2007. based criteria, we propose to apply supervised machine- learning to connect the layout of buildings to subjective [Ching, 1979] Francis F.D. Ching. Architecture: Form, design criteria. The application of these evaluation tools will space, and order. John Wiley & Sons. 1979 testify their usefulness or uselessness. Especially for [Chirkin & Koenig, 2016] Artem Chirkin, and Reinhard accessing perception-based criteria, it is crucial to have easy König. "Concept of Interactive Machine Learning in understandable and distinct labels for the proposals. Urban Design Problems." Proceedings of the SEACHI A further option is to apply unsupervised machine learning 2016 on Smart Cities for Better Living with HCI and UX. to the geometries to identify different categories of ACM, 2016. proposals or similarities in the design. [Ester et al., 1996] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. "A density-based algorithm for discovering clusters in large spatial databases with Acknowledgments noise." Kdd. Vol. 96. No. 34. 1996. The research was conducted at the Future Cities Laboratory [Gaventa & Barrett 2010] John Gaventa, and Gregory at the Singapore-ETH Centre, which was established Barrett. "So what difference does it make? Mapping the collaboratively between ETH Zurich and Singapore's outcomes of citizen engagement." IDS Working Papers 2010.347: 01-72. 2010 [Getis & Ord, 1992] Arthur Getis, and J. Keith Ord. "The analysis of spatial association by use of distance statistics." Geographical analysis 24.3: 189-206. 1992 [Golooba-Mutebi 2004] Frederick Golooba-Mutebi. "Reassessing popular participation in Uganda." Public Administration and Development 24.4: 289-304. 2004 [Herbrich et al., 2007] Ralf Herbric, Tom Minka, and Thore Graepel. "TrueSkill™: A Bayesian Skill Rating System." In Advances in Neural Information Processing Systems, pp. 569-576. 2007. [Grace et al., 2015] Kazjon Grace, Mary Lou Maher, Douglas Fisher, and Katherine Brady. "Data-intensive evaluation of design creativity using novelty, value, and surprise." International Journal of Design Creativity and Innovation 3.3-4 (2015): 125-147. [Lloyd & Bolnick, 2015] Scott Lloyd, and Andy Bolnick. “Empower: An Integrated Development for the Upgrade of BT Section, Site C, Khayelithsa – Design Updates”. ETH Zurich [Mansbridge 1997] Jane Mansbridge. "Normative theory and voice and equality." American Political Science Association, 91.2: 423-425. 1997 [Mitchell, 2005] Andy Mitchell. "The ESRI Guide to GIS Analysis: Spatial Measurements and Statistics. Vol 2. Redlands." (2005). [Moran, 1950] Patrick AP Moran. "Notes on continuous stochastic phenomena." Biometrika 37.1/2: 17-23. 1950 [Mueller et al., 2017] Johannes Mueller, Hangxin Lu, Artem Chirkin, Bernhard Klein, and Gerhard Schmitt. “Citizen Design Science: A strategy for crowd-creative urban design.” In Cities, 2017 (accepted, not yet published) [Ord & Getis, 1995] J. Keith Ord, and Arthur Getis. "Local spatial autocorrelation statistics: distributional issues and an application." Geographical analysis 27.4: 286-306. 1995 [Pateman, 1970] Carole Pateman. Participation and democratic theory. Cambridge University Press, 1970. [Peffers et al., 2007] Ken Peffers, Tuure Tuunanen, Marcus A. Rothenberger, and Samir Chatterjee. "A design science research methodology for information systems research." Journal of management information systems 24.3: 45-77. 2007 [Sanders, 2002] Elizabeth Sanders. "From user-centered to participatory design approaches." Design and the social sciences: Making connections. CRC Press, 2002. 1-8. [Yu & Nickerson 2013] Lixiu Yu, and Jeffrey V. Nickerson. "An internet-scale idea generation system." ACM Transactions on Interactive Intelligent Systems (TiiS) 3.1 : 2. 2013