=Paper=
{{Paper
|id=Vol-2099/CAID17_paper6
|storemode=property
|title=Designing with Citizens: Challenges and Evaluation Methods for Crowd-sourced Urban Layouts
|pdfUrl=https://ceur-ws.org/Vol-2099/paper6.pdf
|volume=Vol-2099
|authors=Johannes Mueller,Hangxin Lu
|dblpUrl=https://dblp.org/rec/conf/ijcai/MuellerL17
}}
==Designing with Citizens: Challenges and Evaluation Methods for Crowd-sourced Urban Layouts==
https://ceur-ws.org/Vol-2099/paper6.pdf
Designing with citizens: Challenges and evaluation methods for crowd-sourced
urban layouts
Johannes Mueller
ETH Zurich, Future Cities Laboratories, Singapore-ETH Centre,
1 CREATE Way, CREATE Tower, 138602 Singapore
johannes.mueller@arch.ethz.ch
Hangxin Lu
ETH Zurich, Chair of Information Architecture
Wolfgang-Pauli-Str. 27, CH-8093 Zurich, Switzerland
lu@arch.ethz.ch
Abstract must be as low as possible. Any discussion that requires
citizens be physically present are exclusionary, and do not
This paper presents analysis tools for evaluating
leverage on contemporary communications technologies to
crowdsourced geometry-based design proposals for
include as many citizens as possible for more representative
urban planning. With the Quick Urban Analysis
feedback. Secondly, tools must be provided to facilitate the
Kit, an online platform, citizens are able to
creation of creative solutions to problems, and in particular
manipulate objects and create a preferred layout
design proposals. Surveys and voting systems are
over a case study area. Given that our case study is inadequate for this purpose.
on a meso-cale, our analysis is focused on the
layout and plot configuration. The proposed
American designer and social scientist Elisabeth Sanders
analysis tools range from simple counting of object
wrote about how people can contribute as co-designers
types and a buffer analysis to clustering and spatial
[Sanders, 2002]. In order for this contribution to happen,
autocorrelation tools. Besides these form-based
however, a designer must faciliate access to the experience
criteria, perception-based criteria are also proposed of the user. For Sanders, people express these experiences
to link the participating subject’s assessment of the
by talking, thinking, doing, using, knowing, feeling and
designs with the layout. Techniques deployed
dreaming. While the first four activities are explicit and
include supervised machine learning methods,
observable, the latter are more tacit and latent. To access
statistical spatial tests, and simple calculations of
these levels of experience, she proposed applying “make
the area size and frequency of objects.
tools”. “Make tools” enable people to express themselves in
many ways. For instance, cognitive toolkits that help people
Keywords: citizen design science, crowd-creative evaluation, create maps and 3D models can show how they perceive
geometry-based evaluation, creative participatory planning and understand a place, as such tools force people to think
and express themselves in novel ways. We elaborate on
1 Introduction and Related Work Sanders’ concept and use make tools in the participatory
urban design process for the layperson. This combination of
citien science and design is what we name ‘Citizen Design
In recent decades, the contribution of citizen participation Science’ [Mueller et al., 2017].
has led to the improvement of democratic governance and
other adjacent fields. These contributions are thought to Crowdsourced participatory urban design may be regarded
establish a sense of citizenship, increase positive attitudes as a specific case in collective intelligence. Previous work
and strengthen responsive and accountable states [Pateman, on collective intelligence is divided into two parts,
1970; Mansbridge, 1997; Gaventa & Barrett, 2010]. participatory design and crowdsourcing, with the former
Conversely, scholars are also skeptical about the perceived practiced in design communities and the latter researched in
autonomy of citizen participation, citing external influences urban computing [Peffers et al., 2007]. Yu and Nickerson
such as elite capture, lack of civic capacities, or other [Yu & Nickerson, 2013] integrate the two domains, namely
factors [Bonfiglioli, 2003; Golooba-Mutebi, 2004; Banerjee human intelligence with machine processes, and postulate a
et al., 2010]. We propose new forms for citizen participation crowdsourced idea generation process that facilitates the
in the urban planning process. Two main factors are combination of ideas. Our work also combines the two
important in our vision. Firstly, the hurdles for participation domains but we focus instead on design evaluation, since
our platform already enables effective idea generation from such that participants can solve it in a few minutes. It is
the crowd. possible to see it as a tool that gamifies design problem-
solving.
This paper describes some options for evaluating crowd-
creative design proposals for redeveloping urban areas. We One of the tool’s drawbacks is that objects cannot be
take a township in South Africa as a case study area. Using directly edited. This reduces participant creativity, but also
a 3D geometry viewer and editor we explain how a Citizen ensures that they only focus solely on the configuration of
Design Science project on meso-scale can be conducted and objects.
analysed.
2.2 Case study area: Empower Shack
2 Tool and case study description
Figure 2 shows a neighbourhood in Khayelitsha, an
2.1 Qua-kit: A 3D object viewer and editor informal settlement 22km southeast of Cape Town. The
project Empower Shack developed prototypes of new shacks
Quick Urban Analysis Kit (qua-kit) was developed by which can be extended to two storey accommodations,
Artem Chirkin at the Chair of Information Architecture at thereby using the space more efficiently. The residents of
the neighbourhood were involved in the rearrangement of
the ETH Zurich [Chirkin & Koenig, 2016]. The software
shacks, although they did not use the qua-kit.
interface is an online viewer retrievable via http://qua-
kit.ethz.ch/viewer. This viewer can show 3D objects which The site was prepared for the MOOC lectures Smart Cities
are either movable or static. The main function of this (https://www.edx.org/course/smart-cities-ethx-ethx-fc-03x-
platform is the manipulation of object positions in two 0) and approximately 500 students submitted their proposals
dimensions, including rotation. It is not possible to place via qua-kit. This paper does not focus on the results of the
blocks on top of one another. The user can make
modifications with the left mouse button, right-click to
change the point of view and use the scroll wheel to zoom in
and out (Figure 1). A mouse is more intuitive than a
touchscreen because the latter would require an additional
key for further object modification.
Figure 2: The informal settlement of Khayelitsha, South
Africa. [Lloyd & Bolnick, 2015]
students’ work as research is ongoing. Instead, it presents
design criteria and techniques to make the mass of designs
useful for designers and decision-makers. The data from the
participants are not pictures, but geo-data. This allows for a
Figure 1: Screenshot of the qua-kit viewer. The object are wide variety of evaluation options for the data which
movable in x- and y-direction as well as rotatable. By right- improves precision, in comparison to pictures which must
clicking and scrolling, the user can change the view be pre-analysed with image recognition methods.
perspective and zoom in or out.
This simple web application enables non-expert designers to
3 Evaluation tools
modify given geometry layouts according to their individual
preferences. The focus is on configuring geometries, not on Our data analysis distinguishes between form-based and
the infrastructure or creating new items. The final layout can perception-based criteria. Form-based criteria quantify the
be saved and submitted with optional comments on the layout of buildings and the appearance of objects.
user’s design motives or further explanations. Participants Perception-based criteria formulate conclusions on the
can also vote and comment on other particpant-proposals participants’ perception of the area that can be made by
and reflect on their own ideas and preferences. analysing the geometry.
To keep the different analysis tools clear and
The ease of use is a key factor for citizen science studies.
commensurable, we present them in form of profiles. We
Qua-kit offers the opportunity for design without any
explain the method and purpose of each analysis and
instructions by designers. Design tasks can be formulated
elaborate on the pros and cons.
3.2 Form-based criteria
Frequency analysis
Method: The objects are counted by object type.
Purpose: To find out preferences for object types. The more
often an object is used by the participant in the proposal, the
more it is prefered.
Pro: The comparisons between different building types
(high-rise, mid-rise, low-rise) can be useful for decision
makers. Several other standard design criteria can be
deduced: given the area, plot size and number of floors,
quantities like the Gross Plot Ratio, Gross Floor Ratio, the
number of units, and the density of objects are easily Figure 3: Space and street detection. Streets are marked as
computed. The advantage of the frequency analysis is that it black lines, public spaces as black polygons. The shaded areas
is simple and thus easy to understand for designers and are private spaces (no street access).
decision makers. Pro: This additional layer helps researchers understand the
organisation of the area.
Contra: Geometric information is ignored.
Contra: Distance-based assignments of streets and space
Buffer analysis demand some general interpretation in advance. If the scale
of the site is clear, the participants’ interpretation of space
Method: The objects in the circuit (buffer) of a particular will match the automatic detection. If, for instance, the area
object type are counted. of our use case would not be introduced as an informal
settlement in South Africa, one could interpret the site as a
Purpose: This analysis considers the position of objects and suburban area in North America instead, for example. Also,
assesses the mutual appearance of objects, and shows the algorithm could misidentify some public squares.
interrelations between objects.
Geometry pattern analysis
Pro: This analysis is the perfect for association rule mining.
The results can reveal insights into the citizen’s Method: According to Ching [1979], the arrangement of
subconscious decision-making processes e.g. what to build buildings usually follows some typical ordering principles.
along a river, or which building typologies should be built Axes and symmetries are quite obvious and easily
next to each other. detectable. While axes can be identified by distance rules
since they can be considered in our example as streets, the
Contra: A carefully considered interpretation is essential to recognition of symmetries requires a specific algorithm.
prevent far-fetched outcomes as not every result from the
assocaition rule mining algorithm is inherently meaningful. Purpose: Axes and symmetries are strong indicators of how
a study area is organised.
Space and streets detection
Pro: The two form criteria are well studied in architecture
Method: Streets, and public and private spaces are and urban planning and already implemented in algorithms
automatically added as an additional layer based on the [Chen, et al., 2007].
distance between buildings. A street is detected if the
distance between buildings are within the boundaries of a Contra: While the axis may be detected using the distance
minimum distance x and a maximum distance y. For our between buildings, strict symmetries are supposed to be
case study, these boundaries could be set to ! = 1$ and rarely identifiable in human-made designs, especially in the
% = 4$. Distances greater than y are interpreted as open qua-kit tool which allows for flexible rotation and
spaces. If an open space is accessible via a street, it is placement of buildings with no “magnetic features”.
public; otherwise, it is private (Figure 3).
Heat maps
Purpose: Researchers can draw indirect conclusions about
the street network and the placement of public and private Method: The 2D plots from different participants (optional:
spaces. all) are superimposed. The merging of data can be done
visually and geometry-based, too. Is is optional to deploy a Pro: The autocorrelation test is an objective measurement
hot spot analysis like the Gets Ord GI* [Mitchell, 2005; for a very subjective aspect for the perception of space.
Getis & Ord, 1992; Ord & Getis, 1995] in advance.
Contra: The dispersion of buildings can have many reasons,
Purpose: The heat map shows preferred areas for particular making simple interpretations hard.
objects and also directly reveals the spatial distribution of
object types.
3.3 Perception-based criteria
Pro: This methods allows the visual merging of proposals
from different participants. All other proposed analyses are Creativity analysis
applied for separate designs.
Method: Human intelligence is required to label design
Contra: The plots of the buildings are decontexualised. proposals regarding creativity. Crowdsourcing internet
platforms like Amazon Turk or Crowdfower are used to
present workers screenshots of the participants’ designs.
One option is to show two designs and ask which is
perceived to be more creative. By using the Microsoft
TrueSkill algorithm [Herbrich et al., 2007], the designs can
be ranked. Kanzjon et al. have evaluated the design
creativity of mobile devices using factors like novelty, value
and surprise [Grace et al., 2015].
Purpose: Creativity is hitherto a very subjective impression
and there are no rules or algorithms for making a decision.
Creativity is an important factor for the uniqueness and
individuality of a design and needs to be assessed by
Figure 4: Heat map exemplarily shown with five different humans.
design proposals.
Clustering Pro: The analysis can be extended for supervised machine
learning. The 2D layout of the plot is labeled with the
Method: The 2D centroids of buildings are calculated and creativity index according to the workers’ result.
used for spatial clustering, e.g. with the DBSCAN algorithm
[Ester et al., 1996]. Contra: A large number of workers need to be employed to
ensure unbiased labeling and to compensate for unreliable
Purpose: On meso-scale, the space is subdived into smaller works.
neighbourhoods like blocks. The preferred number and
block sizes can be accessed through clustering. Meta information analysis
Pro: The clustering approach allows segmentation to happen Method: Similar to the creativity analysis, we want objective
objectively. criteria that can be adhered to when human feedback is
given for design proposals. The concern of the meta
Contra: A cluster is not neccesarily meant to be a block. information analysis is not how other humans perceive the
The examples in Figure 5 show that parameters in the participant’s design but what the participant thinks of their
cluster algorithm need to be carefully adjusted for the case own proposal. By proposing categories for the main purpose
study area, but even this does not guarantee a satisfactory or idea of the design (e.g. safety, dominance of greenery,
result. accessibility,...), the designs can be labeled, and used for the
application of supervised machine learning to the geometry
Autocorrelation test and purpose.
Method: A spatial autocorrelation, e.g. Global Moran's I Purpose: The idea is to identify characteristics in the
[Moran, 1950] is applied for different objects types. geometry in order to infer the main purpose of the design.
Purpose: The test reveals if object typologies appear Pro: This analysis allows very subjective characteristics of
dispersed or clustered in the area. If they are clustered, the designs to be quantified.
buffer analysis can give some indication of the interrelation
between objects. Contra: The success of this method is not predictable.
Participants may not be clear about what the main idea
behind their designs is.
4 Discussion National Research Foundation (FI 370074016) under its
Campus for Research Excellence and Technological
The qua-kit tool is designed for online participation with Enterprise programme. Moreover, we wish to thank Artem
citizens. The methods in this paper provide options to
Chirkin who has developed the Quick Urban Analysis Kit,
precisely evaluate a large number of designs which are
for giving us access to the toolkit and geometric data from
based on geo-information. Questions may be asked about
the edX MOOC Smart Cities to conduct our research.
whether these presented online design tools offer enough
options for the participants to express their ideas, or if
designers can gain any valueable inputs from these tools. References
The form-based criteria are simple to understand, but how [Banerjee et al., 2010] Abhijit V. Banerjee, Rukmini
can the results be interpreted? Especially the opportunity of Banerji, Esther Duflo, Rachel Glennerster, and Stuti
letting non-experts design provides the chance to access Khemani. "Pitfalls of Participatory Programs: Evidence
knowledge that cannot be expressed semantically (e.g. by from a randomized evaluation in education in India."
surveys). The evaluation should therefore concentrate on American Economic Journal: Economic Policy 2.1: 1-
identifying the design’s semantic meaning how to make 30. 2010
conclusions from the layout to the design idea beyond the
[Bhatt et al., 2012] Mehul Bhatt, Carl Schultz, and Minqian
proposal. This question is about to be answered by findings
Huang. "The shape of empty space: Human-centred
about how space is perceived [Bhatt et al. 2012, 2013,
cognitive foundations in computing for spatial design."
2014]. Assuming that the design process is guided by the
In Visual Languages and Human-Centric Computing
participant’s experience of space (i.e. the perception of
(VL/HCC), 2012 IEEE Symposium on, pp. 33-40. IEEE,
space), it makes sense to analyse the design proposals by
2012.
criteria that are found out to have a significant influence on
the space organisation [Ching, 1979]. Axes and symmetries [Bhatt, et al., 2013] Mehul Bhatt, Carl Schultz, and
are already mentioned and it is for discussion, how and to Christian Freksa. "The ‘Space’ in spatial assistance
which extent the other perception-based criteria should be systems: conception, formalisation and computation."
applied. Representing space in cognition: Interrelations of
behavior, language, and formal models. Series:
5 Conclusion and future work Explorations in Language and Space. 2013: 978-0.
This paper presents a brief summary of evaluation methods [Bhatt, et al., 2014] Mehul Bhatt, Carl Schultz, and
for crowdsourced geometric data on the meso-scale. The Madhura Thosar. "Computing narratives of cognitive
techniques are to be used in participatory urban planning user experience for building design analysis: KR for
studies. The introduced design tool qua-kit allows non- industry scale computer-aided architecture design." 14th
experts to view 3D geometries and make simple edits. We International Conference on the Principles of
use a case study from a neighbourhood near Cape Town to Knowledge Representation and Reasoning. 2014.
show options to evaluate the data on meso-scale. The most [Bonfiglioli, 2003] Angelo Maliki Bonfiglioli. Empowering
comprehensible instruments for analysis are the frequency the poor: Local governance for poverty reduction.
and buffer analysis. Streets and public or private space can United Nations Publications, 2003.
be determined by appropriate distance rules while cluster [Chen, et al., 2007] Po-Chun Chen, J. H. Hays, Seungkyu
algorithms make small neighbourhoods automatically Lee, Minwoo Park, and Yanxi Liu. "A quantitative
identifyable. The overlays of plots from different evaluation of symmetry detection algorithms."
participants can be regarded as a heat map and visualise Pennsylvania State University, Department of Computer
favoured spots for particular objects. Besides these form- Science and Engineering. 2007.
based criteria, we propose to apply supervised machine-
learning to connect the layout of buildings to subjective [Ching, 1979] Francis F.D. Ching. Architecture: Form,
design criteria. The application of these evaluation tools will space, and order. John Wiley & Sons. 1979
testify their usefulness or uselessness. Especially for [Chirkin & Koenig, 2016] Artem Chirkin, and Reinhard
accessing perception-based criteria, it is crucial to have easy König. "Concept of Interactive Machine Learning in
understandable and distinct labels for the proposals. Urban Design Problems." Proceedings of the SEACHI
A further option is to apply unsupervised machine learning 2016 on Smart Cities for Better Living with HCI and UX.
to the geometries to identify different categories of ACM, 2016.
proposals or similarities in the design.
[Ester et al., 1996] Martin Ester, Hans-Peter Kriegel, Jörg
Sander, and Xiaowei Xu. "A density-based algorithm for
discovering clusters in large spatial databases with
Acknowledgments noise." Kdd. Vol. 96. No. 34. 1996.
The research was conducted at the Future Cities Laboratory [Gaventa & Barrett 2010] John Gaventa, and Gregory
at the Singapore-ETH Centre, which was established Barrett. "So what difference does it make? Mapping the
collaboratively between ETH Zurich and Singapore's
outcomes of citizen engagement." IDS Working Papers
2010.347: 01-72. 2010
[Getis & Ord, 1992] Arthur Getis, and J. Keith Ord. "The
analysis of spatial association by use of distance
statistics." Geographical analysis 24.3: 189-206. 1992
[Golooba-Mutebi 2004] Frederick Golooba-Mutebi.
"Reassessing popular participation in Uganda." Public
Administration and Development 24.4: 289-304. 2004
[Herbrich et al., 2007] Ralf Herbric, Tom Minka, and
Thore Graepel. "TrueSkill™: A Bayesian Skill Rating
System." In Advances in Neural Information Processing
Systems, pp. 569-576. 2007.
[Grace et al., 2015] Kazjon Grace, Mary Lou Maher,
Douglas Fisher, and Katherine Brady. "Data-intensive
evaluation of design creativity using novelty, value, and
surprise." International Journal of Design Creativity and
Innovation 3.3-4 (2015): 125-147.
[Lloyd & Bolnick, 2015] Scott Lloyd, and Andy Bolnick.
“Empower: An Integrated Development for the Upgrade
of BT Section, Site C, Khayelithsa – Design Updates”.
ETH Zurich
[Mansbridge 1997] Jane Mansbridge. "Normative theory
and voice and equality." American Political Science
Association, 91.2: 423-425. 1997
[Mitchell, 2005] Andy Mitchell. "The ESRI Guide to GIS
Analysis: Spatial Measurements and Statistics. Vol 2.
Redlands." (2005).
[Moran, 1950] Patrick AP Moran. "Notes on continuous
stochastic phenomena." Biometrika 37.1/2: 17-23. 1950
[Mueller et al., 2017] Johannes Mueller, Hangxin Lu, Artem
Chirkin, Bernhard Klein, and Gerhard Schmitt. “Citizen
Design Science: A strategy for crowd-creative urban
design.” In Cities, 2017 (accepted, not yet published)
[Ord & Getis, 1995] J. Keith Ord, and Arthur Getis. "Local
spatial autocorrelation statistics: distributional issues and
an application." Geographical analysis 27.4: 286-306.
1995
[Pateman, 1970] Carole Pateman. Participation and
democratic theory. Cambridge University Press, 1970.
[Peffers et al., 2007] Ken Peffers, Tuure Tuunanen, Marcus
A. Rothenberger, and Samir Chatterjee. "A design
science research methodology for information systems
research." Journal of management information systems
24.3: 45-77. 2007
[Sanders, 2002] Elizabeth Sanders. "From user-centered to
participatory design approaches." Design and the social
sciences: Making connections. CRC Press, 2002. 1-8.
[Yu & Nickerson 2013] Lixiu Yu, and Jeffrey V. Nickerson.
"An internet-scale idea generation system." ACM
Transactions on Interactive Intelligent Systems (TiiS) 3.1
: 2. 2013