Police photo lineups are an important part of criminal proceedings, where the task is to identify the perpetrator among photos of other persons (fillers). In order to prevent major errors in criminal proceedings, lineups should be unbiased (i.e. the suspect and fillers should share similar appearance characteristics). Capability to assemble unbiased lineups is often hindered by the lack of efective methods to explore the database of fillers (i.e. good fillers are hard to be found), but also by the insuficient size of the database itself (i.e. no good fillers exist). In this demo, we present LiGAN application aiming on onthe-fly recommendation of artificial fillers for police photo lineups. We consider this to be a highly novel recommending task, where items can be generated with arbitrary density and arbitrary precision to the (estimated) user's needs. LiGAN utilizes StyleGAN2 architecture to generate images, identity-preserving autoencoder for suspect seeding and optional model finetuning for individual lineups. It recommends fillers based on the semantic proximity to the suspect, or as an interpolation between suspect and filler images. As such, LiGAN aims to contribute towards both the fillers existence and the fillers ifndability problems.
ifgure size, etc.). Examples of biased and unbiased lineups are depicted on Figure 1.
Eyewitness identification of suspects is an important part In the current police praxis, photo lineups are still of criminal proceedings. It often leads to the prosecu- mostly constructed manually via browsing through the tion and eventual conviction of crime perpetrators, but (limited) database of available fillers. This brings two it is also prone to the human errors [1]. There are doc- problems. First, manual browsing is rather tedious, so eiumented cases, where incorrect eyewitness testimony ther the construction of unbiased lineups takes excessive led to false accusation and conviction of innocent sus- amount of time, or (partially) biased lineups are produced. pects and therefore, error-proof methods for eyewitness The second problem comes with the features of the fillers identification is an intensively studied research subject. database, which (mainly due to various legal constraints)
One of the recommended approaches is identification often contains only several thousands of photos. In addivia photo lineup. In this case, a witness receives a selec- tion to that, various appearance characteristics are often tion of several photos (usually four to eight), where one not represented evenly, so suitable fillers may be undepicts the suspect and others depict additional persons available for some suspects and constructing an unbiased (so-called fillers ), that are known not to be on the crime lineup is not possible. [3] scene. The idea behind photo lineup is that only a rea- In our previous work, we focused on the first problem sonably certain witness can identify the perpetrator if and considered it from the perspective of content-based similar fillers are present [ 2]. As such, the requirement recommender systems (RS) [4]. We utilized the semantic for the suspect-fillers similarity is crucial. In criminal psy- similarity of photos induced by a pre-trained convoluchology literature, the suspect-fillers similarity problem tional network and recommended fillers similar to the is often formulated as (un)biased lineups: the lineup is suspect as well as other members of so-far constructed biased if the suspect’s photo poses considerably diferent lineup. This approach led to a reduction of task’s temappearance characteristics. Those can be both features poral complexity, but we did not tackle the database size of the person (age, skin color, face shape, haircut, etc.), problem. but also features of the photography (background, angle, In this demo paper we present LiGAN, an experimental application based on Generative Adversarial Networks (GANs). LiGAN provides on-the-fly generation and recommendation of artificial fillers for police photo lineups.
With this approach, we aim to contribute towards solving both the problem of database size as well as the problem of fillers discovery. Nonetheless, recommending artificial objects, which (in theory) can be constructed with an
1For the sake of feature disentanglement and generator stability, StyleGAN2 uses a mapping network to transform the seed vector ∈ into a style vector ∈ (512 dimensions), that is supplied to all layers of the StyleGAN architecture. unlimited density and unlimited proximity to the user’s generator to construct artificial images w.r.t. the supneeds brings interesting theoretical challenges as well. plied style. Recommending component is responsible In the next section we describe LiGAN application, while for the modifications of style vectors, so suitable fillers we briefly present some of the theoretical challenges in are provided to the user. LiGAN features a REST-like the discussion section. webserver that encapsulates generator components and
While the proposed application domain (recommend- tracks individual user sessions (e.g. for the sake of model ing artificial fillers for photo lineups) is brand new, there ifne-tuning). Due to space limitations we only briefly are some related approaches in other domains. GANs describe the main principles behind LiGAN, details can themselves are frequently present in RS literature, but be found in [10]. rarely used for image synthesis [5]. One notable exception is the fashion domain, where GANs are often used 2.1. On-Demand Fillers Generation to construct artificial clothing [ 6, 7, 8, 9]. An underlying motivation of these approaches is to help designers to For image generation, we utilized a state-of-the-art Styleifnd new styles of products that users might like although GAN2 [11] architecture. StyleGAN2 training is conducted they do not exist yet. as a zero sum game between two model components: The
Kang et al.[6] use conditional GAN where the gen- generator receives a random seed vector ∈ (512 dierator receives a user and an item category and then mensions)1 and aims to generate images that fits into the produces items that are most consistent with the given training dataset. The discriminator aims to distinguish category as well as user preferences. The main diference between real and generated images. We trained the model to our approach is the usage of conditional GAN (i.e. gen- from scratch based on the dataset of missing and wanted erator is directly conditioned on product category) while persons from two Central European countries. After the we do not utilize conditioning, but instead employ an pre-processing steps, the dataset contained over 90000 identity-preserving encoder to reconstruct the suspect passport-style photos with the resolution of 256 × 256 image. Analogical diferences can be found also between pixels. our approach and the work of Yang et al.[7], Shih et al.[8] Instead of constructing a simple dataset of generated and Kumar et al. [9] who focus on generating compatible ifgures, we decided to embrace the opportunity to generfashion items. ate fillers on-demand based on the suspect’s photography. This approach provide more versatility than just selecting from a fixed dataset (e.g. it allows to fine-tune the model 2. LiGAN Application for particular suspect or react on user’s feedback). We relied on StyleGAN’s similarity-preservation feature, i.e. that similar input vectors produce similar output images.
In order to exploit this feature, we trained an identitypreserving encoder that aims to minimize distances between and ¯ , where ¯ = (()).
web application (see Figure 2. It allows to upload suspect’s photo, select recommended fillers or provide additional feedback and iteratively construct the lineup.
Main components of LiGAN’s backend are StyleGAN2 generator , identity-preserving encoder and recommending component . The encoder transforms images into corresponding style vectors that are utilized by the
Several encoder architectures and distance metrics were already selected fillers as well. This could help to get rid considered, but training an encoder to the original input of the "centering" efect, i.e. that the suspect is in an imagvector space or the style vector space was not suc- inary center of all fillers’s appearance characteristics and cessful. Resulting ¯ s were either of insuficient quality therefore easier to be identified. In both cases, the necesor too diferent from the original (see Figure 3 left). sary levels of diversity and filler-based recommendations We suspected that too much information is lost with are not known upfront and should be assessed online the reduction into or and therefore, we extended based on user’s feedback. Recommending component dethe encoder’s output space to allow supplying diferent scribed in the next section is responsible for appropriate style vectors for each StyleGAN’s layers similarly as in selection of filler’s style vectors. [12]. I.e., the encoder produces a matrix of style vectors The crucial part of LiGAN design is the identity pre+ ∈ + with 12 × 512 dimensions. This extension serving encoder. The quality of suspect’s reconstruction considerably improved the identity preservation. from learned style matrix directly afects the ability to
Once the + mapping is obtained, similar fillers can propose relevant fillers. However, despite our efort, the in theory be generated by small variations of the vector. results were sometimes not satisfactory (see Figure 3 In our early attempts, we implemented these variations right). In order to cope with this problem, we allowed as a random sampling from a hyperball around the partic- to fine-tune the encoder and the generator for ular + vector. Nonetheless, sampling from + space the particular suspect’s image. Such fine-tuning is rather often provided poor results (see Figure 3 middle). There- fragile as if suficient steps are performed, it would evenfore, we prepend a PCA dimensionality reduction before tually cause a mode collapse. Therefore, the time allowed the sampling phase. PCA was trained w.r.t. + vectors for fine-tuning is limited and user is allowed to modify it corresponding to the sample of 10000 randomly gener- if necessary. Nonetheless, in several cases, fine-tuning ated images and after the hyperparameter tuning, the subjectively improved the results of identity-preserving output dimensionality was set to 256. This helps to focus transformation as can be seen on Figure 3 right-bottom. the sampling procedure towards images that resemble Overall, the fillers generation procedure is as follows: real persons better. upon the receipt of suspect’s photo a corresponding
In theory, we can generate persons with infinitely close reduced style vector is generated = (()). style vectors that would be rather indistinguishable from This vector, together with vectors of already sesuspect. However, this is not a desired output as it would lected lineup members is supplied to the recommendrender the lineup identification impossible. Instead, cer- ing component that outputs vectors of recommended tain level of noise needs to be introduced in the fillers ifllers ,1, ..., ,. Then, all fillers’ vectors are generation procedure. Also, newly generated fillers may transformed back to the + space via inverse PCA and be sampled from the space around the style vectors of StyleGAN2’s generator is used to generate individual images: , = ( − 1(,)). These images are then presented to the user.
User has several feedback options (asking for less similar, more like this or more similar recommendations, triggering interpolation between a filler and the suspect or initiating a fine-tuning ) that may modify the internal model of the LiGAN and trigger a new recommendation process.
with recommenders 0, 1 and 2, each of them receiving equal initial consumption statistics (i.e. 0 and 0 parameters from Eq. 1). For each recommended position and each eligible recommender , a random value from a beta distribution of its convergence statistics is sampled and the recommender with the highest value is selected to fill this position. Specifically, = ( 0 + , 0 + ℎ − ) (1) 2.2. Fillers Recommendation where denotes the sum of positive feedback (e.g.
selecting recommended filler for the lineup) received by Once the image generator and the identity preserving recommender and ℎ denotes the total volume encoder are established, the important question is how of recommendations given by . to select fillers (or their corresponding style vectors). With this solution alone, recommendations can be We assume that two key concepts should be considered tuned over time to have a desired distance from the susduring the selection process. First, fillers should maintain pect, but only within a fixed pre-defined range. This is certain level of diversity from the suspect, but the user impractical as estimating such range is very tricky and it should have some means to tune this diversity. Second, may also difer for various areas of the style vector space. ifllers should be mainly generated based on the suspect, Therefore, we provide users users with explicit options but already selected fillers may play some role in the to increase / decrease the distance between the suspect recommendation process as well. and recommended fillers (i.e., "More similar" and "Less
As the expected level of diversity is unknown up front, similar" buttons). Each time the button is pressed, the recwe decided to learn it on-line based on the Thompson ommender selection process is performed as usual, but sampling multi-armed bandits [14]. Specifically, we con- the actual recommender that provides recommendation struct a series of recommenders ∈ ℛ. Each recom- is shifted in the direction of expressed user desire. For exmender , upon receiving a source vector , sam- ample, if the user clicked on "Less similar" button and is ples a filler from a hollow hyperball around it, i.e. from a selected via Thompson sampling to fill the position, + space bounded by two spheres, with the center at recommender is used instead. If user hits the "Less simiand diameters − 1 and . I-th diameter is constructed lar" button again, +2 is used and so on. Furthermore, as = * , where is an initial diameter and if user selects a filler supplied by +2 recommender, it is a steepness hyperparameter governing how quickly is added to the pool of initially eligible recommenders should we converge towards more/less similar recom- with appropriate consumption statistics, so the next time mendations. As such, the previous recommender to the the suspect is submitted, more appropriate initial recomcurrent one, − 1, generates strictly more similar fillers, mendations are given. The hyperparameter governs while the next recommender, +1, generates strictly less the steepness of similarity traversal steps. We set = 3, similar fillers than the current one. In the current version i.e., in the initial case the adjacent triple of recommenders of LiGAN, we kept = 1.2 and leave experiments with would be utilized. In addition to the selection-based posthe steepness factor on future work. itive feedback, we also consider that simple asking for
We follow the same approach to generate the final list more / less similar results is a form of (weaker) positive of recommendations as proposed by Broden et al. [14] feedback. Therefore, all recommenders involved in the with one important distinction: the list of eligible rec- generation of the next list of recommendations receive a ommenders changes based on user feedback. We start small volume of positive feedback. As such, convergence towards proper diversity thresholds is secured even if no ciently represented in the training data. Legal challenges ifller is selected and the user, e.g., starts to fine-tune the (although interesting) are out of scope of our research. model. However, we believe that before such questions may be
Next, for each recommended position, we select at even risen, the technical feasibility have to be suficiently random with a fixed probability, whether the suspect demonstrated. Nonetheless, even before legal issues are (p=0.7) or one of the fillers (p=0.3) should be utilized solved, artificial fillers may prove beneficial e.g. for poas a center of the sampling process. We opted for this lice training (no need to consider privacy issues as with simple procedure mainly to gain some initial feedback real person’s photos). on both approaches. For the future work, we would like Fillers recommendation in LiGAN is rather basic at the to focus on modelling a joint probability based on both moment. We approached the problem as session-based suspect and fillers similarly as [4] does for a fixed set of recommendation with on-line learning and a background candidates. knowledge represented by the person’s style vectors. Ac
Finally, LiGAN also allows users to manually decrease cording to the common nomenclature, suspect’s and sethe desired diversity between the suspect and a selected lected filler’s photos play the role of items "visited" in the ifller through image interpolations. In this case, two pho- current session. From this perspective, asking for more / tos (, ) are supplied and a linear interpolation less similar recommendations as well as interpolations between the corresponding and vectors is can be considered as a special cases of recommendation calculated. LiGAN then displays fillers corresponding to critiquing. the individual interpolated points. Due to the reasonable Furthermore, we would like to note that once there level of feature disentanglement in StyleGAN architec- is an unbound volume of candidates for recommendation, ture, interpolated fillers empirically provide a smooth many commonly utilized recommending approaches have transition of one person into another. to be re-considered before application. For instance, recommending items most similar to the user’s profile (i.e. 3. Discussion and Outlook suspect’s photo) does not seem sensible as we can easily generate near-duplicates with no practical applicability.
The need for diversity, novelty, coverage or fairness of representation greatly increased, but many paradigms used to incorporate these metrics were tailored for a ifnite set of items [ 15, 16, 17]. Sampling from the recommendable objects and subsequent post-processing is a plausible first approach, but it may be more interesting to incorporate e.g. diversity or fairness preservation into the sampling process itself.
In the current version of LiGAN we only tackled this problem via on-line learning of the sampling radius, but we believe that re-formulating e.g. per-list diversity preservation into a continuous probability distribution problem may be an interesting future work. Also, several directions of long-term user preference may be explored as well, e.g. learning the personalized sampling radius for individual style dimensions, or focusing on an interplay between the suspect-based and fillers-based distances.
towards both the practical problem of unbiased lineups construction, but also provide foundations for a novel subarea of RS: recommending artificially generated objects.
Artificial fillers has the potential to improve the lineup construction process if the following conditions are met: 1) we can generate images of suficient quality, 2) potential witnesses cannot reliably distinguish between real and artificial photos, 3) we can pre-select suitable filler candidates automatically and 4) legal conditions has to be met. Although additional improvements are necessary, we believe that LiGAN shows that first three conditions are feasible. The first condition is mainly the question of computational power and data availability as shown in other StyleGAN2 applications [11]. We consider the current LiGAN’s generator as suficient for a showcase, but plan to expand both image’s resolution as well as train data diversity in the future.
For the second condition, we conducted a user study with 80 participants to evaluate their capability to distinguish between real and generated photos. Participants received a list of photos both real and generated and their task was to select the generated ones. Average precision per user was 0.65, while average recall was 0.39, so users performed slightly better than random guessing, which can be considered as a success.
Ability to recommend reasonable fillers should be further tested, but first empirical results seems promising as long as suspect’s appearance characteristics are sufi
ence Foundation project GACR-19-22071Y and by Charles University grant SVV-260588. Source codes can be obtained from https://gitlab.mf.cuni.cz/dokoupipa/ligan-thesis/ -/tree/recsys. LiGAN application can be accessed from http://gpulab.ms.mf.cuni.cz:7022/.
CVPR42600.2020.00813. [12] E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, [1] J. Mansour, J. Beaudry, N. Kalmet, M. I. Bertrand, Y. Azar, S. Shapiro, D. Cohen-Or, Encoding in style: R. C. L. Lindsay, Evaluating lineup fairness: a stylegan encoder for image-to-image translation, Variations across methods and measures, Law CoRR abs/2008.00951 (2020). URL: https://arxiv.org/ and Human Behavior 41 (2016). doi:10.1037/ abs/2008.00951. arXiv:2008.00951. lhb0000203. [13] C. E. Thomaz, G. A. Giraldi, A new ranking method [2] S. Clark, M. Erickson, J. Breneman, Probative value for principal components analysis and its applicaof absolute and relative judgments in eyewitness tion to face image analysis, Image and Vision Comidentification, Law and human behavior 35 (2010) puting 28 (2010) 902–913. doi:https://doi.org/ 364–80. doi:10.1007/s10979-010-9245-1. 10.1016/j.imavis.2009.11.005. [3] A. N. Bergold, P. S. Heaton, Does filler database [14] B. Brodén, M. Hammar, B. J. Nilsson, D. Paraschakis, size influence identification accuracy?, Law and Ensemble recommendations via thompson samHuman Behavior 42 (2018) 227–243. pling: An experimental study within e-commerce, [4] L. Peška, H. Trojanová, Lineit: Similarity search in: IUI ’18, ACM, 2018, pp. 19–29. and recommendation tool for photo lineup assem- [15] J. Carbonell, J. Goldstein, The use of mmr, diversitybling, in: Database and Expert Systems Appli- based reranking for reordering documents and procations, Springer International Publishing, Cham, ducing summaries, in: SIGIR ’98, ACM, New York, 2019, pp. 199–209. NY, USA, 1998, pp. 335–336. [5] Y. Deldjoo, T. D. Noia, F. A. Merra, A survey [16] L. Malecek, L. Peska, Fairness-preserving group on adversarial recommender systems: From at- recommendations with user weighting, in: Adtack/defense strategies to generative adversarial junct Proceedings of the 29th ACM Conference networks, ACM Comput. Surv. 54 (2021) 35:1–35:38. on User Modeling, Adaptation and Personalization, doi:10.1145/3439729. UMAP ’21, Association for Computing Machinery, [6] W. Kang, C. Fang, Z. Wang, J. J. McAuley, Visually- New York, NY, USA, 2021, p. 4–9. doi:10.1145/ aware fashion recommendation and design with 3450614.3461679. generative image models, in: 2017 IEEE Interna- [17] H. Steck, Calibrated recommendations, in: RecSys tional Conference on Data Mining, ICDM 2017, ’18, ACM, 2018, pp. 154–162.
New Orleans, LA, USA, November 18-21, 2017, IEEE Computer Society, 2017, pp. 207–216. doi:10.
1109/ICDM.2017.30. [7] Z. Yang, Z. Su, Y. Yang, G. Lin, From recommendation to generation: A novel fashion clothing advising framework, in: 2018 7th International Conference on Digital Home (ICDH), 2018, pp. 180–186.
doi:10.1109/ICDH.2018.00040. [8] Y. Shih, K. Chang, H. Lin, M. Sun, Compatibility family learning for item recommendation and generation, in: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), AAAI Press, 2018, pp. 2403–2410. [9] S. Kumar, M. D. Gupta, c+gan: Complementary fashion item recommendation, CoRR abs/1906.05596 (2019). URL: http://arxiv.org/abs/ 1906.05596. arXiv:1906.05596. [10] P. Dokoupil, Generating synthetic data for an assembly of police lineups, Master’s thesis, Charles University, 2021. URL: https://dspace.cuni.
cz/handle/20.500.11956/127394. [11] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, T. Aila, Analyzing and improving the image quality of stylegan, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 1319, 2020, IEEE, 2020, pp. 8107–8116. doi:10.1109/