meets Industry: New Challenges and Opportunities at AImageLab

Abstract

The application of Artificial Intelligence (AI) is becoming increasingly ubiquitous in industrial fields, posing some of the major issues and challenges of the future decades. In this respect, the old dichotomy Industry vs. Research appears tired: instead, a strong synergy between these two worlds represents a key aspect toward the most breakthrough incoming innovations. This manuscript presents an example of such a connection: namely, some of the activities carried out at the AImageLab laboratory, one of the most active research lab in Italy and Europe in the fields of Computer Vision and Artificial Intelligence.

Keywords

• successful case of research/industry integration
• repre-

1. Introduction

Artificial Intelligence (AI) has emerged as the next break

through technology and is going to shape the innovation landscape of various fields, in ways not yet fully underthat dates back to the early 1950s (with the definition of the former idealized artificial neurons).

In this respect, the landscape has radically changed

in the last two decades: the major research fields falling under the umbrella term AI – i.e. Deep Learning and

Computer Vision – have crossed the chasm, switching

from frontier academic topics to mature sources of operative and foundational business tools. Indeed, there is a disruptive change of perspective in how industry players look at AI: not just as a cutting-edge technology, but as a as already holds for electricity and water.

In such a context, the boundaries between academic research and the industry field have been blurring: to

nEvelop-O recognized research lab of the Department of Engineering ”Enzo Ferrari” located within the University of Modena and Reggio Emilia. The research covers topics of Computer Vision, Pattern Recognition & Machine Learning,

Human Interaction, and Multimedia applied to optical

images and videos as well as data from diferent sensors.

Since its foundation – which dated more than 25 years ago – AImageLab has joined activities for industrial research with many manufacturers and health companies, recently as a partner of the Artificial Intelligence

Technopole. The projects carried out and the topics cov

DESIGNER ered have achieved a strict connection with the economic fabric of the Emilia-Romagna region, tied principally to manufacturing production. However, in the last years, the AImageLab has been evolving to meet the growing demands of the service industry: more and more startups and established IT service companies are active in producing digital solutions for local industry and for export in Italy and the world.

Therefore, we kindly refer the reader to the next sections, each of which is presenting a cross-section of some of the industry-oriented research activities carried out within the AImageLab laboratory. sensitive information useful for data analytics purposes.

Featuring cutting-edge AI technology, GoatEye is able to deliver powerful video analysis capabilities while ensuring that user privacy is always protected. With GoatEye, customers enjoy all the benefits of AI without worrying about people privacy being compromised. It utilizes advanced algorithms to perceive humans in real-time, 2. GoatEye: the Privacy Camera assessing position, posture, and even action performed with high accuracy. What sets GoatEye apart from the Among the industry-oriented activities of AImageLab, competition is its commitment to user privacy. It uses the one carried out along with the startup GoatAI is with techniques to ensure that personal information is never no doubt the more emblematic. GoatAI has been con- collected or shared without users’ consent. This means ceived within AImageLab and, as such, fully embraces that customers are able to use GoatEye with confidence, a mindset where research and industry are strictly en- knowing that the privacy of people is always respected. twined. GoatAI purpose is to apply the most modern AI techniques for Human Behavior Understanding in health care, fitness, retail, safety and security. 3. AI for the Ceramic Industry

As AI continues to advance, it is crucial that it is used in ways that respect individuals and their personal infor- Image enhancement In the design industry, the aumation. That’s why GoatAI is on a mission to develop tomation of color management and image enhancement innovative solutions that not only provide benefits, but are challenging tasks, with several applications related to also prioritize and safeguard the privacy of those who use the creation and the print of design surfaces. As outlined them. Nowadays and even more in the future, applying in the next paragraphs, automated tools would speed up the wrong data privacy strategy can cost an organization the work of graphic designers in the creation of novel billion in fees and damages. ceramic tiles, with particular focus on the editing and the

Additionally, with the advent of the AI Act, several preparation of images fed to the printing pipeline. measures will be enforced to protect individual rights Indeed, one of the most time-consuming activities of and privacy when processing sensitive data through AI designers is to modify the image so that, once printed, models, intended to ensure that AI systems will be used its quality and level of details are consistent with what in a responsible and ethical manner. The AI Act will customers see on the monitor (or on paper). Such an have a significant impact on those developing AI systems, operation – currently carried out manually through adespecially for Human Behavior Understanding. vanced image editing tools – is not only complex and

That is why GoatAI wants to change the way people time-consuming, but also hard to formalize, as it is based see cameras with GoatEye: the Privacy Camera. GoatEye on the experience and sensitivity of designers. wants to provide a new layer of abstraction by producing The AImageLab team has recently proposed novel AI anonymized video material while maintaining all the non- methods – based on Convolutional Neural Networks – to enhance the visual quality of industrial design surfaces, RefereRnceeferenTcrey-OnTry-OnTry-OnTry-On ModelModGealrmeGntasrmenRtsesult Result

ReferenRceeferenTcrey-OnTry-OnTry-OnTry-On ModelModGealrmeGntasrmenRtsesult Result

ReferenRceeferenTcrey-OnTry-OnTry-OnTry-On ModelModGealrmeGntasrmenRtsesult Result R

F

F

4. Generative AI for Fashion

With the advent of e-commerce, the variety and avail

according to specific properties dictated by color profiles ability of online garments have become increasingly (for a visual of the obtained results, see Fig. 2). These overwhelming for the final user. Consequently, userapproaches were developed in collaboration with Digital oriented services and applications such as virtual tryDesign Srl, a leading design company located in Fiorano on [ 2, 3, 4 ], customer-to-shop garment retrieval, and Modenese (Modena) specialized in creating high quality vision-and-language interactions [ 5, 6 ] are increasingly design surfaces for diferent applications, in particular important for online shopping, helping fashion compaceramic tile printing. nies to tailor the e-commerce experience and maximize Generative design In order to provide further support customer satisfaction. to designers, a team of researchers at AImageLab devel- In this context, the task of image-based virtual tryoped a new approach to synthesize novel ceramic tiles. on aims at synthesizing an image of a reference person By generating new patterns and textures, the approach wearing a given try-on garment while preserving the allows to generate the diferent tile faces that make up person’s intrinsic information such as body shape and an entire ceramic surface. Such a case study is a practical pose. Existing datasets for the task are either propriexample of how AI technologies can support the creation etary and therefore not publicly available, or feature a and evolution of new products, which represent the main very limited number of images which are usually low goals of generative design. The latter is an important mile- resolution [ 2 ]. To tackle these problems, AImageLab stone in industrial production, especially for processes researchers recently presented Dress Code [ 4 ]: a new involving an important creative component. dataset of high-resolution images (1024 × 768) contain

To do so, the researchers have leveraged Generative ing more than 50k image pairs of try-on garments and Adversarial Networks (GANs): these models, by virtue corresponding catalog images where each item is worn of the extremely high-quality and high-fidelity images by a model. Diferently from existing public datasets, provided, have seen a significant growing interest for a which contain only upper-body clothes, Dress Code feavariety of applications, ranging from super-resolution tures upper-body, lower-body, and full-body clothes, as to image-to-image translation, from image inpainting to well as full-body images of human models. the field of image synthesis [ 1 ]. In addition to the dataset, the researchers have also in

The idea of supporting designers and engineers by troduced a novel image-based virtual try-on architecture using a generative AI-based system opens new ways in that can anchor the given garment to the right portion the so-called Industry 5.0 paradigm, where humans and of the body. As a consequence, it is possible to perform AI-based engines should collaborate for the industry of a “complete” try-on over a given person by selecting tomorrow. However, the proposed system has already diferent garments (Fig 4). been patented and is currently in use in the ceramic tile industry, whose designers, who supervise both the training 5. AI for Interior Design and the generation process, are exploiting with considerable satisfaction the flexibility and creative capabilities of our generative model.

An additional research and industrial innovation field on which AImageLab works is that of empowering aug

DEPDLEOPYLMOYEMNTENTONDOEDE ADAADPATPT

DEPDLEOPYLMOYEMNTENTONDOEDE

DODNOONTOT TRATNRASMNSITMIT

ADAADPATPT TRATIRNAININGINNGONDOEDE

TRATIRNAININGINNGONDOEDE

Autonomous Learning

DEPDLEOPYLMOYEMNTENTONDOEDE

CLCTLRTARIANIN MM E E R R G G E E

SMSAMLLALL

WOWRKOIRNKGING TRATIRNAININGINNGONDOEDE MEMEOMRYORY TRATIRNAININGING DATDAASTEATSET

TRATIRNAININGING

DATDAASTEATSET RE-RTER-ATIRNAMINOMDOELDEL

RE-RTER-ATIRNAMINOMDOELDEL

RE-RTER-ATIRNAMINOMDOELDEL mented and virtual reality application for interior design. 6. Continual Learning In this context, a fundamental task which is addressed by our research activities is that of automatically parsing Human intelligence is distinguished from Artificial Neuand understanding pictures of indoor scenes. The goal of ral Networks (ANNs) by the former’s ability to acquire the task is that of providing detailed information about knowledge incrementally and retain long-term memthe objects in a scene, the layout of the space, and how ory. Conversely, ANNs sufer sudden performance deobjects interact with each other. terioration, termed catastrophic forgetting, in response

One of the core subtasks which need to be solved in to changes in training data distribution [9]. Continual this context is that of performing a semantic segmen- Learning (CL) [10] is a rapidly growing area of machine tation over the input image. The research on semantic learning which focuses on bridging this gap by devising segmentation models has focused on the introduction of technical solutions that allow AI systems to overcome either fully convolutional networks or Vision Transform- forgetting. In recent years, the AImageLab team has ers which leverage upsampling operations to increase the actively contributed to CL research, focusing especially output resolution. Although this architectural choice is on the development of novel CL methods belonging to necessary to encode contextual information and deal with performant rehearsal-based category [11, 12]. objects at large scales, it also leads to feature smoothing While apparently abstract, this emerging discipline has across object boundaries, and thus to a degraded quality significant implications for MLOps practices. Today, a in the final result. model’s life cycle typically begins with training on a ded

With the aim of improving the quality of semantic icated training node (e.g., a high-compute server), with segmentation in indoor scenarios, especially in boundary the result being frozen and used for inference on a separegions, we have investigated the design of boundary- rate deployment node (a machine that operates on the aware losses for the optimization of semantic segmen- edge, close to the source of operational data). Herein, we tation architectures, both in CNN-based and ViT-based examine three scenarios (see also Fig. 5) that demonstrate architectures. We started from two recently proposed the potential of CL algorithms to alter this paradigm. loss functions, namely the Boundary loss and the Ac- Drift Prevention. In-deployment model re-training is tive Boundary loss, and designed two improved versions required when significant diferences occur between inthat can significantly increase the overall quality of the ference and training data distributions. As this process insegmentation at boundary level [ 7 ]. volves recording new data, transmitting it to the training

Our semantic segmentation solutions have been ap- node and learning a new model from scratch, the model’s plied to the recognition of walls and floors in indoor response to new data may be delayed or unreliable. By images, and to the recognition of particular surfaces (e.g. applying a CL algorithm on the deployment node, the counter-tops or steps). In addition to that, we also work- model can adapt to incoming data gracefully controlling ing towards the recognition of long-tail objects on the its performance degradation until an update is available scene, and to the generation of natural language descrip- (Fig. 5a). Recent advancements permit this procedure tions from indoor images [ 8 ]. even with limited supervision on new data [13]. Decoupled Adaptation. Due to their physical separation, it is likely that some novel data-points recorded on the deployment might not be transmitted back to the training node (e.g., due to security constraints or technical limitations). In such a scenario (Fig. 5b), CL allows the model to fit these additional data-points without them leaving the node. When a re-trained model (unaware of secure data) becomes available, the CL learner cannot be trivially replaced, but must follow an adequate procedure to allow knowledge merging [14].

Autonomous Learning. As CL methods improve, it

becomes increasingly feasible to let them operate over very long periods of time. Ideally, this will allow the in-deployment model to fully adapt to upcoming data dynamically by only leveraging a small working memory, removing the need of re-training.

While such a degree of resilience has not yet been achieved, AImageLab has developed a strong focus on removing bias from continual learners, allowing them to operate for longer timespans [15, 16]. In the context of Industry 4.0, experts agree that the cooperation between humans and intelligent robots [17], rather than the complete removal of human operators, will be the best solution for the advancement in manufacturing. Therefore, safe interaction is a crucial element, especially regarding social and physical coordination between coworkers. In this scenario, the ability to predict the 3D pose of the agents in a collaborative environment is an enabling technology for real-world safety applications, e.g. collision avoidance and anomaly detection.

Thinking about a surveillance system installed in an industrial setting, vision-based pose estimation approaches can be exploited to retrieve the 3D pose of an articulated object with respect to the camera viewpoint. However, a model that jointly predicts human and robot poses is still an open problem in the research community. Thus, the current approach is to consider humans and robots as separate agents. Since the human pose task has been extensively investigated in the computer vision community [18], the focus of our work is on the robotic scenario.

7. Theft Recognition

Robot Pose Estimation (RPE). In robotics, the common approach to estimate the absolute pose of a robot The evolution of self-checkout methods in retail stores with respect to the camera is the Hand-Eye Calibration. presents new challenges that computer vision and deep However, with recent advances in deep learning, many learning technologies are able to address. As there no works have been proposed to estimate the camera-tocashier are involved in scanning the items, it is important robot pose using CNNs. Recently, Lee et al. [19] demonto develop a system able to monitor the customer actions strate that learning-based approaches could replace clasand detect anomalous behaviours such as shoplifting at- sic marker-based calibration also for standard manipulatempts or even good-faith mistakes. Modern embedded tors (e.g. Rethink Baxter and Franka Emika Panda). Using systems allow such solutions to work on the edge, pro- synthetic data for training, they feed RGB images into viding useful feedback to on-site staf. an encoder-decoder network that predicts the 2D loca

To achieve this goal, researchers of AImageLab devel- tion of the robot joints. Assuming the camera intrinsics oped an approach to track the movement of customers and the configuration of the joint angles are known, the at the self-checkout station and used this data to clas- camera-to-robot transform is computed via PnP. sify their actions, distinguishing between malevolent and Going beyond learning-based methods, recent works benevolent behaviours. This system can be divided into propose approaches based on rendering. Labbe et al. [20] three main parts: (a) human behaviour detector, (b) ac- pave the way to this field of research presenting the tion classifier, ( c) retrieval network. ifrst method for robot pose estimation based on the ren

The role of the first part of the system is to extract from der&compare paradigm. This optimization algorithm itvisual data useful information that are able to describe the eratively refines an initial robot state defined as the joint type of actions of the person involved in the self-checkout. angles configuration and the pose of an anchor part with Once a good descriptor of the events has been computed, respect to the camera. it is fed to a more complex architecture, able to link each scanning action to a set of pre-defined behaviors. Finally, SPDH Pose Representation. In contrast with the disthe latter module attempts at bridging the gap between cussed works, our work aims to regress the camera-tothe camera and the traditional laser bar code scanner: robot pose using a novel heatmap-based representation briefly, it is carried out by learning a neural network of the 3D pose. In this way, we can exploit any CNNspecialized on a retrieval task, i.e., identifying objects and based architecture trained to predict heatmaps and adapt ascertaining whether its visual appearance corresponds it to predict our pose representation. Moreover, since with the reading of its bar code. deep learning applied to robotics requires a lot of data

The system has shown promising results in identify- and it is not feasible to record real data covering all the ing diferent types of theft, demonstrating that research possible scenarios, we exploited depth images to close results can earn a relevant place in the retail sector. the domain gap between the synthetic domain of the sim

References

ulator and the real world. An overview of the proposed memory transformer for image captioning, in: CVPR, approach is depicted in Figure 6 . 2020 . Regarding the pose representation , we propose the [9] M. McCloskey , N. J. Cohen , Catastrophic interference in Semi-Perspective Decoupled Heatmaps (SPDH) [21] that connectionist networks: The sequential learning probrely on projections of the 3D space in two 2D spaces: lem, in: Psychology of learning and motivation, voland space. The space, i.e.the camera image plane , ume 24 , 1989 , pp. 109 - 165 .

and the space, composed by quantized Z-values and [10] M. De Lange , R. Aljundi , M. Masana , S. Parisot , X. Jia, the u dimension.A pose estimation algorithm is trained A . Leonardis , G. Slabaugh, T. Tuytelaars , A Continual to generate two heatmaps for each joint, one for each Learning Survey: Defying Forgetting in Classification space. The setting is a Sim2Real scenario, so the method Tasks , IEEE Trans. PAMI 44 ( 2022 ) 3366 - 3385 .

is trained on synthetic data and tested directly on real [11] A. Robins, Catastrophic forgetting, rehearsal and pseudata. Thus, we presented SimBa, a dataset containing dorehearsal , Connection Science 7 ( 1995 ) 123 - 146 .

RGB-D synthetic and real sequences with a Baxter robot [12] P. Buzzega , M. Boschini , A. Porrello , S. Calderara , Reperforming pick-n-place movements. We proved that thinking experience replay: a bag of tricks for continual using depth maps as input reduces the domain gap ob- learning , in: ICPR, 2021 .

taining promising results for the RPE task . [13] M. Boschini , P. Buzzega , L. Bonicelli , A. Porrello , S. Calderara , Continual semi-supervised learning through contrastive interpolation consistency , PRL 162 References ( 2022 ) 9 - 14 . [14] M. Boschini , L. Bonicelli , A. Porrello , G. Bellitto, M. Pen-

[1] P. Shamsolmoali , M. Zareapoor , E. Granger, H. Zhou, nisi, S. Palazzo, C. Spampinato , S. Calderara , Transfer R. Wang , M. E. Celebi , J. Yang , Image synthesis with without forgetting , in: ECCV , 2022 . adversarial networks: A comprehensive survey and case [15] M. Boschini , L. Bonicelli , P. Buzzega , A . Porrello, studies, Information Fusion 72 ( 2021 ) 126 - 146 . S. Calderara, Class-incremental continual learning into

[2] X. Han , Z. Wu , Z. Wu , R. Yu , L. S. Davis , Viton: An the extended der-verse , IEEE Trans. PAMI ( 2022 ) 1 - 16 . image-based virtual try-on network , in: CVPR , 2018 . [16] L. Bonicelli , M. Boschini , A. Porrello , C. Spampinato,

[3] M. Fincato , M. Cornia , F. Landi , F. Cesari , R. Cucchiara , S. Calderara , On the Efectiveness of Lipschitz-Driven Transform, Warp, and Dress: A New Transformation- Rehearsal in Continual Learning , in: NeurIPS, 2022 . guided Model for Virtual Try-on , ACM TOMM 18 ( 2022 ) [17] A. Weiss , R. Buchner , M. Tscheligi , H. Fischer , Exploring 1-24. human-robot cooperation possibilities for semiconductor

[4] D. Morelli , M. Fincato , M. Cornia , F. Landi , F. Cesari , manufacturing, in: CTS, 2011 . R. Cucchiara, Dress Code: High-Resolution Multi- [18] W. Liu , Q. Bao , Y. Sun , T. Mei, Recent advances of monocCategory Virtual Try-On, in: ECCV , 2022 . ular 2d and 3d human pose estimation: a deep learning

[5] M. Stefanini , M. Cornia , L. Baraldi , S. Cascianelli , G. Fi- perspective, ACM Computing Surveys 55 ( 2022 ) 1 - 41 . ameni, R. Cucchiara, From Show to Tell: A Survey on [19] T. E. Lee , J. Tremblay , T. To, J. Cheng, T. Mosier, O. KroeDeep Learning-based Image Captioning , IEEE Trans. mer , D. Fox, S. Birchfield , Camera- to- robot pose estimaPAMI 45 ( 2022 ) 539 - 559 . tion from a single image , in: ICRA , 2020 .

[6] N. Moratelli , M. Barraco , D. Morelli , M. Cornia , L. Baraldi, [20] Y. Labbé , J. Carpentier , M. Aubry , J. Sivic , Single-view R. Cucchiara , Fashion-Oriented Image Captioning with robot pose and joint angle estimation via render & comExternal Knowledge Retrieval and Fully Attentive Gates, pare , in: CVPR, 2021 . Sensors 23 ( 2023 ) 1286 . [21] A. Simoni , S. Pini , G. Borghi, R. Vezzani , Semi-

[7] P. Bruno , R. Amoroso , M. Cornia , S. Cascianelli, perspective decoupled heatmaps for 3d robot pose estimaL . Baraldi, R. Cucchiara , Investigating bidimensional tion from depth maps , IEEE RA-L 7 ( 2022 ) 11569 - 11576 . downsampling in vision transformer models , in: Image Analysis and Processing-ICIAP 2022 , 2022 .

[8] M. Cornia , M. Stefanini , L. Baraldi , R. Cucchiara , Meshed-