In the era of Big Data, manufacturing companies are overwhelmed by a lot of disorganized information: the large amount of digital content that is increasingly available in the manufacturing process makes the retrieval of accurate information a critical issue. In this context, and thanks also to the Industry 4.0 campaign, the Italian manufacturing industries have made a lot of e ort to ameliorate their knowledge management system using the most recent technologies, like big data analysis and machine learning methods. This paper presents the on-going work done within the ADA project, with special emphasis on the speci c image analysis work carried out to extract information from images contained in the so di erent document of the manufacturing companies, partners of the project.
Manufacturing companies, which produce complex products and manage large plants, generate a consistent ow of data and information throughout the company processes, from acquisition to production and maintenance of the products themselves. In this amount of data, a signi cant part consists of texts, graphics, and images obtained as a transposition of the know-how of human personnel.
Collecting and retrieving all this data and information quickly and easily is vital for speeding up internal business activities. For example, during the design phases of a new product, it is useful to be able to identify, in past projects, speci cations, data and information contained in lessons learned, in risk analysis, to carry out more reliable and innovative design activities. In case of plants maintenance, it is invaluable for operators to have immediate access to the information necessary to carry out their work quickly and e ectively.
Copyright c 2019 for the individual papers by the papers authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. SEBD 2019, June 16-19, 2019, Castiglione della Pescaia, Italy.
The needs of the manufacturing companies described above, however, clash with the complexity of knowledge management, due both to the large quantity and to the heterogeneity of the data and documents to be processed: the technological tools currently available on the market are not able to rise this challenge and e ectively meet these needs. In this context, therefore, the main target of the ADA project is to design and develop a platform based on big data analytics systems that allows for the acquisition, organization, and automatic retrieval of information from technical texts and images in the di erent phases of acquisition, design & development, testing, installation and maintenance of products.
In this paper, we illustrate the work carried out in the ADA project focusing on the image content retrieval part: the images contained in the corporate documents constitute a relevant source of information that could be relevant in the manufacturing work ow. On this context, we developed speci c techniques for the extraction, classi cation, recognition, and tagging of images within technical documentation. The architecture and the methodologies of our work are presented on Section 3 and Section 4, respectively. 2
Image recognition techniques have been extensively studied in the last decade
in the eld of computer vision and the recovery of multimedia information.
Deep learning techniques, such as those based on Convolutional Neural
Networks (CNNs), represent today the state of the art for the most varied computer
vision activities such as image classi cation, image recovery and object
recognition [
In the project, the development of tools to search for graphic symbols
belonging to technical schemes within the technical documentation is of particular
importance. To this end, some works [
Elyan et al. [
The technical documents of manufacturing companies often contain a large number of heterogeneous images such as graphics, wiring diagrams, mechanical drawings, etc. While on the one hand the images are almost always accompanied by text such as captions, descriptions, and labels, being able to recognize their content without using text is an increasingly requested feature: images represent a rich source of information.
Provides image search capabilities via category matching, object presence, and visual similarity Image Dataset
The Image Analysis Module (Fig. 1) aims to enable companies to extract information from images contained within their technical documentation. This information is necessary to perform a search and classi cation of images based on visual content. The module in particular deals with the identi cation and extraction from the images the content, classifying them, recognizing some graphic elements, such as symbols, within them and looking for similar ones.
The Image Analysis Module is composed by the following sub-components: FExtractor Module. This component is intended to extract from one image one or more visual descriptors (features) that allow one to perform searches based on visual similarity using an image as a query.
Classi er Module. The classi er component deals with the classi cation of the images in the various possible types in the eld of technical and patent documentation (e.g. technical drawing in perspective, sections, electronic circuit, ow chart, etc.). The tool relies on modern Machine Learning techniques based on Deep Learning. The automatic classi cation is based on training a neural network on a number of examples for each of the types of images to be classi ed. The accuracy of the tool is strictly correlated to the number of training examples provided.
Recognizer Module. This component allows the automatic recognition of objects within the images classi ed by the classi cation tool: once the images are classi ed through the classi cation tool, the objects are automatically recognized within the image. For each type of image, the tool will have a number of examples related to the objects to be identi ed. Another result of this component is to automatically associate one or more tags with an image: the tags can be further enriched or corrected by analyzing the text in the document that refers to the image itself.
Indexer Module. This component deals with the appropriate indexing of both the visual features of the images and the context features related to the image: the image is decomposed into a visual part and into a symbolic part. For each image in fact, we will have both the global visual features necessary to perform visual similarity queries, and local features to perform queries able to detect and localize speci c symbols contained in the image.
Searcher Module. This module deals with sorting the various types of search supported (see the details below).
The search component allows the user to perform di erent types of searches using the index created by the Indexer module: 1. Textual Search on the text correlated or extracted from the images. 2. Similarity Search on images: search by similarity of the basic elements (symbols) in an image archive using an example as a query. 3. Search for the categories to which an image belongs.
Furthermore, the Searcher Module is able to automatically handle external queries, i.e. using images that are not present in the index, as well as internal queries, i.e. using images already present in the database as queries. For the external queries, the Searcher Module uses FExtractor Module to extract features from the query image.
In the next section, we will describe the speci c methodologies exploited for the realization of the four main mentioned modules: FExtractor, Classi er, Recognizer, and Indexer. 4
Due to their astonishing e ectiveness on perceptual tasks, we resort to stateof-the-art Deep Learning techniques based on Convolutional Neural Networks
VCNO VCNO
CNN
ResNet-101
VNCO VNCO
CNN Features Extractor
Recognizer
R O S S E R G E R 0.2 -1.5 5.4
… 1.0 0.0 RegL(iaoRsn-taMLlAaPCyoe)orling DeIms-c8ar.gi3petor
VCNO VCNO
CNN Features Extractor n o itt a u m r e P p e e D (CNN) to implement the visual analyses conducted by the aforementioned modules.
In the following, we will describe in details the methods chosen to implement the main functions of the Image Analysis Module, and how they are distributed among its sub-components. As depicted on Figure 2, for each image on the dataset: { we extract the features (FExtractor) as descriptors of the image on the index (Indexer); the extracted features are also made available to the Recognizer and Classi er modules; { for speci c categories, we use ad-hoc techniques for the object recognition (Recognizer) and the classi cation of the image (Classi er); we rely on simpler techniques based on previously extracted features otherwise; { all the information deriving from FExtractor, Recognizer and Classi er are memorized in the Image Record on a speci c index (Indexer). 4.1
Visual Similarity Search
For the implementation of the visual similarity search, we employ
state-of-theart global image descriptors extracted from CNN speci cally tailored for image
retrieval. Speci cally, we select the Region Maximum Activation of Convolution
(R-MAC) descriptors [
Given an image, the FExtractor module rst feeds it to a pre-trained convolutional network to extract the output of the last convolutional layer, which is composed by multiple feature maps having two spatial dimensions. Then, we compute the R-MAC descriptor by pooling and aggregating di erent parts (or regions) of those feature maps. The feature maps are max-pooled over several regions on their spatial dimensions, and the obtained vectors are aggregated by summation and the result l2-normalized.
The choice of the pre-trained CNN for the extraction of the feature maps
is essential: instead of choosing a network trained on generic object recognition
tasks (such as ImageNet), we select the model developed by [
As already mentioned on Section 3, part of the work done for the ADA project is to make use also of the text correlated or extracted from the image. In the following paragraph, we will explain how we realized an appropriate index supporting both the visual and textual features of the images in an e cient way. 4.2
Indexing of Image Descriptors
While textual information, such as labels and tags, can be stored in a relational
database, we need a similarity search index structure to store high-dimensional
image features and e ciently compute the cosine-similarity to perform retrieval.
Despite open-source indexes for e cient similarity search exist [
Seeking for simplicity, we propose to adopt Elasticsearch in the Indexer and
Searcher modules as database for all the information extracted by the visual
analysis modules, and adapt our similarity search needs to the full-text search
engine provided by the software. Speci cally, the adoption of two techniques,
Deep Permutations [
The Deep Permutations technique is based on the fact that in high-level
deep-learned features, each dimension represents an abstract visual concept, and
its value speci es the importance of that concept. Similar features tend to have
the same relative importance among visual concepts, and thus, we can
approximate a oat vector of features by sorting its dimensions in descending order and
keeping the sorting permutation (an integer vector). By truncating the
permutation to the top elements only, Amato et al. [
The Surrogate Text Representation technique aims at encoding a given integer vector in the term frequency values of a full-text search engine by generating an appropriate textual string. Used in combination with Deep Permutations, it permits us to store sparse truncated permutation in the inverted index for textual data and leverage the vector document model of full-text search engines to perform cosine similarity searches. 4.3
Image Categorization and Object Detection
For certain documentation, the Classi er module needs to assign each image to
one or more categories de ned by the system users. Thus, we have to implement
a multi-class multi-label image classi er. For the categories with a considerable
number of training examples, a CNN can be trained to implement the image
classi er. Instead of de ning and training a new model from scratch, we propose
to leverage the transfer learning practice, exploiting the knowledge of popular
CNNs that have been successfully trained in large-scale generic object
recognition tasks. Speci cally, we plan to use the ResNet-50 [
For speci c categories of images, the Recognizer module needs to detect
the presence of particular objects of interest. If the location of the object is not
required, R-MAC features matching can be used to reveal object presence. Being
an aggregation of local region descriptors, matching the R-MAC descriptor of
a sub-region of an image against the whole image descriptor will yield a high
score. Thus, we can perform object detection via instance-level similarity search
reusing the same modules. Instead, if more ne-grained information about
detected objects is required, such as the speci c localization in the image, and
enough training samples are available, we resort to more complex object
detection techniques based on Deep Learning, such as region-based CNNs [
Both the information extracted by the Classi er and Recognizer modules are stored by the Indexer module in Elasticsearch as additional elds of the image record, and accordingly searchable through the Searcher module. 5
This paper brie y introduces the on-going work on image analysis in the ADA project, whose main aim is to support the innovation of the production process of the manufacturing companies. We focused on the description of the methodologies carried out to extract relevant information from images contained in the di erent document provided by the project partners. This work was partially funded by \Automatic Data and documents Analysis to enhance human-based processes" (ADA), CUP CIPE D55F17000290009. We gratefully acknowledge the support of NVIDIA Corporation with the donation of a Tesla K40 GPU used and a Jetson TX2 board used for this research.