Introduction

Reflectance Transformation Imaging (RTI) is a non-contact, non-destructive, computational imaging technique used to study objects under varying illumination directions. This provides an enhanced visualization experience which is its primary use. RTI was first developed at Hewlett Packard Laboratories (HP Labs) in 2001 [1]. Since then, many new RTI tools (PTMFitter, RTIBuilder and RTIViewer) and methods have been developed to exploit this imaging technique for various applications. RTI has three stages. The first stage is Acquisition. It encompasses the acquisition setup and planning algorithms, calibrations, preparation, and dataset handling. Figure 1 demonstrates the concept of RTI acquisition setup. Each image is captured with a different illumination direction where the camera and object are fixed. This collection of acquired images is called the RTI dataset. RTI datasets are also known as Multi Light Image Collection (MLIC) or Single Camera Multi Light (SCML), depending on the research community [2,3]. The second stage is called Modelling. It primarily processes the dataset to model the data from discrete to continuous space [4,5,1,6]. The third stage is called Application. It involves utilizing the modeled data to get insights about the surface [7,8,9,10,11,12]. For example, to study the geometry of a surface, Normal maps created from modeled RTI data can be one application that can further help in the deeper investigation of surfaces [13]. Another application can be interactive visualization of surfaces in continuously varying illumination [14,15]. RTI has profound applications in CH of which change detection and digitization are the most popular ones [16,17]. The main idea behind RTI is exploiting the reflectance information of the surface under varying illumination. Illumination is incident light of some wavelength (visible spectrum) that hits the surface. Part of the light is absorbed by the surface and part of the light is reflected. If most of the reflected light is concentrated in one direction, it is called specular scattering. Conversely, if the light is more uniformly reflected in all directions it is called a diffuse or lambertian scattering. The comprehensive dataset of reflectance measurements of a surface point, obtained from different light directions as observed by a fixed optical sensor (camera) in a Reflectance Transformation Imaging (RTI) setup, is termed as the reflectance profile. Figure 2 demonstrates this reflectance model. Reflectance modeling is a well-studied problem in computer graphics and notable models are Phong Reflectance Model [18], Cook-Torrance Reflectance Model [19] and modern deep-learning based reflectance models like NeRF [20].

The reflectance profile captures how the appearance of a surface point changes as the light direction varies, while the viewing angle remains constant. This concept is closely related to the Bidirectional Reflectance Distribution Function (BRDF) used in computer graphics. BRDF explains how light is reflected for all possible combinations of incoming light direction, viewing direction and wavelength whereas RTI is focused on a fixed viewing direction and wavelength with only variable incoming light direction. BRDF therefore has many applications in color analysis in computer graphics [21] and surface classification in digital imaging [22] however BRDF measurement is a complex task and is often very difficult to do for many CH objects.

Classification in computer vision has been revolutionized in this decade due to recent advancements in deep learning methods [23,24,25]. Most of the classification is based on the geometrical features of the object. The question we ask ourselves is, while the existing methods are very good at identifying objects, can these methods provide information about the composition of the object, its tactile properties, and other physical characteristics? Humans can predict how an object feels to the touch and comprehend its properties from an image. For example, we can identify the material of the table; we can predict the feeling of touch, anticipate its tactile sensation, and understand the state of the table if it's dirty or clean from an image. Our brains have the remarkable capacity to infer the physical attributes of surfaces from visual input, and humans develop this skill from a young age.

RTI can provide an extra dimension of reflectance in the data, making it unique relative to other imaging modalities. This dimension can capture the reflectance of the surface and can also be used to estimate BRDF parameters. This BRDF can be estimated by various surface reflectance models such as the Ward model [26] shown in figure 2 which models diffuse as well as specular surfaces and can provide surface understanding beyond identification. This can help us gain insights into the material properties and conditions of the surface under study. The classification of surfaces with respect to their reflectance properties can also help in optimizing the acquisition process in RTI. RTI acquisition is conventionally done manually using homogenous, equally spaced light directions, pre-planned without using any surface information. This can produce redundant light directions, potentially leading to huge-size RTI datasets. The optimization of light directions can save a lot of resources and time for acquiring RTI datasets [17,27,28].

We propose to exploit the illumination variation property of RTI to classify different areas in the The illustration depicts the interaction of illumination direction with a surface, showing both lambertian or diffuse scattering and specular scattering from a light source as shown in the RTI setup. Some light is also absorbed by the surface. This interaction varies with the change in incident illumination direction surfaces. The differences in reflectance properties within a surface could be caused by various factors. For example, rust can alter the reflectance profiles of surfaces, and dust or dirt can also affect the reflectance profile of a surface. Developing a tool to classify surfaces using RTI can aid in the study, detection, and investigation of changes or degradation. Surface classification can also lead to material identification. Material information is extensively exploited by curators, restorers, conservators, and researchers to gain insights into cultural heritage objects. Our framework's initial outcome is identifying interesting features and anomalies from the RTI data by leveraging self-supervised learning techniques like K-means and Self-Organizing Maps (SOMs). This framework can be later integrated to RTI visualization tools for interactive visualization and aims to provide a powerful tool for the analysis and visualization of surface characteristics. This tool will be valuable for applications in cultural heritage preservation, industrial quality control, and beyond, enabling users to gain comprehensive insights into the surfaces they study. Section 2 familiarizes with the related work and section 3 presents the methodology designed for classification. The paper then introduces the dataset in section 4 used in this study, followed by section 5, which discusses the experiments conducted using the proposed methodology. Finally, section 6 presents the conclusions drawn from this study. The code for this work is available at github1 .

Related Work

Classification of surfaces is a complex task. Multiple modalities have been used to learn and understand the surfaces. Some related works in this field use texture with visual modality to classify surfaces using deep learning [29]. Surfaces tend to have a unique texture, which can serve as critical information in classification. A computational imaging method was presented for raw material classification using features of Bidirectional Texture Functions (BTF) in [30]. They proposed to learn discriminative illumination patterns and texture filters to directly measure optimal projections of BTFs for the classification of surfaces. The multispectral polarimetric imaging technique captures both the spectral and polarimetric properties of the light, adding dimensions to the spatial intensity that is normally acquired. This provides unique and discriminatory information that can help in material classification. This classification method based on multispectral polarimetric BRDF characteristics is proposed in [22]. In particular, near-infrared (NIR) range wavelengths were used for materials classification [31]. A hyperspectral imaging system was developed in [31] to classify and sort papers according to their material quality. Another paper proposed a Deep Convolutional Neural Network (DCNN)-based 60-GHz radar material classification system that uses images from the radar sensor as input features [32]. A classification accuracy of above 97 % was achieved while classifying 1875 radar images from ten different materials. These materials included books, chocolate cream, glass, the aluminum surface of a laptop, wood, wool, carpet, tile, laminate, and water. However, all these techniques are expensive because they utilize specific hardware like radar, multispectral cameras, hyperspectral cameras, or texture measurement tools and are not trivial methodologies. Furthermore, studying the object's state (whether the object is clean or dirty, etc.) has not yet been explored in these above-mentioned methods.

Some inspiration comes from the medical field. The classification of ECG signals using machine learning has been an important milestone in the diagnostic and medical analysis field [33,34]. The ECG signals have hidden properties and profiles for a certain class of disease [35,36]. Rastgoo et al. [37] proposed a classification framework for melanoma lesions using sparse-coded features and Random Forests. Melanoma is a type of skin cancer that can be treated if diagnosed.

Another source of inspiration comes from multispectral and hyperspectral image analysis. This type of analysis and classification has led to numerous applications in cultural heritage, medical imaging, quality control, and industrial inspection. Mandal et al. used unsupervised learning to classify pigments on Edvard Munch's self-portrait painting by utilizing hyperspectral data [38]. The classification of pigments using hyperspectral imaging provides valuable insight for understanding the painting as well as for its conservation [39,40,41,42]. It also offers opportunities to estimate the era of paintings and identify forgery in paintings. Devassy et al proposed unsupervised clustering to classify hyperspectral data of paper, which has significant applications in document forgery investigations [43]. Another important technique for pigment identification in CH is Near Infrared Imaging (NIR). NIR imaging reveals information about the chemical composition and physical properties of pigments in CH which can also be used for surface analysis and classification [44,45].

Methodology

In this paper, we propose a comprehensive methodology to classify the surfaces with respect to reflectance profiles using RTI. We divide our methodology into four major parts. The first part is data acquisition and processing. The second part includes modeling data using machine learning algorithms. The third part consists of result extraction and mapping into the spatial domain for interpretation. The final part is a visualization of results using a segmentation mask on the reference image. Figure 3 demonstrates the pipeline of the methodology.

This study primarily focuses on qualitative data (such as anomalies, degradation, rust, etc). Our objective is to identify the surfaces that are otherwise not identifiable as distinguished ones based on their appearance. This is a latent property and reasons for the change in reflectance profile might include anomaly, rust, material degradation etc. It is also important to note that this methodology is for Visible Spectrum (ViS) imaging. We have not utilized any multispectral sensor.

The virtual and real RTI both have been utilized to test our methodology. We started with a virtual RTI dataset simulated in Blender [46]. The Blender RTI toolbox mimics the RTI setup where a stationary object is illuminated from a desired number of illumination directions. This virtual approach allows us to precisely control the illumination conditions and surface properties, providing a clear and noisefree dataset for initial algorithm development and testing. The use of Blender enables us to simulate various material properties and reflectance behaviors, which are crucial for validating our classification framework in a controlled environment before applying it to real-world data. The real and simulated datasets were used to train the Self-Organizing Maps (SOMs) and K-means from scratch, ensuring that the models could correctly classify surface reflectance profiles. No pre-trained SOMs were used in this study. Section 4 explains more about the dataset. This is real-time training for each dataset and is self-supervised without the need for any labeled data. The first step in our methodology is signal extraction. In this step, we acquire the Multi Light Image Collection (MLIC) data from a simulated environment or real acquisition in the RTI dome that is available in the lab. The acquired data needs to be pre-processed. We reduce the sample space by extracting the Region of Interest (ROI) from the dataset using a reference image. Figure 4 demonstrates the signal extraction process.

K-Means

The second and most important part of the methodology is the machine learning methods for the classification of surfaces. Initially, we used K-means clustering to classify the signals based on their standard deviation. K-means is a machine learning algorithm that divides the n-dimensional data into k-clusters using variance inside the dataset [47,48]. While it works well for many cases, it can be vulnerable to wrongly classifying the surfaces because two different signals might have the same standard deviation. This motivated us to utilize SOM neural networks. The K-means algorithm that we employed for our classification is explained in pseudocode 1.

Self Organizing Maps (SOMs)

SOM's are inspired by biological neural systems. They are an abstract mathematical model of topographic mapping from visual sensors to cerebral cortex [49]. SOMs are a type of artificial neural network that is trained using unsupervised learning to produce a low-dimensional discretized representation of the training samples from input space [50,51]. Figure 5 demonstrates the vanilla architecture of SOMs.

SOM's are trained using competitive learning rather than the error-correction learning (e.g., backpropagation using gradient descent) conventionally used by other artificial neural networks like ResNets [24], VGG [52], and Tranformers [53,54] etc. SOM's work in two phases. The first one is training and the second one is mapping. Initially, the training phase learns from the dataset or input space to create a simplified, lower-dimensional version of "map space". Finally, this established map space is used to classify new input data in the mapping phase. The SOM algorithm is summarized in pseudocode 2 adapted from [55,49] The third part of the algorithm is the extraction of results. The signals that are classified with their surface labels are mapped back to the spatial space of the image. Finally, a segmentation mask is built based on labels and visualized in the fourth and last part. The section 5 explains and demonstrates the experiments and results by applying the discussed methodology.

Dataset

We have prepared some in-house data, and other data has been obtained from published works [56,57]. For the acquisition of MLIC, we selected the ring setup (varying azimuth, fixed elevation) due to its capability to showcase unpredictable patterns in pixel intensity because of surface characteristics. This lets us focus more on the data depending on surface characteristics. The data is explained in the following sub-sections. At each iteration 𝑡, present an input signal 𝑥(𝑡), and select the winner using competition.

𝜈(𝑡) = arg min 𝑘∈Ω ‖x(𝑡) − w 𝑘 (𝑡)‖

6:

Update weights of the winner and its neighbors.

Δw 𝑘 (𝑡) = 𝛼(𝑡)𝜂(𝜈, 𝑘, 𝑡) [x(𝑡) − w 𝜈 (𝑡)]

◁ Ω is a set of neuron indexes where:

𝜂(𝜈, 𝑘, 𝑡) = exp [︁ − ‖r𝜈 −r 𝑘 ‖ 2 2𝜎(𝑡) 2 ]︁ ◁ neighbourhood function 7:

Learning rate decreases monotonically and satisfies the following equations:

0 < 𝛼(𝑡) < 1 lim 𝑡→∞ ∑︀ 𝛼(𝑡) → ∞ lim 𝑡→∞ 𝛼(𝑡) → 0 8: return updated 𝑊 = [𝑤 1 , 𝑤 2 , . . . , 𝑤 𝑚 ] 9: 𝑌 = 𝑊 (𝑋) 10: 𝐶𝑙𝑎𝑠𝑠𝑒𝑠 = 𝑣𝑒𝑐2𝑖𝑛𝑑(𝑌 )

Coin of Emperor Nicolas

The first surface consists of a cultural heritage object. It is a coin called the "Coin of Emperor Nicolas II" from the year 1897 AD, depicted in figure 6. This coin's 3D model is sourced from Sketchfab under a Creative Commons license and was used in our study [56]. We used Blender software to capture the RTI images.

Rust Course Metal

The second surface we used in our experiments was "Rust Course Metal", which is obtained from a dataset published by [57]. It consists of a metallic surface with some rust on it. The figure 7 shows the metal. This surface is important to understand how the reflectance properties of metals change when rust comes on them. This has significance in industrial quality control and material inspection.

Two Black Surfaces

This dataset was prepared to compare the algorithm's performance in simulated data and real data. Two black surfaces with the same color perception to the human eye but different material properties were prepared by 3D printing. We have used the grayscale camera, primarily a one-channel camera, to acquire its MLIC dataset. Figure 8 demonstrates the real "Two black Surfaces". The synthetic/simulated "Two black surfaces" were also created in blender software by putting the same constraint of the same color and different material. The metallic and roughness properties were kept different for both surfaces. Figure 9 shows the two black surfaces on the blender. Since it is a simulated environment, there was no fabrication constraint and hence the surface did not have any texture lines like the real "Two black surfaces".

Experiments and Results

This section employs the methodology discussed in the section 3 to experiment on data explained in section 4. The results are obtained and discussed.

The first experiment was done on the "Coin of Emperor Nicolas". Since the coin is old and a cultural heritage object, it has undergone various aging processes, and its surface is no longer homogenous. We aim to differentiate the surface within the coin based on its illumination variation information. This can further help cultural heritage experts understand the level of rust or other degradation process that can be present and not apparent otherwise. We used 27 distinct illumination directions at a constant elevation of 30 ∘ and variable equally spaced azimuth to generate 27 simulated images in the blender. Figure 10 demonstrates and explains the results. The same experiment was repeated with the K-means algorithm, and results are shown in figure 11.

The second experiment was performed on the "Rust Course Metal". We used 10 distinct illumination directions with a constant elevation of 30 ∘ and equally spaced variable azimuth angles to generate 10 simulated RTI images. Figure 12 explains the results. Since it is a synthetic flat surface with symmetric reflectance profile, ring acquisition was not helpful in getting illumination variation data. We found that using a varying elevation acquisition, where lights are positioned at different elevation angles, would be more suitable for obtaining meaningful reflectance data from this type of surface.

The third experiment was performed on the real "Two Black Surfaces". The two black surfaces were assembled in the RTI dome to get MLIC. We captured 24 distinct images from 24 illumination directions with a constant elevation of 45 ∘ and variable equally spaced azimuth. The camera in the RTI dome is very high quality and produces 4000 by 6000-pixel images. Hence, it was difficult to process big regions of interest. While the algorithm struggles to classify real data, as shown in figure 13, it was able to differentiate one surface from another to some extent.

The fourth experiment was performed on the synthetic "Two Black Surfaces". We used 10 distinct illumination directions with a constant elevation of 30 ∘ and equally spaced variable azimuth angles to generate 10 simulated RTI images. This surface is discussed in section 4 and figure 9. The algorithm classifies the two different materials in "Two Black Surfaces" as shown in figure 14 because it is a clean synthetic surface with a distinguished reflectance profile.

Conclusion

This paper presents an approach to surface classification using RTI data. The suggested method uses SOM and K-means, two self-supervised learning algorithms, to classify surfaces according to their reflectance profiles. Surface classification using RTI has great potential for several applications, such as industrial quality control and the preservation of cultural heritage. We believe surface classification is surfaces. This methodology is designed for 2D plain surfaces and struggled with real "Two Black Surfaces" whose geometry was not completely flat due to 3D printer fabrication constraints. The algorithm was identifying shadow of elevations as matt surface which was the wrong classification as discussed in section 5. However, the algorithm was still able to classify surfaces according to their reflectance profiles. We observed that the K-means algorithm does not perform as well as the SOM neural networks in our experiments, however this observation is primarily based on visual inspection of the classification results. This qualitative assessment, though not quantified by classification mismatch metrics, can be supported by expert validation of the results. Our approach's motivation is to exploit the illumination variation data to better understand the surfaces and discover their anomalies. This is the prime reason we are using the self-supervised method, and since we are exploring the surface, we do not have the ground truth to compare our algorithm.

In summary, the experiments presented here contribute to the advancement of techniques for surface analysis, with implications for fields such as cultural heritage preservation and material characterization.

In the future, our research will focus on refining and creating methodologies to address computational challenges, expanding the scope of materials analyzed and addressing geometrical 3D properties of the surfaces in the algorithm. We plan to extend this classification model to use RGB images. RGB images offer extra color information that can help the model better distinguish between different materials, (a) Segmentation of two black surfaces using K-means. It can be seen that the K-means algorithm has classified the right side (PLC matt surface with red) mostly correctly but is struggling with classifying the relatively specular left side (Nylon). This is due to the involvement of texture geometry.

(b) Segmetation of two black surfaces using SOM neural network. It can be seen that the SOM algorithm mostly classifies the right PLC side as one class (red) based on its matt response. However, it has misclassified some parts of the left side surface because it mistakes shadow a matt surface due to texture geometry. potentially improving accuracy. We also plan to extend this framework for 3D RTI acquisition. This means the algorithm will be extended for RTI acquisitions with variable elevations and variable azimuth angles. This can provide invaluable reflectance variation information.