Reflectance Transformation Imaging (RTI) is an imaging technique used to analyze objects or surfaces by capturing their appearance under varying illumination directions. This paper proposes two self-supervised learning algorithms to classify surfaces according to their reflectance profiles. The classification problem is addressed using K-means and Self Organizing Map (SOM) neural networks. The proposed methodology is evaluated using both real and synthetic datasets. The primary motivation for our approach is to exploit illumination variation data to enhance surface understanding and detect anomalies. Given the exploratory nature of this task and the lack of ground truth for comparison, a self-supervised method was deemed most suitable. The classification of surfaces using reflectance information has immense applications in fields such as Cultural Heritage (CH) preservation, digitization, and industrial quality control.
Reflectance Transformation Imaging (RTI) is a non-contact, non-destructive, computational imaging
technique used to study objects under varying illumination directions. This provides an enhanced
visualization experience which is its primary use. RTI was first developed at Hewlett Packard Laboratories
(HP Labs) in 2001 [
The main idea behind RTI is exploiting the reflectance information of the surface under varying illumination. Illumination is incident light of some wavelength (visible spectrum) that hits the surface. Part of the light is absorbed by the surface and part of the light is reflected. If most of the reflected light is concentrated in one direction, it is called specular scattering. Conversely, if the light is more uniformly reflected in all directions it is called a difuse or lambertian scattering. The comprehensive dataset of reflectance measurements of a surface point, obtained from diferent light directions as observed by a fixed optical sensor (camera) in a Reflectance Transformation Imaging (RTI) setup, is termed as the reflectance profile. Figure 2 demonstrates this reflectance model. Reflectance modeling is a well-studied problem in computer graphics and notable models are Phong Reflectance Model [ 18], Cook-Torrance Reflectance Model [19] and modern deep-learning based reflectance models like NeRF [20].
The reflectance profile captures how the appearance of a surface point changes as the light direction varies, while the viewing angle remains constant. This concept is closely related to the Bidirectional Reflectance Distribution Function (BRDF) used in computer graphics. BRDF explains how light is reflected for all possible combinations of incoming light direction, viewing direction and wavelength whereas RTI is focused on a fixed viewing direction and wavelength with only variable incoming light direction. BRDF therefore has many applications in color analysis in computer graphics [21] and surface classification in digital imaging [ 22] however BRDF measurement is a complex task and is often very dificult to do for many CH objects.
Classification in computer vision has been revolutionized in this decade due to recent advancements in deep learning methods [23, 24, 25]. Most of the classification is based on the geometrical features of the object. The question we ask ourselves is, while the existing methods are very good at identifying objects, can these methods provide information about the composition of the object, its tactile properties, and other physical characteristics? Humans can predict how an object feels to the touch and comprehend its properties from an image. For example, we can identify the material of the table; we can predict the feeling of touch, anticipate its tactile sensation, and understand the state of the table if it’s dirty or clean from an image. Our brains have the remarkable capacity to infer the physical attributes of surfaces from visual input, and humans develop this skill from a young age.
RTI can provide an extra dimension of reflectance in the data, making it unique relative to other imaging modalities. This dimension can capture the reflectance of the surface and can also be used to estimate BRDF parameters. This BRDF can be estimated by various surface reflectance models such as the Ward model [26] shown in figure 2 which models difuse as well as specular surfaces and can provide surface understanding beyond identification. This can help us gain insights into the material properties and conditions of the surface under study. The classification of surfaces with respect to their reflectance properties can also help in optimizing the acquisition process in RTI. RTI acquisition is conventionally done manually using homogenous, equally spaced light directions, pre-planned without using any surface information. This can produce redundant light directions, potentially leading to huge-size RTI datasets. The optimization of light directions can save a lot of resources and time for acquiring RTI datasets [17, 27, 28].
We propose to exploit the illumination variation property of RTI to classify diferent areas in the
Illumination Scattering ray
surfaces. The diferences in reflectance properties within a surface could be caused by various factors. For example, rust can alter the reflectance profiles of surfaces, and dust or dirt can also afect the reflectance profile of a surface. Developing a tool to classify surfaces using RTI can aid in the study, detection, and investigation of changes or degradation. Surface classification can also lead to material identification. Material information is extensively exploited by curators, restorers, conservators, and researchers to gain insights into cultural heritage objects. Our framework’s initial outcome is identifying interesting features and anomalies from the RTI data by leveraging self-supervised learning techniques like K-means and Self-Organizing Maps (SOMs). This framework can be later integrated to RTI visualization tools for interactive visualization and aims to provide a powerful tool for the analysis and visualization of surface characteristics. This tool will be valuable for applications in cultural heritage preservation, industrial quality control, and beyond, enabling users to gain comprehensive insights into the surfaces they study.
Section 2 familiarizes with the related work and section 3 presents the methodology designed for classification. The paper then introduces the dataset in section 4 used in this study, followed by section 5, which discusses the experiments conducted using the proposed methodology. Finally, section 6 presents the conclusions drawn from this study. The code for this work is available at github1.
Classification of surfaces is a complex task. Multiple modalities have been used to learn and understand
the surfaces. Some related works in this field use texture with visual modality to classify surfaces using
deep learning [
Some inspiration comes from the medical field. The classification of ECG signals using machine
learning has been an important milestone in the diagnostic and medical analysis field [
Another source of inspiration comes from multispectral and hyperspectral image analysis. This type
of analysis and classification has led to numerous applications in cultural heritage, medical imaging,
quality control, and industrial inspection. Mandal et al. used unsupervised learning to classify pigments
on Edvard Munch’s self-portrait painting by utilizing hyperspectral data [
In this paper, we propose a comprehensive methodology to classify the surfaces with respect to reflectance profiles using RTI. We divide our methodology into four major parts. The first part is data acquisition and processing. The second part includes modeling data using machine learning algorithms. The third part consists of result extraction and mapping into the spatial domain for interpretation. The final part is a visualization of results using a segmentation mask on the reference image. Figure 3 demonstrates the pipeline of the methodology.
This study primarily focuses on qualitative data (such as anomalies, degradation, rust, etc). Our objective is to identify the surfaces that are otherwise not identifiable as distinguished ones based on their appearance. This is a latent property and reasons for the change in reflectance profile might include anomaly, rust, material degradation etc. It is also important to note that this methodology is for Visible Spectrum (ViS) imaging. We have not utilized any multispectral sensor.
The virtual and real RTI both have been utilized to test our methodology. We started with a virtual
RTI dataset simulated in Blender [
Signal Analysis Extraction of Pixel
Signals Selection of
Region of Interest(ROI) K-means
Self Organizing Map Neural
Network Classification of Signals Mapping signals back to spatial location
Segmentation
Mask
the models could correctly classify surface reflectance profiles. No pre-trained SOMs were used in this study. Section 4 explains more about the dataset. This is real-time training for each dataset and is self-supervised without the need for any labeled data.
The first step in our methodology is signal extraction. In this step, we acquire the Multi Light
Image Collection (MLIC) data from a simulated environment or real acquisition in the RTI dome that
is available in the lab. The acquired data needs to be pre-processed. We reduce the sample space by
extracting the Region of Interest (ROI) from the dataset using a reference image. Figure 4 demonstrates
the signal extraction process.
3.1. K-Means
The second and most important part of the methodology is the machine learning methods for the
classification of surfaces. Initially, we used K-means clustering to classify the signals based on their
standard deviation. K-means is a machine learning algorithm that divides the n-dimensional data
into k-clusters using variance inside the dataset [
SOM’s are trained using competitive learning rather than the error-correction learning (e.g.,
backpropagation using gradient descent) conventionally used by other artificial neural networks like ResNets[ 24],
VGG [
The third part of the algorithm is the extraction of results. The signals that are classified with their surface labels are mapped back to the spatial space of the image. Finally, a segmentation mask is built based on labels and visualized in the fourth and last part. The section 5 explains and demonstrates the experiments and results by applying the discussed methodology.
Algorithm 1 Classifying RTI signals using K-means with Standard Deviation 1: Input: Signal Standard deviation dataset = [1, 2, . . . , ], number of clusters , threshold , iterations 2: Output: Classification of signals into cluster and centroids 3: Initialize cluster centroids 1, 2, . . . , ∈ R randomly 4: repeat 5: for each signal () do 6: {︁ : ‖ − ‖2 ≤ ‖ − ‖2 for all = 1, 2, . . . , }︁ ◁ is set of data points 7: 8:
= assigned to for each cluster do
= |1| ∑︀∈ 9: Convergence Check: 10: if || − || < then 11: break 12: if == then 13: break 14: until convergence 15: return cluster assignments and centroids
◁ Update cluster centroids
We have prepared some in-house data, and other data has been obtained from published works [
C I L M s s o r c a P l a n g i S l e x I P
P1 P2 P3 P4 P5 Pn
C1 C2 Cm s e s s a l c m o t n i s e c a f r u S f o n o it a c iif s s a l C 1: Input: RTI Signals X = [1, 2, . . . , ], iteration , learning rate 2: Output: Classification of signals into classes 3: Initialize the weight vectors W = [1, 2, . . . , ] randomly 4: for = 1 to do 5: At each iteration , present an input signal (), and select the winner using competition.
() = arg min∈Ω ‖x() − w()‖ 6: Update weights of the winner and its neighbors.
Δw() = () (, , ) [x() − w ()] ◁ Ω is a set of neuron indexes
where:
(, , ) = exp [︁− ‖r2 − (r)2‖2 ]︁ ◁ neighbourhood function
7: Learning rate decreases monotonically and satisfies the following equations:
0 < () < 1
lim→∞ ∑︀ () → ∞
lim→∞ () → 0
8: return updated = [1, 2, . . . , ]
9: = ()
10: = 2( )
4.0.1. Coin of Emperor Nicolas
The first surface consists of a cultural heritage object. It is a coin called the "Coin of Emperor Nicolas II"
from the year 1897 AD, depicted in figure 6. This coin’s 3D model is sourced from Sketchfab under a
Creative Commons license and was used in our study [
(d) Measuring L*a*b* values of both surfaces.
(e) Color measurement values of Nylon.
The synthetic/simulated "Two black surfaces" were also created in blender software by putting the same constraint of the same color and diferent material. The metallic and roughness properties were kept diferent for both surfaces. Figure 9 shows the two black surfaces on the blender. Since it is a simulated environment, there was no fabrication constraint and hence the surface did not have any texture lines like the real "Two black surfaces".
This section employs the methodology discussed in the section 3 to experiment on data explained in section 4. The results are obtained and discussed.
The first experiment was done on the "Coin of Emperor Nicolas". Since the coin is old and a cultural heritage object, it has undergone various aging processes, and its surface is no longer homogenous. We aim to diferentiate the surface within the coin based on its illumination variation information. This can further help cultural heritage experts understand the level of rust or other degradation process that can be present and not apparent otherwise. We used 27 distinct illumination directions at a constant elevation of 30∘ and variable equally spaced azimuth to generate 27 simulated images in the blender. Figure 10 demonstrates and explains the results. The same experiment was repeated with the K-means algorithm, and results are shown in figure 11.
The second experiment was performed on the "Rust Course Metal". We used 10 distinct illumination directions with a constant elevation of 30∘ and equally spaced variable azimuth angles to generate 10 simulated RTI images. Figure 12 explains the results. Since it is a synthetic flat surface with symmetric reflectance profile, ring acquisition was not helpful in getting illumination variation data. We found that using a varying elevation acquisition, where lights are positioned at diferent elevation angles, would be more suitable for obtaining meaningful reflectance data from this type of surface.
The third experiment was performed on the real "Two Black Surfaces". The two black surfaces were assembled in the RTI dome to get MLIC. We captured 24 distinct images from 24 illumination directions with a constant elevation of 45∘ and variable equally spaced azimuth. The camera in the RTI dome is very high quality and produces 4000 by 6000-pixel images. Hence, it was dificult to process big regions of interest. While the algorithm struggles to classify real data, as shown in figure 13, it was able to diferentiate one surface from another to some extent.
The fourth experiment was performed on the synthetic "Two Black Surfaces". We used 10 distinct illumination directions with a constant elevation of 30∘ and equally spaced variable azimuth angles to generate 10 simulated RTI images. This surface is discussed in section 4 and figure 9. The algorithm classifies the two diferent materials in "Two Black Surfaces" as shown in figure 14 because it is a clean synthetic surface with a distinguished reflectance profile. 5 10 15 MLIC Image no 20
25
This paper presents an approach to surface classification using RTI data. The suggested method uses SOM and K-means, two self-supervised learning algorithms, to classify surfaces according to their reflectance profiles. Surface classification using RTI has great potential for several applications, such as industrial quality control and the preservation of cultural heritage. We believe surface classification is
Cluster 2 5 10 15 MLIC Image no 20 25 250 200 y it tsne150 n lIna100 g i S 50 0
Cluster 3 5 10 15 MLIC Image no 20
25 the first step to material classification using RTI.
Machine learning plays a crucial role in surface classification problems. We used K-means classification initially, but it had drawbacks because signals can be mistakenly misclassified since K-means classifies signals based on standard deviation. It does not learn the patterns of the signal. This resulted in the use of artificial neural networks known as SOM neural networks, which is a self-supervised learning method and ofers a better low-dimensional representation of data. The examination of coin of Emperor Nicolas II provides us with valuable information about the reflectance properties of the (c) (a) (b) (d) surfaces. This methodology is designed for 2D plain surfaces and struggled with real "Two Black Surfaces" whose geometry was not completely flat due to 3D printer fabrication constraints. The algorithm was identifying shadow of elevations as matt surface which was the wrong classification as discussed in section 5. However, the algorithm was still able to classify surfaces according to their reflectance profiles. We observed that the K-means algorithm does not perform as well as the SOM neural networks in our experiments, however this observation is primarily based on visual inspection of the classification results. This qualitative assessment, though not quantified by classification mismatch metrics, can be supported by expert validation of the results.
Our approach’s motivation is to exploit the illumination variation data to better understand the surfaces and discover their anomalies. This is the prime reason we are using the self-supervised method, and since we are exploring the surface, we do not have the ground truth to compare our algorithm.
In summary, the experiments presented here contribute to the advancement of techniques for surface analysis, with implications for fields such as cultural heritage preservation and material characterization.
In the future, our research will focus on refining and creating methodologies to address computational challenges, expanding the scope of materials analyzed and addressing geometrical 3D properties of the surfaces in the algorithm. We plan to extend this classification model to use RGB images. RGB images ofer extra color information that can help the model better distinguish between diferent materials, (a) Segmentation of two black surfaces using K-means. It can be seen that the
K-means algorithm has classified the right side (PLC matt surface with red) mostly correctly but is struggling with classifying the relatively specular left side (Nylon). This is due to the involvement of texture geometry. (b) Segmetation of two black surfaces using SOM neural network. It can be seen that the SOM algorithm mostly classifies the right PLC side as one class (red) based on its matt response. However, it has misclassified some parts of the left side surface because it mistakes shadow a matt surface due to texture geometry. potentially improving accuracy. We also plan to extend this framework for 3D RTI acquisition. This means the algorithm will be extended for RTI acquisitions with variable elevations and variable azimuth angles. This can provide invaluable reflectance variation information.
We would like to express our deepest gratitude to ISITE-BFC project. (Initiatives Science Innovation Territoire Economie en Bourgogne-Franche-Comté) of Université de Bourgogne for their unwavering support and funding throughout the course of this research.
The demo code and data is available on the GitHub project webpage.