Vehicle-based mobile mapping systems capture co-registered imagery and 3D point cloud information over hundreds of kilometres of transport corridor. Methods for extracting information from these large datasets are labour intensive. These need to be easily con gured by non-expert users to process images and develop new work ows. Image Engineering provides a framework to combine known image processing, image analysis an image understanding methods into powerful applications. Such a system was built using Orange an open source toolkit for machine learning onto which image processing, visualisation and data acquisition methods were added. The system presented here enable users who are not programmers to manage image data and to customise their analyses by combining common data analysis tools to t their needs. Case studies are provided to demonstrate the utility of the system. Co-registered imagery and depth data of urban transport corridors provided by the Earthmine dataset and laser ranging systems are used.
applications. Point clouds are more expensive to acquire, have no colour information and often have gaps in the data. Most point clouds for mobile mapping have low density along the direction of travel unless multiple scans are performed e.g. six, three in each direction. Finally, imagery, stereo derived point clouds and laser scanning can be combined to exploit the strength of each method.
Data from two di erent systems was used in this paper. The Earthmine system captures panoramic imagery and uses stereo algorithms to generate co-registered 3D point clouds. Figure 1(a) shows the Earthmine system on a car. The other system, AAM, generates depth maps from laser ranging sensor. The AAM system provides generated co-registered imagery and 3D point clouds captured from mobile scanning systems such as shown in Figure 1(b).
Earthmine has developed a server-based system that allows the querying via location to obtain data about a particular location. The data can be processed using a number of methods e.g. randomly, from a number of known locations, or by \driving" along the capture pathway.
In many applications of computer vision, work ow is an important consideration. A user would typically read in some acquired data, process it interactively and produce the desired result. In many recognition applications, much tuning is needed requiring a user skilled in such techniques. The challenge is to produce a work ow that an non-expert user can use to con gure a complex process such as object detection. To do this, a user must be able to view selected parts of the data, identify objects of interest and train a system to use the best features for recognition through a feature selection process combined with some form of pattern recognition method such as a decision tree.
This paper presents a system that gives the non-expert the opportunity to develop powerful image processing applications. The system builds on existing open source frameworks and libraries. Case studies are provided to demonstrate the utility of the system. 2
IP primarily includes the acquisition, representation, compression, enhancement, restoration, transformation and reconstruction of images. IP manipulates an image to produce another (improved) image. IA is concerned with the extraction of information from an image. IA takes an image as input and outputs data. Here, the extracted data can be the measurement results associated with speci c image properties or the representative symbols of certain object attributes. IU transforms the data into descriptions allowing for decisions and actions to be taken according to the interpretation of the images.
The development of software is a complicated and tedious process, requiring highly specialized skills in systems
programming
The user interface of an image processing environment is a key aspect of the proper functioning of an
environment. A good user interface can signi cantly reduce the development e ort of new image processing applications.
The user interface also determines the usability of the environment to the various classes of users
One approach is the use of graphics as the programming language, or visual programming. By connecting
the graphic components a user can visualise the data and the path of the data. This provides a clear expression
of the ow of control in an application. The data ow metaphor is the best choice for the visual programming
interface in image processing
The Image Engineering system developed in this paper takes this approach. Using a visual environment
allows for fast prototyping, interactive exploration of algorithms and work ow, and development of applications.
The system presented exhibits characteristics and provides the bene ts of visual programming, program
visualisation
Figure 3(a) is a fragment of python code that creates a Histogram of Gradients (HoG)
The system was built using Orange an open source toolkit for machine learning
Orange is a general-purpose machine learning and data mining tool
Orange Canvas provides a graphical interface for these functions. Its basic ingredients are widgets. Each widget performs a basic task, such as reading the data from a le in one of the supported formats or from a data base, showing the data in tabular form, plotting histograms and scatter plots, constructing various models and testing them, clustering the data, and so on. from skimage.feature import hog from skimage import data, color, exposure image = color.rgb2gray(data.lena()) desc, hog_image = hog(image, orientations=8, pixels_per_cell=(16, 16), cells_per_block=(1, 1), visualise=True) # Process/save descriptor, perhaps display HoG image # ....
(a)
The widgets expose the parameters of the underlying API. The advantage of widgets is in their modularity.
Widgets can be connected through channels and communicate with each other by sending and receiving data.
The output of one widget is used as an input for one or several other subsequent widgets. Communication
channels are typed (i.e. the data type is determined to be integer, text, table, etc.) and the system establishes
the proper type of data connections automatically. This property relieves the user from the need to design data
structure, which is one of the greatest obstacles for lay users
A collection of widgets and their communication channels is called a schema, which is essentially a program designed by the user for a speci c data analysis task. The programming process creates a schema with widgets and their connections is done visually through an easy-to-use graphic interface. Schemas can be saved and compiled into executable scripts for later reuse. The power of Orange Canvas is its interactivity. Any change in a single widget for example, loading another data set, changing the lter, modifying the logistic regression parameters can instantly propagates down the scheme. 4
Orange does not provide any image processing facilities but does allow you to develop your own widgets, extend scripting interface or even create your own self-contained add-ons, all seamlessly integrating with the rest of Orange, allowing components and code reuse. This allows anyone to leverage the power of the orange canvas to suit speci c needs.
Figure 4(b) shows the Orange schema of the code fragment in Figure 4(a). Figure 4(c) shows the details of the Test Learners widget and the parameters that can be explored. Notice that unlike the code fragment details about the Area Under the Curve and Classi cation Accuracy and various other metrics available to the user.
The Image Engineering extension developed provides widgets designed to process image streams. The extension provides a complete work ow from image stream, to image processing, to feature vector, to machine learning, to classi er, to application. Some bespoke widgets were developed for functionality not provided by image processing libraries. For example in our systems, widgets were developed to calculate curvature, surface normals etc. from range images and statistical description of images.
Machine learning algorithms in Orange require a numerical representation of objects. If machine learning is to be performed then a widget that outputs numerical values need to be placed in the work ow as an interface between the machine learning and image processing data ow. When representing images as numerical values, the values might correspond to the pixels of an image, or the frequency of edge pixels, or perhaps a statistical description of an image region depending on the widget used. Widgets developed generally output a response image and numerical representations. Table 1 provides an overview of some of the widgets that have been developed. 5
Recognising objects in a scene is a primary goal of computer vision. The di culty of this tasks depends on several factors such as: the number of objects, the complexity of the object etc. The appropriate techniques for object recognition depend on the di culty of the task. One approach is to identify image patches, or regions of interest, that may contain objects of interest and then focus later processing, potentially computationally intensive, these regions of interest. Using a classi er, built prior, process the image patches to verify or con rm that the object of interest if present. To demonstrate the utility of the Image Engineering extension developed three case studies are provided demonstrating the above approach.
The case studies are based on the task of asset management of street furniture in urban transport corridors using the co-registered imagery and depth data. They have been simpli ed to focus on tra c lights but should be obvious to the reader how they could be extend to manage di erent assets.
In each case study, no knowledge of how to program is required and a basic understanding of image processing is assumed. Work ows can be discovered through experimentation. Simple dragging, dropping and connecting widgets is all that is required. In one example the output from one case study is reused in another. 5.1
This case study uses AAM depth maps to identify interesting regions in co-registered imagery. In a scene objects exist at di erent depths. The intuition is that local peaks in a depth map indicate that an object is closer to the from orange import BayesLearner, kNNLearner, ExampleTable from orngTest import crossValidation from orngStat import computrCDT # set up the learners bayes = BayesLearner(name=’naive bayes’) knn = kNNLearner(name=’knn’) learners = [bayes, knn] # compute accuracies on data data = ExampleTable("voting") results = crossValidation(learners, data, folds=10) cdt = computeCDT(results) # output the results # ...
(a) camera than its surroundings. The following method segments out things that are closer than their surroundings. A detailed description of the process can be found in Borck et al. (2014)
First a histogram equalization algorithm applied to the depth map. Then a maximal lter is used so local peaks stand out. The image is then dilated to expand the size of the local peaks. To remove the background, which essentially is unaltered in the process, the dilated image is subtracted from the output of the maximal lter. This creates an image that contains only local peaks. This local peak image is then thresholded and bounding boxes for each thresholded segment are determined. These bounding boxes are the hypothesised interesting regions and overlaid on the co-registered imagery to visually evaluate the process. For each step a widget is used to process the image stream. Figure 5 show the schema for detecting interesting regions as well as widgets for loading, dilation, background subtraction and the overlay of bounding boxes. Observe that the work ow is capable of processing multiple images without the user needing to understand and complex programming constructs such as looping or data structures.
The process of using positive and negative examples with machine learning algorithms to build a object detection
system is well de ned in the computer vision literature. The real question is which combination of features and
which machine learning algorithm is t for the purpose. This case study describes a work ow to assess image
and depth features, machine learning algorithms and infer an optimal set of features and a classi er to produce
an tra c light detection system. Imagery and depth data were used from the Earthmine system. This case study
implements a simpli ed work ow of a more complex multi class example as described in
The optimal classi er is determined by examining the graphs, confusion matrix, classi er metrics and other visualisations provide by the widgets use in the work ow. Once an optimal classi er has been identi ed it can be saved and used in other work ows. In this case the Support Vector Machine (SVM) was the optimal classi er and is connected to the Save Classi er widget. Observe the two paths for the positive (Tra c Lights) and negative (Background) training examples. The feature vector is built using the HoG, Gabor and Statistics widgets. Also, since the output of the Gabor widget is an image, this need to be transformed into a descriptor. In this case a statistical summary, mean and variance, is used as the numerical representation of the Gabor feature. Notice in the schema where the output of the Gabor widget is connected to the input of the statistics widget. The HoG widget has two outputs, a descriptor and a HoG image. The HoG image allows you to visualise the feature and the descriptor is included in the feature vector. Figure 6(a) is the schema training a classi er to detect tra c lights from background. Canvas widgets can be opened by double clicking on the widget. Figure 6(b) shows open widgets for the Tra c light images, HoG feature, and the data table where each row is a feature vector. This is an example of visually exploring the work ow. 5.3
This case study demonstrates a simple asset veri cation system and is a simpli ed version of a prototyped developed for a government department. Using the tra c light detection system developed earlier it is possible to label an image patch as either a tra c light or background. The Earthmine widget allows access you to specify a location. By using the Earthmine widget known locations were visited and tra c lights were extracted by manually by drawing bounding boxes around the tra c lights. A feature vector of each extraction region was built using the Hog, Gabor and Statistics widgets. The classi er developed in Section 5.2 was loaded and the newly created feature vector of unseen tra c lights were connected to the prediction widget. The prediction widget labels each feature vector as being either a tra c light or background. Figure 7 show the schema of the asset veri cation system. 6
The extension developed is immediately useful for processing imagery and co-registered depth maps. Work is
under way to include functionality provided by the Point Cloud Library
There is an obvious overhead of using the visual environment, from maintaining internal data structures to updating visualisations. Many of the tasks in a schema could be executed in parallel. The functions could be distributed over several processors. The scheduling overhead would be small compared to the computational complexity of of typical image processing functions.
Widget functionality is in uenced by the conceptual grouping of functions from the underlying library. This
may not be the most suitable mapping onto higher level tasks performed by the widgets. Icons are used to express
meaning graphically. The design of good icons is essential. Research into appropriate conceptual mapping of
task and suitable expressive icons needs to be conducted. The cognitive dimensions framework of
Initial feedback on prototypes developed has been positive and the perceived bene t from the user is high but a more detailed investigation into tangible bene ts need to be conducted. 7
An Image Engineering extension to the Orange data analysis framework was developed. This extensions includes widgets for image acquisition, image processing and image analysis. The extension is modular and gives a non programmer user the opportunity to develop powerful image processing applications in a two-dimensional, data ow metaphor visual environment. This allows for fast prototyping and interactive exploration of algorithms and work ows without the need to learn a textual programming language. To demonstrate the utility of the system three case studies were presented. Imagery and depth data of urban transport corridors were used from the Earthmine dataset and laser ranging system.
This work is supported by the Cooperative Research Centre for Spatial Information, whose activities are funded by the Australian Commonwealths Cooperative Research Centres Programme. It provides PhD scholarship for Michael Borck and partially funds Professor Geo West's position. The authors would like to thank John
Ristevski and Anthony Fassero from Earthmine and Landgate, WA for making available the dataset used in this work.
Van Rossum, G. (2003, September). The Python Language Reference Manual. Network Theory Ltd.