Progress in statistical learning in recent years has enabled computers to recognize objects with near-human ability. However, recent studies have revealed particular drawbacks in current computer vision systems which suggest there exist considerable differences between the way these systems function compared with human visual cognition. Major differences are that: 1) current computer vision systems learn high-level notions directly from the low-level feature space and ignore the mid-level representations, which makes them difficult to incorporate background knowledge. 2) typical computer vision systems learn visual concepts discriminatively instead of encoding the knowledge necessary to produce a visual representation of the class. In this paper, we introduce a framework referred as Logical Vision which is demonstrated on learning visual concepts constructively and symbolically. Given a set of images, a set of first-order logic formulae of background knowledge and a set of examples of target visual concepts, Logical Vision extracts logical facts concerning geometrical elements from an image by sampling low-level features guided by the background knowledge and conjecturing geometrical elements as output. It first extracts logical facts of mid-level features, then generative Meta-Interpretive Learning technique is applied to learn high-level notions because it is capable of learning recursions, inventing predicates and so on. Owing to its symbolic representation paradigm, in our implementation, Logical Vision is fully implemented in Prolog apart from low-level image feature extraction primitives. In our implementation, Logical Vision was used to extract polygon edges as mid-level symbols, and a generalized Meta-Interpreter Learner was applied to learn high-level geometrical notions. Experiments are conducted on learning shapes (e.g. triangles, quadrilaterals, etc.), regular polygons and right-angle triangles. These demonstrates that learning visual concepts constructively and symbolically is effective.
Computer vision is a sub-field of Artificial Intelligence which aims at the analysis and interpretation of visual information. It is an information-processing procedure that receives in input raw and low structured data, and gives outputs of explicit, meaningful descriptions of the structured information.
Computer vision can be categorized into high-level and low-level vision. Low-level
vision was aimed at image processing tasks and identification of local features. By
contrast, high-level vision was aimed at delivering overall scene analysis in terms of
relations between objects in the scene [
Recently, deep neural networks (DNNs) [
For example, it is easy to produce images that are completely unrecognizable to
humans, though state-of-the-art visual learning algorithms believe them to be recognizable
objects with over 99% confidence [
ySource: http://www.papua-insects.nl
Moreover, humans can typically learn from a single visual example [
In this paper we consider the approach of using modern ILP techniques to support
the incorporation of background knowledge in the generation of scene analysis in terms
of high-level relations. For this purpose we propose a novel visual concept learning
framework, called Logical Vision, to realize this symbolic visual processing paradigm.
Logical Vision first uses background knowledge on mid-level symbols to guide the
sampling of low-level features, then it uses the sampled results to revise previously
conjectured mid-level symbols. With the extracted mid-level feature symbols as background
knowledge, a generalized Meta-Interpretive learner [
The rest of this paper is organized as follows. Section 2 presents some related works. Section 3 proposes the Logical Vision framework. Section 4 describes the implementation of the M etagolLogicalV ision approach, followed by experimental results in Section 5. Finally, Section 6 concludes and discusses about future works. 2
State-of-the-art computer vision algorithms are mostly deep neural networks (DNN)
which have been trained from large-scale datasets, such as [
Besides of the end-to-end visual learning paradigms, there exists another classical
idea which tries to parse images hierarchically like human cognition, e.g. [
To incorporate background knowledge into statistical learning, many algorithms has
been proposed in the last decade [
More closely related works are those approaches which also adopt symbolic
learning paradigm like Logical Vision. A representative work in this branch is [
In this section we introduce the framework of Logical Vision. The input for Logical Vision consists of a set of geometrical primitives BP , one or a set of images I as background knowledge, and a set of logical facts E representing the examples as the target visual concepts. The task is to learn a hypothesis H that defines the target visual concept where BP ; I; H j= E.
Given an input image, Logical Vision first alternately conjecture about mid-level visual objects and samples low-level features to support or revise those conjectures. After obtaining mid-level features, a meta-interpreter can be executed for learning the target visual concepts. 3.1
The purpose of mid-level features extraction is to obtain necessary logical facts BA representing mid-level features of I 2 I for target visual concepts learning by ILP.
The mid-level feature extraction in Logical Vision is realized by repeatedly executing a “conjecturing and sampling” procedure. It uses mid-level feature conjectures to guide the sampling of low-level features, they are then used to revise previously constructed conjectures.
Here the low-level features are referred to local visual metrics such as color information, gradients, SIFT and SURF descriptors, etc. The term “mid-level feature/symbol” is a relative concept: they are logical facts that represent possible sub-parts or components of higher-level concepts. For example, a low-level feature “color gradient” can be useful for describing mid-level features such as edge discovery or contour extraction. However, if the target concept is “butterfly”, the mid-level feature “edge” can be seen as sub-parts of other higher-level concepts like “shape” and “region”. Together with more features like “color pattern”, “region size” and background knowledge about “position relations” and so on, we can finally learn the concept of butterfly within a pure symbolic paradigm by ILP (see Figure 1).
The intuition of the “conjecturing and sampling” process is an analogy to human vision process. Suppose a man stands in front of a huge wall painting, which is very large that we can only clearly observe a small region at one time. To get a whole picture of this painting, he can try to move his eyes around to see different small regions in the painting and guess about the entire view of it. During the observation, he either can sample more details to support his conjecture, or can revise them by doing more sampling. After doing enough samples, he will believe that his final conjecture is the ground truth.
Formally, mid-level feature extraction of an image I 2 I could be described as follows: 1. Sample low-level features F in a subarea (e.g. surrounding a focal point) of I, then add F into the sampled low-level features set F . 2. Conjecture a mid-level feature (edge, region, texture, etc.) C according to F . 3. Validate the conjecture C on image I by doing few more samples. If the validation failed, reject C and go to 1, otherwise go to 4. 4. When C is valid, add it to mid-level feature set BA, then remove the low-level features f (C) that encapsulated by C, the rest of low-level features F 0 = F f (C). For example, if the low-level features are pixels whose local area have a large color variance, the mid-level features to be extracted are contours on the image: once a conjectured contour C has been accepted to BA, we should remove all other pixels on the contour, so f (C) = fpixeljpixel is on contour Cg. 5. If F 0 = , terminate the construction procedure and return BA, otherwise go to 1.
Briefly speaking, Logical Vision uses mid-level feature conjectures to guide the sampling of low-level features, then uses sampled results to revise previously obtained conjectures. The low-level features themselves like pixel colors, local color variances or gradient directions are usually redundant and trifling for representing higher-level visual concepts. After the background-knowledge-guided extraction, they can be compactly abduced into logical symbols such as edges, regions, textures, etc., to serve as bases for higher-level concepts learning.
This human-cognition-mimic paradigm assumes that the mid-level conjectures are
constructed by pre-defined predicates, i.e. background knowledge. This assumption is
reasonable because a person need certain knowledge about particular simple primitives
to learn more complicated concepts. For example, we have to define “gradient” before
we learn the notion “contour”, and we have to understand “line segment” before we
learn the concept “polygon”. This procedure follows a symbolic learning paradigm,
thus it can be easily implemented by logic programming tools like Prolog. This
ensures that Logical Vision will hardly introduce typical computer vision operations such
as sliding windows or image filtering. This is because we believe that in human
cognition, observation is a process that happens only when necessary rather than a exhaustive
enumeration, and background-knowledge/conjecture-guided abduction is a proper way
to model this kind of action. By incorporating first-order formalism in feature
extraction procedure, we wanted to show that symbolic reasoning is able to solve particular
computer vision tasks which have been long time considered as a statistical problem.
After obtaining mid-level logic symbols BA, Logical Vision uses a generalized
MetaInterpretive Learner to learn target visual concepts. The input of generalized
MetaInterpretive Learning (MIL) [
Output: Hypothesis of the target visual concept H; Start: Initialize edge points set F = and sampled edges set BE = ; Randomly sample some edge points fP1,P2,: : :g, let F = F [ fP1,P2,...g; repeat
Select a pair of edge points P1,P22 F ; Validate whether P1P2 forms an edge by querying edge(P1,P2,N ); if edge(P1,P2,N ) succeeded then
Extend P1P2 on both of its directions to form a conjecture of edge C; BE = BE [ C;
Remove all edge points P2 F that lies on edge C; else
Randomly sample a line which crosses the line segment P1P2 for new edge points, if
they are not encapsulated by any sampled edge in BE then add them into F ;
end if
until F = ;
Find connected edges in BE to construct facts of polygons BA;
Learn a hypothesis H with BA; M etagolLogicalV ision; BP ; E through MIL;
Return: H.
this form we see that B; :E is given to the meta-interpreter where :E is a goal and the
resulting abducted program :H represents a headless Horn clause. The Prolog
implementation of generalized meta-interpreter is showed in Table 1. In this work, we used
dyadic meta-rules [
Below we describe the implementation of Logical Vision for the task of polygon shapes learning. The target concepts of this task are definitions of different kinds of polygons (e.g. triangles, regular polygons, etc.). Our implementation is displayed as Algorithm 1, which is referred to as LogicalV isionP oly. 4.1
To learn the concepts of polygon shapes, we targeted the mid-level features BA to be extracted as polygons. They are denoted as polygon(Pol i, [Edge1,...,EdgeN]). The process of polygon extraction can be split into two stages: edge discovery and polygon construction. For simplicity, in the polygon construction stage, LogicalV isionP oly groups of connected edges as a list as polygon.
Therefore, the major challenge is to discover those edges. Following the framework in section 3.1, we introduce some primitives as background knowledge to perform the conjecturing and validation. For example, the background knowledge for “edge” is defined as follow: (a) (b) (c) edge(P1,P2,0):-midpoint(P1,P2,P),edge point(P1),edge point(P2), edge point(P). edge(P1,P2,N ):-midpoint(P1,P2,P),edge point(P1),edge point(P2),
N 1 is N 1,edge(P1,P,N 1),edge(P,P2,N 1). in which P1 and P2 are the input conjectured end points of an edge, N is the recursion limit that controls the depth of edge validation and midpoint/3 finds the midpoint between two pixels. By conjecturing (P1, P2) to be an edge, the predicate would recursively sample the points between (P1, P2) and test whether do they have a high local variance (color gradient) as well. The color gradient test is performed by edge point/1, it is the only primitive that interacts with low-level features on images, it returns true when the color gradient magnitude of pixel P exceeds a pre-defined threshold. The color gradient is computed as follow: where
q G(A) =
Gx(A)2 + Gy(A)2
are Sobel filters, A is the focal point that has been queried, denotes the 2-dimensional
signal processing convolution operation. Gx and Gy represents the image derivatives
in vertical and horizontal directions respectively. We implemented an image-processing
program by OpenCV [
The domain specific primitives BP of M etagolLogicalV ison include some necessary background knowledge for learning polygon shape related target concepts. For example, in order to learn the concept of regular polygon, M etagolLogicalV ision uses angle list/2 to obtain all inner angles of a polygon and uses std dev bounded/2 to test whether the standard deviation of a list of double numbers is bounded by an automatically learned threshold.
Moreover, the stochastic implementation of polygon extraction occasionally discovers redundant edges (see Figure 4), therefore the BP of Logical Vision included a postprocessing primitive rmv rdndnt(L1, L2, T). It enables M etagolLogicalV ision to learn a threshold T for automatically removing extra edges. Here L1 and L2 are the input and output edge lists respectively, the threshold T controls the degree of refinement: consider 2 connected edges AB and BC, if (jABj + jBCj)=jACj T, then rmv rdndnt/3 choose to omit them and construct a new edge AC. 5
In this section we describe our experiments which compare M etagolLogicalV ision with statistics-based computer vision approaches on tasks of learning simple geometrical concepts from binary colored images. 5.1
Materials 3 labeled image datasets are generated for polygon shape learning. For simplicity, the images are binary-colored, each image contains one polygon. Target concepts of the 3 tasks are: 1) triangle/1, quadrilateral/1, pentagon/1 and hexagon/1; 2) regular poly/1 (regular polygon); 3) right tri/1 (right triangle). Note that in the third task, we used the best hypothesis of triangle/1 learned in the first task as background knowledge. Part of the datasets are presented in Figure 3. All datasets were partitioned into 5-folds respectively, 4 of them were used for training and the remainder for testing. The details of all tasks are showed as follows:
Learning polygon shapes: We randomly generated 40 images of triangles, quadrilaterals, pentagons and hexagons respectively. Each image contains one polygon. Each (a) Learning triangles, quadrilaterals, etc.
(b) Learning regular polygons (c) Learning right-angle triangles polygon is black-colored, displays on white canvas. Target concepts in this task are triangle/1, quadrilateral/1, pentagon/1 and hexagon/1.
Learning regular polygons: We randomly generated 10 images of regular and irregular triangles, quadrilaterals, pentagons and hexagons respectively, so there are 80 images in total. In this task, the target concept is regular/1.
Learning right-angle polygons: We randomly generated 40 images of right-angle triangles as positive examples. The negative examples consist of 10 triangles, quadrilaterals, pentagons and hexagons respectively. In order to learn a correct definition of right-angle triangle, the generated quadrilaterals, pentagons and hexagons in negative example set may contain right-angles with probability 0:4. Please note that in this task, M etagolLogicalV ision re-used the best hypothesis of triangle/1 that learned from the first task as background knowledge. Target concept of this task is right tri/1. (a) (b) (c) LogicalVisionPoly: This is the proposed approach. In order to handle the noise caused by polygon extraction (e.g. Figure 4), for each image we ran the extraction procedure five times independently to duplicate the input instances (both for training and testing). During evaluation, the learned hypotheses were tested on all the five extracted polygons and the final prediction was based on an equal weighted vote.
Statistics-based Learning: We used a popular statistics-based computer vision
toolbox VLFeat [
In our experiments, it produces a 34048-dimensional vector for each image.
– LBP: Local binary pattern (LBP) [
where list length/2 returns the length of a list; angles list/2 returns a list of sizes of polygon’s angles. std dev bounded/2 examines whether the standard deviation of a set of real numbers is bounded by a threshold; has angle/2 checks whether a list (of angle sizes) contains a real number within an error bound; rmv rdndnt/3 removes redundant edges as section 4.2 introduced. For the right tri/1 task, we included the best hypothesis in the triangle/1 task as background knowledge.
From the results we can see that the performance of LogicalV isionP oly is significantly better than the compared statistics-based vision learning methods on these tasks, which suggests that symbolic learning can be of benefit to visual concepts learning.
This paper studies a novel approach to the problem of visual concept learning, distinct from that employed by traditional computer vision learning algorithms. By using the proposed Logical Vision approach, we are able to exploit background knowledge flexibly and effectively in visual concepts learning tasks. Owes to its symbolic paradigm, the background-knowledge-guided mid-level features extraction and high-level visual concepts learning can be simply implemented by logic programming languages such as Prolog. The experimental results indicate that the proposed framework has potential to analyze which are traditionally hard for more statistically-oriented approaches.
The main reason for LogicalV isionP oly outperforming statistics-based computer vision learners in our experiments is because it enables the incorporation of first-order background knowledge which is very useful for learning the target concepts. Logical Vision exploits background knowledge in two ways: the first one is using high-level background knowledge to guide the observation of low-level features. This mechanism is reflected by the usage of the predicate edge/3 in section 4.1. The second one is making use of background knowledge as primitive predicates like general Inductive Logic Programming approaches, e.g. the predicates list length/2, angles list/2, etc. defined in M etagolLogicalV ision. However, it is difficult for the statistical learning algorithms to incorporate these kinds of prior knowledge in its learning processes. For example, the HOG feature descriptor exploits local gradient information exactly same as the edge point/1 predicate, but its statistical learning framework lacks of a effective methodology to include other complex background knowledge to organize them into more informative mid-level features. On the other hand, the CNN feature descriptors can encode mid/high-level features of certain degrees, yet this ability could hardly be obtained from a small dataset.
Different to the statistics-based computer vision systems, Logical Vision treats the visual learning problem more likely to human cognition. Its learning process relies much more on the background knowledge than the quantity of data. Besides, the learned theory also can serve as background knowledge for subsequent tasks. Furthermore, it learns a constructive theory to define the target concept, rather than discriminative models that only tell differences between classes. From this angle of view, it is not hard to understand why did LogicalV isionP oly outperforms the statistics-based methods in the experiments. 6.2
In this work, we built a primary system targeting for simple geometric concepts learning
tasks such as polygon shapes learning. Although polygon identification itself cannot be
directly applied to general computer vision tasks, it can be seen as a basic step for
further visual recognition. As we illustrated in section 1, complex cognition tasks can be
decomposed into easier sub-problems hierarchically [
In recent future extensions of this work we hope to extend our study to more complicated tasks such as images involving a multiplicity of overlapping colored polygons as the first step. 7
This research was supported by the National Natural Science Foundation of China (61333014, 61321491), the RAEng Newton Research Collaboration Programme (NRCP/1415/133) and the Program B for Outstanding PhD candidate of Nanjing University.