Theories of the processing and representation of spatial form have to take into account recent results on the importance of holistic properties. Numerous experiments showed the importance of “set properties”, “ensemble representations” and “summary statistics”, ranging from the “gist of a scene” to something like “numerosity”. These results are sometimes difficult to interpret, since we do not exactly know how and on which level they can be computed by the neural machinery of the cortex. According to the standard model of a local-to-global neural hierarchy with a gradual increase of scale and complexity, the ensemble properties have to be regarded as high-level features. But empirical results indicate that many of them are primary perceptual properties and may thus be attributed to earlier processing stages. Here we investigate the prerequisites and the neurobiological plausibility for the computation of ensemble properties. We show that the cortex can easily compute common statistical functions, like a probability distribution function or an autocorrelation function, and that it can also compute abstract invariants, like the number of items in a set. These computations can be performed on fairly early levels and require only two well-accepted properties of cortical neurons, linear summation of afferent inputs and variants of nonlinear cortical gain control.
Recent evidence shows that our representation of the world is essentially determined by
holistic properties [
The major difficulty concerns the question of what exactly is a low-level and a
highlevel property. Gestalt theorists already claimed that features considered high-level
according to a structuralistic view are primary and basic in terms of perception. Further
doubts have been raised by global precedence effects [
One of the most abstract ensemble properties seems to be the number of elements
in a spatial configuration. However, the ability to recognize this number is not restricted
to humans with mature cognitive abilities but has also been found in infants and animals
[
The above observations on ensemble properties raise a number of questions, from which the following are addressed in this paper: Sect. 1: Can the cortex compute a probability distribution? Sect. 2: And also an autocorrelation function? By which kind of neural hardware can this be achieved? Sect.3: Can the shape of individual objects also be characterized by such mechanisms? Sect. 4: What is necessary to compute such an abstract property like the number of elements in a spatial configuration? Can this be achieved in early sensory stages?
Formally, the probability density function pe(e) of a random variable e is defined via the cumulative distribution function: pe(e) , dPe(e) with Pe(e) = Pr[e e]. Their empirical de counterparts, the histogram and the cumulative histogram, are defined by use of indicator functions. For this we divide the real line into m bins (e(i), e(i+1)] with bin size D e = e(i+1) e(i). For each bin i, an indicator function is defined as An illustration of such a function is shown in Fig. 1a. From N samples ek of the random variable e we then obtain the histogram as h(i) = N1 Â kN=1 Qi(ek). The cumulative histogram He(e) can be computed by changing the bins to (e(1), e(i+1)] (cf. Fig. 1b), and by performing the same summation as for the normal histogram. The reverse cumulative (a) (b) (c) histogram H¯ (i) is simply the reversed version of the cumulative histogram. The corresponding bins are D ei = (e(i), e(m+1)] and the indicator functions are defined as (Fig. 1c) Albrecht and Hamilton (1982) (b)
How does all this relate to visual cortex? Has the architecture shown in Fig. 2a any neurobiological plausibility? The final summation stage is no problem since the most basic capability of neurons is computation of a linear sum of their inputs. But how about the indicator functions? They have two special properties: First, the indicator functions come with different sensitivities. An individual function does only generate a non-zero output if the input e exceeds a certain level, a kind of threshold, which determines the sensitivity of the element e(i) in Eq. (2) and Fig. 1c. To cover the complete range of values, different functions with different sensitivities are needed (Fig. 2a). Second, the indicator functions exhibit a certain independence of the input level. Once the input is clearly larger than the threshold, the output remains constant (Fig. 1c).
Do we know of neurons which have such properties, a range of different
sensitivities, and a certain independence of the input strength? Indeed, cortical gain control (or
normalization), as first described in early visual cortex (e.g. [
The computation of a reverse cumulative histogram thus is well in reach of the cortex. We only have to modify the architecture of Fig. 2a by the smoother response functions of cortical neurons. The information about a probability distribution available to the visual cortex is illustrated in Fig. 3. The reconstructed distributions, as estimated from the neural reverse cumulative histograms, are a kind of Parzen-windowed (lowpass-filtered) versions of the original distributions.
A key feature of the recent statistical summary approach to peripheral vision [
(4)
where autocorrelation results if e(k) = g(k) and where indicates multiplication. With
respect to their neural computation, the outer summation is no problem, but the
crucial function is the nonlinear multiplicative interaction between two variables. A
neural implementation could make use of the Babylonian trick ab = 14 [(a + b)2 (a b)2]
[
Of course there will be differences between a formal autocorrelation function and the neurobiological version, but the essential feature, the signaling of good matches in dependence of the relative shifts will be preserved (Fig. 5).
We extracted different features sr,q from the image luminance function l = l(x, y) by applying a Gabor-like filter operation sr,q (x, y) = (l ⇤ F 1(Hr,q ))(x, y) where F 1 denotes the inverse Fourier transformation and the filter kernel Hr,q is defined in the spectral space. We distinguish two cases (even and odd) which can be seen in the following definition in polar coordinates:
Hre,vqen( fr, fq ) := ( cos2 ⇣ p fr r ⌘ cos2 ⇣ p fq q ⌘
2 2 fr,h 2 2 fq ,h 0 , ( fr, fq ) 2 W r,q , else, with W r,q := {( fr, fq )| fr 2 [r 2 fr,h, r + 2 fr,h] ^ fq 2 [q 2 fq ,h, q + 2 fq ,h] \ [q + p 2 fq ,h, q + p + 2 fq ,h]}, where fr,h denotes the half-bandwidth in radial direction and fq ,h denotes the half-bandwidth in angular direction. Hro,qdd is defined as the Hilbert transformed even symmetric filter kernel.
Various AND combinations of these oriented features (see caption Fig. 6) are obtained by the gain-control mechanism described in Eq. (5). The integration over the whole domain results in global features Fk := RR2 gk(x, y) d(x, y) which capture basic shape properties (Fig. 6).
One of the most fundamental and abstract ensemble properties is the number of elements
of a set. Recent evidence (see Introduction) raised the question at which cortical level
the underlying computations are performed. In this processing, a high degree of
invariance has to be achieved, since numerosity can be recognized largely independent of other
properties like size, shape and positioning of elements. Models which address this
question in a neurobiologically plausible fashion, starting from individual pixels or neural
receptors instead of an abstract type of input, are rare. To our knowledge, the first approach
in this direction has been made in [
The key factor of our approach is a relation between surface properties and a topological invariant as described by the famous Gauss-Bonnet theorem. In order to apply this to the image luminance function l = l(x, y) we interpret this function as a surface S := {(x, y, z) 2 R3|(x, y) 2 W , z = l(x, y)} in three-dimensional real space. We then apply the formula for the Gaussian curvature
K(x, y) =
lxx(x, y)lyy(x, y) lxy(x, y)2
(1 + lx(x, y)2 + ly(x, y)2)2
,
(6)
(7)
where subscript denotes the differentiation in the respective direction (e.g. lxy = ∂∂x2∂ly ).
The numerator of (6) can also be written as D = l 1l 2 where l 1,2 are the eigenvalues of
the Hessian matrix of the luminance function l(x, y) which represent the partial second
derivatives in the principal directions. The values and signs of the eigenvalues give us
the information about the shape of the luminance surface S in each point, whether it
is elliptic, hyperbolic, parabolic, or planar. Since Gaussian curvature results from the
multiplication of the second derivatives l 1,2 it is zero for the latter two cases. It has been
shown that this measure can be generalized in various ways, in particular towards the use
of neurophysiologically realistic Gabor-like filters instead of the derivatives [
The following corollary from the Gauss-Bonnet theorem is the basis for the invariance properties in the context of numerosity.
Corollary 4.1 Let S ⇢
R3 be a closed two-dimensional Riemannian manifold. Then
Z S K dA = 4p (1 g) where K is the Gaussian curvature and g is the genus of the surface S.
We consider the special case where the luminance function consists of multiple objects (polyhedra with orthogonal corners) with constant luminance level. We compare the surface of this luminance function to the surface of a cuboid with holes that are shaped like the polyhedra. The trick is that the latter surface has a genus which is determined by the number of holes in the cuboid and which can be determined by the integration of the local curvature according to Eq. (7). If we can find the corresponding contributions of the integral on the image surface, we can use this integral to count the number of objects. We assume the corners to be locally sufficiently smooth such that the surfaces are Riemannian manifolds. The Gaussian curvature K then is zero almost everywhere except on the corners. We hence have to consider only the contributions of the corners. It turns out that these contributions can be computed from the elliptic regions only if we use different signs for upwards and downwards oriented elliptic regions. We thus introduce the following operator which distinguishes the different types of ellipticity in the luminance function. Let l 1 l 2, then the operator N(x, y) := | min(0, l 1(x, y))| | max(0, l 2(x, y))| is always zero if the surface is hyperbolic and has a positive sign for positive ellipticity and a negative one for negative ellipticity. We thus can calculate the numerosity feature which has the ability of counting objects in an image by counting the holes in an imaginary cuboid as follows:
F =
Z
N(x, y) W (1 + lx(x, y)2 + ly(x, y)2) 23 d(x, y). (8) The crucial feature of this measure are contributions of fixed size and with appropriate signs from the corners. The denominator can thus be replaced by a neural gain control mechanism and an appropriate renormalization. For the implementation here we use a shortcut which gives us straight access to the eigenvalues. The numerator D(x, y) of (6) can be rewritten as D(x, y) = lxxlyy
(luu lvv)2 = (9) with u := x cos(p /4) + y sin(p /4) and v := x sin(p /4) + y cos(p /4). The eigenvalues then are l 1,2 = 12 (D l ± |e |) and we can directly use them to compute N(x, y). Application of this computation to a number of test images is shown in Fig. 7.
50 100 150 200 250 50 100 150 200 250 5 100 0−5 150 −10 200 −15 50 100 150 200 250 −20 250 4.0 rechteckstruktur01c
Recent evidence shows that ensemble properties play an important role in perception and
cognition. In this paper, we have investigated by which neural operations and on which
processing level statistical ensemble properties can be computed by the cortex.
Computation of a probability distribution requires indicator functions with different
sensitivities, and our reinterpretation of cortical gain control suggests that this could be a basic
function of this neural mechanism. The second potential of cortical gain control is the
computation of AND-like feature combinations. Together with the linear summation
capabilities of neurons this enables the computation of powerful invariants and summary
features. We have repeatedly argued that AND-like feature combinations are essential
for our understanding of the visual system [
A somewhat surprising point is that linear summation and cortical gain control, two
widely accepted properties of cortical neurons, are the only requirements for the
computation of ensemble properties. These functions are already available at early stages of
the cortex, but also in other cortical areas [
Acknowledgement This work was supported by DFG, SFB/TR8 Spatial Cognition, project A5-[ActionSpace].