Sign languages (SL) require a fundamental rethinking of many basic assumptions about human language processing because instead of using linear speech, sign languages coarticulate facial expressions, shoulder and hand movements, eye gaze and usage of a three-dimensional space. SL researchers have therefore advocated SL-specific approaches that do not start from the biases of models that were originally developed for vocal languages. Unfortunately, there are currently no processing models that adequately achieve both language comprehension and formulation, and the SL-specific developments run the risk of becoming alienated from other linguistic research. This paper explores the hypothesis that a construction grammar architecture offers a solution to these problems because constructions are able to simultaneously access and manipulate information coming from many different sources. This claim is illustrated by a proof-of-concept implementation of a basic grammar for French Sign Language in Fluid Construction Grammar.
Sign languages (SL) require a fundamental reassessment of
many of the basic assumptions about human language
processing because instead of using linear streams of speech,
sign languages coarticulate facial expressions, shoulder and
hand movements, and eye gaze
Unfortunately, while lexical signs can now reasonably be
recognized and produced in their base forms, many hard
problems remain unsolved when it comes to grammatical
processing
This paper proposes that a construction grammar
architecture
The paper illustrates its claims through a
proof-ofconcept implementation of a basic grammar for French
Sign Language in Fluid Construction Grammar
Sign languages are so-called under-resourced languages,
which means that there are only few reference grammars and
small corpora available
Most traditional theories invite SL-researchers to analyze
SL utterances as a sequence of lexical gestures that are
combined with each other in a phrase-structural analysis.
However, as argued by
Besides rule-based grammars, the increasing availability
of annotated corpora
Sign language researchers have therefore started to
advocate and implement SL-specific methods and technologies.
One example, discussed in more detail by
Construction grammar is a family of cognitive-functional approaches to language that, amongst other reasons, emerged as a reaction against the assumption in mainstream linguistics that grammatical phenomena can be divided into a core (which can be described as a set of rules) and a periphery (which is a list of exceptions). Construction grammarians also reject the assumption that language is divided into different, largely independent modules such as phonology, syntax and semantics. Instead, they argue that all linguistic knowledge can be described using constructions as the central representation unit. Charles J. Fillmore, widely recognized as the father of construction grammar, defined a construction as follows:
pattern which is assigned to one or more conventional functions in a language, together with whatever is linguistically conventionalized about its contribution to the meaning or the use of structures containing it. (Fillmore, 1988, p. 36, italics added)
I italicized the words any and whatever in this quote to emphasize the fact that constructions are able to represent any kind of mapping between meaning/function and form. For instance, as Fillmore (1988, p. 35) explains, constructions are not limited to the immediate-dominance constraints of phrase-structure rules (i.e. every phrase-structure rule is a relation between a parent and its immediate children in a local tree configuration) but can make direct reference to the linguistic information they require, wherever this information may be located.
Constructional approaches are currently thriving in all
areas of linguistics
In recent work, Steels (to appear) has offered a formal specification of a construction that is independent from how constructions are implemented in the FCG-system. I will briefly summarize this specification in this section in order to demonstrate its relevance for the challenges of sign languages. Let us first start from the well-known representation of a phrase structure grammar rule, as shown in example (1).
1It should be noted that Sign-Based Construction Grammar
V P !
A phrase structure rule is a declarative rule that
specifies an immediate-dominance constraint between its left-hand
side and its right-hand side
Example (2) shows how example (1) can be reformulated with feature-value pairs and explicit word ordering constraints using the formal notation that I will apply throughout this paper. The symbol < specifies a linear precedence constraint between the NP and VP constituents, and symbols that start with a question mark are variables whose values are underspecified. For instance, the variables ?p and ?n indicate that the rule underspecifies which values should be assigned to the person and number features of the NP and VP, but the fact that the same variable symbols are used in both constituents imposes an equality constraint between them.
2 category: NP 3 2 category: VP 3 [ category: S ] ! 64 agreement: 7 < 6 agreement: person: ?p 5 4 person: ?p number: ?n number: ?n 7 (2) 5
Constructions, as operationalized by Steels (2004, to appear), are different from such phrase structure rules in two important ways. First, the right-hand side of a construction is divided between a formulation lock and a comprehension lock. These locks specify the conditions under which a construction can be applied in processing. The formalization thus changes the direction of the arrow to the left in order to indicate that information on the left-hand side is only made available if the conditions on the right-hand side are satisfied. Example (3) shows a lexical construction for the word ball (corresponding to the rule Noun ! “ball”) with the formulation lock above the single line in the right-hand side of the construction, and the comprehension lock under the line. The formulation lock specifies that the construction may only apply in formulation if the meaning ball(?x) needs to be verbalized (here we use a first-order logic calculus for representing meaning), and the comprehension lock specifies that the construction may only apply in comprehension if the string “ball” has been observed in the input. 64 rseefmer-ecnatt:: ?idxentifier 57 syn-cat: noun 3
66 # mperaendiincgat:es: fball(?x)g 777 (3) 64 # form: 5 string: “ball”
The second major difference is that the arrow symbol in a construction no longer implies an immediate-dominance relation,2 but should be read as “the constraints on the lefthand side can be imposed on the constraints on the right-hand side.” In other words, constructions can directly access information that falls beyond the scope of local tree configurations without resorting to additional formal machinery such as transformations or feature passing.
This second difference is technically achieved by reifying feature-value pairs as units that have a unique name, as can be seen in example (3) above the double line (?ball-unit). Traditional formalisms typically lack such an explicit notion of units so the only way of accessing information is through path descriptions of nested feature structures. In a construction, feature structures can be retrieved directly by either using a unit’s name, or by finding a unit whose features match with either the formulation or comprehension lock of a construction.
Crucially, there are no restrictions on which feature-value pairs can form a unit. For example, the Subject-Verb construction in example (4) imposes a linear precedence constraint between a clause’s subject (i.e. accessing the utterance’s functional structure) and its main verb phrase (i.e. accessing the utterance’s constituent structure). Example (5) shows the Topic construction, which imposes a clause’s topic to take up clause-initial position, thereby accessing the utterance’s information structure.
2 ?clause-unit 3 666666 smurpebearufenendirinietcsgna::tt:fe?s?:seuvfbvjaenctta-ugne-itp,o?ivnpt(-?uenvi,t xg)g 777777 4 sem-cat: proposition 5
syn-cat: clause 2 ?subject-unit 3 2 ?vp-unit 3 66666664 ssmeyrrsemneeua-ff-bnf-ecujieraennxetcgcn:pt:ttri:eos?nsx:ion 77777775 < 666666664 svmearpfemleearsfe-unnecdbciran-eejteeg:n:xc:tpt:(r??esesuvsbiojenct-unit)g 777777775 syn-cat: NP syn-cat: VP (4) " ?topic-unit pragm-function: clause-topic # 2 ?topic-unit 3 466666666 #mfr#eeoappfrnofemriscrenei:ldtganiiu:oct:snaet?:e-xctsol:apuics(e?-einvi,t?iaxl)g 577777777 (5)
syn-cat: NP
2In other words, a construction does not impose any restrictions
on its left-hand and right-hand sides and therefore has the
expressive power of an unrestricted grammar
As demonstrated by
Since there are no constraints on which sets of
featurevalue pairs can be reified as units, constructions can easily
specify units that pertain to SL-specific properties such as eye
gaze, hand and shoulder movements, and the signing space.
Moreover, form constraints in the FCG-system make
reference to units rather than to actual words, so there is no need
to align for instance gestural units with lexical boundaries,
which was one of the main problems of traditional formalisms
Parametric Variation. One important obstacle in sign
language modeling is parametric variation in the expression of
gestural units. That is, gestural units do not simply appear in
their citation form when used in continuous signing, as
illustrated before in Figure 1. Instead of “reading off” lexical units
from a linear representation device such as a phrase structure
tree, SL-specific models such as AZee therefore take a more
procedural approach in which language formulation yields a
set of form specifications that can be visualized using avatar
software
Example (6) shows a construction that captures the citation form of the sign for [BALL]. Let us first look at the right-hand side of the construction. The formulation lock contains the same meaning representation as used for the vocal word ball in example (1). If the speaker wishes to express this meaning, the remainder of the construction is thus unlocked. Since the sign for ball is a highly lexicalized sign, the comprehension lock contains a pointer to the sign’s citation form. That is, in language comprehension the information of this construction is unlocked if the sign for [BALL] is recognized.
If a sign is conventionally associated with a number of parameters, these can be incorporated as feature-value pairs in the left-hand side of the construction, as shown for the parameters loc (location) and rad (radius). Note, however, that the values of these features are variables, which means that their actual values are underspecified. The construction also specifies a type for the sign, which I will explain in more detail in a few paragraphs.
3There is also technical congruence between FCG and state-ofthe-art SL systems: FCG has been developed in Common Lisp, which uses S-expressions that can be translated quite straightforwardly into the XML specifications of other systems.
As mentioned before, French Sign Language users can express the concept of a big ball by increasing the radius of the circular movement made with both hands. In order to capture this parametric variation, we can implement a construction that does not have its own citation form, but instead applies to the citation form of another sign and thereby modifies it. Example (7) shows how such a construction might look like. In its comprehension lock, the construction specifies that if a sign is recognized with up as a value for the form-feature rad (that is, a radius was detected that is significantly larger than the sign’s prototypical radius), then this can be mapped onto the concept of bigness. The left-hand side of the construction is empty because no additional information needs to be imposed on the sign.
When applied in formulation, the construction specifies that the concept of bigness can be expressed by increasing the radius of a sign. Obviously, this construction is not general enough because the radius feature is not appropriate for all signs. The concept of [BIG] can thus be better implemented as an operation whose outcome depends on the type of the sign that it modifies, similar to method specialization in object-oriented programming. For example, in Lisp we can implement a generic function for the modifier modify-size that takes a type and a value as arguments: (defgeneric modify-size (type value) (:documentation "Controls the size
parameters of a sign.")) ‘(rad: ,value))
Sup++pose that the sign for ball is typed as circle, we can now write a specialized method that has to return an appropriate feature structure: (defmethod modify-size ((type (eql ’circle)) (value t))
The FCG-system includes special operators for calling
such methods at processing time. In the formal notation, a
call to a method is indicated by the string ++ followed by
the method name and its arguments. For example, the
notation ++modify-size(circle, up) will call the method that
specializes on the type circle and return the feature-value pair
rad: up. The new definition of the construction for bigness
is shown in example (8). Note that the call to the operator
modify-size passes the variable ?type as the operator’s first
argument. This variable name is also used as the value of
the feature type just above, so its value will be bound to the
actual type of the sign that is being modified.
2 ?unit-x 3
6466666666 fsmoetr#+ermyema+pfp-nem:creirean:eodtg?n:di:tctiiy:fdaypt?ee-exnssti:izfifeeb(r?igty(p?xe,)gup) 7757777777
syn-cat: noun
(8)
Multilinearity. A second important challenge is to handle
“the multilinearity and the complex synchronisation patterns
involving all (manual and non-manual) articulators of SL”
Following an OSV pattern, the first part of the utterance
involves the two-handed sign CAR followed by its placement
in the signing space.
2 ?placement-unit 3 66 sufb?unnoitusn: , ?eye-gaze, ?classifierg 777 66 referent: ?x 7 666664 fsoepormmlrai-e:ccnaettma:terienoftn-::e?x?tpoarrregiesenstitoatnion 777757 syn-cat: NP 2 ?noun
3 66 rseefmer-ecnatt:: ?x 7 466 syind-ecnatti:fier 5777 < noun 2 ?eye-gaze 3 / 0 466 foermye:-gaze-direction: 577 ?target
/ 0 66664666 foahoprmrlratainie:ccdneustmlahataetiopnoretn::::????tahoararrtginiceeduntslthaaattoiporen 77777577
syn-cat: classifier
Let us again first look at the right-hand side of the construction. The pattern consists of three units (which I called ?noun, ?eye-gaze and ?classifier) that need to be synchronized with each other. First, ?noun is a gestural unit that in our example matches with the sign CAR. The symbol < indicates that it precedes the ?eye-gaze unit, which specifies that the eye gaze should be directed at a particular location in the signing space (9) (here: underspecified through the variable ?target). The symbol indicates that this unit in turn must immediately precede the ?classifier unit with potentially some temporal overlap between the units. The ?classifier unit is bound to the same location in the signing space by repeating the variable ?target as the value of its placement feature. This unit can also be parametrized in terms of which articulator was used (here: the signer’s weak hand), which handshape is required (here: a specific classifier for vehicles), and which orientation the signed object takes.
The left-hand side of the construction specifies that these three units can be grouped together into a higher-level unit. For want of a more appropriate terminology, the construction states that together these units can be seen as a referring expression or a noun phrase. The target and orientation of the signed object are percolated upwards to this ?placementunit by repeating the variables ?target and ?orientation in the unit’s form feature.
One important side-note is that grouping together the three units on the right-hand side as subunits of the ?placement-unit does not prevent those units to be member of other groups as well. Indeed, instead of the single dominance relations of phrase structure trees, constructions allow units to be member of multiple higher-level units at the same time in order to facilitate information access and to make the formal synchronisation across units more flexible.
Sign languages have unique properties that make them an indispensable object of study for our understanding of human cognition in all of its complexities. The differences with vocal languages are so vast that many assumptions in (computational) linguistics about linguistic representations and language processing must be reconsidered. State-of-the-art models in sign language research have therefore been developed from the ground up in order to eliminate the biases of traditional approaches based on speech. However, as a result, the field risks becoming alienated from the rest of linguistics, and there currently are no models available that adequately handle both language formulation and comprehension.
This paper proposed that construction grammar offers a solution to both problems because constructions are able to simultaneously access and manipulate various sources of linguistic information, which makes it possible to handle issues such as parametric variation and multilinearity that are at the heart of sign language grammar. The paper supported this argument through a proof-of-concept implementation in Fluid Construction Grammar that works for both formulation and comprehension. While it is clear that much fundamental work remains to be done in all areas of SL research (in terms of modeling, learning and corpus annotation), this paper has shown that a constructional approach holds much promise for achieving broad coverage grammars of sign languages.
The research reported in this paper was conducted at and funded by the Sony Computer Science Laboratory Paris. I wish to thank my colleagues for their constructive feedback and comments, particularly Luc Steels (ICREA Institut de Biologia Evolutiva, UPF-CSIC), Katrien Beuls (VUB AI-Lab), Miquel Cornudella Gaya and Paul Van Eecke (Sony CSL Paris). I would also like to thank Annelies Braffort (LIMSICNRS) for the interesting discussion and for sharing with me her expertise on the challenges and issues in Sign Language modeling.