Search, retrieval, and management of multimedia contents are challenging tasks for users and researchers alike. The development of efficient systems to navigate through content has recently become an important research topic. Since domains as parliaments, courts, ministries, or security and military forces are producing enormous masses of video, audio and text files, the requirement of a specific content management solution have arisen naturally.
The aim of E-Sentencias is to develop a software-hardware system for the global management of the multimedia contents produced by Spanish civil courts. The Civil Procedure Act of January 7th, 2000 (1/2000) introduces the video recording of oral hearings. As a result, Spanish civil courts are currently producing a massive number of DVDs which have become part of the judicial file, together with suits, indictments, injunctions, judgments and pieces of evidence. This audiovisual material is used by lawyers, prosecutors and judges to prepare, if necessary, appeals to superior courts. Nevertheless, there is no available system at present to automatically annotate audiovisual contents within the judicial domain. E-Sentencias proposes a meta-search engine to manage text (legislation, jurisprudence, procedural documents, etc.), images, graph materials, and audiovisual contents in a dynamic way that combines algorithmic techniques with legal ontologies. Both automatic and semiautomatic processes facilitate the exploitation of the stored information by the users’ website. In this regard, e-Sentencias involves technologies such as the Semantic Web, ontologies, NLP techniques, audio-video segmentation, and IR. The ultimate goal is to obtain an automatic classification of images and segments of the audiovisual records that, coupled with textual semantics, allows the efficient navigation and retrieval of judicial documents and additional legal sources.
Section 2 below describes the current situation concerning the audiovisual recording of civil cases in Spain. In Section 3 we offer an overview of the steps followed towards the construction of a conceptual structure to classify video segments and the development of legal ontology applications. Sections 4 and 5 depict respectively the structure and architecture of the video system prototype at the present stage of research and, finally, we conclude by offering some expected results and conclusions in sections 5 and 6.
The provisions made by the 1/2000 Civil Procedure Act for the video recording of civil proceedings in Spain do not include a homogeneous protocol establishing how to obtain audiovisual records. Rather, and since an ever growing number of Autonomous Governments in Spain hold competencies on the organization of the judicial system there is a plurality of standards, formats, and methods to produce audiovisual records. As a result, analogical and digital standards coexist with different recording formats. The support in which copies are provided to legal professionals (i.e. to prepare an appeal) may also consist of either VHS videotapes or CDs. And, finally, the procedures to store, classify, and retrieve audiovisual records may vary even from court to court.
As regards the basic typology of civil proceedings, the 1/2000 Act sets two declarative processes: the ordinary proceeding and the verbal proceeding. The main differences between the two lie in the value of the case – more or less than Ä3000, respectively – and the legal object at dispute.
The steps of the process also vary depending on the specific proceeding. On the one hand, the ordinary proceeding starts with a separate, independent oral hearing called “audiencia previa” to resolve pre-judiciary issues (documents, evidences to be accepted, etc.), while verbal proceedings take place in the same judicial event. On the other hand, in the ordinary proceeding the claim of the plaintiff is contested in written terms, while in the verbal proceeding is replied orally in the same act.
One of the core objectives of e-Sentencias is to develop automatic classification strategies to classify video segments. To do so, we have started from scratch by transcribing a small set of oral hearings (corresponding to fifteen civil cases). Textual transcriptions also mark the different steps of the oral hearing and include a manual coding of legal concepts (i.e. judgment, injunction, cause of necessity, deed, etc.) and legal expressions (i.e. “with the permission of your Honor”. In addition, they facilitate the coding of practical rules of procedure that are implicit in the video sequences, such as the following piece of transcription shows:
This is only a first level of textual and visual annotation of judicial hearings, but it is also the basis to create specific annotation templates at different levels (concepts, legal formulae, practical rules of interaction, etc.) that facilitate the construction of different types of ontologies.
In practice the use of ontologies for different tasks and purposes
requires to consider the particular task as context for the ontology. The
reason is that ontologies are often not really designed independent of
the task at hand
When talking about viewpoints, we can distinguish two basic use cases: In the first case, the aim is to provide means for maintaining and integrating different existing viewpoints. In the second use case, one may want to extract a certain viewpoint from an existing model that best fits the requirements of an application.
In many application domains (such as law) it is acknowledged that
the creation of a single universal ontology is neither possible nor
beneficial, because different tasks and viewpoints require different, often
incompatible conceptual choices. As a result, we need to support
situations where different parties commit to different viewpoints that
cannot be integrated by imposing a global ontology. This situation
demands for a weak notion of integration, in order to be able to exchange
information between the viewpoints
Law is also a complex domain, where several roles are involved (judge, prosecutor, defendant . . . ). They must be represented from different points of view, thinking of the possible use of the images of the hearings for multiple (and adversarial) purposes.
We find in the recent literature several approaches to this perspective
problem and the so-called ‘semantic gap’: (i) multi-context ontologies
vs. mono-context ontologies
From the multimedia researchers point of view, context is defined
currently as ‘the set of interrelated conditions in which visual entities
(e.g. objects, scenes) exist’
2 Directionality of information flow : keeping track of the source and the target
ontology a specific piece of information; local domains: giving up the hypothesis
that all legal ontologies are interpreted in a single global domain; context mapping:
stating that two elements (concepts, roles, individuals) of two ontologies, though
extensionally different, are contextually related, e.g., because they both refer to the
same object in the word
3 ‘The main idea of our approach lies in a way to associate concepts with instances
that are deemed to be prototypical by their annotators with regard to their visual
characteristics’ (ibid. 2005: 593).
time and space convey stories, so, a video program (raw video data)
must be viewed as a document, not a non-structured sequence of frames’
However, the audiovisual documents that are recorded in Spanish courtrooms do not convey actions, but legal narratives. Motion and colour are generally uniform, since they are not considered the relevant aspect of those documents. Thus, court records are technically very poor (see fig. 5), filmed using a one-shot perspective (the camera is situated above and behind the judge, who never appears on the screen). Rather than telling a story, the video structures a single framework in which a story is referred, conveyed and constructed by the procedural actors (judge, counsels, testimonies, secretary, and court clerks).
Here lies the layered exophoricity of the legal discourse. Actions, events and stories are referred into a contextually embedded discourse, procedurally-driven, and hierarchically conducted by the judge (judgecentered). Therefore, a strong décalage is produced between audio and video as sources of information. A legal court video record would be completely useless without the audio, because we may only infer procedural (but not substantial) items from the motion. What is important is what is said in court, not what is done. Visual images are only ancillary related to the audio stream. This is an important feature of the records, which has to be taken into account in the tasks of extracting, merging and aligning ontologies, because what the different users require (judges, lawyers, citizens) is the combination of different functionalities focused on the legal information content (legislation quoted, previous cases and judgements –precedent-, personal professional records, and so on). This is the reason for a hybrid user-centred approach that is the kernel of our theoretical approach.
The development of an intuitive user interface constitutes a central requirement of the system. While preserving the simplicity of use, the application allows: a) access to the legally significant contents of the video file; b) integration of all procedural documents related to the oral hearing; c) management of sequential observations, and d) semantic queries on the contextual procedural aspects.
The structure of the application is based on two intuitive and semantically powerful metaphors: the oral hearing line and the oral hearing axe. The oral hearing line presents a timeline divided into segments. Each segment represents a different speech, produced by one of participants in the process: judge, secretary, attorneys, witnesses, etc. Each participant is represented by a different color to obtain an identification at first glance of their interventions. Therefore, it is possible to visualize specific contents of the video by merely clicking on a particular colored sequence. Moreover, it is possible to add textual information to any instant of the intervention.
The oral hearing axe consists of a column representing the different phases of the event as defined by procedural legislation. Different phases (as opening statements, presentation of evidences, concluding statements, etc) are represented by different colors, allowing a quick access. It is also possible to access to legal documents related to each phase (i. e. pieces of evidence such as contracts, invoices, etc.) as well as to jurisprudence quoted in the oral hearing and detected through phonetic analysis. This legal information is also structured in directories and folders.
As Figure 6 shows, the user interface is divided into two main parts: the upper part contains the video player, the oral hearing axe and the oral hearing line. The lower part is devoted to external information layers (i.e. references to articles, documents annexed, manual annotations, links to jurisprudence, etc.). This part is divided into two tabs. The first one contains important information of the selected phase, allowing the addition of the different documents presented during the phase. The second tab contains historical information of the process and all the related information available in advance.
The main functionalities offered in the upper part of the user interface are: 1) The information tab: this is a scrollable tab containing the most relevant data of the process. 2) The oral hearing line: the timeline of sequences and interventions assigned to the different actors of the process. One single sequence of the video may contain interventions of different actors. Therefore, sequences may be either mono-colored (intervention of one single part) or multi-colored (more than one part intervening in the same sequence). The horizontal length of each segment of the timeline is proportional to its length in seconds. The application includes two modes of playing video, apart of the usual one. It is possible to select either the visualization of all the interventions by a single participant or, in turn, all the interventions on a given phase. 3) The list of intervening parties: Each actor intervening in the process is represented by an icon. As in the case of the oral hearing line, we may choose to visualize only those sequences appearing one specific participant (i.e. the judge or de defense attorney). 4) The oral hearing axe: this is the vertical line representing the procedural phases of the process. The judicial process is therefore divided in procedural phases which can, as well, be subdivided in interventions. The vertical axe has the advantage of providing quick access to interventions belonging to a given phase.
In addition to these functionalities, it is possible make a manual annotation of the sequence. Double-clicking with the right bottom of the mouse over a sequence running on the video screen opens a pop-up with a manual annotation tool.
As regards the lower part of the user interface, this area contains all the relevant information and documents of the process, but also enables the user to add and organize the information appearing during the different phases. This part is divided into two different sections: 1) An area enabling the visualization of all the references related to each phase of the process. References consist of data (i.e. Civil Code articles, judgments, Internet links, etc.) automatically introduced through semantic annotation. 2) An area including all manual annotations of the sequences made by the user.
The architecture of the system is based on a web system including the following components: 1) Video server WMS: a server based on Windows 2003 Enterprise server with a streaming Windows Media services which allows video broadcast of audiovisual content of the judicial processes under demand. Application server TOMCAT: the application serves web contents and provides the required interaction with the database by means of Java Server Pages; 2) Mysql Database: the Mysql database contains the information related to all processes and their respective annotations; 3) Client browser IE 7.0: It allows the management of the user interface and the management of the user interaction with the embedded Windows Media Player 11 that streams the video.
In the E-Sentencias project we expect to obtain two different types of results. On the one hand, a fully annotated legal corpus of multimedia oral hearings classified in 15 procedural classes, as regulated by the 1/2000 Act. On the other hand, an operational system with a human-computer interface as described in this paper. Using the system prototype, the automatic capabilities of speaker interventions and phases detection will be tested against manually annotated corpus. It will also be evaluated in cross-oral hearings retrieval based on hardware accelerated and specifically implemented multimedia ontologies.
E-Sentencias (E-Sentencias. Plataforma hardware-software de aceleración del proceso de generación y gestión de conocimiento e imágenes para la justicia) is a Project funded by the Ministerio de Industria, Turismo y Comercio (FIT-350101-2006-26). A consortium of: Intelligent Software Components (iSOCO), Wolters Kluwer España, IUAB Institute of Law and Technology (IDT-UAB), Centro de Prototipos y Soluciones Hardware - Software (CHEPIS - UAB) y Digital Video Semantics (Dpt. Computer Science UAB).