Data-driven investigation intelligence systems are critical to the Intelligence and Defense domains to detect and prevent existing, evolving, and emerging criminal activities such as cyber-threats, human traficking, money laundering, frauds, and crime rings. Several technical and usability challenges must be tackled to implement an efective investigative intelligence system. For instance, the ever-increasing volume and complexity of information form a fertile ground for malicious actors to blend in and operate, and investigations often involve connecting the dots on both large quantities of structured (well-defined records), semi-structured (logs), and unstructured data (textual and other media). In a Law Enforcement scenario, this could mean connecting tables of vehicles, cases, and trafic camera license plate readings, while it is common to use concepts such as IPs, MD5 hash values, or user IDs to tie together security logs in a Cybersecurity scenario. Therefore, investigative systems must provide intuitive and scalable ways to search, explore, and analyze large relational datasets. With information that is often interconnected in essence, such systems must also enable a mixed workload of search, data and graph analytics to support users in examining records and their relationships from diferent perspectives. Finally, analysts often interact with the system by following an explorative and iterative process that represents their train of thoughts. Consequently, investigative systems must have fast response times to avoid impeding the mental process of the analysts. With roots in Information Retrieval research applied to the faceted exploration of Linked Data [1, 2], Siren provides an investigative intelligence platform called Investigate [3, 4] which is based on Elasticsearch and on the Siren's Elasticsearch plugin called Federate [3, 5]. The platform gives analysts several data interaction paradigms, such as search, analytic dashboard, set-to-set navigation, and graph visualization, and combines them into a unified and coherent interaction model. For example, the set-to-set navigation helps analysts to express their information needs visually, by connecting together diferent areas of interest. Applying filters on a set has an impact on all (in)directly connected sets, which allows an analyst to view only the relevant information for a given need. For instance, in a Financial Investigation scenario, we could have data on companies that have received investments from investors but are also mentioned by articles and have headquarters in cities. Siren Investigate can move from a set of companies to the set of records connected to it - for example, the investments received by Irish companies
that are mentioned in articles regarding Artificial Intelligence. The graph visualization instead helps an analyst to have an overview of the computed sets, to see if any pattern or clusters emerge. In our example, graph visualization can help to picture the answers to questions like “Which investors invested in which companies? Are they investing in pairs or groups? Are there groups that appear to be investing in competing companies?”.
Eficient relation search capabilities are essential for implementing both set-to-set and graph
visualization paradigms. Regrettably, document-oriented databases like Elasticsearch present
some limitations in joining data. For instance, joining must be planned in advance at document
indexing-time, and the documents to be joined must reside in the same index and on the
same shard for scalability reasons [
In this talk, we will explore how the Siren Federate plugin integrates relational search capabilities into Elasticsearch and how it empowers the Siren Investigate platform to tackle some of the challenges in Investigative Intelligence. Specifically, we will discuss: • Enabling analysts to devise custom data models that describe relationships between records, • Supporting set-to-set navigation for interactive exploration and defining information needs, • Utilizing graph visualization to uncover patterns within the data.