<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Low-Latency Streaming Evaluation of JSONPath Queries</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Jana Kostičová</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Faculty of Mathematics</institution>
          ,
          <addr-line>Physics and Informatics</addr-line>
          ,
          <institution>Comenius University</institution>
          ,
          <addr-line>Mlynská dolina, Bratislava</addr-line>
          ,
          <country country="SK">Slovakia</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>The recent standardization of the JSONPath language underscores the need for eficient JSONPath query evaluation over JSON (JavaScript Object Notation) data. In this work, we focus on real-time scenarios where available methods may have a too high latency. We identify a subset of JSONPath queries that enable eficient evaluation with a constant number of steps per event in the input JSON stream. This constant time evaluation guarantees low and predictable latency, making the approach highly suitable for real-time applications. The algorithm uses a streaming tree transducer (SST) as the underlying formal model. The JSONPath query is compiled into a finite state automaton (FSA) and the transducer keeps track of the evaluation status by storing FSA states in its stack. We present formal background, complexity analysis and implementation details of the algorithm.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;JSON</kwd>
        <kwd>JSON querying</kwd>
        <kwd>JSONPath</kwd>
        <kwd>real-time data processing</kwd>
        <kwd>low latency</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>memory to the output stream. The memory footprint is
typically proportional to the size of the input data.</p>
      <p>Fast and low-latency solutions are becoming a crucial During the streaming processing of JSON data the
part of today’s data pipelines to satisfy the increasing input is read sequentially and similarly the output is
demand for real-time (RT) or near-real-time processing generated sequentially. A stack is typically used to store
(NRT). Such solutions are required to handle data contin- information related to the current tree level of the input
uously, with minimal delay, and overcome the inherent data. If a part of the input data needs to be processed
challenge of limited data availability at any given mo- in memory, it must be temporarily stored in dedicated
ment [1]. Querying data streams emerges as a versatile memory bufer(s). Basically, these parts are processed in
tool in this context - beyond direct output generation of a tree-based manner that inherently results in processing
ifltered data, it serves as a necessary first step for var- delays, as described above. Therefore, real-time scenarios
ious intricate processing tasks, such as data cleaning, demand minimal input bufering.
entity resolution, data validation, data transformation, The presented algorithm is intended to be used in
realand feature engineering [2, 3]. time data pipelines including rapid data transformation,</p>
      <p>This paper focuses on querying JSON data. JSON such as restructuring data for aggregation purposes (e.g.,
(JavaScript Object Notation) [4, 5] is an extremely pop- combining social media posts into a unified format) and
ular tree-based format for data storage and data trans- entity resolution tasks (e.g., matching customer records
mission thanks to its lightweight nature, easy readability, across disparate datasets). However, it is useful at each
and straightforward mapping to object-oriented models. data processing task that requires low-latency querying
In case of tree data, the streaming processing involves JSON data streams.
traversing the data in a specific order (preorder) and us- The main contribution of this paper is two-fold. First,
ing queries to address locations within the tree. The we present formal foundations that enable us to analyze
common query language for JSON is the recently stan- the complexity of the streaming processing of JSONPath
dardized JSONPath language [6]. It is based on the XPath queries. Second, on the top of this formalism, we identify
language used for querying another tree format XML the JSONPath subset that can be evaluated consistently
(Extensible Markup Language) [7]. with low latency and design the algorithm that realizes</p>
      <p>
        Most of the JSONPath processors are tree-based, i.e., such an evaluation. The algorithm consists of two steps:
they load all of the input data into memory and then (
        <xref ref-type="bibr" rid="ref1">1</xref>
        ) the compilation step when the query is compiled into
perform the evaluation. This approach is apparently not a finite automaton, and (
        <xref ref-type="bibr" rid="ref2">2</xref>
        ) the evaluation step based
suitable for RT / NRT scenarios, especially for large input on a tree transducer, when the query is evaluated over
data streams. Delays are introduced by the initial loading an input JSON stream and an output JSON stream is
step, by evaluating the JSONPath queries against the in- generated. The evaluator performs a constant number of
memory representation, or by writing the data from the steps per single event in input JSON stream that ensures
ITAT 2024 Information Technologies – Applications and Theory 2024 predictable consistent latency for RT / NRT processing.
$ kosticova@dcs.fmph.uniba.sk (J. Kostičová) The proposed solution is compared with other streaming
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License JSONPath evaluators.
      </p>
      <p>CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org)</p>
      <sec id="sec-1-1">
        <title>1.1. Related Work</title>
        <p>• A literal value without internal structure that is
either a number value, string value, or one of the
literal names (true, false, null).</p>
        <p>To our best knowledge, ours is the first approach to
analyzing the JSONPath language for its suitability in
scenarios with low-latency requirements. There are two JSON data can be represented as a tree that is obtained
streaming solutions that focus particularly on JSONPath by a natural one-to-one mapping between values and
queries. Both of them were presented before JSONPath internal nodes of the tree. Reading the JSON data stream
standardization, and thus the languages being evaluated then exactly corresponds to the preorder traversal of the
difer from the resulting standard in a few aspects. Json- constructed tree.</p>
        <p>Surfer [8] processes incoming events on the fly, but still We base our JSON abstraction on the model introduced
builds a partial internal representation of input data. It in our previous work [13]. We label the nodes
correcannot process queries with descendat axis and star se- sponding to the array elements by their index - similar
lector. JPStream [9] compiles queries into a finite state technique is used in [14] . We omit literal values since
automaton and combines it with parsing automaton to they are not used in our evaluation algorithm. Section 5
obtain the resulting streaming automaton. The authors describes how they are handled in implementation. We
also present a parallelization mechanism to speed up the formally define two JSON structure types: subtrees with
evaluation. Both of these solutions lack latency analysis, arbitrary root identifier labeled by any object name, and
and it generally varies for diferent queries. trees with the root identified by " 0" and labeled by the</p>
        <p>Much prior research has focused on analyzing the designated root symbol "$".
streaming processing of XPath language. In [10] authors Let Σ be an alphabet of object names. The set of JSON
present a streaming algorithm that evaluates a subset of subtrees over Σ is denoted by Σ. We use dynamic level
XPath called Univariate XPath in (||||) time, where numbers [15] to identify nodes within the JSON subtree.
|| is the size of the input document and || is the size The node identifier (id) is a sequence of numeric
valof the query. The SPEX evaluator [11] compiles XPath ues, separated by point. The root node has some root
query into a network of deterministic pushdown trans- id and the id of any other node consists of the parent’s
ducers and processes it in polynomial combined complex- id, point, and the position of the node among its
sibity. The algorithm proposed in [12] is based on determin- lings. This approach allows us to trivially determine
istic nested word automata and runs in polynomial time. the relations between any two given nodes such as the
Although several of the approaches to XPath streaming ancestor-descendant relation, the parent-child relation,
evaluation possess strong formal foundations, their al- the sibling relation. We include zero in node ids to get
gorithms are not directly applicable to JSONPath due to consistency with the indices of JSON array elements. The
inherent diferences between the two languages. Further- JSON subtree always contains at least the root node, as
more, these solutions focus primarily on overall time or the JSON data must consist of exactly one root value. One
space complexity rather than latency and often support ambiguity exists in this model - the JSON subtree
consistquery constructs like backward axes and predicates that ing of the root node only can represent a root value equal
can lead to unpredictable latency behavior and hinder to either an empty object or to an empty array. However,
their applicability in RT/NRT processing. this does not afect the design of our algorithm, as shown
later.
2. Formal Foundations The JSON subtree  ∈ Σ is then defined as a pair
 = (,  ) where</p>
        <sec id="sec-1-1-1">
          <title>This section presents a formal basis for analyzing the complexity of the streaming JSONPath evaluation. We establish a set of customized formal models represeting JSON data, JSONPath queries and JSON transformations.</title>
        </sec>
      </sec>
      <sec id="sec-1-2">
        <title>2.1. JSON Abstraction</title>
        <p>At the root level, JSON data contain a root value. This
value can be one of the following:
• An object that consists of zero or more unordered
name/value pairs (also called members).
• An array that consists of zero or more ordered
values (also called array elements).
•  ⊆ 2(N0∪{.})* is a set of node ids (shortly
nodes), and
•   :  → (Σ
function.</p>
        <p>∪ N0), Σ
∩ N0, is the labeling</p>
        <sec id="sec-1-2-1">
          <title>Note that the set of nonnegative integers N0 is used both</title>
          <p>
            for creating node ids and for labeling nodes by array
indices. It is required that (
            <xref ref-type="bibr" rid="ref1">1</xref>
            ) node ids in  form a tree,
and (
            <xref ref-type="bibr" rid="ref2">2</xref>
            ) in case of array node, the labels of its children
form an ordered sequence of indices starting from zero.
          </p>
          <p>A JSON tree over Σ is a JSON subtree  = (,  ) over
Σ where
0 ∈ , $ ∈ Σ and  (0) = $.</p>
        </sec>
        <sec id="sec-1-2-2">
          <title>We denote the set of JSON trees over Σ by Σ. An exam</title>
          <p>ple of a JSON tree is depicted in Fig. 1.</p>
          <p>The following notions and notations are used
throughout this paper: Let  be a JSON subtree. We denote the
(finite) set of all labels that occur in  by Λ( ). We refer to
the root node of  by () and to the preorder relation
between nodes ,  ∈  by  ≺ . We define the set of
events over some label alphabet Λ as
(Λ) =</p>
          <p>{( ) |  ∈ Λ } ∪ {}.</p>
          <p>An event stream at  ∈  is then defined recursively as
follows:
() =
 ::= $  *
 ::=  [(, )* ]
 ::= . (child) | .. (descendant)
 ::=  (name selector)
|  (index selector)
| * (”any” selector)
| :: (array slice selector)
| ?() (filter selector )
Filter selectors contain predicate - a boolean expression
over other nodes and literal values. They are evaluated
with respect to the current node. We do not provide
details since their specification is rather complex and, as
mentioned later, they are excluded from the queries
under consideration. We shall use the common term label
selector for the name and index selectors. We use dot
notation for JSONPath queries throughout this paper.</p>
          <p>
            A query starts with a special root selector $. It selects
the root of the input JSON tree where the evaluation
starts. Then a sequence of segments follows. Each
segment consists of (
            <xref ref-type="bibr" rid="ref1">1</xref>
            ) an axis (child, descendant) that selects
the node set to be processed (either children or
descendants of the current node) and (
            <xref ref-type="bibr" rid="ref2">2</xref>
            ) a sequence of selectors
that apply further node filtering on a given node set.
Example 1. Consider the JSONPath query
$.[0].addresses..street with three segments. When
evaluated over JSON tree at Fig. 1, first segment selects
the child of the root node being an array element indexed
by 0, the second segment selects child object named
, and the last segment selects all descendants
named . Formally, the query returns literal values
at nodes 0.0.2.0.0 and 0.0.2.1.0.
( ()) (.0) . . . (.) ,
          </p>
        </sec>
        <sec id="sec-1-2-3">
          <title>Let  be a query and  a JSON tree. We denote the list of</title>
          <p>where .0, . . . , . are children of . The event stream JSON subtrees returned by evaluating  over  according
of the JSON tree  equals to the event stream at its root: to the semantics described in the JSONPath standard by
() = (0). (,  ).</p>
          <p>The structure of the JSON data can optionally be
constrained by a schema. Based on this aspect, we distin- 2.3. Simple Streaming Transducer
guish schema-agnostic and schema-aware approaches
to JSON processing. Since it is pretty common for real- Tree transducers are well-established formal models for
world JSON data streams to be schemaless, in this paper performing tree or forest transformations. We adapt the
we focus on schema-agnostic JSONPath evaluation. This streaming model from our previous work [13] due to
approach introduces some additional issues and restric- its simplicity and ability to cover aspects of streaming
tions; most prominently, it is necessary to deal with an transformations of tree data without any input bufering.
unknown input alphabet and the query set that can be We customize it to work directly on event streams rather
processed with guaranteed low latency is more restricted. than trees. We consider a basic model with a single output
In Section 6 we outline future research that would benefit stream. The simple streaming transducer (SST) is a
7from schema availability. tuple  = (, Λ , Λ , Γ , 0, 0, ), where</p>
        </sec>
      </sec>
      <sec id="sec-1-3">
        <title>2.2. JSONPath</title>
        <p>Based on recently published IETF RFC 9535 [6]
(hereafter referred to as the JSONPath standard), we use the
following grammar to describe the JSONPath queries:
•  is a finite set of states,
• Λ  is the alphabet of input labels,
• Λ  is the alphabet of output labels,
• Γ is an alphabet of stack symbols,
• 0 is the initial state, and
•  is a finite set of rules of the form:
 × (Λ ) × Γ</p>
        <p>→ ((Λ ))* × ( × Γ * ).</p>
        <p>In addition, if (, , ) → (′, ′) ∈  then 
must be a substring of an event stream of some JSON
tree.</p>
        <p>The transducer rules can modify the state of the
transducer, manipulate the top element of the stack, and
optionally generate a sequence of output events. The data
storage mechanism is limited to the state and the stack.
We restrict a single rule to generate a JSON substream.
This, however, does not guarantee that the overall output
is correct JSON stream, as that must be ensured by the
transducer rules.</p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>3. Low-Latency Processing</title>
    </sec>
    <sec id="sec-3">
      <title>Constraints</title>
      <p>In our previous work [13] we have discussed properties
of XML transformations that allow for streaming
processing without input bufering. We adapt them to the
context of low-latency JSONPath query evaluation. We
add the property of local evaluability to classify the query
constructs according to the context necessary for their • Negative step in the array slice selector results in
evaluation. We analyze the constructs of the JSONPath the selection of the array elements in reverse
orlanguage and identify which of them violate these condi- der. This in general leads to reordering of some
tions for some input JSON trees. input subtrees.</p>
      <p>The following facts should be pointed out: First, this • Sequence of reordering selectors within a segment
work is focused on a schema-agnostic approach, where again results in reordering some input subtrees.
no information about the structure of the input data is For example, a sequence of index selectors in
reavailable. When analyzing specific JSONPath construct, verse order is clearly reordering.
the worst-case input data is searched within the group of
all valid JSON data. This can result in the identification Note that the order-preserving property is violated only
of worst-case scenarios that are potentially more severe when dealing with array-based selectors due to the
inthan those encountered in schema-bound environments. herent order of array elements.</p>
      <p>Second, members of an object are unordered in JSON
format. This means that we must consider any member 3.2. Branch-Disjoint Queries
order when looking for the worst-case scenarios.
2 is encountered, it is outputted directly. After
processing 2, the stored copy of 1 is outputted, introducing a
delay proportional to its size. Since we consider
schemaagnostic approach, the size of 1 and consequently the
length of the delay is not upper-bounded.</p>
      <p>The following JSONPath constructs violate the
orderpreserving property for some input JSON tree:</p>
      <sec id="sec-3-1">
        <title>The query  is branch-disjoint if and only if the subtrees</title>
        <p>3.1. Order-Preserving Queries returned by  do not overlap for any input JSON tree
. That is, if 1, 2 ∈ (,  ) are two subtrees
reThe query  is order-preserving if and only if the roots of turned by evaluating  over , it must hold (1) ̸=
the subtrees returned by  are in preorder with respect (2).′. See Fig. 2 b) for an example of a
non-branchto an arbitrary input JSON tree . That is, if 1, 2 ∈ disjoint query that results in copying the subtree 2. Here
(,  ) are two subtrees returned by evaluating  over upon encountering the subtree 2, it needs to be stored
 such that 1 is returned before 2, then it must hold in the memory and outputted later that results in a delay
(1) ≺ (2). of the length proportional to the size of 2. In
schema</p>
        <p>See Fig. 2 a) for an example of a query that results in agnostic approach, both the size of 2 and the delay are
reordering input subtrees 1 and 2 and thus is not order- not upper-bounded.
preserving. Consider that an evaluator with additional This property is violated (for some input JSON tree) by
memory is employed for processing such query. Upon using a descendant axis in some of the query segments
encountering 1 the evaluator copies its contents into because it allows one to return two or more overlapping
memory to store it for later usage (1 needs to be out- subtrees by the evaluation of that query.
putted later) while continuing query evaluation. When</p>
        <sec id="sec-3-1-1">
          <title>3.3. Locally Evaluable Queries</title>
          <p>The query  is locally evaluable if and only if no
forward context is needed to evaluate  for arbitrary
input JSON tree . Formally: let  =  ′  
where  is a segment and  is a list of
segments (possibly empty), let 1 ∈ ( ′, ), 2 ∈
($  , 1), then  does not refer to any
node  such that (2) ≺ .</p>
          <p>In case of query that cannot be evaluated locally, when Figure 3: Low-latency evaluator
the evaluator encounters the subtree 1, it needs to visit
some forward context node  to be able to decide whether
2 needs to be outputted. Thus, it is necessary to store 2 standard JSONPath semantics is necessary to
guaranin the memory to be able to evaluate such query in our tee low-latency evaluation. Filter selectors are excluded
processing model (enhanced with additional memory). although some array-based filter selectors not necessarily
In case of schema-agnostic approach, the size of 2 is not violate local evaluability, it would require a separate
comlimited and introduces delay proportional to that size. plex analysis to define this subset clearly and distinguish</p>
          <p>The following JSONPath constructs violate this prop- it from non-locally evaluable ones.
erty for some input JSON tree: We refer to the identified JSONPath subset as the
lowlatency subset later in this paper, and we mention several
approaches to address some of its limitations in Section
6.
• Negative indices in array-based selectors allow the
access to array elements relative to the end of the
array. To process a node at index − , the
evaluator must locate the ( − )-th child of the current
node, where  is the total number of children.
• Filter selectors can contain relative or absolute
subqueries that may need some forward context
to be evaluated.</p>
        </sec>
        <sec id="sec-3-1-2">
          <title>3.4. Identified Low-Latency JSONPath</title>
        </sec>
        <sec id="sec-3-1-3">
          <title>Subset</title>
        </sec>
      </sec>
      <sec id="sec-3-2">
        <title>Based on the analysis above, we define the subset of</title>
        <p>JSONPath language suitable for low-latency streaming
processing by the following rules for queries:
• A segment contains either a single selector or a
sequence of label and array slice selectors that
preserve the ascending order of selected array
indices.
• Only nonnegative values for array indices in
index/array slice selectors are allowed.
• Only a positive value for the  operator in the
array slice selector is allowed.</p>
        <p>• Filter selectors (predicates) are excluded.</p>
      </sec>
      <sec id="sec-3-3">
        <title>Within the JSONPath query segment, the order of in</title>
        <p>dex selectors must be preserved, while name selectors
can appear in any order due to the unordered nature of
JSON object members. The descendant axis is allowed in
queries, as it typically satisfies the branch-disjoint
condition in most practical scenarios. In cases where the
branch-disjoint condition is violated, a runtime error is
explicitly reported, since the query cannot be processed
within our model’s constraints. This deviation from</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Low-Latency Evaluator</title>
      <sec id="sec-4-1">
        <title>The proposed SST-based evaluator consists of the query compilation step and the query processing step (see Fig. 3.). This section presents both tasks in detail and discusses their complexity.</title>
        <sec id="sec-4-1-1">
          <title>4.1. Query Compilation</title>
        </sec>
      </sec>
      <sec id="sec-4-2">
        <title>The low-latency subset of JSONPath queries defined in</title>
        <p>Section 3 can be rewritten to regular expressions, but
ifrst we need to establish a finite alphabet that constrains
the language described: Let  be a query; then the query
label alphabet Λ(  ) consists of all labels that appear
explicitly within the query  (both the object names and
the indices) and a finite set of indices that fall within the
ranges defined by the array slice operators present in  .</p>
        <p>As mentioned earlier, in the context of
schemaagnostic streaming processing, neither the set of object
names nor the set of array indices is known in advance.</p>
        <p>Since traditional finite automata operate on finite input
alphabets, we need to overcome this issue. It is easy to
observe that when evaluating  , handling of input
symbols not present in Λ(  ) is identical. Therefore, we can
parse these symbols into a single token represented by
a special placeholder label  ℎ. Compilation of a query
into a regular expression is then based on the following
principles:
• Label selector: maps to the label itself.
• Star selector: represents a choice of all labels.
• Array slice selector: represents a choice of indices 4.2. Query Processing
contained within the defined range (we allow only
ifxed upper bound for the array slice selector).
• Selector sequence: represents a choice of regular</p>
        <p>subexpressions for particular selectors.
• Child axis: maps to a concatenation operation.
• Descendant axis: represents a concatenation
followed by a Kleene star over the query alphabet
and the placeholder  ℎ, and another
concatenation (in this order).</p>
      </sec>
      <sec id="sec-4-3">
        <title>We present a query processing algorithm by means of</title>
        <p>a construction of a streaming tree transducer. Let  be
a query and ( ) = (, Σ ,  , 0, ). Then the
simple streaming transducer evaluating  over an input
JSON stream is constructed as follows:</p>
        <p>( ) = (, Λ , Λ , Γ , , 0, )
•  = {, , },</p>
      </sec>
      <sec id="sec-4-4">
        <title>Formally, we then rewrite queries to regular expressions as follows:</title>
        <p>Let  = $ be a query without any segment, then
( ) = $.</p>
        <p>Let  =  ′  [1, . . . , ] be a query with at least
one segment, then
( ′)  ((1) | . . . | ()),
where
•  =  for the child axis,
•  ∈ (Λ(  ) ∪ { ℎ})* for the descendant axis,
and (),  ∈ {1, . . . , } is the regular
expression for the selector  defined as follows:
Label selector: If  =  or  =  then
() = .</p>
        <p>Star selector: If  = * then
() = ( 1 | . . . |  ),</p>
        <p>∈ Λ(  ) ∪ { ℎ},  ∈ {1, . . . , }.</p>
        <p>Array slice selector: If  = (::),
,  ≥ 0,  &gt; 0 then
() = (1 | . . . | ), where
 ≤  &lt; ,
( − ) ≡ 0 (mod ),
 ∈ {1, . . . , }.</p>
      </sec>
      <sec id="sec-4-5">
        <title>We denote the deterministic finite automaton (DFA) rec</title>
        <p>ognizing the language described by ( ) by ( ).
Note that this is the language of all label sequences that
can appear in a path starting from the root of the tree
to the root of the matched subtree. The automaton
processes words over the alphabet Λ(  ) ∪ { ℎ} and can
be constructed by standard algorithms [16]. We assume,
without loss of generality, that ( ) is complete.
• Λ  = Λ  = Σ ,
• Γ =  ∪ , , where , = { |  ∈ },
•  contains the following rules:
a) Evaluation rules:
(, ( ), )
(, , )
b) Generation rules:
(, ( ), )
(, , )
(, , )
4.2.1. Evaluation
(, ′), where
→
→
 (,  ) = ′, and
′ ∈/  ⇒  = ,
′ ∈  ⇒  = ,</p>
        <p>(, ).
→ ( )(, ′), where
 (,  ) = ′, and
′ ∈/  ⇒  = ,
′ ∈  ⇒  = ,
→ (, ), where
 ∈ ,,
→ (, ), where
 ̸= ,.</p>
      </sec>
      <sec id="sec-4-6">
        <title>The  state is used to evaluate the query. At the</title>
        <p>beginning, the stack contains the initial state of DFA ( )
and we encounter the ($) event of the JSON stream.
The first transition is made based on the transition of
( ) for the initial state and the label $. Let assume ( )
moves into the state ′.</p>
        <p>• If ′ is one of the final states of ( ), it means that
we have found a match for the given query and
that we need to send the subtree of the current
node to the output. Therefore, the transducer
moves to the generating state  puts the next
state of ( ) on the stack.
• If ′ is not a final state of ( ), it means that we
have not found a match and we need to proceed
with the evaluation. Thus, the transducer remains
in the evaluation state  and puts the next
state of ( ) on the stack.
4.2.2. Generation
In the generating state , the transducer puts the
whole subtree to the output. It continues in evaluation
only in order to detect the scenarios when input query
violates the branch-disjoint condition on given input. In
that case it enters the error state . The evaluation is
based on the copies of DFA states (denoted by the
subscript ) so that the transducer is able to recognize the
end of the subtree and return to the evaluation state.
4.2.3. Complexity Summary</p>
      </sec>
      <sec id="sec-4-7">
        <title>It is easy to see that the transducer performs a single</title>
        <p>transition at each event. Implementation of this
transition actually consists of several operations (reading the
input stream, parsing, DFA transition, manipulating the
stack, generating the output stream). However, as shown
in the following section, the latency remains consistent.
A stack of the size proportional to the depth of the input
JSON tree is used. In addition, the size of the constructed
DFA counts towards the overall memory footprint used
by the evaluator. Since this work is not focused on
minimizing the memory usage, further optimizations might
be possible to reduce the DFA’s size.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Implementation</title>
      <sec id="sec-5-1">
        <title>We prototyped a low-latency SST-based evaluator in Java</title>
        <p>to validate our design of an SST-based algorithm for
JSONPath evaluation, to ensure that our design
translates to actual performance as intended and to compare
it to other JSONPath evaluators. We used the Jackson
streaming parser [17] for event-based parsing. Java was
chosen as it is well-suited for rapid prototyping that
facilitates our primary goals mentioned above.</p>
        <p>The following mapping issues have been identified and
resolved:
• Root symbol. Since the root symbol $ does not
occur in JSON data, it is generated at the beginning
of the evaluation.
• Arrays. JSON does not explicitly index array
elements, thus the indices are generated at
evaluation time. For each array value, we store the
current size of the array () on the stack. When a
new array element is encountered, the  value
is popped out of the stack, and a surrogate
label  is generated that represents the index The evaluation results are shown in Table 1. For Test
of given element. The  value is incremented suite 1, the time per event (including parsing and output
and pushed back on to the stack. Array delim- generation) exhibits lowest values and highest
consisiters are filtered out since they are not used by tency in the case of the SST-based evaluator presented
the transducer. in this paper. The values for the other two evaluators
• Placeholder symbol. All symbols not mentioned are still reasonably low and consistent. Although they
in regular expression are replaced by the place- are highest in case of the JPStream, it must be taken into
holder symbol  ℎ. account that diferent measurement method was used.
• Literal values. The proposed algorithm abstracts
from literal values. As the low-latency JSONPath
subset does not use literal values in queries, they
are skipped in the evaluation mode and outputted
in the generating mode.</p>
      </sec>
      <sec id="sec-5-2">
        <title>Focusing primarily on the measuring the latency of query processing step, additional features (e.g., compiled query storage), although important in real-world use cases, have not been implemented.</title>
        <p>We compared the presented evaluator with both
JSONPath streaming evaluators mentioned in Section 1.1. For
the Java-based evaluators (our SST-based implementation
and JsonSurfer [8]), we employed the JMH framework
[18] to measure running time per single event, mitigating
GC impact through extended warmup, forced GC, and
JMH’s built-in techniques. The C-based JPStream [9] was
evaluated using the clock_gettime system call,
providing suficient precision for our comparative analysis.</p>
        <p>As a proof-of-concept optimized for benchmarking, the
implementation of SST-based evaluator strictly follows
the code structure required by JMH. On the other hand,
JsonSurfer needed some code restructuring to make
JMHbased evaluation possible. As JPStream makes two passes
over the input and the latency is then inherently high,
only the first pass was measured and the second one was
skipped,which resulted in excluding filter expressions
from the test queries. Two test suites were used:
• Test suite 1 consists of test data and queries from
the JPStream project [9], while excluding
constructs outside the low-latency subset. Extra
queries with descendant axis were added. This
suite was aimed to ensure that the SST-based
evaluator exhibits low and consistent latency for these
queries and to enable comparison with existing
solutions.
• Test suite 2 uses the same test data, but queries
with constructs beyond the low-latency subset
were chosen (negative indices, negative steps,
filter expressions, reordering selectors, descendant
axis violating branch-disjoint condition). This
suite was intended to confirm the need for
highlatency processing of these constructs by other
evaluators. It was not meant to be processed by
SST-based evaluator as it does not support them
intentionally.</p>
        <p>SST-based
evaluator
JsonSurfer
JPStream</p>
        <p>Test
suite 1
queries from the low-latency subset and any input JSON
data.</p>
        <p>The work is in progress; to enhance the baseline
algorithm, future work will focus on refining JSONPath
predicate analysis to identify a processable subset and
optimizing the compiled query’s space complexity.</p>
        <p>The presented evaluator is designed to process a
welldefined subset of JSONPath queries based on a specific
set of rules. These limitations can potentially be relaxed
by exploring several extensions to the basic processing
model, such as
• incorporating knowledge about the structure of
the input JSON data (e.g., by employing
schemaaware or learning approaches),
• introducing parallelism and /or parallel output
streams.</p>
      </sec>
      <sec id="sec-5-3">
        <title>Moreover, since we operate at the nanosecond level, these</title>
        <p>particular diferences might be considered not significant.</p>
        <p>Evaluation showed that JsonSurfer and JPStream are not
able to process the whole low-latency JSONPath subset, As mentioned earlier, making assumptions about the
so they are less powerful than the SST-based evaluator structure of the input JSON data may considerably limit
when latency matters. JsonStream does not support star the set of possible input streams. If this leads to the
selector and descendant axis, JPStream does not support exclusion of input streams that participate in the
worstmultiple selectors and step in array slice selector. Al- case scenarios identified in this work, a larger subset of
though these two evaluators demonstrate low latency JSONPath can be possibly processed by the low-latency
for supported test queries, it is important to mention evaluator. Parallelism and parallel output streams allow
that they lack formal latency analysis, preventing a clear us to process multiple tasks at the same time, and such an
understanding of their latency guarantees for diferent approach could eliminate some situations where one task
query types. is waiting for another in a sequential model. However,</p>
        <p>Surprisingly, queries in Test suite 2 failed to process the exact benefit of this approach depends on the
particcorrectly with both JsonSurfer and JPStream, so it was not ular definition of the parallel processing model (e.g., the
possible to perform planned measurements. JsonSurfer way in which the output streams are synchronized).
produced incorrect results, while JPStream either does Another possible direction for future work involves
exnot support the constructs explicitly (negative indices, tending the processing model to evaluate multiple queries
multiple selectors, step operator), requires additional pass in parallel. While the current design focuses on
singleover the input (filter expressions), or yields unhandled query processing, a straightforward approach to achieve
exception (queries with descendant axis violating the multiquery evaluation could involve: a) compiling each
branch-disjoint condition). query into DFA, and b) maintaining a stack of state
tuples, where each tuple tracks the evaluation state for a
specific query’s DFA. However, this approach presents
6. Conclusion and Future Work several challenges: new branch conflicts may arise due
to parallel evaluation of multiple queries, and the overall
space overhead of DFAs can increase significantly. For
the latter case, techniques on combining DFA states could
be explored [19].</p>
        <p>This work analyzes a new JSONPath standard and
identiifes a powerful subset of JSONPath queries that can be
evaluated over JSON data streams performing a constant
number of steps at each event. An evaluation algorithm
has been presented that is based on finite automata and
tree transducers. The constant time complexity per input
event is guaranteed and leads to low-latency processing
as required in real-time / near-real-time scenarios. The
algorithm operates within a basic processing model
characterized by schema-agnosticism, single-query execution,
sequential processing, and a single output stream. The
presented evaluator has been compared with two other
JSONPath streaming evaluators, and the results show that
it not only supports the larger low-latency subset, but
even more importantly, based on its formal background
it gurantees the latency to be low and consistent for all
[3] A. Zheng, A. Casari, Feature Engineering for Ma- [15] T. Böhme, E. Rahm, Supporting Eficient Streaming
chine Learning: Principles and Techniques for Data and Insertion of XML Data in RDBMS, in: Proc.</p>
        <p>Scientists, O’Reilly Media, 2018. 3rd Int. Workshop Data Integration over the Web
[4] ECMA-404 The JSON data interchange (DIWeb) 2004, 2004.</p>
        <p>syntax, 2nd Edition, ECMA, 2017. [16] A. V. Aho, S. Ravi, J. D. Ullman, Compilers:
Prinhttps://ecma-international.org/ ciples, Techniques, and Tools (1st ed.),
Addisonpublications-and-standards/standards/ Wesley, 1986.</p>
        <p>ecma-404/. [17] Main Portal page for the Jackson project, 2024.
[5] The JavaScript Object Notation (JSON) https://github.com/FasterXML/jackson,
Data Interchange Format, Internet En- Retrieved: 2024-06-08.
gineering Task Force (IETF), 2017. [18] Java Microbenchmark Harness (JMH), 2024.
https://datatracker.ietf.org/doc/html/ https://github.com/openjdk/jmh,
Rerfc8259. trieved: 2024-06-08.
[6] JSONPath: Query Expressions for JSON, In- [19] Y. Diao, M. J. Franklin, High-Performance XML
ternet Engineering Task Force (IETF), 2024. Filtering: An Overview of YFilter, IEEE Data Eng.
https://datatracker.ietf.org/doc/html/ Bull. 26 (2003) 41–48.</p>
        <p>rfc9535.
[7] Extensible Markup Language (XML) 1.0 (Fifth</p>
        <p>Edition), W3C Recommendation, W3C, 2008.</p>
        <p>http://www.w3.org/TR/REC-xml.
[8] A streaming JsonPath processor in Java, 2024.</p>
        <p>https://github.com/jsurfer/JsonSurfer/,</p>
        <p>Retrieved: 2024-06-08.
[9] L. Jiang, X. Sun, U. Farooq, Z. Zhao, Scalable
Processing of Contemporary Semi-Structured Data on
Commodity Parallel Processors - a
Compilationbased Approach, in: Proceedings of the
TwentyFourth International Conference on Architectural
Support for Programming Languages and
Operating Systems, ASPLOS ’19, Association for
Computing Machinery, New York, NY, USA, 2019, pp.</p>
        <p>79–92.
[10] G. Gou, R. Chirkova, Eficient Algorithms for
Evaluating XPath over Streams, in: Proceedings of the
2007 ACM SIGMOD International Conference on
Management of Data, SIGMOD ’07, Association for
Computing Machinery, New York, NY, USA, 2007,
pp. 269–280.
[11] D. Olteanu, SPEX: Streamed and Progressive
Evaluation of XPath, IEEE Transactions on Knowledge
and Data Engineering 19 (2007) 934–949.
[12] O. Gauwin, J. Niehren, Streamable Fragments of</p>
        <p>Forward XPath, in: B. Bouchou-Markhof, P. Caron,
J.-M. Champarnaud, D. Maurel (Eds.),
Implementation and Application of Automata, Springer Berlin</p>
        <p>Heidelberg, Berlin, Heidelberg, 2011, pp. 3–15.
[13] J. Dvořáková, Automatic Streaming Processing of</p>
        <p>XSLT Transformations Based on Tree Transducers,
in: C. Badica, M. Paprzycki (Eds.), Advances in
Intelligent and Distributed Computing, Springer Berlin</p>
        <p>Heidelberg, Berlin, Heidelberg, 2008, pp. 86–94.
[14] P. Bourhis, J. L. Reutter, F. Suárez, D. Vrgoč, JSON:</p>
        <p>Data model, Query languages and Schema
specification, PODS ’17, Association for Computing
Machinery, New York, NY, USA, 2017, pp. 123–135.</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>S.</given-names>
            <surname>Chakravarthy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>Jiang</surname>
          </string-name>
          ,
          <source>Stream Data Processing: A Quality of Service Perspective: Modeling</source>
          , Scheduling,
          <string-name>
            <given-names>Load</given-names>
            <surname>Shedding</surname>
          </string-name>
          , and Complex Event Processing,
          <source>Advances in Database Systems</source>
          , Springer US,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>V.</given-names>
            <surname>Christophides</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Efthymiou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Palpanas</surname>
          </string-name>
          , G. Papadakis,
          <string-name>
            <given-names>K.</given-names>
            <surname>Stefanidis</surname>
          </string-name>
          ,
          <article-title>An Overview of End-to-End Entity Resolution for Big Data</article-title>
          ,
          <source>ACM Comput. Surv</source>
          .
          <volume>53</volume>
          (
          <year>2020</year>
          ).
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>