This paper proposes a neural-symbolic framework for analyzing a large volume of sequential hi-speed images of combustion flame for early detection of instability that is extremely critical for engine health monitoring and prognostics. The proposed hierarchical approach involves extracting low-dimensional semantic features from images using deep Convolutional Neural Networks (CNN) followed by capturing the temporal evolution of the extracted features using Symbolic Time Series Analysis (STSA). Furthermore, the semantic nature of the CNN features enables expert-guided data exploration that can lead to better understanding of the underlying physics. Extensive experimental data have been collected in a swirlstabilized dump combustor at various operating conditions for validation.
Recent advancements in deep learning shows that neural approaches are excellent at low-level
feature extraction from raw data, automated learning and discriminative tasks. However, such models
still may not be suited as much for logical reasoning, interpretation and domain knowledge
incorporation. On the other hand, symbolic approaches can potentially alleviate such issues as they are
shown to be effective in high-level reasoning and capturing sequence of actions. Therefore, a
hybrid neural-symbolic [
• A novel data-driven framework, with CNN at lower layer and STSA at upper layer, is
proposed for early detection of thermo-acoustic instability from hi-speed videos.
• In the above framework, the CNN layers extract meaningful shape-features to represent the
coherent structures of varied sizes and orientations in the flame images. This phenomenon
enables STSA at the temporal modeling layer to capture all the fast time scale precursors
before attaining persistent instability.
• The proposed theory and the associated algorithms have been experimentally validated for
transition data at multiple operating conditions in a swirl-stabilized combustor by
characterizing the stable and unstable states of combustion.
• Training and testing of the proposed framework have been performed on different operating
conditions (e.g., air flow rate, fuel flow rate, and air-fuel premixing level) of the combustion
process to test the transferability of the approach. Performance of the proposed framework
(‘CNN+STSA’) have been evaluated by comparison with that of a framework, where CNN
is replaced by another extensively used dimensionality reduction tool, principal component
analysis (PCA) [
To collect training data for learning the coherent structures, thermo-acoustic instability was induced
in a laboratory-scale combustor with a 30 mm swirler (60 degree vane angles with geometric swirl
number of 1.28). Figure 1 (a) shows the setup and a detail description can be found in [
(a) optical access module (IOAM), 4 - test section, 5 & 6 - big and small extension ducts, 7 - pressure transducers, Xs - swirler location, Xp - transducer port location, Xi - fuel injection location, (b) Visible coherent structure in greyscale images at 900 lpm AFR and full premixing for 45 lpm FFR 3
This section describes the proposed architecture for early detection of thermo-acoustic instability in a combustor via analyzing a sequence of hi-speed images. Figure 2 presents the schematics of the framework where a deep CNN is stacked with symbolic time series analysis (STSA). In the training phase, images (or a segment of the images) from unstable state for various operating conditions are used as the input to the CNN. re su a e iilty b tsa In Stable
time Early detection of Instability ߙ ߛ
ߚ ݍ ଵ ߙ ߛ ߙ ߛ ݍ ସ ߚ ߙ ߛ ݍݍ _2ଶ ݍ ଷ ߚ ߚ time ߙ … ߚߛߙ ߚ ߛ ߙ ݍ ଵ ߚߙ ݍ ଶ
Partitioning Symbol sequence ߝ ߚ ߚ ݍ ଷ State merging ݍ ସ State transition matrix
Generalized D-Markov Machine
State splitting
edges and corners to form part of an image [
In CNNs, data is represented by multiple feature maps in each hidden layer as shown in the figure 2. Feature maps are obtained by convolving the input image by multiple filters in the corresponding hidden layer. To further reduce the dimension of the data, these feature maps typically undergo non-linear downsampling with a 2 × 2 or 3 × 3 maxpooling. Maxpooling essentially partitions the input image into sets of non-overlapping rectangles and takes the maximum value for each partition as the output. After maxpooling, multiple dimension-reduced vector representations of the input is acquired and the process is repeated in the next layer to learn a higher representation of the data. At the final pooling layer, resultant outputs are linked with the fully connected layer where sigmoid outputs from the hidden units are post-processed by a softmax function in order to predict the class that possesses the highest joint probability given the input data. This way, coherent structures in the unstable flame can be learned at different operating condition. 3.2
Symbolic Time Series Analysis (STSA)
STSA [
Generalized D-Markov Machine [
P [sn | · · · sn−D · · · sn−1] = P [sn | sn−D · · · sn−1] (1) D is called the depth. Q is the finite set of states with cardinality |Q| ≤ |Σ|D, i.e., the states are represented by equivalence classes of symbol strings of maximum length D, where each symbol belongs to the alphabet Σ. δ : Q × Σ → Q is the state transition function that satisfies the following condition: if |Q| = |Σ|D, then there exist α, β ∈ Σ and x ∈ Σ⋆ such that δ(αx, β) = xβ and αx, xβ ∈ Q.
Definition 3.2 (D-Markov Entropy Rate) The D-Markov entropy rate of a PFSA (Σ, Q, δ, π) is defined in terms of the conditional entropy as:
H(Σ|Q) , X P (q)H(Σ|q) = − X X P (q)P (σ|q) log P (σ|q) q∈Q q∈Q σ∈Σ where P (q) is the probability of a PFSA state q ∈ Q and P (σ|q) is the conditional probability of a symbol σ ∈ Σ given that a PFSA state q ∈ Q is observed. 3.2.2
Construction of a D-Markov Machine [
1 + N (q) |Q| + P N (q′) ∀q ∈ Q q′∈Q where Pσ∈Σ πˆ(σ, q) = 1 ∀q ∈ Q. Then, the D-Markov entropy rate (see Definition 3.2) is computed as:
H(Σ|Q) = − X X P (q)P (σ|q) log P (σ|q) ≈ − X X Pb(q)πˆ(q, σ) log πˆ(q, σ) q∈Q σ∈Σ q∈Q σ∈Σ State Merging: While merging the states, this algorithm aims to mitigate this risk of degraded precision via a stopping rule that is constructed by specifying an acceptable threshold ηmrg on the distance Ψ(·, ·) between the merged PFSA and the PFSA generated from the original time series. The distance metric Ψ(·, ·) between two PFSAs K1 = (Σ, Q1, δ1, π1) and K2 = (Σ, Q2, δ2, π2) is as follows:
n Ψ(K1, K2) , lim X n→∞
P1(Σj ) − P2(Σj ) ℓ1 2j+1 (2) (3) j=1 where P1(Σj ) and P2(Σj ) are the steady state probability vectors of generating words of length j from the PFSA K1 and K2, respectively, i.e., P1(Σj ) , [P (w)]w∈Σj for K1 and P2(Σj ) , [P (w)]w∈Σj for K2. States that behave similarly (i.e., have similar morph probabilities) have a higher priority for merging. The similarity of two states, q, q′ ∈ Q, is measured in terms of the respective morph functions of future symbol generation as the distance between the two rows of the estimated morph matrix Πb corresponding to the states q and q′. The ℓ1-norm has been adopted to be the distance function as seen below.
M(q, q′) , kπˆ(q, ·) − πˆ(q′, ·)kℓ1 = X |πˆ(q, σ) − πˆ(q′, σ)|
σ∈Σ
Hence, the two closest states (i.e., the pair of states q, q′ ∈ Q having the smallest value of M(q, q′))
are merged using the merging algorithm explained in [
This section discusses the results that are obtained when the proposed framework is applied on the experimental data of hi-speed video for early detection of thermo-acoustic instability. 4.1
CNN training The network is trained using flame images with 4 different unstable combustion conditions mentioned in the section 2. The data consists of 24,000 examples for training and 12,000 examples for cross-validation. In the first convolutional layer, 20 filters of size 33 × 34 pixels (px) reduce the input image of dimension 100 × 237 pixels to feature maps of 68 × 204. Next, the feature maps are downsampled with a 2×2 max-pooling, resulting in pooled maps of 34×102 px. Each of these maps undergoes another convolutional layer with 50 filters of 23 × 23 px which produces feature maps of 12 × 80 px (before 2 × 2 max-pooling), and 6 × 40 pooled maps after max-pooling. All generated maps are connected to the fully connected layer of 100 hidden units followed by 10 output units where the sigmoid activations are extracted. Training is performed with a batch size of 20 and learning rate of 0.1. Convolution is done without any padding with a stride-size of 1. Visualization of few filters at first and second convolutional layer is shown in figure 3 (a), (b). Second layer visualization shows that it captures fragments of the flame coherent structures. Figure 3 (c) presents couple of the feature maps of a stable frame (top) and an unstable frame (bottom) after convolving with first layer filter. Red outline at the bottom exhibits how the mushroom-shaped coherent structure is highlighted on the unstable frame feature map.
(a) (b) (c) Once the CNN is trained on the sets of unstable data, every frame of the transition data sets(i.e., 50040to38, 50040to30, 40500to600, 60050to35 and 50700to800 as mentioned in section 2) are fed to the CNN. Each sigmoid activation unit out of ten at the last fully connected layer generates a time series for one transition data set. For capturing the fast change in a transition data, a window of 0.5 seconds (1500 frames) is traversed over the hi-speed video with an overlap of 80% to keep the response speed at 10 Hz, which is necessary for real-time combustion instability control. The time window output of a sigmoid activation unit is symbolized by maximum entropy partitioning (MEP) with an alphabet size of |Σ| = 3. Considering the first window to be reference stable state, a generalized D-Markov machine is constructed by state splitting with Nmax = 10 and state merging with ηmrg = 0.05. Nmax is chosen as 10 because window length is not enough to learn a large state machine. For the alphabet {1, 2, 3}, the set of states after state splitting is {11, 21, 31, 2, 113, 213, 313, 23, 133, 233, 333} and state merging leads to {11, 21, 31, 2, {113, 313}, 213, {23, 133}, {233, 333}} for one of the sigmoid activation outputs in the transition video 60050to35. State probability vector, arising from D-Markov machine at each time window, is the feature capturing the extent of instability which is transmitted through the corresponding sigmoid hidden unit. Instability measure of a time window is defined as the l2 norm distance from the reference stable time window. Figure 4 shows an aggregated progression (summation of individual instability measure obtained from each sigmoid activation unit) of instability measure for 60050to35. The rms curve of the pressure on right of the figure 4 gives a rough idea about the ground truth regarding stability. Two fold advantages of the proposed instability measure over Prms are as follows: (i) intermittancy phenomenon (region 2 and 3 on figure 4) is captured by this measure because it can detect variablesize mildly-illuminated mushroom-shaped coherent structure (bounded by red box in the figure 4) in the ‘CNN+STSA’ framework whereas Prms ignores these important precursors to instability and (ii) region 4 of the figure 4 shows that the proposed measure rises faster towards instability. Other transition data sets also exhibit similar nature regarding this measure. Hence, the proposed measure performs better in early detection of instability than other commonly used measures such as Prms. 4.3
Comparison with ‘PCA+STSA’
To compare with the proposed approach, Principal Component Analysis (PCA) [
5.5 e r sau 5 e m ty4.5 ili b tsa 4 n i ipn3.5 m jun 3 o iitsn2.5 a r T
The paper proposes a framework that synergistically combines the recently introduced concepts of CNN and STSA for early detection of thermo-acoustic instability in gas turbine engines. Extensive set of experiments have been conducted on a swirl-stabilized combustor for validation of the proposed method. Sequences of hi-speed greyscale images are fed into a multi-layered CNN to model the fluctuating coherent structures in the flame, which are dominant during unstable combustion. Fragments of coherent structures are observed in the CNN filter visualization. Therefore, an ensemble of time series data is constructed from sequence of images based on the sigmoid activation probability vectors of last hidden layer at the CNN. Then, STSA is applied on the time series that is generated from an image sequence and ‘CNN+STSA’ is found to exhibit larger change in instability measure while transition to instability than ‘PCA+STSA’. The proposed framework detects all the intermittent precursors for different transition protocols, which is the most significant step towards detecting the onset of instability early enough for mitigation. In summary, while CNN captures the semantic features (i.e., coherent structures) of the combustion flames at varied illuminations, sizes and orientations, STSA models the temporal fluctuation of those features at a reduced dimension. One of the primary advantages of the proposed semantic dimensionality reduction (as opposed to abstract dimensionality reduction, e.g., using PCA) would be seamless involvement of domain experts into the data analytics framework for expert-guided data exploration activities. Developing novel use-cases in this neural-symbol context will be a key future work. Some other near-term research tasks are: (i) dynamically tracking multiple coherent structures in the flame to characterize the extent of instability, (ii) multi-dimensional partitioning for direct usage of the last sigmoid layer and (iii) learning CNN and STSA together.
Authors sincerely acknowledge the extensive data collection performed by Vikram Ramanan and Dr. Satyanarayanan Chakravarthy at Indian Institute of Technology Madras (IITM), Chennai. Authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of the GeForce GTX TITAN Black GPU used for this research.