<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Expert system of food sensory evaluation for mobile and tablet</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>M A Nikitina</string-name>
          <email>nikitinama@yandex.ru</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Y A Ivashkin</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Moscow Technical University Communication and Informatics</institution>
          ,
          <addr-line>Aviamotornaya str., 8a, Moscow, Russia, 111024</addr-line>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>V.M. Gorbatov Federal Research Center for Food Systems of Russian Academy of Sciences</institution>
          ,
          <addr-line>Talalikhina str., 26, Moscow, Russia, 109316</addr-line>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2019</year>
      </pub-date>
      <fpage>332</fpage>
      <lpage>339</lpage>
      <abstract>
        <p>One of the main directions of statistics in sensory evaluation is an assessment of the dependence between experimental variables and measured characteristics. Statistical criteria are used to assess a degree of interaction between variables, a level of experimental e ects, and allow accepting or rejecting hypothesis proposed. In sensory evaluation, people act as measurement instruments, and a variation associated with the human factor arises. This proves that the use of statistical methods is necessary. This article represents a network computer system for collection and evaluation of food sensory indicators based on the methods of rank correlation and multifactorial analysis of variance in real time. The article describes information technology of expert sensory evaluation of food quality by individual panelists and sensory panels regarding the indicators that are not measured by technical means of control, based on client-server network architecture. The software implementation of system for collecting and statistical processing of sensory data based on the principles of multifactorial analysis of variance in real-time mode makes it possible to evaluate the in uence of the human factor on objectiveness and reliability of sensory evaluation results, as well as to visualize the data of expert scores by various expert panels.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Sensory analysis and evaluation of food quality is the basis for commodity examination of
food products and prediction of consumer demand. The usage of modern methods of sensory
analysis requires from panelists not only specialized knowledge of methodology and application
of, procedures for generating lexical dictionaries or scaling, but also the provision of consolidated,
consistent scores that con rm the objectiveness of the results [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
      </p>
      <p>
        Regardless of the experience and degree of training of panelists, individual di erences always
arise in their scores of food quality, associated with sensory sensitivity, knowledge, and the
subjectivity of the scales for sensory indicator measurement [
        <xref ref-type="bibr" rid="ref2 ref3">2, 3</xref>
        ]. Therefore, the adequate
interpretation and objecti cation of individual panelist scores requires the development of
IT technologies for factor analysis of food sensory evaluation results based on mathematical
statistics methods [
        <xref ref-type="bibr" rid="ref4 ref5 ref6">4-6</xref>
        ] and computer software for processing and visualizing data from sensory
evaluation of a product [
        <xref ref-type="bibr" rid="ref7 ref8">7-8</xref>
        ].
      </p>
      <p>An objective assessment of food quality, taking into account a variety of parameters,
alternatives and criteria, may be implement by using intelligent computer technologies for
processing and formalizing knowledge with the adoption of optimal decisions based on statistical
methods of multivariate data analysis with an objective assessment of adequacy and con rmation
of hypotheses.</p>
      <p>
        The probabilistic spread of characteristics and properties of initial biological raw material,
as well as subjectivity of individual panelist opinions determine the complexity of this problem.
The samples for testing cannot be identical in nutritional and biological value (protein, fat,
connective tissue, etc.). If the expert does not have enough knowledge and competencies, then
even the most sophisticated software will not conduct a qualitative analysis of the data [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
      </p>
      <p>
        Therefore, the adequacy of the scores in each case essentially depends on the in uence of
the human factor with the individual psycho-physiological capabilities of sensory perception
and needs correction and checking the objectivity of expert scores, considering the coherence of
certain panel opinions and experience [
        <xref ref-type="bibr" rid="ref10 ref11 ref12 ref13">10-13</xref>
        ].
      </p>
      <p>
        Information technology in a client-server network architecture is proposed based on the
methods of ranking correlation and multivariate analysis of variance, for sensory evaluation of
food quality by individual panelists and sensory panels regarding the indicators that are not
measured by any instruments [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ].
      </p>
    </sec>
    <sec id="sec-2">
      <title>2. Rank correlation in sensory evaluation of food quality</title>
      <p>
        Formal characterization of nal product sensory evaluation obtained as a result of expert survey is
achieved by rank correlation [
        <xref ref-type="bibr" rid="ref11 ref15 ref16 ref8 ref9">8,9,11,15,16</xref>
        ], according to which a group of quantitatively
nonmeasurable factors is ranked by each expert independently of each other in order of decreasing or
increasing their in uence on the assessment of product quality. The ranking results are recorded in
a rank matrix xij , i = 1; m, j = 1; n, specifying the place of j-th parameter among n other
parameters by i-th expert.
      </p>
      <p>Since opinions of panelists not always match each other, total ranks are determined to obtain
an objective assessment.
and the coe cient of concordance W is calculated, which characterizes the objective relationship
between the scores of m independent panelists using the equation:
(1)
(2)
(3)
(4)
(5)
W =</p>
      <p>12 S(d2)
m2 (n3
n)</p>
      <p>;
n " m
S(d2) = X X xi;j
j=1 i=1
gj =</p>
      <p>M
Pim=1 xij
; j = 1; n;
n
Q = Q0 + X gj xj ;
j=1
where Q0 { score of reference product, and allows to objectively identify the most signi cant
factors of deviations from the speci ed quality.</p>
      <p>The vector of weight coe cients is presents as a bar chart in accordance with the index
number of the indicator or in descending order of the absolute values of the coe cients. If the
distribution of weights is uniform or close to it, then the level of a priori knowledge is low and
further accumulation and processing of statistical data is necessary. In turn, uneven distribution
with an exponential decrease in the weights corresponds to a high degree of a priori knowledge
about the product quality.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Multifactorial analysis of variance for sensory evaluation</title>
      <p>
        The data with a multi-level structure is analyzed by multifactorial statistical procedures [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ],
which allow determining di erences between two or more data sets for all dependent variables
simultaneously. This helps to reduce the level of overall mistake of the rst kind and to assess the
degree of relationship between dependent variables, and makes it possible to establish
combinations of sensory variables, which allow distinguishing samples in the case of
nonmanifestation of di erences in each variable separately[
        <xref ref-type="bibr" rid="ref18 ref19 ref20 ref21 ref22 ref23 ref24 ref25 ref26 ref27 ref28">18-28</xref>
        ].
      </p>
      <p>The general algorithm for processing the results of sensory evaluation includes the
determination of the sample size from the general population and formulation of a null (H0) and
alternative (H1) hypotheses; choice of signi cance level ( = 0:01; or = 0:05; or = 0:1) and
conducting an assessment; data collection and calculation of total statistical criteria; acceptance
or rejection of the null (H0) hypothesis and result interpretation.</p>
      <p>
        In the case of two-factor analysis of variance, pooling by two factors is used, and in addition to
the experimental error, the variance of scores due to individual di erences between panelists in
the panel is taken into account [
        <xref ref-type="bibr" rid="ref29">29</xref>
        ]. In this case, the order of samples A, B and C should be
individual for each panelist, and their combinations ABC, ACB, BAC, BCA, CAB, CBA are
distributed in equal proportions to ensure complete randomization. The total sample variance for
all experiments is equal to sum of the intergroup and intragroup variances. The value of Fisher
test is calculated not only as the ratio of the intergroup and intragroup variances of panelist
scores, but also as the ratio of the variance of scores between individual experts to the intragroup
variance.
      </p>
      <p>
        Analysis of variance [
        <xref ref-type="bibr" rid="ref17 ref18">17, 18</xref>
        ] is the basis for software development with a client-server
architecture for collection and statistical processing of sensory data. Fisher's exact test goes
through all possible options of contingency table with the same total frequencies in rows and
columns, i.e. carries out all kinds of construction of null-models, which built on the assumption of
no in uence on the factor under study [
        <xref ref-type="bibr" rid="ref30 ref31 ref32">30-32</xref>
        ].
      </p>
      <p>The general algorithm for processing the results of sensory evaluation (Figure 1) includes
determining the sample size from the general population and formulating null (H0) and
alternative (H1) hypotheses; selecting the level of signi cance ( = 0:01; or = 0:05; or
= 0:1) and conducting an assessment; data collection and calculation of total statistical
criteria; accepting or rejecting the null (H0) hypothesis and interpreting the results.</p>
      <p>The in uence of the factor is estimated by the Fisher-Snedecor test at the chosen level of
signi cance, not only as the ratio of the intergroup variance of scores from the relationship
of factors to the intragroup variance from the experimental error F =
MMSSx212x2 , but also as</p>
      <p>E</p>
      <p>MMSSx2E22 of factor variances of expert scores for intragroup</p>
      <p>MMSSx2E21 and F =
a relationships F =
variance.</p>
    </sec>
    <sec id="sec-4">
      <title>4. Software implementation of expert system for food sensory evaluation with a client-server network architecture</title>
      <p>The client-server expert software consists of two subsystems: the server software and the client
software (Figure 2). The client software is installed on the user's computer and transmits
requests to the server subsystem to process data and requests from clients and to return them
back to user's computer.</p>
      <p>The functional structure of the system includes six modules that enter parameters and
evaluate product descriptors; creating a data set for analysis; sensory pro le; comparison with
reference; help (for user and administrator).</p>
      <p>The list of parameters (Figure 3) determined by the purpose of the sensory evaluation,
includes:
the number of samples and evaluated descriptors;
type of scale (structured or unstructured); structured ve- and nine-point scales are used,
according to which each indicator has 5 or 9 degrees of quality, respectively. According to a
vepoint scale: 5 { excellent quality; 4 { good; 3 { satisfactory; 2 { unsatisfactory, but acceptable;
1 { unsatisfactory.</p>
      <p>Nine-point scale recommended in the V.M. Gorbatov Federal Research Center for Food
Systems expands the range of sensory scores with the introduction of quantitative characteristics:
9 { optimum quality; 8 { very good; 7 { good; 6 { above average; 5 { medium; 4 or 3 { acceptable,
but undesirable; 2 or 1 { unacceptable.</p>
      <p>name of the evaluated descriptors;
folder for saving les with sensory evaluation results (text format *.txt);
instruction for the sensory panel.</p>
      <p>After setting the evaluation parameters, the panelist connects to the server software and
enters his identi cation data (for example, full name) and further, using the intensity scale of the
descriptors in the product samples, individually evaluates the intensity of the product descriptors
recording the results from the beginning of the scale. After evaluating all the descriptors in the
rst product, the panelist proceeds with the next product or nishes the sensory evaluation.</p>
      <p>Figure 4 represents a table with results of two-factor analysis of variance by the descriptor
of \smoke odor".</p>
      <p>As a null hypothesis (H0), the system proposes: { tthe products do not a ect the \smoke
odor" tdescriptor, and the alternative hypothesis (H1) { the products a ect the \smoke odor"
descriptor. To verify them, Fisher's exact test was used at a signi cance level of = 0:05.</p>
      <p>From the data in Figure 5, the calculated value of F-test for x1 factor (products) as F 19:85,
and the critical region is formed by the right-hand interval (4:46; +1). Since F falls into the
critical region, the null hypothesis (H0) is rejected and the alternative (H1) hhypothesis is
accepted, i.e. x1 factor (products) a ects the \smoke odor".</p>
      <p>Similarly, assessment of the second factor takes place, i.e. \panelists". With a null (H0)
hypothesis, panelists do not a ect the \smoke odor" descriptor and with alternative (H1)
hypothesis, panelists a ect the \smoke odor" descriptor.</p>
      <p>The values in Figure 5 show, that calculated F -test for x2 factor (panelists) is F = 1, and
the critical region is formed by the right-hand interval (3:84; +1). Since F does not fall in the
critical region, the null (H0) hypothesis is accepted, i.e. in uence of x2 factor (panelists) on the
\smoke odor" was not con rmed.
shows that 77% of the total sample variation in the descriptor (smoke odor) is related to the
in uence of the product type on it.</p>
      <p>In the P-value column P-value is determined, which corresponds to the calculated value of
F-test.</p>
      <p>In our example, P-value for x1 factor (products) depends on the values of F ; df and MS of
this factor in the rst row of the table, and has a value of 0.00079.</p>
      <p>P-value for the x2 factor (panelists) depends on the values of F ; df and MS of this factor in
the second row of the table, and is equal to 0.46.</p>
      <p>According to the Fisher-Snedecor test, when P-value is less than 0.05 (P &lt; 0:05), the data are
not consistent. Based on the calculation, analysis and comparison, the system makes conclusion
\Products di er in this descriptor; the scores of the panelists are consistent".</p>
      <p>In the case of a consistent and reliable evaluation, the software allows to build a sensory
pro le (pro logram) of the product characteristic being evaluated (Figure 6) with a number of
intensity score axes equal to the number of speci c descriptors.</p>
      <p>Figure 5 shows an example of the sensory pro le for three samples of cooked sausage in the
form of a polygon with vertices combining the obtained product characteristics.</p>
      <p>Using similar procedures, the software allows to determine the position of a product among
competitors based on a comparison of its pro le with competitors' product pro les.</p>
      <p>For comparison of the product pro le with reference, reference product is preliminarily
produced, which is the basis for comparing all the products involved in the evaluation. The
characteristics of the reference sample determine the reference sensory pro le, which is compared
with the pro le of a similar sample from another batch (Figure 6).</p>
      <p>The computer software also allows to identify changes in the sensory characteristics of the
product when replacing food ingredients, additives or spices in the formulation or using new
types of packaging, etc.</p>
    </sec>
    <sec id="sec-5">
      <title>5. Experimental testing of the software</title>
      <p>The given example of the network expert system and its dialog interface along with the individual
numerical scores and statistical evaluation by panelists provides the objectivized conclusion and
recommendations concerning the product quality based on processing of the subjective data from
expert panels of up to 20 panelists by 15 descriptors and 6 product types with the construction
of pro lograms with up to 15 descriptors and a possibility of data export to MS Excel. Thus,
the accuracy and reliability of the objectivized scores presented in Figure 4 is determined by the
criteria values for the speci c case, as well as by the degree of agreement and competence of the
opinions from the quali ed panelists evaluating technically uncontrollable sensory properties of
food products and their in uence on evaluation.</p>
    </sec>
    <sec id="sec-6">
      <title>6. Conclusion</title>
      <p>Therefore, the computer software with the client-server architecture based on the multivariate
analysis of variance realizes the information technology for support of decision making in sensory
food evaluation contrary to the traditional expert systems and software packages. It performs
real-time collection, accumulation and statistical processing of sensory data from individual
panelists and geographically distributed panels and visual presentation of the objecti ed results in
di erent graphic forms.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <surname>Lawless H T and Heymann</surname>
            <given-names>H 2010</given-names>
          </string-name>
          <article-title>Sensory evaluation of food: principles and practices</article-title>
          (New York: Springer Science &amp; Business Media) p
          <fpage>596</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <surname>Piggott J R and Hunter</surname>
            <given-names>E A</given-names>
          </string-name>
          <year>1999</year>
          <article-title>Review: Evaluation of assessorperformance in sensory analysis</article-title>
          <source>Italian Journal of Food Science</source>
          <volume>4</volume>
          (
          <issue>11</issue>
          )
          <fpage>289</fpage>
          -
          <lpage>303</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <surname>Tomic</surname>
            <given-names>O</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Forde</surname>
            <given-names>C</given-names>
          </string-name>
          ,
          <article-title>Delahunty Cand Naes T 2013 Performance indices in descriptive sensory analysis Acomplimentary screening tool for assessor and panel performance Food Quality</article-title>
          and
          <source>Preference</source>
          <volume>28</volume>
          (
          <issue>1</issue>
          )
          <fpage>122</fpage>
          -
          <lpage>133</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <surname>Johnson</surname>
            <given-names>R A</given-names>
          </string-name>
          and
          <string-name>
            <surname>Wichern D W 2007 Applied Multivariate Statistical Analysis</surname>
          </string-name>
          (New York: Prentice-Hall) p
          <fpage>800</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <surname>Kochan</surname>
            <given-names>A</given-names>
          </string-name>
          and
          <string-name>
            <surname>Grimm</surname>
            <given-names>M</given-names>
          </string-name>
          <source>1985 Die sensorische Profilmethode Lebensmittelindustrie</source>
          <volume>32</volume>
          (
          <issue>6</issue>
          )
          <fpage>255</fpage>
          -
          <lpage>269</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <surname>Surmacka-Szczesniak</surname>
            <given-names>A</given-names>
          </string-name>
          1998
          <article-title>Sensory texture proffiling-historical and scientific perspectives</article-title>
          <source>Food Technology</source>
          <volume>52</volume>
          (
          <issue>8</issue>
          )
          <fpage>14</fpage>
          -
          <lpage>16</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <surname>Labbe</surname>
            <given-names>D</given-names>
          </string-name>
          ,
          <article-title>Rytz A andHugi A 2004 Training is a critical step to obtain reliable product profiles in a real food industry context Food quality</article-title>
          and preference
          <volume>15</volume>
          (
          <issue>4</issue>
          )
          <fpage>341</fpage>
          -
          <lpage>348</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <surname>Savela-Huovinen</surname>
            <given-names>U</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Muukkonen</surname>
            <given-names>H</given-names>
          </string-name>
          and
          <article-title>Toom A 2018 Sensory expert assessor's learning practices at workplace: Competencies and contexts in sensory evaluation Journal of sensory studies 33(2</article-title>
          ) DOI:
          <fpage>10</fpage>
          .1111/joss.12315
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <surname>Mitchell</surname>
            <given-names>M N</given-names>
          </string-name>
          <year>2005</year>
          <article-title>Strategically using General Purpose Statistics Packages: A Look at Stata</article-title>
          ,
          <source>SAS and SPS Technical Report Series</source>
          . Statistical Consulting Group: UCLA Academic Technology Services URL: http://www.ats.ucla.edu/stat/technical-reports
          <source>/umber1_editedFeb_2_2007ucla_ATSstat_trl_1</source>
          .1_ _207.
          <string-name>
            <surname>pdf</surname>
          </string-name>
          (
          <volume>08</volume>
          .
          <fpage>09</fpage>
          .
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <surname>Running</surname>
            <given-names>C A</given-names>
          </string-name>
          and
          <string-name>
            <surname>Hayes J E 2016</surname>
          </string-name>
          <article-title>Individual Differences in Multisensory Flavor Perception Multisensory flavor perception: from fundamental neuroscience through to the marketplace (Woodhead Publishing Series in Food Science Technology</article-title>
          and Nutrition)
          <volume>298</volume>
          <fpage>185</fpage>
          -
          <lpage>210</lpage>
          DOI: 10.1016/ B978-0
          <source>-08-100350-3</source>
          .
          <fpage>00010</fpage>
          -
          <lpage>9</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <surname>da Silva R D D N</surname>
          </string-name>
          ,
          <string-name>
            <surname>Minim</surname>
            <given-names>V P R</given-names>
          </string-name>
          ,
          <string-name>
            <surname>da Silva A N and Minim</surname>
            <given-names>L A</given-names>
          </string-name>
          <year>2014</year>
          <article-title>Number of judges necessary for descriptive sensory tests Food quality</article-title>
          and preference
          <volume>31</volume>
          <fpage>21</fpage>
          -
          <lpage>27</lpage>
          DOI: 10.1016/j.foodqual.
          <year>2013</year>
          .
          <volume>07</volume>
          .010
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <surname>Turner</surname>
            <given-names>R</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ledwith</surname>
            <given-names>A</given-names>
          </string-name>
          and
          <string-name>
            <surname>Kelly J 2012</surname>
          </string-name>
          <article-title>Project management in small to medium-sized enterprises: Tailoring the practices to the size of company Management Decision 50(5</article-title>
          )
          <fpage>942</fpage>
          -
          <lpage>957</lpage>
          DOI: 10.1108/00251741211227627
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <surname>Yemelyanov</surname>
            <given-names>S V</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Burovoi I F and Levada F Yu 1998</surname>
          </string-name>
          <article-title>Control of Indefinite Nonlinear dynamic systems</article-title>
          .
          <source>Induced internal feedback Lecture Notes in Control and Information Sciences 231 196</source>
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <surname>Brockhoff P B 2003</surname>
          </string-name>
          <article-title>Statistical testing of individual differences in sensory profiling Food Quality</article-title>
          and
          <source>Preference</source>
          <volume>14</volume>
          (
          <issue>5-6</issue>
          )
          <fpage>425</fpage>
          -
          <lpage>434</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>Naes</surname>
            <given-names>T</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brockhoff P B andTomic O 2010</surname>
          </string-name>
          <article-title>Statistics for Sensory and Consumer Science</article-title>
          (John Wiley &amp; Sons, Ltd) p 287 DOI: 10.1002/9780470669181
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <article-title>Brand Z 2003 Data analysis</article-title>
          .
          <source>Statistical and Computational Methods for Scientists and Engineers</source>
          (Moscow: Mir) p
          <fpage>686</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <surname>Stahle</surname>
            <given-names>L</given-names>
          </string-name>
          and
          <string-name>
            <surname>Wold</surname>
            <given-names>S 1990</given-names>
          </string-name>
          <article-title>Multivariate analysis of variance (MANOVA) Chemometrics</article-title>
          and Intelligent
          <source>Laboratory Systems</source>
          <volume>9</volume>
          <fpage>127</fpage>
          -
          <lpage>141</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <surname>Lee S M</surname>
            , Chung
            <given-names>S J</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lee</surname>
            <given-names>O H</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lee H S</surname>
            , Kim Y K and Kim
            <given-names>K O</given-names>
          </string-name>
          <year>2008</year>
          <article-title>Development of sample preparation, presentation procedure and sensory descriptive analysis of green tea</article-title>
          <source>Journal of Sensory Studies</source>
          <volume>23</volume>
          <fpage>450</fpage>
          -
          <lpage>467</lpage>
          DOI: 10.1111/j.
          <fpage>1745</fpage>
          -
          <lpage>459X</lpage>
          .
          <year>2008</year>
          .
          <volume>00165</volume>
          .x
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <surname>Hoehl</surname>
            <given-names>K</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schoenberger</surname>
            <given-names>G U</given-names>
          </string-name>
          ,
          <article-title>Schwarz K and Busch-Stockfisch 2013 Is Perception of Sucrose and Caffeine Affected by Training or Experience? Monitoring Training Effects in Female Subjects over a Half-</article-title>
          <source>Year Period</source>
          <volume>28</volume>
          <fpage>1</fpage>
          -
          <lpage>13</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <surname>Brockhoff P B and Sommer</surname>
            <given-names>N A</given-names>
          </string-name>
          <year>2008</year>
          <article-title>Accounting for scaling differences in sensory profile data Proceedings of 10th European Symposium on Statistical Methods for the Food Industry (Louvain-LaNeuve</article-title>
          , Belgium)
          <fpage>283</fpage>
          -
          <lpage>290</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <surname>Smith</surname>
            <given-names>A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cullis</surname>
            <given-names>B</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brockhoff</surname>
            <given-names>P</given-names>
          </string-name>
          and
          <string-name>
            <surname>Thompson</surname>
            <given-names>R 2003</given-names>
          </string-name>
          <article-title>Multiplicative mixed models for the analysis of sensory evaluation data Food Quality</article-title>
          and
          <source>Preference</source>
          <volume>14</volume>
          (
          <issue>5-6</issue>
          )
          <fpage>387</fpage>
          -
          <lpage>395</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <surname>Bi</surname>
            <given-names>J 2003</given-names>
          </string-name>
          <article-title>Agreement and reliability assessments for performance of sensory descriptive panel</article-title>
          <source>Journal of Sensory Studies</source>
          <volume>18</volume>
          <fpage>61</fpage>
          -
          <lpage>76</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <surname>Naima</surname>
            <given-names>H A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mehdi</surname>
            <given-names>S</given-names>
          </string-name>
          and
          <string-name>
            <surname>Morched C H 2013 Training and</surname>
          </string-name>
          <article-title>Control of a Tactile Sensory</article-title>
          <source>Panel Journal of Applied Sciences</source>
          <volume>13</volume>
          <fpage>366</fpage>
          -
          <lpage>376</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <surname>Bi</surname>
            <given-names>J</given-names>
          </string-name>
          and
          <string-name>
            <surname>Kuesten C 2012 Intraclass Correlation</surname>
          </string-name>
          <article-title>Coefficient (ICC): A Framework for Monitoring and Assessing Performance of Trained Sensory Panels</article-title>
          and
          <source>Panelists Journal of Sensory Studies</source>
          <volume>27</volume>
          (
          <issue>5</issue>
          )
          <fpage>352</fpage>
          -
          <lpage>364</lpage>
          DOI: 10.1111/j.
          <fpage>1745</fpage>
          -
          <lpage>459X</lpage>
          .
          <year>2012</year>
          .
          <volume>00399</volume>
          .x
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <surname>Grawford C M and Di Benedetto C A 2014 New Products</surname>
          </string-name>
          <article-title>Management (New York: McGraw-Hill Education</article-title>
          ) p
          <fpage>608</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <surname>Moskowitz</surname>
            <given-names>H R</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Porretta</surname>
            <given-names>S</given-names>
          </string-name>
          and
          <string-name>
            <surname>Silcher M 2005</surname>
          </string-name>
          <article-title>Concept research in food product design and development (Ames: Blackwell Publishing</article-title>
          ) p
          <fpage>612</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <surname>Stone</surname>
            <given-names>H</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bleibaum N R and Thomas A N 2012 Sensory Evaluation Practices</surname>
          </string-name>
          (San Diego: Elsevier Academic Press) p
          <fpage>438</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [28]
          <string-name>
            <surname>Johnson</surname>
            <given-names>R A</given-names>
          </string-name>
          and
          <string-name>
            <surname>Wichern D W 2007 Applied Multivariate Statistical Analysis</surname>
          </string-name>
          (New York: Prentice-Hall) p
          <fpage>800</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [29]
          <string-name>
            <given-names>O</given-names>
            <surname>'Mahony M 1986 Sensory</surname>
          </string-name>
          <article-title>Evaluation of Food. Statistical Methods</article-title>
          and
          <string-name>
            <surname>Procedures</surname>
          </string-name>
          (New York: Marcel Dekker) p
          <fpage>488</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          [30]
          <string-name>
            <surname>Pate eld W M 1981 Algorithm AS</surname>
          </string-name>
          <article-title>159</article-title>
          .
          <article-title>An e cient method of generating r x c tables with given row</article-title>
          and
          <source>column totals Applied Statistics</source>
          <volume>30</volume>
          (
          <issue>1</issue>
          )
          <fpage>91</fpage>
          -
          <lpage>97</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          [31]
          <string-name>
            <surname>Gotelli</surname>
            <given-names>N J</given-names>
          </string-name>
          and
          <string-name>
            <surname>Entsminger N J 2003</surname>
          </string-name>
          <article-title>Swap algorithms</article-title>
          in
          <source>null model analysis Ecology</source>
          <volume>84</volume>
          <fpage>532</fpage>
          -
          <lpage>535</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          [32]
          <string-name>
            <surname>Strona</surname>
            <given-names>G</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ulrich</surname>
            <given-names>W</given-names>
          </string-name>
          and
          <string-name>
            <surname>Gotelli N J 2018</surname>
          </string-name>
          <article-title>Bi-dimensional null model analysis of presenceabsence binary matrices</article-title>
          <source>Ecology</source>
          <volume>99</volume>
          (
          <issue>1</issue>
          )
          <fpage>103</fpage>
          -
          <lpage>115</lpage>
          DOI: 10.1002/ecy.2043
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>