Testing Computer Vision Applications An Experience Report on Introducing Code Coverage Analysis in the Field Iulia Nica and Franz Wotawa1 and Gerhard Jakob and Kathrin Juhart2 Abstract. In this paper we present our work in progress in defining when the obtained results are still correct and when we are dealing a suitable testing and validation methodology to be used within com- with an abnormal behavior. puter vision (CV) projects. Typical quality assurance (QA) measures, Regarding the evaluation of the complete, often very complex vi- targeting the applicability in real-world scenarios, are meant here sion system on the other side, the QA team has to manage and run a to complement the research on specific computer vision methods. high amount of tests on all levels - from unit tests, to integration, While inspecting the existing literature in the domain of CV perfor- function and system tests. Therefore one needs to understand the mance evaluation, we first identified the main challenges the CV re- system as a whole, as well as all of its components and their in- searchers have to deal with. Second, as every vision algorithm even- terdependencies. Furthermore we have to cover also possible hard- tually takes the form of a software program, we followed the classic ware faults when identifying use cases, based on the defined system software development process and performed an in depth code cov- requirements and specifications. Fortunately, today there are well- erage analysis in order to assure the quality of our test suites and established QA practices and many quality management tools avail- pinpoint code areas that need to be reviewed. This further leaves us able on the market, meant to ease the generic evaluation of products with the questions of which test coverage tool to prefer in our situa- and processes, that the only challenge is to find a proper manner to tion and whether we can introduce some specific evaluation criteria integrate them in the vision project. for identifying the right tool to be used within a CV project. In this The remainder of this paper is organized as follows. In Section 2 article we also contribute to answering these questions. we review the existing literature in the domain of CV performance evaluation and introduce some basic quality assurance terms. After- wards, in Section 3, we identify and discuss the requirements a code 1 Motivation coverage tool has to fulfill in order to be used in the CV domain. Computer vision (CV) is used today in a wide range of real-world Further on, we give a short overview of our four best ranked tools. applications, from industrial inspection and safety relevant vehicle In Section 4 we first introduce the case study and compare the tools functions to 3D model generation by photogrammetric methods, based on their integration with the example application. Additionally, medical imaging and fingerprint recognition. Although a vast vari- we present the first success story in improving our code coverage. ety of literature covering evaluation techniques in subfields of the With Section 5 we conclude this paper. whole topic is available, still no study reports on testing a complete vision system, i.e., comprising hardware, software, data communica- tion and control. Obviously the high quality of CV applications has a 2 Related Research great impact on their usability in real world scenarios. Hence beside In our work, we have first inspected the existing literature in the traditional CV evaluation techniques such as using test data sets as domain of performance evaluation in computer vision. General input and comparing the algorithms output against a manually estab- overviews of empirical evaluations were found in [5], [2], [3], and lished ground truth - we have to control the quality of the involved [11] and will be further presented here in a chronological order. They applications by means of applying a more generic evaluation strategy. all review the commonly used techniques for performance character- In this context, quality assurance (QA) activities like peer reviews, ization of algorithms in different subfields of CV. coding guidelines, or the usage of software quality tools (static and In the early 90s, [5] was discussing the evident lack of perfor- dynamic analyzers) offer many benefits, from being able to track the mance evaluation in the literature on vision algorithms. In the au- CV projects progress and estimate its relative complexity to helping thor’s opinion, this situation has been tolerated because the ability to us realize when we have achieved the desired state of quality [4]. perform a CV task was interesting enough, so that the performance Still, what is different about testing CV applications and why is it of the new algorithm became a secondary issue. In order to quickly so difficult to test whether computer vision algorithms can live up to design a machine vision system, which works efficiently and meets their claims? requirements, [5] suggests an analogy with a system’s engineering Regarding algorithmic correctness on one side, it is often very hard methodology. Thus, a well-defined protocol containing a modeling to get a consistent and exact definition of the desired output for a component, an experimental component and a data analysis compo- specific input. Especially in classification tasks, it is tough to decide, nent was envisioned. The modeling component would describe the 1 Technische Universität Graz, Austria, email: {inica,wotawa}@ist.tugraz.at ideal input image population (real or synthetic images), the random 2 JOANNEUM RESEARCH, email:{gerhard.jakob,Kathrin.Juhart}@joanneum.at perturbation model (by which non-ideal images arise), the random perturbation process (that characterizes the output random pertur- 3. Are there any known algorithms that can be used as benchmarks bation as function of input random perturbation) and the criterion for comparison? function (by which one can quantify the difference between the ideal 4. What should we be measuring to quantify performance? What output and the computed output). The experimental component de- metrics are used? scribes the performed experiments, whilst the data analysis deter- Though the analysis in [11] touches also other aspects of build- mines the performance characterization based on the experimentally ing a complete vision system, it excludes testing the hardware. Fur- observed data. thermore, the mentioned software validation is limited to ensuring In the absence of acknowledged methods for the evaluation of al- that the software implementation of an algorithm correctly instanti- gorithmic performance, [2] proposed the definition of performance ates its mathematical foundation [11]. Hence, the collected answers as function of mathematical sophistication. However, as the number for each of the considered visual tasks indicate the fact that perfor- and specificity of assumptions made in the mathematics underlying a mance characterization techniques are mostly application/algorithm vision algorithm increase (i.e., the sophistication of an algorithm in- specific and that currently they do not refer to the integrated system creases), the performance of the CV application not necessarily does. as a whole, i.e., comprising hardware, software, data communication This is the case when the assumptions made do not match the appli- and control. cation characteristics. Furthermore, the need of standard databases, More currently published research like [13], [14] emphasizes the evaluation protocols and scoring methods/performance metrics avail- role of test data generation and test data validation in vision testing. able to researchers was identified by the authors. For the purpose of evaluating CV algorithms, there are today some Regarding the typology of test data, [3] differentiate first between publicly available data sets, such as the FERET database [9] for face data without noise and data with noise. Moreover, they mention three recognition algorithms, Middlebury [10] and KITTI [7] test data sets types of empirical testing: testing using real data with full control, for stereo vision, or VOT datasets [6] for visual tracking. The us- empirical testing with partially controlled test data and testing in an age of this large amount of test images brings yet some problems. uncontrolled environment. Depending on the distribution of the avail- One of them is that the test data sets are not specially designed for able data into training and testing sets, test protocols have been pro- a particular vision application, but for a class of algorithms. Hence, posed. Another discussed issue in [3] is again the necessity to define a 100% coverage of the possible scenarios can not be guaranteed. a metric, which can be used to quantify performance. The authors As introduced in [13] and further elaborated in [14], a solution to associate such performance metrics with the failure modes of an al- this problem would be the automatic generation of datasets, so that gorithm. For each type of vision algorithm, specific evaluation met- they contain all the typical scenes and hazards, without including too rics were defined according to the function performed by the given much redundancy, so that the testing effort could be manageable. algorithm. Some examples are the ROC (Receiver Operating Char- For a change, as the vision algorithm will take eventually the form acteristic) curve in case of a feature detector, the confusion matrix in of a software program, we see no reason why we should not take ad- case of object recognition, or the true and false matches when deal- vantage of the great progress in the domain of quality assurance and ing with matching algorithms, such as those used in stereo or motion software testing in particular. The usage of standardized QA meth- estimation. ods, metrics and tools can ease the work of any CV developer and Similarly to [3], the authors of [11] outline two different levels of quickly improve the overall process, especially in terms of system’s analysis for vision systems: resilience and end user’s satisfaction. • technology evaluation, which concerns the characteristics of the ”Quality control activities determine whether a product conforms algorithms using generic metrics, such as ROC curves. Standard- to its requirements, specifications, or pertinent standards”[12]. In ad- ized data sets are used and the results are therefore repeatable dition to the traditional testing practices, QA activities encompass and depend on the size and scope of the test data sets. Generally, peer reviews, coding guidelines, and also the usage of software qual- this evaluation stage requires simple metrics related to the fulfilled ity tools, like static analyzers that examine source code for possible function- detection, estimation, classification. errors or code coverage analysis tools, that can measure the actual • scenario evaluation, which concerns the system’s behavior in par- coverage of the software with the available test data sets. For more ticular situations - for a specific functionality with its sets of vari- information on software testing and other QA techniques we refer ables (e.g, number of users, type of lighting). The test data is the interested reader to [8], [1], [12]. based on a controlled real world and is therefore only partly re- producible. More complex metrics are to be used here, e.g., sys- 3 Code Coverage Analysis tem reliability expressed as mean time between failures. Among the first quality assurance metrics invented for systematic [11] takes the topic of technology evaluation a step further by software testing, code coverage is used to describe the degree to defining a set of eight key questions, thought to highlight the best which the source code of a program is tested by a particular test practices and the state of evaluation methodology in several repre- suite. Test coverage can be used in unit testing, regression testing, sentative areas of computer vision discipline: sensor characterization, for test case order optimization, test suite augmentation or test suite feature detection, shape- and grey-level-based object localization, minimization. shape-based object indexing: recognition, lossy image and video The code coverage analysis process is generally divided into code compression, differential optical flow, stereo vision, face recognition, instrumentation, data gathering, and coverage analysis. Code instru- measuring structural differences in medical images. From the guid- mentation consists in inserting some additional statements, that mon- ing questions formulated in [11], we selected those, which are in our itor the execution of the source code. The instrumentation can be opinion first to be answered in algorithmic testing: done basically at code level in a separate pre-processing phase or at runtime. 1. Is there a data set for which the correct answers are known? In order to be self-contained, we briefly introduce here the most 2. Are there data sets in common use? commonly used code coverage metrics, as they might be new to the computer vision community. We further refer to the following small code snippet to quickly highlight their major advantages/ disadvan- tages in practice: if (x>1 && y==0) { z=z+1; } if (x==2 || z>1) { z=z+2; } As already mentioned, several kinds of instrumentation are possi- ble. The most common are for: • line or statement coverage: where the tool instruments the execu- Figure 1. One-to-one correspondence between development and testing tion of every executable source code line; this coverage criterion processes. is a rather poor one, as it is completely insensitive to some control structures and logical operators. For instance, one could execute every statement (reaching a 100% plates defined in CV applications and which have to be taken into line coverage) from our example by writing a single test case: consideration when analyzing the code coverage. Tools that can- T1(x=2, y=0, z=4). Now, let us assume that the second de- not handle templates appropriately are dismissed. cision should have stated z>0. If so, this error would not be de- 2. unit testing support: in our case, CPPUnit unit testing framework tected. Or perhaps in the first decision should be an or rather than support is needed, as this is the most frequently used framework an and. This error would also go undetected. in C/C++ CV applications. • decision or branch coverage: it reports whether each decision has 3. excluding 3rd party libraries from the coverage analysis: as most a true and a false outcome at least once; this criterion is of the CV applications make use of third party libraries, whose stronger than line coverage, but it is still rather weak. analysis is obviously not desired, the tool has to provide a simple For instance, with our previous test-case inputs T1(x=2, y=0, way to hook/instrument only certain files. z=4) and a new one T2(x=3, y=1, z=1), we can reach full 4. automated testing/non-interactive testing: taken into considera- decision coverage. However, if in the second decision we should tion the high complexity of the currently developed CV software, have had z<1 instead of z>1, the mistake would not be detected an easy automation of the test coverage analysis is essential. by the two test cases. 5. performance under big test data amounts: There is no doubt that • condition coverage: in this case, one has to write enough test cases the insertion of instrumentation will increase the code size and to ensure that each condition in a decision takes on both true and affect the instrumented applications performance, i.e., it will use false outcomes at least once; this metric is similar to decision more memory and run slower. A low performance overhead is of coverage, but has better sensitivity to the control flow. However, course desired, however, considering the complexity of the target full condition coverage does not guarantee full decision coverage. programs, our requirement is that the analysis tool does not crash. For instance, the following test cases: T3(x=1, y=0, z=4) and T4(x=2, y=1, z=1) cover all conditions’ outcomes, but they cover only two of four decisions’ outcomes. 3.2 Four state-of-the-art Code Coverage Tools • function coverage: reports whether each function is called (and Identifying the right tool for code coverage analysis in vision appli- how many times); it is useful during preliminary testing to quickly cations can lead to major productivity improvements and implicitly find coarse deficiencies in a test suite. to increases in the release quality of the overall computer vision sys- tem. Hence, various free and commercial coverage analyzers have 3.1 CV tailored Evaluation Criteria been inspected and compared. As a large variety of coverage metrics exist (see the preceding summary), the QA team imposed as require- Following the classic software development process depicted in Fig- ment that the code coverage tool should be able to measure at least ure 1, we first learned that code coverage analysis does not exist in condition coverage. This requirement together with the previously most of the CV projects. As a result, we tried to identify the must- presented CV tailored evaluation criteria have led to limiting our have and nice-to-have features of a code coverage tool to be used in comparative evaluation to the following four state-of-the-art com- the CV application domain. Like in any tool selection process, one mercial coverage tools: C++ coverage validator1 , Squish Coco code has to clarify first the user’s requirements. We will further present coverage tool2 , BullseyeCoverage tool3 , Testwell CTC++ analyser4 only those particular requirements related to computer vision soft- . ware, and neglect general questions such as: what platforms can the tool run on, what is the target application’s language or which are the supported compilers. We will not mention here requirements com- 4 Case Study: Dibgen and Dibgiom Libraries ing from the quality assurance team, which are to be discussed in the Dibgen is a collection of basic C++ libraries used particularly, next section. but not exclusively, in computer vision applications implemented The following list ranks the priorities of these specific features, as discussed with CV software developers: 1 http://www.softwareverify.com/cpp-coverage.php 2 http://www.froglogic.com/squish/coco/index.php 1. working with templates: due to the great variety of data types 3 http://www.bullseye.com/measurementTechnique.html 4 http://www.verifysoft.com/de cmtx.html (pixel and parameter types), there is a tremendous number of tem- by JOANNEUM RESEARCH (JR). Included libraries cover ba- Core(TM) i7-4500U CPU 1.80 GHz and 8 GB of RAM running un- sic, mostly matrix based mathematical operations, color handling der Windows 10 Pro. Although the running time for the program and evaluation, as well as generic parameter storage, progress in- invoked by Coverage Validator is approximately six times higher, we formation handling, different types of basic file IO methods often have no source code instrumentation involved, i.e., there is no need to used in computer vision, and value-to-string conversion (and back- recompile or relink the target program. The only requirement is the conversion). All the libraries are implemented using template-heavy existence of PDB files with debug information and/or MAP files with C++ code allowing the usage of different data types (pixel types, line number information. Therefore we chose to further use the Cov- parameter types) for most of the operations. In terms of volume, erage Validator tool for the first Dibgiom experiments. The results Dibgen consists of approximately 100000 LOC. can be seen in Table 5. The other partially analyzed collection was Dibgiom. Seen as OpenCV counterpart and based on Dibgen, it contains 15 libraries, Table 4. Running Times for the unit tests defined for the Dibgen Solution which are all used for image processing tasks. The library consists For non-instrumented programs 68,44 sec of approximately 9 MB of source code and approximately 255000 For programs invoked by Coverage Validator 475,68 sec LOC. We further provide a brief description of those Dibgiom li- For CTC++ instrumented programs 74,16 sec braries, which were yet analyzed: For Bullseye instrumented programs 68,97 sec For Squish Coco instrumented programs 70,81 sec • Band: Various representations of image data in the memory (tiled with FileIO for huge satellite data, pure memory-based for rapid CPU access, specially aligned memory layout for acceleration us- ing Intel Performance Primitives, special layout for CUDA accel- Table 5. Dibgiom Coverage Results computed with C++ Coverage eration), transparently accessible via the same interface to both Validator user and algorithms. Library Decision Coverage Function Coverage • BandIterator: Generic access iterators for bands regardless Band 36,84% 53,55% of memory layout (see above) BandIterator 13,94% 67,12% • Calibration: Simple radiometric calibration methods Calibration 73,58% 13,33% Convolve 14,52% 34,77% • Convolve: Image filter based on convolution (Gauss, Laplace, Detect 56,25% 41,72% etc.) Filter 30,02% 73,66% • Detect: Various detectors (Extrema, Bright Spot, Corner, etc.) KernelFilter 40,08% 48,86% • Filter: General image-filter (arithmetic, logic, etc.,), that con- KeyPoint 86,99% 70,64% vert, in principle, a pixel in the source image(s) to a pixel in the Operation 54,53% 68,21% target image Pyramid 2,01% 12,71% Segmentation 87,72% 87,27% • KernelFilter: Core-based image filters (mean, median, etc.), Sift 78,10% 66,34% which do not calculate any convolution • KeyPoint: Description of key points for various detectors • Operation: Operations on images whose result, or whose source is not an image (source no image: filling images, etc., or target no image: the sum of all the pixels in the image) 4.1 First Success Stories • Pyramid: Generation of pyramid representations (Gauss, etc.) • Segmentation: Image-based operations that compute segmen- One of the most complex and frequently used basic libraries in tations from arbitrary source images (Watershed, RegionGrowing, the JR’s CV applications is the library ParameterPool from the etc.) Dibgen collection. With about 17.000 LOC, the library is used to • Sift: Special version of a Sift detector. store any kind of parameters of arbitrary types in one container. Each parameter can be combined with validity information, access level For the Dibgen experiments we used the same unit test suites permission for user interface based parameter modifications, as well and the same configuration for all the four coverage tools. Although as several kinds of descriptive text (unit, help text). Additionally, pa- each tool features more than just decision and function coverage, we rameters can be grouped together and it is possible to define several will merely present the comparison of these two types of coverage types of parameter dependencies. Since this library is used heavily in measurements, as only they are computed by all the four tools. nearly every JR CV application, the JR developers particularly paid The tests carried out for the Dibgiom experiments are also unit attention to test it thoroughly from the very start of development. tests, in which the source data is generated either directly by means of However, first code coverage analysis showed dissenting results, using unit-test programs (usually only for very simple algorithms), or especially in branch, function and line coverage, while at least file by reading the image data from files. In the latter case, the expected coverage could reach nearly acceptable results (see Figure 2). More outcome is generated with other reference implementations chosen detailed analysis showed that only 12 out of 36 source code files from the literature (like MATLAB, OpenCV, etc.) and it is further had a line coverage better than 90%, while 9 files were not tested compared with the outcome produced by Dibgiom. at all (see Figure 4). Although the remaining 15 files were tested at In Table 1 we list the global results for the whole Dibgen test least partially from the line coverage point of view, especially their application, while in Table 2 and Table 3 we present the coverage re- branch coverage showed very poor results. After particular review of sults per directory. It is worth noting that with Testwell CTC++, the the tested source code, the used test code as well as the used test data, coverage results are extremely low, while the other three tools com- the test code has been adapted in some places and some test data sets pute comparable coverage results. Table 4 depicts the running times have been slightly modified. for the normal, uninstrumented program and for the instrumented Additionally, some new test functions were developed, especially programs. Note that the tests were run on a notebook with Intel(R) for previously untested files or functions. One meanwhile unused Table 1. Overall Dibgen Coverage Results C++ Coverage Validator Testwell CTC++ BullseyeCoverage Squish Coco Decision Coverage 31,27% 9% 40% 45,30% Function Coverage 39,86% 8% 52% 51,95% Table 2. Dibgen Coverage Results per Library (Decision Coverage) C++ Coverage Validator Testwell CTC++ BullseyeCoverage Squish Coco Color 52,91% 9% 75% 78,65% Exception N.A. 30% 12% 40% Fileio 15,15% 15% 44% 39,65% Internationalisation 55,81% 77% 61% 70,89% Math 62,36% 3% 30% 39,54% ModuleInterface 14,05% 5% 23% 28,54% ParameterPool 12,39% 6% 46% 58,95% ParameterPoolDocumentation N.A. 0% 0% N.A. ProgramOptions 0% 0% 59% 0% Progress 55,07% 12% 47% 54,51% ResultDataPool 5,31% 1% 20% 29,51% Serialization 10,14% 11% 86% 76,31% Strings 45,50% 36% 44% 50,66% Types 70% 1% 8% 20,28% UserDataBase 10,94% 77% 77% 81,51% Utilities 0% 3% 0% 23,18% Table 3. Dibgen Coverage Results per Library (Function Coverage) C++ Coverage Validator Testwell CTC++ BullseyeCoverage Squish Coco Color 59,41% 6% 81% 81,17% Exception 32,50 36% 45% 45,45% Fileio 16,81% 26% 60% 56,52% Internationalisation 77,67% 95% 94% 94,44% Math 74,22% 3% 47% 47,08% ModuleInterface 23,49% 4% 37% 35,68% ParameterPool 36,13% 6% 72% 69,17% ParameterPoolDocumentation N.A. 0% 0% N.A. ProgramOptions 0% 0% 50% 0% Progress 14,30% 9% 63% 68,46% ResultDataPool 8,60% 2% 37% 34,69% Serialization 11,22% 6% 86% 86,20% Strings 49,46% 37% 65% 64,66% Types 51,87% 1% 22% 22,88% UserDataBase 10,45% 80% 86% 86,20% Utilities 0% 7% 26% 30,76% Figure 2. Coverage Validator’s summary tab before improvements. Figure 3. Coverage Validator’s summary tab after improvements. Figure 4. Coverage Validator’s Files and Lines tab before improvements. Figure 5. Coverage Validator’s Files and Lines tab after improvements. source code file could be entirely removed. Two of the untested however, the classical hierarchy of vision systems, which groups the source code files contained only source code that is used to dis- them into low-, mid- and high-level processing levels, could serve as able default class behavior (make default constructor, copy con- a starting point. At low-level vision, code structure and data represen- structor and/or assignment operator private), which makes this code tation are still in close correlations (in other words, every pixel has to untestable by design. Altogether, all of these mentioned modifica- be treated by some kind of operation/code), thus code improvement tions did not touch more than 10% of the test code, but resulted in a by QA directly affects the data quality. For example, filtering oper- huge improvement in all code coverage measures (see Figure 3). As ations by convolutions (as those contained in our Dibgiom library) one can see in Figure 5, now from the remaining 35 source code files, are many simple code snippets executed many times sequentially or 32 reach a line coverage above 90% (23 of which even reach 100% in parallel, thus even small code discrepancies produce a large effect, - compared to only 9 before the modifications were made). The 2 which easily propagate further to higher processing levels. Mid- and still remaining untested files contain the above mentioned disabling high-level vision algorithms on the other hand, are more difficult to source code. tackle, because the representations fall into one of the exponentially By improving the test code and the test data for the exemplarily many branches of different meta-data types, where often the same chosen library, 3 implementation errors were found and corrected, 2 meta-data can be produced by fundamentally different code pieces. of which can be considered to potentially cause major problems in applications. Spending some effort in QA and improving the cover- ACKNOWLEDGEMENTS age of the tested source code will already pay off in the near future in several stages of the testing process; especially in regression and This work was partly funded by BMVIT/BMWFW under COMET integration tests. programme, project no. 836630, by ”Land Steiermark” trough SFG under project no. 1000033937, and by the Vienna Business Agency. 5 Conclusions REFERENCES In this paper we presented our first steps in the direction of construct- [1] B. Beizer, Black-box testing: techniques for functional testing of soft- ing a generic testing and evaluation protocol for CV applications. In ware and systems, NY, USA, 1995. our view, the performance characterization methodology in the do- [2] Kevin Bowyer and P. Jonathon Phillips, Empirical Evaluation Tech- main can successfully be complemented with well known techniques niques in Computer Vision, IEEE Computer Society Press, Los Alami- tos, CA, USA, 1st edn., 1998. borrowed from a typical quality assurance process. [3] Patrick Courtney and Neil A. Thacker, ‘Imaging and vision systems’, The conducted experiments on JR’s source code demonstrated that chapter Performance Characterisation in Computer Vision: Statistics with little effort, by means of using a code coverage analysis tool for in Testing and Design, 109–128, Nova Science Publishers, Inc., Com- the available unit tests, the CV developers can considerably improve mack, NY, USA, (2001). their code, and implicitly the release quality of the overall CV sys- [4] Robert B. Grady, Practical Software Metrics for Project Management and Process Improvement, Prentice-Hall, Inc., Upper Saddle River, NJ, tem. USA, 1992. After finishing unit/module-testing the program, we have to per- [5] Robert M. Haralick, ‘Performance characterization in computer vision’, form higher-order testing, as for instance integration and system tests CVGIP: Image Underst., 60(2), 245–249, (September 1994). (see Figure 1), in order to complete the testing process. Therefore, [6] Matej Kristan, Jiri Matas, Ales Leonardis, Tomas Vojir, Roman P. Pflugfelder, Gustavo Fernández, Georg Nebehay, Fatih Porikli, and together with JR, we analyzed the requirements and possible use Luka Cehovin, ‘A novel performance evaluation methodology for cases/hazards of one CV application, which was chosen as repre- single-target trackers’, CoRR, abs/1503.01313, (2015). sentative candidate in the Vision+ project3 . We paid particular at- [7] Moritz Menze and Andreas Geiger, ‘Object scene flow for autonomous tention to the process of test case definition, with focus on: require- vehicles’, in Conference on Computer Vision and Pattern Recognition ment(s)(from the requirements specification) related to a particular (CVPR), (2015). [8] G. J. Myers, The Art of Software Testing, New Jersey, Second Edition test case, its prerequisites (any conditions that must be fulfilled prior edn., 2004. to executing the test), its detailed setup and preferred execution pro- [9] P. Jonathon Phillips, Hyeonjoon Moon, Syed A. Rizvi, and Patrick J. cedure (automated/manual). However, as usually a test management Rauss, ‘The FERET Evaluation Methodology for Face-Recognition Al- tool is used to accomplish the task, we further encourage CV devel- gorithms’, IEEE Trans. Pattern Anal. Mach. Intell., 22(10), 1090–1104, (October 2000). opers to consider the integration of such a tool in their projects. Our [10] Daniel Scharstein, Heiko Hirschmller, York Kitajima, Greg Krathwohl, colleagues from JR have already started out on analyzing the test Nera Nesic, Xi Wang, and Porter Westling, ‘High-resolution stereo management tools available for managing functional software and datasets with subpixel-accurate ground truth.’, in GCPR, eds., Xiaoyi hardware testing in agile development projects. Some of the benefits Jiang, Joachim Hornegger, and Reinhard Koch, volume 8753 of Lec- one gains are the assurance of the complete test cycle, the repeata- ture Notes in Computer Science, pp. 31–42. Springer, (2014). [11] N. A. Thacker, A. F. Clark, J. Barron, R. Beveridge, C. Clark, P. Court- bility of tests as well as the automatic generation of statistics and ney, W. R. Crum, and V. Ramesh. Performance characterisation in com- reports. puter vision: A guide to best practices, 2005. Finally, we would like to summarize the main ideas, which will [12] K. Wiegers, Peer Reviews in Software: A Practical Guide, Addison- further lead our work presented in this paper. On one hand, as re- Wesley, 2002. [13] Oliver Zendel, Wolfgang Herzner, and Markus Murschitz, ‘VITRO - sources are always limited, we have to find the right mixture of QA vision-testing for robustness’, ERCIM News, (97), (2014). techniques and to focus towards specific CV pain points. In order to [14] Oliver Zendel, Markus Murschitz, Martin Humenberger, and Wolfgang do this, it is important to determine the desired quality attributes for Herzner, ‘CV-HAZOP: introducing test data validation for computer CV applications. On the other hand, we have to find a way to derive vision’, in 2015 IEEE International Conference on Computer Vision, applicability rules for certain sets of CV algorithmic classes. Due to ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 2066–2074, (2015). the vast diversity of CV algorithms, these tasks are rather difficult, 3 http://comet-visionplus.at/