In solving the problem of automated analysis of football match video recordings, special video cameras are currently used. This work presents a comparative characterization of known algorithms and methods for video camera calibration, including those utilizing machine learning and neural networks, with the aim of identifying their shortcomings and forming a theoretical foundation for developing modern, more effective methods and algorithms. Specifically, it examines both algorithms that require more input data but operate quickly [1, 2] and more accurate algorithms using machine learning [3, 4, 5, 6, 5]. It is demonstrated that their main drawback is either accuracy or speed. More accurate algorithms using machine learning often do not specify the algorithm's operational speed, which precludes their use in realtime applications. The examined works that emphasize speed frequently lack the accuracy necessary for practical use in real-life scenarios.
2.1.
For the classic pinhole model, the basic formula of perspective projection is given by: λm m= K [ R T ] M , where:
M denotes a 3D point and m denotes the corresponding 2D point on image. They are both expressed in homogeneous coordinate and λm is an arbitrary scale factor.
R is 3 x 3 rotation matrix that describes the rotational mapping from the world coordinate system into the camera coordinate system.
T is a 3 x 1 vector that describes the translational mapping from the world coordinate system into the camera coordinate system.
K is a 3 x 3 matrix describing the internal camera parameters: where:
Scale factor f applies to both the u and v axes of an image, while s describes the skew between these two axes.
Beta accounts for non-isotropic scaling, and the coordinates (uo,vo) denote the principal point.
When the observed 3D points lie on a plane, this projection can be simplified to a homography, which is a 3x3 matrix mapping between two planar surfaces. When considering perspective projection, an interesting phenomenon occurs with parallel lines in 3D space. If these lines are not parallel to the image plane, their 2D projections converge to a single point in the image. This point of convergence is termed the vanishing point. Notably, the line that connects this vanishing point to the optical center of the camera runs parallel to the corresponding 3D lines in space. Consequently, all sets of parallel lines in 3D space that share the same direction will correspond to the same vanishing point in the 2D image. This principle is fundamental to understanding how 3D scenes are projected onto 2D images in perspective projection. 2.2.
Camera calibration involves determining its internal parameters (focal length, pixel ratio, projection center) and external parameters (rotation and translation expressing the camera’s position and orientation relative to the world coordinate system).
Early approaches rely on matching local features in combination with direct linear transformation (DLT) to estimate homography. One of the first algorithms for determining these parameters is vanishingpoint based calibration (VPBC).
The study [
In [
Both studies - [
One of the first applications of the aforementioned algorithms in football is described in [
The first stage involves detecting straight lines or their segments. For this purpose, Hough methods [11] or edge segmentation methods [12] are used. The detected lines are grouped into vertical and horizontal sets.
The second stage involves matching these two sets of image segments with segments in the football field model. Matching occurs by identifying segments that intersect with each other. Having two vanishing points, the algorithm selects segments that best correspond to the field model constructed using these vanishing points. After this, rotation (R) and translation (T) matrices are calculated to obtain the final camera model. An example of the algorithm’s sequential operation on real images is shown in Figure 2.
However, all the aforementioned algorithms have limitations and work effectively only under certain conditions and with specific input data. In the modern world, this is insufficient for a fully functional analytics system, the foundation of which is determining the homography matrix in each frame of the video stream.
Further discussion will be devoted to methods that not only find key points but also continue calculating the homography matrix in subsequent frames. These methods allow for the creation of more reliable and flexible systems for analyzing football matches, capable of working in various conditions and with different types of input data. A comparison of the aforementioned algorithms in terms of their application in real conditions of modern football is presented in Table 1. 2.3.
The study [13] is one of the pioneering works in the field of not only determining the homography matrix for individual frames but also tracking its changes in a sequence of frames. The authors first proposed a combined approach for automatic computation of homography during camera motion. This approach includes using the KLT system [14, 15, 16] for automatic detection of correspondences between frames by extracting characteristic features. Since these correspondences are not perfect and contain outliers, the RANSAC algorithm [17] is applied to filter out incorrect matches. After this additional selection, the DLT algorithm is used to compute a new homography matrix for the current frame. An example of the algorithm’s output is shown in Figure 3.
However, this method has a significant drawback: with each new frame, the reprojection error accumulates, leading to inaccuracies in the homography matrix. To address this issue, the authors propose periodically adjusting the homography matrix using key points on the field lines.
It is important to note that the study does not specify the accuracy of the proposed algorithm. This approach laid the foundation for further research in the field of dynamic homography determination in video sequences.
The study [18] proposes one of the first approaches to determining not only the homography matrix but also the camera rotation angles. The authors developed a method that uses prior information about key points in the goal area to calculate these parameters with a fixed focal length. Experimental verification of the algorithm was conducted on a sample of 500 frames, and the researchers claim to have achieved reprojection accuracy within 2 pixels. It is important to note that the work focuses on theoretical aspects of determining rotation angles without considering the practical application of this method to frames that do not contain the goal area. A visual representation of the reprojection results obtained using this algorithm can be seen in Figure 4.
Despite using lines and camera parameters, [
The accuracy of this approach is also not specified.
The study by Zhang et al. [21] describes a method for simplifying camera calibration and finding the transformation matrix by leveraging the specifics of a particular task. Traditionally, the DLT algorithm required four non-collinear key points to obtain the homography matrix. Instead, the authors propose the PCC (Pan-tilt camera calibration) algorithm, which takes into account the specifics of game filming where the camera remains stationary and only pan-tilt parameters change. This approach allows reducing the number of required key points for calibration to two. The PCC calibration process consists of two stages: first, initial camera calibration is performed using four points to determine fixed parameters, and then the Levenberg-Marquardt algorithm [22] is applied to find the homography matrix using only two key points. An important innovation in this work is the use of the offset line as a source of key points for determining the transformation matrix. The authors conducted a comparative analysis of the accuracy of both algorithms using computer simulation, which showed that the accuracy of the PCC algorithm surpasses that of the DLT algorithm.
A comparison of the aforementioned algorithms in terms of their application in real conditions of modern football is presented in Table 2.
Machine learning is a powerful approach in the field of artificial intelligence that uses statistical methods to analyze large volumes of data. This technology allows algorithms to detect complex patterns and make accurate predictions, finding applications in many areas - from speech recognition to autonomous vehicle control.
In the context of camera calibration and finding the homography matrix, [
Examples of the algorithm’s results are shown in Figure 6.
An alternative approach using machine learning was proposed [
In [23], a method is proposed that uses two neural networks: the first determines the initial
homography matrix, while the second estimates the registration error. The process involves
transforming the sports field template to the current perspective, combining the transformed image
with the current one, and iteratively updating the homography parameters to minimize the error. The
authors evaluated the accuracy of their method on WorldCup and hockey match data [
A similar method is presented in [
The researchers in [26] introduce a novel approach using evenly spaced keypoints as field-specific features, framing the task as an instance segmentation problem with dynamic filter learning. To validate their method, they created the TS-WorldCup dataset, which comprises 3,812 sequential images from 43 videos of the 2014 and 2018 FIFA World Cup tournaments, featuring precise field markings. The method employs a standard encoder-decoder architecture similar to U-Net, with a ResNet-34 backbone for the encoder. It introduces a keypoints-aware label condition, using 91 pre-defined keypoints and dynamically generated convolution kernels. The approach utilizes a keypoints-specific controller and dynamic head to predict keypoint heatmaps, which are then merged to estimate the final homography using DLT and RANSAC. The proposed method demonstrated strong performance when evaluated on the WC and TSWC datasets, achieving IOUpar and IOUwhole scores of 0.96 and 0.91 for WorldCup, and 0.97 and 0.93 for TSWC, respectively.
In their study, [27] introduce speed metrics as a measure of performance in keypoint detection. The authors propose a dual-model approach, employing separate deep learning models for point and line detection, both utilizing heatmap-based techniques. For keypoint extraction, they adopt the widelyused HRNetV2-w48 model as their backbone architecture. The researchers report a processing time of 33.6 ms per image using a single Nvidia GeForce RTX 3090 GPU. While this performance approaches real-time capabilities, further optimization is necessary for widespread practical application.
In contrast, [
Static calibration methods paved the way for more accurate and powerful approaches. The methods of
[
The importance lies not only in the accuracy and speed of algorithms but also in their field coverage. A football field does not always contain enough lines to determine key points based on their intersections. The existing field coverage issues of previous approaches are partially addressed in [13]. However, the authors do not specify the algorithm’s accuracy, only its speed - 1900 frames per hour, or 2 frames per second on a 2.8 GHz Pentium IV processor. Modern cameras record at a minimum of 24 frames per second, which would result in a very long wait time to process a 90-minute football match.
Camera parameter determination in [18] does not resolve accuracy or speed issues of systems,
focusing more on adding content to existing broadcasts or videos, which does not require high
precision. Accuracy assessment is mentioned in [
However, improvements in accuracy and coverage depend on well-annotated key frames required for the algorithm’s operation. This condition precludes application to real football match videos, as new annotated key frames would need to be created for each new camera or stadium.
With the advancement of machine learning, camera calibration methods have also evolved. In [
Table 3 Comparison table for machine learning based calibration methods
Optimizing Through
Learned Errors for
Accurate Sports
Field
Registration [23]
TVCalib: Camera
Calibration for
Sports Field
Registration in
Football [
In [
Recent advancements in real-time keypoint detection algorithms have demonstrated significant
progress in processing speed and efficiency. Falaleev et al. [27] and Gutierrez et al. [
However, the performance characteristics of these smaller models in real-time keypoint detection
tasks remain unexplored in the current literature. Furthermore, the studies by Falaleev et al. [27] and
Gutierrez et al. [
An analysis of current literature in the field of machine learning for camera calibration and homography matrix determination reveals a significant shortcoming in algorithm description and evaluation: a considerable portion of the presented methods lacks comprehensive information regarding quality metrics or performance speed. This limitation is particularly noticeable in machine learning algorithms, where information about the algorithm’s operational speed is often absent. Such a situation creates serious obstacles for objective assessment of algorithm efficiency and their comparison.
The lack of performance data significantly limits the application of these algorithms in real-time systems, where data processing speed is a critical factor. Many algorithms that demonstrate high accuracy on test datasets may prove unsuitable for practical use due to low operational speed.
To overcome these limitations, it is necessary to focus on developing algorithms in the direction of improving their performance. This includes optimizing existing algorithms, developing new approaches with an emphasis on computational efficiency, as well as applying parallel computing and specialized hardware. It is important for researchers and developers to pay more attention to comprehensive algorithm evaluation, including both quality and performance metrics, which will expand their scope of application and increase efficiency in real conditions.
It is worth noting that the authors of the aforementioned works do not consider a number of important optimization techniques for computer vision models. In particular, methods such as Pruning, Quantization, Knowledge Distillation, and Sparsity remain overlooked. These techniques have significant potential for substantially accelerating model performance while maintaining their effectiveness.
The use of these methods allows for the optimization of large and powerful models, reducing their computational requirements without significant loss of quality. For example, Pruning allows for the removal of the least important weights in a neural network, Quantization reduces the precision of parameter representation, Knowledge Distillation transfers knowledge from a large model to a smaller one, and Sparsity introduces sparseness into the network architecture.
The absence of consideration of these techniques in the analyzed works indicates a potential direction for further research and improvements in the field of computer vision model optimization.
In this work, studies related to camera calibration for determining the homography matrix for subsequent transformation of 2D coordinates into 3D coordinates were analyzed. The analysis encompassed both foundational works associated with general camera calibration algorithms and more contemporary developments utilizing machine learning. It was ascertained that works without using of machine learning do not specify algorithm accuracy. Machine learning approach demonstrate sufficient accuracy but lack the necessary speed for comfortable use on offline recordings, as well as for potential future real-time application during broadcasts. The study analyzed and presented the main shortcomings of these works, conducted a comparative analysis of solutions, and identified directions and ideas for future improvements in this field.
The research covered a range of approaches, from basic camera calibration methods to advanced machine learning techniques. It revealed that traditional methods often lack precise accuracy metrics, while machine learning approach, though accurate, fall short in terms of processing speed. This speed limitation hinders their practical application in both offline video analysis and real-time broadcast scenarios.
A critical evaluation of existing methodologies highlighted their respective strengths and weaknesses. The comparative analysis provided insights into the effectiveness of various solutions, considering factors such as accuracy, field coverage, and computational efficiency. This comprehensive review served to pinpoint areas where current approaches fall short and where future research efforts should be concentrated.
Based on the findings, several directions for future research and development were identified. These include the need for improved algorithm speed optimization, especially for machine learning-based methods, without compromising accuracy. Additionally, the potential for incorporating advanced optimization techniques such as pruning, quantization, knowledge distillation, and sparsity in model architectures was emphasized as a promising avenue for enhancing both accuracy and computational efficiency.
The work underscores the importance of developing algorithms that not only achieve high accuracy but also demonstrate practical applicability in real-world scenarios, particularly in the context of sports analytics and broadcast technologies. By highlighting these areas for improvement, this analysis provides a valuable foundation for future research aimed at advancing the task of camera calibration and homography matrix determination. [11] D. Farin, S. Krabbe, W. Effelsberg, et al., Robust camera calibration for sport videos using court models, in: Storage and Retrieval Methods and Applications for Multimedia 2004, volume 5307, SPIE, 2003, pp. 80–91. [12] J.-B. Hayet, J. Piater, J. Verly, Incremental rectification of sports fields in video streams with application to soccer, in: Advanced Concepts for Intelligent Vision Systems (ACIVS 2004), 2004. [13] K. Okuma, J. J. Little, D. G. Lowe, Automatic rectification of long image sequences, in: Asian conference on computer vision, volume 9, 2004. [14] S. T. Birchfield, Depth and motion discontinuities, stanford university, 1999. [15] J. Shi, C. Tomasi, Good features to track, Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 600 (2000). doi:10.1109/CVPR.1994.323794. [16] C. Tomasi, T. Kanade, Detection and tracking of point, Int J Comput Vis 9 (1991) 3. [17] M. A. Fischler, R. C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM 24 (1981) 381–395. [18] V. Babaee-Kashany, H. R. Pourreza, Camera pose estimation in soccer scenes based on vanishing points, in: 2010 IEEE International Symposium on Haptic Audio Visual Environments and Games, IEEE, 2010, pp. 1–6. [19] W. Förstner, T. Dickscheid, F. Schindler, Detecting interpretable and accurate scale-invariant keypoints, in: 2009 IEEE 12th International Conference on Computer Vision, IEEE, 2009, pp. 2256–2263. [20] D. G. Lowe, Distinctive image features from scale-invariant keypoints, International journal of computer vision 60 (2004) 91–110. [21] S. Zhang, Research on camera calibration in football broadcast videos, International Journal of uand e-Service, Science and Technology 8 (2015) 89–98. [22] Z. Niu, X. Gao, D. Tao, X. Li, Semantic video shot segmentation based on color ratio feature and svm, in: 2008 International Conference on Cyberworlds, IEEE, 2008, pp. 157–162. [23] W. Jiang, J. C. G. Higuera, B. Angles, W. Sun, M. Javan, K. M. Yi, Optimizing through learned errors for accurate sports field registration, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 201–210. [24] L.-C. Chen, G. Papandreou, F. Schroff, H. Adam, Rethinking atrous convolution for semantic image segmentation, arXiv preprint arXiv:1706.05587 (2017). [25] A. Cioppa, A. Deliege, S. Giancola, B. Ghanem, M. Van Droogenbroeck, Scaling up soccernet with multi-view spatial localization and re-identification, Scientific data 9 (2022) 355. [26] Y.-J. Chu, J.-W. Su, K.-W. Hsiao, C.-Y. Lien, S.-H. Fan, M.-C. Hu, R.-R. Lee, C.-Y. Yao, H.-K. Chu, Sports field registration via keypoints-aware label condition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3523–3530. [27] N. S. Falaleev, R. Chen, Enhancing soccer camera calibration through keypoint exploitation, arXiv preprint arXiv:2410.07401 (2024). [28] A. Cioppa, S. Giancola, V. Somers, F. Magera, X. Zhou, H. Mkhallati, A. Deliège, J. Held, C.
Hinojosa, A. M. Mansourian, et al., Soccernet 2023 challenges results, Sports Engineering 27 (2024) 24.