Pallets are critical components in the logistics of food and beverage products transportation, and their stability is essential to ensure reliability of the handling system as well as safety during road and rail freight. This work aims to develop an innovative solution for evaluating pallet stability very early, ideally during the design phase of the pallet schema and the wrapping format. Diferently from other investigations, the goal of our work is to analyse the dynamics of pallets wrapped in an envelope of paper material instead of plastic. By collecting raw video data from an acceleration test bench, and using computer vision and machine learning techniques, we develop a physically realistic multi-body simulation. The simulation is completely virtual and capable to evaluate the stability of the pallet under diferent configurations and loading conditions.
LLDPE (Linear Low Density Polyethylene) stretch film, or the heat-shrink wrap, is still the favorite choice in the automatic wrapping of pallets stacked with food or beverage products. Such a wrapping is called tertiary packaging, a term which refers to the external envelope that surrounds both a wooden pallet and the products on it. The single product’s actual container and their grouping into a larger set for handling purposes are called, respectively, primary and secondary packaging. lution and plastic usage, like the European Strategy for
LLDPE plastic film with Kraft paper in automatic pallet wrapping on FSE REACT-EU, by Ministero dell’Università e della Ricerca (MUR), under the Programma Operativo Nazionale (PON) “Ricerca e Innovazione” 2014-2020–Azione Green. E. Iotti is funded by this project. E. Iotti and A. Dal Palù are members of the INdAM Research group GNCS. Partially supported by INdAM–GNCS projects CUP E55F22000270001 and CUP E53C22001930001. ∗∗Corresponding author. Palù) avoid loss of products, and for the safety of truck workers.
Rules to certify an adequate safety during transports are encoded in the European Road Worthiness Directive, and in particular in the European Standard EUMOS 40509 [3].
The stability analysis requires a deep understanding of the wrapping material behavior in diferent conditions.
For example, it is crucial to realize how many layers of paper are needed for wrapping and how they should be stratified, and how much pulling tension has to be applied to paper while wrapping. Also, the design of the pallet loading schema considerably impacts on the dynamic of Figure 1: The ESTL Machine in the R&D Department of ACMI the system, therefore the analysis must also take into wS.opo.Ad.enThpealalectceislelroaatidoend bweinthchsohmoledslaoynersa osflepigrohdtuwcthse,raenda account that information. Finally, acquiring knowledge wrapped with Kraft paper. about the dynamic of a paper-wrapped envelope is of key importance to provide engineers hints for the actual development of the automatic wrapping machine and its controlling software. with constant accelerations from 0.2 up to 0.5 , which
Our work in collaboration with ACMI S.p.A. is aimed is the acceleration to be supported. The acceleration may at searching methods to perform such a stability analysis cause elastic deformations and permanent deformations, on pallets wrapped with Kraft paper. The idea is to inves- which are, respectively, the deformation of the load unit tigate pallet stability by means of a multi-body dynamics during the test, and the residual deformations of the load simulation, capable to virtually reproduce the behavior of unit after the test ended. the load unit when subject to external forces. This paper EUMOS 40509 quantifies thresholds and limits of these describes the set up of such a virtual simulation, starting deformations and shifting, beyond which the unit could with the collection of real-world data from the observa- impact severely on the stability of the overall truck, thus tion of actual physical tests. Such data are analysed with making unsafe the transport. The specifications of EUcomputer vision techniques. In order to setup the physics MOS 40509 state that (i) the permanent displacement simulation, we need to learn parameters that controls of all parts of the test load unit (after the test) must not the dynamics: in particular, the retrieval of (i) static and exceed 5% of the total height of load unit; (ii) permanent dynamic friction forces at work between each bundle of displacement in the lowest 20 of the test load unit is the test and (ii) elastic coeficient that model tensions at to be less than 4 on the wooden pallet; (iii) the elastic each point of a mesh representing the wrapping envelope. displacement of all parts of the test load unit (during the We design the resolution of two inverse problems, one test) must not exceed 10% of the total height of load unit; for each kind of parameters. (iv) there must be no visible structural damage and/or leakage of products at the end of the test.
Such a physical test could be made by actually loading 2. Background a truck with the load units to be tested, and driving that truck in a safe environment. The alternative, adopted by The European Road Worthiness Directive is the Euro- us and ACMI S.p.A., is to use a special testing machinery, pean norm aimed at assuring the security during rail or produced by ESTL Company [5] and shown in Figure 1, road freight. In particular, the European Standard EUMOS which consists in a movable platform, the sleight, on 40509 [3] is devoted to quantify the rigidity of the pallet which is carried the pallet, and an engine that can genwhen it is subject to a force (due to an acceleration) along erate a constant and controlled horizontal acceleration a direction. Its goal is to investigate the motion dynamic impulse over such a platform. The acceleration can be of a truck loaded with one or more EUR pallets. The sta- set between 0/ 2 and 10/ 2 in steps of 0.5/ 2. The bility of the load unit, in fact, depends on the rigidity of duration of the acceleration is at least 500 . Usually the the load, i.e., how much the load is sensitive to perma- pallets are tested at diferent acceleration levels: tests nent deformations or excessive shifting [ 4], and on the start at a low acceleration level of 0.2 or 0.3 (about holding strength of the external envelope, which could 1.962/ and 2.943/ 2, respectively). Then, if the rebe deformed or ripped during motion. The Acceleration sult is successful (w.r.t. EUMOS 40509), the constant acBench Test of EUMOS 40509 defines a physical test setup celeration impulse is increased by a value of 0.1 , heading and some test acceptance criteria. Such a test consists in for the legal requirement of 0.5 for load safety. These subjecting the load unit to an acceleration impulse that parameters permit to simulate road transport events such immediately stops and gives rise to a constant deceler- as diverting maneuvers and/or emergency stops. Such a ation, until the unit stops. Typical tests are performed machinery also records high-quality video of the motion, providing summarized information about elastic and permanent deformations. Three markers are attached to the load unit and two markers on the sleight, as in Figure 1, so that the ESTL Machine vision system can detect fluctuations of the pallet. Acceleration and displacement of the sleight is known at each time instant thanks to an X-Y accelerometer, and the acceleration profile data are recorded and plotted as well. Figure 2: An example result of the tracking phase of the vision
The ESTL machine method is surely advantageous system, searching for bundles in a two layer non-wrapped w.r.t. the road test in terms of both fuel saving and safety. configuration. Then, optical flow is used to extract centers of Also, recordings from high-speed cameras and data from gravity of pallet and packages. the accelerometer allow to precisely identify when EUMOS 40509 conditions are matched. However, the very high energy consumption of each ESTL machine test makes it not convenient to perform extensive experimentation. Therefore the necessity of providing a virtual— simulated—testing bench at the core of our work. vision, for which there are plenty of AI approaches. Most of the recent literature focuses on deep learning techniques, given the fact that novel neural architectures and learning algorithms outperform standard vision methods in mainstream (i.e. standard) settings, such as the recog3. Our Proposal nition of common objects [6, 7]. Unfortunately, except for some notable examples [8], deep learning methods Given the problem of pallet stability, the standard EU- usually require a huge amount of homogeneous data, MOS 40509, and the ESTL machine as testing bench, our that have to be carefully annotated in case of supervised work aims at developing an intelligent and automatic learning. In our case, the object to be recognized are not recommendation system that is able to suggest the safer, standard ones, and even if we could fine-tune some prerobust, and most reliable wrapping format to the ACMI trained networks, mainstream datasets on which those paper wrapping machine. To do so, however, it is cru- networks are trained are too general, and making eforts cial to gain a thorough knowledge about the dynamics to switch from their to our domain could result in a global of the acceleration test bench. This paper illustrates the lowering of the network performances. Moreover, our ifrst steps taken in such a direction, i.e., the collection of raw data are produced by physical experiment with the raw data from the ESTL machine and the usage of such acceleration testing machine, and each experiment has data to virtually reproduce the dynamics of any physical a high cost in terms of time and power consumption, setup. The proposed methodology is AI-based and com- failing the requirement of a vast amount of data to be bines computer vision and machine learning techniques available for network training and testing. Hence, our to model a realistic multi-body simulation. system was crafted for the specific task, with the aid of
The developing of such a simulation starts from the standard computer vision algorithms and techniques. acquisition by the ESTL machine of raw video record- The experimental settings regard the selection of few ings of the load unit subjected to horizontal accelerations. influential tests to be physically performed with the accelThen, a low-level vision system is devoted to extract the eration testing machine, their analysis with the computer centers of gravity of each visible bundle (the wooden vision system and the modeling of the relative virtual pallet and the secondary packages on it) in those videos, multi-body simulation. We take two diferent kind of exand their possible rotations. With the aid of a multi-body periments, with or without the external envelope. We physics engine, the base building blocks of the simulation choose bundles of six Coca-Cola Zero™ and maintained could be modeled. At the beginning of this phase, how- the same product for all the experiments. Diferent prodever, the simulation cannot be realistic, due to the lack ucts would have diferent shapes and dynamics, thus of knowledge about crucial physical parameters such as making it impossible to compare experiments among static and dynamic friction forces at work. Using mea- each other. The absence of wrapping, in this early phase, surements retrieved by the vision system, and centers of allows us to analyse more accurately the behavior of each gravity of each body of the simulation, those parameters package above the pallet. Then, tests with the same conare learned. ifgurations but wrapped with paper and/or plastic are also taken, to compare them with unwrapped configurations. Recordings of experiments last 8–10 seconds, at 3.1. The Vision System a rate of 20 frames per second, with 1920 × 1200 frames The problems of object detection and tracking of their size. position are well-known tasks in the field of computer The computer vision system we developed consists of a program which processes raw videos frame by frame. must be flexible enough to shape the parameters of such an envelope. Kraft paper, and in general, paper dynamics are still an open challenge for those types of engines. For these reasons, we choose a multi-physics engine that is also able to perform Finite Element Analysis (FEA) and model smooth collisions. Such an engine is called Project Chrono [23, 24], developed by University of Wisconsin Figure 3: Use of ArUco markers to detect the movement of (Madison) and University of Parma. layers. Each body can be a simple shape (e.g. a box, a sphere) and/or a user defined 3D model. Each body has a center of gravity, a mass, a moment of inertia and a collision The system was developed in Python 3.8.12, using the model. Masses of pallets and bundles are easily obtainPython versions of OpenCV [9] open-source library. For able from real measurements. Initial centers of gravity of the first type of tests, i.e. without wrapping, the goal is objects depend on their shape and their initial position in to retrieve the centers of gravity of each package and the the simulation. We choose to approximate bundles with pallet. The program thus identifies a Region of Interest boxes, so the center of gravity could be easily calculated. (ROI), which is automatically detected based on the accel- A flexible material developed using FEA consists of a set eration profile data of the physical test, from which the of nodes that composes a mesh, and elements that conposition of the load unit in the video is predicted. Then, nect the nodes among each others. On each node could normalized cross-correlation is used to perform a multi- be applied a resulting force, and a link to other elements ple matching of a template (an example of a package to be of the simulation. Then, a linear motor engine is initialrecognized) inside the ROI, and to prevent the explosion ized to model the sleight. Chrono has a facility to create of computational times, a Non-Maximum Suppression functions for vertical and/or horizontal motion, and in (NMS) algorithm [10] follows the template matching. We our case a constant acceleration could be modeled by used several state-of-the-art methods for multi-object imposing the ramp length and height (of the speed functracking [11, 12, 13, 14, 15, 16, 17] to maintain the recog- tion), the ending time of the acceleration and the starting nition of the template along the frames. An example is time of the deceleration. All such parameters could be shown in Figure 2 on the left. Finally, an optical flow de- easily obtained from the acceleration profile provided by tection methods [18] is employed to measure the actual the ESTL machine. displacements and rotations of packages. For each bundle The ESTL machine, the wooden pallet, and each of the and the wooden pallet, an approximation of the center bundles are composed of diferent materials. For each of gravity is computed, by taking a weighted mean of object/material a value of static friction and a value of all displacements centered in the center of the bounding kinetic friction must be set. From previously analysed box of the tracked object. Figure 2 on the right shows an videos, it is easy to see that the motion of the load unit is example of results, where colored dots are the computed delayed compared to the motion of the sleight, due to the centers of gravity of bundles and the wooden pallet. friction force between those two rigid body, depending
For the second type of tests, i.e. with wrapping, the on their material. So, we used the sleight displacement goal is to retrieve the movement of each layer of packages as a reference and computer the diference between that separately. During the setup of the experiments, we put and the displacement of the load unit. The same was ArUco markers over the paper wrapping, at the corners of done for the relative displacements of the pallet w.r.t. the each layer. We also performed tests with plastic wrapping, packages of product of the first layer, then the first w.r.t. to compare the two materials. In such a case, ArUco the second layer and so on. Such diferences are strongly markers were posed inside the wrapping, on each bundle. related to friction coeficients (static and dynamic) of the Using ArUco detection [19], approximated positions of pallet over the sleight, of each bundle over the pallet, layer were extracted, as in Figure 3. and of bundles with each other. The displacements are measured using the extracted data (centers of gravity) 3.2. The Simulation () = ( () , () ) of each relevant object visible in the video recordings of ESTL machine at time . We choose to use the centers of gravity retrieved by Boosting tracker [13], that achieves the better accuracy in the previous phase. The predicted centers of gravity are the computed positions () = ( ( ) , ( ) , () ) of all the objects in the simulation at time . To made such predicted coordinates to match with video data, we run the simulation with the
(rigid)-body dynamics simulations. There are both free and commercial softwares capable of reproducing rigid bodies motion, such as AutoDesk AutoCAD [20], MathWorks Simscape Multibody [21], NVIDIA PhysX [22], and so on. Nevertheless, our system needs also a nonrigid part to model the external envelope and the engine those in front of the camera, could be compared to simu- as in previous discussion. displacements. Unfortunately, only the visible bundles, input of a gradient descent algorithm with momentum, friction for the pallet and each package. However, in- that reproduces the behavior of a three layer unit with a with 2 cost function to find static friction and kinetic () ,
() ), as put data frames are few (∼ 50 positions for each bundle) and also very noisy due to previous calculations (matching, tracking, optical flow detection). A statistical method to denoise data is Exponential Moving Average (EMA), that defines a novel sequence from raw data depending on the value of a parameter ∈ (0, 1) . Larger (close to 1) values of produce smoother sequences. The machine learning method is a variation of the gradient descent algorithm, which uses EMA on gradient sequence. In deep learning field, such a method is known as gradient descent with momentum [25].
The simulated wrapping envelope is a Chrono’s FEA object modeled as a closed mesh around the whole load unit. Given the standard dimensions of an EUR pallet, i.e. 800 × 1200 mm, nodes are generated with a ratio of 4/6 around the perimeter of the box shape of the pallet. In fact, the lower round of nodes is anchored to the wooden pallet, while the upper ones are shrinked (in negative or positive, depending on the dimensions of the layers of product) around each layer. Project Chrono provides several materials to model elements between FEA nodes, and we choose the ElastoPlastic one to deal with the behavior of a well-known material during the fitting phase. The actual envelope produced by ACMI’s wrapping machine has a diferent number of layers at diferent heights, i.e. the material is stratified to be more resistant at critical points. Such a stratification is modeled by increasing the pulling tension of nodes. A standard wrapping format divides into three phases: the lower layer phase, where many stratification occur to attach the products to the wooden pallet; the central layers phase, which usually requires a single passage; and the upper layer phase, when some overlaps are made to close the envelope. Therefore, we choose to highly increase the pulling tension of lower nodes and slightly increase the pulling tension of upper nodes, w.r.t. the central ones. At the current stage of development, it is not possible to know the real value of the tensions working on the paper envelope, to compare them with those applied to each point of the mesh and computed by Chrono. In particular, the forces we want to retrieve are those acting at each corner of the wrapping.
ters of gravity in tests without wrapping. Similarly, we look at the displacements between layers in wrapping experiments, where we put ArUco markers. We plan to use the positions of selected key points of FEA mesh as
The virtual simulation was developed using the Python versions of Project Chrono, PyChrono [26], with IRRLicht [27] engine to render the simulation. Both programs run on an Anaconda environment on a laptop with a 6-cores 10ℎ gen. i7 CPU, base speed 1.61 GHz up to 3.60 GHz, and 16 GB of RAM. Figure 4 shows a snapshot of a simulation columnar layout, without wrapping.
This paper proposes the development of a virtual multibody simulation to analyse the stability of pallets wrapped with Kraft paper instead of LLDPE. The design of the simulation strongly relies on the EUMOS 40509 requirements of safety for rail and road transport of packages. The main idea is to simulate the acceleration test bench, to minimize high energy and time consumption experiments that have to be done with the ESTL machine. A complete AI pipeline is proposed to study the dynamics of pallet and packages without wrapping. The analysis starts from the actual physical experiment and combines standard computer vision and machine learning techniques to reproduce the behavior of elements.
Preliminary results are encouraging, and thanks to the convergence of a gradient descent method, the simulation is able to adhere to the reality of tests, at least for visible bundles. Even if further investigations are needed, future work will be devoted to the identification of critical points of tension which could impact the paper wrapping and the use of such insight to tell wrapping machine software engineers how many wrapping layers are needed, at which heights, with what tension, and so on.
Recognition, IEEE, 2009, pp. 983–990. [15] Z. Kalal, K. Mikolajczyk, J. Matas, Tracking[1] European Commission: A European Strategy for learning-detection, IEEE transactions on patPlastics in a Circular Economy. 2018a., Avail- tern analysis and machine intelligence 34 (2011) able online: https://ec.europa.eu/environment/ 1409–1422. circular-economy/pdf/plastics-strategy-annex.pdf, [16] Z. Kalal, K. Mikolajczyk, J. Matas, Forward2018. Accessed on March 28, 2023. backward error: Automatic detection of tracking [2] C. Matthews, F. Moran, A. K. Jaiswal, A review on failures, in: 2010 20th international conference on european union’s strategy for plastics in a circular pattern recognition, IEEE, 2010, pp. 2756–2759. economy and its impact on food safety, Journal of [17] D. S. Bolme, J. R. Beveridge, B. A. Draper, Y. M. Lui, Cleaner Production 283 (2021) 125263. Visual object tracking using adaptive correlation [3] Eumos, the european safe logistics association. qual- iflters, in: 2010 IEEE computer society conference ity standards., Available online: https://eumos.eu/ on computer vision and pattern recognition, IEEE, quality-standards/, 2020. 2010, pp. 2544–2550. [4] S. Tkaczyk, M. Drozd, Ł. Kędzierski, K. Santarek, [18] G. Farnebäck, Two-frame motion estimation based Study of the stability of palletized cargo by dynamic on polynomial expansion, in: Scandinavian confertest method performed on laboratory test bench, ence on Image analysis, Springer, 2003, pp. 363–370.
Sensors 21 (2021) 5129. [19] S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid[5] Engineering & solutions for transport & logistic Cuevas, M. J. Marín-Jiménez, Automatic generanv (estl nv), wafelstraat 46, 8540 deerlijk, belgium., tion and detection of highly reliable fiducial markAvailable online: www.estl.be, accessed on March ers under occlusion, Pattern Recognition 47 (2014) 28, 2023. 2280–2292. [6] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You [20] Autocad, Available online: https://www.autodesk. only look once: Unified, real-time object detection, it/solutions/simulation/overview, accessed on in: Proceedings of the IEEE conference on computer March 28, 2023.
vision and pattern recognition, 2016, pp. 779–788. [21] Simscape, Available online: https://www. [7] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Dar- mathworks.com/products/simscape-multibody. rell, S. Xie, A convnet for the 2020s, in: Proceed- html, accessed on March 28, 2023. ings of the IEEE/CVF Conference on Computer [22] Physx, Available online: https://github.com/ Vision and Pattern Recognition (CVPR), 2022, pp. NVIDIAGameWorks/PhysX, accessed on March 28, 11976–11986. 2023. [8] A. Nichol, J. Achiam, J. Schulman, On first- [23] M. Anitescu, A. Tasora, An iterative approach for order meta-learning algorithms, arXiv preprint cone complementarity problems for nonsmooth dyarXiv:1803.02999 (2018). namics, Computational Optimization and Applica[9] Opencv, Available online: https://opencv.org/, ac- tions 47 (2010) 207–235.
cessed on March 28, 2023. [24] A. Tasora, R. Serban, H. Mazhar, A. Pazouki, [10] A. Neubeck, L. Van Gool, Eficient non-maximum D. Melanz, J. Fleischmann, M. Taylor, H. Sugiyama, suppression, in: 18th International Conference D. Negrut, Chrono: An open source multi-physics on Pattern Recognition (ICPR’06), volume 3, IEEE, dynamics engine, in: International Conference on 2006, pp. 850–855. High Performance Computing in Science and Engi[11] A. Lukezic, T. Vojir, L. ˇCehovin Zajc, J. Matas, neering, Springer, 2015, pp. 19–49.
M. Kristan, Discriminative correlation filter with [25] I. Sutskever, J. Martens, G. Dahl, G. Hinton, On the channel and spatial reliability, in: Proceedings of importance of initialization and momentum in deep the IEEE conference on computer vision and pat- learning, in: International conference on machine tern recognition, 2017, pp. 6309–6318. learning, PMLR, 2013, pp. 1139–1147. [12] J. F. Henriques, R. Caseiro, P. Martins, J. Batista, [26] Pychrono, Available online: https://www.
High-speed tracking with kernelized correlation projectchrono.org/pychrono/, accessed on March iflters, IEEE transactions on pattern analysis and 28, 2023.
machine intelligence 37 (2014) 583–596. [27] Irrlicht, Available online: https://irrlicht. [13] H. Grabner, M. Grabner, H. Bischof, Real-time track- sourceforge.io/, accessed on March 28, 2023. ing via on-line boosting., in: Bmvc, volume 1, Citeseer, 2006, p. 6. [14] B. Babenko, M.-H. Yang, S. Belongie, Visual tracking with online multiple instance learning, in: 2009 IEEE Conference on computer vision and Pattern