Pedestrian injuries and fatalities are one of the most signi cant problems related to travel and road safety. Pedestrians are vulnerable users of roads and due to the very di erent velocities and mass when compared to vehicles like cars and trucks, and very often they undergo serious injuries in case of collisions. Older pedestrians are even more vulnerable to injuries and fatalities due to (i) their reduced mobility and re exes and (ii) their increased fragility when compared to young individuals. Crosswalks are the point where pedestrians face lower level of safety because they have to cross the street and must be aware of the incoming tra c. Such kind of awareness becomes di cult in case of old pedestrians because of their reduced physical and perceptive capabilities. Besides other factors, lower speed of an old pedestrian is an important factor that limits the mobility of old pedestrians and it also increases the risk of fatalities while crossing the road. In this paper, we developed vision based intelligent system that can detect low speeds and directions of pedestrians and can help him/her by (a) increasing the time associated to a green light for pedestrians, (b) using audible signals to help the pedestrians understanding that there are cars approaching the crossing.
Pedestrian injuries and fatalities are one of the most signi cant problem related
to travel and road safety. Pedestrians are the vulnerable users of road and due
to the very di erence in speed and mass when compared to vehicles like cars and
trucks, are very often they undergo serious accidents. Studies have shown that
more than one fth of the pedestrians killed while crossing the roads [
Motion segmentation is the most important pre-processing step for detecting the moving objects from the video. Traditionally in video surveillance with a xed camera, researchers tend to nd some sort of motion in the video. There are two part of such of videos, background and foreground part. The object in motion is the foreground part of the video and the rest static part is the background. Motion detection is used to extract foreground part from the video. Such kind of extraction is useful for detecting, tracking and understanding the behavior of the object. A survey on motion detection techniques can be found in [18]. There are two types of motion: 1) large-scale body movements like movement of head, legs and arms [19], and 2) small scale body movements like hand gestures and facial expressions [20], [21].
A. Foreground Segmentation: A popular and traditional foreground object segmentation method is a background subtraction. It calculates the di erence between current image and background image and detects the foreground by setting up the threshold value. Pixels in the current frame that deviate signi cantly from the background are considered to be moving objects. These foreground pixels are further processed for object localization and tracking. This technique is prone to errors when there is a change in illumination in the video.
B. Approximate Median: median lter is another technique for motion segmentation. It bu ers N number of frames and median of these frames are calculated which will be the background reference frame. This method is e ective in some cases but many frames have to be stored for calculating median frame which makes it not suitable in most of cases. Also this method will end up with errors if there is change in illumination.
C. Gaussian Mixture Model : The GMM is one of the most commonly used methods for background subtraction in most of visual surveillance applications. A mixture of Gaussians is maintained for each pixel in the image. with time to time, new pixel values updates the mixture of Gaussians using an online Kmeans approach. This updating of mixture of Gaussians is used to account for illumination changes, sensor movements and noise.
D. Temporal Di erencing : Another common approach for motion segmentation is the temporal di erencing. In temporal di erencing, video frames are separated by a constant time interval and compared to nd the regions that are changed. A small time interval between the frames can increase the robustness to illumination changes. Temporal di erencing approach is computationally inexpensive but in some cases it fails to extract the shape of the object and cause small holes.
E. Optical Flow : optical ow estimates the motion by matching points on
objects over multiple frame using vectors. Optical ow technique gives more
accurate estimation of motion if the frame rate is high. Horn and Schunck [21],
Lucas and Kanade [
Tracking is de ned as the problem of estimating the trajectories of objects in image plane. Video surveillance has motivated many researchers to explore tracking techniques. Tracking moving objects in a video sequence is a di cult job. Occlusion makes tracking a di cult problem for the researchers. In video surveillance, normally some features of the moving objects are extracted and tracking that object using those features. Selecting good features that can be used for tracking is very important, since the object appearance, color and orientation may change from frame to frame. So we need to extract those features that can be tracked for a long period of time. 4
Pedestrian injuries and fatalities are most signi cant problems related to road safety. Crosswalks are the point where pedestrians face lower level of safety and most of time they end up with serious accidents. Therefore, as a solution an automatic monitoring system based on computer vision technology is needed that can automatically detect the behaviors of pedestrians on crosswalks and alarm the system in order to prevent pedestrian vehicle collisions. Figure 1 shows the methodology, video is streamed from the camera to the system and system processes the video frame by frame. From each frame, foreground is segmented which represents the objects of interest (pedestrians). Blob analysis is performed to nd out the independent blobs of a particular size. Corner points are extracted from each bounding box. Later on, these points are tracked through number of frames using Luckas Kanade point tracker. Instantaneous velocity of each point related to bounding box is calculated and average speed of the object is determined by equation discussed in the following section. 4.1
Identifying moving objects in video sequence is a fundamental and critical task in video surveillance and gesture recognition in human-machine interface. Foreground segmentation is an important pre-processing step for detecting moving objects from the video. Traditionally, background subtraction method is used for extracting moving objects from the video frame where pixels in the currents frame that deviate signi cantly from the background are considered as part of moving objects. Such kinds of methods are usually prone to errors due to unpredicted and changing behavior of the pixels. In addition, this method can not accurately detect fast moving or slow moving as well as multiple objects. Also these methods are a ected by change in illumination in the video frame. Some time change in illumination in static background will be detected as part of moving object. Such errors and noise must be removed from the foreground objects before applying blob analysis and tracking.In order to extract valid and accurate foreground objects, we employed both Gaussian mixture model and Lucas Kanade optical ow as in [24]. There are ve popular optical ow methods: gradient based algorithms, block based algorithm, energy based, phase-based algorithm and neuro-dynamic algorithms [25].As optical ow can not get rid of change in illumination so we use Gaussian Mixture Model in combination with optical ow to extract an accurate foreground objects. GMM is more robust to light changes and slight movements for small image sequence. In order to obtain a foreground object without noise, we make use of both LK optical ow and GMM method as show in Figure 2.As shown in Figure 2, Lucas Kanade optical ow and GMM are applied in parallel to the input image.LK optical ow is applied to two adjacent images of video i.e. f (x, y, t1) and f (x, y, t) and output of LK optical ow will be in the form of magnitudes. LK optical ow tends to nd the corresponding points on f (x, y, t1) on next frame f (x, y, t).we discuss LK optical ow in more detail in the following section. After calculating LK optical ow between two frames, we use threshold Thof to segment motion from static background. Thof value can be calculated experimentally. The range of Thof is [0.005 0.002] which is calculated experimentally. The range of Thof for di erent videos is di erent. For slow moving objects the value of Thof will be low and for fast moving objects the range of Thof will be high. It is matter of fact that fast moving objects generates optical ow vectors with high magnitudes and slow moving objects generate ow vectors of lower magnitudes. So for the videos, where objects moving with di erent speeds, mean value of all ow vectors should be taken as Thof . In extracting moving part from the image, the pixel with large magnitude than Thof will be classi ed as foreground while the pixels whose magnitudes are less than Thof will be the part of background. In the same way, we get foreground objects by applying GMM, but the GMM ends up with errors as shown in Figure 2. So in order to extract accurate foreground we apply logical product of foreground mask generated by LK optical ow and GMM. Later on, we apply Morphological processes like morphological opening and closing on the binary image generated by logical product of LK optical ow and GMM. The morphological open operation is erosion followed by dilation eliminates smooth contours and protrusions while morphological close is dilation followed by erosion smooth the section of contours, eliminates small holes and lls gaps in contours. These operations are dual to each other. Later on, ood ll algorithm is applied to ll small holes. The output image f out (x, y, t) from Morphological processing block contains accurate foreground objects while will applied to Blob analysis block for detecting moving objects. 4.2
Blobs are the connected regions in the binary image. The purpose of blob analysis is to detect those points or regions in binary image that are di erent from other part of image in terms of brightness or area etc. Following are the steps for nding connected components in the binary image. 1. Search for unlabeled pixel 2. Label all the pixels in connected region containing p by ood ll algorithm 3. Repeat step 1 and step 2 until all pixels are labeled.
In the next step, we measure the area of each connected component. Area of connected component is the number of pixels in the region. There may be di erent moving objects in the video frame with di erent area sizes. In Transportation surveillance system, video frame contains multiple objects like pedestrians, vehicles of di erent sizes. In this paper, we are interested only in pedestrians whom we want to detect and track over multiple frames. The area of pedestrians is normally contains less number of pixels than vehicles and trucks in other words the size of pedestrian is less than vehicle and trucks. On the basis of this assumption, we set the upper and lower bound of blobs area. The upper and lower bound of blobs area can be found experimentally. The size of blob also depends on the resolution of frame. In this paper, we use videos of resolution 576 x 768 pixels and lower and upper bound of blobs area is [1000 2500] pixels. The connected component (object) will be classi ed as pedestrian if its area lies within the upper and lower bound otherwise it will be discarded. 4.3
In order to track objects detected in blob analysis step, we use KLT feature
tracker [26] where motion is detected by using pyramidal Lucas- Kanade
optical ow method, using Shi and Tomasi feature detection algorithm [27]. They
pyramidal implementation of Lucas- Kanade gives more robustness against huge
movement with di erent speeds. There are two categories of optical ow
algorithms 1) sparse optical ow 2) dense optical ow. Dense optical ow methods
like Horn- Schunck [21] estimates the displacement of all pixels of image while
sparse optical ow algorithms, such as Lucas-Kanade approach estimates the
displacement for selected number of pixels. Sparse optical ow gives more
robustness towards noise. In sparse optical ow methods, the selection of pixels
should be done automatically and done wisely. There are various methods in
literature for selecting features to be traced correctly. Robust features can be
found and tracked over multiple frames using scale invariant feature transform
(SIFT) algorithm [28]. Speeded-Up Robust Features (SURF) [29] also presents
a method for extracting interest points and describe these points for fast
comparison. Canny edge detector [30] detects edges in image. Then image is divided
into small blocks and we search for the closest edge pixel from the center of the
block. If the pixel is found, it will be regarded as feature point for that block.
Harris Corner detector [31] detects the points where two edges are detected.
One of the disadvantage of SIFT and SURF algorithms is that they are
computationally expensive. The comparison of all these feature detection techniques
can be found in [32]. In this paper, we use Shi and Tomasi algorithm for
extracting corner points. Figure 3 (a) shows the pedestrian detected during blob
analysis step. In the next, we detect corner points of each bounding box using
Shi and Tomasi corner detector as shown in Figure 3 (b). But through our
experiments we realize that points detected in the rst frame may not be tracked
over multiple frames. This is due to dramatically change in the appearance of
the objects and change in intensity values of pixels. Such kind of change always
results in tracking failure. Here, it should be noted that our aim is to nd the
instantaneous velocity of valid pixels. A pixel will be valid pixel if its forward and
backward trajectory does not di er signi cantly. [33] Purposes a method that
can automatically detect the tracking failure by forward and backward tracking
of pixels. As shown in Figure 4 (a) , point p on frame t is to be tracked on frame
t + 1. Let p is a point detected on frame t + 1. This is a forward trajectory of
point p. in order to check the validity of the pixel, the point location p will track
back the point on frame t. this is backward trajectory. Let p is the point location
tracked during backward trajectory. The di erence between the two trajectories
is d. In ideal case, the value of d is zero. So here, we de ne a threshold value
for d. experimentally the value of d is in range of [
Figure 4 (b) shows the points of one of pedestrian in frame that were detected by Shi Tomasi corner detector. 0 represents the points (pixels) whose d is greater than threshold range when tracked over next frame and hence regarded as invalid pixels while 1 represents the valid pixels. We consider only valid pixels for nding the speed, direction and understanding the behavior of pedestrian. 4.4
In order to nd the speed of pedestrian, we must use the valid points. To nd the speed of pedestrian, instantaneous speed can be found by using successive image frames of the video. This instantaneous speed for each pixel can be found by Equation 1.
v = d= t (1) Where v is the instantaneous velocity of a valid point. d is the change in displacement of valid point over successive frames. t is time interval between two successive video frames and is equal to the frame capture rate of the camera. In our experiments, t is 33.33 milliseconds. To nd the accurate speed of pedestrian, only one valid point is not enough. We need to nd out valid points for the same pedestrian, calculate the instantaneous velocities of those points. Then by averaging instantaneous velocities of all valid points. Lets assume that n valid points are selected from the pedestrian and let vi represents the instantaneous velocity of point i. where i = 1::::n. By using those instantaneous velocity vectors, we can nd the instantaneous velocity of pedestrian by Equation 2: V iv(t) = n X vi(t) i=1 (2)
Where V iv is the instantaneous velocity of a pedestrian at time t, vi(t) is the instantaneous velocity vector of ith point and n is the number of valid points that tracked. vi(t) in Equation 2, is calculated in pixels per second because we measure the displacement d between two frame in pixels which has no correspondence with real displacement. As we know, that objects closer to the camera will have large pixel displacement than the objects far from the camera although both objects are moving with same speed in real world. Thus, it is very di cult to nd the accurate speed of pedestrians using only optical ow. In order to overcome this limitation, we calculate the displacement of each pedestrian in the world coordinates. In our case, we calculate the distance between entry and exit point of pedestrians. Later on, we calculate the displacement in pixels and derive a linear scale factor to relate displacement in images to the motion in the world. 4.5
At intersections and pedestrian crossings, pedestrians frequently change their speed and directions. It is even more dangerous and prone to pedestrian vehicle collisions if the pedestrians change their walking directions in the middle of the road. Therefore to avoid such collisions, driver must know when the pedestrian is going to change his/her walking direction to dangerous area. Therefore, estimating the orientation of pedestrian on pedestrian crossing becomes very important. In this paper, we estimate the walking direction of pedestrian by making use of optical ow vectors of the valid points. Let vel (x, y, u, v) is the velocity vector of a pixel. Where (x, y) is the coordinate of a pixel and u and v are the horizontal and vertical movements of the pixel. As the optical ow eld of the foreground image contains all the velocity vectors vel (x, y, u, v), therefore it is easy to get the magnitude r (x, y) and angle. Let (x, y) be a direction of optical ow vector at pixel (x, y) in frame t and is given by Equation 3.
(x; y) = tan 1(u=v) (3) As we are interested in nding the orientation of the pedestrian, therefore we consider angle information. 5
We carried out our experiments on a PC of 2.6 GHz (Core i5) with 4.0 GB memory and data set from UCF. As shown in Figure 5 (a), pedestrians cross the road in the opposite directions while the car is moving towards the pedestrians. Studying such kind of scenario becomes very important for understanding the pedestrian/vehicle interactions. And in order to avoid collisions we develop a system that automatically nds the speeds and directions of pedestrians and vehicles. Figure 5 (a) shows a sample frame taken from a video sequence. Figure 5 (b) and (c) shows the objection detection and tracking results after applying our algorithm mentioned in section 4. The most important result are shown in Figure 6. Figure 6(a) shows the di erent speeds of pedestrians. The color bar shows amount of speed a pedestrian is travelling with. The dark regions shows that pedestrians are moving with high speed while cyan and yellow regions represents relatively low speed. Figure 6(b) shows the directions of pedestrians. As shown in Figure 6 (b) there are two dominant ows, one towards the East and other towards west. Red color shows that pedestrians are moving towards west while green color shows pedestrians moving towards east. The vehicle in cyan color is moving towards south 6
In this paper we proposed a framework to nd the speed and direction of people moving in the video. Foreground objects (people) are extracted by applying Gaussian mixture model and optical ow. Blob analysis is performed to detect pedestrians. Lucas Kanade Tracker is used to track each point across multiple frames. Later on, average speed (pixels/second) of each pedestrian is computed. It is observed that proposed framework worked very well in low density scenarios. We are developing methods and techniques that can automatically map image coordinates to world coordinates and nd the actual speed(meter/seconds) of pedestrians and is a part of our future works. 18. T. B. Moeslund and E. Granum, \A survey of computer vision-based human motion capture," Computer Vision and Image Understanding, vol. 81, no. 3, pp. 231{268, 2001. 19. V. I. Pavlovic, R. Sharma, and T. S. Huang, \Visual interpretation of hand gestures for human-computer interaction: A review," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 19, no. 7, pp. 677{695, 1997. 20. B. Fasel and J. Luettin, \Automatic facial expression analysis: a survey," Pattern
Recognition, vol. 36, no. 1, pp. 259{275, 2003. 21. B. K. Horn and B. G. Schunck, \Determining optical ow," Arti cial intelligence, vol. 17, no. 1, pp. 185{203, 1981. 22. R. Szeliski and J. Coughlan, \Spline-based image registration," International Journal of Computer Vision, vol. 22, no. 3, pp. 199{218, 1997. 23. J. L. Barron, D. J. Fleet, and S. S. Beauchemin, \Performance of optical ow techniques," International journal of computer vision, vol. 12, no. 1, pp. 43{77, 1994. 24. W. Li, X. Wu, K. Matsumoto, and H.-A. Zhao, \Crowd foreground detection and density estimation based on moment," in Wavelet Analysis and Pattern Recognition (ICWAPR), 2010 International Conference on. IEEE, 2010, pp. 130{135. 25. M. Cristani, R. Raghavendra, A. Del Bue, and V. Murino, \Human behavior analysis in video surveillance: a social signal processing perspective," Neurocomputing, vol. 100, pp. 86{97, 2013. 26. C. Tomasi and T. Kanade, Detection and tracking of point features. School of
Computer Science, Carnegie Mellon Univ., 1991. 27. J. Shi and C. Tomasi, \Good features to track," in Computer Vision and Pattern Recognition, 1994. Proceedings CVPR'94., 1994 IEEE Computer Society Conference on. IEEE, 1994, pp. 593{600. 28. D. G. Lowe, \Object recognition from local scale-invariant features," in Computer vision, 1999. The proceedings of the seventh IEEE international conference on, vol. 2. Ieee, 1999, pp. 1150{1157. 29. H. Bay, T. Tuytelaars, and L. Van Gool, \Surf: Speeded up robust features," in
Computer Vision{ECCV 2006. Springer, 2006, pp. 404{417. 30. J. Canny, \A computational approach to edge detection," Pattern Analysis and
Machine Intelligence, IEEE Transactions on, no. 6, pp. 679{698, 1986. 31. C. Harris and M. Stephens, \A combined corner and edge detector." in Alvey vision conference, vol. 15. Manchester, UK, 1988, p. 50. 32. N. Nourani-Vatani, P. Borges, and J. Roberts, \A study of feature extraction algorithms for optical ow tracking," in Australasian Conference on Robotics and Automation, 2012. 33. Z. Kalal, K. Mikolajczyk, and J. Matas, \Forward-backward error: Automatic detection of tracking failures," in Pattern Recognition (ICPR), 2010 20th International Conference on. IEEE, 2010, pp. 2756{2759.