The number of adult salmon that escape predators and fishing to successfully return to their natal streams to spawn, known as escapement counts, are vital to ensure sustainability of the salmon fisheries. Sufficient numbers of salmon must reach the spawning grounds each year to ensure the long term viability of the various species. Once a critical threshold is reached, the fishery is opened to commercial exploitation. To ensure the viability of salmon in Alaska, the Alaska Department of Fish and Game conducts escapement counts using a variety of techniques. One well established technique is to use remote video stations to record salmon passing upstream and periodically use human operators to view the video sequences to determine the escapement counts. This paper presents the initial stages of the development of a technique to automate the escapement counts and thereby relieve the demand for human operators. The technique uses a global segmentation and image morphology operations followed by a Kalman filter to identify and track the candidate salmon as they pass through the region of interest on the video recording. Candidates that complete the journey across the region of interest are counted. Finally, some of the future challenges posed by real time processing, water surface turbulence and lighting changes are discussed.
Salmon escapement counts are vital to ensure sustainability of the fisheries in Alaska. Sufficient numbers of salmon must reach the spawning grounds each year to ensure the long term viability of the various species. In remote areas the salmon must escape their natural predators such as salmon sharks, seals and orca in open water, and then bears and eagles at the entrances to streams and rivers. In more populated areas the salmon are also subject to recreational and subsistence fishing, usually at or near stream and river mouths. Catches are limited by daily or seasonal limits per person/household. Only when sufficient numbers have passed through the predation zones and will reach the spawning grounds are the fisheries opened to commercial exploitation. Commercial operators apply pressure to open the fisheries as early as possible to maximise the fishing season, consequently escapement counts have to be both accurate and reliable.
A number of techniques are used to monitor escapement. Traditionally, artificial weirs and manual counts by a
resident fishery officer have been the mainstay for salmon escapement
This paper describes the progress made to develop an automatic video counting system that operates in real time or rapidly analyses pre-recorded videos. The developed algorithm has the capability detect, track and count the salmon passing through a defined region of interest in the video frame. The first step in the processing is to segment the fish objects in the current video frame using a global threshold technique. Morphological image processing (dilation, erosion and median filtering) is then used to eliminate speckle noise, eliminate small regions which are likely not related to fish objects, and connect parts of the same fish object separated by water surface turbulence. The third step of the system is to track the fish objects using a Kalman filter to overcome occlusions and gaps in identification. The Kalman filter predicts the track up to a certain number of frames, beyond which the object is eliminated as a false positive. Finally, fish objects with a full track across the region of interest are counted.
In the 1990s the ADF&G at Homer began developing stand-alone remote video salmon escapement monitoring
stations to supplement aerial surveys of remote, clear water streams
The video is recorded at 2-5 frames per second to a digital video recorder (DVR) inside a bear-proof strongbox. The DVR receives a NTSC composite (analogue) video signal from the camera but records digital video sequences onto a hot-swappable hard disk drive. Electrical power is provided by a bank of 12V lead-acid batteries connected in parallel. Wind and solar power generators are used to maintain the battery charge level. An automatic alarm system and satellite communications relay is used to alert ADF&G if equipment failures occur.
The field of view of the overhead cameras is optimised to match the panel area as far as possible. Figure 2 shows the fields of view from the systems at Chenik Creek and Mikfik Creek, both on the western side of Cook Inlet. Video sites are chosen based on a range of factors such as accessibility, stream morphology, minimisation of surface turbulence and reflections, availability of solar, wind or hydropower, and many others. As noted previously, the optimal direction for the camera view is to the north and this requirement can lead to a compromise between the field of view and the resolution of the view. The orientation of the format of the camera is physically limited by the weatherproof housing design, so in general the substrate panel is not aligned with the highest resolution axis of the camera. At some sites other requirements, for example a desire to monitor human traffic near the site, forces a wider than optimum field of view.
The DVR has four input channels and some sites also have a low-light camera mounted on the quadrapod or an
underwater camera with a view across the stream bed. The underwater camera is used primarily to positively
identify fish enumerated with the above-water camera and to apportion the total escapement among co-occurring
species with overlapping run timing. The low light camera does not have a polarizing filter and is used primarily
during twilight periods to minimize missing counts caused by the 4-5 hours of darkness during June to August.
Previous research that compared video counts to those derived by a weir at Delight Creek in South-central
Alaska indicated that less than 2% of the salmon escapement passed the video counter during the hours of
darkness
The hard disks have sufficient capacity to record video for 2-3 weeks. Regular visits to the remote sites are carried out to check the equipment and swap hard drives, with visits increasing in frequency as escapement counts approach the threshold necessary to open the commercial fishery. Maintenance visits are commonly pooled with aerial survey flights to efficiently use the float plane travel, as this is a significant component of the budget for escapement counts.
The hard disks are returned to the ADFG office for analysis. The disks are mounted in a multi-screen video station for replay. If multiple cameras are available then all captured videos can be shown on the screens with synchronous replay. An operator manually records the numbers of salmon passing over the substrate panel using a hand counter. The operator can fast forward through periods of inactivity or step slowly through the individual frames of groups of fish passing over the panel. An operator can review a full day of video in as little as 10-20 minutes during periods of low fish count. However, 1-3 hours are often needed to review peak passage days which may involve 2000-4000 fish. Salmon runs span as many as 40 days and may require a total of 15 hours of analysis by an operator, dependent on the frequency profile and total count. A typical profile of daily salmon escapement counts is shown in figure 3.
The quality of the video imagery recorded by the remote stations is, in most conditions, satisfactory for an operator to identify and count the salmon. The resolution of the video is limited to 720 by 480 pixels by the NTSC broadcast signal, which was adopted in the design of the remote stations for compatibility with the low cost surveillance cameras and security industry DVRs that are available off-the-shelf. This technology has many advantages, particularly that it has a proven track record, is very robust and there is a vast knowledge base of systems and technological solutions. In addition, the ability to readily inject a time code into the video image or the audio track allows a permanent record of the date and time, and allows accurate synchronisation of multiple cameras.
Nevertheless, the NTSC resolution is a limitation that is more evident for some stations, such as the Mikfik Creek site shown in figure 2. The field of view could be narrowed using a longer focal length lens, but in some cases the width of the stream is too wide to permit sufficiently high resolution images.
A further limitation on the quality of the video imagery is the video compression. The DVR units used in the remote video stations use a proprietary Motion JPEG compression to maximise the recording time by the units. The compression, combined with the relatively low number of frames per second, allows the units to be deployed for 2-3 weeks of 24 hour recording. This is particularly important during off-peak periods when the hard drives are not swapped out as frequently.
As is typically the case for surveillance DVRs, the operator cannot disable the video compression but does have some control over the level of compression. However the compression levels are high even within the allowable range. For example the lowest to the highest compression ratios per frame are 30:1 to 65:1. Visually there is little discernible difference between the extremes of quality recorded by the DVR. An apparent characteristic of the proprietary Motion JPEG algorithm is that adjacent odd and even horizontal lines in the image are adjusted to have similar RGB intensities, leading to an effective halving of the vertical resolution and further deterioration of the image quality through loss of resolution. Unfortunately testing with alternative compression algorithms, such as MPEG, produced even greater deterioration in image quality and also had a greater propensity for dropped frames when the video was reviewed at faster replay speeds.
As discussed above, operators can review an entire 24 hour video recording in a span of 10-180 minutes depending on the number of salmon to be counted. This level of efficiency is admirable for a human operator, however over a full season with many remote video stations the total time for review can be measured in hundreds of hours.
Computer-based image analysis has been used in marine and freshwater science for many tasks associated with
detection and identification of fish from still images and video sequences. For example,
More recently,
The behaviour of the salmon facilitates the RoI approach, rather than attempting the virtually impossible task of uniquely identifying each fish, because the salmon rarely deviate from a direct path across the substrate panels. The fish often pause at the threshold of the substrate panel because it is an unnatural feature, but then swim directly across the panel to the natural creek bed on the far side. The salmon are driven by instinct to swim upstream, so multiple factors militate against the fish repeatedly swimming or hesitating over the panel. Accordingly, a feasible approach is to track the fish objects across the RoI and count each detected object that completes the passage.
The video processing and tracking approach described in the previous section has been implemented using Matlab1 as a development and testing platform. Once a mature, generalised solution is developed the intent is to implement the video processing algorithms as a stand-alone application that can be deployed on any PC. Each step in the process is described in more detail below.
The first step in the processing is to define the RoI within the video frame. This approach is feasible because of the fixed nature of the field of view of the remote video stations. The Matlab function roipoly allows the operator to specify a polygonal RoI within an image from the video sequence and creates a binary image that is used as a mask. The mask is then applied to every subsequent video frame to eliminate areas outside the RoI from further processing. The RoI reduces the computational burden on the system, increases the effectiveness of the segmentation performance and avoids the false detection of fish objects in the cluttered image areas outside of the RoI.
An example of the operator selected polygon and the subsequent mask are shown in figures 4 and 5. The RoI should match the substrate panel, but should avoid the edges and shallow sections near the shore to provide uniform background intensity. If the salmon are hesitating, or entering and then re-entering the substrate panel area, the downstream edge of the RoI can be positioning further inside the substrate panel area to minimise tracking errors.
Segmentation is the next step in the video processing. This commences with converting the video frame from
RGB colour into an average greyscale intensity for each pixel using the ColorSpaceConverter function. The
medfilt2 median filter function is applied next to remove noise and smooth the image. Under the simple
assumption that the substrate panel will always be a relatively high intensity value and the fish objects will be a
relatively low intensity value, a simple average of the minimum and maximum intensities in the RoI is used to
globally segment the image and create a binary image of the candidate fish objects. This approach is based on
the principle of a bi-modal distribution of intensity used in the Otsu method
The Matlab BlobAnalysis function is then applied, with empirically determined minimum and maximum blob sizes, to analyse the RoI and identify the fish candidates. The blob analysis identifies the connected groups of pixels and computes characteristics such as the area, centroid and bounding box. The effectiveness of the blob analysis algorithm relies completely on the quality of the segmentation. A poor segmentation will result in no candidates being detected or many false positives due to lighting conditions or noise in the image. In turn this will lead to inaccurate results from the tracking and counting processes, described in the next section.
In computer vision moving objects are tracked for variety of applications such as activity recognition, traffic
monitoring and automotive safety
The multiple objects in the RoI are tracked with a Kalman filter
In this project a Kalman filter has been used for object tracking by predicting the position from the previous information and verifying the existence of the object at the predicted location. In Matlab, the estimates are computed using the vision.KalmanFilter system object. The association of candidate detection to the same object is based solely on motion. The motion of each track is estimated by the Kalman filter which determines the likelihood of each candidate detection being assigned to each track.
Track maintenance becomes an important aspect of counting the salmon. In any given frame, some candidate detections may be assigned to tracks, while other candidates and tracks may remain unassigned. The assigned tracks are updated using the corresponding detections. An unassigned candidate detection begins a new track. Candidates must be tracked for a certain number of consecutive frames to be assigned to a track. Tracks that are unassigned for a certain number of consecutive frames are deleted. False positives caused by noise or surface turbulence tend to result in short-lived tracks. The limit is set empirically and in most circumstances a value of five consecutive frames is effective.
Assigning object detections in the current frame to existing tracks is done by minimizing a cost function. The cost is defined as the negative log-likelihood of a detection corresponding to a track. The algorithm involves two steps:
Compute the cost of assigning every detection to each track using the distance method of the vision.KalmanFilter system object. The cost takes into account the Euclidean distance between the predicted centroid of the track and the centroid of the detection. It also includes the confidence of the prediction, which is maintained by the Kalman filter. The results are stored in an MxN matrix, where M is the number of tracks and N is the number of detections.
Solve the assignment problem represented by the cost matrix using the assignDetectionsToTracks function. The function takes into account the cost matrix and the cost of not assigning any detections to a track.
The value for the cost of not assigning detection to a track depends on the range of values returned by the distance method of the vision.KalmanFilter. This value must be tuned experimentally. Setting it too low increases the likelihood of creating a new track, and may result in track fragmentation. Setting it too high may result in a single track corresponding to a series of separate moving candidates.
Figure 8 shows an example of fish candidates mapped onto the original frame, so that the results of segmentation and counting can be shown within the RoI. The top left annotation of the bounding box shows the fish counter and top right annotation shows the life of the candidate in number of frames. When the candidate is occluded or fragmented by surface turbulence, or is touching an edge of the RoI, the top right annotation indicates that the location is ‘predicted’. The green cross marks show the centroids of the candidate blobs.
Candidates that complete an unbroken track across the RoI are added to an accumulation count of salmon.
In the limited testing of good quality video sequences that has been conducted to date, the counter has been 100% reliable. However this perfect reliability has to be strongly qualified by the quality of the video and the heuristic nature of the approach. To establish a baseline of effectiveness, the initial testing has concentrated on video sequences in which the salmon are clearly visible and counts can be easily verified by a human operator. Moreover, factors such as the limits on blob size, the upper limit for the elimination of small blobs and the selected number of frames used to eliminate false positives must all be chosen very carefully.
The approach to salmon escapement counts described in this paper has the potential to relieve the demand for manual counts by operators. The approach has been successfully applied to pre-recorded video sequences. However it is yet to be tested on live video and has been applied in a limited range of water surface conditions. Further, testing is needed to validate the results against reference data provided by experienced human observers. The most difficult circumstances for salmon counts are twilight periods, wind-induced surface turbulence, suninduced surface glare and discoloration of the substrate panel. Figure 9 illustrates the range of different conditions experienced over the course of a single day at Chenik Creek. It is evident that, as conditions deteriorate, identification and counts of salmon become commensurately more difficult. Despite the use of a polarizing filter, surface turbulence from the wind and upstream obstacles can exacerbate sun-induced surface glare to the point where the ability to see and count fish is severely compromised. The substrate panels can become discoloured by deposition of silt or algae growth, or may be interfered with by bears or floods. Fortunately, interference with the panels is infrequent and discoloration is rectified during regular maintenance visits.
Further development and testing is necessary to extend the effectiveness of the approach to the full range of conditions that will be experienced at the remote video stations and decrease the reliance on empirically chosen factors to control the algorithms. The critical step in the video processing is the segmentation within the RoI, as this determines the success of the following steps. Different approaches, such as segmentation within the colour space and frame differencing from an averaged background image will be the subject of future research.