New England Marine Monitoring (NEMM) and our partners at CVision AI and Teem Fish Monitoring have been actively working on training, scoping, and implementing artificial intelligence (AI) into electronic monitoring systems. Our work in this area has been funded through a NFWF Electronic Monitoring and Reporting Grant Program award.
While much progress has been made in developing AI for New England EM projects, the utility of current tooling is still very narrow. To achieve desired efficiencies from AI we believe it needs to be adaptable and portable. We envision a future where we have a variety of AI tools in our toolbox that can be broadly and strategically deployed to maximize impact. Through our NFWF award we set out to test the feasibility of porting AI tools between programs, review platforms and vessels. An example of this interoperability is illustrated below. We chose specific algorithms due to our intimate experience with them, and ready access to code and experimental data.
We limited our project to trawl vessels in an effort to streamline the need for additional training and in hopes of achieving appreciable results. The two vessels of focus were active EM vessels, one from the New England Audit Program (NEAP) and one from the Maximized Retention EM program (MREM). Both programs are focused on accounting for undersized groundfish discards.
Figure 1.) Example of algorithm interoperability between review platforms and across vessels
In the NEAP program, fishermen measure all of their groundfish discards before discarding them at sea. In the MREM program, fishermen are required to land all of their groundfish discards and meet a dockside monitor. While building review protocols and considering our approach it was critical to keep these different program elements in mind.
Tools, integration and trials
Through previous work with MREM vessels, NEMM and our collaborators with GMRI and CVision AI had developed activity recognition(AR) algorithms that can flag on-deck activities within certain categories for reviewers. The goal is to identify when a specific activity is happening within a video scene (e.g discarding of fish, loading or unloading of the hold, etc.) This effort was developed in the maximized retention program, and passed to SnapIT for integration into their review platform known as Engagement. Collaboration between SnapIT and NEMM review staff resulted in an integrated tool that was used for experimental AI review trips. Figure 2 shows the activity-flagging interface developed by CVision AI and implemented in the Maximized Retention program now integrated and functioning in trials on SnapIT’s software for the NEAP Program.
Figure 2.) Screenshot from Teem Fish/SnapIT Review software Engagement. The yellow bar (enlarged below screenshot of fishing) represents activity flagged for review.
Through previous work, NEMM and other partners had access to object detection, object tracking, and species ID algorithms. These algorithms were integrated into the current Maximized Retention review platform workflow with the goal of detecting, tracking and classifying objects being discarded. This required additional training of species ID classification algorithms to produce reasonable results on the Maximum Retention data set. The MREM program protocols require groundfish to be retained, and the resulting lack of groundfish discards had left little training data for our project team to work with. A substantial volume of non-groundfish species are discarded on each trip, however, so we decided to shift towards training the classifier to recognize those species and only flag individuals that could not be determined as non-groundfish. Figure 3 is flagging a discard for identification by a human reviewer on the CVision AI review platform.
Figure 3.) Shown here, our system flags a discard for a human reviewer to confirm as a non-groundfish species.
Through this work we were able to demonstrate the ability to transfer development efforts across programs, platforms and vessels, dramatically increasing the utility and return on investment of AI tools. While there were technical achievements in this project we feel the most important result is the collaboration among EM companies and the commitment to improving EM capabilities for all. We are hopeful that the success in this project will encourage continued collaboration among stakeholders to maximize efficiencies made possible by technology.
We are currently conducting experimental AI-assisted reviews of trips that had been previously reviewed (for their respective programs) without AI, in order to make direct comparisons. We look forward to sharing all of our results and lessons learned in our final report, which we will share on EM4Fish.
Projects in the Field is a series of independently produced articles profiling work supported by NFWF’s Electronic Monitoring & Reporting Grant Program, and is meant to raise awareness and support for these important initiatives. The author welcomes your questions and comments.
