The “AI for Orcas” project (#ai4orcas) represents a 2-decade effort by the Orcasound and open-source communities to develop machine learning models to classify signals made by killer whales (calls, clicks, and whistles), in real-time, starting with the calls of the endangered orcas known as the Southern Resident Killer Whales (SRKWs). An important part of the process is generating high-quality and open-access training and testing data sets. Along the way, we are building and sharing bioacoustic software that may help conserve other types of orcas (which make similar sounds), other marine mammals, and even soniferous species in non-marine environments.
Key links
- How to contribute to Orcasound as a “hacker”
- Orcasound Github repositories
- Orcadata wiki: orca-specific machine learning resources & efforts
- Google Summer of Code 2020: Orcasound page
Latest updates
- L90 and her new calf vocalize & echolocate alone (9/17/2024)
- J pod calls, but doesn’t click? (5/22/2024)
- Exciting (S10) L pod calls as the sun sets over Orcasound Lab (8/25/2021)
- Acoustic inference as J pod departs Puget Sound & heads to Canada (10/1/2020)
- My Journey as a Google Summer of Code participant with Orcasound (8/26/2020)
Project history
From its beginnings at the turn of the century, Orcasound has promoted a “friendly competition” between humans and machines to reliably detect endangered southern resident killer whales (SRKWs) acoustically, in real-time. Some twenty years later, technology has advanced dramatically and we are building new synergistic solutions that combine the power of crowd-sourcing and the human mind with the power of artificial intelligence (AI).
In this rapidly-evolving arena, Orcasound is helping to organize the open development of new bioacoustic solutions. The evolution towards a “friendly cooperation” via increasingly-open efforts — from ~2002 through 2019+ — is depicted chronologically below. A similar story is told by the Orcasound human-machine learning slide deck (presented at the 2019 ONC/Merdian workshop on Detection & Classification in Marine Bioacoustics with Deep Learning).
2000-2005: Visual Basic & mp3 streams
Orcasound Lab on the west side of San Juan Island began streaming live audio data from SRKW habitat using underwater microphones (hydrophones) in 2002. On a PC running Windows XP, a Winamp Shoutcast plugin generated the audio stream in mp3 format, allowing citizen scientists to listen live through a wide range of devices and programs. Custom software written in Visual Basic also monitored the hydrophones in real-time, not only computing calibrated noise statistics, but also making short recordings whenever an amplitude or tonal detector was triggered.
2005-2015: QT & Google sheets
The initial “friendly competition” took the form of Val Veirs iteratively improving the detection software at each hydrophone node while his son, Scott Veirs trained human listeners to detect SRKWs and log their observations in a Google spreadsheet. Val’s original Visual Basic software was known as “WhoListener.” Later, he implemented similar detection functionality and added classifcation schemes in platform-independent QT and dubbed the software “Zorbita.” At the same time, Scott organized partnerships with Orcasound organizational members to advance online resources and in-person exhibits to train humans to be better listeners (see the Orcasound learning page). Val’s machines got smarter (but still generated a lot of false positives) and so did Orcasound citizen scientists — first logging their observations in a collaborative live Google spreadsheet and then using social networks to discuss and share their detections in real-time.
(with Audacity showing waveforms & spectrograms of orca calls)
Most of these research and education efforts were funded by NOAA, especially during 2008-2012. Lots of volunteers and — after 2012 — other funding sources, like WDFW and Beam Reach, kept the Orcasound hydrophones working.
2017: cloud archive & browser listening
With new funding from a 2017 Kickstarter campaign, Orcasound developed an open-source audio streaming solution that made it easy to listen for orcas through a web browser. The first version of the Orcasound web app launched publicly in November, 2018. Version 2.0 entered beta testing in November 2019.
The orcanode streaming software archives compressed audio data from each hydrophone location in cloud-based storage (Amazon S3 buckets). In contrast, raw data from the old streaming system was only archived when listeners recorded the mp3 stream, or the automated detection software triggered recording. This shift opened the door to real-time machine learning in the cloud.
2018: “hack” events & machine learning
In 2018, Orcasound began participating in “hack” events — one-day events that bring volunteer coders (aka “hackers”) together to help good causes. Monthly hackathons were organized by DemocracyLab at the Code Fellows facility near the Seattle Center. Occasional hack days at the University of Washington were organized by Valentina Staneva and Shima Abadi and hosted by the UW’s e-Science Institute.
The hackathons brought an intense burst of talent and new ideas to the AI side of Orcasound, fueled by the emergence of new machine learning tools and a concentration of talented data scientists working for local tech firms or at the UW. In the fall of 2018, UW computer science graduate students like Jennifer Rogers began attending hackathons and contributing to the Orcasound software repositories on Github. Independent data scientists like Erika Pelaez worked during and between hackathons, advancing their own machine learning models for SRKW calls. Erika was the first to publicly present her results — at ML4All in April, 2019 (see video)!
2019: Microsoft hackathons+ & Canadian-catalyzed collaborations
Orcasound continued to benefit from DemocracyLab hackathons in 2019. Between hackathons, Mike Castor — a Code Fellow graduate who learned about Orcasound at a DemocracyLab hackathon — put in a heroic volunteer effort to implement version 2 of the Orcasound web app UI. Orcasound 2.0 entered beta-testing in November, 2019, and delivers new features for citizen scientists: a button to push when interesting signals are detected, plus the option to receive real-time notifications about network activity. Giving humans this new way to tag the live audio stream — identifying “interesting” periods when signal density is high — was a key step towards a more-efficient machine learning pipeline and a more-automated real-time detection system for Orcasound.
In July, 2019, Orcasound was invited to participate in the annual internal hackathon organized by Microsoft. In a 4-day sprint, a team of volunteers (Akash Mahajan, Prakruti Gogia, and Nithya Govindarajan) created a new tool that makes labeling Orcasound data much more efficient. At the same hackathon, a separate but related effort to use AI to detect SRKW calls in real-time was led by David Bain of Orca Conservancy and a ~10-person team of Microsoft employees led by Chris Hanke. Incremental advances in machine learning, most importantly additional labeled Orcasound training and testing data, followed in fall, 2019 — thanks to the combined talent of these teams during and between additional Microsoft hackathons.
In the latter half of 2019, Orcasound — both our machine learning efforts and the network in general — began benefiting from generous financial support from Microsoft. Internal hackathon hours were donated by employees, and their value was matched by Microsoft donations to Orcasound via our non-profit organizational members. For example, Orca Network received support to repair the Bush Point hydrophone.
Motivated by previous collaborative efforts in 2018 to fund UW students to advance Orcasound with machine learning innovations, Orcasound members joined our University of Washington partners in applying to the 2019 Leonardo DiCaprio Foundation and Microsoft AI for Earth Innovation Grant with a proposal entitled “Detect2Protect.” Orcasound also supported David Bain’s successful 2019 application for an AI for Earth grant of Azure credits and data-labeling funds that would leverage existing non-Orcasound archives of SRKW calls labeled by from Monika Wieland of the Orca Behavior Institute (an Orcasound member) and Candice Emmons NOAA/NWFSC to advance open ML models.
Orcasound’s connections with the broader community of bioacousticians and data scientists were strengthened by our participation in the Canadian deep learning and bioacoustic metadata workshops in November, 2019. Hosted by Ocean Networks Canada and Meridian, the workshops were attended by colleagues from the UW, DFO, and Europe — many of whom have since begun collaborating with Orcasound. Here are the two contributions from Orcasound:
- Detection & classification of SRKW calls through human & machine learning, in real-time
- Orcasound metadata and annotation practices
2020-21: Open-source collaboration
Google Summer of Code 2020
In continued pursuit of a way to get students more involved in open-source software development for orcas, including ML models and associated tools, in early 2020 Orcasound became a host organization for Google Summer of Code. The two GSOC students selected to work with Orcasound in 2020 were Kunal Mehta (from Mumbai, India) and Diego Rodriguez (from Monterrey, Mexico). Together with their mentors, they built a new active learning tool, OrcaAL, to crowd source the annotation of killer whale calls in audio data from Orcasound.
Latest posts from GSoC 2020:
Google Summer of Code 2021
In our second year as a host organization for Google Summer of Code, Orcasound worked with four amazing students from around the globe. Two students worked on aspects of the machine or active learning pipelines while the other two advanced the web infrastructure that supports and trains community scientists.
Most relevant to ai4orcas, Jose Giraldo worked on embeddings and visualizations within the OrcaAL web app. Dmitry Volodin pioneered Github actions as a means of improving data access and easing deployment of new algorithms. Learn more about their projects, as well as the work done by Dhananjay Purohit and Isabella Macchiavello, on the Orcasound GSoC project page.
Microsoft hackathons & AI for Earth support
- Microsoft hackathons continue about monthly in 2020, including remote-only hackathons during the Covi19 pandemic
- Microsoft’s AI for Earth program is supporting data labeling, some facets of the Detect2Protect proposal, and Azure resources for ML model development & real-time pipeline(s)
- In 2021, the combined OrcaHello team meets primarily in March and October, but coordinates occasional progress via Teams and tracks PRs and issues via the open aifororcas-livesystem repo.
Canadian-led HALLO project
- Orcasound partners with a group of Canadians from Dalhousie, Simon Fraser, and Carlton Universities, funded by DFO
- Through bi-weekly meetings in 2021 and occasional Meridian webinars, Orcasound collaborators help trans-boundary effort to share open audio data, annotations, call catalogs, models, and deployment schemes
I spent 10 years developing an artistic experiment of inviting musicians to perform live improvised music into the waters off johnstone strait to interact with northern residents. The best recorded examples of those interactions have recently been invited to be added to the permanent collection of the smithsonian institute. I have over 50 hours of recordings on cassette and DAT. If you have any interest to do AI analysis of these sound documents, let me know. Be aware that I have no idea what your analysis would either look at, or show.
Hi Jim!
It would be wonderful if you could share a link to the archive here once it the excerpts are in place with the Smithsonian. The ORCA-SPOT AI, trained using OrcaLab annotated audio data from northern residents would probably be the best algorithm to try first in an analysis of the musical interactions. The OrcaHello AI we’ve developed has been trained only with southern resident calls.