Here are a few things I've done that might be
Audio and Video
I built one of the first systems using speech
recognition to retrieve video content. This work earned the Best
Paper awards at SIGIR
Multimedia. Here's a video
demonstration of the system (8 minutes
MPG, 80 MB). Years before most people were thinking
about it, I built a system
retrieve news video from a keyword search of the subtitles.
I've been inventing ways to search and retrieve musical audio for
possibly longer than anyone else. I built the first content-based
search engine specifically for music. I wrote a widely-cited review
paper about audio information retrieval. I've invented ways
retrieving music by rhythmic
similarity as well as long-term
structure, and one of my better inventions automatically
segments popular music using matrix factorization.
I invented an information-theoretic model for
distributions, allowing audio and video data to be classified and ranked
similarity for retrieval. I've released the code on
SourceForge as TreeQ.
I developed a fundamental approach to analysing audio,
text based on similarity matrices. I've used this for many
interesting applications. Many others
now use this approach.
I came up with the beat
spectrum, a new approach to characterizing rhythm in musical
Panoramic Video Maps
For those interested in GIS and mapping, Don Kimber
and I linked GPS
data with recorded panoramic video for a photorealistic
virtual travel system called FlyAbout.
This grew out of a multiple-camera panoramic
video system I developed.
I've come up with some interesting ways of
interacting with video and audio. A few examples:
- A video
browser showing keyword relevance on a timeline.
is an automatically generated printable summary of video
makes video an active interface for remote control.
remote users draw on real-world objects through an active video image.
turns an image collection into a video animation synchronized to
- The AMV
automatically generates short music videos with synchronized music
For fun, I
things. I'm unafraid of doing stupid
with higher mathematics. Finally, DON'T
somewhat less professional stuff, I have delusions
Oh yeah, I have a bunch of patents
few of the more interesting
ones are here),
page, and if you care about my degrees and awards, you can
them in my bio.
Credit and thanks to co-authors and collaborators too numerous