OT: Has anyone tried the opposite - ask AI to listen to music and determine the notes or chords being played? Or watch someone playing an instrument and give a textual output of what notes/chords they are playing.
I did this for my graduate capstone (https://www.deepjams.com/)
We extracted chord progressions from existing music you would upload and then riffed based on those chords. there are open source libraries for this.
I think these are all using old machine learning techniques and not the modern transformer based architectures that underlie LLMs. These tools won't be able to match the abilities of an expert musician replicating a song by listening to a live recording of it. Check this video channel where they ask professional drummers to replicate a song after only one listen [1].
I use https://moises.ai/ multiple times a week for practicing / figuring out chords being played. For the notes (say in a guitar riff), I dont know if such a thing exists
I would love this! There's a song I like by a band that broke up in 2013 and I am transcribing it by watching a live performance they did and trying my best but realizing I'm trying to take a mandolin/guitar and put it to acoustic. Even just being able to do a similar rendition would be nice by telling the AI "hey, do a twist on this and give me the chords/tabs".
I'm very interested in this too. We're beginning to see models that can deeply understand a single image, or an audio recording of human speech, but I haven't seen any models that can deeply understand music. I would love to see an AI system that can iteratively explore musical ideas the same way Claude Code can iterate on code.
Reminds me of an example in a similar direction, where AI was used for audio processing to filter out everything except a person's voice. If I remember right, it was able to focus on different people in a crowded room. It might have been also for music, to pick out an instrument and listen to it, filtering out the rest of the band.
There are several tools that do this already - AnthemScore, Spleeter, CREPE, and even Google's AudioLM can transcribe music to MIDI with varying accuracy depending on instrument complexity and audio quality.
It's not as lucrative. For human produced songs you can usually get the sheet music for them. If not, musicians can listen and do it manually, but it's not common enough to need AI to do it. Just transcribing for one instrument isn't that useful for many cases. Often they need an arrangement for multiple instruments, and depending on which instruments, the key may need to be transcribed. This is mostly referring to classical music and traditional western songs.
LLMs can do this well, though, and there are such. They weren't calling themselves AI when I last looked a couple of years ago, but I'll bet any of them looking for VC money have rebranded since.
