well, as far as “proprietary vs free” software needs, there are very few free vocal tuning (“Auto-tune”) VSTs out there. It turns out fixing vocals is pretty important to all kinds of production, it’s not just for turning vocalists into robotic auditory paste.

Tuners are just super useful tools to begin with. The math behind recognizing pitch is, evidently, rather tricky. (I have more than one guitar tuner that will take about a half-second to decide that my E string is tuned a little bit low.) So the first step would be writing a pitch detector and getting it to work on a guitar, and a bass guitar, and then a voice.

Once you’ve got an algorithm for pitch detection working, then you’d need to get it to respond VERY quickly (which would take some next-level cleverness to do.) After that you’d have it analyze an input signal to graph what frequency the signal is at, and then choose which notes those frequencies correspond to. Those notes could probably be stored as MIDI data. By this point you’ve already achieved a “sing your own MIDI notes” VST, which I’ve already seen people asking for.

Lastly you’d need editable parameters for each note (or group of notes) to describe how to adjust the pitch from the detected frequency to the desired frequency. One parameter is how quickly it changes the note (which gives that characteristic “robotic” feel that is just pervasive in pop music these days.)

I think this could be a fantastically useful plug-in; it’d certainly be nice to have a useful free alternative for people who can’t afford hundreds of dollars of software.