I’m glad you’re concerned with learning this skill. Bluegrass music depends on the musicians knowing how to pick things up by ear.
There's a Homespun video out called Playing Banjo By Ear, or some such, by Bill Keith. I have never watched it, and can't vouch for it, unfortunately. That's the only thing I have seen like that.
I teach ear learning as well as I can at my jam camps. A lot of it is based on just practice. One of the best exercises: Get a songbook with familiar songs in it, and pick one with just three chords. Now, close the book and try (trial and error style) to GUESS the chords, both when they change, and which ones they change to. With only three chords to guess from, your choices are limited, and you might find you can do it, at least some of the time. If that's hard, go from the list of 2-chord songs on DrBanjo.com. Once you think you have it, then peek at the answers, and review the correct changes. If you can do it at all, keep doing it, and your ear for chord changes will slowly improve.
Finding melodies by ear is another important ear skill, and the trial and error method is the main way to go there as well. If you can carry a tune, hum or sing along with the right chords for a song, then start at the top and see if you can find the melody notes, one by one. Go slowly, and as you get each note, then hum from the beginning up to the one you don't know yet. Hum just that one note, and look for it, keeping in mind if it's lower, higher, or the same as the previous one. If it’s higher, move up the string, or switch to a higher string, and so on. Slow going, but it brings results.
If you have trouble carrying a tune in the first place, that's a more fundamental skill that you should probably work on first. Try chording along with familiar songs from a songbook, and singing as you chord. Important: Most of the main melody notes are actually among the notes that comprise whatever chord is going as that note is sung. So if you hold the chords they tend to pull your singing toward the correct note (which is likely right there in the chord).
Likewise, when finding melodies on the instrument, start with the correct chord. Often the main melody note will already be sounding as part of the chord.
Sorry there's no "just follow these simple exercises" answer to your question. It's trial and error, basically experience, that gets you there. Ear skills are *very* important in bluegrass, and you can't just learn them out of a book. Having a patient guide (needn’t be a banjo teacher) can be a good help, telling you when you’ve got it right or not (but not too quickly), and helping you out when you’re in left field.
If you can’t carry a tune, have a look at the newly posted article on DrBanjo.com under The Doc’s Prescriptions, "Learning to sing in tune".
Best of luck with your learning!