Dear Dave and Paul,
This is a big topic! You have already thought of some of the main things to consider. I think this problem is a tough one for banjo players, more so than other acoustic instruments, due to the head being so sensitive to vibrations from monitors. Tony Furtado tapes his head up so much it seriously deadens the instrument and changes its tone pretty drastically. Bela Fleck when he plays with the Flecktones uses a mounted-on-banjo mic and I think a pickup, and blends them. Still doesn't sound as great to me as his great tone as a player, through a regular mic.
Best solution from some points of view is if the whole band is on ear monitors. Ear monitors don’t feed back. Mics can work pretty well in these situations. There are important drawbacks, but the band can really hear itself, and with separate monitor mixes, it can be pretty cool. But it's weird hearing only through headphones and not hearing the room at all. Also, ear monitors are pretty expensive when you consider that they require a wireless system to send and receive the signals. Maybe there's a way you alone could use ear monitors with no speaker-monitor, thus possibly reducing the likelihood of feedback.
I use a Prucha Elban plug-in banjo (with pickup) for sit-in-with-loud-bands gigs. It is a "real" banjo with normal head size but a slightly smaller resonating cavity. It sounds pretty good, but I have to watch out for having it too loud in the monitors, or... feedback.
For what it's worth, most all of the times I play (even with my band Flexigrass, which has a drummer -- a relatively quiet one I might add), I do fine with a microphone. My favorite, AKG 414, actually has a wide pickup pattern, but rarely causes feedback in the situations I use it. Shure SM-57 is another favorite, less feedback-prone, with a much smaller pickup pattern.>>
Of your 5 suggested solutions, I think these two are best:
- Get a better pickup?
- Get a fancy amp with frequency filters and the like?
For me, the Prucha works, but I only use it in occasional situations, not my main band.
In general, it's a good idea, as you did, to ask people who you think can be helpful. Sorry I can't be more helpful, but at least you now what I know.
Best of luck, and if you find a good solution, please let me know!