My quickest answer is that if you want to play an instrumental that no one else knows or can take a break on, you may want to skip it altogether. At jams, it's best if everyone gets a chance to shine, and if there are people present who can improvise breaks to easy songs, that gives them that chance. Instrumentals are much harder to improvise breaks on (the breaks are longer length, and the instrumentals are typically played faster than typical songs). But if people urge you to "pick one" even if they can't take a solo on it, make sure they can follow the chords before launching into it, then go for it. Three times is probably about right.
When it comes to songs, as long as you've picked one that is not too hard to follow, it's not so bad if it's one no one has heard before. But you need to make sure there's a dependable way for everyone to follow the chords. For guitar players introducing songs, that's little problem-- everyone can recognize, or should be able to, guitar chords. For banjo players, though, most people can't follow their left hand chording, so you need either:
1. A guitar player in the jam who knows the chords, which everyone can then follow, or,
2. A run-through or two where you preview the song to everyone while calling out the chords. Some people are pretty good at singing the song and inserting chord names where they belong, substituting for song lyrics. Such as: "Yonder stands little F, with a G glass D her G. She's drinking away her F while G a D G."
These kinds of protocols are really helpful at jams. If people don't know how to do them it can quickly increase the discomfort level of the novices in the group. If people do them well, everyone's confidence and fun level goes way up. Your asking the question in the first place is a good sign that you are gaining experience and have your antenna up.
For lots more practical hints about making jams work smoothly, check out my new Bluegrass Jamming video.