First of all, congratulations on getting the group going to the point where this kind of thing happens. Believe me, this is not an uncommon situation, and it's not an easy one to cope with, for anyone. I have had similar problems with jam sessions being crippled by a stringed instrument (banjos are the most likely culprit) playing too loud or too busily. Washboards and especially spoons are pretty controversial, even if played well and tastefully.
Let's remember that when well-played (not too loud, not on every song), these instruments can enhance or at least not detract from the music. So it's a little extreme to me to "outlaw" those instruments. The first approach, which you have used successfully with the spoons player, is to ask them to tone down.
I think what's called for is a discreet talk one-on-one with each annoying player. Such a direct approach takes some definite skill and kindness to pull off. It's not easy for most people, but in fact, as jam leader, you are the logical choice. A responsibility you hadn't bargained on! If you just can't bring yourself to do that, and have a trusted "lieutenant" who could, go ahead and delegate. I suggest you (or him/her) pleasantly and very discreetly mention to the culprit that at some point, maybe afterwards, you want to talk with him/her (or maybe "I have a question for you" without saying what it is). Maybe it would work on a break, or when the thing is over, or even by getting a phone number and asking a good time to call. It really needs to be away from other people so there's a minimum chance for embarrassment. If two culprits can be talked to at once, it might be easier for everyone embarrassment-wise.
By the time that "meeting" happens, they may suspect what you're going to say. Before you talk, please set your mind to being as kind to this person as possible. Thank him/her for being there and contributing, and then admit how awkward it is to ask this question. The question might be, "Would you mind playing a lot quieter at these jams?" You could say that as the jam leader you try to represent the interests of the whole group, and you sense that you are not alone in your feeling. (It is a "bluegrass norm" after all.) You can say that if it came down to it, you could poll everyone, but that would be possibly embarrassing for all. You can mention that while there are some exceptions, percussion is rarely heard in bluegrass because it is considered "string band music". Anyone inserting it should do so tentatively and with the attitude "is this ok?" before digging in.
For what it's worth, in all my years of playing bluegrass, I don't recall ever seeing a spoons player on stage with a "legit" bluegrass band. There's a great bones player (same idea, but less "cloppy" (wood) / "clangy" (metal) than spoons, and quieter) who has sat in with some top groups for a tune or two. A friend of mine known as Washboard Chaz is a good and tasteful washboard player and has sat in with Hot Rize on occasion. John Hartford amplified a board he performed on so the percussion of his dancing could blend with his music. And certainly a number of famous bluegrass bands have used drums (sometimes just a snare) on stage and on records (Flatt & Scruggs, Jimmy Martin, Osborne Bros. Jim & Jesse, J.D. Crowe, etc.). Monroe was strict and quite explicit about a lot of stuff, but he had on some recordings the sound of a bass player hitting the string pretty hard onto the fingerboard, which definitely made a wooden percussion sound. And when I asked him once if it was ok to include a jug in a bluegrass band (as he had had for a while in the early 40s) he just said, "If it don't override the mandolin." (!!)
So I feel there's no "automatic rule" about percussion and bluegrass. You can make one, as jam leader, for your own group, and even post it on the wall where you jam. The rule might be "Everyone watch your backup playing volume, especially percussionists." or "no spoons after 9," as you said. But what do you then do when Mr. LOUD BANJOIST shows up?
Keep being especially nice to the percussionists when they start complying. You might even suggest they take up mandolin, which is the understood main "percussion" instrument in bluegrass. They can add a lot with a crisp chord-chop backbeat and don't need to solo to contribute.
I know you know that it's really all about people having a good time with each other. If that is obvious and implicit in what you do, it should work out. Kindness!