First of all, by no means should you quit if you love it. Many good players just can't play fast, which limits what they can play, and sometimes whom they can be comfortable playing with-- but they can still enjoy playing really neat music on the instrument. No reason to not try. Your chances are good. Possible proof:
Yes, 120 beats per minute is 8 notes a second, which sounds fast, at least verbally. But a second is kind of a long time -- if you hurry you can count to almost 8 in a second. Try drumming your three fingers on a tabletop 3 times a second. That should prove to you you can move your fingers that fast. Of course, to have them learn patterns well enough to play them at that speed does take time and work. But lots of players older than you can do it, and the chances are that you can too.
Over a period of time, work on raising the speed by tiny increments above what you've established before. Even improving 1% a week will amount to a noticeable change after a couple of months. Like any exercise, you can tone your muscles, and speed and agility does come with practice.
At my banjo camps we often work on this. I usually have a person take a song they like and already play cleanly and want to play faster, and subject the song to this program of increasing speed. Often there'll be a weak spot that sounds bad when played at the faster speed, even while the rest of the tune sounds ok. That weak spot needs special practice. I often start the person at a slower than normal speed so they can really make sure that part is mastered well enough to be flawless, even if at the slow speed. First step is to make a practice loop that can be played uninterrupted repeatedly along with a rhythm device. Often there is something awkward going on that needs to be smoothed out before the section can be played fast and cleanly. Once it's smoothed out, it ceases to be the weak spot, and the whole tune can now be played comfortably at a higher speed.
It might take a while to work out these flaws, but once they are worked out and the piece can be played very well at slower speed, it's often easy to just do it all faster, with good results.
Enjoy your picking (whatever the speed).