Yes, panic attacks are common in that at one time or another pot smokers will experience the symptoms. Usually when the pot is much better than usual or if it's a been awhile since you've toked, you'll increase your chance of having a panic attack. It can also be manifest as a generalized paranoia, not necessarily due to a specific fear. This can also be due to nervousness, stress, unusual situations, strange people.

When you feel a panic attack coming on, a good idea is to isolate yourself, pick out some soothing music and just groove. In other words find a calming distraction (a walk in the woods, meditation, yoga), or just take long slow breaths and watch your breathing. Just focus on that and keep it slow and even and you'll calm down very quickly.

A really cool thing to learn is how to calm yourself, and refresh yourself through some form of yoga or meditation. Once you get proficient you can then use whenever you feel "out of balance" and it will restore your balance and overall health and attitude. It is the best prevention there is. And since you pointed out that certain people make you uncomfortable you can avoid those situations and be with people you enjoy.

Hope this helps!
-The Old Hippy