It's definitely possible, but I don't think it's likely. Believe me, I know what panic attcks are, and it sounds like you do too. (I get so pissed and offended when I hear someone use that term as a figure of speech.) My attacks actually decreased in frequency when I began using whey consistently. I think anxiety has more to do with your brain's frontal lobe chemistry. Serotonin, gaba, dopamine, and other neurochemicals can wreak havoc when one or more of them are out of whack.
Have you ever been prescribed medication... SSRI, benzodiazepine, Buspar, beta-blocker, etc.?
Btw, after making nutritional changes and learning how to meditate, I haven't had panic in almost 6 years... so there's hope, dude. You're not crazy... I think.