Hello Jen,
No, it's not a nasty story. I was diagnosed with ulcerative colitis when i was 20. I am now 34. Without alarming you, this is the reason i keep getting anxiety, i have just recently been going through another flare up and my anxiety came back with it.
My suggestion to you would be to definately go ahead with the test. You need to know what it is you are dealing with so it can be trated properly. When you get to your appointment(or before if you like)ask to speak to someone about some concerns you have and then explain your anxiety/panic to them and they will help to settle you and to make you feel more at ease with what is going on. The test does not hurt. Let us know how you go and if I can be of any further support whether it be panic or colits/crohns realted, just holler!!
Cheers
ConquerQueen