Lymphoma/or nothing???

I am a 25 year old male and have noticed a small (about 1 centimeter in diameter) movable lymph node behind my right ear. Its been there for about 3 months. I haven't been sick or had an infection since I have found it.

I was freaking out. My family doctor said its nothing to worry about, but would send me to a specialist because I am worried.

I went to the Ear nose throat doctor and he told me its just a lymph node, and many times said, "he doesn't think its lymphoma, but if you are worried about it, I can take it out for you".

I felt great that day and the next day, but now a week later, I still have my doubts about it. Why can't I trust these doctors?