What kind of vitamin D are you taking? Vitamin D3, cholecalciferol, is the natural form and is usually safe as long as you don't overdose. It is very inexpensive and can be purchased anywhere OTC and is produced by being exposed to sunlight. But of course if it bothers you then don't take it.
However there is also artificial vitamin D2, ergocalciferol, which is what a doctor usually prescribes if he finds that you are deficient. This D2 form of vitamin D can have many more undesired side effects and is more easily overdosed.
/www.drugs.com/mtm/vitamin-d2.html