I used stevia for all the reasons you mentioned here. I don't do regular sugar and I don't do man-made sweeteners. Due to some other meds I take, I have regular blood work done. My blood work was showing some kidney disease. After an exhaustive search as to what could be causing this, I googled stevia and kidney disease. There are lots of issues there for some people. I stopped stevia completely and my blood work promptly returned to normal.
It could be a coincidence. There are so many variables, however I no longer use stevia. I just refrain from using sweeteners. Your body adjusts after a while. Honey in small amounts as well as agave nectar seem to be safe. By the way, just because it says its natural doesn't mean it is safe. Many plant products are natural and completely toxic!!