I agree with Cody in that this is completely your choice to make.
Though I'm confused as to why you are so against going to see the cardiologist? I'm just thinking of ways they could maybe think you have a cardio problem. Could it be to do with your blood pressure or pulse? Maybe an ultrasound as they do sometimes have a glance at the heart, or an xray. Maybe listening to your chest with a stethoscope they've heard something? Perhaps infection frequency, or lung function decline. Could be many things. Might not even be CF related, which is why they want you to see a cardiologist as its out of their specialist area. Heart murmours seem to be somehting commonish with CF, I've heard of a few people having them, most heal quite easily and quickly on there own.
It might be something really small that just needs to be kept an eye on. It could be something bigger, that you need tablets to keep it under control. I don't think they will be wanting you to go to a cardio, just for the sake of it. It could be for a reason to prevent future problems.
For me, the risks of what it could be, would be well worth it. But then I do lean towards overcautious always. I'd probably need to go, to stop myself panicking about what could be. I think a good first step though, is asking your CF team, why they want you to see a cardiologist. What have they spotted that you can't feel.