If you haven't started using it at all, you don't have anything to worry about. Nothing that your contacts (meaning people whose email addresses you have or people who have your email address) don't already know about you is shown, and there's no concept of linked accounts. If you did start using it, you can turn off displaying followers/following lists on your profile (even then, it only shows names, not email addresses).

I'm going to do a post about this at some point because I've seen so much confusion on my flist. :-/