No, there's not really a down side to using yeast nutrients. What is not consumed by the yeast falls out of suspension.
But in beer it is usually not required since there is plenty of nutrition in the wort. But in mead, you need to add the nutrients since there's not much in honey.
What I do is only use it when it is required. It's cheap, and there's no real downside, but I don't like to add anything that's not required.
