No, as long as you use it at the right time. If you're planning on EV-training your Pokemon, it is best to not use them until after they are fully EV-trained.
Once the pokemon is EV trained, it makes no difference if you use a Rare Candy or train it. So as long as your pokemon is EV trained they're not bad in any way.
