EV's don't come into play straight away. It takes time for your pokemon to gain significant stat boosts.
Plus, I find battling Digletts and Dugtrio's in Digletts Cave to be a lot easier than Pidgeys.
And you say you're training a Charmander? Well, pre-evolutions have very low stat gains per level, I mean, for example, while Attack EV training my Adamant Dratini, after giving it 200 attack EV's, it grew a level and only gained 2 points in attack.
So your Charmander WILL get higher speed, it just won't happen straight away.