If I wanted to take that logic further, already even less than 10% of the population are the ruling elite. Who are these? The Swinging voters. The vast majority of voters always stick to one party - for example, while you may deny it Manaphyman will almost always vote democrat, and you BigLutz will almost always vote Republican. However there are some people who only make up their mind at the last minute, and swing from election to election. Seeing how small swings in elections are ("landslides" are elections with over a 5% swing generally, and they are uncommon), one could easily and justifiably say that these swinging voters, less than 5% of the population determine the fate of a normal (non-landslide) election. Wouldn't it be this group that is a ruling elite? Or another example. With this ridiculous caucus system, wouldn't it be places like Iowa and New Hampshire that become the "ruling elite"?
Completely irrelevant. Under the Electoral College system, "Swing States" do become major campaign targets, however, due to their limited number of votes, their sway in the overall campaign is still limited.
Under a direct popular vote, however, campaigns would focus entirely on the most heavily populated areas, because winning these would ensure victory regardless of remaining swing voters.
Both systems grant more power to a certain region/voting block. The difference is how much the power is skewed. Under Electoral College, Minnesota recieves more attention than it otherwise would, but not enough to steal the campaign away from other states. Under a direct popular vote, every single campaign would pander almost entirely to California and New York.
Why does it matter if the candidates go only to the population centers?
Because they will then tailor their political platforms to fit the region with the highest voting population. You'd see alot of special interest politics, with desicions being made to appease population centers at the price of other states. It doesn't just skew the campaign trail, it skews political power.
The Electoral College keeps political focus balanced, and usually represents the popular vote anyway. Sure, once in a century you'll have a Bush/Gore scenario, but it's by far the more desirable system.