Wally/May and Pearl are awful rivals. They're too chummy and, "this is just a friendly battle between friends, friend". Wally disappears for ages before showing up for only two battles (plus rematches), and May loses interest halfway through, which is terrible behaviour for a rival. Pearl teams up with the player a couple of times, and again is still too chummy to count. I think it's the attitude, rather than the position they take, that makes someone a rival.
Blue was cool, and I'd forgotten what Silver got up to (aside from the theft at the start of the game). Obviously I'm going to choose one of these two, and at random I'll choose Blue. He's the only one who became a Champion (and keeps on regaining that position as soon as you leave the building), so he's obviously strong, and he never properly helps you or goes into friendship speeches, so they're big pluses.