First runs aside, Steven somehow manages to annoy me every time. I always go into the Elite 4 without grinding, because who needs that stuff? Kanto, Johto, Sinnoh? I'm perfectly fine this way, but Steven's Metagross always manages to screw me over somehow, leading me to PP stall it more often than not. Bringing a Fire or Fighting type doesn't exactly guarantee a victory either, since only half of his team is actually Steel type. He doesn't have the greatest team, but that Metagross alone is awful to fight unless I can manage a crit KO before it wrecks whatever I'm using with Psychic, Hyper Beam, Earthquake, or a potentially Attack boosting Meteor Mash, which is all sorts of bad. So I guess Metagross is the most annoying Champion, not Steven.
I'd say Wallace was annoying, but only because his team seemed to be composed of things meant to annoy you like that Double Team Ludicolo. Even a Grass or Electric type alone doesn't stomp him, though, so his team is as diverse as a mono-water team can get. That said, a mono-water team is reasonably easy to get around. Blue was never any bit annoying, because seriously, it's the same team you've curbstomped about 6-7 before. There's not much you don't already know, since your team is probably fit to beat him at this point, levels ignored. Lance, despite the stupid level curve in Johto, was never that awful to fight, either. If you have an ice type, you've pretty much won, though I know having an ice type isn't exactly common because of Johto's iffy Pokemon distribution. I've never had a terrible time with him, though. I've actually had tougher times with Karen, to be honest.
Cynthia has a great team, probably the best, but she comes a slight bit short in terms of being annoying outside of a Nuzlocke with that Togekiss of hers. Garchomp hurts, no doubt about that, but that moveset comes a bit short in terms of just how threatening it can get. Having beaten that thing with an underleveled Bibarel, Garchomp doesn't bother me. She's got a great team, but it's not annoying to beat. Alder almost doesn't count because he's a post-game fight, but letting Volcarona set up is the worst thing you could ever do. That's the only massive threat he has, though. Iris and Diantha have that nice level curve that I like, but their teams do share a relatively common weakness, so they fall into the same boat as Lance, apart from leveling.