Yes, the champion is hard but the E4 is getting ridiculously easy. Let's see (talking about the E4 only):
gen 1: using legendary birds plus a psychic type (or electabuzz) makes them so easy to take out. But they were hard, no doubt, especially for me who didn't grind back then. I had pokemon 6-10 levels under lorelei.
gen 2: Dayum, that's some hard foes. even with SE moves against them, they were very hard. For me they were the hardest. And don't even get me started on HGSS E4.
gen 3: Drake doesn't forgive. The other 3 were not easy but drake...these were not as hard but still challenging.
gen 4: Kind of depressing, all of them are easy to beat except for lucian and he's not that hard. Cynthia though was overkill compared to the E4.
gen 5: Hey E4, you're here because...? Drayden is harder than all of these guys. A decent dark type, a psychic type and a fighting type. 3 pokemon are enough to beat them the 1st time. 2nd time you need a fire type for metagross and bronzong but still...very easy to beat.
So in conclusion...If they increase the AI and put decent movepools in the teams, i believe there's possibility to make the E4 some fearsome adversaries.