If you imagine these League conferences like tennis Grand Slams, ‘dropping a rank’ from one league conference to another isn't that unheard of. All the trainers at each conference are strong because they have obtained 8 Gym Badges, and most matches are single elimination. Just like in tennis Grand Slams, any player can beat another, and the margin of victory is slim. If you have a bad draw, you can lose very early. Therefore, it’s very competitive and to actually win one is incredible. To reach the Top 8 out of a field of 128+ players is pretty darn good. I mean, returning to tennis, even Federer has lost in the QFs at several Grand Slams.
Still, Ash was on the verge of reaching the Top 8 way back in Kanto (if not for Charizard’s disobedience). He narrowly lost to two very strong trainers in the Top 8 in Johto and Hoenn. And even though he lost in the Final 4 in Sinnoh, he was basically the second best trainer in the entire conference. So him losing in this fashion (where his opponent is completely scatterbrained and has one less Pokemon) makes his Top 8 performance no longer as impressive, and actually reflects negatively upon him.