Gary was definetly the hardest for me, personally because I was shocked by how fast his team had grown. In one battle after the SS:Anne, his pokemon were around level 25 in the pokemon tower, then when I fight him at Silph Co. he has a level 40 starter and a level 37 Pidgeot? Man that is crazy, especially when his old pokemon used to grow up slowly.
Silver was alright for me, sometimes annoying but sometimes easy. The whole generation was easier because the pokemon were lower levels than the first.
The only time May ever annoyed me was before Mauville City with her Lombre and Combusken, I swear it was seriously difficult to defeat a combination of nature power from Lombre and then double kick and peck from Combusken. Her Pelliper was easy but it was just those two pokemon. After that though, she was seriously easy, dissapointed that her starter didn't fully evolve.
Barry was ridiculously easy. His starter and the elemental types were pretty easy as Floatzels, Roserades and Rapidashs defenses aren't that high. Snorlax was difficult, the pokemon I tried to defeat that at first was my Weavile but night slashes and ice shards did nothing, so I used Staraptor and his close combat for him. Heracross was also too easy, just send in a fire pokemon to burn him and if everyone else is dead, revive staraptor. His Staraptor ended up being annoying at the battle zone with U-Turn, but it was eventually another lamb to the slaughter.
And Cheren and Bianca on Black and White? Easy just like all the other trainers in that game. The levels in black and white and less than even gold and silver, which is saying something.