Didn't Fujitora speak at Reverie as well alongside Kings Cobra and Riku's testimonies? He was one of the admirals advocating for the abolishing of the warlord system, after all. I also feel like Luffy becoming the fifth Emperor is what helped throw the system "off balance", so to "balance" it out, they had to cut out that which was, frankly, a revolving door system of replacements. They had, what, three-or-so Warlords having to get replaced in such a short amount of time before the time skip? That was making the Marines and World Government edgy, and the Summit War happened in which Blackbeard threw a wrench into everything. And when Doflamingo went to Impel Down, they hadn't found a replacement for him before they decided to get rid of the system.
I think Mihawk's reaction to being "relieved of boredom", Doflamingo laughing in his cell, and Boa Hancock's words on her throne says a lot about their feelings toward the Warlord system. They aren't particularly worried that they're "wanted" again, unlike the newbies Buggy and Weevil, because they had a good reason for being Warlords in the first place. So shit's going down for sure, but it's feeling much more one-sided than we're being led to believe.