Gameplay balance.
The more playable characters you have, the harder it is to properly balance them. Pokémon in particular suffer a lot because of it, with many Pokémon having obvious oversights in their movepools or stats (e.g. Cyrogonal can use Attract despite being genderless). By rotatint the roster, it's possible to fix that, since the Pokémon that DO appear in the game are very likely to have their movepools and stats fixed.
That's exactly what other Mon franchises do, and just to mention two examples:
1) Medarot 2 (GB) has 150 Medarots, it's remake, Medarot Ni CORE (GBA) only has 120.
2) Yo-kai Watch 4 (Switch/PS4) only has 213 Yo-kai, while Yo-kai Watch 1 (3DS/Switch/Mobile) has 245 Yo-kai. And keep in mind that in day 1, there were only around 110-120 Yo-kai in 4, the game that had to be delayed multiple times even.
No, the reason is time. Pokémon games come out on an annual schedule, with a new game coming out around the holiday season every year, or in the case of 2020, multiple DLC expansions.
There are well over 1,000 Pokémon designs accounting for every different form. You'll need a different model for each of these different forms, whether it be the 28 variants of Unown, Eiscue with and without its Ice Face, the different Furfrou trims, etc. Each of these takes time to make, and more importantly, each of these takes time to test to make sure they look okay;
Sword and Shield in particular use Dynamax, which enlarges a Pokémon's 3-D model, and low-poly flaws REALLY stand out when the model is enlarged. Each of them also needs time to animate in different ways (contact move, non-contact move, introduction, fainting, eating, walking, running, being happy).
Mechanically, there are numerous edge cases as well. For instance, Diglett, Dugtrio, Sandygast, Palossand, and Gigantamax Gengar are anchored to or embedded within the floor, meaning moves, abilities, and other mechanics involving lifting the Pokémon off the floor won't work, which has to be determined on a case-by-case basis. Mawile has its back facing its opponents but is turned to face the audience in Pokémon Contests of varying types; should Turtonator be usable in contests, this would happen too. If Neutralizing Gas is in effect and Golisopod's HP is below half, Emergency Exit wouldn't trigger, but if the Pokémon with Neutralizing Gas is removed from play, when does the switch-out occur? (The answer is immediately after the Pokémon with Neutralizing Gas goes away.) Can Trace copy Neutralizing Gas? (No, it cannot.) Do Abilities that reduce physical damage, like Fluffy and Fur Coat, increase the power of Body Press? (No, they do not.) Does activation of a Kee Berry increase the power of Body Press? (Yes, it does.) If you use Cherrim in a double battle and its partner is holding a Utility Umbrella, does that partner still get the Attack and Sp. Defense boost from Flower Gift? (Yes, it does.)
There are hundreds, if not thousands more cases like these, and new ones come up every generation while older ones are sometimes revised if a problem comes up. (A notable one is the decrease of Confusion from 1/2 to 1/3 chance of hurting oneself due to the rise of Swagger followed by Foul Play, a combination that the developers probably didn't account for at the time. Another one is the havoc Smeargle unleashed at VGCs because it can learn Dark Void from Darkrai, which led to a severe nerf in Dark Void.) The last thing they want to see happen is someone crashing, softlocking, or otherwise rendering unplayable the game because they were looking for new strategies (which has happened in the past involving using Gravity when Sky Drop was in effect).
Game Freak BARELY got every Pokémon in to
Pokémon Sun and Moon; there were programmers literally working 24/7 up to their scheduled end of production fitting Pokémon into the games that weren't previously there. They were actually ready to announce that the Alola games won't have all of the Pokémon until this occurred. If they made it in the nick of time for Generation VII, they certainly couldn't do it for Generation VIII, as there are more Pokémon and on a brand new system they needed to understand first.