It's not determined by the trainer number... otherwise more people would have been able to visit the place...
it's actually determined by a number that's hidden by the player, every pokemon is assigned a number when it's generated, and it's pretty much permanent.
This secret value determines plenty of things with pokemon, such as Spinda's spot placement, what wurmple evolves into, shiny pokemon, and mirage island's apperance.
the Mirage island's "number" is changing each day while your pokemon's number remains permanient. If you manage to bring the right pokemon to the man who looks for Mirage island, he'll spot it and you can go there.