One quickly gets into trouble with that position since we do not take any action based on the wide range of physiological advantages of some males over others. We don't say "well, that 7.5 foot guy shouldn't be allowed in the NBA because few males are that tall". And if we don't, it makes it harder to justify saying "ok, so you're a bit taller than women on average, and while we wouldn't utter a peep if you were born female, we are going to utter a peep because you've transitioned."
And as far as I'm aware, we actually do not have indication that the average differences between males and females actually translates into M-F females winning more on average than those born female. The media certainly blares on when a M-F trans person "breaks records", but they don't make much noise when they don't unless you scroll down a while. I
remember noise on DP about a M-F trans body builder who was supposedly breaking records. Well, she bombed out of the Olympics. Didn't even make it in to compete there. One anecdote, but anecdotes are what we mostly seem to have.
I find it very hard to justify barring M-F trans people from female sports when we take no such action with any other biological advantage. (Mechanical ones, yes.
See Pistorious. And that was of course its own debate...
did he actually have an advantage?). And that's not to say that we
should. We'd ruin sports if we barred particularly tall, broad-shouldered, strong, explosive males from the NBA out of a concern that they had unfair biological advantages over 5"8 males who really want to play.