Actually, crossdressing is closely linked to the concept of the masculine identity and how it is obtained, in my opinion.
Consider the concept of a "tomboy" as opposed to a "girly boy". While a girl who indulges in masculine activities may not be cheered on completely, this tendency is generally portrayed as something cute, or even a form of female empowerment. On the other hand, the man who takes on femenine activities is generally shunned unequivocally, and will have to reject the notion that what he is doing is effeminate or lose his masculine identity. In the media, the effeminate man is portrayed as the Hollywood gay stereotype; an overly sensitive, loud queen who most likely works as a hairdresser.
Taking upon a femenine identity is seen tacitly as "taking a step down", while taking on a masculine identity is seen as a "step up" (although that step can never really be taken by women). This is also why the concept of women crossdressing is generally not seen as nearly as striking or as offensive as the concept of men crossdressing, which, if anything, is usually played for laughs, and any man who crossdresses seriously instantly loses his masculine identity. So the 'set' of activities, customs, and clothing related to women is not only separated from those that are related to men, but the 'set' reserved for men is also seen as superior. If a man takes upon a woman's 'set' of activities, customs and clothing, society is almost asking him, "Why would you do that?", while if a woman does the same with the 'set' of a man, it's a kind of "Aww, that's adorable. You go, girl, although you'll never really get there" response. The concept of a male-dominated world isn't exactly new to anyone, but thinking of the male crossdressing stigma under that light is pretty revealing.
Your basic sociology course will tell you that the male identity is something that is constantly challenged and needs to be proven, while the female identity is something that is almost a given. This is related to a male's necessity to separate himself from his mother if he is to become a man, while a female can look up to her mother as a role model. The phenomenon is related to the idea that "girls mature faster than boys" in school; girls already have a "default" identity which they can be comfortable with, while boys have to earn theirs through trials of strength, courage, robbing some girl's panties and what have you.
This fleeting and fragile quality of the male identity obviously relates to why men simply cannot crossdress. It is an immediate surrender of their standing as men and relegates them as femenine. On the other hand, women already have a 'default' identity; they don't have to fear falling into the other side unless they actively try to be more manly. This, and the whole Women's Liberation movement in part, have contributed to the ability of women to trascend gender roles with greater ease and comfort than men, even in a patriarchal society.
And to a lesser degree there's the obvious aesthetic difficulties. Women's clothing is typically revealing in places where men's clothing is not; the legs, the armpits, the chest, all places in which men differ drastically from women, in part due to biology and in part due to socially-imposed practices such as shaving body hair. Seeing a man expose these features, which are generally concealed by men's clothing, is, needless to say, shocking.
On the other hand men's clothing tends to cover more and be more subdued and reserved, for which the woman who dresses as a man is in a way "toning it down", and if anything covering herself up more than usual, for which it isn't nearly as striking.
