Your son likes picking out his clothes. I can't see that as a gender-defining issue. One of my (now married) sons had such a great eye for arranging flowers in the most interesting and imaginative ways that I would sometimes send him to the garden with a pair of clippers and a free hand (something my other children didn't get (blush)!

Playing with a wide variety of toys without pressure has been shown to foster creativity, being pressured to play with gender conforming toys alone inhibits creativity.

The real problem here, as you know, is the father's negative attitude towards his son. If he doesn't have visitation rights, you can specify that he can see his son only when you are present, so that you can see how the father acts towards your son. Having a number of such visits might let you make a more informed decision on how much of a roll you want to allow the father to play in your son's life. Having a father play a role is important, if it's not a destructive role. Only you are in a position to decide that.