First, I think it is worth noting that gender identity and gender roles are not the same as sexual orientation--liking girly things is not the same as being gay.
As far as your specific issues go, I agree with Rhea that these are not big signs that your son is experimenting with an alternate gender identity. Hanging out with girls, using female avatars in video games, and playing volleyball are not unusual behaviors in heterosexual men.
But I'll assume that there is something more going on here. Even so, adolescents and teens sometimes experiment with different roles, just to see where they fit in. Your son knows what things are girly and what things aren't, what he doesn't know is how he thinks and feels about everything in the world. He needs space and time to figure this stuff out.
He also might make fashion choices--for example, wearing makeup or nail polish--that are more about what music he likes than about gender identity. Punk fashion involved makeup, nail polish, and skirts for guys when I was in high school, and I'm sure there's some kind of fashion statement like that now.
The important thing is that he needs to feel comfortable talking to you about whatever is going on in his life. If his behavior isn't dangerous to himself or others, and if he is still doing the things he needs to do like chores and homework, try just leaving him alone about this sort of thing. The important thing is that he needs to feel comfortable talking to you... if he is being bullied at school, for example, you don't want him to think that you'll blame him for acting "girly". You want him to trust you so that he'll tell you if he's having problems. Making him feel like you think he is gay–and that you think that's a bad thing–because he wants a diary is probably going to make it harder for him to talk to you about whatever is going on in his head.
Of course it is possible that your son is gay. It's also possible that one of his siblings is gay and he is not. Being gay is not a lifestyle choice or the result of a persons upbringing, it's a sexual orientation, something people are born with. Stereotypes aside, gay men are not always effeminate, gay women are not always butch, effeminate men are not always gay, butch women are not always lesbians. You truly can't judge a book by its cover.
But if one of your children is gay, the best thing you can do is to accept them as they are. This is difficult for some parents, so there are organizations out there that can help you adjust, notably PFLAG (Parents, Families, and Friends of Lesbians and Gays).
Overall, I wouldn't worry about this too much. As parents, the best we can hope for is to raise children who turn into happy adults with their own place in the world. Keep an open dialog and focus on things that matter.