I don't know what the "official" age is where one can be diagnosed with Aspergers with certainty. I'm sorry I can't answer that question for you. Regarding the second aspect of your question:
assuming our suspicion doesn't fade, at which age is it worth it given that there is not much that can be done about it?
My answer to that is as soon as possible.
I disagree with your statement that "not much that can be done about it." True, there is no cure, but that's hardly the same as not having any options.
You should trust your instincts and get your child evaluated. The worst thing that could happen is that it turns out that your child is normal. However, if your child is NOT normal, whether it is because of Asperger or something else, you can begin seeking treatment/support right away. Just because there is no "cure" for Aspergers does not mean you should be sitting around doing nothing about it. There are tons of resources available to help your child and your family get the most out of life no matter what.
I had added this link in a comment (which I deleted in order to post this answer) and I do hope its relevant; if you are in the US it should be. I really think that Early Intervention can probably help you. The site has a lot of good information about how to navigate the various services and options that may be available, and also has lots of good links to other resources.
The advantages of early intervention cannot be overemphasized. Children who receive intensive therapy can make tremendous strides in their overall functioning and go on to lead productive lives.
The above is quoted from the Autism Society's website's screening page, which also mentions that the American Academy of Pediatrics recommends autism (the spectrum of which includes Aspergers) screenings at 18 months of age and again at 24 months. If your doctor hasn't already brought it up, it is NOT unreasonable for you to do so the next time you are there, if you choose to wait until your next scheduled check-up. To prepare for the visit, you can start by keeping notes of the situations where your child's behavior concerns you. Also, here is a questionnaire from the Easter Seals that you can take that may help you identify whether or not your child is on track developmentally.
Finally, if your child is too old for the EI program, your local school district may be able to help you. See if you can find a local outreach organization (usually associated with the local school district), such as Child Find, that does screenings of pre-school aged children. These screenings evaluate children in various areas of development- social, emotional, physical, academic- and usually also for hearing and vision problems.
I am sorry if you are not in the US and all of this is not applicable to you. What I'm trying to say essentially is, no matter where you are, you need to be an advocate for your child. You don't necessarily need to rely on your doctor to get the ball rolling- that is up to you.
I am not a special ed professional, nor am I social worker; I am a mother who felt strongly that her child was a little off when he was very young. I didn't wait for someone to validate or confirm my suspicions. I found a local outreach center and I called them up. It was an intensive process- the evaluations, that is- but it was worth it. My son had a speech delay. He was my first child and I didn't even realize he was that far behind with his language, but the frustration that he felt by not being able to communicate was the source of his other symptoms. He was 2 when he was "diagnosed" and by the time he was 4 he was developmentally equivalent with his peers. I realize that what you are concerned about is a little bit more serious than a speech delay, however, the lesson learned is that the earlier you start, the sooner you see results. And, you can prevent what could be a minor problem from compounding into something much more complex and difficult to resolve/treat.