I don't have first hand experience with having a startup in Silicon Valley, but I would imagine that:
1. Local talent is really hard to find and hire, given the competition, so you better have at least a founding team locked.
2. Cost of living is probably higher than in many other places, so bootstrapping could be a challenge, especially for founders with families.
3. You are surrounded by a lot of self-made millionaires and serial entrepreneurs, so as a first time founder you could feel intimidated or at least have some self-confidence issues.
4. I am under impression dating is really hard in the area as you sort of have to find someone in a similar stage of success.
That said, there are so many advantages that absolutely have to be considered as well.
Here are some: