I'm a US citizen, and would like to start an internet company in the US, however I don't live in the US. As my kind of work can be done virtually, a physical presence isn't necessary. Typically when people ask questions about which state to incorporate it, the standard advice is to incorporate in the state that you live in.
However, since I don't live in any state, does it matter? Couldn't I just pick a place like Wyoming that has no state corporate taxes, instead of just going with the state I was born in?
I think you should see a qualified tax adviser (fx a chartered accountant who is specialized in international tax).
The US system by default taxes US citizens of their global income. But there are double taxation agreements with many nations, which may dramatically change the taxation in your situation (depending on where you're living now).
Normally, I don't advise to consider tax planning when doing a startup, but to focus on creating value. However, if you have truly left USA for good, then you may be able to dramatically reduce your tax payments by setting up a company in the 'right' country. So what I'm saying is, in your situation, don't default to setting up your business in the US without thinking -- look at the business requirements and the tax situation first.
No, it "doesn't matter" insofar as restrictions go for most cases***. Yes, you can pick any state (like Wyoming) without restriction aside from registered agent requirements. However, as you suggested, the state you choose can influence the amount you pay in annual fees and state taxes. Be aware that low/no-corporate tax states try to make up for it by charging higher annual renewal fees.
There are also, of course, differences in corporate law between states, if legal protections factor into your decision.
***Although you are living abroad, do you have an official residency in your home state? Some states, like California, consider residency to be a sufficient basis for taxing a company based on where its employees live.
I've incorporated a Delaware corp while not on US soil. US is one of the few countries that make it extremely easy to incorporate, and protect corporations aggressively as an individual entity.
You might find this state-by-state breakdown of limitation of liability interesting: