I understand that it is accepted as dogma that a small business needs to have a niche to be successful (in my particular case, with respect to web design).
However, I've only read "hand-waving"/empirical arguments that explain why this is so. Basically, every explanation boils down to, "You need to have a niche because small businesses that have a niche are successful, and those that don't have one, are not".
Could someone please provide a more logical, rationally justified explanation? Some deductive logic would be perfect, if possible.
You need a niche market if you're in a hyper competitive industry because as you can imagine, the more competition there is, the harder it is to get a share of the market. And the smaller the niche, the lower the competition.
The second solution is simply to have a better marketing than all the competition, this way you can steal the leadership. (see Dominos' pizza)
The third solution is to create a new category. This way you can become the leader of that category. (see Dyson vacuum cleaner)
Niche strategy is not a dogma, it's simply one of generic strategies described and popularized by Michael Porter (see "Competitive Strategy: Techniques for analyzing industries and competitors", 1980).
Porter argues that to make a profit in the competitive marketplace you either need to be able to charge higher prices without incurring higher costs (differentiation strategy), or have lower costs than your competitors while charging the same prices (low cost strategy).
Both of these generic strategies could be applied to a broad market or to a particular segment (a niche). If you can have a lower costs or be able to differentiate your services within the broadly defined market, then why limit your scope to a niche? A small slice of a bigger pie (in your terms) could be bigger or more accessible than the big slice of a very small pie.
Usually, it may be easier to have a differentiation or cost advantage in a narrowly defined niche (especially for a small business). That said, there are small businesses which also target broader market without specialization on the needs of a particular niche. In fact, web design could be used as an example of appealing to a wider audience.
Another reason for focusing on a niche market is a marketing one. If you marketing strategy depends on the referrals, then a referral within the same niche segment is more likely to result in new sale than referral across the segment boundary (see Moore "Crossing the Chasm", 1991). This is particularly important when introducing new technology or a product as you want to dominate in at least one sufficiently large niche market before moving on to another one. But it's just as applicable for any business which growth depends on a word of mouth advertising.