This is two questions in one, but they're so closely related that they ought to be addressed together.
A. If I were founding a non-technological start up -- say a financial services group -- how would I go about setting up the IT side of things? As it happens, I'm a comp-sci major, and I'm far from computer ignorant. But that's beside the point-- unless I were a true guru, I still most likely wouldn't know the full range of technologies needed and how to set them up as a cohesive corporate framework -- company email, databases and servers, company website and its interaction with the servers and databases et cetera, security -- the list goes on.
When someone is starting a company that's going to need all that stuff (as many companies do), and they are computer ignorant, how do they manage it? They wouldn't even know who to hire, because they wouldn't know what skills they need. How is this done?
B. If I were starting an entirely technologically-oriented company, i.e. one which operates around a website -- let's imagine an eBay imitator -- how do I scale? For instance, Google was started by two people out of a garage with a server or two. And now obviously they have dozens of warehouses' worth of servers and databases. What's a good place to start? Just set up a personal MySQL server and work with that until you outgrow it, then hire professionals with the necessary skills to scale you up without losing any information or compromising security along the way?
B.) You almost had an answer in your question.
The cost for technology for a startup is initially high compared to your revenue. But, as you grow the cost of it will considerably go down compared to other expenses. So concentrate on your growth. If your idea is an instant hit, you can go to option 2.) immediately.
A) One hires a local PC / tech services company. Depending on how many people / technical savvy on staff, setup / training / on call service agreements are signed to keep business flowing.
B) How to scale is the last question one should ask when starting out. More important is validating what your product / market / opportunity is, then addressing the scaling issue when it becomes necessary. Yes, you need to make sensible choices at first, but typically what you start with to support 10's of customers isn't remotely similar to what you deploy to support 10M customers. Cash enables you to consider alternative approaches unavailable during lean times.
A) I would employ a "IT/Business Analyst" type person to help. To find them I would search for companies that help you set up Google Apps for email, docs, etc. That would give you someone that understands using cloud technologies and services for businesses. They likely would then be able to help in other areas like that too (eg. accounting tools, CRM, etc.)
As for how people find them, I don't think they do. That's why so many companies have terrible IT infrastructure and systems.
B) Use cloud services to host your application/website. Try Amazon Web Services for example, or a fully managed service like Heroku. This way you can "flick a switch" to increase capacity. In practice it's not as simple as "flick a switch", but it's as close as you get.
Those other two answers are the correct answers but not what happens on Day One.
Usually when someone gets an office space, there are one of two things that happen:
Once things become a mess and/or there's a major failure, they hire the local IT/business services group to do the right thing.