In the context of start-ups it has to be boot strapped to the max yet still ergonomically superior as people are going to be spending crap loads of time at the desk.
Tables - Go Jeff Bezo stlye and use doors as tables. Customize the height using bricks, drill round holes for cables etc.
Seating - Don't think Aeron's are the answers to all your ergo problems. Everyone is different and Aerons are expensive. Do your research and find second hand sturdy adjustable height chairs, customize with air filled cushion and strap on lumber support. Leave a couple of foam rollers in the corner.
Coffee - Don't skimp. Calcanis is a fan of keeping it in house so staff don't leave to often for out of work straysies. Prices are coming down for decent coffee makers, find a product that maximizes coffee use and keep away from the one time use throw away capsule coffee machines.
Stationary - Do not go though big companies like office works or corporate express. Find what you want in a store then go online using e-bay or the a like. It is not worth anyones time to go running errons for stationary.
Desktop Hardware - If you are tech based buy a nvidia personal super computers (second hand). You won't need to upgrade for ever and performance is mind blowing at a great price.
http://www.nvidia.com/object/personal_supercomputing.html Alternatively, only use laptops and use docking stations.
Power Consumption - obviously timers, low energy light bulbs, techniques to maximize light
Water Cooler \ Plants - The first thing that disappears in corporate settings are the spring water and plants. Don't pay for them work a cheaper to system to have these luxuries.
Backup Infra-structure - Pay for unlimited net access and synch on the cloud real time. Obviously keep top secret stuff closer to home.
R & D - Find out everyone who still has their college subscriptions to the libraries resources or other subscriptions from previous employers and pool them so everyone can use them.
OK.... better stop I could go and curious to find out what others recommend.
Don't have an office space - only get one when you can pay for a year in advance with existing revenues.
If you can't afford the power, paper, etc then you shouldn't dream of getting a dedicated office.
Also - don't be penny-wise and pound foolish here. Having people skimp on some of the things mentioned above by cameron could mean loss of productivity.
Use your dorm or home office as an office until you get revenues.
Heating and cooling. You can either set an electronic schedule or manually do it yourself, but turn off the heater/aircon at night. Start it up first thing in the morning. Leave it off on the weekends if nobody is coming in.
Paper. For inter-office use, buy the cheapest bulk copy paper you can. Have a box or bin for re-using paper as well. Just because you've printed on one side, doesn't mean you can't use the other for notes! We used to cut stacks of used paper into 4 smaller squares as cheapo notepads.
Supplies. Find a good reliable local office supply place who gives discounts for bulk. They are usually much cheaper than the outlets like Office depot or Officemax. Oh and buy anything not perishable but required often in bulk. Even coffee.
Labor. Labor can be one of the biggest expenses for any company. If you have hourly paid employees, watch their hours closely. Cut their hours wherever you can, even if it means you have to pick up a phone or two.
Bills. Audit your monthly payouts. You'd be surprised what things sneak in and get paid. Cut out anything you don't need.