I've long been of the opinion that the core assets of a software company (e.g. source code and any system that touches it, continuous integration environments, internal wikis containing design/planning docs, etc. etc) should always be closely-held. In other words, hosted in-house, maintained by employees, and not managed by outsourced vendors.
However, I'm starting to revisit this line of thinking. Some counterarguments, in no particular order:
EDIT: Just FYI, by "cloud/outside hosts", I mean operations like Amazon AWS, Rackspace, et al.; decidedly not second or third tier hosts, startups, mom-and-pops, etc.
We are a very successful Silicon Valley startup and we don't have a single server on our office network. Everything is in the cloud. Gmail and Google Apps, bug tracking, SVN hosting, dev/qa servers, CRM, telecom...everything. The fact is, your secrets probably aren't very valuable and a snapshot of your company is of little use. Companies create value over the long term by having a great team execute on good ideas on a continuous basis.
The key to answering this question is that as a startup, you don't have the resources to innovate on a non-functional concern.
Outsource what is not your core competency to someone who does it better, cheaper, etc.
The truth is that Amazon, Google, etc. are not going away any time soon, if ever. You're data, email, etc. is likely much safer there than it would be on your private network which probably has a wireless access point that could be hacked by other close by companies/employees.
The most important thing here besides security is continuity and the cloud can offer that to you, cheaply.
Off the top of my head, I can think of 4 approaches to hosting.
For a startup #4 will never be as reliable as either 1 or 2. Not by an order of magnitude. Do you really want to deal with servers, UPSs, connectivity from multiple vendors, etc, when you can get a small instance from ec2 for $72/month?
Looking at #1 and #2, they're really equivalent. It's a question of who you let access your data. Overall, I believe it's certainly worth it. We're fully hosted on ec2/s3.
I find mysef to be pretty trusting when it comes to data passing over other networks through the cloud: I really don't think that a nosey sysadmin on their end will snoop our data and, if they do, it will take a lot of snooping before they find any "juicy" stuff.
However, I don't trust the cloud to keep our data safe. I hear story after story of cloud failures, insane theft stories (Ocean's 11-style ), floods, etc. And then there's the fact that a cloud provider could just all of a sudden shut down. No warning, just gone.
That scares me. We're a sotware company and spent hundreds of thousands of dollars to develop soft assets. I trust a federally-regulated and insured bank with my assets, but not Google or any other company with "we can change it whenever we feel like it" terms of service which state their not liable for loss, blah blah blah...
But I do believe that both our in-house apps and the cloud will notfail simultaneously. Whatever we put in the cloud, we have a redudant copy in the office. Exchange server, local mailbox. Local back-ups that go to the cloud for backup. Etc.
Yes, it costs more in terms of time... but it's worth it to know the data is safer.
Cloud hosting and data center companies are professional companies that earn their bread-n-butter through providing these services. "For a startup" , It is safer to host the data at one of these companies than to manage it yourself on counts of uptime, cost efficiency , reliability, flexibility and even data security. I also agree with Alex that a redundant copy of the same needs to be maintained locally as well. The beauty and bane of digital data is that it can be copied easily and backups maintained.
If you are worried that some of your data is proprietary then you can perhaps encrypt it, but the decryption would be on the software in the cloud so it is only limited protection.
As more people move to doing software in the cloud then it will be harder for the companies to really be able to mine the data as easily.
But, if you are really concerned then just host it yourself.
We are looking at putting the software in the cloud mainly for uptime, as we are developing software to be used by safety engineers, and if they need to make a change while the crane is waiting, they can't wait for a computer to reboot.
So, the convenience is more important than what they may learn from the database.
All good advice... especially regarding backups & encryption.
I'll just add a point regarding security:
Enforce secure connections. Most systems can be set this way, to require https: and SFTP, for example. This is even a good idea for things like your blog platform control panel and other such "non-critical" resources.
If they are holding your data/code, you need some agreement where they can place it in escrow and agree to send you backups on DVD's if they go out of business.