Do you need permission to scrape a site and build a mashup?


Should we contact any site we want to scrape to build our market intelligence app?

No company would say yes if we were to contact them.

While Google obviously doesn't ask for permission while scraping (indexing) the entire web.

Any suggestions?

Legal Scraping

asked Mar 1 '14 at 20:33
Michal Harper
25 points
Get up to $750K in working capital to finance your business: Clarify Capital Business Loans

2 Answers


I think its a stretch to say that because google does it, its okay for you to do so.

Check what the target site(s) have in regard to copyright notices / data reuse notices. Perhaps they have an api. IF you are not going to compete against them, why not provide an avenue where you promote their site as a data provider / provide links back to their source? Simply "do it and apologize later" IS a strategy, but it comes with risks.

answered Mar 1 '14 at 21:14
Jim Galley
9,952 points


Scraping is almost always illegal when collected info is republished identically (styling doesn't save you here). Even so, it is rarely prosecuted - but the frequency is increasing.

There have recently been a number of law suits by companies who were the 'victim' of scraping. They have been able to win large sums based on the "temporary or partial thievery of datalines, servers, administrative hours and other resources". This as opposed to IP rights.

Instead of scraping, look for sites which allow you to download RSS, JSON or other feeds or public APIs. You can update your local copy of the feeds based on date to avoid taxing their servers; be sure to check their terms of use for proper attribution.

answered Mar 2 '14 at 02:15
Garet Claborn
324 points

Your Answer

  • Bold
  • Italic
  • • Bullets
  • 1. Numbers
  • Quote
Not the answer you're looking for? Ask your own question or browse other questions in these topics:

Legal Scraping