How does Hubspot get Google statistics?


Put your website into Website Grader. The free report they come back with has:

  • Google pagerank
  • Google indexed pages
  • Last Google crawl date
  • Traffic rank (from Alexa)
  • Inbound links (matches neither Alexa nor Google links: operator)

How does Website Grader get this info? Are they screen scraping? Is it possible to get this information under a commercial license?

Do people have experiences to share about using screen-scraped data for their businesses?


asked Jan 16 '10 at 05:43
303 points
  • Dharmesh should know that one ;) – Olivier Lalonde 14 years ago
  • Dharmesh, any comment? – Dan 14 years ago

2 Answers


Most of this data is easily retrievable using various APIs, ex. Google Ajax Search API.

Furthermore, the links: method is not very accurate, rather the better way is to fetch the indexed pages, number; hint: the one found in Google webmaster tools.

Furthermore, I dont think a user initiated report request would anyways get classified as screen-scraping. though I am in no way qualified to comment on the legal aspects.

Also, perhaps you should take the technical/programming aspect of the question over to Stackoverflow.


answered Jan 16 '10 at 07:09
Dheer Gupta
191 points
  • "The implementation of the Service on your Property must be generally accessible to users without charge and must not require a fee-based subscription or other fee-based restricted access." Hmm, so all the Google features they use are free to their end-users. – Dan 14 years ago


It's interesting to see how they still get Page Rank. Page rank is no longer available publicly on the API and Google blocked SeoMoz from using Page rank data for their tool.

The rest is available from the various API's publicly. The links on Hubspot's report is from Yahoo Site Explorer.

answered Jan 16 '10 at 10:05
Romy Misra
71 points
  • Yahoo Site Explorer a) only allows you to get info on your own verified site, b) is limited to noncommercial use. Are they really using YSE? – Dan 14 years ago
  • I should say, what makes you think they are using YSE? – Dan 14 years ago
  • Because the data matches perfectly :) Try it for your self. – Romy Misra 14 years ago
  • So do you think they are illegal, or got a commercial license from Yahoo somehow? – Dan 14 years ago
  • The Yahoo site explorer api is free, anyone can use it! – Romy Misra 14 years ago
  • Romy, I believe the terms of service say limited to noncommercial use. Do you have a ref saying otherwise? – Dan 14 years ago

Your Answer

  • Bold
  • Italic
  • • Bullets
  • 1. Numbers
  • Quote
Not the answer you're looking for? Ask your own question or browse other questions in these topics: