How does Hubspot get Google statistics?


1

Put your website into Website Grader. The free report they come back with has:

  • Google pagerank
  • Google indexed pages
  • Last Google crawl date
  • Traffic rank (from Alexa)
  • Inbound links (matches neither Alexa nor Google links: operator)

How does Website Grader get this info? Are they screen scraping? Is it possible to get this information under a commercial license?

Do people have experiences to share about using screen-scraped data for their businesses?

Data

asked Jan 16 '10 at 05:43
Blank
Dan
303 points
  • Dharmesh should know that one ;) – Olivier Lalonde 9 years ago
  • Dharmesh, any comment? – Dan 9 years ago

2 Answers


1

Most of this data is easily retrievable using various APIs, ex. Google Ajax Search API.

Furthermore, the links: method is not very accurate, rather the better way is to fetch the indexed pages, number; hint: the one found in Google webmaster tools.

Furthermore, I dont think a user initiated report request would anyways get classified as screen-scraping. though I am in no way qualified to comment on the legal aspects.

Also, perhaps you should take the technical/programming aspect of the question over to Stackoverflow.

Cheers!!

answered Jan 16 '10 at 07:09
Blank
Dheer Gupta
191 points
  • "The implementation of the Service on your Property must be generally accessible to users without charge and must not require a fee-based subscription or other fee-based restricted access." http://code.google.com/apis/ajaxsearch/terms.html. Hmm, so all the Google features they use are free to their end-users. – Dan 9 years ago

0

It's interesting to see how they still get Page Rank. Page rank is no longer available publicly on the API and Google blocked SeoMoz from using Page rank data for their tool.

The rest is available from the various API's publicly. The links on Hubspot's report is from Yahoo Site Explorer.

answered Jan 16 '10 at 10:05
Blank
Romy Misra
71 points
  • Yahoo Site Explorer a) only allows you to get info on your own verified site, b) is limited to noncommercial use. Are they really using YSE? – Dan 9 years ago
  • I should say, what makes you think they are using YSE? – Dan 9 years ago
  • Because the data matches perfectly :) Try it for your self. – Romy Misra 9 years ago
  • So do you think they are illegal, or got a commercial license from Yahoo somehow? – Dan 9 years ago
  • The Yahoo site explorer api is free, anyone can use it! – Romy Misra 9 years ago
  • Romy, I believe the terms of service say limited to noncommercial use. Do you have a ref saying otherwise? – Dan 9 years ago

Your Answer

  • Bold
  • Italic
  • • Bullets
  • 1. Numbers
  • Quote
Not the answer you're looking for? Ask your own question or browse other questions in these topics:

Data