How to be data-driven startup (and handle spiders from search engines)?


I am curious on how some of you adopt data-driven approach to your startups.

Recently, i did an experiment with simple AB testing tool like A/Bingo: Through that, i found that majority of our traffics are search engine's spider.

Example: instead of 100 visitors with 10% conversion rate (10 users), it become 10000 visitors and 0.001%. 9900 "visitors" are spiders.

How do you handle that? Any tools you would recommend?

It would be great if you have any reference to blog post or article that explain in detail on how to adopt data-driven approach to startups.

PS. Have been reading heaps about Dave McClure AARRR, Eric Ries, Andrew Chen, Mix Panel etc. I am looking for "more specific, with real-world scenario" kind of example.

Thank you.

Analytics Metrics Ab Testing

asked Nov 15 '09 at 13:25
Hendro Wijaya
1,408 points
  • +1 I'd love to learn more about that too. – Slav Ivanov 14 years ago

3 Answers


I think robots.txt works well with "genuine" bots, and evil bots can be handled using htaccess rules.

See robots.txt examples from some top sites -

See this for .htaccess rules -
answered Nov 15 '09 at 19:37
Arpit Tambi
1,050 points
  • Robots.txt. Isn't that also affect my search ranks in Google? – Hendro Wijaya 14 years ago
  • Yeah, they're likely to go up up up. ;) cuz you will be blocking duplicate and canonical pages. – Arpit Tambi 14 years ago


Use a web analytics tool that's Javascript-based -- those aren't triggered by search bots.

Beyond that, you'll have to discard all results from search engines. Maybe Patrick can help make that change to A/Bingo.

P.S. +1 on the robots.txt

answered Nov 16 '09 at 03:05
16,231 points
  • Thanks. I just know that JS-based analytics aren't triggered by search bots. – Hendro Wijaya 14 years ago


Here's what i found recently about adopting data-driven method. it helps articulate the concept a bit:

answered Nov 18 '09 at 11:50
Hendro Wijaya
1,408 points

Your Answer

  • Bold
  • Italic
  • • Bullets
  • 1. Numbers
  • Quote
Not the answer you're looking for? Ask your own question or browse other questions in these topics:

Analytics Metrics Ab Testing