I am a small software company and I'm thinking about buying display ads on sites that are relevant to my target audience.
Those ads are usually priced by CPM, so how many impressions do I need to decide if a particular ad/site works for me?
I don't have a large budget and there are many sites I'd like to try, so I can't invest a lot of money per test, does going really low (buying several 1000 impressions for $10, for example) gives any meaningful results? and if not how many impressions do I need before I know this wouldn't work?
You're most common desktop/tablet banner is going to be 728x90 (Leaderboard). The most common mobile size is mostly 300x50 or 320x50. 300x250 can typically target all smartphone/tablet/desktops.
You can gain pretty meaningful data at about $30-50 spend. Since it seems like you already know which sites you want to target, are you buying directly from them? If not, you might want to explore working with a DSP (Demand Side Platform) to give you the ability to buy inventory across thousands of sites and target on various criteria, or purchase intended users for your market. You might want to check out SiteScout or Lean (no affiliation to either site) as they tend to work with non-agency advertisers.
Regarding display ad size:
Like Eric said, ad position is more important. You want above the fold (does not require use o the vertical scroll bar), but you also want an ad that will show up on mobile devices. If the websites you advertise on are already optimized, your ad will be okay. But if not, your ad will be cut off and possibly blurry on a mobile screen. That can be averted by advertising on mobile optimized sites or buying mobile ads.
So ad position may be more important, but the ad size becomes more important when you consider mobile ads.
Ad position is more important than banner size, but assuming you're talking about banners above the fold (ads that the visitor will see without scrolling down) both a 768x80 leaderboard and a 300x250 or 336x280 rectangle will have higher click through rates. The 160x600 skscrapers on the right side of pages are ok, but the leaderboard and rectangle are superior.
As for results and testing, doing 1000 impressions on a bunch of sites will provide no meaningful data. There is a ton of randomness in online advertising and you really need the highest amount of data for the lowest number of options. This is also a tough question to answer because we don't know your market or your goal. Are you trying to gather leads? To make sales? You're also testing two things: the click through rate (CTR) for your ads and the conversion rate of your landing page or website. For both tests, you need lots of data, but the second test is somewhat dependent on the first.
Maybe try this: Get 100 visitors to your landing page. My guess is that if your ads are well constructed and the sites are well targeted you will need to run 3k-6k impressions. You will hopefully have a CTR of 2-3%.
Most reputable self serve ad networks will allow you to view available placements before you fund the account. Take a look at the positions available within your niche/category and you'll get a good idea.
Since you're looking at different ad sizes, don't forget to tag them so that you can later tell which versions are performing best.