Google Ad campaigns have an option to block an IP address. I don't recall the specifics, but these instructions should be valid.
By excluding their IP address, your competitors will quit seeing your ads. It will appear to them as though you've quit advertising. They'd still be able to get to your site directly, but won't cost you in PPC ads.
Keep in mind they're probably not doing it maliciously. I mean, maybe they are, but the most logical explanation is that they're on the lookout for competition and constantly search keywords that matter to them, and click on ads to see what their competition is up to. Most likely they aren't even aware (at the very least, don't care) that it's costing you a little every time they do so.