I like answers from both rbwhitaker and Bruce, BUT technical knowledge alone is only one of the factors, that most people assume is most important. There is a lot to consider.
Here are some factors about the hiring team that are important to define an ideal candidate:
1. Current size of the team. On a small team you need more cross-skilled developers. On a large team you might prefer people who specialize and are good at very specific range of work.
2. What is the current composition of a dev team and ratio of seniors (people who can design solutions), noobs (eager to learn and grow), working bees (people who don't mind maintenance)? What role is missing and/or what pain point needs to be addressed? If you got a bunch of juniors you need a senior person to architect and mentor, if you got a bunch of rockstars you might want someone for maintenance or low skill tasks.
3. Stage of the company and product. Are you hiring someone to work on a core product - building or extending/improving? Or, are you looking to upgrade infrastructure? Knowledge of patterns and practices is one thing, experience with designing and scaling applications comes with practice.
4. Company's ability to compete for top talent in a given local market, based not only on compensation but also brand recognition, location, risk factors (profitability and funding), and growth potential.
Things to evaluate a candidate on besides technical knowledge:
1. Motivational fit and attitude. You could be jumping up and down with joy about your candidate acing technical interview and accepting your offer, only to find out that once inside he/she doesn't ship code at a pace you would hope, perhaps over designing everything for scale that is not yet needed or suggesting to re-write things from scratch because they don't want to extend the current codebase. Or perhaps they do everything explicitly asked of them, but aren't interested in innovation or ownership, and are not proactive with problem-solving (they always bring the problem to their manager to solve).
2. Emotional maturity. Experienced managers can deal with it, but recently promoted engineers might not be able to. Imagine your new rockstar a) bitching about how bad all the code is non-stop instead of diving in and fixing any issue they can get their hands on b) openly criticize your company's strategy and leadership c) blogging about his/her work in a non-flattering way d) takes care of his/her past clients/obligations/projects on your company's time.
A hiring manager should consider all factors above and look for an overall fit. You always want a rockstar if you can find one, but when you can not find enough rockstars to build a team you may choose to hire for attitude and train for skill.
Make them write code in the interview.
Since that's what they'll be doing most of the day, you better make sure they can handle it. Even doing something as simple as fizzbuzz is something far too many people who claim to be programmers can't do, even with a compiler, and IDE, and an hour in front of them.
Then make them discuss code. It's also something they'll be doing a lot about. Have them explain why their version of fizzbuzz is correct, or give them another piece of sample code to discuss. It tells you really quickly whether they understand important fundamentals of programming (like what a variable or loop is) and depending on their responses and comments, you can dig into deeper things to see how deep their understanding is (design patterns, optimization, details of how the language itself functions at a deep level).
Let me be clear: this doesn't make the rockstars stand out from the crowd. It makes the people who are absolutely clueless stand out.
Seeing that you are a non-technical person doing the initial filtering, here are some suggestions for you:
1. Ask them what interests them. Software engineers who will talk passionately about what they love in programming likely have a higher change of actually knowing how to code.
2. Get someone technical in your team to give them these questions over the phone: Five Essential Phone-Screen Questions