Don't know if this qualifies or not, but I enjoy the weekly StackOverflow Podcast.
Joel & Jeff basically just talk about the design decisions, implementation, and problems the encounter while building StackOverflow.com.
It seems unfocused and has a 'water cooler chat' feel to it, but I learn something everytime I listen to it.
I am really surprised IT-Conversations has not been listed. They have the probably best list of interesting podcasts.
They where the ones who spotted companies like Evernote and Etherpad way before anyone else.
I am listening to a Podcast called Advanced Sales, sourced on ITunes. Really good stuff, very actionable for software companies though not aimed squarely at the tech startup scene.
I like listening to "Help, my business sucks!". It's usually entertaining. Andrew definitely tries to drive traffic to some of his paid content, but you do get some nice tips and thoughts from the podcast as well.
Manager-Tools also has some good content on general management issues. And, whether you are a start up or part of a large corporation, it can never hurt to brush up on your management skills.
There are also a lot of great audio books out there. I know they aren't free like most podcasts, but you could check out Jim Collins' Good to Great and How the Mighty Fall. They are easy to listen to and provide some great information.
The Stanford Series that Hendro mentioned has a lot of great options: http://www.stanford.edu/group/edcorner/uploads/podcast/EducatorsCorner.xml - the recent one with Mark Pincus is a must listen.
This podcast from Duke's Entreprenurship 101 series with Aaron Patzer - the founder of Mint - is also a must listen: http://deimos3.apple.com/WebObjects/Core.woa/Browse/new.duke.edu.1738990054?i=1422259961 Mixergy, as mentioned before, aslo has a bunch of good interviews.
My Startup related podcasts come and go, but I really like Evernote 's podcast, it tends to be very product related, but their CEO, COO and VP Marketing are the hosts and you get a lot of raltionale and candid discussions about how they are building their business.
37signals.com podcast is interesting, I love how they are approaching their business.