Not NYC specifically, but New York Metro area (aka tri-state area). The metro area is a huge job market and people commute from every direction to NYC. Some startups let engineers work remotely from Philadelphia, DC, and Boston area and come up to NYC office on some consistent schedule. Many people commute 2 hours each way from South Jersey and/or Connecticut.
Northern New Jersey has some excellent school districts with school ranking 10 out of 10 and commute to NYC is fairly short. NYC has great schools, but those are very hard to get into.
NYC consists of 5 boroughs (Brooklyn, Queens, Staten Island, Manhattan, and Bronx) with Manhattan being the center and most expensive for office space, but there are many co-sharing spaces for new startups as well. You could setup in outer boroughs at first and then move to Manhattan eventually.
NYC taxes are high for both corporations and individuals, but NY State recently announced a program with tax-free zones for startups, that I have been meaning to look into. People who work in NYC but live outside do not pay NYC tax (around 5%).
Salaries in Manhattan are the highest and are 30-40% lower in the immediate metro area, so some family folks actually look for local jobs to shorten their commute and spend more time with family. Not a lot of high-profile companies in suburbs though, so great local jobs are close to impossible to find for engineers.
It would be best to know someone in the city you are considering a move to, so you can get advice on a variety of things. The change of pace alone could be a good shock to the system.
Some benefits I've heard from startups that moved from other cities to the bay area:
Many startups move to the bay area after launch when they want to scale. For example, Yammer moved from LA to San Francisco.
After Silicon Valley, the next two best options are either New York or Boston.
Webbi and Bruce already gave out excellent answers.
But, if cost was a consideration here are some stats you may want to check out: What does it cost to run a startup?
The infographic shows how much it costs to establish a basic startup (with a designer, developer and an office) in 8 major cities in the world.
It doesn't cover everything. But, it should give you an idea of how location really makes a drastic difference in where you decide to put up your startup.
Hope that helps your decision in any way.