This is a strongly opinionated post, reflect my personal standpoint only.
The point of scaffolding is to get you up and running real quick, which is a good thing, right? Well, not always.
Let's say you are trying to get something done in a fairly new way (this is what we love about javascript development: every month there's a new way to do the same thing). You found the perfect yeoman generator for it, downloaded it down and have half dozen of config files generated for you. Now you are ready to develop the next killer app, except you have no idea which files are needed in the git repo, what changes needed for a production ready config and many more similar questions. Now you have to backtrack whatever template you have, understand why it is there and what does it actually do, and possibly spend more time on it than doing it step by step.
The point is: if you are doing something the first time, it probably worth doing it without scaffolding.
For the nth similar javascript project scaffolding is just great!
