While most here worry about affordability, neighbor groups ask ‘How Big Is Too Big?’