Families say a local council is making a terrible mistake over its plans for a treasured community space, which is set to become a hub for specialist dementia services.