A senior member of Conservative leader Kemi Badenoch's shadow cabinet made the comment