The Duke has opened up about his family relationships in a candid interview but has warned to it 'too soon' to build bridges