As trees have struggled to get enough water, their fruit has been falling off smaller and earlier.