A majority of federal grants handed out as part of an invitation-only scheme to upgrade community infrastructure was directed to Labor seats, a report has found.