Sir Keir Starmer has defended his trip to Brazil amid domestic concerns as he insisted there was a “direct link between the work I do internationally and the impact back at home”.