David Miliband: The nearly man of British politics

"The most important thing for any of us is the people that we love and that is what makes us human," David Miliband told me during the final stages of the leadership campaign he hoped would see him crowned leader of the Labour party.

"That's not something we go around advertising. Now, we then love our football teams, we have our holiday we enjoy, we have our favourite places and favourite people. That's something that comes out in due course.

"I think there's a real danger of underestimating the British public in this. They want to see beneath the surface. And it's important that we respect that. They want consistency and they want clarity. The real person comes from those virtues."

For a man whose private thoughts and public actions have been at the centre of what makes him fascinating for so many years, this was a revealing observation. Miliband was on the brink of being usurped by the ambition of his younger brother – and the decisive influence the trade unions hold over the system the party uses to elect its leaders. He was so close – and yet so far.

The Conservative party is, oddly, the best judge of what David Miliband might have been like as leader of the Labour party. Their peculiar authority in this regard stems from their reaction to Ed Miliband's victory in September 2010. The Tories were, put simply, delighted. David Miliband's closeness to the New Labour government would have helped them, but his obvious political abilities were feared as much as they were admired.

The 2010 leadership upset was a hammer blow. After being usurped by his younger brother, the ensuing party conference in Manchester was dominated by his internal struggles. When Ed declared in his speech he believed the Iraq invasion was "wrong", David turned to Harriet Harman to confront her for her automaton applause. "Why are you clapping? You voted for it," he hissed. It was already becoming clear life in the shadow Cabinet was untenable.

Days later David was appearing in an awful shirt outside his London home acknowledging exactly that. Stories of splits, however unfounded, meant it just wasn't feasible, he claimed, even when they had no grounding. After allowing his bitterness to show, that was a bit rich.

In opposition, the gap between his work from the backbenches and his private thoughts was always going to be hidden from public view. He did a good job convincing those in Westminster he viewed himself as a plausible future big player, remaining active in carefully selected Commons appearances.

"It is hard to stomach a government who take absolutely no responsibility for their mistakes," Miliband said in one of his last appearances in the Commons.

"It is intolerable – [Interruption.] Government members are laughing, but I am ready to say what we did wrong; I have not heard them say a word about what they are doing wrong."

That just about sums up the man chosen by his party's membership and MPs to be their party's leader: a serious figure, frustrated by the ease with which his enemies could laugh at him, and saddled by a mixed record in office.  That regrettable 2008 image of him grinning while clutching a banana seemed to epitomise his inability to make the most of an abortive leadership attack on Gordon Brown. He will be happier across the Atlantic helping make a difference on the world stage.

Happy in the short-term, at least. All of the above applied to William Hague, who had his own traumatic comeuppance in 2001 and took many years to recover. He is now one of Britain's most respected politicians, his stature enhanced by his past life in politics. Miliband is 48 this year, giving him enough time to take a break off and return rejuvenated. Much depends on the fortunes of his younger brother in 2015, but the ambition which fuelled him throughout his career so far could rekindle in time for a return in 2020.

The difference is Hague remained an MP, while Miliband is making a much more decisive break. What makes him human, as he explained to me, is the people he loves. He is refocusing on his immediate family – and leaving his brother to get on with trying to win power in the UK without him. It is a powerful story – and, for now, one with an oddly saddening ending.

All politicians have to create some sort of a barrier between their private and public lives. David Cameron and Gordon Brown, who have both lost children, have dealt with the challenge in very different ways. Miliband has never had to cope with that sort of tragedy, but in his case it proved impossible to create a barrier between the public and private. Ed Miliband was, is and will be his younger brother. As David has known for a long time, he just can't escape that.

"I now have to make a choice about how to give full vent to my ideas and ideals," David wrote in his letter to his local party chair, Alan Donnelly. As on so many occasions previously in his career, we're left wondering: surely there is more to this decision than that.