Prince Philip, the Duke of Edinburgh, died on Friday, nearly four years after officially retiring from public duties in August 2017. He was 99.

He had been married to Queen Elizabeth II for more than 70 years, since the two wed at Westminster Abbey on Nov. 20, 1947. To outsiders, it looked as if they had the perfect romance, between their fairy tale wedding—the first British royal wedding to be broadcast live—and their shared love for the Rodgers and Hammerstein musical Oklahoma! (especially the song “People Will Say We’re in Love”).

On the occasion of their 50th anniversary in 1997, the Queen noted in a speech that he had been an often-hidden force in keeping her steady at the helm for half a century:

Yesterday I listened as Prince Philip spoke at the Guildhall, and I then proposed our host’s health. Today the roles are reversed.

All too often, I fear, Prince Philip has had to listen to me speaking. Frequently we have discussed my intended speech beforehand and, as you will imagine, his views have been expressed in a forthright manner.

He is someone who doesn’t take easily to compliments but he has, quite simply, been my strength and stay all these years, and I, and his whole family, and this and many other countries, owe him a debt greater than he would ever claim, or we shall ever know.

Princess Elizabeth and Prince Philip walk during their honeymoon at Broadlands, Romsey, Hampshire, Nov. 24, 1947.Princess Elizabeth and Philip Mountbatten, Duke of Edinburgh, walk during their honeymoon at Broadlands, Romsey, Hampshire, Nov. 24, 1947.Topical Press Agency/Getty Images

But royal insiders knew that the marriage wasn’t quite as easy to keep up as some might have thought. That side of the story has more recently been depicted on the hit Netflix series The Crown. In the fictional show’s first season, the actor Matt Smith depicts the royal as struggling to play second fiddle to his more high-profile wife after she becomes the Queen of England. TIME’s Oct. 21, 1957, cover story on Prince Philip also noted that Elizabeth’s family scoffed at his less well-off upbringing—though it was actually a boon to the family from a public relations perspective because he was seen as the more relatable one.

In 2016, in light of the show’s debut, TIME explained what to know about how their marriage began: “Philip met Elizabeth, his third cousin, when they were children. From the time she was 13, she was besotted…She never fell for another man, though he had other relationships while she grew into adulthood. As suggested in the first episode of The Crown, Elizabeth’s family did not approve of the match.”

Royal Couple At BalmoralQueen Elizabeth II and Prince Philip at Balmoral, Scotland, 1972.

When the show returned for its second season, TIME took a deeper look at one of the tensest times in their relationship — and how perhaps their marriage was stronger than the show suggested:

Yes, Philip did embark on a solo royal tour (described in the show as a “five-month stag do,” the British equivalent of a bachelor party) of commonwealth countries including New Guinea, Ceylon (now known as Sri Lanka) and the Malay Peninsula in 1956, nine years after he married the Queen.

It was a trying time for the pair’s relationship, particularly as their only communication during Philip’s lengthy period away from England was by telegram, letter and the occasional hard-to-hear telephone conversation, as the show portrays.

…The show suggests that Philip continued to be unfaithful during the royal tour, with encouragement from his fellow travelers. Most of this egging-on comes from his right-hand man, lieutenant commander Michael Parker, whose wife files for divorce from him once she gains evidence of his adulterous nature. (This bit is true: according to Tim Heald’s The Duke: A Portrait of Prince Philip, Eileen Parker did sue her husband for divorce while he was away.)

But despite rumors of Philip’s infidelity, with his name linked to women like the writer Daphne du Maurier and the cabaret star Helene Cordet, there is no evidence to prove that he ever did have an affair.