ED, THIS IS all you.
Whatever anger, distraction or embarrassment long-rampant speculation about an affair with your beauty-queen employee causes you, your marriage, focus on work or even life after leaving office, it's self-inflicted.
This is not the act of some political enemy. This is not a gotcha moment, despite your comments to Harrisburg's Patriot-News that you've "had it with the media."
No male in high-profile public office, especially one with a history of seen-in-public "friendships" with attractive women, can expect to be seen repeatedly in public with an attractive woman and not fuel eyebrow-raising rumors.
And anyone with a modicum of media savvy understands that, right or wrong, fair or not, the longer that such rumors simmer, the better the odds that they will get reported.