At 170 pounds, calculators estimate his Baldy effort at 423w for 23 minutes; at 180 pounds, he'd have had to put out 454w.
You may feel you can fit that particular number in your pipe and smoke it without worry, but keep in mind that this 20-minute effort--on flat ground, the kind of effort that would put you at 31+ mph, comes at the end of stage with two prior category 2 climbs, and starts at a point where the field of 130 riders had been whittled down to six.
You and I would have been dropped far, far before Sagan even started his 450w effort. Even Ben Friggin' King lost contact as Sagan started his 450w 20 minute effort (you can see him in the distance here)!
To go the same speed up a 8% climb like Baldy, Sagan had to put out 80-100 watts more than a skinny climber like Joe. We have some data to back this up. In 2012 Joe put out 338w for almost 23 minutes to earn a KOM he held for several years until Ian Boswell took it from him Saturday. Robert Gesink, I'm guessing between 5-10 pounds heavier than Joe, finished about 15 seconds down on Joe, and he averaged 408w on Saturday.
Michele Ferrari once said that, given the choice between doping and a losing weight, a Grand Tour cyclist would be better off losing weight. The world's most notorious doping doctor makes the case that being skinny is more of a performance boost than dope.
For all Sky's talk of advanced training, from what I can tell, their method is simply to take time trial specialists and starve them. That's what the did with Wiggins, Geraint Thomas, and Porte.
Geraint Thomas 2008
Geraint Thomas 2015
Richie Porte (with Contador) 2011
Sagan is just like those guys, except he didn't bother to lose the weight, which leads me to conclude that Sagan could be the greatest stage racer around if he'd drop 20 pounds. He might lose some top-end power, but his threshold power would probably remain close to his current level--if the examples of Wiggins, Porte, and others hold true.