Colchester are competing at this level for the first time in their history, and they were left wondering how they didn't secure at least a point from their first ever Championship home game.
The U's beat keeper Luke McCormick on no fewer than three occasions, only for them to strike the woodwork once and to be thwarted twice by goal line clearances.
And to rub salt into the wounds, Plymouth's rookie midfielder Summerfield, who was making his full debut, came up trumps with a spectacular goal to win the game in the 30th minute.
Wayne Brown could only half clear the cross from David Norris with his head, but there still seemed little danger as the ball bounced into the path of Summerfield who was standing 22 yards from goal.
The 18-year-old tried his luck with an instinctive half volley which gave home keeper Aidan Davison no chance as it flew into the far corner of the net.
Otherwise, Plymouth did not really threaten Colchester's goal, while at the other end they had to ride their luck.
Even so, Ian Holloway's men are now unbeaten in two matches, following Saturday's home draw against Wolves.
Argyle were thanking their lucky stars after only four minutes when Richard Garcia, who had scored in the 2-1 defeat at Birmingham on Saturday, struck the woodwork with a thunderbolt.
The first goal line clearance occurred in the 13th minute. McCormick was in no-man's land as U's skipper Karl Duguid beat him with a high lob that would have dropped into the net if it wasn't for the quick reactions of defender Hasney Aljofree.
Against the run of play, Summerfield blasted Plymouth ahead with his superb goal just before the half hour mark, but this didn't knock Colchester out of their stride.
In the last minute of the first half, Jamie Cureton was poised to celebrate his first goal of the season, only to look on in despair as midfielder Paul Wotton nodded off the line with McCormick again beaten.
Colchester, who clinched promotion from League One last season, did not have quite so many chances in the second half although substitute Chris Iwelumo nearly grabbed an equaliser within a minute of coming on the pitch.
Scotsman Iwelumo was just a whisker away from diverting home a dangerous cross from the excellent Garcia from deep inside the six-yard box.