Sunderland were undone by a quick free kick in the dying seconds of the game as a tense finale ended in heartbreak for the Black Cat supporters.
They have now lost seven times in a row at the national stadium.
The Black Cats had taken the lead after beeing handed an almighty gift just four minutes into the game.
Both sides looked evenly matched with Lee Bowyer sticking with the 3-5-2 that had delivered a victory over Doncaster Rovers in the semi final.
Centre-back Naby Sarr was under no pressure as he collected the ball on the left flank, but surprised his goalkeeper Dillon Phillips by playing the ball back to him.
It should have still been routine for Phillips, but he missed the ball as he attempted to gather with his feet and Sunderland had a shock lead.
Bowyer's side were visibly rattled by that blow, coughing up possession and looking remarkably nervy.
Even an early injury to Max Power, which saw Lewis Morgan introduced inside ten mintues, did not disrupt the flow of the game.
Sunderland went close to a second when a cross was cleared just to the edge of the area, where Grant Leadbitter rifled a first-time effort towards the bottom corner. Phillips did well to get down and turn it wide.
Ross and his side were just unable to keep exerting that early pressure, Charlton slowly but surely starting to control possession and build patient attcks from the back.
Their first effort of note came 26 minutes in when an early ball across the pitch found Lyle Taylor free at the back post, the striker taking the effort on the bounce and firing over.
It was a warning sign for Sunderland and they were punished eight minutes later when Taylor combined neatly with Joe Aribo on the edge of the box. Taylor fired a low cross into a dangerous area and Ben Purrington, overlapping from the wing-back position, had an easy finish.
Potential investor Mark Campbell watches Sunderland at Wembley alongside Stewart Donald - report
Sunderland steadied as the half came to a close, and at the beginning of the second periof Bowyer was forced to withdraw Sarr after he collected a booking for fouling George Honeyman.
The key for the Black Cats was to regain control of a game being largely dictated by Aribo and the impressive Josh Cullen in midfield.
Charlton switched to a back four and Sunderland started brightly, a clever move from the back ending when Morgan was just unable to control a cross from Maguire.
They had to survive a tricky spell with ten men as Luke O'Nien received stitches for a head wound, the right-back returning with a bandage as the game entered the final half hour.
Both managers played their attacking trump card, with Jonny Williams and Aiden McGeady entering the fray.
Williams immediately began to cause problems, drawing a yellow card from O'Nien as he glided down the right flank.
The Welshman was a constant threat on the ball but Sunderland were beginning to dominate territory, Grigg increasingly influential and teeing McGeady up to fire over with five minutes to go.
The game looked certain to go to extra time but the Black Cats were stunned when Cullen took a quick free kick.
Charlton got two bodies free at the back post and after the initial effort was blocked, Patrick Bauer swooped to seal the win.
Sunderland XI: McLaughlin; O'Nien, Ozturk, Flanagan, Oviedo; Leadbitter, Cattermole, Power (Morgan, 8), Honeyman, Maguire (Grigg, 56); Wyke (McGeady, 72)
Subs: Stryjek, Matthews, Gooch, Dunne
Charlton Athletic XI: Phillips; Dijksteel, Bauer, Sarr (Pearce, 45), Purrington; Bielik, Cullen, Pratley (Williams, 71), Aribo; Parker, Taylor
Subs: Maxwell, Forster-Caskey, Reeves, Solly, Lapslie
Bookings: Sarr, 43 Grigg, 64 O'Nien, 72 Leadbitter, 82 Flanagan, 90
Attendance: 76, 155