The Eagles secured their spot with a 2-0 win over Doncaster, courtesy of goals from Daniel Munoz and Justin Devenny.
While shifting focus to the Premier League and Saturday’s match against Everton, Glasner stressed that his side “have to win” when Millwall visit Selhurst Park in March.
“I am told it will be a big derby,” he told BBC Sport.
“The game happened a few years ago? But it is good playing at home and it will be a great game for our fans.
“They will be very aggressive but playing at home to a Championship side means we have to win.
“I want to go to Wembley."
“Adam Wharton played 60 minutes after a long injury – he was excellent,” he added.
“It was planned for him to 45 minutes but he did so great and felt well, so we did 60.
“Even then, he wanted to play more. Ben Chiwell had his debut and Romain Esse had more minutes.
“It’s important for them. I’m really pleased too that Matheus Franca got a few minutes after nine or 10 months of being injured.
“Winning 2-0, having a clean sheet and with all this individual positive aspect, I’m really pleased with this evening.”