Potter was hired by the Hammers on Thursday to replace Julen Lopetegui, who was sacked earlier this week after just six months in charge.
But the former Chelsea and Brighton boss was unable to make a winning return to the dugout as West Ham squandered the first-half lead given to them by Lucas Paqueta.
Amadou Onana equalised in the second period before Morgan Rogers bagged the match-winner to leave Potter holding his head in frustration.
Potter will also be sweating on the fitness of Niclas Fullkrug, who he was forced into subbing off in the first period.
Aston Villa struggled to get going in the tie, but were vastly improved in the second half and deserving of their win.
That said, West Ham will be left frustrated by referee Tim Robinson's incorrect decision to award a corner in the build-up to Onana's goal.
In tonight's earlier kick-off, Wycombe Wanderers beat Portsmouth 2-0.
Both Villa and Wycombe will now wait to find out who they'll face in the fourth round of the FA Cup.