West Ham had an early lead through Lucas Paqueta before Villa struck in the second-half through Amadou Onana and Morgan Rogers.
Potter said, "The first half was really good.
"We scored a good goal, we were organised defensively, showed a great attitude, togetherness on the pitch, good communication, good understanding of what we're trying to do, carried a threat and were probably a little bit unlucky not to get a second goal.
"Aston Villa certainly didn't carry any threat in the first half. Obviously losing Niclas was a blow, and then Cree as well at half-time, probably affected our attacking intent.
"Credit to Aston Villa, they stepped up as well I think and pushed us back probably a little bit more than we'd have liked, but the boys dug in.
"We were a little bit unlucky I think, with the manner of their first goal in terms of it didn't look like it was a corner and it's one of those things. But after going behind, we still pushed and you could see the personality of the players, and you could see what they were trying to do and we had a couple of chances at the end.
"So we're disappointed to go out, disappointed to lose, but we'll have to carry the positives forward.
"Ollie Scarles came in for his first domestic start and I thought he was really, really good.
"He was playing at a difficult place and dealing with Leon Bailey and the overload down that side, so there were lots of positives.
"I couldn't have asked for anything more from the preparation of the players, how they applied themselves, how they were open to the ideas and how they tried to carry them out was really, really good.
"It was just some executions that of course we're going to get better with but, like I said, overall I think there were positives to the game.
"It just doesn't feel very nice at the moment because we're out and we've lost, but we have to stick to the principles, think about those and dust ourselves down and prepare for Fulham, which is a big game."