Morgan Rogers had Villa ahead before Spurs overran the visitors through goals from Dominic Solanke (2), Brennan Johnson and James Maddison.
Emery said afterwards: "We won two times here and we knew how to win here but they are a good team.
"We had to keep going and try to focus again on our way because in the league the most important thing is to keep the position we are in.
"They scored so quick in the second half. When they scored the second goal it is difficult because their transition is so, so strong. Winning was more difficult. If we score the second goal maybe it is a possibility to win here.
"When you are losing you have to try and get one step forward to try and change the result. But we know how difficult it is against Tottenham. 4-1 is a tough result but the match we played was more or less in line with the expectation I had before the match.
"We are disappointed, frustrated but we are accepting it. We know our way, it is 38 matches, the league is very tight."