Walker insists Oliver allowed Arsenal to take a free-kick without allowing him to get back into position after originally calling him over to speak with him.
Walker told Radio 5 Live: "I'm not ever going to come on national TV, national radio and start slagging people off because there's enough of that going on around in the world.
"What I'm saying is I've not gone over to the referee. Me and Bukayo (Saka) have not gone over to the referee - he's called us over.
"So if I'm being called over to the referee, he then should wait and allow me to get back in position before the ball is then gone over my head.
"So if I go over to the referee by my own accord and I'm out of position, it's my fault. But I'm in position, he's called the two captains obviously to calm the players down.
"I'm walking back saying, 'Lads, concentrate, nothing stupid, make sure we get through this'. The ball is then going over my head.
"Me and Mike have a very good relationship and I think he has a very hard job, especially at the top level. So that's not me being critical of him. I know he has a hard enough job, he's got 22 angry players screaming at him continuously.
"If he calls me over, if I go over of my own accord, fair enough. But for him to call both captains over and then not let me get back in position...
"If I was a goalkeeper, does he let me get back in my net? Of course. I'm a defender, I'm the first line of defence he should let me get back in and be set and then blow the whistle."