Vardy, 38, has announced he will leave Leicester at the end of the season.
Ramsdale told talkSPORT: "Wow, well he's my biggest nightmare.
"He's closing me down from a pass back. I’m like ‘what are you doing?’ 92nd minute, 4-0 down!"
"Vardy scrambles your head!"
He continued: "I remember the first time I played against him we worked on balls over the top for his pace. Ball over the top takes the touch lobs me, horrible. I'm like ‘oh my God’.
"So don't get me wrong I've faced strikers who can probably finish a bit better than him but he has a lot of things which go on where it makes your head scrambled and I'm guessing because he's a natural goal scorer.
"From a goalie's perspective you don't really know what he's gonna do. He could chip, he could sprint with the ball, he could just do anything. Horrible.
"Go around you, chip you, take it first time. Creates fouls for his team, runs in behind, runs all day long."
"Always gives you a bit"
Ramsdale added: "He scored a penalty against me at the start of the season.
"I've always had good battles against him. He's always someone as well on the football pitch who gives you a little bit.
"You have some good chats with him and it's not always banter. You can just have a chat with him and see he always has time for people."