Kompany moved to Bayern this summer from Burnley, though also was former Bayern coach Guardiola's Manchester City captain.
Rummenigge told Kicker: "Max Eberl (the sporting director) asked me to talk to him because I am very close friends with Pep.
"I told Pep that I needed an analysis that was authentic and honest. We then spoke on the phone for almost two hours. To make a long story short, he said that he was 100 percent convinced that Vincent Kompany was absolutely the right coach for Bayern.
"Sometimes fate helps you. The number of rejections that Max Eberl unfortunately had to deal with meant that we may have gotten exactly the coach we needed for this team.
"There were two decisive factors: the team likes the style of football, and so do the fans. Football should be fun. This is exactly why children play in the street. And you can see that in our team too."