Seatacus,
Hitler saw the whole of the 1936 Summer Olympics as a means to demonstrate the superiority of Aryan people, not necessarily to promote the Nazi party. While many Nazi leaders, including Hitler, did see the Olympics as an opportunity to promote Nazi ideology, much of that was hidden away during the games. e.g. the removal of anti Jewish signs, suspending the enforcement of discriminatory laws, and the outward acceptance of other races.
Even so, there are many examples of symbols that have represented vastly different things and ideas in different times and places. The torch relay was originally thought up by Nazi leaders as a grandiose means to open the games, yet it has been used at each of the 30 games since then, and always as a tie-in with the ancient games, not as a symbol promoting Nazism.
BTW, the 1936 Summer Olympics was attended by athletes representing 49 countries; more than any previous games. The 1932 summer games, for example, only had athletes from 37 countries.
The Olympics did see a massive boycott, of sorts, in 1940. Those games were supposed to be held in Tokyo, Japan, but the IOC retracted the award and gave it to the second place selection, Helsinki, Finland.