[Nyororin]

You most definitely do not take your shoes off in most public buildings...
It is a home or certain restaurant sort of thing. The rule is more "Am I going to be walking on tatami?" - if so, then you will be taking your shoes off either at the door or at the edge of the tatami area. Most public buildings are not tatami, so taking your shoes off is fairly rare.
Another note - even if there are slippers provided, you never wear them on tatami. They are for walking in areas other than tatami, such as hallways, kitchen, etc. You will often see it written that you should always take your shoes off when inside, and that there are separate slippers for the toilet... But it is pretty rare to see the "no slippers on tatami (or carpet when in a private home)" thing mentioned.