Open your Preferences (Ctrl-P) and go to Prefs > Chat > Chat Windows. Find "Automatically ignore and leave all conference (ad-hoc) chats" near the bottom of the tab. Enable it!
If you'd still like to receive conference chats from friends and only ignore those from nonfriends, then also enable "Don't ignore conference chats invitations from my friends."