Realistically speaking, it should not. I am not actually sure if it also relies on host performance pretty much.
I can see that it would affect your performance when you have a shitty host with low specs. Since every time a post is loaded with user posts, it has to call all the user groups the users are in and what not.
I would also assume that these XenForo Developers stress tested their software before they released it, and It would make sense to me if they did test this scenario as well as having 100 addons installed and run at the same time.
So in essence, I can't see why it would make problems, unless you really really have a shitty host.