I see, so i noticed this, i went into alerts and opened it and saw that it have some unread alerts. Than i marked all alerts as read and that solved the issue for me, no more those alert numbers showing up and going.
So with this i think, this is what happening, when the forum is opened, it checks for the unread alerts and shows them over there. Than through JavaScript xenforo checks if the alert button is opened after those alerts, than if it was opened, than through JavaScript it's removing the old unread alert notification.
As of now the solution seems to be marking all notifications as read.