Fans don't stay your fans forever, they lose interest, die or other terrible things can happen to them. This causes a decrease of your fanbase. Fanclub members stay with you the longest, while casual fans lose interest relatively quickly.
Each day you lose 0.5% of your casual fans, 0.25% of your normal fans and 0.08% of your fanclub members.
You shouldn't worry too much about this since you'll get new fans at a much faster rate. This just makes it increasingly harder to get a better popularity.
