Al-Qasas (The Story)
Chapter 28: Verse 50 (partial)
Physical action depends on one's capacity. When the desire and repugnance of the heart are complete and perfect, and the creature acts upon them to the best of his ability, he receives the reward due to one who performs perfectly. For there are some whose love, hatred, desire and repugnance are purely personal feelings, not in accordance with the love and hatred of God and His Messenger.
The basis of capriciousness is selfish love and hate. This in itself is not blameworthy, since it is beyond our control. It is blameworthy to act upon it. Love and hate lead to the acquiring of tastes when their objects are present, to passion, desire and so on. To follow these without the sanction of God and His Messenger is to follow one's whim without guidance from God. Worse, things may escalate to the point where one makes a god of one's whim. Capriciousness in religious matters is more serious than following the whims of the flesh, since the former is the condition of the unbelievers.
This is why those who depart from the requirements of the Quran and Sunnah - whether scholars or ordinary people - are classed as People of Caprice, just as they were named by the early believers. For whoever does not follow knowledge follows his own whim. Religious knowledge is possible only through God's guidance.
"Public Duties in Islam" - Ibn Taymiya, pp. 83-85