Bhagavad-gita As It Is

by A.C. Bhaktivedanta

Though the embodied soul may be restricted from sense enjoyment, the taste for sense objects remains. But ceasing such engagements by experiencing a higher taste, he is fixed in consciousness. PURPORT Unless one is transcendentally situated, it is not possible to cease from sense enjoyment. The process of restriction from sense enjoyment by rules and regulations is something like restricting a diseased person from certain types of eatables. The patient, however, neither likes such restrictions nor loses his taste for eatables. Similarly, sense restriction by some spiritual process like aṣṭāṅga-yoga, in the matter of yama, niyama, āsana, prāṇāyāma, pratyāhāra, dhāraṇā, dhyāna, etc., is recommended for less intelligent persons who have no better knowledge. But one who has tasted the beauty of the Supreme Lord Kṛṣṇa, in the course of his advancement in Kṛṣṇa consciousness, no longer has a taste for dead, material things. Therefore, restrictions are there for the less intelligent neophytes in the spiritual advancement of life, but such restrictions are only good until one actually has a taste for Kṛṣṇa consciousness. When one is actually Kṛṣṇa conscious, he automatically loses his taste for pale things.

Trying to restrict the senses can do more harm than good. you just havr to find a higher form of happiness to transcend the senses