Yes, in Galatians 4, Paul is basically saying, "What is good for the goose is good for the...other goose." Something like that. He says that the legalists "zealously affect you", but not for good. Not for a good reason, not with a right motive.
But, he says, it is good for the heart to be zealously affected in a good thing, always! Paul, a master of sarcasm, is doing a "back atcha, Pharisee". The more the pseudo law-lovers try to zealously affect God's people, the more Paul is going to zealously affect them.
It isn't a bad thing, at all, to be zealously affected. It isn't necessarily wrong for someone to try to zealously influence you. It depends on 1. What you are being affected for, or what you are being affected about, and 2. The motive of the person influencing you.
Paul's Great Goal was Grace. The legalists Great Goal was Performance.
And so it is today. The scary thing is, you do get to choose what you will be zealously affected by.
Reread Galatians, if you get the chance.