Maybe a different religion, or especially political beliefs seems to be a big deal-breaker. Do you still find it worthwhile to keep them in your life?
I do. I have e.g. Christian Conservative friends, and Atheist Liberal ones, etc. I enjoy each one for what they are. I mean, nobody is perfect! (like me 😁)
There is little use blaming yourself for much of anything you’ve done in the past - you can’t go back and change any of it, and perhaps you’ve thought about it since much more than they have even:-). The only thing to do is decide what way seems best to move forward to.