St. Paul gives us clear guidance on when we are to work for unity and when we can allow liberty with our brothers and sisters in Christ. That's found in Romans 14.
St. Augustine summarizes that chapter nicely,
In essentials unity;
In non-essentials liberty;
In all things charity (love).
Challenge: Read Romans 14 and ask God to show you what nonessential in church you're treating as an essential.