UNC is not leaving Duke behind, and vice versa. Clemson would be stupid to leave the ACC, and UVA is the prototypical ACC school.
Notre Dame is going to join the Big Ten - that is their only option to maintain a semblance of their media rights revenue stream. There wouldn't be a single incentive for NBC to pay massive money for exclusive Notre Dame rights if Notre Dame can't schedule any Big Ten or SEC schools because Big Ten and SEC schools won't have an incentive to play any tough schools outside of their respective conference.
The ACC is in an interesting spot. Their niche is being the Academic Power Conference, with private schools and academic standards galore. They also have a nice deal with Disney/ESPN that gives them a financial foundation, their basketball is top class, their football has a National Champion contender every couple of years, and they offer a higher-end level of off-brand sports like soccer and lacrosse. The PAC12 & Big12 don't have this niche. Combining into one conference and becoming the 4th Power Conference might be their only way to survive as a Power Conference.