The primary difference between Class A and Class B asbestos removal lies in the type of asbestos material being handled, the risk level, and the strictness of the safety controls required by Workplace Health and Safety (WHS) regulations.