The exact worries vary, although they tend to be a mix of tangible security and privacy problems with teachers and staff who may not be familiar with how Zoom works. There's no end-to-end encryption for Zoom meetings, making it possible to intercept videoconferences. Teachers have also had to grapple with "Zoombombing" incidents where bad actors disrupt virtual classes -- sometimes by guessing the meeting number, sometimes because staff share the meeting details on public sites. Add and people uploading videos to unprotected websites and it may be all too easy for a determined intruder to spy on calls or collect recordings.
In a statement to the Washington Post, Zoom reiterated that it takes privacy and security "extremely seriously." It said it had been "proactively engaging" to outline best practices and policies, and pointed out changes taking effect that require waiting rooms for K-12 classes and limit who can share data by default. These should help, but they're clearly coming too late for some schools. They've had untrustworthy experiences with Zoom, and they're unlikely to come back in the near future even if all their complaints are addressed.