It has to do with dm-verity. Taken from XDA Developers here:
The problem (itxe2x80x99s a problem if you like root and modifying devices) stems from something I pointed out a long time back, when it first hit AOSP xe2x80x93 the introduction of dm-verity to Android. Verity is a security feature, originally found in ChromeOS, designed to provide assured and trustworthy computing devices, preventing malicious software from modifying a device. Back in Android 4.4, Google announced verity for Android, and then all remained quiet. While there has been some research into using verity, for the most part, things have been quiet. Until now, that is.
With Android 6.0, Google has begun to up their game on device security. One of the fundamental requirements for this is to prevent the software on a device from being modified without a userxe2x80x99s knowledge xe2x80x93 while many here at XDA take root for granted, imagine a userxe2x80x99s device being rooted without their knowledge or consent, and root access being used to steal their data. For this reason, Google has started to implement verification of the system partition on some devices.
To answer your question of "How does it know..."...any changes to the system partition at boot (or when accessed) could cause one of three warning/error messages.