If you disconnect the IR board, then the camera has no input from the light sensor. This is likely causing your camera to assume that it should be in night mode all of the time.
Most of the sPoE cameras do not have the ability to disable the IR lights on the camera. If the light is sufficient where the camera is mounted (and the IR board is still connected), then the IR lights will turn off on their own.
Some of the newest versions have the ability to turn the IR lights off through the NVRs settings in the Zmodo app. You may want to check that.
It isn’t possible to get to the IR filter on the camera’s image sensor without voiding the warranty, and likely breaking the camera.