Servant Reduced Difficulty wont apply

As title says and shown in the video below, the Reduced Difficulty bonus somehow sometimes does not apply (sometimes it does tho).
https://youtu.be/WH7TnaUBowI