Even if Toby said he wasn't going to freak, Sam was still looking for any indications of that emotion in his head. Toby's colors were showing regular human, nothing weird about him, which would explain Sam not picking up the telepathy thing.

"Why didn't you say anything when you picked up that up from me?" Sam asked. "Because you didn't know the word?"