Conversational biometrics and speech recognition are used by an IVR contact/call center during call hold to affect call handling/routing. A caller's (user's) behavioral response, such as an utterance or other spoken reaction to being put on hold by the IVR contact/call center, etc., can be recognized, captured and analyzed. Business rules analyze the caller's behavior in real time during a caller hold period. Such business rules can be fed back into the call center system and this information can be used to provide opportunities to affect routing priorities for a particular caller based upon information learned by the data that is captured and analyzed.