Mechanisms are provided for performing load balancing of question processing in a Question and Answer (QA) system, implemented by the data processing system, having a plurality of QA system pipelines. The mechanisms receive an input question for processing by the QA system and determine a predicted question difficulty of the input question. The mechanisms select a QA system pipeline from the plurality of QA system pipelines based on the predicted question difficulty and route the input question to the selected QA system pipeline for processing. In addition, the mechanisms process the input question with the selected QA system pipeline to generate an answer for the input question.