Transformers have sparked a paradigmatic shift in question-answering training practices by simplifying its architectures. As models became larger and better important usability shortcomings appeared. This includes their computational costs and degraded performance with limited training data (e.g., domainspecific or low-resourced language tasks). Considering this resource trade-of, we (i) explore training strategies such as data augmentation, hyperparameter optimization, cross-lingual transfers and crossdataset mixing, (ii) perform an in-depth analysis to understand the contribution of each on model performance maintenance and (iii) provide a question-answering corpus and a compressed pre-trained model for French1. Our experimental results attest to the merit of a flexible paradigm for a low-resource scenario.