Drought as a slow-onset phenomenon inflicts important losses to agriculture where the degree of vulnerability depends not only on physical variables such as precipitation and temperature, but also on societal preparedness. While the scopes of physical and social vulnerability are very different in nature, studies distinguishing these two aspects have been lacking. In this study we address the physical and social aspects of drought vulnerability of maize (CDVIphy and CDVIsoc) in Sub-Saharan Africa (SSA). To quantify vulnerability, we applied a probabilistic framework combining a Drought Exposure Index (DEI) with a physical or social Crop Failure Index, CFIphy or CFIsoc, respectively. DEI was derived from the exceedance probability of precipitation. Maize yields, simulated using the Environmental Policy Integrated Climate (EPIC) model, were used to build CFIphy, whereas the residual of simulated and FAO recorded yields were used to construct CFIsoc. The results showed that southern and partially central Africa are more vulnerable to physical drought as compared to other regions. Central and western Africa, however, are socially highly vulnerable. Comparison of CDVIphy and CDVIsoc revealed that societal factors cause more vulnerability than physical variables in almost all SSA countries except Nigeria and South Africa. We conclude that quantification of both drought vulnerabilities help a better characterization of droughts and identify regions where more investments in drought preparedness are required.