Individual differences in responsiveness toward infant crying are well understood. Little research, however, has examined the effects of situational risk factors (e.g., social stress, cognitive load) and possible interactions between situational and dispositional factors on response toward infant crying. This study examined if trait empathy (conceptualized as empathic concern; EC, and personal distress; PD) moderate situational factors’ relationship with people’s intentions in response to infant crying. Social stress was manipulated using the Trier Social Stress Test. Cognitive load was manipulated by requiring participants to keep syllable-strings of either two or eight syllables in mind while listening to an infant crying. Participants responded to question items examining their caregiving and neglect intentions in response to the crying stimulus. Multilevel regression analyses demonstrated that trait empathy (empathic concern in particular) was strongly associated with neglect intention under cognitive load. Participants with high EC showed strong neglect intention with increasing cognitive load. Furthermore, results also showed that social stress increased neglect intentions and reduced care intentions; these effects were more remarkable among participants with both low EC and low PD. These results suggest that dispositional factors moderate situational factors’ effect on response to infant crying.