The COVID-19 pandemic is impacting communities worldwide, with direct effects of illness and mortality, and indirect effects on economies, workplaces, schools/daycares, and social life. However, we understand very little about the effects of this pandemic on families of young children. We used a risk and resilience model to evaluate the effects of the pandemic on mental health in diverse caregivers (N = 286) with children ages birth to 5. We evaluated the hypotheses that (a) pandemic stress and caregiver-reported child psychosocial concerns correlate with caregivers’ mental health symptoms and (b) caregivers’ pandemic-related self-efficacy and coping mediate these relationships. Caregivers completed surveys in April–May 2020 assessing pandemic stress (e.g., health, finances, and housing), child psychosocial problems, coping strategies, and self-efficacy to manage family needs. Our primary outcome was caregivers’ self-reported changes in mental health symptoms since the outbreak. Path analysis revealed that higher pandemic stress was associated with caregivers’ reduced confidence in meeting their family’s needs related to COVID-19, which correlated with worse caregiver mental health symptoms. Greater child psychosocial problems also predicted worse caregiver mental health symptoms. Findings suggest that pandemic stress, child psychosocial problems, and caregiver self-efficacy are interrelated in their influence on caregivers’ mental health. While further research is needed to examine strategies to foster resilience and buffer the pandemic’s effects on caregiver mental health, this is a first step in evaluating the psychosocial effects of this pandemic in families of young children. Clinical implications are discussed for a tiered response to mitigate the pandemic’s impacts on family functioning.