First of all, he didn't even father all those people behind him. He is a father of just one toddler. That's all it takes to instill the fear within that reportedly never leaves.
The 49-year-old actor - who has a two-year-old son, Exton, with his pregnant wife Susan - has confessed to being "exhausted" by the tot and found it scary when the toddler learnt to walk and climb.
He told Access Hollywood: "Yeah, he just wants to sling open drawers and grab knives. It's really not stressful at all! When he first started walking, it was kind of like, alright, this is a little bit exhausting and terrifying, and then he kind of smoothed out and now he can climb things."
Seemingly Iron Man's worries stem from an accident he had as a youngster, which landed him in the doctor's surgery.
Here's what he remembers: "I just remember when I was 2 or 3 or 4 years old, I found my way to the top of the refrigerator and I hung out there, that was like my spot until I lost my footing and banged my head on the radiator. I remember some doctors squeezing my head... I don't want him to fall off the refrigerator." Worse, he could fall off the refrigerator and into the cutlery drawer filled with knives, not that we're ones to perpetuate fear, or anything.
Robert - who is expecting a baby girl with his wife in November - also revealed he is excited to get to know his little girl and is curious about her personality.
He said: "It should be great. I wonder [what] a little girl Downey will be like. I'm sure she'll let us know."
And then some. The omnipotent "they" say that "boys wreck your home... girls wreck your head."