A senior barrister has urged a jury to “avoid the assumption” his client was more likely to have caused significant injuries to an infant just because he is a man.