LW1, you SHOULD have said something, but not what Kathy/Marcy recommend. That can work for very young children, but the boy is old enough to understand what you're saying when you call him a handful, and it justifies the Mom's belief that he, as the older sib, was to blame for younger's crying and annoying strangers in public. You may be setting him up for worse treatment later.
Instead, youcould have approached Mom & praised the boy, either immediately after the incident (don't let on that you saw it) or when you saw the family again. Not in a way that blames Mom for her absence, but just to say you saw how protective he was of his little sis and how distressed he was by her crying, and how lucky she is to have such a caring big brother.

Slapping a child can cause hearing loss--it happened to my brother, while my mother stood by and did nothing. Years ago, before my son who is 24 was born, in a store I saw a little girl about 4, crying, while her pig of a father threatened to "break her finger" and her stupid mother just stood there. I've always regretted not finding a payphone and calling the police or getting the security guard involved. I will never let that happen again. If I see a child being abused, I'll call the police.

The congregation can be SOMETHING ELSE! I know of a mid 40s age minister who was called on the carpet by members of the congregation for going to see elderly widows without his wife along. People would think there was hanky panky going on. A man did not visit a decent woman alone!