Colbert grew up with a lot of brothers and sisters, most of whom got to hear some of the comedian’s earliest jokes for the first time.