Olmos was born and raised in Los Angeles, California, the son of Eleanor (née Huizar) and Pedro Olmos, who was a welder and mail carrier. His father was a Mexican immigrant who moved to California in 1945 and his mother was Mexican American. His parents separated when he was seven years old, and he was mostly raised by his great-grandparents. He grew up wanting to be a professional baseball player, and at age 13 he joined the Los Angeles Dodgers' farm system, playing as a catcher. He left baseball at age 15 to join a rock and roll band, which caused a rift with his father. He graduated from Montebello High School in 1964.
From 1979 to 1987, Olmos lived in West New York, New Jersey. In 1971, he married Katija Keel, the daughter of actor Howard Keel. They had two children, Bodie and Mico, before divorcing in 1992. Olmos has four adopted children: Daniela, Michael, Brandon, and Tamiko. He married actress Lorraine Bracco in 1994. She filed for divorce in January 2002 after five years of separation. Olmos also had a long term relationship with actress Lymari Nadal. They married in 2002.
In 1996, he was awarded an honorary Doctorate of Fine Arts from California State University, Fresno. In 2007, after a seven-year process, he obtained Mexican nationality. Asteroid 5608 Olmos is named in his honor.