We combined spectroscopic ellipsometry, Raman scattering spectroscopy, and first-principles calculations to explore the optical properties of YBaCuFeO5 single crystals. Measuring the optical absorption spectrum of YBaCuFeO5 at room temperature revealed a direct optical band gap at approximately 1.41 eV and five bands near 1.69, 2.47, 3.16, 4.26, and 5.54 eV. Based on first-principles calculations, the observed optical excitations were appropriately assigned. Analysis of the temperature dependence of the band gap indicated anomalies in antiferromagnetic phase transition at 455 and 175 K. Additionally, a hardening in the frequency of the Eg phonon mode was observed at 175 K. The value of the spin–phonon coupling constant was 15.7 mRy/Å2. These results suggest a complex nature of spin–charge–lattice interactions in YBaCuFeO5.