Understanding the lattice dynamics and phonon transport from the perspective of chemical bonds is essential for improving and finding high-efficiency thermoelectric materials and for many applications. Here, the coexistence of global and local weak chemical bonds is elucidated as the origin of the intrinsically low lattice thermal conductivity of non-caged structure Nowotny–Juza compound, α-MgAgSb, which is identified as a new type of promising thermoelectric material in the temperature range of 300–550 K. The global weak bonds of the compound lead to a low sound velocity. The unique three-centered MgAgSb bonds in α-MgAgSb vibrate locally and induce low-frequency optical phonons, resulting in “rattling-like” thermal damping to further reduce the lattice thermal conductivity. The hierarchical chemical bonds originate from the low valence electron count of α-MgAgSb, with the feature shared by Nowotny–Juza compounds. Low lattice thermal conductivities are therefore highly possible in this series of compounds, which is verified by phonon and bulk modulus calculations on some of the compositions.