We present a detailed study of the impact of ligand passivation on the electronic structures and optical properties of plasmonic Ag nanoclusters using density functional theory (DFT) and time-dependent density functional theory (TD-DFT). The clusters studied are Ag135+, Ag25SH18−, Ag25NH218−, Ag3214+, and Ag44SH304−. We find that the highest occupied ligand orbitals from S (3p) and N (2p) appear just above the conduction band, and this leads to significant ligand-to-metal charge transfer transitions at high energies. Dielectric screening associated with ligand passivation results in reduced HOMO–LUMO gaps and in an increased gap between the HOMO and the valence band associated with the Ag 4d orbitals. Ligand field effects result in splitting of plasmonic peaks, leading to reduced mixing between nearby single-particle excitations. The magnitude of these effects is found to decrease when thiolate ligands are replaced with amine ligands. We also find that, in the case of the Ag44SH304− cluster, the ligands localize plasmonic excitations into the core of the cluster.