We introduce a second quantization scheme based on quasinormal modes, which are the dissipative modes of leaky optical cavities and plasmonic resonators with complex eigenfrequencies. The theory enables the construction of multiplasmon or multiphoton Fock states for arbitrary three-dimensional dissipative resonators and gives a solid understanding to the limits of phenomenological dissipative Jaynes-Cummings models. In the general case, we show how different quasinormal modes interfere through an offdiagonal mode coupling and demonstrate how these results affect cavity-modified spontaneous emission. To illustrate the practical application of the theory, we show examples using a gold nanorod dimer and a hybrid dielectric-metal cavity structure.