We discuss the failure of the Markov approximation in the description of atom-surface fluctuation-induced interactions, both in equilibrium (Casimir-Polder forces) and out of equilibrium (quantum friction). Using general theoretical arguments, we show that the Markov approximation can lead to erroneous predictions of such phenomena with regard to both strength and functional dependencies on system parameters. In particular, we show that the long-time power-law tails of two-time dipole correlations and their corresponding low-frequency behavior, neglected in the Markovian limit, affect the prediction of the force. Our findings highlight the importance of non-Markovian effects in dispersion interactions.