We describe a numerical time-domain approach for high-accuracy calculations of Casimir-Polder forces near microstructured materials. The use of a time-domain formulation enables the investigation of a broad range of materials described by advanced material models, including nonlocal response functions. We validate the method by a number of example calculations for which we thoroughly investigate the convergence properties of the method, and comparing to analytical reference calculations, we find average relative errors as low as a few parts in a million. As an application example, we investigate the anisotropy-induced repulsive behavior of the Casimir-Polder force near a sharp gold wedge described by a hydrodynamic Drude model.