In this paper, we introduce a high order finite element (FEM) implementation using perfectly matched layer (PML) for the scattering by plasmonic structures inside layered media. The PML is proven to be very accurate and efficient by a comparative analysis with a commercial FEM software and the Multiple Multipole Program (MMP). A convergence analysis using hp-adaptive refinement inside the PML layer
shows that adaptive mesh refinement inside the PML layer is most efficient. Based on this convergence analysis an hp-strategy is proposed, which shows a remarkable error reduction for small additional computational costs.