Historically, humans have cleared many forests for agriculture. While this substantially reduced ecosystem carbon storage, the impacts of these land cover changes on terrestrial gross primary productivity (GPP) have not been adequately resolved yet. Here, we combine high-resolution datasets of satellite-derived GPP and environmental predictor variables to estimate the potential GPP of forests, grasslands, and croplands around the globe. With a mean GPP of 2.0 kg C m−2 yr−1 forests represent the most productive land cover on two thirds of the total area suitable for any of these land cover types, while grasslands and croplands on average reach 1.5 and 1.8 kg C m−2 yr−1, respectively. Combining our potential GPP maps with a historical land-use reconstruction indicates a 4.4% reduction in global GPP from agricultural expansion. This land-use-induced GPP reduction is amplified in some future scenarios as a result of ongoing deforestation (e.g., the large-scale bioenergy scenario SSP4-3.4) but partly reversed in other scenarios (e.g., the sustainability scenario SSP1-1.9) due to agricultural abandonment. Comparing our results to simulations from state-of-the-art Earth System Models, we find that all investigated models deviate substantially from our estimates and from each other. Our maps could be used as a benchmark to reduce this inconsistency, thereby improving projections of land-based climate mitigation potentials.