Probing and engineering the environment of near-surface nitrogen-vacancy centers in diamond for quantum sensing and simulation