The Trump administration’s plan to deploy some of the most restrictive investment limits in its economic arsenal against China marks a setback for Treasury Secretary Steven Mnuchin’s effort to take a less confrontational approach toward Beijing.