According to the HUD, the GPU load is actually higher in the bad cases. Have you checked what clocks the GPU runs at in each case?