"He's causing a lot of frustration and even pain with his words,” the senator said