tl;dr - This is (mostly) intended functionality by the browser. Though the devs could have fixed it, it's way easier and faster to just leave it like that.
The HTML input element allows you to set the type of data you expect people to enter, which provides some basic validation. In this case, it's using the type of number, for obvious reasons. This disallows all letters except e, (for scientific notation) and all symbols except a single .. It also causes the number entry keyboard to appear on mobile devices, if supported.
There is no specific phone number data type, so the developers of that site used the more generic number type. The browser can't tell the difference, and so adds the individual up/down buttons as it would with any other standard number input.
It's possible to remove by using a more generic input type of text and adding in validation manually to limit entry to only digits, but it seems like a reasonable shortcut to take.
Also, I just found out that there is in fact an input type called tel for phone numbers but it doesn't include any validation (apparently because of how varied the formatting of phone numbers can be throughout the world). So it only does the numeric keyboard on mobile devices. This, plus some validation, would have been the best choice I think.