There is a field called Ref (for Reference) below the Email Address field in the Bill To area. This Ref field is for anything you'd like to enter, anything you can use in your conversations with your client. You could use this field for the manually entered number.
We can't let people enter the Invoice Number because that needs to be a unique number and humans might enter the same number for 2 or more invoices.
But if you'd like, if the Invoice Number is too distracting, we could add a functionality to hide it. Then it won't show in your invoices. What do you think?