Validation firing on Textbox tab out

I am having trouble preventing validation when I tab out of a Textbox.

If I have 2 x Textboxes + 2 x RequiredFieldValidators and type into the 1st Textbox, then fine.
However, if I tab back into the Textbox, remove the text and tab out, then the validator fires.
I only want the validation to fire when I click on a link button (ie, I try to submit the form).

This is a very simple example without the link button:

<asp:TextBox ID="x" runat="server" CausesValidation="false" AutoPostBack="false" /><asp:TextBox ID="y" runat="server" CausesValidation="false" AutoPostBack="false" /><asp:RequiredFieldValidator ID="rfvX" runat="server" ControlToValidate="x" ErrorMessage="*" ValidationGroup="z" /><asp:RequiredFieldValidator ID="rfvY" runat="server" ControlToValidate="y" ErrorMessage="*" ValidationGroup="z" />

There is nothing else in the form.

How can I prevent the validator firing except for when a link button is pressed?