Create a first step in your SQL Agent job which runs this:
DECLARE @result INT; SET @result = (select sys.fn_hadr_is_primary_replica ( 'your datbase name' ) ) IF @result = 0 RAISERROR('Server not primary for database',16,0);
And in Advanced properties set "On failure action:" to "Quit the job reporting success".
Then the job will only run on the primary server.