CS18 : Validate and Verify


https://docs.google.com/presentation/d/1sPMQSFoLJeQMDOZbeLaTCrfNqBF7AumB482GswNEQaI/preview
In this section, we will learn how to use validation and verification routines to help prevent syntax and semantic errors in our code.

We are learning ...
  • About the need for validation and verification
  • Some practical techniques for validation and verification
So that we can ...
  • Discuss the need for input sanitisation, validation and verification
  • Apply suitable input sanitisation techniques
  • Apply suitable validation techniques
    - presence checks
    - type checks
    - length checks
    - range checks
    - sequence checks
  • Exception handling
  • Verification techniques
    - Double entry
    - Email verification

CGP The Revision GuidePage 57, 58
CGP Exam Practice WorkbookPage 66, 67

# Get Ready.png

Activity 1
Validation vs verification 

https://drive.google.com/file/d/0B83yXMOilskab2stWjJuWFVnQUk/preview
Click to enlarge

A program can be described as robust if it is difficult to cause it to crash by entering incorrect data. Robust programs rely on validation, or input sanitation, to make sure that the data entered is present, of the correct type, length or range. Validation ensures that data is sensible, even reasonable, but not accurate. Verification ensures that the data looks sensible by either asking for it twice or manually proofreading it.

Task 1.1 BBC Learning and a Quiz!
Where we learn about the difference between validation and verification

On the BBC Website

Visit the BBC Education page for Data Validation and Verification. This is an GCSE ICT topic but it covers all that you need to know about the theory of these two essential pathways to robustness. Make sure that you run through the 'revision' lesson and try the quick test at the end. When you have done, you should try this ...

https://goo.gl/forms/wGMfS0CNBX1MduAS2

In your notebooks / folders

Make some notes on validation and verification, specifically, how they are different. If you want to use some of the content from the BBC learning page, feel free but don't just copy and paste - you can use the screen clipping tool to cut down your work.


INPUT SANITISATION TECHNIQUES HERE

removing control characters, non-ascii characters
use of the lstrip(), rstrip() and strip() methods

messy = input('Give me a messy string : ')
print('Wow! That is a messy string ...\n')
messyencoded = bytes(messy,"utf-8").decode("unicode_escape")
print(messyencoded)

print('\nLet\'s clean this up!')

print('Original String     : "',messy,'"',sep='')
print('Strip leading       : "',str(bytes(messyencoded.lstrip(),"utf-8"))[2:-1],'"',sep='')
print('Strip trailing      : "',str(bytes(messyencoded.rstrip(),"utf-8"))[2:-1],'"',sep='')
print('Strip both          : "',str(bytes(messyencoded.strip(), "utf-8"))[2:-1],'"',sep='')
print('Only ASCII          : "',''.join([i for i in messyencoded if 32<=ord(i)<=127]),'"',sep='')
print('Lower case          : "',str(bytes(messyencoded, "utf-8"))[2:-1].lower(),'"',sep='')
print('Only ASCII + strip  : "',''.join([i for i in messyencoded if 32<=ord(i)<=127]).strip(),'"',sep='')
print('Lower, ASCII + strip : "',''.join([i for i in messyencoded if 32<=ord(i)<=127]).strip().lower(),'"',sep='')


Activity 2
Practical validation techniques 

https://drive.google.com/file/d/0B83yXMOilskaZE1TSC1vRXN0NTA/preview
Click to enlarge

Again, there is no substitute for trying out some validation functions. Make sure that you engage with the following exercises. Ideally, these should be written as functions but, to keep things simple, I've given you them as simple script snippets. I have provided links to the function based versions in the extension activity.

Just like most things to do with programming, the only way you learn about validation is to do some! The following tasks require you to code lots of simple scripts - there is a focus on testing in this exercise. You should confidently choose suitable testing values to make sure that the scripts work.

Task 2.1 Presence check
Where we learn how to carry out a practical presence check in Python

A presence check makes sure that you actually type something - that the input is actually 'present'.



In a Python script

Enter the following in a new script. Save the script and run it.

valid = False
while not valid:
  value = input('Enter a value : ')
  if value == '':
    print('Validation error - please enter a value!')
  else:
    valid = True

You will be presented with a prompt. Test the script by pressing the  ENTER  key - you should be taken back to the prompt after a suitability derisive message. Try entering a value and pressing the  ENTER  key. Then ...

>>> print(value)

In your notebooks / on paper

Add your name and class in a # comment at the top of the script and print it out for your folder. Write down, in your own words ...
  • how the script works and 
  • describe a specific situation in which you might use it.


Task 2.2 Type check
Where we learn how to perform a practical type check using Python

Type checks make sure that the data you enter is present and of the correct data type. There are two which are relatively easy to do using Python ...
  • Alpha type check - makes sure that the data contains only letters (alphabetical)
  • Digit type check - makes sure that the data contains only number (digits)



 ALPHA TYPE CHECK 

In a Python script

Enter the following in a new script. Save the script and run it.

valid = False
while not valid:
  value = input('Enter a letter : ')
  if value == '':
    print('Validation error - please enter a value!')
  elif value.isalpha() == False:
    print('Validation error - please enter only LETTERS!')
  else: # Must be OK!
    valid = True

Again, you will be presented with a prompt. Try testing the script by typing in a number and pressing the  ENTER  key. You can carry on doing this forever, until you enter a letter. 

Try typing a letter (lower or uppercase) and press the  ENTER  key. Then ...

>>> print(value)

In your notebooks / on paper

Add your name and class in a comment at the top of the script and print it out for your folder. Write down, in your own words ...
  • how the script works and 
  • describe a specific situation in which you might use it.

 DIGIT TYPE CHECK 

In a Python script

Enter the following in a new script. Save the script and run it.

valid = False
while not valid:
  value = input('Enter an letter : ')
  if value == '':
    print('Validation error - please enter a value!')
  elif value.isdigit() == False:
    print('Validation error - please enter only NUMBERS!')
  else: # Must be OK!
    valid = True

Again, you will be presented with a prompt. Try testing the script by typing in a letter and pressing the  ENTER  key. You can carry on doing this forever, until you enter a number. 

Try typing a number and press the  ENTER  key. Then ...

>>> print(value)

In your notebooks / on paper

Add your name and class in a comment at the top of the script and print it out for your folder. Write down, in your own words ...
  • how the script works and 
  • describe a specific situation in which you might use it.


Task 2.3 Length check
Where we learn how to perform a practical length check using Python

A length check ensures that the input is the required number of characters long. There are three standard length checks that you can perform ...
  • Minimum length check which ensures that the data is equal to or greater than a minimum length;
  • Maximum length check which ensures that the data is less than or equal to a maximum length;
  • Range length check which ensures that the data is no less than the minimum length and no more than the maximum length.



 MINIMUM LENGTH CHECK 

In a Python script

Type the following snippet in a new Python script. Save it, run it and test it.

valid = False
while not valid:
  length = 5
  value = input('Enter password (minimum {} characters) : '.format(length))
  if value == '':
    print('Validation error - please enter something!')
  elif len(value) < length:
    print('Validation error - minimum characters {}. Try again.'.format(length))
  else:
    valid = True


In your notebooks / on paper

Add your name and class in a comment at the top of the script and print it out for your folder. Write down, in your own words ...
  • how the script works and 
  • describe a specific situation in which you might use it.

 MAXIMUM LENGTH CHECK 

In a Python script

Type the following snippet in a new Python script. Save it, run it and test it.

valid = False
while not valid:
  length = 5
  value = input('Enter password (maximum {} characters) : '.format(length))
  if value == '':
    print('Validation error - please enter something!')
  elif len(value) > length:
    print('Validation error - maximum characters {}. Try again.'.format(length))
  else:
    valid = True

In your notebooks / on paper

Add your name and class in a comment at the top of the script and print it out for your folder. Write down, in your own words ...
  • how the script works and 
  • describe a specific situation in which you might use it.

 RANGE LENGTH CHECK 

In a Python script

Type the following snippet in a new Python script. Save it, run it and test it.

valid = False
while not valid:
  minimum = 5
  maximum = 10
  value = input('Enter password ({0} and {1} characters) : '.format(minimum,maximum))
  if value == '':
    print('Validation error - please enter something!')
  elif (len(value) < minimum) or (len(value) > maximum):
    print('Validation error - expect {0}-{1} characters. Try again.'.format(minimum,maximum))
  else:
    valid = True

In your notebooks / on paper

Add your name and class in a comment at the top of the script and print it out for your folder. Write down, in your own words ...
  • how the script works and 
  • describe a specific situation in which you might use it.


Task 2.4 Range check
Where we learn how to perform a practical range check using Python

A range check is different to a length check because it checks the numerical range of a number is valid. This example allows you to specify a range but you could easily change this so that it only checked for a minimum or a maximum limit. More often than not, you need both - if you want a minimum only, set the maximum to 999 or something and if you want a maximum, set the minimum at 0 (or whatever, mate).


In a Python script

Enter the following in a new script. Save the script, run it and test it.

valid = False
while not valid:
  minimum = 3
  maximum = 6
  value = input('Enter a number ({0}-{1})'.format(minimum,maximum))
  if value == '':
    print('Validation error - please enter something!')
  elif (float(value) < minimum) or (float(value) > maximum):
    print('Validation error - expected {0}-{1}. Try again.'.format(minimum,maximum))
  else:
    valid = True


In your notebooks / on paper

Add your name and class in a comment at the top of the script and print it out for your folder. Write down, in your own words ...
  • how the script works and 
  • describe a specific situation in which you might use it.


Task 2.5 Sequence check
Where we learn how to perform a practical sequence check using Python

Provide your script with a sequence of values and a sequence check can check that the value you entered is in the list or not. Again, this script also performs a presence check (which they all do).



In a Python script

Enter the following in a new script. Save the script, run it and test it.

valid = False
while not valid:
  sequence = ['1','2','3','4','5'] # Must be strings!
  value = input('Enter a valid value ({0}) : '.format(','.join(sequence)))
  if value == '':
    print('Validation error - please enter something!')
  elif value not in sequence:
    print('Validation error - choose from \'{0}\''.format(','.join(sequence)))
  else:
    valid = True

In your notebooks / on paper

Add your name and class in a comment at the top of the script and print it out for your folder. Write down, in your own words ...
  • how the script works and 
  • describe a specific situation in which you might use it.


Task 2.6 Demonstrating your understanding
Where we learn how to match up a particular validation check to a particular scenario

That's what you reckon!

Demonstrate your understanding

If your teacher has not provided you with a copy of Validation match up, download yourself a copy now, print it out (single sided!) and follow the instructions. You will be expected to code as many validation routines as you can because that's the only way you can demonstrate your understanding!


Activity 3
Exception handling

https://drive.google.com/file/d/0B83yXMOilskaZlQ0LVBlbWQyODQ/preview
Click to enlarge

One way of trapping errors uses a technique called exception handling to try to force Python to crash but trap the error before it displays on the screen. A simple exception block looks like this ...

valid = False
while not valid:
  try:
    perform_operation_that_might_crash  # This could cause an error ...
    valid = True                        # ... but it didn't!
  except:
    display_error_message               # ... but it did!

Task 3.1 Practical exception blocks
Where we learn how to program a practical exception handling routine using Python

The following simple Python snippets are worth nothing outside a full program. Remember, these are techniques!


In a Python script

Open up your favourite Python programming environment and enter the following simple script. You don't need to enter the comments. Save it, run it and test it thoroughly.

valid = False
while not valid:
  try:
    number = int(input('Enter a number : '))   # This could cause an error ...
    valid = True                               # ... but it didn't!
  except:
    print('That\'s not a number - try again!')

What happens when you try ...
- a letter
- a float
- a boolean (True / False)
- a word
- a whole number

In your notebooks / on paper

Make sure you have printed out this script and described the results of the 5 tests you carried out.

A (ridiculously easy) challenge

In a Python script

Write a simple exception block which will accept a float. Hint : Use float() instead of int()

Yup!


# Faster Workers.png


Your challenge, if you choose to accept it, is to create a validation routine to accept a valid date string in the form dd/mm/yy. Don't worry about leap years (oh, really!), but your validation routine must only allow valid months and only allow the correct number of days in each month.

HINTS ...
  • Use a dictionary to store valid days for each month.
  • Validate ...
    1. Presence!
    2. Length (must be 8 characters long)
    3. Position of the '/'s
    4. Type of the year, month and day (must be digit)
    5. Year (must be between 00 and 99 but you will have to convert to INT first)
    6. Month (must be between 01 and 12)
    7. Days (must be between 01 and the days in the selected month)
  • I've left it until now to ask you to do this because you probably need to use try ... except to solve it!

If you give up, ask your teacher for a copy of their solution. In order to deepen your understanding of the way in which the script works, you should create a flowchart of it.

Activity 4
Verification techniques

https://drive.google.com/file/d/0B83yXMOilskaUm9jbzZLcFoyQ1k/preview
Click to enlarge

Task 4.1 Verification check
Where we learn how to perform a practical verification check using Python

If the data that is entered into a computer system matches the original source of the data then it is said to be verified and the techniques used to ensure this are called verification.

Verification can be performed by proof reading / checking the input manually or using a procedure called double entry because you enter the same piece of data twice and the computer checks whether it is the same. This doesn't confirm that the data is actually correct, only that the likelihood is that you didn't make any errors when you typed it in and that it matches the original source of the data.


In a Python script

Open up your favourite Python programming environment and type the following simple script. Save it, run it and test it thoroughly.

valid = False
while not valid:
  value = input('Enter email address : ')
  if value == '':
    print('Please enter your email address - try again!')
  else:
    check = input('Enter your email address again : ')
    if value != check:
      print('Sorry - both values must be the same. Start again!')
    else:
      valid = True

Again, thoroughly test this code snippet using 4 tests. Try ...
- Not entering any email address
- Entering the first one and leaving the second one blank
- Entering the first one and a slightly different second one
- Entering them both exactly the same

In your notebooks / On paper

Make sure you have a copy of the script for your notes. Describe how it works and make sure you give details of the 4 tests you have performed to investigate its operation.


Task 4.2 Email verification
Where we how email verification works and practice drawing a flowchart.

Often, when you sign up for a new web account, the website uses a technique called email verification to send you a email containing a unique, time-limited token in order to check that the email address exists and that you are happy to continue using it to sign up to the website ...


... and the email contains a link which you click which contains the unique token generated at the time you signed up.


If, for some reason, you type in the wrong verification token (why would you?) or you don't verify your email within a certain time, the code expires and you have to ask for a new verification email to be sent.

In a flowchart

Firstly, copy up the algorithm into your notebooks. Next, create a flowchart on paper, using yEd Live or an alternative flow diagram editor (not Flowgorithm - this isn't really a program) which explains the method of email verification. My flowchart is based on the following algorithm ...

01 START
02 GET email address
03 WHILE email not verified
04   GENERATE unique token
05   SEND verification email
06   WAIT for user to click link
07   IF token is not valid
08     DISPLAY error message
09   ELSE IF token has expired
10     DISPLAY error message
11   ELSE
12     email verified
13 DISPLAY confirmation message
14 STOP

HINT : Line 03 WHILE email not verified is a 'decision' symbol. The 'loop' simply leads back to line 04.

Complete the worksheet

Your teacher will give you a copy of a worksheet which contains my solution. Complete the worksheet and then compare it to your solution. Does it look similar?


Assessment Task (Homework)


Leanne, John, Tristen and Georgia are still waiting for their explanations! In your notebooks / on paper, make sure you have a copy of each of their questions at the top of the page and write some notes underneath to help answer their questions. 

Grading rubric

MASTER : Your explanations to our friends are clear and concise demonstrating a deep understanding of the concepts - your teacher is proud of you.
APPRENTICE : Your explanations to our friends are brief and demonstrate a simple understanding of the concepts.
NOVICE : Your explanations to our friends are very simple and only demonstrate a partial understanding of the concepts. They are really none the wiser ...

# Flash cards.png
Click to load key word list to help you make your own flash cards 


Hungry for more?

For those of you who are eager to get more complicated (and more efficient), here are links to the structured versions of the validation routines that you have learnt about in this lesson. We'll be focusing on getting these little fellas integrated into our scripts soon when we look at the Break it down topic, don't worry!
There is one more validation check that we've met before in the filehandling section - using the os library to check whether a file exists. Look back at the scripts for this section to refresh your memory!

FAQ

Q : What other things can you use for a type check?
A : In Python, you can easily check for digits using variable.isdigit(), letters using variable.isalpha(), and both letters and numbers using variable.isalnum(). Checking for other variable types is a bit trickier ...

Q : How do you set up a practical email validation routine for a website?
A : This is a bit tricky - you would need to learn to write webpages with HTML and CSS, client side scripting with Javascript and serverside scripting with PHP. There are lots of tutorials on the web which could help you.

Q : Are there other methods of verification?
A : The two main methods are email verification and double entry which we've looked at.

Q : Is this a popular topic on the exam?
A : It may come up on the exam but it is more important for programming tasks.

Q : Do you need to know regular expressions at GCSE level?
A : No, not for the examination but it doesn't do any harm to learn about it!