8c. Prof. Rosenthal’s Review#
A review of Chapter 8: Strings and Regular Expressions, for students who have already studied the chapter. Use the two lists under Concepts covered as a checklist (click a list’s title to hide it), then work through the Questions, which go from easy to hard. Each question has a suggested answer under Answer. Try the question first, then open the Answer.
Concepts covered#
Python Syntax and Semantics
Indexing
A string is a sequence of characters.
STRING[INDEX]selects the character at position INDEX. The first character has index0: iffruit = 'banana', thenfruit[0]is'b'andfruit[1]is'a'.The index can be any expression whose value is an integer, such as
fruit[i+1]. An index that is not an integer, such asfruit[1.5], is aTypeError.len(STRING)returns the number of characters in the string. The indices of a string go from0tolen(STRING) - 1. An index after the last character, such asfruit[len(fruit)], is anIndexError.A negative index counts backward from the end:
STRING[-1]is the last character,STRING[-2]is the second to last, and so on.fruit[-1]is'a'andfruit[-2]is'n'.This runs the body once for each index of the string, with
ireferring to0,1,2, … up to the length minus 1:for i in range(len(STRING)): BODY
For example, this displays
0 b,1 aand so on, on separate lines:for i in range(len(fruit)): print(i, fruit[i])
Slices
STRING[N:M]returns the part of the string from index N up to, but not including, index M:fruit[0:3]is'ban'andfruit[3:6]is'ana'.If you omit the first index, the slice starts at the beginning of the string:
fruit[:3]is'ban'. If you omit the second index, the slice goes to the end of the string:fruit[3:]is'ana'.fruit[:]is the whole string.If the first index is greater than or equal to the second, the slice is the empty string,
'', which has length0:fruit[3:3]is''.An index after the end of the string is not an error in a slice: the slice stops at the end of the string.
fruit[3:100]is'ana'.Negative indices also work in slices:
fruit[:-1]is'banan'(all the characters except the last).
Strings are immutable
A string cannot be changed. An assignment to a character of a string, such as
fruit[0] = 'B', is aTypeError.To get a variation of a string, create a new string, for example with
+and a slice:'B' + fruit[1:]is'Banana'.String methods never change the string they are invoked on: they return a new string.
Comparing strings
==and!=check whether two strings are equal. Uppercase and lowercase letters are different:'Cat' == 'cat'isFalse.<,>,<=and>=compare strings in alphabetical order:'apple' < 'banana'isTrue.All the uppercase letters come before all the lowercase letters:
'Pineapple' < 'banana'isTrue. To compare words in alphabetical order, convert them to lowercase first.A string that is the beginning of a longer string comes before it:
'ban' < 'banana'isTrue.
String methods
A method is invoked with the dot operator:
STRING.METHOD_NAME(ARGUMENTS).STRING.upper()returns a new string with all the letters in uppercase:'banana'.upper()is'BANANA'. (STRING.lower()does the same with lowercase.)STRING.startswith(PREFIX)returnsTrueif STRING starts with PREFIX, otherwiseFalse:'banana'.startswith('ban')isTrue.STRING.endswith(SUFFIX)checks the end of the string in the same way:'banana'.endswith('ban')isFalse.STRING.count(PART)returns the number of times PART appears in STRING:'banana'.count('a')is3. The parts that it counts do not overlap:'banana'.count('ana')is1.STRING.replace(OLD, NEW)returns a new string in which every OLD is replaced with NEW:'banana'.replace('a', 'o')is'bonono'.PART in STRINGisTrueif PART appears anywhere in STRING:'nan' in 'banana'isTrue.
Writing files
open(FILE_NAME)opens a file for reading.open(FILE_NAME, 'w')opens a file for writing: if the file does not exist, it is created; if it exists, its contents are replaced.FILE_OBJECT.write(STRING)writes the string to the file. It does not add a newline: to end a line, put'\n'at the end of the string.FILE_OBJECT.close()closes the file. Close a file when you are done writing it: what you wrote is not certain to be in the file until it is closed.A file object keeps track of where it is in the file. A second loop over the same file object continues after the last line that the first loop read. To start again at the beginning, open the file again.
The break statement
breakinside a loop ends the loop immediately. The program continues with the first statement after the loop, and the loop variable keeps the value it had when the loop ended:for LOOP_VARIABLE in SEQUENCE: if CONDITION: break BODY
For example, this displays
banda, and stops at the firstn:for letter in 'banana': if letter == 'n': break print(letter)
Regular expressions
import remakes the functions of the modulere(regular expressions) available.re.search(PATTERN, STRING)returns aMatchobject if the pattern matches some part of the string, andNoneif it does not. To check, compare the result withNone:re.search('nan', 'banana') != NoneisTrue.If
resultis aMatchobject,result.stringis the string that was searched,result.group()returns the part of the string that matched, andresult.span()returns two numbers in parentheses: the index where the match starts and the index just after it ends.re.search('nan', 'banana').span()is(2, 5).In a pattern, letters, digits and spaces match themselves. These characters are special:
|matches either the pattern on its left or the pattern on its right:'Mina|Murray'matchesMinaorMurray.^matches the beginning of the string:'^Dracula'matches a string that starts withDracula.$matches the end of the string (before a newline at the end):'Harker$'matches a string that ends withHarker.Parentheses enclose the part of the pattern that
|applies to:'cent(er|re)'matchescenterandcentre.?makes the character before it optional:'colou?r'matchescolorandcolour.
re.sub(PATTERN, REPLACEMENT, STRING)returns a new string in which every part of STRING that matches PATTERN is replaced with REPLACEMENT:re.sub('colou?r', 'color', 'colour and color')is'color and color'.A regular expression is often written as a raw string, with the letter
rbefore the quotation mark, such asr'colou?r'.
Shell commands
In Jupyter and Colab, a line that starts with
!is a shell command, not Python.!head FILE_NAMEdisplays the first 10 lines of a file,!head -100 FILE_NAMEdisplays the first 100, and!tail FILE_NAMEdisplays the last 10.>writes the result to a file instead of displaying it:!head pg345.txt > first_lines.txt. These commands are not available on all operating systems.
Definitions, etc.
Definitions
DEFINITION: A sequence is an ordered collection of values, in which each value is identified by an integer index.
DEFINITION: A character is an element of a string: a letter, a digit, a punctuation mark or white space.
DEFINITION: An index is an integer that selects an item in a sequence, such as a character in a string. In Python, indices start at
0.DEFINITION: A slice is a part of a string specified by a range of indices.
DEFINITION: The empty string,
'', is a string that contains no characters and has length0.DEFINITION: An object is something a variable can refer to. An object has a type and a value.
DEFINITION: An object is immutable if its elements cannot be changed. Strings are immutable.
DEFINITION: An invocation is an expression, or part of an expression, that calls a method, as in
word.upper().DEFINITION: A regular expression is a sequence of characters that defines a search pattern.
DEFINITION: A pattern is a rule that specifies the requirements a string has to meet to be a match.
DEFINITION: String substitution is the replacement of a string, or part of a string, with another string.
DEFINITION: A shell command is a statement in a shell language, a language used to interact with the operating system.
Other ideas
Indices start at
0, so the last index of a string is its length minus 1.Convert strings to a standard format, such as all lowercase, before comparing them.
If you know exactly which characters to look for, the
inoperator and thereplacemethod are enough. Regular expressions can do more: match one of several words, the beginning or the end of a string, or an optional character.When a program works with a large file, start with a small part of the file (for example its first 10 or 100 lines), get the program working, and then run it with the whole file.
Questions#
The questions use indexing and slices, string methods, files, break and regular expressions. The module re (regular expressions) is imported in the setup cell below with import re.
One question reads words.txt, a list of about 114,000 English words, one per line, in alphabetical order (the first two are aa and aah). The setup cell below downloads it.
from os.path import basename, exists
def download(url):
filename = basename(url)
if not exists(filename):
from urllib.request import urlretrieve
local, _ = urlretrieve(url, filename)
print("Downloaded " + str(local))
return filename
download('https://raw.githubusercontent.com/AllenDowney/ThinkPython/v3/words.txt');
import re
Summary and questions by Prof. Y. Rosenthal, based on Think Python: 3rd Edition by Allen B. Downey. Text license: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
Question 1 (easy): what is displayed?#
Predict what this code displays, then open the Answer to check.
fruit = 'banana'
print(fruit[0], fruit[1], fruit[5])
print(fruit[-1], fruit[-2])
print(len(fruit), fruit[len(fruit) - 1])
Answer
b a a
a n
6 a
The first character has index 0, so fruit[1] is the second character, 'a'.
fruit has 6 characters, with indices 0 to 5. So the last character is fruit[5], which is the same as fruit[len(fruit) - 1] and fruit[-1].
A negative index counts backward from the end: fruit[-2] is the second to last character, 'n'.
Question 2 (easy): what is displayed?#
word = 'Python'
print(word[0:2], word[2:4], word[4:6])
print(word[:2], word[2:])
print(word[:])
print('[' + word[3:3] + ']', len(word[3:3]))
Answer
Py th on
Py thon
Python
[] 0
word[N:M] contains the characters from index N up to, but not including, index M. So word[0:2] has 2 characters, the ones with indices 0 and 1.
Without the first index, the slice starts at the beginning; without the second, it goes to the end. So word[:2] + word[2:] is the whole string, and so is word[:].
When the first index is greater than or equal to the second, the slice is the empty string: word[3:3] is ''.
Question 3 (easy): what is displayed?#
s = 'banana'
print(s.upper(), s)
print(s.startswith('ban'), s.startswith('Ban'), s.endswith('na'))
print(s.count('a'), s.count('n'), s.count('an'))
print(s.replace('a', 'o'), s.replace('an', ''))
Answer
BANANA banana
True False True
3 2 2
bonono ba
upper returns a new string; s itself does not change.
startswith and endswith return True or False. Uppercase and lowercase letters are different, so 'banana' does not start with 'Ban'.
count returns the number of times its argument appears: 'an' appears twice in 'banana'.
replace replaces every occurrence. Replacing 'an' with the empty string removes both ans: s.replace('an', '') is 'ba'.
Question 4 (easy): write a function#
Write a function called capitalize_first that takes a string word and returns a new string: the same as word, but with the first letter in uppercase.
Start from this header (replace ... with the body):
def capitalize_first(word):
...
Examples
print(capitalize_first('hello'))
Hello
print(capitalize_first('python'), capitalize_first('Already'))
Python Already
Answer
Make a new string from the first letter in uppercase and the rest of the word:
def capitalize_first(word):
return word[0].upper() + word[1:]
Then try: print(capitalize_first('hello')).
word[0] = word[0].upper() does not work: strings are immutable, so an assignment to one character is a TypeError.
The version above fails for the empty string, because ''[0] is an IndexError. A slice never fails, so this version also works for '' (it returns ''):
def capitalize_first(word):
return word[:1].upper() + word[1:]
Question 5 (easy): what is displayed?#
print('apple' < 'banana', 'cherry' < 'banana')
print('Zebra' < 'apple', 'zebra' < 'apple')
print('ban' < 'banana', 'cat' == 'Cat', 'cat' == 'Cat'.lower())
Answer
True False
True False
True False True
< compares strings in alphabetical order, but all the uppercase letters come before all the lowercase letters. So 'Zebra' comes before 'apple', but 'zebra' does not.
A string that is the beginning of a longer string comes before it, so 'ban' < 'banana' is True.
== is True only if the two strings are exactly the same, so 'cat' == 'Cat' is False. Convert to lowercase first to ignore the difference.
Question 6 (medium): what happens?#
For each part, predict what happens: does it display something, or is it an error? Then open the Answer to check.
Part a
fruit = 'banana'
print(fruit[6])
Part b
fruit = 'banana'
print(fruit[6 / 2])
Part c
greeting = 'Hello'
greeting[0] = 'J'
Part d
fruit = 'banana'
print(fruit[2:100])
Answer
Part a: fruit has 6 characters, with indices 0 to 5. There is no character with index 6, so this is an IndexError. The last character is fruit[5], or fruit[-1].
1 fruit = 'banana'
----> 2 print(fruit[6])
IndexError: string index out of range
Part b: 6 / 2 is 3.0, a float, and an index must be an integer, so this is a TypeError. With integer division, fruit[6 // 2] is 'a'.
1 fruit = 'banana'
----> 2 print(fruit[6 / 2])
TypeError: string indices must be integers, not 'float'
Part c: Strings are immutable: an assignment to one character of a string is a TypeError. To get 'Jello', make a new string: 'J' + greeting[1:].
1 greeting = 'Hello'
----> 2 greeting[0] = 'J'
TypeError: 'str' object does not support item assignment
Part d: This is not an error. In a slice, an index after the end of the string is allowed: the slice stops at the end of the string.
nana
Question 7 (medium): what is displayed?#
greeting = 'Hello, world!'
new_greeting = 'J' + greeting[1:]
greeting.replace('world', 'Python')
shout = greeting.upper()
print(new_greeting)
print(greeting)
print(shout)
Answer
Jello, world!
Hello, world!
HELLO, WORLD!
Strings are immutable, so nothing in this code changes greeting.
'J' + greeting[1:] makes a new string, which is assigned to new_greeting.
replace also returns a new string, but on the third line that string is not assigned to anything, so it is lost. To keep it, assign it: greeting = greeting.replace('world', 'Python').
upper is invoked on the original greeting, which still contains world.
Question 8 (medium): what is displayed?#
word = 'stop'
for i in range(len(word)):
print(i, word[i], word[-1 - i], word[i:])
Answer
0 s p stop
1 t o top
2 o t op
3 p s p
len(word) is 4, so i is 0, 1, 2 and 3.
word[i] goes forward through the string. word[-1 - i] is word[-1], then word[-2], and so on, so it goes backward from the end.
word[i:] is the rest of the string, starting at index i.
Question 9 (medium): write a function#
Write a function called rotate that takes a string word and an integer n from 0 to len(word), and returns a new string: the same letters, with the first n letters moved to the end.
Start from this header (replace ... with the body):
def rotate(word, n):
...
Examples
print(rotate('python', 2))
thonpy
print(rotate('banana', 1), rotate('abc', 0), rotate('abc', 3))
ananab abc abc
Answer
Two slices: the part from index n to the end, then the first n letters:
def rotate(word, n):
return word[n:] + word[:n]
Then try: print(rotate('python', 2)).
Another way: a loop that adds the letters one at a time. The letter at index i of the result is the letter at index i + n of word; the % operator makes the index go back to 0 after the end of the word:
def rotate(word, n):
result = ''
for i in range(len(word)):
result += word[(i + n) % len(word)]
return result
Question 10 (medium): write a function#
Write a function called censor that takes two strings, text and word, and returns a new string: the same as text, but with each occurrence of word replaced by as many * characters as word has letters.
Start from this header (replace ... with the body):
def censor(text, word):
...
Examples
print(censor('the darn cat is darn cute', 'darn'))
the **** cat is **** cute
print(censor('banana', 'an'))
print(censor('banana', 'x'))
b****a
banana
Answer
The replace method replaces every occurrence, and '*' * len(word) is a string of as many * as word has letters:
def censor(text, word):
return text.replace(word, '*' * len(word))
Then try: print(censor('the darn cat is darn cute', 'darn')).
replace does not change text: it returns a new string, and the function returns that string.
Question 11 (medium): what is displayed?#
for letter in 'programming':
if letter == 'g':
break
print(letter, end=' ')
print()
print('stopped at', letter)
Answer
p r o
stopped at g
The loop displays each letter until it gets to the first g. Then break ends the loop immediately, before the print in the loop, so the g is not displayed by the loop, and the rest of the word is never reached.
After the loop, letter still refers to the letter that made the loop end, 'g'.
print() with no arguments ends the line that end=' ' left open.
Question 12 (medium): what is displayed?#
writer = open('notes.txt', 'w')
writer.write('one\n')
writer.write('two ')
writer.write('three\n')
writer.write('four\n')
writer.close()
count = 0
for line in open('notes.txt'):
count += 1
print(count, line.strip())
Answer
1 one
2 two three
3 four
open('notes.txt', 'w') creates the file (or replaces its contents), and write writes each string to it.
write does not add a newline: only '\n' ends a line. So 'two ' and 'three\n' are on the same line, and the file has 3 lines.
close makes sure that everything written is in the file before the loop reads it.
Question 13 (medium): what is displayed?#
re is the module for regular expressions; it was imported with import re.
text = 'I am Dracula; and I bid you welcome.'
result = re.search('Dracula', text)
print(result.group())
print(result.span())
print(text[5:12])
print(re.search('Count', text))
Answer
Dracula
(5, 12)
Dracula
None
re.search finds 'Dracula' in text and returns a Match object. group returns the part of text that matched.
span returns two numbers in parentheses: the index where the match starts, and the index just after it ends. These are the indices of a slice, so text[5:12] is the part that matched.
'Count' does not appear in text, so re.search returns None: re.search('Count', text) == None is True.
By the way: a pair of values in parentheses, such as (5, 12), is a tuple. Tuples are a topic of their own.
Question 14 (medium): write a function#
Write a function called reverse that takes a string and returns a new string with the same characters in reverse order.
Start from this header (replace ... with the body):
def reverse(word):
...
Examples
print(reverse('stressed'))
desserts
print(reverse('abc'), reverse('a'), '[' + reverse('') + ']')
cba a []
Answer
Start with the empty string, and put each letter in front of the letters so far:
def reverse(word):
result = ''
for letter in word:
result = letter + result
return result
Then try: print(reverse('stressed')).
Or loop through the indices, and add the letters from the end, with negative indices:
def reverse(word):
result = ''
for i in range(len(word)):
result += word[-1 - i]
return result
For the empty string, the loop runs zero times, so both versions return ''.
Question 15 (hard): what is displayed?#
The function has_match returns True if the regular expression pattern matches some part of text, and False otherwise (re is the module for regular expressions; it was imported with import re).
Run this cell to define has_match.
def has_match(pattern, text):
return re.search(pattern, text) != None
Predict what this code displays:
print(has_match('cat|dog', 'hotdog'), has_match('^dog', 'hotdog'), has_match('dog$', 'hotdog'))
print(has_match('^hot$', 'hotdog'), has_match('^hotdog$', 'hotdog'))
print(has_match('colou?r', 'color'), has_match('colou?r', 'colouur'))
print(has_match('cent(er|re)', 'recent'), has_match('center|re', 'recent'))
Answer
True False True
False True
True False
False True
Line 1: 'cat|dog' matches cat or dog, and hotdog contains dog. ^ matches only at the beginning of the string and $ only at the end, so '^dog' does not match, but 'dog$' does.
Line 2: with ^ at the beginning and $ at the end, the pattern must match the whole string.
Line 3: ? makes only the u optional: colou?r matches color and colour, but not two us.
Line 4: in 'cent(er|re)', the parentheses make | apply only to er and re, so the pattern matches only center and centre. Without them, 'center|re' matches center or re, and recent starts with re.
Question 16 (hard): what is displayed?#
re is the module for regular expressions; it was imported with import re.
text = 'My favourite colour is the colour of the sea.'
print(re.sub('colou?r', 'color', text))
print(re.sub('favou?rite', 'favorite', text))
print(re.sub('^My', 'Your', text))
print(re.sub('sea$', 'sky', text))
print(text)
Answer
My favourite color is the color of the sea.
My favorite colour is the colour of the sea.
Your favourite colour is the colour of the sea.
My favourite colour is the colour of the sea.
My favourite colour is the colour of the sea.
re.sub replaces every part of the string that matches the pattern, so both colours become color. Each call starts from the original text, so the first line does not change favourite.
'^My' matches My at the beginning of the string.
'sea$' does not match anything: text ends with sea., and the period is the last character. So re.sub returns the string unchanged.
Strings are immutable, so re.sub returns new strings and text does not change.
Question 17 (hard): what is displayed?#
writer = open('story.txt', 'w')
writer.write('Title\n*** START\nOnce upon a time\nThe end\n*** END\nLicense\n')
writer.close()
reader = open('story.txt')
for line in reader:
if line.startswith('*** '):
break
for line in reader:
if line.startswith('*** '):
break
print(line.strip())
print(line.strip())
reader.close()
Answer
Once upon a time
The end
*** END
The file has 6 lines. The first loop reads Title and *** START, and break ends it at *** START.
reader keeps track of where it is in the file, so the second loop continues with the next line, Once upon a time. It displays each line until it gets to *** END, and break ends it there.
After the loop, line still refers to '*** END\n', so the last print displays it. The line License is never read.
Question 18 (hard): write a function#
A palindrome is a word that reads the same backward and forward, such as racecar.
Write a function called is_palindrome that takes a string and returns True if it is a palindrome and False otherwise. Uppercase and lowercase letters count as the same letter.
Start from this header (replace ... with the body):
def is_palindrome(word):
...
Examples
print(is_palindrome('racecar'), is_palindrome('Noon'))
True True
print(is_palindrome('python'), is_palindrome('a'), is_palindrome(''))
False True True
Answer
Compare each letter with the letter at the same distance from the end. As soon as two letters are different, the answer is False:
def is_palindrome(word):
word = word.lower()
for i in range(len(word)):
if word[i] != word[-1 - i]:
return False
return True
Then try: print(is_palindrome('racecar'), is_palindrome('Noon')).
Or make the reversed string and compare it with the word:
def is_palindrome(word):
word = word.lower()
backward = ''
for letter in word:
backward = letter + backward
return word == backward
Both versions call lower first, so that N and n are the same letter.
The first version checks each pair of letters twice; a loop over range(len(word) // 2) would be enough.
Question 19 (hard): write a function#
Write a function called count_words that takes a regular expression pattern and returns the number of words in the file words.txt that the pattern matches.
words.txt has one word per line, in lowercase.
Start from this header (replace ... with the body):
def count_words(pattern):
...
Examples
print(count_words('^q'), count_words('ology$'))
568 100
print(count_words('^(un|re)'), count_words('colou?r'))
6614 57
This question uses has_match from Question 15.
Answer
A counter, and a loop through the lines of the file:
import re
def count_words(pattern):
count = 0
for line in open('words.txt'):
word = line.strip()
if re.search(pattern, word) != None:
count += 1
return count
Then try: print(count_words('^q'), count_words('ology$')).
Each line is stripped first, so that the newline is not part of the word.
With the function has_match from Question 15, the test is shorter:
def count_words(pattern):
count = 0
for line in open('words.txt'):
if has_match(pattern, line.strip()):
count += 1
return count
Question 20 (hard): write a function#
Write a function called wordle_feedback that takes two strings of the same length, target and guess, and returns a string with one character for each letter of guess:
G if the letter is at the same position in target, Y if it is somewhere else in target, and - if it is not in target.
Start from this header (replace ... with the body):
def wordle_feedback(target, guess):
...
Examples
print(wordle_feedback('mower', 'tried'))
-Y-G-
print(wordle_feedback('spade', 'spade'), wordle_feedback('spade', 'clerk'))
GGGGG --Y--
By the way: this is the information that the word game Wordle gives after each guess. (The real game handles letters that appear twice in a different way.)
Answer
Loop through the indices, so that the letters of guess and target at the same position can be compared:
def wordle_feedback(target, guess):
result = ''
for i in range(len(guess)):
if guess[i] == target[i]:
result += 'G'
elif guess[i] in target:
result += 'Y'
else:
result += '-'
return result
Then try: print(wordle_feedback('mower', 'tried')).
Or loop through the letters of guess, and keep the index in a variable of your own:
def wordle_feedback(target, guess):
result = ''
i = 0
for letter in guess:
if letter == target[i]:
result += 'G'
elif letter in target:
result += 'Y'
else:
result += '-'
i += 1
return result
The order of the tests is important: check for G first, because a letter at the right position is also in target.
Summary and questions by Prof. Y. Rosenthal, based on Think Python: 3rd Edition by Allen B. Downey. Text license: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International