Monash University Faculty of Information Technology 2nd Semester 2020 FIT2014 Assignment 1 Linux tools, regular expressions, induction DUE: 11:55pm, Friday 28 August 2020 How to manage this assignment • Do as much as possible of it before your week 4 prac class. There will not be time during the class itself to do the assignment from scratch; there will only be time to get some help and clarification. Instructions Please read these instructions carefully before you attempt the assessment: • To begin working on the assignment, download the asgn1.tar.gz workbench from Moodle and unpack it. Be sure to configure and test the submission-builder before you begin working. Refer to Lab 0 for a reminder on how to do all of these tasks. • The workbench provides locations and names for all solution files. These will be empty, needing replacement. Do not add or remove files from the workbench. All solutions must be submitted to Moodle by the deadline above, and must be packaged using the workbench’s makefile system. • Solutions to written questions must be submitted as PDF documents. You can create a PDF file by scanning your legible (use a pen, write carefully, etc.) hand-written solutions, or by directly typing up your solutions on a computer. • Start work on this assignment early, and try to make substantial progress before your sched- uled lab class: your tutor won’t have time to help you start from scratch. Before you attempt any problem—or seek help on how to do it—be sure to read and understand the question, as well as any accompanying code. • To aid the marking process, you must adhere to all naming conventions that appear in the assignment materials, including files, directories, code, and mathematics. Not doing so will cause your submission to incur a one-day late-penalty (in addition to any other late-penalties you might have). Be sure to check your work carefully. Your submission must include: • a sed script, decomposeSyllablesIntoParts, for Problem 1(a); • a one-line text file, MandarinHanyuPinyin-NameStructure2, for Problem 1(a); • a sed script, decomposePartsIntoLetters, for Problem 1(b); • a one-line text file, MandarinHanyuPinyin-NameStructure3, for Problem 1(b); • an awk script, matchMandarinHanyuPinyin, for Problem 1(c); • the output file outputFile you produced by running your awk script on the provided input file, inputFileOfNames, in Problem 1(c); • a file prob2.pdf with your solution to Problem 2. To submit your work, simply enter the command ‘make’ from within the asgn1 directory, and then submit the resulting .tar.gz file to Moodle. Make sure that you have tested the submission mechanism and that you understand the effect of make on your directory tree. (You first practised this in Lab 0, §1.3.) 1 Introduction to the Assignment In Lab 0, you met the stream editor sed, which detects and replaces certain types of patterns in text, processing one line at a time. These patterns are actually specified by regular expressions. You will use sed again in Problem 1 of this Assignment, to help construct regular expressions. You will also learn about awk, which is a simple programming language that is widely used in Unix/Linux systems and which also uses regular expressions. In Problem 1, you will construct an awk program to identify a class of Chinese names. Finally, Problem 2 is about applying induction to a problem of counting strings. Introduction to awk In an awk program, each line has the form /pattern / { action } where the pattern is a regular expression (or certain other special patterns) and the action is an instruction that specifies what to do with any line that contains a match for the pattern. The action (and the {. . . } around it) can be omitted, in which case any line that matches the pattern is printed. Once you have written your program, it does not need to be compiled. It can be executed di- rectly, by using the awk command in Linux: $ awk -f programName inputFileName Your program is then executed on an input file in the following way. // Initially, we’re at the start of the input file, and haven’t read any of it yet. If the program has a line with the special pattern BEGIN, then do the action specified for this pattern. Main loop, going through the input file: { inputLine := next line of input file Go to the start of the program. Inner loop, going through the program: { programLine := next line of program (but ignore any BEGIN and END lines) if inputLine contains a string that matches the pattern in programLine, then if there is an action specified in the programLine, then { do this action } else just print inputLine // it goes to standard output } } If the program has a line with the special pattern END, then do the action specified for this pattern. Any output is sent to standard output. You should read about the basics of awk, including the way it represents regular expressions and the main instruction types used in its actions. Any of the following sources should be a reasonable place to start: • A. V. Aho, B. W. Kernighan and P. J. Weinberger, The AWK Programming Language, Addison-Wesley, New York, 1988. 2 (The first few sections of Chapter 1 should have most of what you need, but be aware also of the regular expression specification on p28.) • http://www.grymoire.com/Unix/Awk.html • http://www.hcs.harvard.edu/~dholland/computers/awk.html (a good, short introduc- tion) • the Wikipedia article looks ok • the awk manpage. Introduction to Problem 1 The Master said, ‘What is necessary is to rectify names . . . . If names are not recti- fied, then words are not appropriate. If words are not appropriate, then deeds are not accomplished.’ – Confucius (孔夫子), The Analects (transl. R. Dawson), Oxford University Press, 1993, §13.3. Most organisations today deal with people from many different cultures and language groups, and they must often record and process people’s names in systems that work mainly with English language text. In such contexts, it is helpful to be able to recognise names from different language groups. Example applications include: determining how to pronounce students’ names when reading them out from a list at graduation ceremonies; determining how to greet a person with whom you have an appointment; determining how to enter the various parts of a person’s name into a database; determining how automatically-generated emails, sent to many different people listed in some file, should address each recipient; determining the most likely native language of a person in situations where their name is known but they cannot be spoken to directly at the time (e.g., in some emergency situations). Recognising the language group that a name belongs to is an important first step in all these situations. In this problem you will write some code in sed and awk to try to recognise Chinese language names in a long file of Asian names. More specifically, suppose you are given an input file in which each line starts with a person’s name in some language, with each name transcribed somehow into English text. Your task is to detect which of these names come from Mandarin Chinese transcribed using Ha`nyuˇ P¯ıny¯ın (which is the most standard way of representing Mandarin Chinese using strings of English letters). In the input file, all text from the start of each line until the first colon (:) on the line (but not including the colon itself) is taken to be a person’s name. In most cases, each line ends with a string of non-blank letters specifying which language the name is believed to come from. An example input file is provided, as inputFileOfNames. If you browse through the file, you will notice that it contains names from a variety of Asian languages: Mandarin, Cantonese, Hokkien, Teochew, Hakka, Korean, Japanese, Thai, Vietnamese, Malay and Indonesian. They have been represented in English text using a variety of transcription schemes, and with all extra marks on letters (accents, tone marks, other diacritical marks, etc.) removed. 1 In many cases, the line about a person also contains some other information a
bout them, but our name recognition task will ignore that information.2 Further information about working with names from different cultures can be found in: • Fiona Swee-Lin Price, Success with Asian Names, Allen & Unwin, Crows Nest, NSW, 2007. 1These marks carry important information about meaning and pronunciation. But they are often removed when names are represented using other alphabets. 2The file was compiled from a number of sources, mainly Wikipedia lists of names of type https://en.wikipedia.org/wiki/List of CultureName people, where CultureName is one of the cultures listed above; also https://www.goratings.org/en/. The lists obtained from Wikipedia are rather imperfect, with peo- ple’s names often not written in a form that clearly shows the claimed cultural background. 3 • J. Greenberg Motamedi, Z. Jaffery, A. Hagen, and S. Y. Yoon, Getting it right: Reference guides for registering students with non-English names, 2nd edition. (REL 2016-158 v2), U.S. Depart- ment of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest, Washington, DC, 2017. https://ies.ed.gov/ncee/edlabs/regions/northwest/pdf/REL_2016158.pdf Names in Mandarin Mandarin Chinese names consist of a family name followed by a given name (the reverse of the usual order in English). The family name usually consists of just a single syllable, and the given name usually consists of one or two syllables. We restrict ourselves to names with these numbers of syllables from now on.3 We treat a Mandarin name as consisting of a syllable, followed by a space, followed by either a single syllable or two consecutive syllables (concatenated, so there is no space between that pair of syllables). Each syllable consists of either a prefix followed by a suffix, or a standaloneSuffix (which is a suffix that can be a whole syllable in its own right; other suffixes must follow a prefix).4 • The possibilities for the prefix are: b, p, m, f, d, t, n, l, g, k, h, j, q, x, zh, ch, sh, r, z, c, s, w, y These are also listed, one per line, in the file prefix-file. • The possibilities for the suffix are: a, e, i, o, u, ai, an, ao, ei, en, ia, ie, in, iu, ou, ua, ue, ui, un, uo, ang, eng, ong, ian, iao, ing, uai, uan, iang, iong, uang These are also listed, one per line, in the file suffix-file. • The possibilities for the standaloneSuffix are: ai, ei, ao, ou, an, ang, en, eng, er These are also listed, one per line, in the file standaloneSuffix-file. For example, the syllable zhong is formed from the prefix zh followed by the suffix ong. Note that ong does not appear in the list of standaloneSuffixes, so it cannot be a syllable in its own right; it always needs a prefix before it. The syllable ang consists just of the standaloneSuffix ang.5 Constructing a regular expression for Mandarin names We will use this information, together with sed, to generate a regular expression to try to match Mandarin Ha`nyuˇ P¯ıny¯ın names. (The match will not be perfect, and later you will do some simple computations to get some idea of how well, or poorly, it does.6) Throughout, we will only do case- insensitive matching, so there is no need to capitalise initial letters of syllables in order to match names. 3There are exceptions, though they are uncommon: some family names have two or even more syllables, and some given names have more than two syllables. But in this assignment we consider only one-syllable family names and one- or two-syllable given names. 4This terminology is different to the more usual linguistic terminology of initials and finals. This is deliberate, since they represent slightly different sets of strings here to the standard initials and finals. This change was just a small simplification for the purposes of the assignment. 5The only standaloneSuffix that is not also a suffix is er. 6Some reasons for the imperfect matching include: not all the combinations of prefixes and suffixes listed above give valid syllables in Mandarin; some people write Mandarin Ha`nyuˇ P¯ıny¯ın names with a hyphen or a space between the two syllables of the given name; and some family or given names may have more syllables than we have allowed for. 4 We describe the steps in detail now, and then list the assignment submission requirements for Problem 1. We start with the file MandarinHanyuPinyin-NameStructure0, which just contains one line consisting of the text . Think of this as a special symbol, eventually to be transformed to a regular expression for Ha`nyuˇ P¯ıny¯ın names. We first transform to a regular expression representing the syllable structure of Ha`nyuˇ P¯ıny¯ın names. From our description of the syllable structure above, we can represent this by the regular expression ? where is a special symbol to represent any Ha`nyuˇ P¯ıny¯ın syllable, and putting “?” after a symbol means it can occur just once or not at all at that position.7 Each occurrence of will eventually be transformed to a regular expression to represent such syllables. The first step of this transformation can be done using the following sed command: $ sed “s// ?/” MandarinHanyuPinyin-NameStructure0 > MandarinHanyuPinyin-NameStructure1 Alternatively, you can put the sed instruction “s// ?/” into the file decomposeNameIntoSyllables (provided with this assignment) and do the following: $ sed -f decomposeNameIntoSyllables MandarinHanyuPinyin-NameStructure0 > MandarinHanyuPinyin-NameStructure1 Try these, using the given files, and check that you get the required result in the file MandarinHanyuPinyin-NameStructure1. This use of sed is a template for what you will do next. Now you need to replace each by a regular expression representing possible syllable structures, using , and . Write the sed instruction to do this, in the file decomposeSyllablesIntoParts, and do the transformation from your Linux command line using $ sed -f decomposeSyllablesIntoParts MandarinHanyuPinyin-NameStructure1 > MandarinHanyuPinyin-NameStructure2 Now you need to replace each occurrence of each part, , and , by regular expressions representing possible letter strings, according to the sets of strings given on the previous page. Write the sed instructions to do this, in the file decomposePartsIntoLetters. You will need three lines: one line giving the sed instruction for each part type, , and . Then do the transformation from your Linux command line using $ sed -f decomposePartsIntoLetters MandarinHanyuPinyin-NameStructure2 > MandarinHanyuPinyin-NameStructure3 This should put, into the file MandarinHanyuPinyin-NameStructure3, a regular expression that is intended to match Ha`nyuˇ P¯ıny¯ın names. Matching names in the input file We will now use this regular expression to construct an awk program to apply it to names in the input file, and to do some simple computations to determine how well the regular expression matches Ha`nyuˇ P¯ıny¯ın names. For each name in the input file, checking it against your regular expression gives one of four possible outcomes: 7The operator “?” is a standard feature of regular expression syntax in sed and awk. Question to consider (but not part of the assignment): does using it enlarge the class of languages that can be represented by regular expressions? 5 • Correct Match: a name that belongs to Mandarin Ha`nyuˇ P¯ıny¯ın (according to the last word on its line in inputFileOfNames) is also matched by your regular expression. • False Positive: a name that does not belong to Mandarin Ha`nyuˇ P¯ıny¯ın (according to the last word on its line in inputFileOfNames), but is matched by your regular expression. • False Negative: a name that belongs to Mandarin Ha`nyuˇ P¯ıny¯ın (according to the last word on its line in inputFileOfNames), but is not matched by your regular expression. • Correct Non-match: a name that does not belong to Mandarin Ha`nyuˇ P¯ıny¯ın (according to the last word on its line in inputFileOfNames) and is not matched by your regular expression. Your awk program should compute the number of names, in the input file, of each of thes
e four types, and report those numbers at the end. You can write the awk program manually, cutting-and-pasting the regular expression as needed (and making any necessary modifications to it) from MandarinHanyuPinyin-NameStructure3. The heart of your awk program will be a statement that does the following. • The pattern matches a Mandarin Ha`nyuˇ P¯ıny¯ın name whenever it occurs in the specified position at the start of a line. – Recall the general structure of an awk statement. By default, matches in awk are case- sensitive. To do a case-insensitive match, you need to match the pattern against a version of the current line in which all alphabetic characters have been converted to lower case. This can be done using a statement of this form: tolower($0) ~ /pattern / { action } Here, $0 is a built-in awk variable that always contains the current line of the input file, and tolower() is a function that converts all alphabetic characters to lower case. You can read “ ~ ” as “matches”. • The action increments appropriate variables in order to update the number of correct matches, false positives, false negatives, or correct non-matches, as appropriate. You may need another line or few too, including an END line, at the end, in order to print the total numbers of matches of each type. The files and commands required to complete this problem are shown in Figure 1. Problem 1. [12 marks] First, write the one-line sed script, decomposeNameIntoSyllables, and run sed, as described, to produce MandarinHanyuPinyin-NameStructure1. (a) Write the one-line sed script, decomposeSyllablesIntoParts, and run sed, as described, to produce MandarinHanyuPinyin-NameStructure2. (b) Write the three-line sed script, decomposePartsIntoLetters, and run sed, as described, to produce MandarinHanyuPinyin-NameStructure3. (c) Write the awk script, matchMandarinHanyuPinyin, and run it, as described, on input file inputFileOfNames, to produce outputFile. For bonus marks: can you write a much better regular expression for Mandarin Ha`nyuˇ P¯ıny¯ın names? If you attempt this, then, in addition to the files required above, you should also provide: • your new regular expression, in a one-line file called MandarinHanyuPinyin-NameStructure4; 6 MandarinHanyuPinyin-NameStructure0 sed -f decomposeNameIntoSyllables MandarinHanyuPinyin-NameStructure0 > MandarinHanyuPinyin-NameStructure1 MandarinHanyuPinyin-NameStructure1 sed -f decomposeSyllablesIntoParts MandarinHanyuPinyin-NameStructure1 > MandarinHanyuPinyin-NameStructure2 MandarinHanyuPinyin-NameStructure2 sed -f decomposePartsIntoLetters MandarinHanyuPinyin-NameStructure2 > MandarinHanyuPinyin-NameStructure3 MandarinHanyuPinyin-NameStructure3 write awk code, using regular expression in MandarinHanyuPinyin-NameStructure3 matchMandarinHanyuPinyin awk -f matchMandarinHanyuPinyin inputFileOfNames > outputFile inputFileOfNames decomposeNameIntoSyllables decomposeSyllablesIntoParts decomposePartsIntoLetters outputFile Figure 1: The plan. 7 • your new awk program, in a file called matchMandarinHanyuPinyinBonus; • the output from running your awk program on the input file, as outputFileBonus. To qualify for bonus marks, your approach must use a significantly different approach to the one used above, and offer a significant advantage. If your approach is more complex, then it should give better results (i.e., lower error rate) in general. If your approach is significantly simpler but does almost as well, then that also may qualify for bonus marks. Your solution will not just be evaluated on how well it does on the provided input file. It will be run on other files of names too. Also, do not base your regular expression solely on the specific details of the Ha`nyuˇ P¯ıny¯ın names provided in the input file. They may not be totally representative of the full range of names from that culture. As usual, be sure to cite any sources used. Problem 2. [8 marks] This problem is about counting the number of different strings of South Korean family names, according to the number of characters in the sequence. According to Wikipedia8, there are 286 different Korean family names used in South Korea (using 2015 data). Of these, 274 consist of a single character, while 12 consist of two characters.9 Let fn be the number of strings of South Korean family names in which each name is from the list of 286 names mentioned above, and the entire string consists of exactly n characters. Notes and clarifications: • The number of names in a string of n characters will also equal n only if every name in the string has just one character. The string will have < n names if it includes any two-character names. • There are no separators (such as commas) in the strings. They consist of sequences of charac- ters, with nothing else. • An example string of three characters: “망절김” (using Hangul; in Hanja, it would be “網切金”), which consists of two names: the rare two-character family name망절 (Mangjeol) followed by the one-character family name 김 (Kim) which is the most common of all. (a) Find f1 and good upper bounds for f2 and f3. (b) Give (with justification) an expression for an upper bound for fn in terms of fn−1 and fn−2, that works for all n ≥ 3. (c) Prove, by induction on n, that fn ≤ 274.044n for all n. 8https://en.wikipedia.org/wiki/List_of_Korean_surnames . . . but the data there seems to pertain to c2000, rather than 2015. 9Although the Korean language has its own characters (Hangul), people’s names are usually written using Chinese characters (Hanja). The number of characters used to write a person’s name is almost always the same, whether Hangul or Hanja is used. For simplicity, we assume each name has a unique representation using characters; this is almost correct for Hangul, but less accurate for Hanja. 8
