Write the code to read the raw training data into the data structure in the first approach described in the section called “The Raw Data”. That is, the data structure is a data frame with a column for each MAC address that detected a signal. For the column name, use the last two characters of the MAC address, or some other unique identifier.

Compare the size of two data structures: the data frame created in the section called “The Raw Data” and the data frame created in the previous problem. Which uses less memory? What is the dimension of each? How might this change with different numbers of devices in the building? different number of signals from the less commonly detected devices? Use object.size() and dim() to address these questions.

Compare the time taken to read the raw data and create the data frame using the two approaches described in the section called “The Raw Data” both approaches? Do this for different size subsets of the data (chosen at random) and draw a curve of input size and time taken to read the data? Comment on the memory and speed for the two approaches. Use system.time() and Rprof() to make these comparisons.

Examine the time variable in the offline data. Any change over time in the characteristics of the signal caused by, e.g., reduced battery power in the measuring device as time goes by, or measurements taken on different days may be made by different people with different levels of accuracy. Also, examination of time can give insight into how the experiment was carried out. Were the positions close to each other measured at similar times? Do you see any change in the signal strength variability or mean over time? Try controlling for other variables that might affect this relationship.

Write the readData() function described in the section called “Creating a Function to Prepare the Data”. The arguments to this function are the file name, filename and the MAC addresses to retain, subMacs. Determine whether these parameters should have default values or not. The return value is the data frame described in the section called “Cleaning the Data and Building a Representation for Analysis”. Use the findGlobals() function available in codetools to check that the function is not relying on any global variables.

In the section called “Distribution of Signal Strength” we calculated measures of center and location for the signal strengths at each location angle access point combination. (See for example Figure 1.9, “Comparison of Mean and Median Signal Strength”.) Another possible summary statistic we can calculate is the Kolmogorov-Smirnov test-statistic for normality. If the signal strengths are roughly normal, then we expect the -values to have a uniform distribution. This leads to about 5% of the -values for the 8000 tests to fall below 0.05.

Write the surfaceSS() function that creates plots such as those in Figure 1.10, “Median Signal at Two Access Points and Two Angles”. This function has three arguments: data for the offline summary data frame, and mac and angle which supply, respectively, the MAC address and angle to select the subset of the data which we want smoothed and plotted.

Consider the scatter plots in Figure 1.11, “Signal Strength vs. Distance to Access Point”. There appears to be curvature in the signal strength – distance relationship. Does a log transformation improve this relationship, i.e., make it linear? Note that the signals are negative values so we need to be careful if we want to take the log of signal strength.

The floor plan for the building (see Figure 1.1, “Floor Plan of the Test Environment”) shows six access points. However, the data contain seven access points with roughly the expected number of signals (166 location 8 orientations 110 replications 146,080 measurements). With the signal strength seen in the heat maps of Figure 1.10, “Median Signal at Two Access Points and Two Angles”), we matched the access points to the corresponding MAC address. However, two of the MAC addresses seem to be for the same access point. In the section called “Exploring MAC Address” we decided to keep the measurements from the MAC address 00:0f:a3:39:e1:c0 and to eliminate the 00:0f:a3:39:dd:cd address. Conduct a more thorough data analysis into these two MAC addresses. Did we make the correct decision? Does swapping out the one we kept for the one we discarded improve the prediction?

Write the selectTrain() function described in the section called “Choice of Orientation”. This function has three parameters: angleNewObs, the angle of the new observation; signals, the training data, i.e., data in the format of offlineSummary; and m, the number of angles to include from signals. The function returns a data frame that matches trainSS, i.e., selectTrain() calls reshapeSS() (see the section called “Choice of Orientation” for this function definition).

We use Euclidean distance to find the distance between the signal strength vectors. However, Euclidean distance is not robust in that it is sensitive to outliers. Consider other metrics such as the distance, i.e., the absolute value of the difference. Modify the findNN() function in the section called “Finding the Nearest Neighbors” to use this alternative distance. Does it improve the predictions?

To predict location, we use the nearest neighbors to a set of signal strengths. We average the known values for these neighbors. However, a better predictor might be a weighted average, where the weights are inversely proportional to the ``distance'' (in signal strength) from the test observation. This allows us to include the points that are close, but to differentiate between them by how close they actually are. The weights might be for the -th closest neighboring observation where is the distance from our new test point to this reference point (in signal strength space). Implement this alternative prediction method. Does this improve the predictions? Use calcError() to compare this approach to the simple average.

In the section called “Cross-Validation and Choice of ” we used cross-validation to choose , the number of neighbors. Another parameter to choose is the number of angles at which the signal strength was measured. Use cross-validation to select this value. You might also consider selecting the pair of parameter, i.e., and the number of angles, simultaneously.

The researchers who collected these data implemented a Bayesian approach to predicting location from signal strength. Their work is described in a paper which is available from https://www.informatik.uni-mannheim.de/pi4/publications/King2006g.pdf . Consider implementing this approach to building a statistical IPS.

Other statistical techniques have been developed to predict indoor positions from wireless local area networks. These include[bib:Krishnan, bib:Madigan and bib:Youssef]. Consider employing one their approaches to building and testing a statistical IPS with the CRAWDAD data.

Write a function that uses read.fwf() to read the 28 text tables in MenTxt/ and WomenTxt/ into R. These are called 1999.txt, 2000.txt, etc. and are described in greater detail in the section called “Reading Tables of Race Results into R”. Examine the tables in a plain text editor to determine the start and end position of each column of interest (name, hometown, age, and gun and net time). Use statistics to explore the results and confirm that you have extracted the information from the correct positions in the text.

Revise the extractVariables() function (see the section called “Reading Tables of Race Results into R”) to remove the rows in menTables that are blank. In addition, eliminate the rows that begin with a '*' or a '#' . You may find the following regular expression helpful for locating blank rows in a table

grep("^[[:blank:]]*$", body)

The pattern uses several meta characters. The ^ is an anchor for the start of the string, the $ anchors to the end of the string, the [[:blank:]] denotes the equivalence class of any space or tab character, and * indicates that the blank character can appear 0 or more times. All together the pattern ^[[:blank:]]*$ matches a string that contains any number of blanks from start to end.

Find the record where the time is only 1.5. What happened? Determine how to handle the problem and which function needs to be modified: extractResTable() , extractVariables() , or cleanUp() . In your modification, include code to provide a warning message about the rows that are being dropped for having a time that is too small.

Examine the head and tail of the 2006 men's file. Look at both the character matrix in the list called menResMat and the character vector in the list called menFiles (see the section called “Reading Tables of Race Results into R”). (Recall that the desired character matrix in menResMat and the character vector in menFiles both correspond to the element named "2006"). What is wrong with the hometown? Examine the header closely to figure out how this error came about. Modify the extractVariables() function to fix the problem.

Write the convertTime() function described in the section called “Data Cleaning and Reformatting variables”. This function takes a string where time is in either the format hh:mm:ss or mm:ss. The return value is the time as a numeric value of the number of minutes. Design this function to take a character vector with multiple strings and return a numeric vector.

Modify the createDF() function in the section called “Data Cleaning and Reformatting variables” to handle the formatting problem with the 2006 male file. You will need to carefully inspect the raw text file in order to determine the problem. (Hint: ).

Follow the approach developed in the section called “Reading Tables of Race Results into R” to read the files for the female runners and then process them using the functions in the section called “Data Cleaning and Reformatting variables” to create a data frame for analysis. You may need to generalize the createDF() and extractVariables() functions to handle additional oddities in the raw text files.

Modify the call to the plot() function that created Figure 2.6, “Default Scatter Plot for Run Time vs. Age for Male Runners” to create Figure 2.7, “Revised Scatter Plot of Male Runners”. To do this, read the documentation for plot() to determine which parameters could be helpful; that is, ?plot.default contains helpful information about the commonly used graphical parameters.

Modify the piecewise linear fit from the section called “Fitting Models to Average Performance” to include a hing at 70. Examine the coefficients from the fit and compare the fitted curve to the loess curve. Does the additional hing improve the fit? Is the piecewise linear model closer to the loess curve?

We have seen that the 1999 runners were typically older than the 2012 runners. Compare the age distribution of the runners across all 14 years of the races. Use quantile-quantile plots, boxplots, and density curves to make your comparisons. How do the distributions change over the years? Was it a gradual change?

Normalize each male runner's time by the fastest time for the runner of the same age. To do this, find the fastest runner for each year of age from 20 to 80. The tapply() function may be helpful here. Smooth these times using loess() , and find the smoothed time using predict() . Use these smoothed times to normalize each run time. Use density plots, quantile-quantile plots, and summary statistics to compare the distribution of the age-normalized times for the runners in 1999 and 2012. What do you find? Repeat the process for the women. Compare the women in 1999 to the women in 2012 and to the men in 1999 and 2012.

Clean the strings in home in menRes to remove all leading and trailing blanks and multiple contiguous blanks. Also make all letters lower case and remove any punctuation such as '.' or ',' or ''' characters from the string. Assign the cleaned version of home into menRes. Call it homeClean.

In the section called “Constructing a Record for an Individual Runner across Years” we created an id for a runner by pasting together name, year of birth and state. Consider using the home town instead of the state. What is the impact on the matching? How many runners have competed in at least 8 races using this new id? What if you reduced the number of races to 6? Should this additional restriction be used in the matching process?

Further refine the set of athletes in the longitudinal analysis by dropping those IDs (see the section called “Constructing a Record for an Individual Runner across Years”) who have a large jump in time in consecutive races and who did not compete for two or more years in a row. How many unique IDs do you have when you include these additional restrictions? Does the longitudinal analysis in the section called “Modeling the Change in Running Time for Individuals” change?

Follow the procedures developed in the section called “Constructing a Record for an Individual Runner across Years” to clean the female runners names and hometown and create longitudinal records for the females. Then follow the modeling of the section called “Modeling the Change in Running Time for Individuals” to investigate the run time – age relationship for women who competed in multiple races.

Consider adapting a nonparametric curve fitting approach to the longitudinal analysis. Rice[bib:Rice] suggests modeling an individual's behavior as a combination of an average curve plus an individual curve. That is, the predicted performance for an individual comes from the sum of the average curve and the individual's curve: , where is the performance of individual at age . He suggests a “two-step” process to do this: (a) take a robust average of all of the smoothed curves for the individuals; (b) subtract this average smoothed curve from the individual data points and smooth the residuals.

Rather than using only the individual's run times to produce the individual's curve, Rice also suggests smoothing over a set of nearest neighbors' times. Here a nearest neighbor is a runner with similar times for similar age.

In the section called “Scraping Race Results from the Web”, we discovered that the HTML file for the male 2000 results was so poorly formatted that htmlParse() was unable to fix it to allow us to extract the text table from the <pre> tag. In this exercise, we programmatically edit this HTML file so that we can use htmlParse() as desired. To do this, begin by reading the HTML file located at https://www.cherryblossom.org/cb003m.htm using readLines() . Carefully examine the HTML displayed in the section called “Scraping Race Results from the Web” and come up with a plan for correcting it. Consider whether you want to drop <font>s or close them properly. Once you have fixed the problem so that the <pre> tag contains the text table, pass your corrected HTML to htmlParse() . You may want to use a text connection to do this rather than writing the file to disk and reading it in.

Revise the extractResTable() function in the section called “Scraping Race Results from the Web” so that it can read the male 2009 results. Carefully examine the raw HTML to determine how to find the information about the runners. Work with XPath to locate <div> and <pre> tags and extract the text value. The female 2009 results do not need this special handling. Modify the extractResTable() function to determine whether to perform the special processing of the <div> and <pre> tags.

Revise the extractResTable() function in the section called “Scraping Race Results from the Web” so that it takes an additional parameter: file. Give the file parameter a default value of NULL. When NULL, the parsed results are returned from extractResTable() as a character vector. If not NULL, the results are written to the file named in file. The writeLines() function should be helpful here.

We hand selected email to belong to the sample set in sampleEmail. Instead of this approach, use the sample function to choose messages at random for the sample. Be sure to take files from all five directories of email. Read these files into R and test several functions with these new messages, e.g., getBoundary() and dropAttach() from the section called “Removing Attachments from the Message Body”, to make sure that they work properly.

In the text mining approach to detecting spam we ignored all attachments in creating the set of words belonging to a message (see the section called “Removing Attachments from the Message Body”). Write a function to extract words from any plain text or HTML attachment and include these words in the set of a message's words. Try to reuse the findMsg() function and modify the dropAttach() function to accept an additional parameter that indicates whether or not the words in attachments are to be extracted. Does this change improve the classification?

The string manipulation functions in R can be used instead of regular expression functions for finding, changing, extracting substrings from strings. These functions include: strsplit() to divide a string up into pieces, substr() to extract a portion of a string, paste() to glue together multiple strings, and nchar() which returns the number of characters in a string. Write your own version of getBoundary() (see the section called “Removing Attachments from the Message Body”) using these functions to extract the boundary string from the Content-Type. Debug your function with the messages in sampleEmail.

Write the dropAttach() function for the section called “Removing Attachments from the Message Body”. This function has two inputs, the body of a message and the boundary string that marks the location of the attachments. It returns the body without its attachments. Include in the return value the lines of the body that follow the first boundary string up to the string marking the first attachment and the lines following the ending boundary string. Be sure to consider the idiosyncratic cases of no attachments and a missing ending boundary string.

Write the function findMsgWords() of the section called “Extracting Words from a Message Body”. This function takes as input the message body (with no attachments) and the return value is a vector of the unique words in the message. That is, we only track which words are in the message, not the number of times it appears there. This function should eliminate punctuation, digits, and blanks from the message. Consider whether it is simpler to split the string by blanks first and then process the punctuation, digits, etc. The function should convert capital letters to lower case, and drop all stop words and words that are only one letter long. A vector of stop words is available in the tm package. Use the stopwords() function in this package to create the vector.

Try to improve the text cleaning in findMsgWords() of the section called “Extracting Words from a Message Body” by stemming the words in the messages. That is, make plural words singular and reduce present and past tenses to their root words, e.g., run, ran, runs, running all have the same “stem”. To do this, use the stemming functions available in the text mining package tm. Incorporate this stemming process into the findMsgWords() function. Then recreate the vectors of words for all the email and see if the classification improves.

Consider the treatment of s in the text cleaning in findMsgWords() of the section called “Extracting Words from a Message Body”. Notice that this function often turns a into gibberish. Should we drop s all together from the messages, or should we try to keep the as one whole “word”? Why might these alternatives be better or worse than the approach taken in the section called “Extracting Words from a Message Body”? Try one of these alternatives and compare it to the approach of that section to see if it improves the classification.

In the section called “Computational Considerations” we saw a few alternative mathematical expressions for the naïve Bayes approximation to the log likelihood ratio of the chance a message is spam or ham. Each suggests a different approach to carrying out the computation. Create alternative versions of computeFreqs() and computeMsgOdds() in the section called “Implementing the Naïve Bayes Classifier” to calculate the log odds. Compare the accuracy of these alternatives to the approach used in the section called “Implementing the Naïve Bayes Classifier”. Also consider timing the computation with a call to system.time() to determine if one approach is much faster or slower than the others.

The function computeFreqs() in the section called “Implementing the Naïve Bayes Classifier” uses as a default bag of words constructed from the words passed to it in the argument wordsList. However, it is possible to supply a different bag of words via the bow parameter. When this happens, it may be the case that some words in wordsList are not found in the bag of words. This causes an error when running the function. Determine which lines of code in computeFreqs() are problematic and update the function to handle this situation. Be sure to test your code.

Use the typeIErrorRates() function in the section called “Classifying New Messages” as a guide to write a function that computes the Type II error rates, i.e., the proportion of spam messages that are misclassified as ham. As with the Type I error rates, convince yourself that the error rate is monotonic in , that it changes only at the values of the LLR in the provided messages, and that you only need to consider these values for the spam messages. This function, called typeIIErrorRates() , has the same inputs as the typeIErrorRates() function and returns the same structure. The only difference is that the rates returned are based on Type II errors.

In the section called “Processing the Header”, we used the read.dcf() function to read the key: value data in the email headers. In this exercise, we use regular expressions to extract the keys and their values from the header.

The first step in the process is to find the continuation lines for a value, and then collapse them with the first line of the value. These continuation lines start with either a blank space or a tab character. Use regular expressions to locate them. Then paste them to the first line of the value.

Next break the revised set of key:value strings into the keys and values. Again use regular expressions to do this. Then create the names vector from these keys and values.

Write the processAttach() function for the section called “Processing Attachments”. This function has two inputs, the body of a message and the Content-Type value. It returns a list of two elements. The first is called body and it contains the message body without its attachments. Be sure to consider the idiosyncratic cases of no attachments and a missing ending boundary string. The second element is a data frame called attachDF. This data frame has one row for each attachment in the message. There are two variables in this data frame, aLen and aType, which hold the number of lines and the MIME type of each attachment, respectively.

Write a function to handle an alternative approach to measure yelling: count the number of lines in the email message text that consist entirely of capital letters. Carefully consider the case where the message body is empty. How do you modify your code to report a percentage rather than a count? In considering this modification, be sure to make clear how you handle empty lines, lines with no alpha-characters, and messages with no text.

Write alternative implementations for the isRe() function in the section called “Deriving Variables from the Email Message”. For one alternative, instead of only checking whether the subject of a message begins Re:, look also for Fwd: Re:. For a second alternative, check for Re: any where in the subject, not just at the beginning of the string. Analyze the output from these three functions, including the original isRe() function in the section called “Deriving Variables from the Email Message”. How many more messages have a return value of TRUE for these alternatives, and are they all ham? Which do you think is most useful in predicting spam?

Choose several of the ideas listed in Table 3.1, “Variable Definition Table” for deriving features from the email and write functions for them. Be sure to check your code against what you expect. Try writing one of these functions in two different ways and compare the output from each. Use exploratory data analysis techniques to check that your code works as expected. Does the output of your function match the values in the corresponding columns of emailDF? If not, why do you think this might be the case? Does it appear that this derived variable will be useful in identifying spam or ham?

Consider other variables that are not listed in Table 3.1, “Variable Definition Table” that might by useful in the classification problem. Write functions to derive them from the messages in emailStruct and add them to emailDF. Refit the classification tree with the enhanced data frame. Were these new variables chosen to partition the messages? Is the error in classification improved?

Carry out additional exploratory analysis as described in the section called “Checking Our Code for Errors”. Include in your analysis a quantile-quantile plot of perCaps for the ham and spam.

Write code to handle the attachments in the message separately from the text in the body of the message. Since each attachment has its own header, try processing the header and body of the attachment in a manner similar to the message header and body. Use the processHeader() function to do this. You may need to revise processHeader() to handle cases that arise in the attachments and not in the main headers of the messages.

Consider the other parameters that can be used to control the recursive partitioning process. Read the documentation for them in the rpart.control() documentation. Also, carry out an Internet search for more information on how to tweak the rpart() tuning parameters. Experiment with values for these parameters. Do the trees that result make sense with your understanding of how the parameters are used? Can you improve the prediction using them?

In the section called “Classifying New Messages” we used the test set that we had put aside to both select , the threshold for the log odds, and to evaluate the Type I and II errors incurred when we use this threshold. Ideally, we choose from another set of messages that is both independent of our training data and our test data. The method of cross-validation is designed to use the training set for training and validating the model. Implement 5-fold cross-validation to choose and assess the error rate with our training data. To do this, follow the steps:

Use the sample() function to permute the indices of the training set, and organize these permuted indices into 5 equal-size sets, called folds.

For each fold, take the corresponding subset from the training data to use as a 'test' set. Use the remaining messages in the training data as the training set. Apply the functions developed in the section called “Implementing the Naïve Bayes Classifier” to estimate the probabilities that a word occurs in a message given it is spam or ham, and use these probabilities to compute the log likelihood ratio for the messages in the training set.

Pool all of the LLR values from the messages in all of the folds, i.e., from all of the training data, and use these values and the typeIErrorRate() function to select a threshold that achieves a 1% Type I error.

Apply this threshold to our original/real test set and find its Type I and Type II errors.

Often in statistics, when we combine two methods, the resulting hybrid out performs the two pure methods. For example, consider a naïve Bayes approach that incorporates the derived variables into the probability calculation. For simplicity, you might try a few variables that result from the important splits in the recursive partitioning tree from Figure 3.8, “Tree for Partitioning Email to Predict Spam”, e.g., whether or not the percentage of capitals in the message body exceeds 13%. These variables have only two values as the splits are based on yes-no questions. Develop a hybrid classifier that uses both the word vectors and these additional features.

An alternative to recursive partitioning is the nearest neighbor method. This method computes the distance between two messages using the values of the derived variables from the section called “Deriving Variables from the Email Message”, or some subset of them. Use the email in trainDF to find the closest messages to each email in testDF. Then use these neighbors to classify the test message as spam or ham. That is, use the neighbors' classifications to vote on the classification for the test message. Compare this method for predicting spam to the recursive partitioning approach. Use both Type I and Type II errors in making your comparison. Include a comparison of the two approaches from a computational perspective. Is one much faster or easier to implement?

Write an alternative function to genKids() (see the section called “Generating Offspring”), called genKidsB() , where the B stands for batch. In this function, generate the inter-arrival times in batches, where the size of the batch depends on the expected number of offspring for a job. The expected number depends on the rate and the birth and completion times of the parent. That is, for a job born at time and complete at time , we expect it to have offspring. The parameters to this function and their default values should be the same as genKids() .

Write the function to genKidsU() described in the section called “Considering Alternative Implementations”. This function generates the birth and completion times for a job that is born at and completes at . The number of children follow a Poisson(. Once the number of children are known, their births can be generated according to the uniform on the interval . As mentioned in the section called “Considering Alternative Implementations”, these times are not generated in order so they need to be sorted. The parameters to genKidsU() and their default values should be the same as genKids() .

Develop a test suite to confirm that the recursive function genKidsR() (see the section called “Generating Offspring”) is consistent with the while-loop version of genKids() (also in the section called “Generating Offspring”).

Use Rprof() to profile the genKidsU() function in the previous exercise. As in the section called “Profiling and Improving Our Code”, profile the code by calling genKidsU() 1000 times with the parameter settings: bTime = 1, cTime = 100, and the default values for and . Also profile the code with one simulation, where cTime = 1000000. Do the profiles look different? Try improving the efficiency of your code based on the profile information. Additionally, does the code include calls to force() and deparse() ? Or some other unexpected functions? Can you determine why these functions are being called?

Develop a set of test cases for genKidsV() (see the section called “Unit Testing”). Write code to check the output from genKidsV() for these test cases.

Figure 7.4, “Visualization of a Randomly Generated Branching Process” is a custom visualization of the birth and completion times for a tree. Design an alternative custom visualization of the return value from familyTree() .

Incorporate into familyTree() a limit on the time that the tree is observed. If we want to observe a process up until time then those offspring with birth times after are discarded. Also, the simulation stops once all of the observed births in a generation are after . See Figure 7.5, “Visualization of a Randomly Generated Branching Process Over a Fixed Time Interval” for a visualization of the first five generations a simulated process that is truncated at time 8.

Update the genKidsV() function developed in the section called “From One Job's Offspring to an Entire Generation” to return one data frame rather than a list of data frames. This data frame needs to include additional columns that supply the parent and offspring identifiers. See the section called “A Structure for the Function's Return Value” for an example of the modified return value.

The branching process was summarized by two statistics: the number of generations and the number of offspring (see the section called “Replicating the Simulation”). Consider other summary statistics for the process. Incorporate them into exptOne() . Carry out a simulation study and create a visualization of the simulation that uses these additional statistics. Do they confirm the earlier findings? Do they offer any new insights?

Carry out a simulation study to see if the re-parameterization suggested in the section called “Analyzing the Simulation Results” is appropriate. For example, fix to be 1, and run the simulation for various values of . Compare the results to other simulations where is , but the ratio of matches one of the values from the earlier simulation when was 1.

Consider other probability functions to describe the lifetime of a process. Revise familyTree() (see the section called “The Family Tree: Simulating the Branching Process”) and genKidsV() (see the section called “From One Job's Offspring to an Entire Generation”) to take as an argument the random number generator for any probability distribution. The functions familyTree() and genKidsV() are to use this probability distribution (with arguments that may be specific to the distribution) to generate the completion times of the jobs.

Redesign the simulation study, where rather than generating one branching process at a time, the processes are generated in a vectorized fashion. This may require rewriting genKidsV() (the section called “From One Job's Offspring to an Entire Generation”) and familyTree() (the section called “The Family Tree: Simulating the Branching Process”).

In the winnings() function in the section called “Blackjack Basics”, we chose to break the function down by the value of the dealer's cards first, and then the player's cards. What happens when we switch this order and start with the player's values? Is the function easier or harder to understand? Write this alternative version of the function and test it using the test code in the section called “Testing Functions” to make sure that your code works as expected.

How does the terse version of the winnings() function in the section called “Blackjack Basics” work? Is it consistent with the original version? Test this new version of the function with the test cases developed in the section called “Testing Functions”. Is it easier to follow the logic in the first or second version of the winnings() function? Why? When do you think using code like this is appropriate?

Consider how you might add checks to the hit() , dd() and splitPair() functions (see the section called “Creating Functions for the Player's Actions”) to ensure that they are being used properly. For example, add code to splitPair() so that an error message is returned if the player tries to split a hand when the two cards are not of equal value. Add other checks as well.

Compile a set of test cases for the checks that you developed in the previous exercise. Write a function that calls these functions (hit() , dd() and splitPair() ) with the test cases and returns an informative message indicating whether the functions pass all of the tests and if not where problems occurred. Create other test cases and test functions for the play_hand() function.

Implement the draw_n() method for the Shoe class of the section called “A More Accurate Card Dealer Shoe”. This function has one input: n, which indicates how many cards should be drawn from the shoe. It returns a vector of n cards taken from the cards field of the Shoe object. This vector of cards begins at the position indicated by the pos fields. This method also increments pos by n so that we do not draw the same cards multiple times. It also calls the helper method decks_left() to see if it is time to reshuffle. If there is less than one deck in the shoe then it calls the shuffle() method to shuffle the cards.

Implement the hi_low() function from the section called “Counting Cards”. This function takes one input, which is a numeric vector cards, and it returns the Hi-Lo count (see Table 9.2, “Card Counting Strategies”). If you want an additional challenge, write the function without using an if statement or any loops. For example, consider taking advantage of the fact that the vector of cards is a numeric vector, and use that to subset into a vector of count values.

Implement the any_count() function for counting cards (see the section called “Counting Cards”). This function has two arguments: a numeric vector cards which contains the cards played, and a numeric vector, strategy, which is of length 10 and contains the card values for one of the strategies in Table 9.2, “Card Counting Strategies”. Modify count_payoff() to use any_count() . This may require modifying the function definition to accept a strategy argument.

Suppose a bet was larger than $1, say $4, how might the bet() function take this into consideration in deciding the size of the bet? Modify bet() (in the section called “Counting Cards”) so that bets are integers, but utilize a standard bet size. What impact does this change have on other functions, e.g., winnings() ? Is there a need to modify any of the earlier code?

Implement other play options that are available in some games of blackjack, such as insurance and surrendering. Think about the play_hand() function: can you simply augment this function with code for these new actions, or do you need to fundamentally re-structure how the function works? See the section called “Strategies for Playing” for the play_hand() function.

A major simplification in our simulation is that we have effectively given the gambler an infinite amount of money. How do things change if we also account for ruin, i.e., the gambler running out of money?

Can you make the code faster? Use profiling techniques to find the slow parts of the code. Also, consider how you might use vectorization to play multiple games at once.

In the merge of infMortDF and popDF in the section called “Integrating Data from Different Sources” some rows in each data frame were excluded from the resulting data frame. Determine which rows in each data frame did not find a match in the other.

The data frame codeMapDF contains a mapping of the ISO 3166 codes to the codes used in the CIA Factbook. It also contains country name. Below are the first few observations in this data frame:

head(codeMapDF)
  cia           name iso
1  af    Afghanistan  AF
2  ax       Akrotiri   -
3  al        Albania  AL
4  ag        Algeria  DZ
5  aq American Samoa  AS
6  an        Andorra  AD

Use this data frame to fix the problem with the merging of the location data, which uses ISO codes, with the demographic data, which uses the CIA Factbook coding. Be sure the final data frame contains both codes and the country name from codeMapDF, as well as the original variables.

Remake the map in Figure 11.6, “Map of Infant Mortality and Population” to include the countries for which there is latitude and longitude, even if there is no demographic information. To do this you need to modify the call to merge() . In particular, the all.x and all.y parameters are helpful here. In the map, use a special plotting symbol to denote those countries where demographic information is missing.

To determine the color of a circle in the map in Figure 11.6, “Map of Infant Mortality and Population”, we discretized the infant mortality rates. Investigate alternative cut points for this discretization that derive from standards used by international organizations such as the United Nations, World Health Organization, and World Bank. Remake the map to incorporate these external values.

To determine the size of a circle in the map in Figure 11.6, “Map of Infant Mortality and Population”, we used a scaling of the square-root of population. We found that the range in population was so large that when the largest countries had a reasonably sized disk, the disks for the smaller countries could not be seen. To address this problem we created a lower bound for the disk size so that no matter how small the country a disk with this minimum size would be plotted. Find an alternative scaling for population that instead caps the size of the larger countries. That is, for very large countries, the no matter how large the population, the disk plotted on the map is no bigger than this cap. Remake the map using this approach to scaling the circles.

The mapproj[bib:mapproj] library contains various projections of the 3D earth into 2D map. Install this package and investigate the projections. To do this you will want to read some resources and tutorials available online. Remake the map in Figure 11.6, “Map of Infant Mortality and Population” using a projection.

There are many other statistics available in the Factbook. Choose demographic information other than infant mortality to extract and visualize.

Wikipedia tables offer another source of data. For example, Wikipedia has a list of infant mortality rates by country at https://en.wikipedia.org/wiki/List_of_countries_by_infant_mortality_rate. These data are from the United Nations World Population Prospects report (https://esa.un.org/unpd/wpp/Excel-Data/mortality.htm) and the CIA World Factbook. Another source of information is the World Bank, which provides infant mortality rates at https://data.worldbank.org/indicator/SP.DYN.IMRT.IN. Create a mashup with data from one of these alternative sources.

XPath is a powerful tool for locating content in an XML document, and there are often many XPath expressions that we can use to access the information we want. For example, rather than use the id attribute of <field> to find the relevant data, try another approach, such as locating the desired nodes by filtering on the value of the name attribute.

Use XPath to extract the mapping between the two types of country codes from the CIA factbook. Once you have this mapping in a data frame, us it to remerge the lat/lon data with the factbook data. Then re-do the world map. It should look like Figure 11.6, “Map of Infant Mortality and Population”. To assist you in this task, we provide a nippet of the Factbook that shows a portion of the table that contains the various codings for countries.

<table title="" lettergrouped="1">
 <columnHeader colspan="1" title="Entity"/>
 <columnHeader colspan="1" title="FIPS 10"/>
 <columnHeader colspan="3" title="ISO 3166"/>
 <columnHeader colspan="1" title="Stanag"/>
 <columnHeader colspan="1" title="Internet"/>
 <columnHeader colspan="1" title="Comment"/>
...
<row>
 <cell country="ch" content="China"/>
 <cell center="1" content="CH"/>
 <cell content="CN"/>
 <cell content="CHN"/>
 <cell content="156"/>
 <cell center="1" content="CHN"/>
 <cell center="1" content=".cn"/>
 <cell content="see also Taiwan"/>
</row>
...
<row>
 <cell country="gb" content="Gabon"/>
 <cell center="1" content="GB"/>
 <cell content="GA"/>
 <cell content="GAB"/>
 <cell content="266"/>
 <cell center="1" content="GAB"/>
 <cell center="1" content=".ga"/>
 <cell content=" "/>
</row>

<row>
 <cell country="sz" content="Switzerland"/>
 <cell center="1" content="SZ"/>
 <cell content="CH"/><cell content="CHE"/>
 <cell content="756"/>
 <cell center="1" content="CHE"/>
 <cell center="1" content=".ch"/>
 <cell content=" "/>
</row>

In the section called “Extracting Latitude and Longitude from an HTML File” we saw that the latitudes and longitudes for the countries are available on the Web, embedded in the page at https://dev.maxmind.com/geoip/legacy/codes/average-latitude-and-longitude-for-countries/. A snippet of the HTML source for this page is shown in Figure 11.10, “Screenshot of the HTML Source of the MaxMind Web Page”. Extract these location coordinates from this HTML file.

Screenshot of the HTML Source of the MaxMind Web Page

This screenshot displays the HTML source of the Web page shown in Figure 11.9, “Screenshot of the MaxMind Web Page with Country Latitude and Longitude”. Notice that the latitudes and longitudes for the countries appear within a <pre> node in the HTML document.

Examine the HTML source, and notice that the data are simply placed as plain text within a <pre> node in the document. If we can extract the contents of this <pre> node, then we can place this information in a data frame. Begin by parsing the HTML document with htmlParse() . Don't download the document, simply pass the function the , https://dev.maxmind.com/geoip/legacy/codes/average-latitude-and-longitude-for-countries/.

Next access the root of the document using xmlRoot() , and use an XPath expression to locate the <pre> nodes in the document. The getNodeSet() function should be useful here. The getNodeSet() function takes the XML tree (or subtree) and an XPath expression as input and returns a list of all nodes in the tree that are located with this expression.

Once you have located these <pre> nodes, check to see how many were found. It should be only one. The latitude and longitude values are in the text content of this node. Extract the text content with the xmlValue() function. It should look something like:

[1] "\n\"iso 3166 country\",\"latitude\",\"longitude\"\n
AD,42.5000,1.5000\nAE,24.0000,54.0000\nAF...

The content is one long character string containing all the data.

Complete the exercise by reading the plain text in your character vector into a data frame. Use read.table() to do this. The parameters, text, skip, header, and sep should be useful here.

Collect all of the steps for generating the KML document (see the section called “Generating KML Directly”) into one function, called makeCIAPlot() . Consider generalizing it so that it can be used with other Factbook variables or other data.

The plotting symbols used in Figure 11.1, “Display of Infant Mortality by Country on Google Earth” were created in R. Write a function as described in the section called “Creating Plotting Symbols” to create a set of PNG files that can be used as placemarks on Google Earth. Rather than simply making circles, consider making other shapes, including a miniature plot of the data. Also, use a different color scheme than the yellow-orange-red palette.

This placemark function creates the icons by setting up a blank canvas that has a transparent background and no axes and no labels. On this canvas draw a circle or some other shape and fill it with the desired color. The function should take as input a vector of colors and create a set of circles (or some other shape) as PNG files, one for each color. The png() , plot() , draw.circle() functions might be helpful. In addition, you might try making the symbols partially transparent so that when they overlap on the plot, they can still be seen.

Make an R plot for the pop-up window of each country. That is, for each country, make a plot that is country specific. Save this plot to a PNG file and add the file name to the <description> node, e.g., the content for the description might be something like

Afghanistan:
<img src = "af.png"/>

The resulting KML document should be a file, i.e., a ZIP archive, that contains all of these PNG files as well as the KML document.

Rewrite the addPlacemark() and makeStyleNode() functions to use the technique described in the section called “Efficiency in Generating KML from Strings”. That is, construct the placemarks by pasting strings together, and then convert the strings into XML nodes with a call to xmlParseAndAdd() .