Introduction to Data Analysis and Graphics in R

Session three

Data Entry, Management and Manipulation in R

• Data sets can be created for any of R's data structure i.e. dimensionless vector, 1 dim vector, matrix, array, data frame or list
• There are two way to create a data set:
  • Using spreadsheet like data editor
  • By coding then in

• R has four functions to invoke a spreadsheet-like data editor, these are:
  • edit()
  • fix()
  • data.entry(), and
  • dataentry()

• Using these function's goes against R's core functionality; programming/coding
• Not a recommended way as it looses on documentation/reproducibility

• To code in data, function scan() can be quite handy in addition to calling functions for any of the data structures; c() for vector, matrix(), array(), data.frame(), and list()
• scan() is also not a good data entry process as it looses on reproducibility as data is entered interactively (console)

• It can be a single variable or multiple variables
• In R, a single variable can be a dimensional vector (created with “c()”) or a 1 dim array
• For multiple variables, if they are all of the same type (especially if numeric), then matrix is a better data structure other wise for multiple types with same length data frame is ideal

• If data is of different length and type, generic lists are appropriate.
• Lists can also be used to store different data sets for a particular project as well as accompanying source code/function

We will look at:

• Spreadsheet data entry using "data.entry()"
• Using "scan()"
• Coding in data using data structure functions c(), matrix(), array(), data.frame(), and list()

• First, need to have variables or list of variables for data entry
• Then Call data entry
• From pop-up data editor, click on individual cell and enter data
• Variable names can be changed from data entry

Spreadsheet data entry demonstration

• Can be used to input 1 dim atomic vectors
• Values entered interactively (on console) if file is not give
• For each entry, type value and click enter, after last value click enter and entry mode will be exited
• Important to assign to variable name and specify type if it not “double”; dataset2 <- scan(what = "character")

Demonstration on data input using function “scan”

• Recommended way to generate data in R (ideally small data)
• Data structure function include:
  • c() for atomic vectors
  • matrix() for matrices
  • array() for 1 or more dimension arrays
  • data.frame for data frames
  • list() for lists

• Used to create individual variables of any type as long as all elements are of the same type e.g all logical or all character

# An integer vector
num <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) # same as 1:10
# A logical vector
logi <- c(TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, FALSE)

# A character vector
R_authours <- c("Douglas Bates", "John Chambers", "Peter Dalgaard", "Seth Falcon", "Robert Gentleman", "Kurt Hornik", "Ross Ihaka", "Michael Lawrence", "Friedrich Leisch", "Uwe Ligges", "Thomas Lumley", "Martin Morgan", "Duncan Murdoch", "Paul Murrell", "Martyn Plummer", "Brian Ripley", "Deepayan Sarkar", "Duncan Temple Lang", "Luke Tierney", "Simon Urbanek")

• 2 dimensional vectors (store data as rows and columns)
• Primarily created with function matrix() but rbind(), cbind() and as.matrix() can be used to convert other vectors to a matrix
• Function matrix() can be called without any input thus creating an empty matrix
• Argument “dimnames” can only be NULL (nothing) or a list

mat1 <- matrix(data = 1:9, nrow = 3, dimnames = list(NULL, c("a", "b", "c")))
mat1

     a b c
[1,] 1 4 7
[2,] 2 5 8
[3,] 3 6 9

• Multi-dimensional structures (1 > dims), but often used for 3 dim structures
• Can only be used with one data type
• Matrices are special form of these data structures (have 2 dims)
• Primarily created with function array()

• Similar to matrices except they can contain different types of data as long as they have the same length (number of elements)
• Though resemble matrices, they are actually list of vectors
• Columns contain measurements on one variable and rows contain cases
• Primarily created by data.frame()

# Example of weight loss data set
dataset3 <- data.frame(ID = 1:5, Exercise = c(TRUE, TRUE, FALSE, TRUE, FALSE), Height = c(5.2, 4.9, 5.1, 5.2, 5.4), Weight = c(69, 72, 75, 67, 77))
dataset3

  ID Exercise Height Weight
1  1     TRUE    5.2     69
2  2     TRUE    4.9     72
3  3    FALSE    5.1     75
4  4     TRUE    5.2     67
5  5    FALSE    5.4     77

• A bit unique as not many statistical programs have similar data structure
• A sort of “carry-all” data structure
• Can also contain sub-list thus referred to as recursive
• Primarily created by list()

lst <- list(vect = 5:9, Matrix = mat1, Array = arry, Dataframe = dataset3, List = list("a", 2:3))

str(lst)

• Everything in R is referred to as an object from data structures to functions and all objects have two types of attributes:
  • Mode and
  • Length
• Mode is the basic type of an object's core constituent
• Length is extent or number of elements in an object
• Function mode() and length() are used to establish mode and length of an object

# Atomic vector
length(num)

[1] 10

# Matrix
length(mat1) 

[1] 9

• Length is not the best attribute for assessing a matrix or an array, “dim” is more appropriate

mat1; dim(mat1)

     a b c
[1,] 1 4 7
[2,] 2 5 8
[3,] 3 6 9

[1] 3 3

# Data frames
length(dataset3) # This shows number of variables not cases/rows

[1] 4

# Lists
length(lst)

[1] 5

• There are 3 functions for checking basic constituents of an object, these are:
  • mode() which is an S compatible function for checking type
  • storage.mode() which is used for compatability when calling functions written in other languages (ensures data is of expected type)
  • typeof() which is basically an R's implementation of S's mode()

• Which function should be used? Depends on why,
  • If it's just a general query, then typeof() is adequate
  • If working with other S objects, use mode()
  • If calling functions written in other languages, use storage.mode()
    

• Attributes are basically meta data about an object in R
• All objects (except NULL) can have at least two or more attributes
• Attributes are stored as a pairlist i.e. name=value
• List of all attributes for an object are given by attributes()
• Individual attributes are given by attr()

• Other than mode and length other often used attributes are:
  • Names
  • Dimensions (dim)
  • Dimnames
  • Classes, and
  • Time series

• Used to name individual elements of a data object
• They are not mandatory, but quite handy when indexing element
• Accessed with name() and set with name(object) <-
• colnames() is used for matrix-like objects

# Creating an unnamed vector
vect1 <- c(12, 54, 98)
names(vect1)

NULL

# Naming vector elements
names(vect1) <- c("a", "b", "c")
names(vect1)

[1] "a" "b" "c"

• There at least two data structures with dimension attribute, these are arrays (including matrices) and data frames
• Function dim() is used to query an objects dimension and dim <- used to set dimension to an object
• There is a difference between an atomic vector and a 1 dim array; latter has a dim attribute while former does not
• Giving a vector dimensions changes it's data structure from a vector to an array

• Gives names to dimensions
• Like “dim” attribute, “dimnames” attributes are given to vectors with dim attribute like matrices, array and data frames
• Dimnames are given as a list of names (same lenth as “dim(x)”)

• Class attribute is a special type of information used functions called “methods”
• Used to determine how an object should be handled/acted upon
• All objects have an intrinsic class attribute which is basically it's data type, but other classes can be added to an object

• Classes are character vectors accessed and added with function class() and class <- respectively or attr(obj, class)
• When a class is added to an object, that object is called an s3 object. This makes it part of R's Object Oriented Programming (OOP)

• Adding classes has it's implications as far as “method dispatch” (selection of suitable function) is concerned
• For example, changing from intrinsic class “numeric"to "myclass” means function/methods for “myclass” if found, will be applied first
• Basically when a generic function such as “plot()” or “mean()” are called, they will look for functions suitable for first listed class in a class vector, it is not until all classes are are checked that a method for it's intrisic class is dispatched
  

• Used for data with time dimensionality like timely, daily, weekly, monthly, quarterly or annual data
• Created by adding a “tsp” attribute
• It ensures time series parameters such as “start”, “end”, and “frequency” are kept and
• For compartability with S version 2

n

• Used to certain conditions are met by some data like observation above a certain value
• They are also called control structures. Include:
  • if-else
  • ifelse
  • for

• Others are
  
  • while
  • repeat
  • break
  • next, and
  • switch
• We discuss frequently used control structure, that is if-else, ifelse and for

• Used to check if a condition evaluated to true, and if so an action is performed
• It can be extended alternative action(s) with “else if” or “else”
• When “else statement” is given, it must be on same line as end of if statement
• Example, check a vector has intrisic type “character”, if it does, we convert it to a factor else leave it as it is
• Note: if-else can only be performed if condition evaluated to one logical value either TRUE or FALSE

x <- c("a", "b", "c")
class(x)

[1] "character"

if (class(x) == "character") {
   x <- as.factor(x)
} else {
   x
}
class(x)

[1] "factor"

• Used when condition evaluates to a logical vector length > 1
• Excellent for recoding variables, hence an example is done under “recoding variables”

• for() is a looping structure used to perform repetitive tasks
• Though in most programming programs, this is a frequently used construct, in R, there more efficient functions like apply group of functions
• for iterates from a certain value through a sequence performing an action defined it's body (body of any function including for loop is what is in between {})
• As a simple example, let's say Hello five time

for(i in 1:5) {     # variable "i" is a counter (conting from 1 to 5)
    cat("Hello \n") # function "cat" is used to print to console 
}

Hello 
Hello 
Hello 
Hello 
Hello 

• Recoding a variable means changing it's values
• It is often recommended to create a new variable instead of overwriting original variable
• Example:
  • Create a dichotomous recoded variable of “feed” variable from “chickwts” data set
  • This variable will have values “casein” and “others” (this is something often done during analysis)

# Recording with function "ifelse"
chickwts$feed2 <- ifelse(chickwts$feed == "casein", yes = "casein", no = "other")

# Conveting to a factor vector
chickwts$feed2 <- factor(chickwts$feed2)

# New levels 
levels(chickwts$feed2)

[1] "casein" "other" 

• using base R, renaming any variables in a data frame requires all variable names to issued to names() <- function
• For example, to rename “feed2” from previous slide:

# Current name
names(chickwts)

[1] "weight" "feed"   "feed2" 

# Renaming variables (all must be proived)
names(chickwts) <- c("weight", "feed", "casein")
names(chickwts)

[1] "weight" "feed"   "casein"

• In R, denoted with Logical value “NA”
• Many operation can not be performed when there are missing value
• is.na() used to check for missing value
• If negated with “!” infront, it output current (non-missing) values
• For matrices and data frames, complete.cases() might be more appropriate

n

# Vector with a missing value
vect1 <- c(letters[1:5], NA); vect1

[1] "a" "b" "c" "d" "e" NA 

# A logical vector checking for missing values 
is.na(vect1)

[1] FALSE FALSE FALSE FALSE FALSE  TRUE

• Initially imported or created as numeric or character vectors
• Conversion (to class for data/time object: POSIXlt/POSIXct) depends on whether they are character or numeric
• One way to convert character vector to date/time object is by using function as.Date() specifying argument format as detailed by ?strftime
• as.Date() can also be used to convert a numeric vector to a date object, by specifying argument origin; origin in R is “1970-01-01”

# Converting a character vector
date1char <- c("3/6/2017", "3/7/2017", "4/7/2017")
class(date1char)

[1] "character"

date1 <- as.Date(date1char, format = "%m/%e/%Y")

date1

[1] "2017-03-06" "2017-03-07" "2017-04-07"

class(date1)

[1] "Date"

# Converting a numeric vector
date1num <- c(17231, 17232, 17263)
class(date1num)

[1] "numeric"

date2 <- as.Date(date1num, origin = "1970-01-01")

date2

[1] "2017-03-06" "2017-03-07" "2017-04-07"

class(date2)

[1] "Date"

• To convert from one data type to another, use as.data_type like as.logical(), as.integer(), as.double(), as.character(), as.raw(), and as.complex()
• But it must be convertible e.g.
  • Can convert from logical to character but if character is not “TRUE/FALSE” or “true/false” it will result in NA
  • Cannot convert character to integer or double

• Sorting an atomic vector is done with sort()
• Sorting a data frame is done with order()
  
• Matrices are actually atomic vectors with dimensions, hence sorted with looping function apply
• By default sort is done in an increasing manner, be nullified by setting argument “decreasing” to TRUE
• Logical values ordered according to their integer form, i.e. TRUE = 1, FALSE = 0 (TRUE > FALSE)
  

set.seed(3)
v1 <- round(rnorm(9, 50, 10))
set.seed(3)
v2 <- round(rnorm(9, 90))
set.seed(3)
logi <- sample(c(TRUE, FALSE), 9, TRUE, c(0.7, 0.3))
df1 <- data.frame(Logi = logi, V1 = v1, V2 = v2)

# Sorted by first variable "logi"
df1[order(df1$Logi, decreasing = TRUE),]

   Logi V1 V2
1  TRUE 40 89
3  TRUE 53 90
4  TRUE 38 89
5  TRUE 52 90
6  TRUE 50 90
7  TRUE 51 90
8  TRUE 61 91
9  TRUE 38 89
2 FALSE 47 90

• Sorting by more than one variable is first done on first listed variable then the second and so on.
• Example:
  • Sort variable Logi in a decreasing manner (TRUE first)
  • Then sort variable “V1” in a decreasing manner

df1[order(df1$Logi, df1$V1, decreasing = TRUE),]

   Logi V1 V2
8  TRUE 61 91
3  TRUE 53 90
5  TRUE 52 90
7  TRUE 51 90
6  TRUE 50 90
1  TRUE 40 89
4  TRUE 38 89
9  TRUE 38 89
2 FALSE 47 90

# Negative sign used to indicate decreasing
df1[order(-df1$Logi, df1$V1), ]

   Logi V1 V2
4  TRUE 38 89
9  TRUE 38 89
1  TRUE 40 89
6  TRUE 50 90
7  TRUE 51 90
5  TRUE 52 90
3  TRUE 53 90
8  TRUE 61 91
2 FALSE 47 90

• Done by similar (intersecting) columns
• Can use database semantics
• Core considerations for merging
  • Default merging done by intersect(names(x), names(y))
  • Otherwise specific columns in each can be given especially if they do not have same name or capitalization

# Additional data set
dataset4 <- data.frame(ID = 6:10, Exercise = c(TRUE, FALSE, TRUE, TRUE, FALSE), Height = c(5.4, 5.4, 5.2, 5.6, 5.4), Weight = c(77, 74, 75, 79, 82))

# Similar columns to be used for merging
intersect(names(dataset3), names(dataset4))

[1] "ID"       "Exercise" "Height"   "Weight"  

# Merging (adding cases)
merge(dataset3, dataset4, all = TRUE)

   ID Exercise Height Weight
1   1     TRUE    5.2     69
2   2     TRUE    4.9     72
3   3    FALSE    5.1     75
4   4     TRUE    5.2     67
5   5    FALSE    5.4     77
6   6     TRUE    5.4     77
7   7    FALSE    5.4     74
8   8     TRUE    5.2     75
9   9     TRUE    5.6     79
10 10    FALSE    5.4     82

Look at:

• Indexing
• Subsetting/extracting operators
• Subsetting different data objects

• Indexing vectors are used to access elements from different data objects, they include:
  • Logical vector
  • Positive integers
  • Negative integers and
  • Character vectors
• Note: It's possible to have 0 index (empty indexing)

• Logical vectors select elements which evaluate to TRUE
• Positive integers select elements at given positions
• Negative integers exclude values at given integers
• Character indices are only appropriate for named elements
• An empty index selects all values, used to replace all entries but at the same time keeping it's attributes

• There three extracting operators and one extracting function
  • [
  • [[
  • $, and
  • getElement()

• "[" can select more than one element and keeps their names if present while "[[" and "$" can only select one element without their names
• "$" is only applicable for recursive objects (generic/list data structures), basically data frames and lists
• "getElement()" function is similar to extracting with "[["

• Essentially atomic vectors with dimensions hence can be subset with [ and [[
• Output is value occurring at given indices when all values are concatenated
• However, the best way to index these structures is by their dimension e.g. [r, c] for 2 dim matrices and [r, c, l] meaning row, column, and layer for 3 dim arrays
• Exampl data set: R's USPersonalExpenditure

# One of R's data set
USPersonalExpenditure

                      1940   1945  1950 1955  1960
Food and Tobacco    22.200 44.500 59.60 73.2 86.80
Household Operation 10.500 15.500 29.00 36.5 46.20
Medical and Health   3.530  5.760  9.71 14.0 21.10
Personal Care        1.040  1.980  2.45  3.4  5.40
Private Education    0.341  0.974  1.80  2.6  3.64

• All subsetting operators ([, [[ and $) can be used
• As before [ can selects more than one element
• Both [[ and $ can select one item, difference is that $ can not be used with computed values like “i + 1” (index + 1)
• x$name is equivalent to x[[“name”, exact = FALSE]]
• Other than these operators, a much more efficient way to subset data frames is with function subset()

• Function subset can be used to subset any vector, but most suitable for data frames
• Here we will use it to subset high_crime states as we did before
• We use function with() to access variables without making reference to data frame name

• List can be subset with all three subsetting operators
• Rule of the thumb is, subsetting with [ returns a list, subsetting with [[ or $ outputs the same type as element being subset i.e. if list has data frame, subsetting with [[ or $ will output a data frame
• Example data set: R's first 10 values of “state.center” data set

# Example data
state.center; class(state.center)

$x
 [1]  -86.7509 -127.2500 -111.6250  -92.2992 -119.7730 -105.5130  -72.3573
 [8]  -74.9841  -81.6850  -83.3736

$y
 [1] 32.5901 49.2500 34.2192 34.7336 36.5341 38.6777 41.5928 38.6777
 [9] 27.8744 32.3329

[1] "list"

# Using `[` outputs a list
state.center[1]

$x
 [1]  -86.7509 -127.2500 -111.6250  -92.2992 -119.7730 -105.5130  -72.3573
 [8]  -74.9841  -81.6850  -83.3736

class(state.center[1])

[1] "list"

# Using `[[` outputs elements type
state.center[[1]]

 [1]  -86.7509 -127.2500 -111.6250  -92.2992 -119.7730 -105.5130  -72.3573
 [8]  -74.9841  -81.6850  -83.3736

class(state.center[[1]])

[1] "numeric"

# Using "$" outputs elements type
state.center$x

 [1]  -86.7509 -127.2500 -111.6250  -92.2992 -119.7730 -105.5130  -72.3573
 [8]  -74.9841  -81.6850  -83.3736

class(state.center$x)

[1] "numeric"

• Database semantics can sometimes be quite handy in subsetting e.g. subset has to meet certain condition
• Core data base statements are :
  • SELECT
  • FROM
  • WHERE
  • ORDER BY

• If interested, read a small introduction to SQL statement from R's Data Import/Export manual (4.2) or go online and learn from “www.sqlcourse.com”
• Discussing this here might take us out scope, but it's good to know it's possible in R using contributed packages like “sqldf” and “dplyr”.