str(), head(), and
dim() commands to explore your data setsThere is already a lot of data in the R library. The
data() command displays all available data sets.
We are going to use the airquality data here:
# data()
str(airquality)## 'data.frame': 153 obs. of 6 variables:
## $ Ozone : int 41 36 12 18 NA 28 23 19 8 NA ...
## $ Solar.R: int 190 118 149 313 NA NA 299 99 19 194 ...
## $ Wind : num 7.4 8 12.6 11.5 14.3 14.9 8.6 13.8 20.1 8.6 ...
## $ Temp : int 67 72 74 62 56 66 65 59 61 69 ...
## $ Month : int 5 5 5 5 5 5 5 5 5 5 ...
## $ Day : int 1 2 3 4 5 6 7 8 9 10 ...dim(airquality)## [1] 153 6head(airquality)## Ozone Solar.R Wind Temp Month Day
## 1 41 190 7.4 67 5 1
## 2 36 118 8.0 72 5 2
## 3 12 149 12.6 74 5 3
## 4 18 313 11.5 62 5 4
## 5 NA NA 14.3 56 5 5
## 6 28 NA 14.9 66 5 6subset commandThe subset function returns subsets of vectors, matrices or data frames which meet specified conditions. The following examples are from the subset function description (slightly modified).
In the following example, we extract data rows with temperature above 70 degrees, and keep only two columns (Ozone and Temp) of the original data frame:
sub1 = subset(airquality, Temp > 70, select = c(Ozone, Temp))
head(sub1)## Ozone Temp
## 2 36 72
## 3 12 74
## 11 7 74
## 22 11 73
## 29 45 81
## 30 115 79Next, we keep only day1 rows, and skip the Temp column:
sub2 = subset(airquality, Day == 1, select = -Temp)
head(sub2)## Ozone Solar.R Wind Month Day
## 1 41 190 7.4 5 1
## 32 NA 286 8.6 6 1
## 62 135 269 4.1 7 1
## 93 39 83 6.9 8 1
## 124 96 167 6.9 9 1We select all columns from Ozone to Wind (using the colon):
sub3 = subset(airquality, select = Ozone:Wind)
head(sub3)## Ozone Solar.R Wind
## 1 41 190 7.4
## 2 36 118 8.0
## 3 12 149 12.6
## 4 18 313 11.5
## 5 NA NA 14.3
## 6 28 NA 14.9The logical drop parameter tells the function if the
original data structure should be preserved or not.
The default value is FALSE. We give an example where only
one column is selected:
sub4 = subset(airquality, Day == 1, select = Temp, drop = FALSE) # drop = FALSE must not be stated explicitely
str(sub4)## 'data.frame': 5 obs. of 1 variable:
## $ Temp: int 67 78 84 81 91We see that sub4 is still a data frame, even though it consists of one vector only (Temp).
Now, different from above, we declare drop = TRUE:
sub5 = subset(airquality, Day == 1, select = Temp, drop = TRUE) # drop = TRUE !
str(sub5)## int [1:5] 67 78 84 81 91As a result, the command keeps only a vector. The data frame structure has been dropped.
merge commandThe merge command can join two data frames by common columns or row names.
Let us create some example data sets:
df1 <- data.frame( firstname = c("hans", "peter", "mary", "steve", "katie"),
surname = c("lund", "mueller", "kaminski", "smith", "brendel"),
profession = c("dentist", "trucker", "midwife", "dentist", "physician"))
df1## firstname surname profession
## 1 hans lund dentist
## 2 peter mueller trucker
## 3 mary kaminski midwife
## 4 steve smith dentist
## 5 katie brendel physicianIn an new data frame df2, we change the column name surname to name:
df2 = df1
colnames(df2)[2] = "name"
# alternative : df2 <- within(df1, { name <- surname; rm(surname) })
df2## firstname name profession
## 1 hans lund dentist
## 2 peter mueller trucker
## 3 mary kaminski midwife
## 4 steve smith dentist
## 5 katie brendel physicianA third data frame:
cars <- data.frame(
name = c("lund", "mueller", "kaminski", "smith", "brendel", "weller", "cooper"),
type = c("kia", "vw", "ford", "bmw", "bmw", "fiat", "audi"),
color = c(NA, "red", NA, "blue", "white", "black", "white"),
miles = c(1200, 400, 650, 1590, 8420, 5000, 7200))
cars## name type color miles
## 1 lund kia <NA> 1200
## 2 mueller vw red 400
## 3 kaminski ford <NA> 650
## 4 smith bmw blue 1590
## 5 brendel bmw white 8420
## 6 weller fiat black 5000
## 7 cooper audi white 7200By default, the data frames are merged on columns with identical names. If we merge df2 and cars, the common column name is name. We have 5 common entries in the name column:
intersect(df2$name, cars$name)## [1] "lund" "mueller" "kaminski" "smith" "brendel"Consequently, we should see 5 rows in the merged data set:
merge1 <- merge(df2, cars)
merge1## name firstname profession type color miles
## 1 brendel katie physician bmw white 8420
## 2 kaminski mary midwife ford <NA> 650
## 3 lund hans dentist kia <NA> 1200
## 4 mueller peter trucker vw red 400
## 5 smith steve dentist bmw blue 1590However, I suggest to specify the column for joining explicitely, because it feels safer. The parameter by declares the column to be used for joining:
merge(df2, cars, by = "name")## name firstname profession type color miles
## 1 brendel katie physician bmw white 8420
## 2 kaminski mary midwife ford <NA> 650
## 3 lund hans dentist kia <NA> 1200
## 4 mueller peter trucker vw red 400
## 5 smith steve dentist bmw blue 1590This presupposes that there is a column with the same name in both data frames.
In the cases above, the profession columns had identical entries, so that we need only one profession column in the output. What if we have profession columns in the datasets which are not identical?
df3 = df2
df3$profession[3] = "director"
df3$profession## [1] "dentist" "trucker" "director" "dentist" "physician"The frames df1 and df3 are once more displayed:
df1## firstname surname profession
## 1 hans lund dentist
## 2 peter mueller trucker
## 3 mary kaminski midwife
## 4 steve smith dentist
## 5 katie brendel physiciandf3## firstname name profession
## 1 hans lund dentist
## 2 peter mueller trucker
## 3 mary kaminski director
## 4 steve smith dentist
## 5 katie brendel physicianmerge(df1, df3, by = "firstname")## firstname surname profession.x name profession.y
## 1 hans lund dentist lund dentist
## 2 katie brendel physician brendel physician
## 3 mary kaminski midwife kaminski director
## 4 peter mueller trucker mueller trucker
## 5 steve smith dentist smith dentistNow, merge has created two profession columns
with the suffixes .x and .y, respectively.
What if we want to join on columns with different names in the two data sets? For instance, df1 and cars have no common column names:
intersect(colnames(df1), colnames(cars))## character(0)…. but the entries in name and surname are of the same nature, so we could join based on them:
df1$surname## [1] "lund" "mueller" "kaminski" "smith" "brendel"cars$name## [1] "lund" "mueller" "kaminski" "smith" "brendel" "weller" "cooper"Using the parameters by.x and by.y, we can specify which columns to use in the two datasets:
merge(cars, df1, by.x = "name", by.y = "surname")## name type color miles firstname profession
## 1 brendel bmw white 8420 katie physician
## 2 kaminski ford <NA> 650 mary midwife
## 3 lund kia <NA> 1200 hans dentist
## 4 mueller vw red 400 peter trucker
## 5 smith bmw blue 1590 steve dentistWhen using surname in df1 and name in cars for joining, we have only 5 entries in the output, the common entries for these columns.
If we want all names/surnames in the merged dataset, we can specify all = TRUE:
m2 = merge(cars, df1, by.x = "name", by.y = "surname", all = TRUE)
m2## name type color miles firstname profession
## 1 brendel bmw white 8420 katie physician
## 2 cooper audi white 7200 <NA> <NA>
## 3 kaminski ford <NA> 650 mary midwife
## 4 lund kia <NA> 1200 hans dentist
## 5 mueller vw red 400 peter trucker
## 6 smith bmw blue 1590 steve dentist
## 7 weller fiat black 5000 <NA> <NA>If we do so, we have to tolerate unassigned fields (NA) in the output, because there is no cooper and weller in the surname column of df1.
We have seen that the frame m2 contains unassigned fields, marked by NA. If we have to remove rows with such fields, we can use the complete.cases command:
complete.cases(m2)## [1] TRUE FALSE FALSE FALSE TRUE TRUE FALSEThe result of complete.cases() is a logical vector
indicating rows with complete data (TRUE), and rows containing
NA (FALSE). We can utilize this function to clean a
data frame from rows with unassigned fields:
m3 = m2[complete.cases(m2), ]
m3## name type color miles firstname profession
## 1 brendel bmw white 8420 katie physician
## 5 mueller vw red 400 peter trucker
## 6 smith bmw blue 1590 steve dentistWe have used the []-operator to select rows with
TRUE-values.
In some situations, it might be necessary to reorder the rows (or columns) of a data frame. This can be accomplished by using the match function. Let us create a small data frame for explanation:
df <- data.frame(name = letters[1:4], value = c(rep(TRUE, 2), rep(FALSE, 2)))
df## name value
## 1 a TRUE
## 2 b TRUE
## 3 c FALSE
## 4 d FALSEAs an example, we define a desired order of the rows and store it in the variable target. Then, we use match to find the positions of target in the name column of the data frame df:
target <- c("b", "c", "a", "d") # desired order of the rows
order = match(target, df$name)
order ## [1] 2 3 1 4The match
function takes two vectors as arguments. It yields the positions of the
first vector in the second vector (here saved in the integer vector
order). We see that b comes second in df$name,
c comes third in df$name, a comes first in df$name,
and d comes forth in df$name.
Using the information stored in the integer vector order, we
can rearrange the rows of the frame df:
df_ordered = df[order,]
df_ordered## name value
## 2 b TRUE
## 3 c FALSE
## 1 a TRUE
## 4 d FALSEThis is the desired order of the data frame.
We can of course shorten the procedure:
target <- c("b", "c", "a", "d") # desired order of the rows
df_ordered = df[match(target, df$name),]
df_ordered## name value
## 2 b TRUE
## 3 c FALSE
## 1 a TRUE
## 4 d FALSEThis procedure can for instance be used to bring the rows (or columns) of two datasets into the same order, by choosing a column of one frame as target vector:
dfa <- data.frame(name = c("a", "b", "c", "d"), value = c(rep(TRUE, 2), rep(FALSE, 2)))
dfb <- data.frame(name = c("c", "d", "a", "b"), value = 1:4)
dfa## name value
## 1 a TRUE
## 2 b TRUE
## 3 c FALSE
## 4 d FALSEdfb## name value
## 1 c 1
## 2 d 2
## 3 a 3
## 4 b 4The data frames do not have the same order when we focus on the name column.
target = dfb$name
order = match(target, dfa$name)
new_dfb = dfb[order,]
new_dfb## name value
## 3 a 3
## 4 b 4
## 1 c 1
## 2 d 2If we want to have this very concise, we can write:
new_dfb = dfb[match(dfb$name, dfa$name),]
new_dfb## name value
## 3 a 3
## 4 b 4
## 1 c 1
## 2 d 2In a similar way, we can also change the order of the columns.
The grep command (or one of its flavours) can be used to search words in character vectors.
v0 = c("A", "B", "C", "D", "A", "a", "B", "B", "D", "E", "E","X", "XX", "DDD", "A", "XX", "FFFF", "0000", "12", "000")
grep("B", v0)## [1] 2 7 8We see that the command outputs the indices of the matches. Using this information, we can create a new vector containing only the matches:
v1 = v0[grep("B", v0)]
v1## [1] "B" "B" "B"The above grep command works only for vectors. If we
want to work on every row or column of a frame or matrix, we can use the
apply
command.
Some example data:
v0 = c("A", "B", "C", "D", "W", "B")
v1 = c("X", "B", "C", "BB", "A", "B")
v2 = c("A", "B", "C", "D", "W", "B")
v3 = c("X", "T", "X", "CC", "S", "U")
Vall = data.frame(one = v0, two = v1, three = v2, four = v3)
rownames(Vall) = c("eins", "zwei", "drei", "vier", "fuenf", "sechs")
Vall## one two three four
## eins A X A X
## zwei B B B T
## drei C C C X
## vier D BB D CC
## fuenf W A W S
## sechs B B B Ugrep("B", Vall)## [1] 1 2 3The output of grep shows only which columns contain the
“B”. If we want the positions in each column of the frame, we
can use the apply command:
apply(Vall, 2, grep, pattern = "B") # "pattern" is a formal parameter of the grep function ## $one
## [1] 2 6
##
## $two
## [1] 2 4 6
##
## $three
## [1] 2 6
##
## $four
## integer(0)We can also search by row. Therefore, we invoke a 1 as second parameter of the command:
apply(Vall, 1, grep, pattern = "B") # "pattern" is a formal parameter of the grep function ## $eins
## integer(0)
##
## $zwei
## [1] 1 2 3
##
## $drei
## integer(0)
##
## $vier
## [1] 2
##
## $fuenf
## integer(0)
##
## $sechs
## [1] 1 2 3There are more flavours of the apply function, all
listed here.
Did you notice that the command also considered “BB” as a match? (See the row named vier.)
If we only want matches with a single “B”, we have to modify the search pattern:
apply(Vall, 1, grep, pattern = "^B$")## $eins
## integer(0)
##
## $zwei
## [1] 1 2 3
##
## $drei
## integer(0)
##
## $vier
## integer(0)
##
## $fuenf
## integer(0)
##
## $sechs
## [1] 1 2 3Now, there isn’t any match in the row named vier.
Construction of such search patterns, called regular expressions, can be quite complicated. Explanation of regular expressions is out of the scope of this document. However, there are lots of manuals and cheatsheets on the web, for instance here or here.
Note that all code snippets are designed for comprehensibility, not for efficiency!