cran_testing_survey.Rmd

---
title: "R Testing Package Survey"
author: "Seth Russell"
date: "6/19/2018"
output:
  pdf_document: default
  html_document: default
editor_options:
  chunk_output_type: console
---

This notebook performs an analysis of use of testing methodologies for all R packages avaialble on CRAN.

Two methods are used and results are compared.

*Method 1:* Grep for non-empty testing directories. The commonly used R testing packages all recommend placing tests in a directory by themselves, which we look for.

*Method 2:* Check for stated dependencies on testing packages. It is considered best practice to list dependencies on a testing framework even though standard usage of a package may not require it.


I've Picked 2008 (10 years) as the last updated cutoff year based on manual inspection. While there are still packages in CRAN that were last updated in 2007 and prior, there are not very many, so ignore them in the visualizations.


```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(dplyr)
library(stringi)
library(ggplot2)

# Testing packages
look_for_packages <- c("svUnit", "RUnit", "testthat", "testit", "unitizer", "unittest")
# testit is a minimal tool for testing packages, but it can be used for unit testing - e.g. tinytex
# Packages considered but excluded:
#   webmockr - Library for mocking (or "stubbing out") HTTP requests. Although it wasn't listed as a dependency when I checked, some packages now do use it to aid in unit testing. 
#   vcr - Package that is for "caching HTTP requests and responses, primarily in unit tests".
#   vdiffr - Is a extension for testthat, so requires testthat to work
#
# Users could use core R functionality for unit tests such as stopifnot() instead of packages
# we don't attempt to quantify the scope of software that does unit testing without relying on
# a package
```

```{r importsharedfn, echo=FALSE, cache=FALSE}
# update this path based on where you have checked out the git repository
source("~/SoftwareEngineeringPrinciples/sourcecode/shared_fn.R")
```

## Method 1 of testing analysis

This method downloads packages from CRAN and evaluates the files and folders for the presence of a non-empty test directory.

Output hidden as when running against entire CRAN it will be very large.

```{r grep_packages,results="hide"}
# if value 100%, then read all packages, otherwise, randomly select number of packages provided
sample_size <- '100%'
# dont need to untar, but is useful for manual analysis
untar_files <- FALSE

# Although most unit testing packages use test/ or tests/ as the common base directory, some
# such as svUnit recommend the directory /inst/unitTests
#
# code at https://github.com/rorynolan/exampletestr/blob/master/analysis/CRAN_test_analysis.Rmd misses some
# test directories since it checks for "^tests/.+". As an example, R packages using svUnit (such as gsubfn)
# often call the test directory unitTests
#

# call code from shared_fn.R to download files and extract them if desired
package_df <- initialize(search_pattern="[Tt]est[^/]*/.+", sample_size=sample_size, untar_files=untar_files)

if (nrow(package_df[package_df$download_error == TRUE, ]) > 0) {
  print('Unable to download or untar these packages - they will not be considered in analysis')
  print(package_df[package_df$download_error == TRUE, ])
} else {
  print('It appears that all files downloaded and untared successfully.')
}

print(paste('As of', date(), 'there are', nrow(package_df), 'packages available on CRAN'))
```

### Visualizations for method 1

Histogram by year with testing packages shown

```{r grep_based_graphics}
# build data needed for plots
gtf <- grep_table_freqs(package_df)
gtt <- grep_table_totals(gtf)

# generate plots
freq_plot <- grep_viz_freq(gtf, image_prefix = 'testing', label = 'Testing')
freq_plot
pct_plot <- grep_viz_pct(gtt, image_prefix = 'testing', label = 'Testing')
pct_plot
```

## Method 2 of testing analysis

```{r dependencies}
# function from shared_fn.R - sets package_df$package_deps
package_df <- calc_dependencies(package_df)

# are there any that have more than one dependency listed?
# these are all counted multiple times in histogram
show_multiple_dependencies(package_df, look_for_packages)
```

### Visualizations for method 2

Histogram by year with testing packages shown

```{r dependency_based_visualizations}
# need to merge transformed_df with package_df to keep those with no dependency and create better histogram
transformed_df <- transform_df(package_df)
transformed_df <- merge(transformed_df, package_df[package_df$package_deps == 'none', ][colnames(transformed_df)], all=TRUE)

dep_freqs <- aggregate(strftime(transformed_df$date, "%Y"),
                       by=list(strftime(transformed_df$date, "%Y"), transformed_df$package_deps),
                       FUN=length)

names(dep_freqs) <- c('Year', 'Dependency', 'Count')
dep_freqs$Dependency <- reorder_none(dep_freqs$Dependency)

fplot <- ggplot(data=dep_freqs[dep_freqs$Year > 2007, ], aes(x=Year, y=Count, fill=Dependency)) +
  geom_bar(stat="identity") +
  scale_fill_viridis_d(option="inferno", end = 0.96) +
  xlab("Year Package Last Updated") +
  labs(fill = "Framework")
fplot
ggsave(filename = paste0(image_base, "testing_dependency_stacked_bar.png"), fplot,
       width = 7.2, height = 5.5, dpi = 600, units = "in", device='png')

dep_totals <- dependency_table(dep_freqs)
print('Dependency Table:')
dep_totals
```

### Method comparison

As the results from method 1 do not match the results from method 2, explore some of the differences.

Some points discovered:
* It is possible for an R package to use a non-standard testing directory, even if they also use a standardized testing framework
* It is possible to list a testing framework as a dependency and not actually use the framework.
* Speculation - Testing can be performed in-line with other code through use of stop() or stopifnot().
* Speculation - Developers may have their own test cases they run while developing their software, but do not commit to a repository or share with others.

As new packages are being released and updated all the time, these numbers will change.

```{r comparison}
# About 859 more appear to have testing code...
print("Difference between packages that have testing code vs dependency on testing framework:")
nrow(package_df[package_df$found == TRUE, ])  - nrow(package_df[package_df$package_deps != 'none', ])

# About 1070 packages that don't list a dependency to a testing framework, but have a testing directory
# Randomly looked at several: some don't use a framework (e.g. xlsx)
# others use a framework but don't list it as a dependency (e.g. redcapAPI)
only_grep <- package_df[package_df$found == TRUE & package_df$package_deps == 'none', ]
print(paste("Number of packages that don't list a dependency to a testing framework, but have a testing directory:", nrow(only_grep)))

# About 211 packages that list a dependency to a testing framework, but don't use one
only_dep <- package_df[package_df$found == FALSE & package_df$package_deps != 'none', ]
print(paste("Number of packages that list a dependency to a testing framework, but don't use one:", nrow(only_dep)))

```