cran_optimization_survey.Rmd

---
title: "R Optimization Survey"
author: "Seth Russell"
date: "8/7/2018"
output:
  pdf_document: default
  html_document: default
editor_options:
  chunk_output_type: console
---


```{r setup, include=FALSE, echo=TRUE}
library(dplyr)
library(stringi)
library(ggplot2)

# these packages are used for a range of items related to optimization - compiled code, multi-processing, etc.
look_for_packages <- c('Rcpp', 'tictoc', 'rbenchmark', 'microbenchmark', 'bench', 'benchr', 'profr', 'profvis', 'snow', 'doSNOW', 'parallel', 'doParallel', 'Rmpi', 'foreach', 'future', 'future.apply', 'SparkR', 'sparklyr', 'batchtools', 'RcppParallel', 'parallelDist', 'parallelMap', 'doMC', 'doMPI', 'partools', 'DSL')
# Notes about package selection
#Timing tools
#  system.time() - time how long something takes
#  Sys.time() - get current time
#  tictoc - package
#  rbenchmark - package
#  microbenchmark - Package
#  benchr - package
#  bench - new package from tidyverse
#  Rprof - function
#  profr - Package
#  profvis - Package
#
#Parallelization packages
#  snow
#  doSNOW
#  parallel - R started with release 2.14.0 which includes a new package parallel incorporating (slightly revised) copies of packages multicore and snow.
#  doParallel
#  Rmpi - Rmpi provides an interface (wrapper) to MPI APIs. It also provides interactive R slave environment.
#  foreach - Executing the loop in parallel.
#  future - A minimal, efficient, cross-platform unified Future API for parallel and distributed processing in R; designed for beginners as well as advanced developers.
#  SparkR - R frontend for Spark.
#  DistributedR - A scalable high-performance platform from HP Vertica Analytics Team. Not available via CRAN, so not part of analysis.
#  ddR - Provides distributed data structures and simplifies distributed computing in R. Not considered as is now listed as status of abandoned on CRAN
#  sparklyr - R interface for Apache Spark from RStudio.
#  batchtools - High performance computing with LSF, TORQUE, Slurm, OpenLava, SGE and Docker Swarm.
#  RcppParallel
#  threadpool - Not considered as not available via CRAN, only available via github
#  parallelDist
#  parallelMap
#  doMC
#  doMPI
#  Rmpi
#  partools
#  DSL - has MapReduce function

```


```{r importsharedfn, echo=FALSE, cache=FALSE}
# update this path based on where you have checked out the git repository
source("~/SoftwareEngineeringPrinciples/sourcecode/shared_fn.R")
```

## Introduction

This notebook performs an analysis of use of optimization related packages and methodologies for all R packages avaialble on CRAN.

Two methods are used and results are compared.

*Method 1:* Grep for non-empty src directories. By convention, packages using compiled code (C, C++, Fortran) place files in the '/src' directory.

*Method 2:* Check for stated dependencies on packages.

Potential problems/limitations:

The best evidence of optimization would be in history of commits and unit test runs. While all packages have source code available, not all have development history available nor unit tests available.

Use of one of these specified packages doensn't mean user is trying to optimize. Package dependencies can be misleading - I've seen CRAN packages that list packages as dependencies and by inspection of their source code, they never use it.

## Grepping for non-empty src directory

liquidSVM is a good example of an optimized and tested package - see https://arxiv.org/pdf/1702.06899.pdf
* has no optimization related dependencies (so wont show up in depedency based section)
* core of package implemented in C++

Iso package has Fortran code that pre-dates 2013/
* Does it use Fortran code for performance reasons, ease of implementation, legacy reasons, or ???

From manual review, it seems that all C++ and Fortran code is in src directory. External/third party libraries are usually included in other directories.

One potential problem is use of Java - while it seems that due to how rJava allows Java code to be called from R, a memory perfomance hit may occur - but some specific packages do mention that using Java threading improves performance - see package 'rmcfs' as an example

At this point, just look for non empty src directory...

```{r grep_packages,results="hide"}
# if value 100%, then read all packages, otherwise, randomly select number of packages provided
sample_size <- '100%'
# dont need to untar, but is useful for manual analysis
untar_files <- FALSE

# call code from shared_fn.R to download files and extract them if desired
package_df <- initialize(search_pattern="src[^/]*/.+", sample_size=sample_size, untar_files=untar_files)

if (nrow(package_df[package_df$download_error == TRUE, ]) > 0) {
  print('Unable to download or untar these packages - they will not be considered in analysis')
  print(package_df[package_df$download_error == TRUE, ])
} else {
  print('It appears that all files downloaded and untared successfully.')
}

print(paste('As of', date(), 'there are', nrow(package_df), 'packages available on CRAN'))
```

### Visualizations and Tables for Grep results

```{r grep_based_visualizations}
# build data needed for plots
gtf <- grep_table_freqs(package_df)
gtt <- grep_table_totals(gtf)

# generate plots
freq_plot <- grep_viz_freq(gtf, image_prefix = 'optimization', label = 'Compiled')
freq_plot
pct_plot <- grep_viz_pct(gtt, image_prefix = 'optimization', label = 'Compiled')
pct_plot
```

## Package Dependencies

```{r dependencies}
# function from shared_fn.R - sets package_df$package_deps
package_df <- calc_dependencies(package_df)

# are there any that have more than one dependency listed?
# these are all counted multiple times in histogram
show_multiple_dependencies(package_df, look_for_packages)
```

### Visualizations and Tables for Dependency results

```{r dependency_based_visualizations}
# need to merge transformed_df with package_df to keep those with no dependency and create better histogram
transformed_df <- transform_df(package_df)
transformed_df <- merge(transformed_df, package_df[package_df$package_deps == 'none', ][colnames(transformed_df)], all=TRUE)

dep_freqs <- aggregate(strftime(transformed_df$date, "%Y"),
                       by=list(strftime(transformed_df$date, "%Y"), transformed_df$package_deps),
                       FUN=length)

names(dep_freqs) <- c('Year', 'Dependency', 'Count')
dep_freqs$Dependency <- reorder_none(dep_freqs$Dependency)

fplot <- ggplot(data=dep_freqs[dep_freqs$Count > 14, ], aes(x=Year, y=Count, fill=Dependency)) +
  geom_bar(stat="identity") +
  scale_fill_viridis_d(option="inferno", end=0.96) +
  xlab("Year Package Last Updated") +
  labs(fill = "Dependency n > 14")
fplot
ggsave(filename = paste0(image_base, "optimization_dependency_stacked_bar.png"), fplot,
       width = 7.2, height = 5.5, dpi = 600, units = "in", device='png')

dep_totals <- dependency_table(dep_freqs)
print('Dependency Table:')
dep_totals
```

## Method comparison

As the results from method 1 do not match the results from method 2, explore some of the differences.

Some points discovered:
* It is possible to list a one of the sought for packages as a dependency and not acutally use the package
* Speculation - Optimization can occur in many ways. Some well known items are (at the time of this writing) are to prefer matrices over data.frame or to avoid using data.frames with the apply family of functions. It's impossible to determine for all packages what process the developer went through and what options were considered when arriving at the currently available package version.

As new packages are being released and updated all the time, these numbers will change.

```{r comparison}
# About 846 more appear to use compiled code
print("Difference between packages that have src directory vs dependency on optimization framework:")
nrow(package_df[package_df$found == TRUE, ])  - nrow(package_df[package_df$package_deps != 'none', ])

# About 1701 packages that don't list a dependency to one of the specified packages, but have a src directory for compiled code
only_grep <- package_df[package_df$found == TRUE & package_df$package_deps == 'none', ]
print(paste("Number of packages that don't list a dependency to one of the specified packages, but have a src directory for compiled code:", nrow(only_grep)))

# About 855 packages that list a dependency to one of the specified packages but don't have compiled code
only_dep <- package_df[package_df$found == FALSE & package_df$package_deps != 'none', ]
print(paste("Number of packages that list a dependency to one of the specified packages but don't have compiled code:", nrow(only_dep)))

```