(M)achine (L)earning (F)unction (A)pproximation
This code implements the fully-connected Deep Neural Network (DNN) architectures considered in the paper "The gap between theory and practice in function approximation with deep neural networks" available at https://arxiv.org/abs/2001.07523
The DNN.py code accepts a variety of command line inputs, specifying everything from the optimizer to use in solving to the width, depth, and activation function of the networks. Various building-blocks are available for constructing the networks, including the default activation block and ResNet blocks in resnet_v1 and resnet_v2. The DNN code accepts MATLAB .mat format data files containing an array X of points and Y of function values, and generates "run_data.mat" and "ensemble_data.mat" files containing information about the runs and testing data, respectively.
This package is compatible with version 1.13 of Tensorflow. Arguments are specified on the command line as follows:
python DNN.py --nb_layers 10
--nb_nodes_per_layer 100
--nb_train_points 750
--train_pointset uniform_random
--nb_epochs 50000
--batch_size 750
--nb_trials 20
--train 0
--make_plots 0
--nb_test_points 609025
--test_pointset CC_sparse_grid
--blocktype default
--activation relu
--example exp_cos
--optimizer Adam
--quiet 0
--input_dim 8
--output_dim 1
--MATLAB_data 1
--trial_num 1
--precision single
--run_ID test_exp_cos
--use_regularizer 0
--reg_lambda 1e-3
--error_tol 5e-7
--initializer normal
--sigma 1e-1
--lrn_rate_schedule exp_decay
We also include a shell script "run_train_test_local.sh" to simplify running ensembles of trials in testing DNN performance. To use the script, modify the variables as desired and type:
bash run_train_test_local.sh
Difficulty of approximating a piecewise continuous function with a ReLU DNN with 15 layers and 200 nodes per layer:
Training a ReLU network with 2 layers and 200 nodes per layer:
Training a ReLU network to approximate a smooth function:
Training a ReLU network to approximate a more oscillatory function:
