Bug fixes. Refactored of scripts & README content.

Author: uc-msft

samples/features/sql-big-data-cluster/README.md

@@ -11,33 +11,46 @@ Installation instructions for SQL Server 2019 big data cluster can be found [her
 
 ## Samples Setup
 
-**Before you begin**, download the sample database [backup file](https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak) and save it locally. Run the CMD script called *bootstrap-sample-db.cmd* or the shell script *bootstrap-sample-db.sh* depending on your platform. This script will restore the database on the SQL Master instance, execute the *bootstrap-sample-db.sql* script, create the database objects needed, export the web_clickstreams & inventory tables to CSV file, and upload the web_clickstreams CSV file to HDFS inside the SQL Server 2019 big data cluster.
+**Before you begin**, run the CMD script called [bootstrap-sample-db.cmd](bootstrap-sample-db.cmd) or the shell script [bootstrap-sample-db.sh](bootstrap-sample-db.sh) depending on your platform. This script does the following operations:
+
+1. Downloads the tpcx-bb 1GB sample database
+1. Restores the database on the SQL Master instance
+1. Executes the bootstrap-sample-db.SQL script
+1. Exports the web_clickstreams, inventory, customer & product_reviews tables to files
+1. Uploads the web_clickstreams CSV file to the HDFS inside the SQL Server 2019 big data cluster
 
 __[data-pool](data-pool/)__
 
+SQL Server 2019 big data cluster contains a data pool which consists of many SQL Server instances to store data & query in a scale-out manner.
+
 ### Data ingestion using Spark
-Connect to the master instance in your SQL Server big data cluster and the SQL Server big data cluster endpoint, and follow the steps in *data-pool/data-ingestion-spark.sql*.
+The sample script [data-pool/data-ingestion-spark.sql](data-pool/data-ingestion-spark.sql) shows how to perform data ingestion from Spark into data pool table(s).
 
 ### Data ingestion using sql
-Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-pool/data-ingestion-sql.sql*.
+The sample script [data-pool/data-ingestion-sql.sql](data-pool/data-ingestion-sql.sql) shows how to perform data ingestion from T-SQL into data pool table(s).
 
 __[data-virtualization](data-virtualization/)__
 
-### External table over HDFS
-Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-virtualization/external-table-hdfs.sql*.
+SQL Server 2019 or SQL Server 2019 big data cluster can use PolyBase external tables to connect to other data sources.
+
+### External table over Storage Pool
+SQL Server 2019 big data cluster contains a storage pool consisting of HDFS, Spark and SQL Server instances. The [data-virtualization/storage-pool](data-virtualization/storage-pool) folder contains samples that demonstrate how to query data in HDFS inside SQL Server 2019 big data cluster.
 
 ### External table over Oracle
-To execute this sample script, you will need following:
-1. Oracle instance and credentials
-1. Create inventory table in Oracle using [data-virtualization/inventory-oracle.sql](data-virtualization/inventory-oracle.sql/) script
-1. Import the inventory.csv file generated by the bootstrap-sample-db script to a table in Oracle
+SQL Server 2019 uses new ODBC connectors to enable connectivity to SQL Server, Oracle, Teradata, MongoDB and generic ODBC data sources.
 
-Connect to the master instance in your SQL Server big data cluster and execute the steps in *data-virtualization/external-table-oracle.sql*.
+The [data-virtualization/oracle](data-virtualization/oracle) folder contains samples that demonstrate how to query data in Oracle using external tables.
+
+__[deployment](deployment/)__
+
+The [deployment](deployment) folder contains the scripts for deploying a Kubernetes cluster for SQL Server 2019 big data cluster.
 
 __[machine-learning](machine-learning/)__
 
-### SQL Server ML Services on master instance
-Connect to the master instance in your SQL Server big data cluster and execute the steps in *machine-learning/sql/book-category-r-ml.sql*.
+SQL Server 2016 added support executing R scripts from T-SQL. SQL Server 2017 added support for executing Python scripts from T-SQL. SQL Server 2019 adds support for executing Java code from T-SQL. SQL Server 2019 big data cluster adds support for executing Spark code inside the big data cluster.
+
+### SQL Server Machine Learning Services
+The [machine-learning\sql](machine-learning\sql) folder contains the sample SQL scripts that show how to invoke R, Python, and Java code from T-SQL.
 
-### Spark ML
-Connect to the SQL Server big data cluster endpoint, and run the notebook files *machine-learning/spark/1-data-prep.ipynb* and *machine-learning/spark/2-build-ml-model.ipynb* cell by cell.
+### Spark Machine Learning
+The [machine-learning\spark](machine-learning\spark) folder contains the Spark samples.

samples/features/sql-big-data-cluster/bootstrap-sample-db.cmd

@@ -1,17 +1,18 @@
 @echo off
 REM bootstrap sample database CMD script
 setlocal enableextensions
+setlocal enabledelayedexpansion
 set CLUSTER_NAMESPACE=%1
 set SQL_MASTER_IP=%2
 set SQL_MASTER_SA_PASSWORD=%3
-set KNOX_IP=%3
-set KNOX_PASSWORD=%4
+set KNOX_IP=%4
+set KNOX_PASSWORD=%5
 set STARTUP_PATH=%~dp0
+set TMP_DIR_NAME=%~nx0
 
 if NOT DEFINED CLUSTER_NAMESPACE goto :usage
 if NOT DEFINED SQL_MASTER_IP goto :usage
 if NOT DEFINED SQL_MASTER_SA_PASSWORD goto :usage
-if NOT DEFINED BACKUP_FILE_PATH goto :usage
 if NOT DEFINED KNOX_IP goto :usage
 if NOT DEFINED KNOX_PASSWORD set KNOX_PASSWORD=%SQL_MASTER_SA_PASSWORD%
 
@@ -23,49 +24,52 @@ for %%F in (sqlcmd.exe bcp.exe kubectl.exe curl.exe) do (
 )
 
 pushd "%tmp%"
+md %TMP_DIR_NAME%
 echo Downloading sample database backup file...
-curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
+%DEBUG% curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
 
 REM Copy the backup file, restore the database, create necessary objects and data file
 echo Copying database backup file...
 %DEBUG% kubectl cp tpcxbb_1gb.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n %CLUSTER_NAMESPACE% || goto exit
 
 del tpcxbb_1gb.bak >NUL
-popd
 
 echo Configuring sample database...
-%DEBUG% sqlcmd -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "%STARTUP_PATH%bootstrap.out" -I -b || goto exit
+%DEBUG% sqlcmd -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -i "%STARTUP_PATH%bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || goto exit
 
 for %%F in (web_clickstreams inventory customer) do (
     echo Exporting %%F data...
     if /i %%F EQU web_clickstreams (set DELIMITER=,) else (SET DELIMITER=^|)
-    %DEBUG% bcp sales.dbo.%%F out "%STARTUP_PATH%%%F.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t"%DELIMITER%" -o "%STARTUP_PATH%%%F.out" -e "%STARTUP_PATH%%%F.err" || goto exit
+    %DEBUG% bcp sales.dbo.%%F out "%STARTUP_PATH%%%F.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t"!DELIMITER!" -o "%%F.out" -e "%%F.err" || goto exit
 )
 
 echo Exporting product_reviews data...
-%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), '\"', '') from sales.dbo.product_reviews" queryout "%STARTUP_PATH%product_reviews.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "%STARTUP_PATH%product_reviews.out" -e "%STARTUP_PATH%product_reviews.err" || goto exit
+%DEBUG% bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "%TMP_DIR_NAME%product_reviews.csv" -S %SQL_MASTER_INSTANCE% -Usa -P%SQL_MASTER_SA_PASSWORD% -c -t, -o "product_reviews.out" -e "product_reviews.err" || goto exit
 
 REM Copy the data file to HDFS
-pushd "%STARTUP_PATH%"
 echo Uploading web_clickstreams data to HDFS...
 %DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || goto exit
-%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create" -H "Content-Type: application/octet-stream" -T "web_clickstreams.csv" || goto exit
+%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "web_clickstreams.csv" || goto exit
 
+echo.
 echo Uploading product_reviews data to HDFS...
 %DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || goto exit
-%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || goto exit
-:: del /q *.out *.err *.csv
-popd
+%DEBUG% curl -i -L -k -u root:%KNOX_PASSWORD% -X PUT "https://%KNOX_ENDPOINT%/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || goto exit
 
+%DEBUG% del /q *.out *.err *.csv
+
+popd
+%DEBUG% rd /q "%tmp%\%TMP_DIR_NAME%"
 endlocal
 exit /b 0
 goto :eof
 
 :exit
     echo Bootstrap of the sample database failed.
+    echo Output and error files are in directory [%TMP%\%TMP_DIR_NAME%].
     exit /b 1
 
 :usage
-    echo USAGE: %0 ^<CLUSTER_NAMESPACE^> ^<SQL_MASTER_IP^> ^<SQL_MASTER_SA_PASSWORD^> ^<BACKUP_FILE_PATH^> ^<KNOX_IP^> [^<KNOX_PASSWORD^>]
+    echo USAGE: %0 ^<CLUSTER_NAMESPACE^> ^<SQL_MASTER_IP^> ^<SQL_MASTER_SA_PASSWORD^> ^<KNOX_IP^> [^<KNOX_PASSWORD^>]
     echo Default ports are assumed for SQL Master instance ^& Knox gateway.
     exit /b 0

samples/features/sql-big-data-cluster/bootstrap-sample-db.sh

@@ -1,8 +1,10 @@
 #!/bin/bash
 set -e
 set -o pipefail
+STARTUP_PATH=$(dirname $0)
+TMP_DIR_NAME=$(basename $0)
 USAGE_MESSAGE="USAGE: $0 <CLUSTER_NAMESPACE> <SQL_MASTER_IP> <SQL_MASTER_SA_PASSWORD> <KNOX_IP> [<KNOX_PASSWORD>]"
-ERROR_MESSAGE="Bootstrap of the sample database failed."
+ERROR_MESSAGE="Bootstrap of the sample database failed. Output and error files are in directory [/tmp/$TMP_DIR_NAME]."
 
 # Print usage if mandatory parameters are missing
 : "${1:?$USAGE_MESSAGE}"
@@ -26,41 +28,51 @@ KNOX_ENDPOINT=$KNOX_IP:30443
 
 for util in sqlcmd.exe bcp.exe kubectl.exe curl.exe
     do
-    echo Verifying $util is in path & which $util 1>NUL 2>NUL || (echo Unable to locate $util && exit 1)
+    echo Verifying $util is in path & which $util 1>/dev/nul 2>/dev/nul || (echo Unable to locate $util && exit 1)
 done
 
 # Copy the backup file, restore the database, create necessary objects and data file
 pushd "/tmp"
+$DEBUG mkdir "$TMP_DIR_NAME"
 echo Downloading sample database backup file...
 $DEBUG curl -G "https://sqlchoice.blob.core.windows.net/sqlchoice/static/tpcxbb_1gb.bak" -o tpcxbb_1gb.bak
 
 echo Copying database backup file...
 $DEBUG kubectl cp tpcxbb_1gb.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n $CLUSTER_NAMESPACE || (echo $ERROR_MESSAGE && exit 1)
-rm tpcxbb_1gb.bak
-popd
+$DEBUG rm tpcxbb_1gb.bak
 
 echo Configuring sample database...
 # WSL ex: "/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn/SQLCMD.EXE"
-$DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -i "bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || (echo $ERROR_MESSAGE && exit 2)
+$DEBUG sqlcmd -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -i "$STARTUP_PATH\bootstrap-sample-db.sql" -o "bootstrap.out" -I -b || (echo $ERROR_MESSAGE && exit 2)
 
-for table in web_clickstreams inventory
+for table in web_clickstreams inventory customer
     do
     echo Exporting $table data...
+    if [ $table == web_clickstreams ]
+    then 
+        DELIMITER=,
+    else
+        DELIMITER="|"
+    fi
     # WSL ex: "/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn/bcp.exe"
-    $DEBUG bcp sales.dbo.$table out "$table.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "$table.err" || (echo $ERROR_MESSAGE && exit 3)
+    $DEBUG bcp sales.dbo.$table out "$table.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t"$DELIMITER" -e "$table.err" || (echo $ERROR_MESSAGE && exit 3)
 done
 
 echo Exporting product_reviews data...
-$DEBUG bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), '\"', '') from sales.dbo.product_reviews" queryout "product_reviews.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "product_reviews.err" || (echo $ERROR_MESSAGE && exit 3)
+$DEBUG bcp "select pr_review_sk, replace(replace(pr_review_content, ',', ';'), char(34), '') as pr_review_content from sales.dbo.product_reviews" queryout "product_reviews.csv" -S $SQL_MASTER_INSTANCE -Usa -P$SQL_MASTER_SA_PASSWORD -c -t, -e "product_reviews.err" || (echo $ERROR_MESSAGE && exit 3)
 
 # Copy the data file to HDFS
 echo Uploading web_clickstreams data to HDFS...
 $DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 4)
-$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create" -H 'Content-Type: application/octet-stream' -T "web_clickstreams.csv" || (echo $ERROR_MESSAGE && exit 5)
+$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/clickstream_data/web_clickstreams.csv?op=create&overwrite=true&noredirect=true" -H 'Content-Type: application/octet-stream' -T "web_clickstreams.csv" || (echo $ERROR_MESSAGE && exit 5)
 
+echo
 echo Uploading product_reviews data to HDFS...
 $DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS" || (echo $ERROR_MESSAGE && exit 6)
-$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || (echo $ERROR_MESSAGE && exit 7)
+$DEBUG curl -i -L -k -u root:$KNOX_PASSWORD -X PUT "https://$KNOX_ENDPOINT/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true&noredirect=true" -H "Content-Type: application/octet-stream" -T "product_reviews.csv" || (echo $ERROR_MESSAGE && exit 7)
+
+$DEBUG rm -f *.out *.err *.csv
+popd
 
-# rm -f *.out *.err *.csv
+$DEBUG rmdir "/tmp/$TMP_DIR_NAME"
 exit

samples/features/sql-big-data-cluster/bootstrap-sample-db.sql

@@ -25,11 +25,14 @@ IF NOT EXISTS(SELECT * FROM sys.external_data_sources WHERE name = 'SqlStoragePo
 	WITH (LOCATION = 'sqlhdfs://service-mssql-controller:8080');
 GO
 
--- Create view used for ML services training stored procedure
-CREATE OR ALTER VIEW [dbo].[web_clickstreams_book_clicks]
+-- Create view used for ML services training and scoring stored procedures
+CREATE OR ALTER  VIEW [dbo].[web_clickstreams_book_clicks]
 AS
 	SELECT
-	  q.clicks_in_category,
+        /* There is a bug in TPCx-BB data generator which results in data where all users have purchased books.
+        As a result, we cannot use the data as is for ML training purposes. So we will treat users with 1-5 clicks
+		in the book category as not interested in books. */
+	  CASE WHEN q.clicks_in_category < 6 THEN 0 ELSE q.clicks_in_category END AS clicks_in_category,
 	  CASE WHEN cd.cd_education_status IN ('Advanced Degree', 'College', '4 yr Degree', '2 yr Degree') THEN 1 ELSE 0 END AS college_education,
 	  CASE WHEN cd.cd_gender = 'M' THEN 1 ELSE 0 END AS male,
 	  COALESCE(cd.cd_credit_rating, 'Unknown') as cd_credit_rating,
@@ -62,15 +65,4 @@ AS
 	) AS q
 	INNER JOIN customer as c ON q.wcs_user_sk = c.c_customer_sk
 	INNER JOIN customer_demographics as cd ON c.c_current_cdemo_sk = cd.cd_demo_sk;
-GO
-
--- Create table for storing the machine learning models
-IF NOT EXISTS(SELECT * FROM sys.tables WHERE name = 'sales_models')
-	CREATE TABLE sales_models (
-		model_name varchar(100) NOT NULL PRIMARY KEY,
-		model varbinary(max) NOT NULL,
-		model_native varbinary(max) NULL,
-		created_by nvarchar(300) NOT NULL DEFAULT(SYSTEM_USER),
-		create_time datetime2 NOT NULL DEFAULT(SYSDATETIME())
-	);
-GO
+GO

samples/features/sql-big-data-cluster/data-virtualization/oracle/README.md

@@ -1,6 +1,6 @@
 # Data virtualization in SQL Server 2019
 
-**Applies to: SQL Server 2019 on Windows or Linux, SQL Server 2019 big data cluster**
+***Applies to:*** SQL Server 2019 on Windows or Linux, SQL Server 2019 big data cluster
 
 SQL Server 2019 introduces new ODBC connectors to data sources like SQL Server, Oracle, MongoDB and Teradata. These connectors can be used from stand-alone SQL Server 2019 on Windows or Linux or SQL Server 2019 big data cluster.
 

samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/README.md

@@ -1,11 +1,9 @@
 # Oracle setup
 
-This folder contains scripts that can be executed on Oracle server to create the necessary objects for data virtualization in SQL Server 2019 big data cluster.
+This folder contains scripts that can be executed on Oracle server to create the necessary objects for data virtualization in SQL Server 2019+ or SQL Server 2019+ big data cluster.
 
 # Instructions
 
-1. Connect to Oracle instance.
+***Before you begin***, you need the Oracle instance name and credentials.
 
-1. Execute the [sales-user.sql](sales-user.sql). This script creates the sample user. If there is name conflict please change the script user/credentials.
-
-1. Execute the [inventory.sql](inventory.sql). This script creates the inventory table.
+1. Execute the [bootstrap-oracle.cmd](bootstrap-oracle.cmd) to create the necessary objects in Oracle

samples/features/sql-big-data-cluster/data-virtualization/oracle/setup/bootstrap-oracle.cmd

@@ -9,19 +9,21 @@ if NOT DEFINED ORACLE_SERVER goto :usage
 if NOT DEFINED ORACLE_USER goto :usage
 if NOT DEFINED ORACLE_PASSWORD goto :usage
 
-echo Verifying sqlplus.exe is in path & CALL WHERE /Q sqlplus.exe || GOTO exit
-echo Verifying sqlldr.exe is in path & CALL WHERE /Q sqlldr.exe || GOTO exit
+for %F in (sqlplus.exe sqlldr.exe) do (
+    echo Verifying %%F is in path & CALL WHERE /Q %%F || GOTO exit
+)
 
-echo Creating user & tables...
-echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @sales-user.sql || GOTO exit
-echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @inventory.sql || GOTO exit
-echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @customer.sql || GOTO exit
 
-echo Loading tables data...
-sqlldr CONTROL=inventory.ctl userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit
-sqlldr CONTROL=customer.ctl userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit
+for %%F in (sales-user.sql inventory.sql customer.sql) do (
+    echo Executing [%%F]...
+    echo exit | sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% @sales-user.sql || GOTO exit
+)
+
+for %%F in (inventory.ctl customer.ctl) do (
+    echo Loading [%%F]...
+    sqlldr CONTROL=%%F userid=%ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SERVER% || GOTO exit
+)
 
-:: del /q *.out *.err *.csv
 endlocal
 exit /b 0
 goto :eof