Merge pull request #3 from microsoft/master

merge back

Author: bweissman

samples/features/optimize-for-sequential-key/README.md

@@ -2,7 +2,7 @@
 
 # OPTIMIZE_FOR_SEQUENTIAL_KEY
 
-In SQL Server 2019, a new index option was added called OPTIMIZE_FOR_SEQUENTIAL_KEY that is intended to address an issue known as [last page insert contention](https://support.microsoft.com/kb/4460004). Most of the solutions to this problem that have been suggested in the past involve making changes to either the application or the structure of the contentious index, which can be costly and sometimes involve performance trade-offs. Rather than making major structural changes, OPTIMIZE_FOR_SEQUENTIAL_KEY addresses some of the SQL Server scheduling issues that can lead to severely reduced throughput when last page insert contention occurs. Using the OPTMIZE_FOR_SEQUENTIAL_KEY index option can help maintain consistent throughput in high-concurrency environments when the following conditions are true:
+In SQL Server 2019, a new index option was added called [OPTIMIZE_FOR_SEQUENTIAL_KEY](https://docs.microsoft.com/sql/t-sql/statements/create-index-transact-sql#sequential-keys) that is intended to address an issue known as [last page insert contention](https://support.microsoft.com/kb/4460004). Most of the solutions to this problem that have been suggested in the past involve making changes to either the application or the structure of the contentious index, which can be costly and sometimes involve performance trade-offs. Rather than making major structural changes, OPTIMIZE_FOR_SEQUENTIAL_KEY addresses some of the SQL Server scheduling issues that can lead to severely reduced throughput when last page insert contention occurs. Using the OPTMIZE_FOR_SEQUENTIAL_KEY index option can help maintain consistent throughput in high-concurrency environments when the following conditions are true:
 
 - The index has a sequential key
 - The number of concurrent insert threads to the index far exceeds the number of schedulers (in other words logical cores)
@@ -80,4 +80,6 @@ The code included in this sample is not intended to be a set of best practices o
 
 For more information, see these articles:
 
-[CREATE INDEX - Sequential Keys](https://docs.microsoft.com/sql/t-sql/statements/create-index-transact-sql#sequential-keys)
+[CREATE INDEX - Sequential Keys](https://docs.microsoft.com/sql/t-sql/statements/create-index-transact-sql#sequential-keys)
+
+[Behind the Scenes on OPTIMIZE_FOR_SEQUENTIAL_KEY](https://techcommunity.microsoft.com/t5/SQL-Server/Behind-the-Scenes-on-OPTIMIZE-FOR-SEQUENTIAL-KEY/ba-p/806888)

samples/features/sql-big-data-cluster/deployment/aks/README.md

@@ -42,7 +42,7 @@ python deploy-sql-big-data-aks.py
 
 When prompted, provide your input for Azure subscription ID, Azure resource group to create the resources in, and Docker credentials. Optionally, you can also provide your input for below configurations or use the defaults provided:
 - azure_region
-- vm_size - we recommend to use a VM size to accommodate your workload. For an optimal experience while you are validating basic scenarios, we recommend at least 8 vCPUs and 32GB memory across all agent nodes in the cluster. The script uses **Standard_L8s** as default.
+- vm_size - we recommend to use a VM size to accommodate your workload. For an optimal experience while you are validating basic scenarios, we recommend at least 8 vCPUs and 64GB memory across all agent nodes in the cluster. The script uses **Standard_L8s** as default. A default size configuration also uses about 24 disks for persistent volume claims across all components.
 - aks_node_count - this is the number of the worker nodes for the AKS cluster, excluding master node. The script is using a default of 1 agent node. This is the minimum required for this VM size to have enough resources and disks to provision all the necessary persistent volumes.
 - cluster_name - this value is used for both AKS cluster and SQL big data cluster created on top of AKS. Note that the name of the SQL big data cluster is going to be a Kubernetes namespace
 - password - same value is going to be used for all accounts that require user password input: SQL Server master instance SA account, controller user and Knox user

samples/features/sql-big-data-cluster/deployment/aks/deploy-sql-big-data-aks.py

@@ -68,7 +68,7 @@ def executeCmd (cmd):
 executeCmd (command)
 
 print("Creating AKS cluster: "+CLUSTER_NAME)
-command = "az aks create --name "+CLUSTER_NAME+" --resource-group "+GROUP_NAME+" --generate-ssh-keys --node-vm-size "+VM_SIZE+" --node-count "+AKS_NODE_COUNT+" --kubernetes-version 1.12.8"
+command = "az aks create --name "+CLUSTER_NAME+" --resource-group "+GROUP_NAME+" --generate-ssh-keys --node-vm-size "+VM_SIZE+" --node-count "+AKS_NODE_COUNT+" --kubernetes-version 1.13.10"
 executeCmd (command)
 
 command = "az aks get-credentials --overwrite-existing --name "+CLUSTER_NAME+" --resource-group "+GROUP_NAME+" --admin"

samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu-single-node-vm-ad/endpoint-patch.json

@@ -1,20 +1,19 @@
 {
   "patch": [
     {
-      "op": "replace",
-      "path": "spec.pools[?(@.spec.type=='Master')].spec",
-      "value": {
-        "type": "Master",
-        "dnsName": "mastersql.contoso.local",
-        "replicas": 1,
-        "endpoints": [
-          {
-            "name": "Master",
-            "serviceType": "NodePort",
-            "port": 31433
-          }
-        ]
-      }
+      "op": "add",
+      "path": "spec.resources.master.spec.endpoints[0].dnsName",
+      "value": "mastersql.contoso.local"
+    },
+    {
+      "op": "add",
+      "path": "spec.resources.gateway.spec.endpoints[0].dnsName",
+      "value": "knox.contoso.local"
+    },
+    {
+      "op": "add",
+      "path": "spec.resources.appproxy.spec.endpoints[0].dnsName",
+      "value": "app.contoso.local"
     }
   ]
 }

samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu-single-node-vm-ad/security-patch.json

@@ -4,10 +4,9 @@
       "op": "add",
       "path": "security",
       "value": {
-        "useInternalDomain": false,
         "ouDistinguishedName":"OU=bdc,DC=contoso,DC=local",
-        "dnsIpAddresses": ["11.11.111.11"],
-        "domainControllerFullyQualifiedDns": ["VM.CONTOSO.LOCAL"],
+        "dnsIpAddresses": ["00.00.000.00"],
+        "domainControllerFullyQualifiedDns": ["DC.CONTOSO.LOCAL"],
         "realm":"CONTOSO.LOCAL",
         "domainDnsName":"contoso.local",
         "bdcAdminPrincipals": [
@@ -20,12 +19,13 @@
     },
     {
       "op": "add",
-      "path": "spec.endpoints/0",
-      "value": {
-        "name": "Kerberos",
-        "serviceType": "NodePort",
-        "port": 30088
-      }
+      "path": "spec.endpoints[0].dnsName",
+      "value": "controller.contoso.local"
+    },
+    {
+      "op": "add",
+      "path": "spec.endpoints[1].dnsName",
+      "value": "serviceproxy.contoso.local"
     }
   ]
 }

samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu-single-node-vm-ad/setup-bdc-ad.sh

@@ -22,19 +22,7 @@ while true; do
     [ "$password" = "$password2" ] && break
     echo "Password mismatch. Please try again."
 done
-echo ""
-# Get docker credentials for private release.
-#
-read -p "Enter Docker username: " DOCKER_USERNAME
-while true; do
-    read -s -p "Enter Docker Password: " docker_password
-    echo
-    read -s -p "Confirm Docker Password: " docker_password2
-    echo
-    [ "$docker_password" = "$docker_password2" ] && break
-    echo "Password mismatch. Please try again."
-done
-export DOCKER_PASSWORD=$docker_password
+
 echo ""
 
 # Get Domain Service Account Username and Password.
@@ -75,9 +63,9 @@ RETRY_INTERVAL=5
 
 # Variables for pulling dockers.
 #
-export DOCKER_REGISTRY="private-repo.microsoft.com"
-export DOCKER_REPOSITORY="mssql-private-preview"
-export DOCKER_TAG="ctp3.2.1"
+export DOCKER_REGISTRY="mcr.microsoft.com"
+export DOCKER_REPOSITORY="mssql/bdc"
+export DOCKER_TAG="2019-RC1-ubuntu"
 
 # Variables used for azdata cluster creation.
 #
@@ -91,9 +79,10 @@ export STORAGE_CLASS=local-storage
 export PV_COUNT="30"
 
 IMAGES=(
-        mssql-app-service-proxy
-        mssql-appdeploy-init
+	mssql-app-service-proxy
+        mssql-control-watchdog
         mssql-controller
+        mssql-dns
         mssql-hadoop
         mssql-mleap-serving-runtime
         mssql-mlserver-py-runtime
@@ -105,10 +94,13 @@ IMAGES=(
         mssql-monitor-influxdb
         mssql-monitor-kibana
         mssql-monitor-telegraf
+        mssql-security-domainctl
         mssql-security-knox
         mssql-security-support
+        mssql-server
         mssql-server-controller
         mssql-server-data
+        mssql-server-ha
         mssql-service-proxy
         mssql-ssis-app-runtime
 )
@@ -320,39 +312,40 @@ echo "Kubernetes master setup done."
 
 # Pull docker images of SQL Server big data cluster.
 #
+
 echo ""
 echo "############################################################################"
 echo "Starting to pull docker images..." 
 echo "Pulling images from repository: " $DOCKER_REGISTRY"/"$DOCKER_REPOSITORY
 
-docker login $DOCKER_REGISTRY -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
 for image in "${IMAGES[@]}";
 do
     docker pull $DOCKER_REGISTRY/$DOCKER_REPOSITORY/$image:$DOCKER_TAG
     echo "Docker image" $image " pulled."
 done
-docker logout $DOCKER_REGISTRY
 echo "Docker images pulled." 
 
 # Deploy azdata bdc create cluster.
 #
 echo ""
 echo "############################################################################"
-echo "Starting to deploy azdata cluster..." 
+echo "Starting to deploy big data cluster..." 
 
 # Command to create cluster for single node cluster.
 #
 azdata bdc config init --source kubeadm-dev-test  --target kubeadm-custom -f
 azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.repository=$DOCKER_REPOSITORY"
 azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.registry=$DOCKER_REGISTRY"
 azdata bdc config replace -c kubeadm-custom/control.json -j ".spec.docker.imageTag=$DOCKER_TAG"
-azdata bdc config replace -c kubeadm-custom/cluster.json -j "$.spec.pools[?(@.spec.type == "Data")].spec.replicas=1"
+azdata bdc config replace -c kubeadm-custom/bdc.json -j "$.spec.resources.data-0.spec.replicas=1"
 azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.data.className=$STORAGE_CLASS"
 azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.logs.className=$STORAGE_CLASS"
 azdata bdc config patch -c kubeadm-custom/control.json -p $STARTUP_PATH/security-patch.json
-azdata bdc config patch -c kubeadm-custom/cluster.json -p $STARTUP_PATH/endpoint-patch.json
+azdata bdc config patch -c kubeadm-custom/bdc.json -p $STARTUP_PATH/endpoint-patch.json
+
 azdata bdc create -c kubeadm-custom --accept-eula $ACCEPT_EULA
-echo "Azdata cluster created." 
+
+echo "Big data cluster created." 
 
 # Setting context to cluster.
 #

samples/features/sql-big-data-cluster/deployment/kubeadm/ubuntu-single-node-vm/setup-bdc.sh

@@ -321,7 +321,7 @@ azdata bdc config replace -c kubeadm-custom/bdc.json -j "$.spec.resources.data-0
 azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.data.className=$STORAGE_CLASS"
 azdata bdc config replace -c kubeadm-custom/control.json -j "spec.storage.logs.className=$STORAGE_CLASS"
 azdata bdc create -c kubeadm-custom --accept-eula $ACCEPT_EULA
-echo "Azdata cluster created." 
+echo "Big data cluster created." 
 
 # Setting context to cluster.
 #

samples/features/sql-clr/Curl/README.md

@@ -9,7 +9,7 @@ One of the most popular tools for calling an API on `http:` endpoints is [curl](
 
 [About this sample](#about-this-sample)<br/>
 [Build the CLR/CURL extension](#build-functions)<br/>
-[Add RegEx functions to your SQL database](#add-functions)<br/>
+[Add CURL functions to your SQL database](#add-functions)<br/>
 [Test the functions](#test)<br/>
 [Disclaimers](#disclaimers)<br/>
 [Appendix](#appendix) - quick install script for your dev environment.<br/>

samples/manage/azure-sql-db-managed-instance/compare-environment-settings/compare-properties.sql

@@ -18,6 +18,11 @@ select property = 'TEMPDB:'+y.v.value('local-name(.)', 'nvarchar(300)'),
 		value = y.v.value('.[1]', 'nvarchar(300)')
 from @source.nodes('//tempdb') x(v)
 cross apply x.v.nodes('*') y(v)
+UNION ALL
+select property = 'INSTANCE:'+y.v.value('local-name(.)', 'nvarchar(300)'),
+		value = y.v.value('.[1]', 'nvarchar(300)')
+from @source.nodes('//instance') x(v)
+cross apply x.v.nodes('*') y(v)
 ),
 tgt as(
 select property = x.v.value('name[1]', 'nvarchar(300)'),
@@ -33,6 +38,11 @@ select property = 'TEMPDB:'+y.v.value('local-name(.)', 'nvarchar(300)'),
 		value = y.v.value('.[1]', 'nvarchar(300)')
 from @target.nodes('//tempdb') x(v)
 cross apply x.v.nodes('*') y(v)
+UNION ALL
+select property = 'INSTANCE:'+y.v.value('local-name(.)', 'nvarchar(300)'),
+		value = y.v.value('.[1]', 'nvarchar(300)')
+from @target.nodes('//instance') x(v)
+cross apply x.v.nodes('*') y(v)
 ),
 diff as (
 select property = isnull(src.property, tgt.property),
@@ -43,7 +53,7 @@ where (src.value <> tgt.value
 or src.value is null and tgt.value is not null
 or src.value is not null and tgt.value is null)
 )
-select *
+select property, source, [target]
 from diff
 where is_missing = 0 or @verbose = 1 -- in the earlier versions you had to comment out this line. Now just set the value of the flag up
 order by property

samples/manage/azure-sql-db-managed-instance/compare-environment-settings/get-properties.sql

@@ -53,6 +53,20 @@ where name in ('cost threshold for parallelism','cursor threshold','fill factor
 for xml raw, elements
 );
 set @result += (select name = 'version', value = @@VERSION for xml raw, elements)
+
+set @result += isnull
+((SELECT scheduler_count, scheduler_total_count FROM sys.dm_os_sys_info
+   for xml raw('instance'), elements),''
+);
+
+set @result += 
+isnull((SELECT name = REPLACE([type], 'MEMORYCLERK_', 'MEMORY:') 
+     , value = CAST(sum(pages_kb)/1024.1/1024 AS NUMERIC(6,1))
+   FROM sys.dm_os_memory_clerks
+   GROUP BY type
+   HAVING sum(pages_kb) /1024. /1024 > 1
+   for xml raw, elements),'');
+
 select cast(@result as xml);
-end;
 
+end;