Skip to content

Commit 41ec742

Browse files
shivsoodshivsood
committed
Merge remote-tracking branch 'upstream/master'
2 parents b2ef5f6 + 6659e27 commit 41ec742

28 files changed

Lines changed: 784 additions & 606 deletions

‎samples/features/Azure Data Studio/Notebooks/Install cluster dependencies.ipynb‎

Lines changed: 316 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,316 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"Install Dependencies to deploy SQL Server 2019 Big Data Clusters\n",
8+
"----------------------------------------------------------------\n",
9+
"\n",
10+
"The following tools are most important for managing, connecting to, and\n",
11+
"querying the cluster:\n",
12+
"\n",
13+
"- **Azure CLI**\n",
14+
"- **kubectl**\n",
15+
"- **mssqlctl**\n",
16+
"\n",
17+
"**The cell below tries to install these dependencies for you. If you\n",
18+
"still experience any issues in executing this cell then please install\n",
19+
"from the following Installation links below.**\n",
20+
"\n",
21+
"<table>\n",
22+
"<colgroup>\n",
23+
"<col style=\"width: 27%\" />\n",
24+
"<col style=\"width: 24%\" />\n",
25+
"<col style=\"width: 24%\" />\n",
26+
"<col style=\"width: 24%\" />\n",
27+
"</colgroup>\n",
28+
"<thead>\n",
29+
"<tr class=\"header\">\n",
30+
"<th>Tool</th>\n",
31+
"<th>Required</th>\n",
32+
"<th>Description</th>\n",
33+
"<th>Installation</th>\n",
34+
"</tr>\n",
35+
"</thead>\n",
36+
"<tbody>\n",
37+
"<tr class=\"odd\">\n",
38+
"<td><strong>mssqlctl</strong></td>\n",
39+
"<td>Yes</td>\n",
40+
"<td>Command-line tool for installing and managing a big data cluster.</td>\n",
41+
"<td><a href=\"deploy-install-mssqlctl.md\">Install</a></td>\n",
42+
"</tr>\n",
43+
"<tr class=\"even\">\n",
44+
"<td><strong>kubectl</strong></td>\n",
45+
"<td>Yes</td>\n",
46+
"<td>Command-line tool for monitoring the underlying Kuberentes cluster (<a href=\"https://kubernetes.io/docs/tasks/tools/install-kubectl/\">More info</a>).</td>\n",
47+
"<td><a href=\"https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-powershell-from-psgallery\">Windows</a> | <a href=\"https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-using-native-package-management\">Linux</a></td>\n",
48+
"</tr>\n",
49+
"<tr class=\"odd\">\n",
50+
"<td><strong>Azure CLI</strong></td>\n",
51+
"<td>For AKS</td>\n",
52+
"<td>Modern command-line interface for managing Azure services. Used with AKS big data cluster deployments (<a href=\"https://docs.microsoft.com/cli/azure/?view=azure-cli-latest\">More info</a>).</td>\n",
53+
"<td><a href=\"https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest\">Install</a></td>\n",
54+
"</tr>\n",
55+
"</tbody>\n",
56+
"</table>\n",
57+
"\n",
58+
"Steps\n",
59+
"-----\n",
60+
"\n",
61+
"### Provide overrides for any default installation parameters\n",
62+
"\n",
63+
"You don’t need to provide any installation parameters, we’ll provide a\n",
64+
"set of default which will provide a good experience.\n",
65+
"\n",
66+
"However, feel free to override any of the defaults provided below here:"
67+
]
68+
},
69+
{
70+
"cell_type": "code",
71+
"execution_count": null,
72+
"metadata": {},
73+
"outputs": [],
74+
"source": [
75+
"mssql_version=\"\"\n",
76+
"docker_registry=\"\"\n",
77+
"\n",
78+
"windows_azcli_url=\"\"\n",
79+
"windows_kubectl_url=\"\"\n",
80+
"\n",
81+
"azure_cli_use_force_install_option = None\n",
82+
"azure_cli_use_user_install_option = None\n",
83+
"\n",
84+
"mssqlctl_url=\"\" # url to download mssqlctl from (without ending filename)\n",
85+
"mssqlctl_use_force_install_option = None\n",
86+
"mssqlctl_use_user_install_option = None\n",
87+
"\n",
88+
"skip_mssqlctl_uninstall = None\n",
89+
"skip_mssqlctl_install = None"
90+
]
91+
},
92+
{
93+
"cell_type": "markdown",
94+
"metadata": {},
95+
"source": [
96+
"### Provide default installation parameters\n",
97+
"\n",
98+
"A default set of parameters to provide a good initial experience of SQL\n",
99+
"Server 2019 big data clusters."
100+
]
101+
},
102+
{
103+
"cell_type": "code",
104+
"execution_count": null,
105+
"metadata": {},
106+
"outputs": [],
107+
"source": [
108+
"if mssql_version==\"\":\n",
109+
" mssql_version=\"ctp-2.5\"\n",
110+
"\n",
111+
"if windows_azcli_url==\"\":\n",
112+
" windows_azcli_url=\"https://aka.ms/installazurecliwindows\"\n",
113+
"\n",
114+
"if windows_kubectl_url==\"\":\n",
115+
" windows_kubectl_url=\"https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/windows/amd64/kubectl.exe\"\n",
116+
"\n",
117+
"if docker_repository==\"\":\n",
118+
" docker_repository=\"aris-p-release-candidate-gb\"\n",
119+
"\n",
120+
"if mssqlctl_url==\"\":\n",
121+
" mssqlctl_url=\"\"http://helsinki/browse/packages/python/{0}/mssqlctl/{1}\".format(docker_repository, 'requirements.txt')\n",
122+
"\n",
123+
"if azure_cli_use_force_install_option is None:\n",
124+
" azure_cli_use_force_install_option = False\n",
125+
"\n",
126+
"if azure_cli_use_user_install_option is None:\n",
127+
" azure_cli_use_user_install_option = True\n",
128+
"\n",
129+
"if mssqlctl_use_force_install_option is None:\n",
130+
" mssqlctl_use_force_install_option = False\n",
131+
"\n",
132+
"if mssqlctl_use_user_install_option is None:\n",
133+
" mssqlctl_use_user_install_option = True\n",
134+
"\n",
135+
"if skip_mssqlctl_uninstall is None:\n",
136+
" skip_mssqlctl_uninstall = True\n",
137+
"\n",
138+
"if skip_mssqlctl_install is None:\n",
139+
" skip_mssqlctl_install = False\n",
140+
"\n",
141+
"print('mssql_version = ' + mssql_version)\n",
142+
"print('docker_registry = ' + docker_registry)\n",
143+
"print('azure_cli_use_force_install_option = ' + str(azure_cli_use_force_install_option))\n",
144+
"print('azure_cli_use_user_install_option = ' + str(azure_cli_use_user_install_option))\n",
145+
"print('mssqlctl_url = ' + mssqlctl_url)\n",
146+
"print('mssqlctl_use_force_install_option = ' + str(mssqlctl_use_force_install_option))\n",
147+
"print('mssqlctl_use_user_install_option = ' + str(mssqlctl_use_user_install_option))\n",
148+
"print('skip_mssqlctl_uninstall = ' + str(skip_mssqlctl_uninstall))\n",
149+
"print('skip_mssqlctl_install = ' + str(skip_mssqlctl_install))"
150+
]
151+
},
152+
{
153+
"cell_type": "markdown",
154+
"metadata": {},
155+
"source": [
156+
"### Install Azure CLI"
157+
]
158+
},
159+
{
160+
"cell_type": "code",
161+
"execution_count": null,
162+
"metadata": {},
163+
"outputs": [],
164+
"source": [
165+
"import sys\n",
166+
"\n",
167+
"force_install_option='--force-install' if azure_cli_use_force_install_option else ''\n",
168+
"user_install_option='--user' if azure_cli_use_user_install_option else ''\n",
169+
"\n",
170+
"print(f'START: !{sys.executable} -m pip install azure-cli {force_install_option} {user_install_option}\\n')\n",
171+
"\n",
172+
"!{sys.executable} -m pip install azure-cli {force_install_option} {user_install_option}\n",
173+
"if _exit_code != 0:\n",
174+
" raise SystemExit('Shell command:\\n\\n\\t!{sys.executable} -m pip install azure-cli {force_install_option} {user_install_option}\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
175+
"\n",
176+
"print(f'\\nSUCCESS: !{sys.executable} -m pip install azure-cli {force_install_option} {user_install_option}')"
177+
]
178+
},
179+
{
180+
"cell_type": "markdown",
181+
"metadata": {},
182+
"source": [
183+
"### Install Kubernetes CLI"
184+
]
185+
},
186+
{
187+
"cell_type": "code",
188+
"execution_count": null,
189+
"metadata": {},
190+
"outputs": [],
191+
"source": [
192+
"import platform\n",
193+
"import requests\n",
194+
"import sys\n",
195+
"\n",
196+
"if platform.system()==\"Darwin\":\n",
197+
" print(f'START: !brew update\\n')\n",
198+
"\n",
199+
" !brew update\n",
200+
" if _exit_code != 0:\n",
201+
" raise SystemExit('Shell command:\\n\\n\\t !brew update\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
202+
"\n",
203+
" print(f'\\nSUCCESS: !brew update')\n",
204+
" print(f'START: !brew install kubernetes-cli\\n')\n",
205+
"\n",
206+
" !brew install kubernetes-cli\n",
207+
" if _exit_code != 0:\n",
208+
" raise SystemExit('Shell command:\\n\\n\\t !brew install kubernetes-cli\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
209+
"\n",
210+
" print(f'\\nSUCCESS: !brew install kubernetes-cli')\n",
211+
"elif platform.system()==\"Windows\":\n",
212+
" print(f'START: !curl -LO {windows_kubectl_url}\\n')\n",
213+
"\n",
214+
" !curl -LO {windows_kubectl_url}\n",
215+
" if _exit_code != 0:\n",
216+
" raise SystemExit('Shell command:\\n\\n\\t !curl -LO {windows_kubectl_url}\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
217+
"\n",
218+
" print(f'\\nSUCCESS: !curl -LO {windows_kubectl_url}')\n",
219+
"elif platform.system()==\"Linux\":\n",
220+
" print(f'START: !sudo apt-get update\\n')\n",
221+
"\n",
222+
" !sudo apt-get update\n",
223+
" if _exit_code != 0:\n",
224+
" raise SystemExit('Shell command:\\n\\n\\t !sudo apt-get update\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
225+
"\n",
226+
" print(f'\\nSUCCESS: !sudo apt-get update')\n",
227+
" print(f'START: !sudo apt-get install -y kubectl\\n')\n",
228+
"\n",
229+
" !sudo apt-get install -y kubectl\n",
230+
" if _exit_code != 0:\n",
231+
" raise SystemExit('Shell command:\\n\\n\\t !sudo apt-get install -y kubectl\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
232+
"\n",
233+
" print(f'\\nSUCCESS: !sudo apt-get install -y kubectl')\n",
234+
"else:\n",
235+
" raise SystemExit(\"Platform '{0}' is not recognized, must be 'Darwin', 'Windows' or 'Linux'\".format(platform.system()))"
236+
]
237+
},
238+
{
239+
"cell_type": "markdown",
240+
"metadata": {},
241+
"source": [
242+
"### Uninstall MSSQLCTL CLI"
243+
]
244+
},
245+
{
246+
"cell_type": "code",
247+
"execution_count": null,
248+
"metadata": {},
249+
"outputs": [],
250+
"source": [
251+
"import sys\n",
252+
"\n",
253+
"if not skip_mssqlctl_uninstall:\n",
254+
" print(f'START: !{sys.executable} -m pip uninstall -r {mssqlctl_url} --yes --trusted-host helsinki\\n')\n",
255+
"\n",
256+
" !{sys.executable} -m pip uninstall -r {mssqlctl_url} --yes --trusted-host helsinki\n",
257+
" if _exit_code != 0:\n",
258+
" raise SystemExit('Shell command:\\n\\n\\t !{sys.executable} -m pip uninstall -r {mssqlctl_url} --yes --trusted-host helsinki\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
259+
"\n",
260+
" print(f'\\nSUCCESS: !{sys.executable} -m pip uninstall -r {mssqlctl_url} --yes --trusted-host helsinki')"
261+
]
262+
},
263+
{
264+
"cell_type": "markdown",
265+
"metadata": {},
266+
"source": [
267+
"### Install MSSQLCTL CLI"
268+
]
269+
},
270+
{
271+
"cell_type": "code",
272+
"execution_count": null,
273+
"metadata": {},
274+
"outputs": [],
275+
"source": [
276+
"import sys\n",
277+
"\n",
278+
"force_install_option='--force-install' if mssqlctl_use_force_install_option else ''\n",
279+
"user_install_option='--user' if mssqlctl_use_user_install_option else ''\n",
280+
"\n",
281+
"if not skip_mssqlctl_install:\n",
282+
" print(f'START: !{sys.executable} -m pip install -r {mssqlctl_url} {force_install_option} {user_install_option} --trusted-host helsinki\\n')\n",
283+
"\n",
284+
" !{sys.executable} -m pip install -r {mssqlctl_url} {force_install_option} {user_install_option} --trusted-host helsinki\n",
285+
" if _exit_code != 0:\n",
286+
" raise SystemExit('Shell command:\\n\\n\\t !{sys.executable} -m pip install -r {mssqlctl_url} {force_install_option} {user_install_option} --trusted-host helsinki\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\n",
287+
"\n",
288+
" print(f'\\nSUCCESS: !{sys.executable} -m pip install -r {mssqlctl_url} {force_install_option} {user_install_option} --trusted-host helsinki')"
289+
]
290+
}
291+
],
292+
"nbformat": 4,
293+
"nbformat_minor": 5,
294+
"metadata": {
295+
"kernelspec": {
296+
"name": "python3",
297+
"display_name": "Python 3"
298+
},
299+
"azdata": {
300+
"test": {
301+
"strategy": "Sequential",
302+
"dri": false,
303+
"ci": false,
304+
"gci": false
305+
},
306+
"publish": {
307+
"access": {
308+
"current": "Internal",
309+
"goal": "Public"
310+
},
311+
"state": "Draft"
312+
}
313+
}
314+
}
315+
}
316+

‎samples/features/sql-big-data-cluster/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,12 @@ Installation instructions for SQL Server 2019 big data clusters can be found [he
99
## Executing the sample scripts
1010
The scripts should be executed in a specific order to test the various features. Execute the scripts from each folder in below order:
1111

12-
1. __[spark](spark/)__
12+
1. __[spark/dataloading/transform-csv-files.sql](spark/dataloading/transform-csv-files.sql)__
1313
1. __[data-virtualization/storage-pool](data-virtualization/storage-pool)__
1414
1. __[data-virtualization/oracle](data-virtualization/oracle)__
1515
1. __[data-pool](data-pool/)__
16-
1. __[machine-learning/r](machine-learning/r)__
17-
1. __[machine-learning/python](machine-learning/python)__
16+
1. __[machine-learning/sql/r](machine-learning/sql/r)__
17+
1. __[machine-learning/sql/python](machine-learning/sql/python)__
1818

1919
## __[data-pool](data-pool/)__
2020

‎samples/features/sql-big-data-cluster/app-deploy/RollDice/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,10 @@ To run this sample, you need the following prerequisites.
4141
## Run this sample
4242

4343
1. Clone or download this sample on your computer.
44-
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
44+
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
4545

4646
```bash
47-
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
47+
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
4848
```
4949
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `roll-dice.R` files are located:
5050
```bash

‎samples/features/sql-big-data-cluster/app-deploy/SSIS/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,10 @@ To run this sample, you need the following prerequisites.
3232
## Run this sample
3333

3434
1. Clone or download this sample on your computer.
35-
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
35+
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
3636

3737
```bash
38-
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
38+
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
3939
```
4040
3. Replace `[SA_PASSWORD]` in the `spec.yaml` file with the password for SQL user `sa`.
4141
4. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `back-up-db.dtsx` files are located:

‎samples/features/sql-big-data-cluster/app-deploy/addpy/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,10 @@ To run this sample, you need the following prerequisites.
4242
## Run this sample
4343

4444
1. Clone or download this sample on your computer.
45-
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
45+
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
4646

4747
```bash
48-
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
48+
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
4949
```
5050
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `add.py` files are located:
5151
```bash
@@ -116,7 +116,7 @@ To run this sample, you need the following prerequisites.
116116

117117
> *Optional*: If you want, you can open the URL for the `swagger` that was returned when you ran `mssqlctl app describe --name addpy --version [version]` in your browser. You will have to log in with the same credentials you used for `mssqlctl login`. The contents of the `swagger.json` you can paste into [Swagger Editor](https://editor.swagger.io). You will see that the web service exposes the `run` method.
118118

119-
You can use your favorite tool to call the `run` method (`https://[IP]:[PORT]/api/app/addpy/[version]/run`), passing in the parameters in the body of your POST request as json. In this example we will use [Postman](https://www.getpostman.com/). Before making the call, you will need to set the `Authorization` to `Bearer Token` and paste in the token you retrieved earlier. This will set a header on your request. See the screenshot below.
119+
You can use your favorite tool to call the `run` method (`https://[IP]:30778/api/app/addpy/[version]/run`), passing in the parameters in the body of your POST request as json. In this example we will use [Postman](https://www.getpostman.com/). Before making the call, you will need to set the `Authorization` to `Bearer Token` and paste in the token you retrieved earlier. This will set a header on your request. See the screenshot below.
120120
![Postman Run Headers](postman_run_1.png)
121121
Next, in the requests body, pass in the parameters to the app you are calling and set the `content-type` to `application/json`:
122122
![Postman Run Body](postman_run_2.png)

‎samples/features/sql-big-data-cluster/app-deploy/magic8ball/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,10 @@ To run this sample, you need the following prerequisites.
4141
## Run this sample
4242

4343
1. Clone or download this sample on your computer.
44-
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `endpoint-service-proxy` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
44+
2. Log in to the SQL Server big data cluster using the command below using the IP address of the `mgmtproxy-svc-external` in your cluster. If you are not familiar with `mssqltctl` you can refer to the [documentation](https://docs.microsoft.com/en-us/sql/big-data-cluster/big-data-cluster-create-apps?view=sqlallproducts-allversions) and then return to this sample.
4545

4646
```bash
47-
mssqlctl login -e https://<ip-address-of-endpoint-service-proxy>:30777 -u <user-name> -p <password>
47+
mssqlctl login -e https://<ip-address-of-mgmtproxy-svc-external>:30777 -u <user-name> -p <password>
4848
```
4949
3. Deploy the application by running the following command, specifying the folder where your `spec.yaml` and `magic8ball.py` files are located:
5050
```bash

0 commit comments

Comments
 (0)