Skip to content

Commit e31ddb1

Browse files
MikeRayMSFTMikeRayMSFT
committed
Merge branch 'release-sqlseattle' of https://github.com/MicrosoftDocs/sql-docs-pr into ctp20-whatsnew
2 parents 770cdef + 1befd6e commit e31ddb1

5 files changed

Lines changed: 296 additions & 10 deletions

File tree

‎docs/big-data-cluster/concept-security.md‎

Lines changed: 83 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
22
title: Security concepts for SQL Server Big Data Cluster | Microsoft Docs
33
description:
4-
author: rothja
5-
ms.author: jroth
4+
author: nelgson
5+
ms.author: ms.author:negust
66
manager: craigg
77
ms.date: 08/06/2018
88
ms.topic: conceptual
@@ -11,7 +11,87 @@ ms.prod: sql
1111

1212
# Security concepts for SQL Server Big Data Cluster
1313

14-
TBD
14+
A secure Big Data cluster implies consistent and coherent support for authentication and authorization scenarios, across both SQL Server and HDFS/Spark. Authentication is the process of verifying the identity of a user or service and ensuring they are who they are claiming to be. Authorization refers to granting or denying of access to specific resources based on the requesting user's identity. This step is performed after a user is identified through authentication.
15+
16+
Authorization in Big Data context is usually performed through access control lists (ACLs), which associate user identities with specific permissions. HDFS supports authorization by limiting access to service APIs, HDFS files and job execution.
17+
18+
This topic will cover the key security related concepts in the Big Data cluster.
19+
20+
## Cluster Endpoints
21+
22+
There are three entry points to the Big Data Cluster
23+
24+
* HDFS/Spark (Knox) gateway – This is an HTTPS-based endpoint. Other endpoints are proxied through this. HDFS/Spark gateway is used for accessing services like webHDFS and Livy. Wherever you see references to Knox, this is the endpoint.
25+
26+
* Controller endpoint – Big Data Cluster management service that exposes REST APIs for managing the cluster. Some tools like the Admin portal is also accessed through this endpoint.
27+
28+
* Master Instance - TDS endpoint for database tools and applications to connect to SQL Server Master Instance in the cluster.
29+
30+
Currently, there is no option of opening up additional ports for accessing the cluster from the outside.
31+
32+
### How endpoints are secured
33+
34+
Securing endpoints in the Big Data Cluster is done using passwords that can be set/updated either using environment variables or CLI commands. All cluster internal passwords are stored as Kubernetes secrets.
35+
36+
# Authentication
37+
38+
Upon provisioning the cluster, a number of logins are created.
39+
Some of this logins are for services to communicate with each other, and others are for end-users to access the cluster.
40+
41+
## End-user authentication
42+
Upon provisioning the cluster, a set of end-user passwords need to be set using environment variables. These are passwords that SQL administrators and cluster administrators use to access services:
43+
44+
Controller username:
45+
+ CONTROLLER_USERNAME=<controller_username>
46+
47+
Controller password:
48+
+ CONTROLLER_PASSWORD=<controller_password>
49+
50+
SQL Master SA password:
51+
+ MSSQL_SA_PASSWORD=<controller_sa_password>
52+
53+
Username for accessing the HDFS/Spark endpoint:
54+
+ KNOX_USERNAME=<know_username>
55+
56+
Password for accessing the HDFS/Spark endpoint:
57+
+ KNOX_PASSWORD=<knox_password>
58+
59+
### Update HDFS/Spark (Knox) and Controller passwords
60+
In order to update Knox and controller admin passwords, the following command can be used:
61+
62+
```bash
63+
mssqlctl set-password service [-h] [-v] servicename clustername
64+
```
65+
66+
Service name indicated which service to update the password for. This can be either "knox" or "controller"
67+
68+
69+
Optional arguments:
70+
* -h, --help show this help message and exit
71+
* -v, --verbose Increase output verbosity
72+
73+
For example, to update the Knox password for a cluster called "testcluster", run the following:
74+
75+
```bash
76+
export KNOX_PASSWORD = <password>
77+
mssqlctl set-password service knox testcluster
78+
```
79+
80+
It is currently not possible to change the Knox user name.
81+
82+
## Intra cluster authentication
83+
84+
Upon deployment of the cluster, a number of SQL logins are created:
85+
86+
* A special SQL login is created in the Controller SQL instance that is system managed, with sysadmin role. The password for this login is captured as a K8s secret.
87+
88+
* A sysadmin login is created in all SQL instances in the cluster, that Controller owns and manages. It is required for Controller to perform administrative tasks on these instances (i.e. HA setup, upgrade etc.). These logins are also used for intra cluster communication between SQL instances, i.e. Master instance communicating with Data Pool.
89+
90+
Please note that in CTP2.0, only basic authentication is supported. Fine-grained access control to HDFS objects, and SQL Big Data Cluster compute and data pools, is not yet available.
91+
92+
## Intra cluster communication
93+
94+
Communication with non-SQL services within the Big Data Cluster, like for example Livy to Spark or Spark to Storage Pool, is secured using certificates. All SQL Sever to SQL Server communication between is secured using SQL logins.
1595

1696
## Next steps
1797

‎docs/linux/sql-server-linux-configure-msdtc.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
---
2+
title: How to configure MSDTC on Linux | Microsoft Docs
3+
description: This article provides a walk-through for configuring MSDTC on Linux.
4+
author: rothja
5+
ms.author: jroth
6+
manager: craigg
7+
ms.date: 09/24/2018
8+
ms.topic: conceptual
9+
ms.prod: sql
10+
ms.component: ""
11+
ms.suite: "sql"
12+
ms.custom: "sql-linux"
13+
ms.technology: linux
14+
monikerRange: ">= sql-server-ver15 || = sqlallproducts-allversions"
15+
---
16+
# How to configure the Microsoft Distributed Transaction Coordinator (MSDTC) on Linux
17+
18+
[!INCLUDE[appliesto-ss-xxxx-xxxx-xxx-md-linuxonly](../includes/appliesto-ss-xxxx-xxxx-xxx-md-linuxonly.md)]
19+
20+
This article describes how to configure the Microsoft Distributed Transaction Coordinator (MSTDC) on Linux. MSDTC support on Linux was introduced in SQL Server 2019 CTP 2.0.
21+
22+
## Supported MSDTC configurations
23+
24+
The following MSDTC configurations are supported:
25+
26+
- OLE-TX Distributed transactions against SQL Server on Linux for both ODBC/JDBC providers.
27+
- XA Distributed transactions against SQL Server on Linux using both ODBC/JDBC provider (requires the use of the latest ODBC provider).
28+
- Distributed transactions on Linked server.
29+
30+
For limitations and known issues for MSDTC in CTP 2.0, see [Release notes for SQL Server 2019 CTP on Linux](sql-server-linux-release-notes-2019.md#msdtc).
31+
32+
## MSDTC configuration overview
33+
34+
Distributed transactions are enabled on SQL Server on Linux by introducing MSDTC and RPC endpoint mapper functionality within SQL Server. By default, an RPC endpoint mapping process listens on port 135 for incoming RPC requests and routes that to appropriate components (such as the MSDTC service). A process requires super user privilege to bind to system ports (port numbers less than 1024) on Linux. To avoid starting SQL Server with root privileges for the RPC endpoint mapper process, system administrators must use iptables to create NAT translation to route traffic on port 135 to SQL Server's RPC endpoint mapping process.
35+
36+
SQL Server 2019 introduces two configuration parameters for the mssql-conf utility.
37+
38+
| mssql-conf setting | Description |
39+
|---|---|
40+
| **network.rpcport** | The RPC port that the RPC endpoint manager process binds to. |
41+
| **network.servertcpport** | The port that the MSDTC server listens to. |
42+
43+
For more information about these settings and other related MSDTC settings, see [Configure SQL Server on Linux with the mssql-conf tool](sql-server-linux-configure-mssql-conf.md#msdtc).
44+
45+
There are three steps to configure MSDTC communication and functionality.
46+
47+
- Configure **network.rpcport** and **distributedtransaction.servertcpport** using mssql-conf.
48+
- Configure Linux server routing so that RPC communication on port 135 is redirected to SQL Server's **network.rpcport**.
49+
- Configure the firewall to allow communication on both **rpcport** and **servertcpport** so the RPC endpoint mapping process and MSDTC process can communicate externally to other transaction managers and coordinators.
50+
51+
> [!IMPORTANT]
52+
> If the neccessary configuration steps are not done, SQL Server will not enable MSDTC functionality.
53+
54+
The following sections provide detailed instructions for each step.
55+
56+
## Configure RPC and MSDTC ports
57+
58+
First, configure **network.rpcport** and **distributedtransaction.servertcpport** using mssql-conf.
59+
60+
1. Use mssql-conf to set the **network.rpcport** value. The following example sets it to 13500.
61+
62+
```bash
63+
sudo /opt/mssql/bin/mssql-conf set network.rpcport 13500
64+
```
65+
66+
1. Set the **distributedtransaction.servertcpport** value. The following example sets it to 51999.
67+
68+
```bash
69+
sudo /opt/mssql/bin/mssql-conf set distributedtransaction.servertcpport 51999
70+
```
71+
72+
1. Restart SQL Server.
73+
74+
```bash
75+
sudo systemctl restart mssql-server
76+
```
77+
78+
## Configure port routing
79+
80+
Configure the Linux server routing table so that RPC communication on port 135 is redirected to SQL Server's **network.rpcport**. The iptable rules may not persist during reboots, so the following commands also provide instructions for restoring the rules after a reboot.
81+
82+
1. Create routing rules for port 135. In the following example, port 135 is directed to the RPC port, 13500, defined in the previous section. Replace `<ipaddress>` with the IP address of your server.
83+
84+
```bash
85+
iptables -t nat -A PREROUTING -d <ip> -p tcp --dport 135 -m addrtype --dst-type LOCAL -j DNAT --to-destination <ip>:13500
86+
87+
iptables -t nat -A OUTPUT -d <ip> -p tcp --dport 135 -m addrtype --dst-type LOCAL -j DNAT --to-destination <ip>:13500
88+
```
89+
90+
1. View the routing rules you created with the following command:
91+
92+
```bash
93+
sudo iptables -t nat -L
94+
```
95+
96+
1. Save the routing rules to a file on your machine.
97+
98+
```bash
99+
iptables-save > /etc/iptables.conf
100+
```
101+
102+
1. Add the following command to `/etc/rc.local` to reload the rules after a reboot.
103+
104+
```bash
105+
iptables-restore < /etc/iptables.conf
106+
```
107+
108+
> [!TIP]
109+
> If you need to recreate or delete existing routing rules, you can use the `-D` parameter of **iptable** and specify the name of the routing rule to remove.
110+
111+
## Configure the firewall
112+
113+
The final step is to configure the firewall to allow communication on both **rpcport** and **servertcpport**. The actual steps for this will vary depending on your Linux distribution and firewall. The following example shows how to create these rules on Ubuntu.
114+
115+
```bash
116+
sudo ufw allow from any to any port 13500 proto tcp
117+
sudo ufw allow from any to any port 51999 proto tcp
118+
```
119+
120+
The following example shows how this could be done on Red Hat Enterprise Linux:
121+
122+
```bash
123+
sudo firewall-cmd --zone=public --add-port=51999/tcp --permanent
124+
sudo firewall-cmd --zone=public --add-port=13500/tcp --permanent
125+
sudo firewall-cmd --reload
126+
```
127+
128+
## Verify
129+
130+
At this point, SQL Server should be able to participate in distributed transactions. To verify that SQL Server is listening, run the following command:
131+
132+
```bash
133+
sudo netstat -tulpn | grep sqlservr
134+
```
135+
136+
You should see output similar to the following:
137+
138+
```bash
139+
tcp 0 0 0.0.0.0:1433 0.0.0.0:* LISTEN 13911/sqlservr
140+
tcp 0 0 127.0.0.1:1434 0.0.0.0:* LISTEN 13911/sqlservr
141+
tcp 0 0 0.0.0.0:13500 0.0.0.0:* LISTEN 13911/sqlservr
142+
tcp 0 0 0.0.0.0:51999 0.0.0.0:* LISTEN 13911/sqlservr
143+
tcp6 0 0 :::1433 :::* LISTEN 13911/sqlservr
144+
tcp6 0 0 ::1:1434 :::* LISTEN 13911/sqlservr
145+
tcp6 0 0 :::13500 :::* LISTEN 13911/sqlservr
146+
tcp6 0 0 :::51999 :::* LISTEN 13911/sqlservr
147+
```
148+
149+
However, after a restart, SQL Server does not start listening on the **servertcpport** until the first distributed transaction. In this case you would not see SQL Server listening on port 51999 in this example until the first distributed transaction.
150+
151+
## Next steps
152+
153+
For more information about SQL Server on Linux, see [SQL Server on Linux](sql-server-linux-overview.md).

‎docs/linux/sql-server-linux-configure-mssql-conf.md‎

Lines changed: 48 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ ms.assetid: 06798dff-65c7-43e0-9ab3-ffb23374b322
2424
| [Agent](#agent) | Enable SQL Server Agent |
2525
| [Collation](#collation) | Set a new collation for SQL Server on Linux. |
2626
| [Customer feedback](#customerfeedback) | Choose whether or not SQL Server sends feedback to Microsoft. |
27-
| [Database Mail Profile](#dbmail) | Set the default database mail profile for SQL Server on Linux |
27+
| [Database Mail Profile](#dbmail) | Set the default database mail profile for SQL Server on Linux. |
2828
| [Default data directory](#datadir) | Change the default directory for new SQL Server database data files (.mdf). |
2929
| [Default log directory](#datadir) | Changes the default directory for new SQL Server database log (.ldf) files. |
3030
| [Default master database file directory](#masterdatabasedir) | Changes the default directory for the master database files on existing SQL installation.|
@@ -37,6 +37,7 @@ ms.assetid: 06798dff-65c7-43e0-9ab3-ffb23374b322
3737
| [Local Audit directory](#localaudit) | Set a a directory to add Local Audit files. |
3838
| [Locale](#lcid) | Set the locale for SQL Server to use. |
3939
| [Memory limit](#memorylimit) | Set the memory limit for SQL Server. |
40+
| [Microsoft Distributed Transaction Coordinator](#msdtc) | Configure and troubleshoot MSDTC on Linux. |
4041
| [TCP port](#tcpport) | Change the port where SQL Server listens for connections. |
4142
| [TLS](#tls) | Configure Transport Level Security. |
4243
| [Traceflags](#traceflags) | Set the traceflags that the service is going to use. |
@@ -441,6 +442,46 @@ The **memory.memorylimitmb** setting controls the amount physical memory (in MB)
441442
sudo systemctl restart mssql-server
442443
```
443444

445+
## <a id="msdtc"></a> Configure MSDTC
446+
447+
The **network.rpcport** and **distributedtransaction.servertcpport** settings are used to configure the Microsoft Distributed Transaction Coordinator (MSDTC). To change these settings, run the following commands:
448+
449+
1. Run the mssql-conf script as root with the **set** command for "network.rpcport":
450+
451+
```bash
452+
sudo /opt/mssql/bin/mssql-conf set network.rpcport <rcp_port>
453+
```
454+
455+
2. Then set the "distributedtransaction.servertcpport" setting:
456+
457+
```bash
458+
sudo /opt/mssql/bin/mssql-conf set distributedtransaction.servertcpport <servertcpport_port>
459+
```
460+
461+
In addition to setting these values, you must also configure routing and update the firewall for port 135. For more information on how to do this, see [How to configure MSDTC on Linux](sql-server-linux-configure-msdtc.md).
462+
463+
There are several other settings for mssql-conf that you can use to monitor and troubleshoot MSDTC. The following table briefly describes these settings. For more information on their use, see the details in the Windows support article, [How to enable diagnostic tracing for MS DTC](https://support.microsoft.com/en-us/help/926099/how-to-enable-diagnostic-tracing-for-ms-dtc-on-a-windows-based-compute).
464+
465+
| mssql-conf setting | Description |
466+
|---|---|
467+
| distributedtransaction.allowonlysecurerpccalls | Configure secure only rpc calls for distributed transactions |
468+
| distributedtransaction.fallbacktounsecurerpcifnecessary | Configure security only rpc calls for distributed |transactions
469+
| distributedtransaction.maxlogsize | DTC transaction log file size in MB. Default is 64MB |
470+
| distributedtransaction.memorybuffersize | Circular buffer size in which traces are stored. This size is in MB and default is 10MB |
471+
| distributedtransaction.servertcpport | MSDTC rpc server port |
472+
| distributedtransaction.trace_cm | Traces in the connection manager |
473+
| distributedtransaction.trace_contact | Traces the contact pool and contacts |
474+
| distributedtransaction.trace_gateway | Traces Gateway source |
475+
| distributedtransaction.trace_log | Log tracing |
476+
| distributedtransaction.trace_misc | Traces that cannot be categorized into the other categories |
477+
| distributedtransaction.trace_proxy | Traces that are generated in the MSDTC proxy |
478+
| distributedtransaction.trace_svc | Traces service and .exe file startup |
479+
| distributedtransaction.trace_trace | The trace infrastructure itself |
480+
| distributedtransaction.trace_util | Traces utility routines that are called from multiple locations |
481+
| distributedtransaction.trace_xa | XA Transaction Manager (XATM) tracing source |
482+
| distributedtransaction.tracefilepath | Folder in which trace files should be stored |
483+
| distributedtransaction.turnoffrpcsecurity | Enable or disable RPC security for distributed transactions |
484+
444485
## <a id="tcpport"></a> Change the TCP port
445486

446487
The **network.tcpport** setting changes the TCP port where SQL Server listens for connections. By default, this port is set to 1433. To change the port, run the following commands:
@@ -451,13 +492,13 @@ The **network.tcpport** setting changes the TCP port where SQL Server listens fo
451492
sudo /opt/mssql/bin/mssql-conf set network.tcpport <new_tcp_port>
452493
```
453494

454-
1. Restart the SQL Server service:
495+
2. Restart the SQL Server service:
455496

456497
```bash
457498
sudo systemctl restart mssql-server
458499
```
459500

460-
1. When connecting to SQL Server now, you must specify the custom port with a comma (,) after the hostname or IP address. For example, to connect with SQLCMD, you would use the following command:
501+
3. When connecting to SQL Server now, you must specify the custom port with a comma (,) after the hostname or IP address. For example, to connect with SQLCMD, you would use the following command:
461502

462503
```bash
463504
sqlcmd -S localhost,<new_tcp_port> -U test -P test
@@ -544,6 +585,9 @@ accepteula = Y
544585
captureminiandfull = true
545586
coredumptype = full
546587
588+
[distributedtransaction]
589+
servertcpport = 51999
590+
547591
[filelocation]
548592
defaultbackupdir = /var/opt/mssql/data/
549593
defaultdatadir = /var/opt/mssql/data/
@@ -563,6 +607,7 @@ memorylimitmb = 4096
563607
forceencryption = 0
564608
ipaddress = 10.192.0.0
565609
kerberoskeytabfile = /var/opt/mssql/secrets/mssql.keytab
610+
rpcport = 13500
566611
tcpport = 1401
567612
tlscert = /etc/ssl/certs/mssql.pem
568613
tlsciphers = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA

0 commit comments

Comments
 (0)