Merge pull request #21388 from shohamMSFT/patch-8

Updates on events and timing

Author: ttorble

docs/relational-databases/security/encryption/troubleshoot-tde.md

@@ -12,7 +12,7 @@ ms.prod: sql
 ms.technology: security
 ms.reviewer: vanto
 ms.topic: conceptual
-ms.date: 11/06/2019
+ms.date: 03/04/2022
 monikerRange: "= azuresqldb-current || = azure-sqldw-latest"
 ---
 # Common errors for transparent data encryption with customer-managed keys in Azure Key Vault
@@ -22,13 +22,13 @@ monikerRange: "= azuresqldb-current || = azure-sqldw-latest"
 This article describes how to identify and resolve Azure Key Vault key access issues that caused a database configured to use [transparent data encryption (TDE) with customer-managed keys in Azure Key Vault](/azure/sql-database/transparent-data-encryption-byok-azure-sql) to become inaccessible.
 
 ## Introduction
-When TDE is configured to use a customer-managed key in Azure Key Vault, continuous access to this TDE Protector is required for the database to stay online.  If the logical SQL server loses access to the customer-managed TDE protector in Azure Key Vault, a database will start denying all connections with the appropriate error message and change its state to *Inaccessible* in the Azure portal.
+When TDE is configured to use a customer-managed key in Azure Key Vault, continuous access to this TDE Protector is required for the database to stay online.  If the logical SQL server or managed instance loses access to the customer-managed TDE protector in Azure Key Vault, a database will start denying all connections with the appropriate error message and change its state to *Inaccessible* in the Azure portal.
 
-For the first 8 hours, if the underlying Azure key vault key access issue is resolved, the database will auto-heal and come online automatically. This means that for all intermittent and temporary network outage scenarios, no user action is required, and the database will come online automatically. In most cases, user action is required to resolve the underlying key vault key access issue. 
+For the first 30 minutes, if the underlying Azure key vault key access issue is resolved, the database will auto-heal and come online automatically. This means that for all intermittent and temporary network outage scenarios, no user action is required, and the database will come online automatically. In most cases, user action is required to resolve the underlying key vault key access issue. 
 
 If an inaccessible database is no longer needed, it can be deleted immediately to stop incurring costs. All other actions on the database are not permitted until access to the Azure key vault key has been restored and the database is back online. Changing the TDE option from customer-managed to service-managed keys on the server is also not possible while a database encrypted with customer-managed keys is inaccessible. This is necessary to protect the data from unauthorized access while permissions to the TDE Protector have been revoked. 
 
-After a database has been inaccessible for more than 8 hours, it will no longer auto-heal. If the required Azure key vault key access has been restored after that period, you must re-validate the access to the key manually, to bring the database back online. Bringing the database back online in this case can take a significant amount of time depending on the size of the database. Once the database is back online, previously configured settings such as [failover group](/azure/sql-database/sql-database-auto-failover-group), PITR history, and any tags **will be lost**. Therefore, we recommend implementing a notification system using [Action Groups](/azure/azure-monitor/platform/action-groups) that allows to become aware of and address the underlying key vault key access issues as soon as possible. 
+After a database has been inaccessible for more than 30 minutes, it will no longer auto-heal. If the required Azure key vault key access has been restored after that period, you must re-validate the access to the key manually, to bring the database back online. Bringing the database back online in this case can take a significant amount of time depending on the size of the database. Once the database is back online, previously configured settings such as [failover group](/azure/sql-database/sql-database-auto-failover-group), PITR history, and any tags **will be lost**. Therefore, we recommend implementing a notification system using [Action Groups](/azure/azure-monitor/platform/action-groups) that allows to become aware of and address the underlying key vault key access issues as soon as possible. 
 
 ## Common errors causing databases to become inaccessible
 
@@ -43,10 +43,10 @@ Most issues that occur when you use TDE with Key Vault are caused by one of the
 ### No permissions to access the key vault or the key doesn't exist
 
 - The key was accidentally deleted, disabled or the key expired.
-- The logical SQL Server instance AppId was accidentally deleted.
-- The logical SQL Server instance was moved to a different subscription. A new AppId must be created if the logical server is moved to a different subscription.
-- Permissions granted to the AppId for the keys aren't sufficient (they don't include Get, Wrap, and Unwrap).
-- Permissions for the logical SQL Server instance AppId were revoked.
+- The server's managed identity (system-assigned or user-assigned) was accidentally deleted.
+- The server was moved to a different subscription. A new managed identity (system-assigned or user-assigned) must be assigned to the server when it is moved to a different subscription.
+- Permissions granted to the server's managed identity for the keys aren't sufficient (they don't include Get, Wrap, and Unwrap permissions).
+- Permissions for the server's managed identity were revoked from the key vault.
 
 ## Identify and resolve common errors
 
@@ -56,21 +56,21 @@ In this section, we list troubleshooting steps for the most common errors.
 
 **Error message**
 
-_401 AzureKeyVaultNoServerIdentity - The server identity is not correctly configured on server. Please contact support._
+_401 AzureKeyVaultNoServerIdentity - The server identity is not correctly configured on server. Please contact support.
 
 **Detection**
 
-Use the following cmdlet or command to ensure that an identity has been assigned to the logical SQL Server instance:
+Use the following cmdlet or command to ensure that an identity has been assigned to the server:
 
-- Azure PowerShell: [Get-AzureRMSqlServer](/powershell/module/AzureRM.Sql/Get-AzureRmSqlServer) 
+- Azure PowerShell: [Get-AzSqlServer](/powershell/module/Az.Sql/Get-AzSqlServer) 
 
 - Azure CLI: [az-sql-server-show](/cli/azure/sql/server#az_sql_server_show)
 
 **Mitigation**
 
-Use the following cmdlet or command to configure an Azure AD identity (an AppId) for the logical SQL Server instance:
+Use the following cmdlet or command to configure an Azure AD identity for the server:
 
-- Azure PowerShell: [Set-AzureRmSqlServer](/powershell/module/azurerm.sql/set-azurermsqlserver) with the `-AssignIdentity` option.
+- Azure PowerShell: [Set-AzSqlServer](/powershell/module/az.sql/set-azsqlserver) with the `-AssignIdentity` option.
 
 - Azure CLI: [az sql server update](/cli/azure/sql/server#az_sql_server_update) with the `--assign_identity` option.
 
@@ -82,7 +82,7 @@ In the Azure portal, go to the key vault, and then go to **Access policies**. Co
 To learn more, see [Assign an Azure AD identity to your server](/azure/sql-database/transparent-data-encryption-byok-azure-sql-configure#assign-an-azure-ad-identity-to-your-server).
 
 > [!IMPORTANT]
-> If the logical SQL Server instance was moved to a new tenant after the initial configuration of TDE with Key Vault, repeat the step to configure the Azure AD identity to create a new AppId. Then, add the AppId to the key vault and assign the correct permissions to the key. 
+> If the logical SQL Server or managed instance was moved to a new tenant after the initial configuration of TDE with Key Vault, repeat the step to configure the Azure AD identity to assign a new managed identity (system-assigned or user-assigned) to the server. Then, add the managed identity to the key vault and assign the correct permissions to the key. 
 >
 
 ### Missing key vault
@@ -95,9 +95,9 @@ _503 AzureKeyVaultConnectionFailed - The operation could not be completed on the
 
 To identify the key URI and the key vault:
 
-1. Use the following cmdlet or command to get the key URI of a specific logical SQL Server instance:
+1. Use the following cmdlet or command to get the key URI of a specific server instance:
 
-    - Azure PowerShell: [Get-AzureRmSqlServerKeyVaultKey](/powershell/module/azurerm.sql/get-azurermsqlserverkeyvaultkey)
+    - Azure PowerShell: [Get-AzSqlServerKeyVaultKey](/powershell/module/az.sql/get-azsqlserverkeyvaultkey)
 
     - Azure CLI: [az-sql-server-tde-key-show](/cli/azure/sql/server/tde-key#az_sql_server_tdekey_show) 
 
@@ -111,7 +111,7 @@ To identify the key URI and the key vault:
 
 Confirm that the key vault is available:
 
-- Ensure that the key vault is available and that the logical SQL Server instance has access.
+- Ensure that the key vault is available and that the server has access.
 - If the key vault is behind a firewall, ensure that the check box to allow Microsoft services to access the key vault is selected.
 - If the key vault has been accidentally deleted, you must complete the configuration from the start.
 
@@ -151,86 +151,115 @@ To identify the key URI and key vault:
 
 **Mitigation**
 
-Confirm that the logical SQL Server instance has permissions to the key vault and the correct permissions to access the key:
+Confirm that the server has permissions to the key vault and the correct permissions to access the key:
 
-- In the Azure portal, go to the key vault > **Access policies**. Find the logical SQL Server instance AppId.  
-- If the AppId is present, ensure that the AppID has the following key permissions: Get, Wrap, and Unwrap.
-- If the AppId isn't present, add it by using the **Add New** button. 
+- In the Azure portal, go to the key vault > **Access policies**. Find the server's managed identity (system-assigned or user-assigned).  
+- If the server identity is present, ensure that it has the following key permissions: Get, WrapKey, and UnwrapKey.
+- If the server identity isn't present, add it by using the **Add New** button. 
 
 ## Getting TDE status from the Activity log
 
-To allow for monitoring of the database status due to Azure Key Vault key access issues, the following events will be logged to the [Activity Log](/azure/service-health/alerts-activity-log-service-notifications) for the resource ID based on the Azure Resource Manager URL and Subscription+ResourceGroup+ServerName+DatabaseName: 
+To allow for monitoring of the database status due to Azure Key Vault key access issues, the following events will be logged to the [Activity Log](/azure/service-health/alerts-activity-log-service-notifications) for the resource ID based on the Azure Resource Manager URL.
+
+> [!NOTE]
+> Events may take at least 15-30 mins to appear in the Activity Log from the time key vault access issue occurs. 
 
 **Event when the service loses access to the Azure Key Vault key**
 
-EventName: MakeDatabaseInaccessible 
+**Azure SQL Database**
+
+EventName: MakeDatabaseInaccessible
 
 Status: Started 
 
-Description: Database has lost access to Azure key vault key and is now inaccessible: \<error message\>   
+Description: Database {database_name} on Server {server_name} has lost access to Azure Key Vault Key and is now transitioning to inaccessible state.  
+  
+**Azure SQL Managed Instance**
 
- 
+EventName: MakeManagedDatabaseInaccessible
 
-**Event when the 8-hour wait time for self-healing begins** 
+Status: Started 
+
+Description: Database { database_name} on managed server {server_name} has lost access to Azure Key Vault Key and is now transitioning to inaccessible state.
 
-EventName: MakeDatabaseInaccessible 
 
-Status: InProgress 
+**Event when the issue wasn’t resolved within 30 minutes and Azure Key Vault key access has to be validated manually** 
 
-Description: Database is waiting for Azure key vault key access to be reestablished by user within 8 hours.   
+**Azure SQL Database**
+  
+EventName: MakeDatabaseInaccessible 
 
- 
+Status: Succeeded 
 
-**Event when the database has automatically come back online**
+Description: Database is inaccessible and requires user to resolve Azure key vault errors and reestablish access to Azure key vault key using Re-validate key. 
 
-EventName: MakeDatabaseAccessible 
+**Azure SQL Managed Instance**
+  
+EventName: MakeManagedDatabaseInaccessible 
 
 Status: Succeeded 
 
-Description: Database access to Azure key vault key has been reestablished and database is now online. 
-
- 
+Description: Database { database_name} on managed server {server_name} is inaccessible and requires user to re-establish access to Azure Key Vault Key.
 
-**Event when the issue wasn’t resolved within 8 hours and Azure Key Vault key access has to be validated manually** 
 
-EventName: MakeDatabaseInaccessible 
+**Event when re-validation of Azure Key Vault key access has succeeded and the db is coming back online**
 
-Status: Succeeded 
+**Azure SQL Database**
+  
+EventName: MakeDatabaseAccessible 
 
-Description: Database is inaccessible and requires user to resolve Azure key vault errors and reestablish access to Azure key vault key using Re-validate key. 
+Status: Started 
 
- 
+Description: Access to Azure Key Vault Key has been re-established, operation to make database {database_name} on server {server_name} accessible started. 
 
-**Event when db comes online after manual key re-validation**
+**Azure SQL Managed Instance**
+  
+EventName: MakeManagedDatabaseAccessible 
 
-EventName: MakeDatabaseAccessible 
+Status: Started 
 
-Status: Succeeded 
+Description: Access to Azure Key Vault Key has been re-established, operation to make database {database_name} on managed server {server_name} accessible started.
 
-Description: Database access to Azure key vault key has been reestablished and database is now online. 
+  
+**Event when the database has successfully come back online**
 
- 
+**Azure SQL Database**
+  
+EventName: MakeDatabaseAccessible
 
-**Event when re-validation of Azure Key Vault key access has succeeded and the db is coming back online**
+Status: Succeeded 
 
-EventName: MakeDatabaseAccessible 
+Description: Access to Azure Key Vault Key has been re-established and database {database_name} on server {server_name} is now online.
 
-Status: Started 
+**Azure SQL Managed Instance**
+  
+EventName: MakeManagedDatabaseAccessible
 
-Description: Restoring database access to Azure key vault key has started. 
+Status: Succeeded 
 
- 
+Description: Access to Azure Key Vault Key has been re-established and database {database_name} on managed server {server_name} is now online.  
+  
 
 **Event when re-validation of Azure Key Vault key access has failed**
 
+**Azure SQL Database**
+  
 EventName: MakeDatabaseAccessible 
 
 Status: Failed 
 
-Description: Restoring database access to Azure key vault key has failed. 
+Description: Access to Azure Key Vault Key has been re-established, operation to make database {database_name} accessible on server {server_name} failed
+
+**Azure SQL Managed Instance**
+  
+EventName: MakeManagedDatabaseAccessible 
+
+Status: Failed 
 
+Description: Access to Azure Key Vault Key has been re-established, operation to make database {database_name} accessible on managed server {server_name} failed 
+  
 
 ## Next steps
 
 - Learn about [Azure Resource Health](/azure/service-health/resource-health-overview).
-- Set up [Action Groups](/azure/azure-monitor/platform/action-groups) to receive notifications and alerts based on your preferences, e.g. Email/SMS/Push/Voice, Logic App, Webhook, ITSM, or Automation Runbook.
+- Set up [Action Groups](/azure/azure-monitor/platform/action-groups) to receive notifications and alerts based on your preferences, e.g. Email/SMS/Push/Voice, Logic App, Webhook, ITSM, or Automation Runbook.