Merge pull request #25573 from PiJoCoder/Error194XX_JoPilov_011322

Create doc files for 19421 and 19419

Author: ttorble

docs/relational-databases/errors-events/includes/database-engine-events-and-errors-19000-20999.md

@@ -90,6 +90,9 @@ ms.topic: include
 |    19148    |    10    |    No    |    Unable to initialize SSL support.    |
 |    19149    |    10    |    No    |    Unable to configure MDAC-compatibility protocol list in registry.    |
 |    19150    |    10    |    No    |    Unable to open SQL Server Network Interface library configuration key in registry.    |
+|    [19407](../mssqlserver-19407-database-engine-error.md)    |    10    |    No    |    The lease between availability group '%.*ls' and the Windows Server Failover Cluster has expired. A connectivity issue occurred between the instance of SQL Server and the Windows Server Failover Cluster. To determine whether the availability group is failing over correctly, check the corresponding availability group resource in the Windows Server Failover Cluster.    |
+|    [19419](../mssqlserver-19419-database-engine-error.md)    |    10    |    No    |    Windows Server Failover Cluster did not receive a process event signal from SQL Server hosting availability group '%.*ls' within the lease timeout period.    |
+|    [19421](../mssqlserver-19421-database-engine-error.md)    |    10    |    No    |    SQL Server hosting availability group '%.*ls' did not receive a process event signal from the Windows Server Failover Cluster within the lease timeout period.    |
 |    20001    |    10    |    No    |    There is no nickname for article '%s' in publication '%s'.    |
 |    20002    |    10    |    No    |    The filter '%s' already exists for article '%s' in publication '%s'.    |
 |    20003    |    10    |    No    |    Could not generate nickname for '%s'.    |

docs/relational-databases/errors-events/mssqlserver-19407-database-engine-error.md

@@ -30,7 +30,7 @@ ms.author: jopilov
 
 Error 19407 is raised in the SQL Server error log when the communication between SQL Server and the Windows Server Failover cluster is lost. Typically a  corrective action occurs - a failover to another Always On node. 
 
-A lease is a time-based communication mechanism that takes place between the SQL Server and the Windows Server Failover Cluster (WSFC) process, specifically the RHS.EXE process. The two processes communicate with each other periodically to ensure the other process is running and responding. This communication takes place using Windows [event objects](/windows/win32/sync/event-objects) and ensures that a failover of the AG resource doesn't occur without the knowledge of the WSFC. If one of the processes doesn't respond to the lease communication based on a predefined lease period, a lease timeout occurs. For detailed information, see [Lease Mechanism](../../database-engine/availability-groups/windows/availability-group-lease-healthcheck-timeout.md). Also see [How It Works: SQL Server AlwaysOn Lease Timeout[(https://techcommunity.microsoft.com/t5/sql-server-support-blog/how-it-works-sql-server-alwayson-lease-timeout/ba-p/317268)
+A lease is a time-based communication mechanism that takes place between the SQL Server and the Windows Server Failover Cluster (WSFC) process, specifically the RHS.EXE process. The two processes communicate with each other periodically to ensure the other process is running and responding. This communication takes place using Windows [Event objects](/windows/win32/sync/event-objects) and ensures that a failover of the AG resource doesn't occur without the knowledge of the WSFC. If one of the processes doesn't respond to the lease communication based on a predefined lease period, a lease timeout occurs. For detailed information, see [Lease Mechanism](../../database-engine/availability-groups/windows/availability-group-lease-healthcheck-timeout.md). Also see [How It Works: SQL Server AlwaysOn Lease Timeout](https://techcommunity.microsoft.com/t5/sql-server-support-blog/how-it-works-sql-server-alwayson-lease-timeout/ba-p/317268)
 
 ### Causes
 
@@ -71,7 +71,7 @@ If there are occurrences of low virtual or physical memory on the system, the SQ
    - **Process\Working Set** - to check individual processes memory usage
    - **Memory\Available MBytes** - to check overall memory usage on the system
 
-   Below is a PowerShell script to identify overall memory usage across all process and the available memory on the system. If you would like to get individual processes memory usage, change it `"\Process(_Total)\Working Set"` to `"\Process(*)\Working Set"`.
+   You can use the following PowerShell script to identify overall memory usage across all process and the available memory on the system. If you would like to get individual processes memory usage, change it `"\Process(_Total)\Working Set"` to `"\Process(*)\Working Set"`.
 
    ```powershell
    $serverName = $env:COMPUTERNAME
@@ -97,7 +97,7 @@ If there are occurrences of low virtual or physical memory on the system, the SQ
 
 ### Reduce or avoid large memory dumps of the SQL Server or cluster process
 
-In some cases SQL Server process may be encountering exceptions, asserts, scheduler issues and so on. In those cases, SQL Server will by default trigger the SQLDumper.exe process to generate a minidump with indirect memory. However, if that dump generation takes a long time, the SQL Server process will stop responding and this may trigger a lease timeout. Frequent causes for a memory dump taking a long time include large memory usage by the process, the I/O subsystem where the dump is written is slow, or the default setting was changed from mini dump to a filtered or full dump. To avoid a lease timeout, do the following on AG systems:
+In some cases SQL Server process may be encountering exceptions, asserts, scheduler issues and so on. In those cases, SQL Server will by default trigger the SQLDumper.exe process to generate a minidump with indirect memory. However, if that dump generation takes a long time, the SQL Server process will stop responding which may trigger a lease timeout. Common causes for a memory dump to take a long time include large memory usage by the process, the I/O subsystem where the dump is written is slow, or the default setting was changed from mini dump to a filtered or full dump. To avoid a lease timeout, use the following steps on AG systems:
 
 - Increase session-timeout, for example, 120 seconds for all replicas
 - Change the auto failover of all replicas to manual failover
@@ -107,7 +107,7 @@ For more information, see [Impact of dump generation](/troubleshoot/sql/tools/us
 
 ### Check virtual machine (VM) configuration for overprovisioning
 
-If you're using a virtual machine, ensure that you aren't overprovisioning or overcommitting CPUs and memory resources. Overprovisioning CPUs or memory may cause the guest OS to run out of resources and show the same problems described above - high CPU and low memory. Frequently if you're viewing things inside the guest OS, you'll have a hard time explaining why you're running out of computing resources because things are happening outside of the virtual machine itself. Overcommitting resources can cause temporary halts of processing, which are likely to cause lease timeouts. For more information on how to address overcommitting, see [Troubleshooting ESX/ESXi virtual machine performance issues (2001003)](https://kb.vmware.com/s/article/2001003) and [Virtualization – Overcommitting memory and how to detect it within the VM](https://techcommunity.microsoft.com/t5/running-sap-applications-on-the/virtualization-8211-overcommitting-memory-and-how-to-detect-it/ba-p/367623).
+If you're using a virtual machine, ensure that you aren't overprovisioning or overcommitting CPUs and memory resources. Overprovisioning CPUs or memory may cause the guest OS to run out of resources and show the same problems described earlier - high CPU and low memory. Frequently if you're viewing things inside the guest OS, you'll have a hard time explaining why you're running out of computing resources because things are happening outside of the virtual machine itself. Overcommitting resources can cause temporary halts of processing, which are likely to cause lease timeouts. For more information on how to address overcommitting, see [Troubleshooting ESX/ESXi virtual machine performance issues (2001003)](https://kb.vmware.com/s/article/2001003) and [Virtualization – Overcommitting memory and how to detect it within the VM](https://techcommunity.microsoft.com/t5/running-sap-applications-on-the/virtualization-8211-overcommitting-memory-and-how-to-detect-it/ba-p/367623).
 
 ### Check for virtual machine (VM) migration or backup
 

docs/relational-databases/errors-events/mssqlserver-19419-database-engine-error.md

@@ -0,0 +1,65 @@
+---
+title: "MSSQLSERVER_19419"
+description: "MSSQLSERVER_19419"
+author: pijocoder
+ms.author: jopilov
+ms.reviewer: pijocoder
+ms.date: 01/13/2023
+ms.service: sql
+ms.subservice: supportability
+ms.topic: "reference"
+helpviewer_keywords:
+  - "19419 (Database Engine error)"
+---
+# MSSQLSERVER_19419
+
+[!INCLUDE [SQL Server](../../includes/applies-to-version/sqlserver.md)]
+
+## Details
+
+| Attribute | Value |
+| :--- | :--- |
+| Product Name | SQL Server |
+| Event ID | 19419 |
+| Event Source | MSSQLSERVER |
+| Component | SQLEngine |
+| Symbolic Name | HADR_AG_LEASE_EXPIRED_WAITING_FOR_RENEW |
+| Message Text | Windows Server Failover Cluster did not receive a process event signal from SQL Server hosting availability group '%.*ls' within the lease timeout period. |
+
+## Explanation
+
+Error 19419 is raised in the SQL Server error log when the lease worker on the SQL Server side didn't get scheduled in time to process event signal from the cluster.
+Specifically, SQL Server calls [WaitForMultipleObjects()](/windows/win32/api/synchapi/nf-synchapi-waitformultipleobjects) waiting for the Lease timeout event to be set in a signaled state. If the function returns WAIT_OBJECT_0, which indicates success, but by this time the lease has expired, then error 19419 is raised.
+
+A lease is a time-based communication mechanism that takes place between the SQL Server and the Windows Server Failover Cluster (WSFC) process, specifically the RHS.EXE process. The two processes communicate with each other periodically to ensure the other process is running and responding. This communication takes place using Windows [Event objects](/windows/win32/sync/event-objects) and ensures that a failover of the AG resource doesn't occur without the knowledge of the WSFC. If one of the processes doesn't respond to the lease communication based on a predefined lease period, a lease timeout occurs. For detailed information, see [Lease Mechanism](../../database-engine/availability-groups/windows/availability-group-lease-healthcheck-timeout.md). Also see [How It Works: SQL Server AlwaysOn Lease Timeout](https://techcommunity.microsoft.com/t5/sql-server-support-blog/how-it-works-sql-server-alwayson-lease-timeout/ba-p/317268)
+
+This error is related to other lease timeout errors and provides more specific detail for error [MSSQLSERVER_19407](mssqlserver-19407-database-engine-error.md)
+
+### Causes
+
+Since Windows Events are light-weight synchronization objects, there's relatively small number of external factors that affect them negatively. Typical issues that can lead to lease timeout involve system-wide problems. Here's a list of possibilities that can cause lease expiration and cause a restart or failover:
+
+- High CPU usage on the system (close to 100%)
+- Out-of-memory conditions - low virtual memory and/or one of the processes is being paged out
+- SQL Server process not responding while generating a large memory dump
+- WSFC going offline (e.g due to quorum loss)
+
+
+The most common reason for error 19419 is high CPU, which causes a delay in scheduling the lease worker thread.
+
+## User action
+
+Check the CPU utilization on the server as SQL Server lease worker seems to be starved for CPU resources. The following PowerShell script will allow you to quickly diagnose CPU usage on the system.
+
+ ```powershell
+  Get-Counter -Counter "\Processor(_Total)\% Processor Time" -SampleInterval 5 -MaxSamples 30 |
+    Select-Object -ExpandProperty CounterSamples | Select-Object TimeStamp, Path, CookedValue
+  ```
+
+For detailed troubleshooting, see User action in [MSSQLSERVER_19407](mssqlserver-19407-database-engine-error.md#user-action)
+
+- Troubleshoot high CPU issues
+- Troubleshoot low memory issues
+- Reduce or avoid large memory dumps of the SQL Server or cluster process
+- Check virtual machine (VM) configuration for overprovisioning
+- Check for virtual machine (VM) migration or backup causing issues

docs/relational-databases/errors-events/mssqlserver-19421-database-engine-error.md

@@ -0,0 +1,56 @@
+---
+title: "MSSQLSERVER_19421"
+description: "MSSQLSERVER_19421"
+author: pijocoder
+ms.author: jopilov
+ms.reviewer: pijocoder
+ms.date: 01/13/2023
+ms.service: sql
+ms.subservice: supportability
+ms.topic: "reference"
+helpviewer_keywords:
+  - "19421 (Database Engine error)"
+---
+# MSSQLSERVER_19421
+
+[!INCLUDE [SQL Server](../../includes/applies-to-version/sqlserver.md)]
+
+## Details
+
+| Attribute | Value |
+| :--- | :--- |
+| Product Name | SQL Server |
+| Event ID | 19421 |
+| Event Source | MSSQLSERVER |
+| Component | SQLEngine |
+| Symbolic Name | HADR_AG_LEASE_RENEWAL_TIMEOUT |
+| Message Text | SQL Server hosting availability group '%.*ls' did not receive a process event signal from the Windows Server Failover Cluster within the lease timeout period. |
+
+## Explanation
+
+Error 19421 is raised in the SQL Server error log when the lease helper on the Windows cluster side didn't signal the SQL Server lease worker thread within the pre-defined lease period. Specifically, SQL Server calls [WaitForMultipleObjects()](/windows/win32/api/synchapi/nf-synchapi-waitformultipleobjects) waiting for the Lease timeout event to be set in a signaled state. If the function returns WAIT_TIMEOUT, because it has exceeded the specified Lease interval, then error 19421 is raised.
+
+A lease is a time-based communication mechanism that takes place between the SQL Server and the Windows Server Failover Cluster (WSFC) process, specifically the RHS.EXE process. The two processes communicate with each other periodically to ensure the other process is running and responding. This communication takes place using Windows [Event objects](/windows/win32/sync/event-objects) and ensures that a failover of the AG resource doesn't occur without the knowledge of the WSFC. If one of the processes doesn't respond to the lease communication based on a predefined lease period, a lease timeout occurs. For detailed information, see [Lease Mechanism](../../database-engine/availability-groups/windows/availability-group-lease-healthcheck-timeout.md). Also see [How It Works: SQL Server AlwaysOn Lease Timeout](https://techcommunity.microsoft.com/t5/sql-server-support-blog/how-it-works-sql-server-alwayson-lease-timeout/ba-p/317268)
+
+This error is related to other lease timeout errors and provides more specific detail for error [MSSQLSERVER_19407](mssqlserver-19407-database-engine-error.md)
+
+### Causes
+
+Since Windows Events are light-weight synchronization objects, there's relatively small number of external factors that affect them negatively. Typical issues that can lead to lease timeout involve system-wide problems. Here's a list of possibilities that can cause lease expiration and cause a restart or failover:
+
+- High CPU usage on the system (close to 100%)
+- Out-of-memory conditions - low virtual memory and/or one of the processes is being paged out
+- SQL Server process not responding while generating a large memory dump
+- WSFC going offline (e.g due to quorum loss)
+
+## User action
+
+Check corresponding Availability Group resource in WSFC cluster to see if it reported any errors.
+
+For detailed troubleshooting, see User action in [MSSQLSERVER_19407](mssqlserver-19407-database-engine-error.md#user-action)
+
+- Troubleshoot high CPU issues
+- Troubleshoot low memory issues
+- Reduce or avoid large memory dumps of the SQL Server or cluster process
+- Check virtual machine (VM) configuration for overprovisioning
+- Check for virtual machine (VM) migration or backup causing issues

docs/toc.yml

@@ -11508,6 +11508,10 @@ items:
         items:
         - name: 19407
           href: relational-databases/errors-events/mssqlserver-19407-database-engine-error.md
+        - name: 19419
+          href: relational-databases/errors-events/mssqlserver-19419-database-engine-error.md
+        - name: 19421
+          href: relational-databases/errors-events/mssqlserver-19421-database-engine-error.md
       - name: Errors 20,000 to 33,999
         items:
         - name: 20557