How to: Debug Exceptions in a Dump file

Introduction

One of the main problems in our applications is exceptions handling.

Right from the planning stage, when architecting a new solution, we are designing an 'Error Handling' mechanism, which naturally effects on the application reliability & stability.

Quite often (especially in large scale solutions) we are encountering lots of different exceptions in our logging systems, and sometimes even experiencing exceptions that crashes our application.

There are 2 types of exceptions:

         1.      First Chance Exception (Non-fatal): An exception was thrown, however the process wasn't crashed.

         2.      Second Chance Exception (Fatal): An exception was thrown and the process was crashed.

We could use debugging tools & debugging techniques to seize 'Crash Dumps', analyze these exceptions and find the root cause of these crashes in our code.

In this post I'll demonstrate how to use debugging extensions in WinDbg (from: 'debugging tools for windows') to find exceptions in our application's Dump file.

Demo – planning

1.      Create a simple application that crashes.

2.      Run the application.

3.      Generate a Dump file.

4.      Analyze the Dump file.

           4. 1          Print all thrown exceptions.

           4. 2          Review the managed Call Stack.

           4. 3          Find the exception in the method disassembly.

           4. 4          Save the source code.

5.      Summary & Conclusions.

  

Demo

1.     Create a simple application that crashes:

I've created a simple application with the following objects and logic:

           -        Person business object (BO):

namespace DebuggingExceptionsExample.BOs {     public class Person     {         public long Id { get; set; }         public string FirstName { get; set; }         public string LastName { get; set; }     } }

           -        PersonException object:

using System; using DebuggingExceptionsExample.BOs; namespace DebuggingExceptionsExample.ErrorHandling {     public class PersonException : Exception     {         private Person _person;         public PersonException(string message,                                Exception innerException = null,                                Person person = null)             : base(message, innerException)         {             _person = person;         }         // TODO: Implement logic.     } }

           -        Main method:

using DebuggingExceptionsExample.BOs; using DebuggingExceptionsExample.ErrorHandling; namespace DebuggingExceptionsExample {     class Program     {         static void Main(string[] args)         {             DebuggingPersonExceptionExample();         }         private static void DebuggingPersonExceptionExample()         {             // 1. Create a person.             Person person = new Person {FirstName = "Yonatan",                                         LastName = "Fedaeli",                                         Id = 1234};             // 2. Validate person details.             if (!ValidatePerson(person))             {                 // 3. Throw a PersonException in case failed validation.                 throw new PersonException("Debugging exceptions example.",                                           person: person);             }         }                 private static bool ValidatePerson(Person person)         {             return false; // For demo purposes.         }     } }

2.     Run the application:

After compiling & running the application we received the expected PersonException, and the application crashed:

The exception message:

Unhandled Exception: DebuggingExceptionsExample.ErrorHandling.PersonException: D ebugging exceptions example. at DebuggingExceptionsExample.Program.DebuggingPersonExceptionExample() in c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\ DebuggingExceptionsExample\Program.cs:line 24 at DebuggingExceptionsExample.Program.Main(String[] args) in c:\Users\Yonatan \Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExcep tionsExample\Program.cs:line 10

From the error message above we could observe the following:

-        Type of exception: DebuggingExceptionsExample.ErrorHandling.PersonException

-        The exception message: Debugging exceptions example.

-        The call stack and method in which the exception was thrown:  DebuggingPersonExceptionExample()

3.     Generate a Dump file:

There are several ways and tools to generate a Dump file:

                1.      Windows Task Manager

                2.      DebugDiag

                3.      ADPlus

                4.      Visual Studio

                5.      ProcDump

                6.      Etc.

In this example, where the application crashes at startup, I'll demonstrate the DebugDiag (Debug Diagnostic) Tool.

          3.1.  Open DebugDiag and add new Rule…

         3.2. Select Target Type:

          3.3.  Select Target:

     I'll type our application name: DebuggingExceptionsExample.exe

          3.4.  Advanced Configuration:

     In our case, we'll select the Exceptions configuration:

          3.5.  Configure Exception:

     In this window we'll set the following:

-        Exception Code: CLR (.Net) 4.x Exception

-        Exception Type Equals…: DebuggingExceptionsExample.ErrorHandling.PersonException

-        Action Type: Full Userdump (so we could perform proper debugging)

          3.6. Save & Close:

          3.7. Select Dump Location And Rule Name:

          3.8. Activate the rule now, and finish:

          3.9.  Now we could observe the Rule we created:

-        Rule Type

-        Rule Name

-        Status

-        Userdump Count

-        Userdump Path

The next time we'll run our application DebugDiag tool would detect it, and based on the rule we just created, would create the Crash Dump file.

4.     Analyze the Dump file:

After generating the Dump file, we could perform different debugging commands to locate the exception and find the root cause of the error in our code.

The basic idea is to explore the application's Threads, their corresponding Call Stacks (Managed & Native) and to review the exceptions that were thrown.

When opening the Dump file in WinDbg we could observe the following:

Remark: For brevity, I omitted some of the irrelevant information.

Microsoft (R) Windows Debugger Version 6.3.9600.17200 X86 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [C:\Program Files\DebugDiag\Logs\Crash rule for all instances of DebuggingExceptionsExample.exe\DebuggingExceptionsExample__PID__9748__Date__01_18_2017__Time_06_14_33PM__148__Second_Chance_Exception_E0434352.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: 'Dump created by DbgHost. Second_Chance_Exception_E0434352' Symbol search path is: *** Invalid *** . . . This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (2614.1dcc): CLR exception - code e0434352 (first/second chance not available) . . . eax=0024ef24 ebx=00000005 ecx=00000005 edx=00000000 esi=0024efdc edi=00000001 eip=7538c6e3 esp=0024ef24 ebp=0024ef74 iopl=0         nv up ei pl nz ac pe nc cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000216 KERNELBASE!RaiseException+0x54: 7538c6e3 c9              leave 0:000> .symfix 0:000> .reload ........................... 0:000> .load psscor4

-        We could clearly see that WinDbg alerts that this Dump file contains an exception.

-        Next, I fixed my symbol path, and loaded the PSSCOR4 (psscor4.dll) debugging extension.

4.1.           Print all thrown exceptions:

Command: !PrintException

Output:

0:000> !PrintException Exception object: 015a3258 Exception type:   DebuggingExceptionsExample.ErrorHandling.PersonException Message:          Debugging exceptions example. InnerException:   StackTrace (generated):     SP       IP       Function     0024F0E0 002E0560 DebuggingExceptionsExample!DebuggingExceptionsExample.Program.DebuggingPersonExceptionExample()+0xe0     0024F104 002E0464 DebuggingExceptionsExample!DebuggingExceptionsExample.Program.Main(System.String[])+0x1c StackTraceString: HResult: 80131500

We could clearly see the:

-        Type of the exception: DebuggingExceptionsExample.ErrorHandling.PersonException

-        The call stack and relevant method: DebuggingPersonExceptionExample()

Remark: Inner Exception

In this case we don't have an inner exception, however, in cases we have inner exceptions; we should investigate all of them up to the root exception, since it probably caused the outer exceptions, and could help resolving this issue.

Remark: Thread Exception

We could print all exceptions in the current thread, since WinDbg opened the Dump file in the thread that caused the crash. We could observe its information and code also in the opening message (presented at section 4 above - Analyze the Dump file).

0:000> !Threads ThreadCount:      2 UnstartedThread:  0 BackgroundThread: 1 PendingThread:    0 DeadThread:       0 Hosted Runtime:   no                                                                          Lock         ID OSID ThreadOBJ    State GC Mode     GC Alloc Context  Domain   Count Apt Exception    0    1 1dcc 00442b70     2a020 Preemptive  015AAD74:00000000 0043b3e0 0     MTA DebuggingExceptionsExample.ErrorHandling.PersonException 015a3258    2    2 1840 004500d0     2b220 Preemptive  00000000:00000000 0043b3e0 0     MTA (Finalizer)

4.2.           Review the managed Call Stack:

Command: !CLRStack –a

Output:

0:000> !CLRStack –a OS Thread Id: 0x1dcc (0) Child SP       IP Call Site 0024f030 7538c6e3 [HelperMethodFrame: 0024f030] 0024f0e0 002e0560 *** WARNING: Unable to verify checksum for DebuggingExceptionsExample.exe DebuggingExceptionsExample.Program.DebuggingPersonExceptionExample() [c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\Program.cs @ 27]     LOCALS:         0x0024f0f0 = 0x015a3240         0x0024f0ec = 0x015a3240         0x0024f0f8 = 0x00000000 0024f104 002e0464 DebuggingExceptionsExample.Program.Main(System.String[]) [c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\Program.cs @ 10]     PARAMETERS:         args (0x0024f104) = 0x015a31fc 0024f280 703d1396 [GCFrame: 0024f280]

We could see that the 'DebuggingPersonExceptionExample()' method has an issue.

4.3.           Find the exception in the method disassembly

Command: !U 002e0560

Output:

0:000> !U 002e0560 Normal JIT generated code DebuggingExceptionsExample.Program.DebuggingPersonExceptionExample() Begin 002e0480, size e5 c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\Program.cs @ 14: 002e0480 55              push    ebp 002e0481 8bec            mov     ebp,esp 002e0483 83ec1c          sub     esp,1Ch 002e0486 33c0            xor     eax,eax . . . c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\Program.cs @ 24: 002e0528 b96c4e2900      mov     ecx,294E6Ch (MT: DebuggingExceptionsExample.ErrorHandling.PersonException) 002e052d e8962bfaff      call    002830c8 (JitHelp: CORINFO_HELP_NEWSFAST) 002e0532 8945e8          mov     dword ptr [ebp-18h],eax 002e0535 b921000000      mov     ecx,21h 002e053a babc3f2900      mov     edx,293FBCh 002e053f e869832870      call    clr!JIT_StrCns (705688ad) 002e0544 8945e4          mov     dword ptr [ebp-1Ch],eax 002e0547 6a00            push    0 002e0549 ff75f4          push    dword ptr [ebp-0Ch] 002e054c 8b55e4          mov     edx,dword ptr [ebp-1Ch] 002e054f 8b4de8          mov     ecx,dword ptr [ebp-18h] 002e0552 ff15544e2900    call    dword ptr ds:[294E54h] (DebuggingExceptionsExample.ErrorHandling.PersonException..ctor(System.String, System.Exception, DebuggingExceptionsExample.BOs.Person), mdToken: 06000008) 002e0558 8b4de8          mov     ecx,dword ptr [ebp-18h] 002e055b e8f50a1970      call    clr!IL_Throw (70471055) c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\Program.cs @ 27: >>> 002e0560 90              nop 002e0561 8be5            mov     esp,ebp 002e0563 5d              pop     ebp 002e0564 c3              ret

-        We could use the !U command to present the method disassembly in order to focus on the root cause of the exception.

-        The !U command expect a 'Code Address', in this case I provided the 'Instruction Pointer Register (EIP)' value, and we could review the exact line in the output using the >>> sign.

4.4.           Save the source code:

In a case we'd like to use the !SaveModule command and to generate the source code from this Dump file (e.g. we don't necessarily possess the source code), we need to first extract the correct executable name of the method that caused the exception.

Most of the time it's like the method namespace, but for cases that the developer didn't follow this convention and wrote a different namespace, we could perform the following:

0:000> !CLRStack OS Thread Id: 0x1dcc (0) Child SP       IP Call Site 0024f030 7538c6e3 [HelperMethodFrame: 0024f030] 0024f0e0 002e0560 DebuggingExceptionsExample.Program.DebuggingPersonExceptionExample() [c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\Program.cs @ 27] 0024f104 002e0464 DebuggingExceptionsExample.Program.Main(System.String[]) [c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\Program.cs @ 10] 0024f280 703d1396 [GCFrame: 0024f280] 0:000> !ip2md 002e0560 MethodDesc:   00294cf8 Method Name:  DebuggingExceptionsExample.Program.DebuggingPersonExceptionExample() Class:        002913e8 MethodTable:  00294d18 mdToken:      0600000a Module:       00293fbc IsJitted:     yes CodeAddr:     002e0480 Transparency: Critical Source file:  c:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\Program.cs @ 27 0:000> !DumpMT 00294d18 EEClass:         002913e8 Module:          00293fbc Name:            DebuggingExceptionsExample.Program mdToken:         02000004 File:            C:\Users\Yonatan\Documents\Visual Studio 2013\Projects\DebuggingExceptionsExample\DebuggingExceptionsExample\bin\Debug\DebuggingExceptionsExample.exe BaseSize:        0xc ComponentSize:   0x0 Slots in VTable: 8 Number of IFaces in IFaceMap: 0

We could see the executable name and path: DebuggingExceptionsExample.exe

Explanation:

           1.     From the !CLRStack command: We extracted the Instruction Pointer Register of the DebuggingPersonExceptionExample() method.

           2.     From the !IP2MD command: We extracted the Method Table address.

           3.     From the !DumpMT command: We extracted the executable full name.

Now we could save the module to a file:

0:000> !SaveModule DebuggingExceptionsExample.exe C:\MyTemp\DebuggingExceptionsExample_CodeFromDump.exe 3 sections in file section 0 - VA=2000, VASize=cb4, FileAddr=200, FileSize=e00 section 1 - VA=4000, VASize=5d0, FileAddr=1000, FileSize=600 section 2 - VA=6000, VASize=c, FileAddr=1600, FileSize=200

Now we could explore the code using a de-compiler such as ILSpy.

Remark: Alternate way to save the source code:

We could also use the WinDbg 'lm' command to first list all the Loaded Modules, and then provide the module 'Start Address' as a parameter to the !SaveModule command:

0:000> lm start    end        module name 00010000 00018000   DebuggingExceptionsExample C (pdb symbols)          C:\ProgramData\dbg\sym\DebuggingExceptionsExample.pdb\9476FA2B9F554AFDA3BC32F124B3CD463\DebuggingExceptionsExample.pdb 0f750000 0f824000   diasymreader   (deferred)             69380000 6a4ca000   mscorlib_ni   (deferred)              6e2f0000 6e32f000   snxhk      (deferred)             70250000 702ce000   clrjit     (deferred)             702d0000 703c5000   MSVCR120_CLR0400   (deferred)             703d0000 70a81000   clr        (pdb symbols)          . . . 77240000 77259000   sechost    (deferred)             77260000 7732c000   msctf      (deferred)          0:000> !SaveModule 00010000 C:\MyTemp\DebuggingExceptionsExample_CodeFromDump_2.exe 3 sections in file section 0 - VA=2000, VASize=cb4, FileAddr=200, FileSize=e00 section 1 - VA=4000, VASize=5d0, FileAddr=1000, FileSize=600 section 2 - VA=6000, VASize=c, FileAddr=1600, FileSize=200

Summary & Conclusions 

-        We learned & demonstrated how to find exceptions in our application using a Dump file.

-        We generated the Dump file using a scheduled Rule in DebugDiag tool.

-        We analyzed the Dump file in WinDbg using the PSSCOR4 debugging extension.

-        We performed commands to analyze the process's Threads and CLR Stack.

-        Finally, we were able to save the source code to a file and decompile it using a de-compiler, in order to explore the code and provide a proper fix.

The End

Hope you enjoyed!

Appreciate your comments…

Yonatan Fedaeli