|
Debugging with Coredump in Linux
A core dump is essentially a file that captures the memory state of a computer program at a specific moment when the program has crashed or terminated abnormally. The core dump file includes a detailed snapshot of the program's memory, including stack traces, contents of processor registers, memory addresses, and other contextual information about the process's state at the time of its failure. This information is crucial for developers to diagnose and debug the causes of the crash.
How Core Dumps Work ?
Here’s a breakdown of how core dumps are generated and how they function to help in debugging:
- Crash or Signal Trigger: A core dump is typically generated when a program crashes due to an unhandled error or when it receives certain types of signals from the operating system that indicate a fault, such as SIGSEGV (Segmentation Fault), SIGABRT (Abort Signal from abort() call), and others. These signals indicate that the program has attempted to perform an illegal operation, such as accessing forbidden memory or dividing by zero.
- Operating System Intervention: When such a crash or critical signal occurs, the operating system interrupts the normal execution of the program. The operating system's signal handler recognizes that the crash would typically produce a core dump and prepares to capture the state of the program.
- Writing the Core Dump: The operating system collects the contents of the program’s memory at the time of the crash. This includes the contents of the registers, the program counter, the stack, the heap, and global variables. It writes all this data into a core dump file. This file can be very large, depending on the size of the program's memory footprint.
- Core Dump File Storage: The location and naming convention of the core dump file can often be configured (as discussed previously using the kernel.core_pattern setting in Linux). By default, it is usually saved in the current directory of the crashing process or a specified directory with a name that might include the process ID, timestamp, and other details.
- Post-Mortem Debugging: After the core dump file is saved, developers can use debugging tools like GNU Debugger (GDB) in Linux to analyze this file. By loading the core dump file into the debugger, they can inspect the state of the program at the time of the crash. This includes viewing the call stack, examining variable values, and performing other inquiries to understand what went wrong.
- Analysis and Fixing Bugs: Developers analyze the data from the core dump to identify the source of the error or bug that caused the crash. They can see exactly what the program was doing at the time of the crash, which functions were called, and what the values of various variables were. This information helps them to fix the bug and prevent future crashes.
How to utilize Core Dumps ?
Following is common procedure to utilize the core dump
Checking if Core Dump is Enabled
To verify if your system is set up to generate core dumps, check the core file size limit with:
or check it with following command:
Example >
[sharetechnote]# ulimit -a | grep core
core file size (blocks, -c) unlimited
[sharetechnote]# ulimit -c
unlimited
|
Enabling Core Dump
If the output for core file size is 0, core dumps are disabled. Enable them by setting the limit to unlimited:
Locating the Core Dump File
Core dumps are usually placed in the process's current working directory unless specified otherwise. Check the core dump pattern to understand the naming and location:
$ cat /proc/sys/kernel/core_pattern
|
Example >
[sharetechnote]# cat /proc/sys/kernel/core_pattern
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
|
- %p: pid
- %u: uid
- %g: gid
- %s: signal number
- %t: UNIX time of dump
- %h: hostname
- %e: executable filename
Listing Core Dumps
To list core dumps, use the coredumpctl command.
If you want to list it from the beginning
If you want to list the last 5 list:
$ coredumpctl list | tail -n 5
|
Example >
[sharetechnote]# coredumpctl list | tail -n 5
Tue 2024-04-09 08:41:03 EDT 84658 0 0 SIGSEGV missing /root/xvx2 n/a
Tue 2024-04-09 11:50:46 EDT 91558 0 0 SIGSEGV missing /root/xxx n/a
Wed 2024-04-24 16:16:05 EDT 428262 0 0 SIGSEGV present /root/xxx 163.4K
Wed 2024-04-24 16:40:21 EDT 428359 0 0 SIGSEGV present /root/xxx 163.3K
Wed 2024-04-24 16:44:43 EDT 428438 0 0 SIGSEGV present /root/xxx 163.3K
|
Analyzing the Core Dump Contents
You can analyze the core dump contents with following command
Example >
[sharetechnote]# coredumpctl gdb 428438
PID: 428438 (myprog)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Wed 2024-04-24 16:44:43 EDT (1min 20s ago)
Command Line: /root/myprogram/myprog
Executable: /root/myprogram/myprog
Control Group: /system.slice/miniGen.service
Unit: miniGen.service
Slice: system.slice
Boot ID: 3821271ea3dc47869f51a5cec6af68ef
Machine ID: 896594a48e744cfc9ffd7fa691b60c72
Hostname: CBC-2023010100
Storage: /var/lib/systemd/coredump/core.myProg.0.3821271ea3dc478438.....zst (present)
Disk Size: 163.3K
Message: Process 428438 (miniGen) of user 0 dumped core.
Stack trace of thread 428444:
#0 0x00007fd71fcffc8e __strcmp_avx2 (libc.so.6 + 0x15bc8e)
#1 0x000000000040332d n/a (/root/myprogram/myprog + 0x332d)
GNU gdb (GDB) Fedora 11.1-5.fc34
Copyright (C) 2021 Free Software Foundation, Inc.
....
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /root/myprogram/myprog...
...
#0 0x00007fd71fcffc8e in __strcmp_xxx () from /lib64/libc.so.6
|
Extracting a Specific Core Dump into a File
you can save a specific items from the coredump list into file as follows.
$ coredumpctl dump <pid> --output=<filename>
|
Example >
[sharetechnote]# coredumpctl dump 428438 --output=coredump_428438
PID: 428438 (miniGen)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Wed 2024-04-24 16:44:43 EDT (28min ago)
Command Line: /root/myprog/myProg
Executable: /root/myprog/myProg
Control Group: /system.slice/miniGen.service
Unit: miniGen.service
Slice: system.slice
Boot ID: 3821271ea3dc47869f51a5cec6af68ef
Machine ID: 896594a48e744cfc9ffd7fa691b60c72
Hostname: CBC-2023010100
Storage: /var/lib/systemd/coredump/core.miniG...428438.1713991483000000.zst (present)
Disk Size: 163.3K
Message: Process 428438 (miniGen) of user 0 dumped core.
Stack trace of thread 428444:
#0 0x00007fd71fcffc8e __strcmp_xxx (libc.so.6 + 0x15bc8e)
#1 0x000000000040332d n/a (/root/myprog/myProg + 0x332d)
|
Analyzing the Extracted/Saved Core Dump File
To analyze an extracted or saved core dump file, load it into GDB with the corresponding executable:
$ gdb </path/to/executable> </path/to/extracted_core.dump>
|
Example >
[sharetechnote]# gdb miniGen coredump_428438
GNU gdb (GDB) Fedora 11.1-5.fc34
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later ....
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from miniGen...
warning: Can't open file /dev/sdr0 during file-backed mapping note processing
[New LWP 428444]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/root/myprog/myProg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fd71fcffc8e in __strcmp_xxx () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7fd717fff640 (LWP 428444))]
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.33-21.fc34.x86_64
|
Errors with Microsoft Compiler
If you get this error in DotNet 2008 (may be working in other versoin as well), Disabling 'Create/Use Precompiled Header' might solve the problem. See Enabling/Disableing Precompiled Header for the details.
This error would take many different form depending on situation and would print out as ''function' : cannot convert parameter number from 'type1' to 'type2''
Example 1 >
Code causing the error:
Error Message :
Solution :
< Case 1 > cause this error because the pointer myStrPtr is not initialized (in this case it means that no memory is allocated to store the array of chars (string). < Case 2 > removes the error.
< Case 1 >
char *myStrPtr;
strcpy(myStrPtr,"Hello World");
< Case 2 >
char *myStrPtr;
myStrPtr=(char *)malloc(20);
strcpy(myStrPtr,"Hello World");
|
|