目錄

一、什么是core dump？

二、coredump是怎么來(lái)的？

三、怎么限制coredump文件的產(chǎn)生？

ulimit

半永久限制

永久限制

四、從源碼分析如何對(duì)coredump文件的名字和路徑管理

命名?

管理

一些問(wèn)題的答案

1、為什么新的ubuntu不能產(chǎn)生coredump了，需要手動(dòng)管理？

2、systemd是什么時(shí)候出現(xiàn)在ubuntu中的？

3、為什么使用腳本管理coredump后ulimit就不能限制coredump文件的大小了。

4、為什么ulimit限制的大小并不準(zhǔn)確？

5、使用腳本就沒(méi)辦法限制coredump的大小了嘛？

6、ubuntu現(xiàn)在對(duì)coredump文件的管理方式

---

一、什么是core dump？

????????core dump是Linux應(yīng)用程序調(diào)試的一種有效方式，core dump又稱為“核心轉(zhuǎn)儲(chǔ)”，是該進(jìn)程實(shí)際使用的物理內(nèi)存的“快照”。分析core dump文件可以獲取應(yīng)用程序崩潰時(shí)的現(xiàn)場(chǎng)信息，如程序運(yùn)行時(shí)的CPU寄存器值、堆棧指針、棧數(shù)據(jù)、函數(shù)調(diào)用棧等信息。在嵌入式系統(tǒng)開(kāi)發(fā)中是開(kāi)發(fā)人員調(diào)試所需必備材料之一。每當(dāng)程序異常行為產(chǎn)生coredump時(shí)我們就可以利用gdb工具去查看程序崩潰的原因。

二、coredump是怎么來(lái)的？

????????Core dump是Linux基于信號(hào)實(shí)現(xiàn)的。Linux中信號(hào)是一種異步事件處理機(jī)制，每種信號(hào)都對(duì)應(yīng)有默認(rèn)的異常處理操作，默認(rèn)操作包括忽略該信號(hào)（Ignore）、暫停進(jìn)程（Stop）、終止進(jìn)程（Terminate）、終止并產(chǎn)生core dump（Core）等。

Signal	Value	Action	Comment
SIGHUP	1	Term	Hangup detected on controlling terminal or death of controlling process
SIGINT	2	Term	Interrupt from keyboard
SIGQUIT	3	Core	Quit from keyboard
SIGILL	4	Core	Illegal Instruction
SIGTRAP	5	Core	Trace/breakpoint trap
SIGABRT	6	Core	Abort signal from abort(3)
SIGIOT	6	Core	IOT trap. A synonym for SIGABRT
SIGEMT	7	Term
SIGFPE	8	Core	Floating point exception
SIGKILL	9	Term	Kill signal, cannot be caught, blocked or ignored.
SIGBUS	10,7,10	Core	Bus error (bad memory access)
SIGSEGV	11	Core	Invalid memory reference
SIGPIPE	13	Term	Broken pipe: write to pipe with no readers
SIGALRM	14	Term	Timer signal from alarm(2)
SIGTERM	15	Term	Termination signal
SIGUSR1	30,10,16	Term	User-defined signal 1
SIGUSR2	31,12,17	Term	User-defined signal 2
SIGCHLD	20,17,18	Ign	Child stopped or terminated
SIGCONT	19,18,25	Cont	Continue if stopped
SIGSTOP	17,19,23	Stop	Stop process, cannot be caught, blocked or ignored.
SIGTSTP	18,20,24	Stop	Stop typed at terminal
SIGTTIN	21,21,26	Stop	Terminal input for background process
SIGTTOU	22,22,27	Stop	Terminal output for background process
SIGIO	23,29,22	Term	I/O now possible (4.2BSD)
SIGPOLL		Term	Pollable event (Sys V). Synonym for SIGIO
SIGPROF	27,27,29	Term	Profiling timer expired
SIGSYS	12,31,12	Core	Bad argument to routine (SVr4)
SIGURG	16,23,21	Ign	Urgent condition on socket (4.2BSD)
SIGVTALRM	26,26,28	Term	Virtual alarm clock (4.2BSD)
SIGXCPU	24,24,30	Core	CPU time limit exceeded (4.2BSD)
SIGXFSZ	25,25,31	Core	File size limit exceeded (4.2BSD)
SIGSTKFLT	16	Term	Stack fault on coprocessor (unused)
SIGCLD	18	Ign	A synonym for SIGCHLD
SIGPWR	29,30,19	Term	Power failure (System V)
SIGINFO	29		A synonym for SIGPWR, on an alpha
SIGLOST	29	Term	File lock lost (unused), on a sparc
SIGWINCH	28,28,20	Ign	Window resize signal (4.3BSD, Sun)
SIGUNUSED	31	Core	Synonymous with SIGSYS

以下情況會(huì)出現(xiàn)應(yīng)用程序崩潰導(dǎo)致產(chǎn)生core dump：

1. 內(nèi)存訪問(wèn)越界 （數(shù)組越界、字符串無(wú)\n結(jié)束符、字符串讀寫(xiě)越界）
2. 多線程程序中使用了線程不安全的函數(shù)，如不可重入函數(shù)
3. 多線程讀寫(xiě)的數(shù)據(jù)未加鎖保護(hù)（臨界區(qū)資源需要互斥訪問(wèn)）
4. 非法指針（如空指針異?；蛘叻欠ǖ刂吩L問(wèn)）
5. 堆棧溢出

三、怎么限制coredump文件的產(chǎn)生？

ulimit

????????說(shuō)到限制就不得不提到ulimit工具了，這個(gè)工具其實(shí)有個(gè)坑，他并不是linux自帶的工具，而是shell工具，也就是說(shuō)他的級(jí)別是shell層的。

? ? ? ? 當(dāng)我們新開(kāi)啟一個(gè)終端后之前終端的全部配置都會(huì)消失，并且他還看人下菜碟，每個(gè)linux用設(shè)置的都是自己的ulimit，比如你有兩個(gè)用戶，在同一個(gè)終端下對(duì)當(dāng)前終端配置

一個(gè) ulimit -c 1024

一個(gè)ulimit -c 2048

他們的限制都作用在各自的范圍

????????即便同一個(gè)目錄的同一個(gè)文件，本來(lái)這個(gè)coredump可以產(chǎn)生4096k個(gè)字節(jié)那么a的coredump就是1024，b的就是2048當(dāng)然，不會(huì)很準(zhǔn)確的限制，后面會(huì)從源碼上給大家說(shuō)明為什么。

? ? ? ? 還有就是可以通過(guò)配置文件，但是這個(gè)優(yōu)先級(jí)雖然高但是可以改，這個(gè)文件只在重啟時(shí)被讀取一次，后面其它人再來(lái)改這個(gè)限制你之前的限制也就無(wú)效了。

半永久限制

vim /etc/profile
相當(dāng)于每開(kāi)一個(gè)shell時(shí)，自動(dòng)執(zhí)行ulimit -c unlimited（表示無(wú)限制其它具體數(shù)字通常表示多少k）

永久限制

編輯/etc/security/limits.conf，需要重新登錄，或者重新打開(kāi)ssh客戶端連接，永久生效

下面就是軟限制和硬限制

四、從源碼分析如何對(duì)coredump文件的名字和路徑管理

命名?

????????在/etc/sysctrl.conf中保存的是用戶給內(nèi)核傳遞的參數(shù)，這里有個(gè)core_pattern參數(shù)是告訴內(nèi)核怎么管理coredump的。

后面直接寫(xiě)等號(hào)就表示這個(gè)coredump文件的命名

%% 單個(gè)%字符

%p 所dump進(jìn)程的進(jìn)程ID

%u 所dump進(jìn)程的實(shí)際用戶ID

%g 所dump進(jìn)程的實(shí)際組ID

%s 導(dǎo)致本次core dump的信號(hào)

%t core dump的時(shí)間 (由1970年1月1日計(jì)起的秒數(shù))

%h 主機(jī)名

%e 程序文件名

管理

kernel.core_pattern=|/usr/bin/coredump_helper.sh core_%e_%I_%p_sig_%s_time_%t.gz

????????像上面那樣配置core_pattern后內(nèi)核就可以使用我們指定的腳本去管理coredump文件了。

這個(gè)路徑必須是絕對(duì)路徑，前面必須是|前后都不能有空格，coredump文件會(huì)作為參數(shù)傳入腳本。

這里我直接將文件以壓縮格式傳遞給了腳本

#!/bin/shif [ ! -d "/var/coredump" ];thenmkdir -p /var/coredump
fi
gzip > "/var/coredump/$1"

上面的代碼就是一個(gè)簡(jiǎn)單的示例可以把coredump都放到var下的coredump目錄中，我們還可以添加一些遍歷判斷操作限制產(chǎn)生coredump的數(shù)量的單個(gè)coredump的大小。?

#!/bin/bash  # Set the maximum number of files allowed in /var/coredump  
max_files=1000  # Check if /var/coredump directory exists  
if [ ! -d "/var/coredump" ]; then  mkdir -p /var/coredump  
fi  # Get the current number of files in /var/coredump  
current_files=$(ls -1 /var/coredump | wc -l)  # Check if the number of files exceeds the maximum limit  
if [ $current_files -ge $max_files ]; then  # Sort the files based on creation time (oldest first)  sorted_files=$(ls -1t /var/coredump | head -n 1)  # Remove the oldest files to reduce the number of files to below the limit  rm $sorted_files  
fi  # Compress the file and store it in /var/coredump using gzip  
gzip > "/var/coredump/$1"

代碼很簡(jiǎn)陋正常還要有出錯(cuò)管理，異常情況打印到日志備份等等。?

一些問(wèn)題的答案

1、為什么新的ubuntu不能產(chǎn)生coredump了，需要手動(dòng)管理？

????????因?yàn)楝F(xiàn)在的ubuntu使用的是systemd，在這里有些服務(wù)會(huì)對(duì)內(nèi)核管理coredump的接口做配置，導(dǎo)致coredump文件被服務(wù)寫(xiě)到接口的腳本管理了。

2、systemd是什么時(shí)候出現(xiàn)在ubuntu中的？

????????systemd是在Ubuntu 15.04版本中首次出現(xiàn)的。在此之前，Ubuntu使用Upstart作為默認(rèn)的init系統(tǒng)。然而，從Ubuntu 15.04開(kāi)始，Ubuntu采用了systemd作為默認(rèn)的init系統(tǒng)和服務(wù)管理器。systemd是一個(gè)用于啟動(dòng)、停止和管理系統(tǒng)進(jìn)程的軟件套件，它提供了更快的啟動(dòng)速度、更好的系統(tǒng)資源管理和更強(qiáng)大的服務(wù)控制能力。?

? ? ? ? 而在Upstart之前用的才是SysVinit。

3、為什么使用腳本管理coredump后ulimit就不能限制coredump文件的大小了。

這個(gè)問(wèn)題就要看源碼了

void do_coredump(const siginfo_t *siginfo)
{struct core_state core_state;struct core_name cn;struct mm_struct *mm = current->mm;struct linux_binfmt * binfmt;const struct cred *old_cred;struct cred *cred;int retval = 0;int flag = 0;int ispipe;struct files_struct *displaced;bool need_nonrelative = false;bool core_dumped = false;static atomic_t core_dump_count = ATOMIC_INIT(0);struct coredump_params cprm = {.siginfo = siginfo,.regs = signal_pt_regs(),.limit = rlimit(RLIMIT_CORE),/** We must use the same mm->flags while dumping core to avoid* inconsistency of bit flags, since this flag is not protected* by any locks.*/.mm_flags = mm->flags,};audit_core_dumps(siginfo->si_signo);binfmt = mm->binfmt;if (!binfmt || !binfmt->core_dump)goto fail;if (!__get_dumpable(cprm.mm_flags))goto fail;cred = prepare_creds();if (!cred)goto fail;/** We cannot trust fsuid as being the "true" uid of the process* nor do we know its entire history. We only know it was tainted* so we dump it as root in mode 2, and only into a controlled* environment (pipe handler or fully qualified path).*/if (__get_dumpable(cprm.mm_flags) == SUID_DUMP_ROOT) {/* Setuid core dump mode */flag = O_EXCL;		/* Stop rewrite attacks */cred->fsuid = GLOBAL_ROOT_UID;	/* Dump root private */need_nonrelative = true;}retval = coredump_wait(siginfo->si_signo, &core_state);if (retval < 0)goto fail_creds;old_cred = override_creds(cred);ispipe = format_corename(&cn, &cprm);if (ispipe) {int dump_count;char **helper_argv;struct subprocess_info *sub_info;if (ispipe < 0) {printk(KERN_WARNING "format_corename failed\n");printk(KERN_WARNING "Aborting core\n");goto fail_unlock;}if (cprm.limit == 1) {/* See umh_pipe_setup() which sets RLIMIT_CORE = 1.** Normally core limits are irrelevant to pipes, since* we're not writing to the file system, but we use* cprm.limit of 1 here as a speacial value, this is a* consistent way to catch recursive crashes.* We can still crash if the core_pattern binary sets* RLIM_CORE = !1, but it runs as root, and can do* lots of stupid things.** Note that we use task_tgid_vnr here to grab the pid* of the process group leader.  That way we get the* right pid if a thread in a multi-threaded* core_pattern process dies.*/printk(KERN_WARNING"Process %d(%s) has RLIMIT_CORE set to 1\n",task_tgid_vnr(current), current->comm);printk(KERN_WARNING "Aborting core\n");goto fail_unlock;}cprm.limit = RLIM_INFINITY;dump_count = atomic_inc_return(&core_dump_count);if (core_pipe_limit && (core_pipe_limit < dump_count)) {printk(KERN_WARNING "Pid %d(%s) over core_pipe_limit\n",task_tgid_vnr(current), current->comm);printk(KERN_WARNING "Skipping core dump\n");goto fail_dropcount;}helper_argv = argv_split(GFP_KERNEL, cn.corename, NULL);if (!helper_argv) {printk(KERN_WARNING "%s failed to allocate memory\n",__func__);goto fail_dropcount;}retval = -ENOMEM;sub_info = call_usermodehelper_setup(helper_argv[0],helper_argv, NULL, GFP_KERNEL,umh_pipe_setup, NULL, &cprm);if (sub_info)retval = call_usermodehelper_exec(sub_info,UMH_WAIT_EXEC);argv_free(helper_argv);if (retval) {printk(KERN_INFO "Core dump to |%s pipe failed\n",cn.corename);goto close_fail;}} else {struct inode *inode;if (cprm.limit < binfmt->min_coredump)goto fail_unlock;if (need_nonrelative && cn.corename[0] != '/') {printk(KERN_WARNING "Pid %d(%s) can only dump core "\"to fully qualified path!\n",task_tgid_vnr(current), current->comm);printk(KERN_WARNING "Skipping core dump\n");goto fail_unlock;}cprm.file = filp_open(cn.corename,O_CREAT | 2 | O_NOFOLLOW | O_LARGEFILE | flag,0600);if (IS_ERR(cprm.file))goto fail_unlock;inode = file_inode(cprm.file);if (inode->i_nlink > 1)goto close_fail;if (d_unhashed(cprm.file->f_path.dentry))goto close_fail;/** AK: actually i see no reason to not allow this for named* pipes etc, but keep the previous behaviour for now.*/if (!S_ISREG(inode->i_mode))goto close_fail;/** Dont allow local users get cute and trick others to coredump* into their pre-created files.*/if (!uid_eq(inode->i_uid, current_fsuid()))goto close_fail;if (!cprm.file->f_op->write)goto close_fail;if (do_truncate(cprm.file->f_path.dentry, 0, 0, cprm.file))goto close_fail;}/* get us an unshared descriptor table; almost always a no-op */retval = unshare_files(&displaced);if (retval)goto close_fail;if (displaced)put_files_struct(displaced);if (!dump_interrupted()) {file_start_write(cprm.file);core_dumped = binfmt->core_dump(&cprm);file_end_write(cprm.file);}if (ispipe && core_pipe_limit)wait_for_dump_helpers(cprm.file);
close_fail:if (cprm.file)filp_close(cprm.file, NULL);
fail_dropcount:if (ispipe)atomic_dec(&core_dump_count);
fail_unlock:kfree(cn.corename);coredump_finish(mm, core_dumped);revert_creds(old_cred);
fail_creds:put_cred(cred);
fail:return;
}

????????從源碼中可以看到在產(chǎn)生coredump的源碼中會(huì)對(duì)上面的core_pattern進(jìn)行判斷，如果傳入了腳本就會(huì)走上面的分支，沒(méi)傳入走下面的分支，上面只判斷了limit是不是1，1表示不產(chǎn)生coredump，如果不是1就賦值為0，0就是沒(méi)有限制。

????????而下面有這個(gè)判斷，當(dāng)已經(jīng)產(chǎn)生的coredump文件超過(guò)ulimit -c的限制時(shí)就會(huì)停止產(chǎn)生。有一個(gè)點(diǎn)需要注意，ulimit-c和這里的limit不是完全對(duì)應(yīng)的，是在shell源碼中有轉(zhuǎn)化的，不過(guò)大體上還是對(duì)應(yīng)的。

4、為什么ulimit限制的大小并不準(zhǔn)確？

還是要看這張圖，它每次寫(xiě)入的都是內(nèi)核設(shè)計(jì)好的，就寫(xiě)這些。

在外面有個(gè)for循環(huán)調(diào)用do_coredump文件.

int get_signal_to_deliver(siginfo_t *info, struct k_sigaction *return_ka,struct pt_regs *regs, void *cookie)
{struct sighand_struct *sighand = current->sighand;struct signal_struct *signal = current->signal;int signr;if (unlikely(current->task_works))task_work_run();if (unlikely(uprobe_deny_signal()))return 0;/** Do this once, we can't return to user-mode if freezing() == T.* do_signal_stop() and ptrace_stop() do freezable_schedule() and* thus do not need another check after return.*/try_to_freeze();relock:spin_lock_irq(&sighand->siglock);/** Every stopped thread goes here after wakeup. Check to see if* we should notify the parent, prepare_signal(SIGCONT) encodes* the CLD_ si_code into SIGNAL_CLD_MASK bits.*/if (unlikely(signal->flags & SIGNAL_CLD_MASK)) {int why;if (signal->flags & SIGNAL_CLD_CONTINUED)why = CLD_CONTINUED;elsewhy = CLD_STOPPED;signal->flags &= ~SIGNAL_CLD_MASK;spin_unlock_irq(&sighand->siglock);/** Notify the parent that we're continuing.  This event is* always per-process and doesn't make whole lot of sense* for ptracers, who shouldn't consume the state via* wait(2) either, but, for backward compatibility, notify* the ptracer of the group leader too unless it's gonna be* a duplicate.*/read_lock(&tasklist_lock);do_notify_parent_cldstop(current, false, why);if (ptrace_reparented(current->group_leader))do_notify_parent_cldstop(current->group_leader,true, why);read_unlock(&tasklist_lock);goto relock;}for (;;) {struct k_sigaction *ka;if (unlikely(current->jobctl & JOBCTL_STOP_PENDING) &&do_signal_stop(0))goto relock;if (unlikely(current->jobctl & JOBCTL_TRAP_MASK)) {do_jobctl_trap();spin_unlock_irq(&sighand->siglock);goto relock;}signr = dequeue_signal(current, &current->blocked, info);if (!signr)break; /* will return 0 */if (unlikely(current->ptrace) && signr != SIGKILL) {signr = ptrace_signal(signr, info);if (!signr)continue;}ka = &sighand->action[signr-1];/* Trace actually delivered signals. */trace_signal_deliver(signr, info, ka);if (ka->sa.sa_handler == SIG_IGN) /* Do nothing.  */continue;if (ka->sa.sa_handler != SIG_DFL) {/* Run the handler.  */*return_ka = *ka;if (ka->sa.sa_flags & SA_ONESHOT)ka->sa.sa_handler = SIG_DFL;break; /* will return non-zero "signr" value */}/** Now we are doing the default action for this signal.*/if (sig_kernel_ignore(signr)) /* Default is nothing. */continue;/** Global init gets no signals it doesn't want.* Container-init gets no signals it doesn't want from same* container.** Note that if global/container-init sees a sig_kernel_only()* signal here, the signal must have been generated internally* or must have come from an ancestor namespace. In either* case, the signal cannot be dropped.*/if (unlikely(signal->flags & SIGNAL_UNKILLABLE) &&!sig_kernel_only(signr))continue;if (sig_kernel_stop(signr)) {/** The default action is to stop all threads in* the thread group.  The job control signals* do nothing in an orphaned pgrp, but SIGSTOP* always works.  Note that siglock needs to be* dropped during the call to is_orphaned_pgrp()* because of lock ordering with tasklist_lock.* This allows an intervening SIGCONT to be posted.* We need to check for that and bail out if necessary.*/if (signr != SIGSTOP) {spin_unlock_irq(&sighand->siglock);/* signals can be posted during this window */if (is_current_pgrp_orphaned())goto relock;spin_lock_irq(&sighand->siglock);}if (likely(do_signal_stop(info->si_signo))) {/* It released the siglock.  */goto relock;}/** We didn't actually stop, due to a race* with SIGCONT or something like that.*/continue;}spin_unlock_irq(&sighand->siglock);/** Anything else is fatal, maybe with a core dump.*/current->flags |= PF_SIGNALED;if (sig_kernel_coredump(signr)) {if (print_fatal_signals)print_fatal_signal(info->si_signo);proc_coredump_connector(current);/** If it was able to dump core, this kills all* other threads in the group and synchronizes with* their demise.  If we lost the race with another* thread getting here, it set group_exit_code* first and our do_group_exit call below will use* that value and ignore the one we pass it.*/do_coredump(info);}/** Death signals, no core dump.*/do_group_exit(info->si_signo);/* NOTREACHED */}spin_unlock_irq(&sighand->siglock);return signr;
}

哪次判斷超出了就會(huì)停止生成.

5、使用腳本就沒(méi)辦法限制coredump的大小了嘛？

????????可以的，但是limit這個(gè)系統(tǒng)調(diào)用肯定是不行了，源碼中我們就看到了人家都不鳥(niǎo)你。但是我們可以u(píng)limit -f，這個(gè)腳本coredump文件從內(nèi)核態(tài)到用戶態(tài)是通過(guò)這個(gè)傳入的腳本，所以直接ulimit -f限制腳本生成的全部文件大小就ok了，但是這里要寫(xiě)成正常兩倍大小因?yàn)槭窍全@取在寫(xiě)入，我們一個(gè)字節(jié)計(jì)算兩次，這次限制的大小就完全準(zhǔn)確了，也有個(gè)壞處就是文件會(huì)損壞限制住了也沒(méi)法去看這個(gè)coredump。

6、ubuntu現(xiàn)在對(duì)coredump文件的管理方式

????????由于使用systemd機(jī)制，所以現(xiàn)在的coredump是以服務(wù)的方式存在的，為了可以更好的保存和規(guī)范化管理，coredump內(nèi)容被放到了日志中，使用coredump@和socket服務(wù)共同實(shí)現(xiàn)，將coredump文件從內(nèi)核態(tài)轉(zhuǎn)移到用戶態(tài)后又用了socket機(jī)制做過(guò)濾和寫(xiě)入日志。完美限制大小和內(nèi)容。只要日志管理的好可以想看任何時(shí)候的有用的coredump。

---------------------------------------------------------------------------------------------------------------------------------

????????這是我工作中遇到的第一個(gè)做了兩周的問(wèn)題的一部分，當(dāng)時(shí)賊痛苦各種查資料翻源碼，好在最后想到了還算不錯(cuò)的解決方案，工作的提升是真的大，不止是技術(shù)，主要是團(tuán)隊(duì)寫(xiě)作能力，做事情的條例，規(guī)范化做事才會(huì)少出錯(cuò)。這些很重要。還有表達(dá)能力，不然你寫(xiě)的永遠(yuǎn)都是垃圾。下周有時(shí)間更新另一半，日志的管理。