CentOS yum Fatal Error 处理一例

环境说明

[root@thatsit ~]# cat  /etc/redhat-release 
CentOS Linux release 7.1.1503 (Core) 
[root@thatsit ~]# uname -r
3.10.0-229.7.2.el7.x86_64
[root@thatsit ~]#

报错信息

[root@thatsit ~]# yum  -y install  docker  
error: rpmdb: BDB0113 Thread/process 38301/139855495489536 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
[root@thatsit ~]#

处理步骤

1, rm -f /var/lib/rpm/__db*    # 删除/var/lib/rpm 目录下的__db开头的rpmdb文件
2, rpm --rebuilddb             # rpm数据库重建
3, yum clean all               # 清理所有yum缓存
4, yum makecache               # 重新生成yum缓存

※: 此解决法方法在RHEL/CentOS 5(未测试),6,7系列的操作系统中都是适用的.

执行过程

[root@thatsit ~]# cd  /var/lib/rpm
[root@thatsit rpm]# ll  -htr
total 42M
-rw-r--r--. 1 root root 8.0K Feb 18 17:31 Triggername
-rw-r--r--. 1 root root  16K Feb 18 17:31 Obsoletename
-rw-r--r--. 1 root root 8.0K Feb 18 17:31 Conflictname
-rw-r--r--. 1 root root  32K Feb 18 17:31 Name
-rw-r--r--. 1 root root 2.4M Feb 18 17:31 Basenames
-rw-r--r--. 1 root root  16K Feb 18 17:31 Group
-rw-r--r--. 1 root root 200K Feb 18 17:31 Requirename
-rw-r--r--. 1 root root 1.5M Feb 18 17:31 Providename
-rw-r--r--. 1 root root 1.1M Feb 18 17:31 Dirnames
-rw-r--r--. 1 root root  12K Feb 18 17:31 Installtid
-rw-r--r--. 1 root root  40K Feb 18 17:31 Sigmd5
-rw-r--r--. 1 root root  52K Feb 18 17:31 Sha1header
-rw-r--r--. 1 root root  35M Feb 18 18:46 Packages
-rw-r--r--  1 root root 1.3M Feb 18 22:47 __db.003
-rw-r--r--  1 root root 104K Feb 18 22:47 __db.002
-rw-r--r--  1 root root 432K Feb 18 22:47 __db.001
[root@thatsit rpm]# rm -f __db*
[root@thatsit rpm]# rpm --rebuilddb
[root@thatsit rpm]# yum clean all
Loaded plugins: fastestmirror
Cleaning repos: base epel extras updates
Cleaning up everything
Cleaning up list of fastest mirrors
[root@thatsit rpm]# yum makecache
Loaded plugins: fastestmirror
base                                                         | 3.6 kB  00:00:00     
epel/x86_64/metalink                                         | 5.7 kB  00:00:00     
epel                                                         | 4.3 kB  00:00:00     
extras                                                       | 3.4 kB  00:00:00     
updates                                                      | 3.4 kB  00:00:00     
(1/17): base/7/x86_64/group_gz                               | 155 kB  00:00:00     
(2/17): base/7/x86_64/primary_db                             | 5.3 MB  00:00:05     
(3/17): epel/x86_64/group_gz                                 | 169 kB  00:00:00     
(4/17): base/7/x86_64/filelists_db                           | 6.2 MB  00:00:06     
(5/17): extras/7/x86_64/filelists_db                         | 246 kB  00:00:00     
(6/17): extras/7/x86_64/prestodelta                          |  16 kB  00:00:00     
(7/17): extras/7/x86_64/primary_db                           |  90 kB  00:00:00     
(8/17): epel/x86_64/other_db                                 | 1.8 MB  00:00:01     
(9/17): epel/x86_64/primary_db                               | 3.9 MB  00:00:01     
(10/17): base/7/x86_64/other_db                              | 2.3 MB  00:00:02     
(11/17): extras/7/x86_64/other_db                            | 256 kB  00:00:00     
(12/17): updates/7/x86_64/prestodelta                        | 239 kB  00:00:00     
(13/17): updates/7/x86_64/filelists_db                       | 1.9 MB  00:00:01     
(14/17): epel/x86_64/updateinfo                              | 488 kB  00:00:00     
(15/17): updates/7/x86_64/primary_db                         | 3.1 MB  00:00:03     
(16/17): epel/x86_64/filelists_db                            | 6.4 MB  00:00:03     
(17/17): updates/7/x86_64/other_db                           |  26 MB  00:00:26     
Determining fastest mirrors
 * base: mirrors.aliyun.com
 * epel: mirrors.yun-idc.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Metadata Cache Created
[root@thatsit rpm]#

故障原因:

(原文地址:http://serverfault.com/questions/616666/why-does-yum-index-get-corrupted)

In the general case, this happens when rpm (or yum) crashes when updating the rpmdb, which is a Berkeley DB key-value store, and very sensitive. When such a crash happens, the rpmdb is left in an inconsistent state and this error occurs. All of the other files in /var/lib/rpm contain the same information, though in a less efficient format, so the database is easily rebuilt.

Two notable bugs you may have seen on older CentOS systems can cause this. The big one, a "nasty and subtle race in shared mmap'ed page writeback" as it appears in the changelog, was quietly fixed in a kernel update in 2007. This one presented itself slightly differently than your report, though.

The one you might see from 2009 happened when PackageKit would kill yum at an inopportune time, and was also fixed. This would be more likely to affect desktop systems or servers with a GUI, though.

All of these bugs predate EL 6, and you should almost never see this occur on EL 6 or 7, nor should you see it if your EL 5 systems are up to date. (I have no idea about EL 4. If you have one, kill it before it spreads.) That said, anything that causes yum or rpm to die while working with the rpmdb could cause it. This includes what you're most likely to see these days, random cosmic rays flipping bits, or someone getting overzealous with kill -9.

In RHEL 7, yum traps more signals during the actual transaction run, and you'll see the message (shutdown inhibited). This should help prevent most situations in which someone or something interrupts the transaction and causes this problem.