节后第一天，开发说一个开发库（11g rac+dg ）不能登录了。报错
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925
ORA-01075: you are currently logged on
[oracle@rac01 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Mon Oct 12 09:38:19 2020 Copyright (c) 1982, 2013, Oracle. All rights reserved. ERROR: ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925 ORA-01075: you are currently logged on
初步怀疑是归档满了，空间不够，（开发库的原因，没有处理）
用rman登录，报错
[oracle@rac01 ~]$ rman target / Recovery Manager: Release 11.2.0.4.0 - Production on Mon Oct 12 09:40:46 2020 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-00554: initialization of internal recovery manager package failed RMAN-04005: error from target database: ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925
这里都提示空间不足：
发生上面错误的原因 有以下三种可能： 1磁盘空间已满，导致无法创建审核文件，使用df -h查询磁盘空间； 2磁盘inodes已经占用完了，导致无法打开新的文件，使用df -i查看inode使用情况； 3路径权限不足，无法创建审核文件，查看路径权限情况ls -l
----查看磁盘 [root@rac02 bin]# ./crsctl start cluster -all CRS-4639: Could not contact Oracle High Availability Services CRS-4000: Command Start failed, or completed with errors. [root@rac02 bin]# ll /dev/asm-disk* brw-rw---- 1 grid asmadmin 8, 16 Oct 12 09:57 /dev/asm-diskb brw-rw---- 1 grid asmadmin 8, 32 Oct 12 09:51 /dev/asm-diskc brw-rw---- 1 grid asmadmin 8, 48 Oct 12 09:46 /dev/asm-diskd brw-rw---- 1 grid asmadmin 8, 64 Oct 12 09:57 /dev/asm-diske brw-rw---- 1 grid asmadmin 8, 80 Oct 12 09:57 /dev/asm-diskf brw-rw---- 1 grid asmadmin 8, 96 Oct 12 09:52 /dev/asm-diskg brw-rw---- 1 grid asmadmin 8, 112 Oct 12 09:51 /dev/asm-diskh [root@rac02 bin]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sda 8:0 0 80G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 79.5G 0 part ├─VolGroup-lv_root (dm-0) 253:0 0 50G 0 lvm / ├─VolGroup-lv_swap (dm-1) 253:1 0 8G 0 lvm [SWAP] └─VolGroup-lv_home (dm-2) 253:2 0 21.5G 0 lvm /home sdc 8:32 0 40G 0 disk sdf 8:80 0 5G 0 disk sdg 8:96 0 5G 0 disk sdb 8:16 0 40G 0 disk sdd 8:48 0 40G 0 disk sde 8:64 0 50G 0 disk sdh 8:112 0 5G 0 disk [root@rac02 bin]# ./crs_stat -t -v CRS-0184: Cannot communicate with the CRS daemon. This command is deprecated and has been replaced by 'crsctl status resource' This command remains for backward compatibility only [root@rac02 bin]# find / -name crsd.log /u01/app/11.2.0/grid/log/rac02/crsd/crsd.log [root@rac02 bin]# tail -n 100 /u01/app/11.2.0/grid/log/rac02/crsd/crsd.log 2020-10-12 10:06:28.576: [ CRSMAIN][677160736] Checking the OCR device 2020-10-12 10:06:28.577: [ CRSMAIN][677160736] Sync-up with OCR 2020-10-12 10:06:28.577: [ CRSMAIN][677160736] Connecting to the CSS Daemon 2020-10-12 10:06:28.577: [ CRSMAIN][677160736] Getting local node number 2020-10-12 10:06:28.577: [ CRSMAIN][670709504] Policy Engine is not initialized yet! 2020-10-12 10:06:28.578: [ CRSMAIN][677160736] Initializing OCR [ CLWAL][677160736]clsw_Initialize: OLR initlevel [70000] 2020-10-12 10:06:34.048: [ OCRASM][677160736]proprasmo: Error in open/create file in dg [OCR] [ OCRASM][677160736]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=9925, loc=kgfokge 2020-10-12 10:06:34.048: [ OCRASM][677160736]ASM Error Stack : ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925
使用 df -h 查看 空间正常
查看asm磁盘是否正常挂载
[root@rac02 ~]# su - grid [grid@rac02 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Mon Oct 12 10:08:44 2020 Copyright (c) 1982, 2013, Oracle. All rights reserved. ERROR: ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925 ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925
有点奇怪。。于是google了一下。发现一篇相似
https://blog.csdn.net/HLandHB/article/details/104705626/
检查
[root@rac02 bin]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/VolGroup-lv_root 3276800 3276800 0 100% / tmpfs 4110420 276 4110144 1% /dev/shm /dev/sda1 128016 39 127977 1% /boot /dev/mapper/VolGroup-lv_home 1411680 144 1411536 1% /home
居然是的，磁盘的inode使用完了。
解决：
cd /u01/app/oracle/admin/orcl/adump/
ls | xargs -n 10 rm -fr ls
删除这个目录的多个文件。正常情况下是不会有很多个文件的。查看该文件
[oracle@rac01 adump]$ more ORCL1_ora_7401_20201012114037123698143795.aud Audit file /u01/app/oracle/admin/orcl/adump/ORCL1_ora_7401_20201012114037123698143795.aud Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1 System name: Linux Node name: rac01 Release: 2.6.32-754.el6.x86_64 Version: #1 SMP Tue Jun 19 21:26:04 UTC 2018 Machine: x86_64 VM name: VMWare Version: 6 Instance name: ORCL1 Redo thread mounted by this instance: 1 Oracle process number: 42 Unix process pid: 7401, image: oracle@rac01 (TNS V1-V3) Mon Oct 12 11:40:37 2020 +08:00 LENGTH : '160' ACTION :[7] 'CONNECT' DATABASE USER:[1] '/' PRIVILEGE :[6] 'SYSDBA' CLIENT USER:[6] 'oracle' CLIENT TERMINAL:[5] 'pts/0' STATUS:[1] '0' DBID:[10] '1640682678'
才想起，之前这个库为了查一个东西，开启了errorstack
SQL> show parameter aud NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ audit_file_dest string /u01/app/oracle/admin/ORCL/adum p audit_sys_operations boolean FALSE audit_syslog_level string audit_trail string DB
关闭errorstack
conn / as sysdba
alter system set events '1438 trace name errorstack off';
贴一下alert log
[oracle@rac01 trace]$ tail -n 100 alert_ORCL1.log ORA-16014: log 1 sequence# 1289 not archived, no available destinations ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:26:48 2020 DIA0 detected that there is an archiving problem on the cluster. Several processes including LGWR and at least one ARC process are hung. Instance termination is not allowed. External intervention is required. DIA0 detected that there is an archiving problem on the cluster. Several processes including LGWR and at least one ARC process are hung. Instance termination is not allowed. External intervention is required. Non critical error ORA-48180 caught while writing to trace file "/u01/app/oracle/diag/rdbms/cad/ORCL1/trace/ORCL1_dia0_13133_1923.trc" Error message: Linux-x86_64 Error: 28: No space left on device Additional information: 1 Writing to the above trace file is disabled for now on... Mon Oct 12 09:27:52 2020 Suspending MMON action 'undo usage' for 82800 seconds Mon Oct 12 09:29:19 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16038: log 1 sequence# 1289 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:29:19 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16014: log 1 sequence# 1289 not archived, no available destinations ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:32:58 2020 Some DDE async actions failed or were cancelled Non critical error ORA-48180 caught while writing to trace file "/u01/app/oracle/diag/rdbms/orcl/ORCL1/trace/ORCL1_m002_26767.trc" Error message: Linux-x86_64 Error: 28: No space left on device Additional information: 1 Writing to the above trace file is disabled for now on... Mon Oct 12 09:34:19 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16038: log 1 sequence# 1289 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16014: log 1 sequence# 1289 not archived, no available destinations ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:35:00 2020 *********************************************************************** Fatal NI connect error 12170. VERSION INFORMATION: TNS for Linux: Version 11.2.0.4.0 - Production Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production Time: 12-OCT-2020 09:35:00 Tracing not turned on. Tns error struct: ns main err code: 12535 TNS-12535: TNS:operation timed out ns secondary err code: 12606 nt main err code: 0 nt secondary err code: 0 nt OS err code: 0 Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.15.4.40)(PORT=2884)) WARNING: inbound connection timed out (ORA-3136) Mon Oct 12 09:38:24 2020 OS Audit file could not be created; failing after 6 retries Mon Oct 12 09:38:26 2020 Non critical error ORA-48180 caught while writing to trace file "/u01/app/oracle/diag/rdbms/orcl/ORCL1/trace/ORCL1_m002_27284.trc" Error message: Linux-x86_64 Error: 28: No space left on device Additional information: 1 Writing to the above trace file is disabled for now on... OS Audit file could not be created; failing after 6 retries Mon Oct 12 09:39:49 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16038: log 1 sequence# 1289 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:39:49 2020 Non critical error ORA-48180 caught while writing to trace file "/u01/app/oracle/diag/rdbms/orcl/CAD1/trace/CAD1_m003_27424.trc" Error message: Linux-x86_64 Error: 28: No space left on device Additional information: 1 Writing to the above trace file is disabled for now on... Mon Oct 12 09:39:49 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16014: log 1 sequence# 1289 not archived, no available destinations ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:40:52 2020 OS Audit file could not be created; failing after 6 retries