Changing Log Files Location In RHEL

Log files often help System Administrators in finding out important information about System related issues.

In RHEL there are a lot of log files under /var/log like messages ,maillog,faillog,cups and cron etc.

Here we will discuss how can we decide the location of log files and forward appropriate messages to these locations.

In RHEL /etc/syslog.conf is the file responsible for deciding log file location.Open this file and you’ll see it. syslogd is the daemon associated with it and after every change you made to this file you need to restart this daemon.

Now you can make entry in this file as we have done in following examples:

mail.err   /var/log/mailerr will redirect mail error messages to /var/log/mailerr.
*.crit    /var/log/critical will redirect all critical messages to /var/log/critical.
kern.* @viny.example.com   will send all kernel messages to a remote host viny.example.com

The first part in this example is known as Facility and the second is called Priority. Following are some important facilities you can use:

authpriv-Security/authorization messages
cron
kern-kernel messages
lpr-Line printer messages
mail-Mail messages
syslog-Syslog internal messages

Lock User Accounts After Failed Login

Open /etc/pam.d/system-auth file.

Now write these lines just above the line stating auth sufficient pam_unix.so:

auth required pam_tally2.so onerr=fail deny=3

account  required   pam_tally2.so reset

Now the account will be locked after three failed login attempts.

There are various other options you can use like:

unlock_time=100 will unlock the account after 100 seconds.

You can use any value for unlock_time field.

If you don’t use this parameter then you’ll have to manually unlock a locked account using
# pam_tally2 -r -u username command.

lock_time=100 will lock the account for 100 seconds after failed attempts to login.

no_magic_root will avoid locking root account.

VCS Points

Service Group: Collection of dependent Resources
Resource: Anything that the end user requires
Resource Type: Collection of the resources with same type
Agents: To manage the Resource Types (Start,Stop and Monitor)
Service Group Online: Child Resource to Parent Resource
Service Group Offline: Parent Resource to Child Resource

LLT Files
/etc/llthosts
/etc/llttab

GAB Files:
/etc/gabtab

Manipulating Service Groups:
1. hagrp -offline AppSG -sys S1 -localclus --> Offline the AppSG only in S1 system (node)
2. hagrp -offline OracleSG -any --> Offline the OracleSG in all the systems
3. hagrp -online AppSG -sys S2 -localclus --> Online the AppSG in node S2
4. hagrp -switch AppSG -to S1  -->  AppSG will be moved to node S1

Manipulating Resources:
1. hares -offline Oralistener -sys S3  -->  Bring offline the Oralistener resource in node S3
2. hares -online ipres -sys S2  ->  Bring online the ipres resource in node S2

Handling VCS services:
haconf -dump -makero --> sync the RAM's main.cf with hardisk's main.cf and make the status as Readonly

hastop -all --> Stop the application and cluster

hastop -all -force --> Application will be continue running but the cluster service has been stopped
hastop -local --> stop the cluster service in local node

Useful VCS Commands

SERVICE GROUPS AND RESOURCE OPERATIONS:
Configuring service groups
hagrp –add|-delete|-online|-offline group_name

Modifying resources
hares –add|-delete res_name type group
hares –online|-offline res_name –sys system_name

Modifying agents
haagent –start|-stop agent_name –sys system_name

BASIC CONFIGURATION OPERATIONS:
Service Goups
hagrp -modify group_name attribute_name value
hagrp –list group_name
hagrp –value attribute_name

Resources
hares -modify res_name attribute_name value
hares -link res_name res_name


Agents
haagent -display agent_name –sys system_name
hatype –modify

VCS ENGINE OPERATIONS:
Starting had
hastart –force|–stale system_name
hasys –force system_name

Stopping had
hastop –local|-all|-force|-evacuate
hastop –sys system_name

Adding Users
hauser –add user_name

STATUS AND VERIFICATION:
Group Status/Verification
hagrp -display group_name|–state|–resource group_name

Resources Status/Verification
hares -display res_name
hares –list
hares -probe res_name –sys system_name

Agents Status/Verification
haagent –list
haagent -display agent_name –sys system_name
ps –ef|grep agent_name

VCS Status
hastatus –group
LLT Status/Verification
lltconfig –a list
lltstat|lltshow|lltdump

GAB Status/Verification
gabconfig –a
gabdiskhb –l

COMMUNICATION:
Starting and Stopping LLT
lltconfig -U
lltconfig -c
lltconfig -a list

Starting and Stopping GAB
gabconfig –c –n #seed number (eg: gabconfig -c -n 2)
gabconfig –U

Administering Group Services
hagrp –clear|-flush|-switch group_name –sys system_name

Administering Resources
hares –clear|-probe res_name –sys system_name

Administering Agents
haagent -list
haagent -display agent_name –sys system_name

Verify Configuration
hacf –verify

VCS Troubleshooting

Linux Log Files

/var/log/messages : General message and system related stuff
/var/log/auth.log : Authenication logs
/var/log/kern.log : Kernel logs
/var/log/cron.log : Crond logs (cron job)
/var/log/maillog : Mail server logs
/var/log/qmail/ : Qmail log directory (more files inside this directory)
/var/log/httpd/ : Apache access and error logs directory
/var/log/lighttpd/ : Lighttpd access and error logs directory
/var/log/boot.log : System boot log
/var/log/mysqld.log : MySQL database server log file
/var/log/secure or /var/log/auth.log : Authentication log
/var/log/utmp or /var/log/wtmp : Login records file
/var/log/yum.log : Yum command log file.

LLT Port Information

[root@vcs1 /]# lltstat -p
LLT port information:
    Port    Usage        Cookie
      0     GAB_Control  0x0
          opens:     0 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
      7     gab          0x7
          opens:     0 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
     31     gab          0x1F
          opens:     0 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
[root@vcs1 /]#


[root@vcs2 /]# lltstat -p
LLT port information:
    Port    Usage        Cookie
      0     GAB_Control  0x0
          opens:     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
      7     gab          0x7
          opens:     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
     31     gab          0x1F
          opens:     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
[root@vcs2 /]#

So the LLT is configured on 3 links running on the port 0, 7 and 31.

Add a link to LLT

[root@vcs1 /]# lltconfig -t eth2 -d eth2
LLT lltconfig INFO V-14-2-15653 No link type is specified, defaulting to link type "ether"
[root@vcs1 /]# lltconfig -a list
Link 0 (eth0):
  Node   0 vcs1       :   00:0C:29:DE:BA:35  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:FB

Link 1 (eth1):
  Node   0 vcs1       :   00:0C:29:DE:BA:3F  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:05

Link 2 (eth2):
  Node   0 vcs1       :   00:0C:29:DE:BA:49  permanent

[root@vcs1 /]#

[root@vcs2 /]# lltconfig -t eth2 -d eth2
LLT lltconfig INFO V-14-2-15653 No link type is specified, defaulting to link type "ether"
[root@vcs2 /]#

[root@vcs1 /]# lltconfig -a list
Link 0 (eth0):
  Node   0 vcs1       :   00:0C:29:DE:BA:35  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:FB

Link 1 (eth1):
  Node   0 vcs1       :   00:0C:29:DE:BA:3F  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:05

Link 2 (eth2):
  Node   0 vcs1       :   00:0C:29:DE:BA:49  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:0F

[root@vcs1 /]#

Removing a link from LLT

[root@vcs1 /]# lltconfig -u eth2----------> Command to remove a link from LLT Layer. (Need to execute the command on both the nodes)

[root@vcs2 /]# lltconfig -u eth2

This command unconfigures the LLT link, but does not permanently remove the link from configuration files.

Upon reboot, or restart of LLT, the cluster node will use the link again.

To remove the link permanently from the configuration, on each cluster node modify the /etc/llttab file and remove the line for device "eth2".

[root@vcs1 /]# lltconfig -a list
Link 0 (eth0):
  Node   0 vcs1       :   00:0C:29:DE:BA:35  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:FB

Link 1 (eth1):
  Node   0 vcs1       :   00:0C:29:DE:BA:3F  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:05

[root@vcs1 /]#


LLT node information:
    Node                 State    Link  Status  Address
   * 0 vcs1              OPEN
                                  eth0   UP      00:0C:29:DE:BA:35
                                  eth1   UP      00:0C:29:DE:BA:3F
     1 vcs2              OPEN
                                  eth0   UP      00:0C:29:52:9B:FB
                                  eth1   UP      00:0C:29:52:9B:05


[root@vcs1 /]# lltstat -l
LLT link information:
link 0  eth0 on ether hipri
        mtu 1500, sap 0xcafe, broadcast FF:FF:FF:FF:FF:FF, addrlen 6
        txpkts 3429  txbytes 370955
        rxpkts 3180  rxbytes 232432
        latehb 0  badcksum 0  errors 0
link 1  eth1 on ether hipri
        mtu 1500, sap 0xcafe, broadcast FF:FF:FF:FF:FF:FF, addrlen 6
        txpkts 3450  txbytes 349877
        rxpkts 3216  rxbytes 227532
        latehb 0  badcksum 0  errors 0
[root@vcs1 /]#

NTP Parameters

node1:/var/log # ntpdc -p (uses sec instead of ms).

     remote           local      st poll reach  delay   offset    disp
=======================================================================
=LOCAL(0)        127.0.0.1       10   64  377 0.00000  0.000000 0.03053
*circle.net          16.88.11.29   2  512  377 0.06363 -0.017075 0.11371
=netservice       16.88.11.29  16 1024    0 0.00000  0.000000 3.99217
=ntp1.iso           16.88.11.29   3  512  377 0.11404 -0.010279 0.10500
node1:/var/log #

“+” – Good and a preferred remote peer or server (included by the combine algorithm)

“*” – The remote peer or server presently used as the primary reference;

poll : NTP polls each peer every “poll” seconds.

when : is relative time of last poll.

reach : track of last successful polls, 377 is best.

Delay : is network delay, this is fine

offset : offset between local and peer clock (ms) and it should be low.

jitter : Mean deviation (jitter) in the time reported for that remote peer or server.

How to configure NTP Server In Linux

Configured NTP Server in 192.168.100.22

#Server Side
============
NTP Server Pool Address
1.server 1.asia.pool.ntp.org
2.server 0.asia.pool.ntp.org

#Permitting local network to use this
restrict 192.168.100.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.20.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.30.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.40.0 mask 255.255.255.0 nomodify notrap

#full access to the local net
restrict 127.0.0.1

#Client Side
===========

#yum install ntp

edit the ntp.conf file
#vim /etc/ntp.conf

#comment the following lines
#server 0.centos.pool.ntp.org
#server 1.centos.pool.ntp.org
#server 2.centos.pool.ntp.org

#add the following lines
restrict default ignore
server 192.168.100.22
restrict 192.168.100.1 mask 255.255.255.255 nomodify notrap noquery

#save and quit the file.

#Before starting the ntp daemon, run the following command

#ntpdate -b 192.168.100.223

#service ntpd start

#check the ntp

[root@node1 ~]# ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================
192.168.100.223 140.109.1.4 11 u 3 64 1 0.257 0.368 0.001
127.127.1.0 .LOCL. 10 l 2 64 1 0.000 0.000 0.001
[root@node1 ~]#

/dev/sdb: read failed after 0 of 4096 at 0: Input/output error

When a storage unit is longer available to the Linux Server, You would see the following error message:

[root@node1 /]# pvs
  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 2147418112: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 2147475456: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
  PV                VG       Fmt  Attr PSize  PFree
  /dev/mapper/disk1          lvm2 a--   1.00g 1.00g
  /dev/mapper/disk2          lvm2 a--   1.00g 1.00g
  /dev/mapper/disk3          lvm2 a--   1.00g 1.00g
  /dev/sda2         vg_node1 lvm2 a--  19.51g    0
[root@node1 /]#

To fix the issue, We need to delete the corresponding /dev/sdb file from OS Level.

[root@node1 /]# echo 1 > /sys/block/sdb/device/delete

[root@node1 /]# pvs
  PV                VG       Fmt  Attr PSize  PFree
  /dev/mapper/disk1          lvm2 a--   1.00g 1.00g
  /dev/mapper/disk2          lvm2 a--   1.00g 1.00g
  /dev/mapper/disk3          lvm2 a--   1.00g 1.00g
  /dev/sda2         vg_node1 lvm2 a--  19.51g    0
[root@node1 /]#

VxVM vxdisk ERROR V-5-1-684 IPC failure: Configuration daemon is not accessible

[root@node2 yum.repos.d]# cd /

[root@node2 /]# vxdisk list
VxVM vxdisk ERROR V-5-1-684 IPC failure: Configuration daemon is not accessible

[root@node2 /]# cd /etc/vx/reconfig.d/state.d/

[root@node2 state.d]# ls -l | grep -i install-db

-rw-r--r--. 1 root root 0 Sep  1 06:21 install-db

[root@node2 state.d]# rm -rf install-db

[root@node2 state.d]# vxconfigd

[root@node2 state.d]# vxdctl init

[root@node2 state.d]# vxdctl enable

[root@node2 state.d]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
sda          auto:LVM        -            -            online invalid
[root@node2 state.d]#

initramfs (RHEL 6)

dracut utility can be used in RHEL 6 to rebuild the initrams image ,It  creates  an initial image used by the kernel for preloading the block device modules (such as IDE, SCSI or RAID) which  are  needed  to access the root filesystem.


1.       It is recommended you make a backup copy of the initrd in case the new version has an unexpected problem:

# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak


2      Now rebuild the initramfs for the current kernel version:

# dracut -f

Note:

If you are in a kernel version different to the initrd we are building (including if you are in Rescue Mode). We must specify the full kernel version, including architecture:


# dracut -f initramfs-2.6.32-71.el6.x86_64.img 2.6.32-71.el6.x86_64

The only purpose of an initramfs is to mount the root filesystem.

The initramfs is a complete set of directories that you would find on a normal root filesystem.

At boot time, the boot loader loads the kernel and the initramfs image into memory and starts the kernel. 

The kernel checks for the presence of the initramfs and, if found, mounts it as / and runs /init. 

The init program is typically a shell script. Note that the boot process takes longer, possibly significantly longer, if an initramfs is used. 

Kdump Configuration

Kump Introduction

Kdump is the Linux kernel crash-dump mechanism. Oracle recommends that you enable the Kdump feature. In the event of a system crash, Kdump creates a memory image (vmcore) that can help in determining the cause of the crash. Enabling Kdump requires you to reserve a portion of system memory for exclusive use by Kdump. This memory is unavailable for other uses.

Kdump uses kexec to boot into a second kernel whenever the system crashes. kexec is a fast-boot mechanism which allows a Linux kernel to boot from inside the context of a kernel that is already running without passing through the bootloader stage.


When the kdump crash dumping mechanism is enabled, the system is booted from the context of another kernel. This second kernel reserves a small amount of memory and its only purpose is to capture the core dump image in case the system crashes.

Configuring the kdump Service:
There are three common means of configuring the kdump service: at the first boot, using the Kernel Dump Configuration graphical utility, and doing so manually on the command line.
Configuring the kdump at First Boot: The menu which we see during OS Installation.

Using the Kernel Dump Configuration Utility:

Select System → Administration → Kernel crash dumps from the panel, or type system-config-kdump at a shell prompt

Configuring kdump on the Command Line:
To configure the amount of memory to be reserved for the kdump kernel, edit the /boot/grub/grub.conf file and add crash kernel=M or crash kernel=auto

How to go previous date from current date

[root@node1 ~]# date
Sun Jul 13 05:29:50 EDT 2014

[root@node1 ~]# date -d "30 Days Ago"
Fri Jun 13 05:33:45 EDT 2014

Who Can Fix This YUM Issue ?

I am getting the above error upon giving the yum command.

CD is mounted.

Please find the YUM conf file:

[root@node1 yum.repos.d]# cat rhel-source.repo
[rhel source]
name=Red Hat Enterprise Linux $releasever - $basearch - Source
baseurl=file:///mnt/
enabled=1
gpgcheck=0

[root@node1 yum.repos.d]#

Who Can Fix This Linux Boot Issue.

After reboot, I get the below message in the console.

What is a Mirror Log ?

LVM maintains a small log which it uses to keep track of which regions are in sync with the mirror or mirrors. 

By default, this log is kept on disk, which keeps it persistent across reboots and ensures that the mirror does not need to be re-synced every time a machine reboots or crashes. 

You can specify instead that this log be kept in memory with the --mirrorlog core argument; this eliminates the need for an extra log device, but it requires that the entire mirror be resynchronized at every reboot.

Linux LVM Mirroring (RAID -1 ) Part II

Now going to mirror the logical volume:

[root@test ~]# lvconvert -m1 /dev/oracle/oracle_lv
  oracle/oracle_lv: Converted: 0.0%
  oracle/oracle_lv: Converted: 100.0%
[root@test ~]#

[root@test ~]# lvdisplay -m /dev/oracle/oracle_lv
  --- Logical volume ---
  LV Name                /dev/oracle/oracle_lv
  VG Name                oracle
  LV UUID                O9W6go-l2AC-0XP9-GvLw-tL2C-bjeI-YoWmDG
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.00 GiB
  Current LE             256
  Mirrored volumes       2
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Segments ---
  Logical extent 0 to 255:
    Type                mirror
    Mirrors             2
    Mirror size         256
    Mirror log volume   oracle_lv_mlog
    Mirror region size  512.00 KiB
    Mirror original:
      Logical volume    oracle_lv_mimage_0
      Logical extents   0 to 255
    Mirror destinations:
      Logical volume    oracle_lv_mimage_1
      Logical extents   0 to 255


[root@test ~]#

[root@test ~]# pvs -o +lv_name

  PV         VG     Fmt  Attr PSize PFree LV

  /dev/sdb   oracle lvm2 a--  5.00g 4.00g [oracle_lv_mimage_1]

  /dev/sdb   oracle lvm2 a--  5.00g 4.00g

  /dev/sdc   oracle lvm2 a--  5.00g 3.99g [oracle_lv_mimage_0]

  /dev/sdc   oracle lvm2 a--  5.00g 3.99g [oracle_lv_mlog]

  /dev/sdc   oracle lvm2 a--  5.00g 3.99g

[root@test ~]#

Linux LVM Mirroring (RAID -1 ) Part I

This post I am going to show how to mirror a logical volume:

[root@test ~]# df -TH /oracle
Filesystem    Type     Size   Used  Avail Use% Mounted on
/dev/mapper/oracle-oracle_lv
              ext4     1.1G   1.1G      0 100% /oracle
[root@test ~]#

Before mirroring:

[root@test ~]# lvdisplay -m /dev/oracle/oracle_lv

  --- Logical volume ---

  LV Name                /dev/oracle/oracle_lv

  VG Name                oracle

  LV UUID                O9W6go-l2AC-0XP9-GvLw-tL2C-bjeI-YoWmDG

  LV Write Access        read/write

  LV Status              available

  # open                 1

  LV Size                1.00 GiB

  Current LE             256

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto

  - currently set to     256

  Block device           253:0

  --- Segments ---

  Logical extent 0 to 255:

    Type                linear

    Physical volume     /dev/sdc

    Physical extents    0 to 255

Initializing and adding a new disk to the VG Oracle:

[root@test ~]# pvcreate /dev/sdb

  Writing physical volume data to disk "/dev/sdb"

  Physical volume "/dev/sdb" successfully created

[root@test ~]# vgextend /dev/oracle /dev/sdb

  Volume group "oracle" successfully extended

[root@test ~]# 

RMAN-03009: failure of backup command on ORA_DISK_1 channel at 07/09/2014 11:21:25 ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode

The reason for the above error is the due to the database is configured to run in NOARCHIVE Mode.

So to fix the error, We need to convert the database to ARCHIVE Mode.

SQL> archive log list;
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            /u01/app/oracle/product/11.2.0/db_1/dbs/arch
Oldest online log sequence     2
Current log sequence           4
SQL>

Converting  to ARCHIVE Mode.

SQL> alter database archivelog;

alter database archivelog

*

ERROR at line 1:

ORA-01126: database must be mounted in this instance and not open in any instance

SQL>

So, doing that before the database should be in Mounted state.

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area  392495104 bytes

Fixed Size                  2213696 bytes

Variable Size             138414272 bytes

Database Buffers          247463936 bytes

Redo Buffers                4403200 bytes

Database mounted.

SQL> alter database archivelog;

Database altered.

SQL> archive log list;

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            /u01/app/oracle/product/11.2.0/db_1/dbs/arch

Oldest online log sequence     2

Next log sequence to archive   4

Current log sequence           4

SQL> alter database open;

Database altered.

SQL> 

Bingo...!!!!!!!!!!!!!

Configure Virtual Serial Port

1)      /etc/securetty  -append
ttyS0
ttyS1

2)      /etc/inittab  - uncomment and modify
S0:12345:respawn:/sbin/agetty -L 115200 ttyS0 vt100

3)      /boot/grub/menu.lst – modify the line of running kernel
root (hd0,0)
    kernel /vmlinuz-2.6.16.60-0.69.1-smp root=/dev/vg00/lvol1 vga=normal nomodeset 3 resume=/dev/vg00/lvol2 splash=silent showopts
crashkernel=256M@16M console=ttyS0,115200
    initrd /initrd-2.6.16.60-0.69.1-smp

Kdump memory requirement

---------------------------------------------------------------------------
    Configuring crashkernel on RHEL6.0 and RHEL6.1 kernels
---------------------------------------------------------------------------

Some mappings of ram and appropriate crashkernel values:

ram size        crashkernel parameter        ram / crashkernel factor
>0GB            128MB                        15
>2GB            256MB                        23
>6GB            512MB                        15
>8GB            768MB                        31

Linux Swapiness

The swappiness parameter controls the tendency of the kernel to move processes out of physical memory and onto the swap disk. Because disks are much slower than RAM, this can lead to slower response times for system and applications if processes are too aggressively moved out of memory.

swappiness can have a value of between 0 and 100

swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible
swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache


The default setting in Linux is swappiness=60. Reducing the default value of swappiness will probably improve overall performance for a typical desktop installation. A value of swappiness=10 is recommended, but feel free to experiment.

To check the swappiness value use command: cat /proc/sys/vm/swappiness

To make a change permanent, edit the configuration file with your favorite editor:

vi /etc/sysctl.conf
and add following parameter to the end of the file like so:
vm.swappiness=10
Save the file and reboot.

System-wide File Descriptors (FD) Limits

Many application such as Oracle database or Apache web server needs this range quite higher. So you can increase the maximum number of open files by setting a new value in kernel variable /proc/sys/fs/file-max as follows (login as the root):
# sysctl -w fs.file-max=100000

Above command forces the limit to 100000 files. You need to edit /etc/sysctl.conf file and put following line so that after reboot the setting will remain as it is:
# vi /etc/sysctl.conf

Append a config directive as follows:
fs.file-max = 100000

Save and close the file. Users need to log out and log back in again to changes take effect or just type the following command:
# sysctl -p

Verify your settings with command:
# cat /proc/sys/fs/file-max

OR
# sysctl fs.file-max

1) Add the following line to /etc/security/limits.conf

webuser hard nofile 64000
then login as webuser

su - webuser
2) Edit following two files for webuser

append .bashrc and .bash_profile file by running

echo "ulimit -n 64000" >> .bashrc ; echo "ulimit -n 64000" >> .bash_profile
3) Log out, then log back in and verify that the changes have been made correctly:

$ ulimit -a | grep open
open files                      (-n) 64000

fsck - check and repair a Linux file system

       The exit code returned by fsck is the sum of the following conditions:
            0    - No errors
            1    - File system errors corrected
            2    - System should be rebooted
            4    - File system errors left uncorrected
            8    - Operational error
            16   - Usage or syntax error
            32   - Fsck canceled by user request
            128  - Shared library error

Improve Faster IO for your filesystem by chaning the READ AHEAD Value

[root@test oracle]# df -Th /oracle
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/oracle-oracle_lv
              ext4   1008M   34M  924M   4% /oracle

[root@test oracle]# time dd if=/oracle/test of=/tmp/test count=500 bs=1M
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 6.2385 s, 84.0 MB/s

real    0m6.249s
user    0m0.001s
sys     0m1.886s
[root@test oracle]#

[root@test oracle]# blockdev --report /dev/sdb
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0      5368709120   /dev/sdb
[root@test oracle]#


I changed the read ahead value from 256 to 1024:

[root@test ~]#  blockdev --setra 1024 /dev/sdb
[root@test ~]# blockdev --getra /dev/sdb
1024
[root@test ~]# blockdev --report /dev/sdb
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  1024   512  4096          0      5368709120   /dev/sdb
[root@test ~]#

[root@test oracle]# time dd if=/oracle/test of=/tmp/test count=500 bs=1M
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 3.79428 s, 138 MB/s

real    0m3.819s
user    0m0.003s
sys     0m1.589s
[root@test oracle]#


The read performace has improved from 6.2 Sec to 3.8 Sec. Faster IO.

Currently set to and Block device in lvdisplay ?

[root@test ~]# lvdisplay /dev/hits/hits1
  --- Logical volume ---
  LV Name                /dev/hits/hits1
  VG Name                hits
  LV UUID                W7uUkN-WAIf-MrY7-u7oE-x9uy-AegG-WYATjI
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                1.00 GiB
  Current LE             256
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

[root@test ~]#

Most of us would wonder few things with the output of # lvdisplay especially the bottom two lines.
currently set to and Block device.

Block device:

Block Devices refers to the DM (Device Multipath) Associated with the logical volume.

[root@test ~]# dmsetup ls --tree
hits-hits1 (253:0)
 └─ (8:16)

[root@test dev]# ls -l | grep -i "253"

brw-rw----. 1 root disk    253,   0 Jul  4 17:50 dm-0

[root@test dev]#

dm-0 is the device multipath associated with the logical volume hits1.

How to check the Block Size and Sector Size of HDD in Linux ?

[root@test ~]# blockdev --report /dev/sdb
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0      5368709120   /dev/sdb
[root@test ~]#

RA : Read Ahead.

SSZ : Sector Size.

BSZ : Block Size.

Size : Size of the disk.

Available commands:

        --getsz                        get size in 512-byte sectors

        --setro                        set read-only

        --setrw                        set read-write

        --getro                        get read-only

        --getss                        get logical block (sector) size

        --getpbsz                      get physical block (sector) size

        --getiomin                     get minimum I/O size

        --getioopt                     get optimal I/O size

        --getalignoff                  get alignment offset

        --getmaxsect                   get max sectors per request

        --getbsz                       get blocksize

        --setbsz BLOCKSIZE             set blocksize

        --getsize                      get 32-bit sector count

        --getsize64                    get size in bytes

        --setra READAHEAD              set readahead

        --getra                        get readahead

        --setfra FSREADAHEAD           set filesystem readahead

        --getfra                       get filesystem readahead

        --flushbufs                    flush buffers

        --rereadpt                     reread partition table

Check The DATABASE_STATUS

SQL> select STARTUP_TIME,STATUS,LOGINS,DATABASE_STATUS from v$instance where INSTANCE_NUMBER=1;

STARTUP_T STATUS       LOGINS     DATABASE_STATUS
--------- ------------ ---------- -----------------
23-JUN-14 OPEN         ALLOWED    ACTIVE

SQL>

This would show when the DB was started, status and the mode of the DB Started.

How to check with the Database is running in Dedicated or Shared mode?

SQL> select username,status,server from v$session where username='SYS';

USERNAME                       STATUS   SERVER
------------------------------ -------- ---------
SYS                            ACTIVE   DEDICATED

SQL>

The Server states would show the Mode of Database Activation.

Oracle Listener Configuration Manually : Part II

Oracle Listener Configuration Manually : Part I

Oracle 11g Installation Using RunInstaller On Linux : Part IV

Oracle 11g Installation Using RunInstaller On Linux : Part III

Oracle 11g Installation Using RunInstaller On Linux : Part II

Oracle 11g Installation Using RunInstaller On Linux : Part I

Delete a Oracle DB using DBCA In Silent Mode

Make Sure You Stop The DB "ORCL" Using > shutdown command.

How To Manually Reboot The HMC

Below are the pictures that would show how to reboot the HMC Manually.

How To Find The Users With Empty Password

/etc/shadow is the file which we look for password information.

/etc/shadow content:
===============

scott:!!:16236:0:99999:7:::
peter:!!:16236:0:99999:7:::
kevin:!!:16236:0:99999:7:::

This is actual content for a default user/new user.

[root@node1 /]# passwd -d scott
Removing password for user scott.
passwd: Success
[root@node1 /]# passwd -d peter
Removing password for user peter.
passwd: Success
[root@node1 /]# passwd -d kevin
Removing password for user kevin.
passwd: Success

scott::16236:0:99999:7:::
peter::16236:0:99999:7:::
kevin::16236:0:99999:7:::

The above is the content after remving the password for the user using the command # passwd -d

[root@node1 /]# cat /etc/shadow | awk -F: '($2==""){print $1}'
scott
peter
kevin
[root@node1 /]#

Now the above script will give the list of users with EMPTY PASSWORD.

pam_tally2 - The login counter (tallying) module

I have set up PAM Authentication to lock a user accoount after 3 attempts of incorrect password.

PAM Module: pam_tally2.so

password-auth-ac:
=============

auth        required      pam_tally2.so deny=3 file=/var/log/tallylog

account     required      pam_tally2.so

[root@node1 /]# pam_tally2 --user test

Login           Failures Latest failure     From

test                0

[root@node1 /]# 

login as: test

test@192.168.183.148's password:

Access denied

test@192.168.183.148's password:

Access denied

test@192.168.183.148's password:

Access denied

test@192.168.183.148's password:

Access denied

test@192.168.183.148's password:

Account locked due to 4 failed logins

Account locked due to 5 failed logins

Account locked due to 6 failed logins

Account locked due to 7 failed logins

Last login: Sun Jun 15 00:14:30 2014 from 192.168.183.1

[test@node1 ~]$

[root@node1 /]# pam_tally2 --user=test

Login           Failures Latest failure     From

test                6    06/15/14 00:23:20  192.168.183.1

[root@node1 /]# pam_tally2 --user=test --reset

Login           Failures Latest failure     From

test                6    06/15/14 00:23:20  192.168.183.1

[root@node1 /]# pam_tally2 --user=test

Login           Failures Latest failure     From

test                0

[root@node1 /]#

login as: test

test@192.168.183.148's password:

Last login: Sun Jun 15 00:20:07 2014 from 192.168.183.1

[test@node1 ~]$ whoami

test

[test@node1 ~]$

Locate Memory Slots In AIX

Using the below commands we can find out the memory slots in AIX.

                     #lscfg -vp |grep -p DIMM

Output:

#lscfg -vp |grep -p DIMM

Memory DIMM:
  Record Name.................VINI
  Flag Field..................XXMS
  Hardware Location Code......U787B.001.DNWG069-P1-C9-C1
  Customer Card ID Number.....312D
  Serial Number...............YH10MM53M07F
  Part Number.................15R7170
  FRU Number.................. 15R7170
  Size........................2048
  Version.....................RS6K
Physical Location: U787B.001.DNWG069-P1-C9-C1

Memory DIMM:
  Record Name.................VINI
  Flag Field..................XXMS
  Hardware Location Code......U787B.001.DNWG069-P1-C9-C3
  Customer Card ID Number.....312D
  Serial Number...............YH10MM5CH07D
  Part Number.................15R7170
  FRU Number.................. 15R7170
  Size........................2048
  Version.....................RS6K
Physical Location: U787B.001.DNWG069-P1-C9-C3

Unix Password-less Login

On source server side:

First generate public key using rsa

#ssh-keygen -t rsa    

( here it will ask some questions, please give all default options, especially we have to press enter (leave as empty) in the passphrase option)

Copying the .pub files content from source server to target server.
 scp ~/.ssh/id_rsa.pub username@destination:~/.ssh/authorized_keys

                                                         (OR)

Better we can follow the below steps
 cp ~/.ssh/id_rsa.pub ~/.ssh/authorize_keys

 next copy this authorize_keys file to client /root/.ssh/
  scp ~/.ssh/authorize_keys username@destination:~/.ssh/.

Now client will be able to login without password.

Linux Buffer Cache Clean Up

Every 1.0s: free -m                                                                                                                                                                   total       used       free     shared    buffers     cached
Mem:          1998        631       1367          0          2        423
-/+ buffers/cache:        205       1792
Swap:         3999          0       3999

echo 3 > /proc/sys/vm/drop_caches --> Command to clear buffer cache.

Every 1.0s: free -m      

         total       used       free     shared    buffers     cached

Mem:          1998        629       1369          0          0        422

-/+ buffers/cache:        206       1792

Swap:         3999          0       3999

Mem:   2046792k total,   652248k used,  1394544k free,     1536k buffers

Swap:  4095992k total,        0k used,  4095992k free,   438744k cached

echo 3 > /proc/sys/vm/drop_caches

Mem:   2046792k total,   648288k used,  1398504k free,      348k buffers

Swap:  4095992k total,        0k used,  4095992k free,   432796k cached

ORA-01078: failure in processing system parameters

This happens very rarely, when you are not able startup the DB.

SQL> startup;
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/11.2.0/db_1/dbs/initTest.ora'
SQL>

The issue is the Oracle could not fine the Pfile or SPfile to startup the DB.

So, if both the files are deleted then restore the file backup and start the DBA.

I just restored,

SQL> startup;
ORACLE instance started.

Total System Global Area  730714112 bytes
Fixed Size                  2216944 bytes
Variable Size             478153744 bytes
Database Buffers          247463936 bytes
Redo Buffers                2879488 bytes
Database mounted.
Database opened.
SQL> select name from v$database;

NAME
---------
TEST

SQL>

SQL> show parameter spfile;

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

spfile                               string      /u01/app/oracle/product/11.2.0

                                                 /db_1/dbs/spfileTest.ora

SQL>

And it just got up from SPfile.

What happens when your SPFILE is missing while starting the DB ?

I have a database called "Test".

The Test DB has its Pfile and SPfile.

Pfile is configured with the Memory of 600MB.
SPfile is configured with the Memory of 700MB.

I have renamed my SPfile.

[oracle@node1 dbs]$ mv spfileTest.ora spfileTest.ora_ORIG
[oracle@node1 dbs]$ ls -l spfileTest.ora*
-rw-r----- 1 oracle oinstall 2560 Jun 14 20:46 spfileTest.ora_ORIG
[oracle@node1 dbs]$

Shutting down and starting the DB:

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup

ORACLE instance started.

Total System Global Area  626327552 bytes

Fixed Size                  2215944 bytes

Variable Size             373297144 bytes

Database Buffers          247463936 bytes

Redo Buffers                3350528 bytes

Database mounted.

Database opened.

SQL> show parameter spfile;

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

spfile                               string

SQL>

SQL> select name from v$database;

NAME

---------

TEST

SQL> sho parameter sga_max_size;

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

sga_max_size                         big integer 600M

SQL>

Oracle DBA Startup Time

SQL> select INSTANCE_NAME,DATABASE_STATUS,STARTUP_TIME,STATUS from v$instance;

INSTANCE_NAME  DATABASE_STATU STARTUP_TIME   STATUS
-------------- -------------- -------------- --------------
Test           ACTIVE         14-JUN-14      OPEN

SQL>

ORA-32001: write to SPFILE requested but no SPFILE is in use

The possible issue could be the DB Started from the PFile or the SPFILE is missing.

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /u01/app/oracle/product/11.2.0
                                                 /db_1/dbs/spfileTest.ora
SQL>

Command to verify whether the DB started from Pfile or SPFile.

ORA-00845: MEMORY_TARGET not supported on this system

When you get the message ORA-00845: MEMORY_TARGET not supported on this system while starting the DB.

It means you are running short of memory for that instance.

1) Increase RAM on your machine.

2) Else rework on the memory assigned for your database.

3) Else stop other database and start the needed one.

Here I get a error while starting a DB called "ORCL"

SQL> startup;
ORA-00845: MEMORY_TARGET not supported on this system
SQL>

Since I already have a DB called "TEST". I am stopping it and starting the "ORCL".

SQL> select name from v$database;

NAME
---------
TEST

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>

SQL> startup;

ORACLE instance started.

Total System Global Area  835104768 bytes

Fixed Size                  2217952 bytes

Variable Size             536872992 bytes

Database Buffers          293601280 bytes

Redo Buffers                2412544 bytes

Database mounted.

Database opened.

SQL> select name from v$database;

NAME

---------

ORCL

SQL>

Oracle Listener TNS-12541: TNS:no listener

[oracle@node1 ~]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 14-JUN-2014 08:29:55

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=node1.oracle.com)(PORT=1521)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
[oracle@node1 ~]$

Solution :

Start the listener.

[oracle@node1 ~]$ lsnrctl start

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 14-JUN-2014 08:30:12

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/11.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.1.0 - Production
System parameter file is /u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/node1/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=node1.oracle.com)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                14-JUN-2014 08:30:14
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/node1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=node1.oracle.com)(PORT=1521)))
The listener supports no services
The command completed successfully
[oracle@node1 ~]$