Changing Log Files Location In RHEL

Log files often help System Administrators in finding out important information about System related issues.

In RHEL there are a lot of log files under /var/log like messages ,maillog,faillog,cups and cron etc.

Here we will discuss how can we decide the location of log files and forward appropriate messages to these locations.

In RHEL /etc/syslog.conf is the file responsible for deciding log file location.Open this file and you’ll see it. syslogd is the daemon associated with it and after every change you made to this file you need to restart this daemon.

Now you can make entry in this file as we have done in following examples:

mail.err   /var/log/mailerr will redirect mail error messages to /var/log/mailerr.
*.crit    /var/log/critical will redirect all critical messages to /var/log/critical.
kern.* @viny.example.com   will send all kernel messages to a remote host viny.example.com

The first part in this example is known as Facility and the second is called Priority. Following are some important facilities you can use:

authpriv-Security/authorization messages
cron
kern-kernel messages
lpr-Line printer messages
mail-Mail messages
syslog-Syslog internal messages

Lock User Accounts After Failed Login

Open /etc/pam.d/system-auth file.

Now write these lines just above the line stating auth sufficient pam_unix.so:

auth required pam_tally2.so onerr=fail deny=3

account  required   pam_tally2.so reset

Now the account will be locked after three failed login attempts.

There are various other options you can use like:

unlock_time=100 will unlock the account after 100 seconds.

You can use any value for unlock_time field.

If you don’t use this parameter then you’ll have to manually unlock a locked account using
# pam_tally2 -r -u username command.

lock_time=100 will lock the account for 100 seconds after failed attempts to login.

no_magic_root will avoid locking root account.

VCS Points

Service Group: Collection of dependent Resources
Resource: Anything that the end user requires
Resource Type: Collection of the resources with same type
Agents: To manage the Resource Types (Start,Stop and Monitor)
Service Group Online: Child Resource to Parent Resource
Service Group Offline: Parent Resource to Child Resource

LLT Files
/etc/llthosts
/etc/llttab

GAB Files:
/etc/gabtab

Manipulating Service Groups:
1. hagrp -offline AppSG -sys S1 -localclus --> Offline the AppSG only in S1 system (node)
2. hagrp -offline OracleSG -any --> Offline the OracleSG in all the systems
3. hagrp -online AppSG -sys S2 -localclus --> Online the AppSG in node S2
4. hagrp -switch AppSG -to S1  -->  AppSG will be moved to node S1

Manipulating Resources:
1. hares -offline Oralistener -sys S3  -->  Bring offline the Oralistener resource in node S3
2. hares -online ipres -sys S2  ->  Bring online the ipres resource in node S2

Handling VCS services:
haconf -dump -makero --> sync the RAM's main.cf with hardisk's main.cf and make the status as Readonly

hastop -all --> Stop the application and cluster

hastop -all -force --> Application will be continue running but the cluster service has been stopped
hastop -local --> stop the cluster service in local node

Useful VCS Commands

SERVICE GROUPS AND RESOURCE OPERATIONS:
Configuring service groups
hagrp –add|-delete|-online|-offline group_name

Modifying resources
hares –add|-delete res_name type group
hares –online|-offline res_name –sys system_name

Modifying agents
haagent –start|-stop agent_name –sys system_name

BASIC CONFIGURATION OPERATIONS:
Service Goups
hagrp -modify group_name attribute_name value
hagrp –list group_name
hagrp –value attribute_name

Resources
hares -modify res_name attribute_name value
hares -link res_name res_name


Agents
haagent -display agent_name –sys system_name
hatype –modify

VCS ENGINE OPERATIONS:
Starting had
hastart –force|–stale system_name
hasys –force system_name

Stopping had
hastop –local|-all|-force|-evacuate
hastop –sys system_name

Adding Users
hauser –add user_name

STATUS AND VERIFICATION:
Group Status/Verification
hagrp -display group_name|–state|–resource group_name

Resources Status/Verification
hares -display res_name
hares –list
hares -probe res_name –sys system_name

Agents Status/Verification
haagent –list
haagent -display agent_name –sys system_name
ps –ef|grep agent_name

VCS Status
hastatus –group
LLT Status/Verification
lltconfig –a list
lltstat|lltshow|lltdump

GAB Status/Verification
gabconfig –a
gabdiskhb –l

COMMUNICATION:
Starting and Stopping LLT
lltconfig -U
lltconfig -c
lltconfig -a list

Starting and Stopping GAB
gabconfig –c –n #seed number (eg: gabconfig -c -n 2)
gabconfig –U

Administering Group Services
hagrp –clear|-flush|-switch group_name –sys system_name

Administering Resources
hares –clear|-probe res_name –sys system_name

Administering Agents
haagent -list
haagent -display agent_name –sys system_name

Verify Configuration
hacf –verify

VCS Troubleshooting

Linux Log Files

/var/log/messages : General message and system related stuff
/var/log/auth.log : Authenication logs
/var/log/kern.log : Kernel logs
/var/log/cron.log : Crond logs (cron job)
/var/log/maillog : Mail server logs
/var/log/qmail/ : Qmail log directory (more files inside this directory)
/var/log/httpd/ : Apache access and error logs directory
/var/log/lighttpd/ : Lighttpd access and error logs directory
/var/log/boot.log : System boot log
/var/log/mysqld.log : MySQL database server log file
/var/log/secure or /var/log/auth.log : Authentication log
/var/log/utmp or /var/log/wtmp : Login records file
/var/log/yum.log : Yum command log file.

LLT Port Information

[root@vcs1 /]# lltstat -p
LLT port information:
    Port    Usage        Cookie
      0     GAB_Control  0x0
          opens:     0 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
      7     gab          0x7
          opens:     0 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
     31     gab          0x1F
          opens:     0 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
[root@vcs1 /]#


[root@vcs2 /]# lltstat -p
LLT port information:
    Port    Usage        Cookie
      0     GAB_Control  0x0
          opens:     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
      7     gab          0x7
          opens:     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
     31     gab          0x1F
          opens:     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
          connects:  0 1
[root@vcs2 /]#

So the LLT is configured on 3 links running on the port 0, 7 and 31.

Add a link to LLT

[root@vcs1 /]# lltconfig -t eth2 -d eth2
LLT lltconfig INFO V-14-2-15653 No link type is specified, defaulting to link type "ether"
[root@vcs1 /]# lltconfig -a list
Link 0 (eth0):
  Node   0 vcs1       :   00:0C:29:DE:BA:35  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:FB

Link 1 (eth1):
  Node   0 vcs1       :   00:0C:29:DE:BA:3F  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:05

Link 2 (eth2):
  Node   0 vcs1       :   00:0C:29:DE:BA:49  permanent

[root@vcs1 /]#

[root@vcs2 /]# lltconfig -t eth2 -d eth2
LLT lltconfig INFO V-14-2-15653 No link type is specified, defaulting to link type "ether"
[root@vcs2 /]#

[root@vcs1 /]# lltconfig -a list
Link 0 (eth0):
  Node   0 vcs1       :   00:0C:29:DE:BA:35  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:FB

Link 1 (eth1):
  Node   0 vcs1       :   00:0C:29:DE:BA:3F  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:05

Link 2 (eth2):
  Node   0 vcs1       :   00:0C:29:DE:BA:49  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:0F

[root@vcs1 /]#

Removing a link from LLT

[root@vcs1 /]# lltconfig -u eth2----------> Command to remove a link from LLT Layer. (Need to execute the command on both the nodes)

[root@vcs2 /]# lltconfig -u eth2

This command unconfigures the LLT link, but does not permanently remove the link from configuration files.

Upon reboot, or restart of LLT, the cluster node will use the link again.

To remove the link permanently from the configuration, on each cluster node modify the /etc/llttab file and remove the line for device "eth2".

[root@vcs1 /]# lltconfig -a list
Link 0 (eth0):
  Node   0 vcs1       :   00:0C:29:DE:BA:35  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:FB

Link 1 (eth1):
  Node   0 vcs1       :   00:0C:29:DE:BA:3F  permanent
  Node   1 vcs2       :   00:0C:29:52:9B:05

[root@vcs1 /]#


LLT node information:
    Node                 State    Link  Status  Address
   * 0 vcs1              OPEN
                                  eth0   UP      00:0C:29:DE:BA:35
                                  eth1   UP      00:0C:29:DE:BA:3F
     1 vcs2              OPEN
                                  eth0   UP      00:0C:29:52:9B:FB
                                  eth1   UP      00:0C:29:52:9B:05


[root@vcs1 /]# lltstat -l
LLT link information:
link 0  eth0 on ether hipri
        mtu 1500, sap 0xcafe, broadcast FF:FF:FF:FF:FF:FF, addrlen 6
        txpkts 3429  txbytes 370955
        rxpkts 3180  rxbytes 232432
        latehb 0  badcksum 0  errors 0
link 1  eth1 on ether hipri
        mtu 1500, sap 0xcafe, broadcast FF:FF:FF:FF:FF:FF, addrlen 6
        txpkts 3450  txbytes 349877
        rxpkts 3216  rxbytes 227532
        latehb 0  badcksum 0  errors 0
[root@vcs1 /]#

NTP Parameters

node1:/var/log # ntpdc -p (uses sec instead of ms).

     remote           local      st poll reach  delay   offset    disp
=======================================================================
=LOCAL(0)        127.0.0.1       10   64  377 0.00000  0.000000 0.03053
*circle.net          16.88.11.29   2  512  377 0.06363 -0.017075 0.11371
=netservice       16.88.11.29  16 1024    0 0.00000  0.000000 3.99217
=ntp1.iso           16.88.11.29   3  512  377 0.11404 -0.010279 0.10500
node1:/var/log #

“+” – Good and a preferred remote peer or server (included by the combine algorithm)

“*” – The remote peer or server presently used as the primary reference;

poll : NTP polls each peer every “poll” seconds.

when : is relative time of last poll.

reach : track of last successful polls, 377 is best.

Delay : is network delay, this is fine

offset : offset between local and peer clock (ms) and it should be low.

jitter : Mean deviation (jitter) in the time reported for that remote peer or server.

How to configure NTP Server In Linux

Configured NTP Server in 192.168.100.22

#Server Side
============
NTP Server Pool Address
1.server 1.asia.pool.ntp.org
2.server 0.asia.pool.ntp.org

#Permitting local network to use this
restrict 192.168.100.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.20.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.30.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.40.0 mask 255.255.255.0 nomodify notrap

#full access to the local net
restrict 127.0.0.1

#Client Side
===========

#yum install ntp

edit the ntp.conf file
#vim /etc/ntp.conf

#comment the following lines
#server 0.centos.pool.ntp.org
#server 1.centos.pool.ntp.org
#server 2.centos.pool.ntp.org

#add the following lines
restrict default ignore
server 192.168.100.22
restrict 192.168.100.1 mask 255.255.255.255 nomodify notrap noquery

#save and quit the file.

#Before starting the ntp daemon, run the following command

#ntpdate -b 192.168.100.223

#service ntpd start

#check the ntp

[root@node1 ~]# ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================
192.168.100.223 140.109.1.4 11 u 3 64 1 0.257 0.368 0.001
127.127.1.0 .LOCL. 10 l 2 64 1 0.000 0.000 0.001
[root@node1 ~]#

/dev/sdb: read failed after 0 of 4096 at 0: Input/output error

When a storage unit is longer available to the Linux Server, You would see the following error message:

[root@node1 /]# pvs
  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 2147418112: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 2147475456: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
  PV                VG       Fmt  Attr PSize  PFree
  /dev/mapper/disk1          lvm2 a--   1.00g 1.00g
  /dev/mapper/disk2          lvm2 a--   1.00g 1.00g
  /dev/mapper/disk3          lvm2 a--   1.00g 1.00g
  /dev/sda2         vg_node1 lvm2 a--  19.51g    0
[root@node1 /]#

To fix the issue, We need to delete the corresponding /dev/sdb file from OS Level.

[root@node1 /]# echo 1 > /sys/block/sdb/device/delete

[root@node1 /]# pvs
  PV                VG       Fmt  Attr PSize  PFree
  /dev/mapper/disk1          lvm2 a--   1.00g 1.00g
  /dev/mapper/disk2          lvm2 a--   1.00g 1.00g
  /dev/mapper/disk3          lvm2 a--   1.00g 1.00g
  /dev/sda2         vg_node1 lvm2 a--  19.51g    0
[root@node1 /]#

VxVM vxdisk ERROR V-5-1-684 IPC failure: Configuration daemon is not accessible

[root@node2 yum.repos.d]# cd /

[root@node2 /]# vxdisk list
VxVM vxdisk ERROR V-5-1-684 IPC failure: Configuration daemon is not accessible

[root@node2 /]# cd /etc/vx/reconfig.d/state.d/

[root@node2 state.d]# ls -l | grep -i install-db

-rw-r--r--. 1 root root 0 Sep  1 06:21 install-db

[root@node2 state.d]# rm -rf install-db

[root@node2 state.d]# vxconfigd

[root@node2 state.d]# vxdctl init

[root@node2 state.d]# vxdctl enable

[root@node2 state.d]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
sda          auto:LVM        -            -            online invalid
[root@node2 state.d]#

initramfs (RHEL 6)

dracut utility can be used in RHEL 6 to rebuild the initrams image ,It  creates  an initial image used by the kernel for preloading the block device modules (such as IDE, SCSI or RAID) which  are  needed  to access the root filesystem.


1.       It is recommended you make a backup copy of the initrd in case the new version has an unexpected problem:

# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak


2      Now rebuild the initramfs for the current kernel version:

# dracut -f

Note:

If you are in a kernel version different to the initrd we are building (including if you are in Rescue Mode). We must specify the full kernel version, including architecture:


# dracut -f initramfs-2.6.32-71.el6.x86_64.img 2.6.32-71.el6.x86_64

The only purpose of an initramfs is to mount the root filesystem.

The initramfs is a complete set of directories that you would find on a normal root filesystem.

At boot time, the boot loader loads the kernel and the initramfs image into memory and starts the kernel. 

The kernel checks for the presence of the initramfs and, if found, mounts it as / and runs /init. 

The init program is typically a shell script. Note that the boot process takes longer, possibly significantly longer, if an initramfs is used. 

Kdump Configuration

Kump Introduction

Kdump is the Linux kernel crash-dump mechanism. Oracle recommends that you enable the Kdump feature. In the event of a system crash, Kdump creates a memory image (vmcore) that can help in determining the cause of the crash. Enabling Kdump requires you to reserve a portion of system memory for exclusive use by Kdump. This memory is unavailable for other uses.

Kdump uses kexec to boot into a second kernel whenever the system crashes. kexec is a fast-boot mechanism which allows a Linux kernel to boot from inside the context of a kernel that is already running without passing through the bootloader stage.


When the kdump crash dumping mechanism is enabled, the system is booted from the context of another kernel. This second kernel reserves a small amount of memory and its only purpose is to capture the core dump image in case the system crashes.

Configuring the kdump Service:
There are three common means of configuring the kdump service: at the first boot, using the Kernel Dump Configuration graphical utility, and doing so manually on the command line.
Configuring the kdump at First Boot: The menu which we see during OS Installation.

Using the Kernel Dump Configuration Utility:

Select System → Administration → Kernel crash dumps from the panel, or type system-config-kdump at a shell prompt

Configuring kdump on the Command Line:
To configure the amount of memory to be reserved for the kdump kernel, edit the /boot/grub/grub.conf file and add crash kernel=M or crash kernel=auto

How to go previous date from current date

[root@node1 ~]# date
Sun Jul 13 05:29:50 EDT 2014

[root@node1 ~]# date -d "30 Days Ago"
Fri Jun 13 05:33:45 EDT 2014

Who Can Fix This YUM Issue ?

I am getting the above error upon giving the yum command.

CD is mounted.

Please find the YUM conf file:

[root@node1 yum.repos.d]# cat rhel-source.repo
[rhel source]
name=Red Hat Enterprise Linux $releasever - $basearch - Source
baseurl=file:///mnt/
enabled=1
gpgcheck=0

[root@node1 yum.repos.d]#

Who Can Fix This Linux Boot Issue.

After reboot, I get the below message in the console.

What is a Mirror Log ?

LVM maintains a small log which it uses to keep track of which regions are in sync with the mirror or mirrors. 

By default, this log is kept on disk, which keeps it persistent across reboots and ensures that the mirror does not need to be re-synced every time a machine reboots or crashes. 

You can specify instead that this log be kept in memory with the --mirrorlog core argument; this eliminates the need for an extra log device, but it requires that the entire mirror be resynchronized at every reboot.