Pig HBase integration on MapR

This procedure helps integrate HBase transactions using Pig on MapR Hadoop clusters.

On the client node where you have installed Pig, in /opt/mapr/conf/env.sh

Add
export PIG_CLASSPATH=$PIG_CLASSPATH:/location-to-hbase-jar

If you are launching PIG on a node where you have hbase-regionserver
or hbase-master installed, then simply add the location of the
hbase-0.92-1.jar to the PIG_CLASSPATH variable above. Eg

export
PIG_CLASSPATH="$PIG_CLASSPATH:/opt/mapr/hbase/hbase-0.92.1/hbase-0.92.1.jar"

If you don't have hbase installed, the HBase jar can be copied over
directly from any node where hbase is installed to some location on
the pig client node. Include the location where you copied it to in
the above definition.Eg
export PIG_CLASSPATH=$PIG_CLASSPATH:/opt/mapr/lib/hbase-0.92.1.jar

Then identify your zookeeper nodes,
maprcli node listzookeepers

and accordingly add this variable to /opt/mapr/conf/env.sh

export PIG_OPTS="-Dhbase.zookeeper.property.clientPort=5181
-Dhbase.zookeeper.quorum=10.10.80.61,10.10.80.62,10.10.80.63"

Launch the pig job and you should be able to access HBase.
NB: Use only the HBase table name directly for accessing tables. Do not use
hbase:// prefixes. Eg script

Sample env.sh

[root@nmk-centos-60-3 ~]# cat /opt/mapr/conf/env.sh
#!/bin/bash
# Copyright (c) 2009 & onwards. MapR Tech, Inc., All rights reserved
# Please set all environment variable you want to be used during MapR cluster
# runtime here.
# namely MAPR_HOME, JAVA_HOME, MAPR_SUBNETS

export PIG_OPTS="-Dhbase.zookeeper.property.clientPort=5181
-Dhbase.zookeeper.quorum=10.10.80.61,10.10.80.62,10.10.80.63"
export
PIG_CLASSPATH="$PIG_CLASSPATH:/opt/mapr/hbase/hbase-0.92.1/conf:/usr/java/default/lib/tools.jar:/opt/mapr/hbase/hbase-0.92.1:/opt/mapr/hbase/hbase-0.92.1/hbase-0.92.1.jar"
export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:$PIG_CLASSPATH"
export CLASSPATH="$CLASSPATH:$HADOOP_CLASSPATH"
#export JAVA_HOME=
#export MAPR_SUBNETS=
#export MAPR_HOME=
#export MAPR_ULIMIT_U=
#export MAPR_ULIMIT_N=
#export MAPR_SYSCTL_SOMAXCONN=
#export PIG_CLASSPATH=:$PIG_CLASSPATH
[root@nmk-centos-60-3 ~]#

Sample hbase insertion script

[root@nmk-centos-60-3 nabeel]# cat hbase_pig.pig
raw_data = LOAD '/user/mapr/input2.csv' USING PigStorage(',') AS (
listing_id: chararray,
fname: chararray,
lname: chararray );

STORE raw_data INTO 'sample_names' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage (
'info:fname info:lname');

Awk display / print file from column N onwards

This can be piped to any output to display the lines from the second column onwards.

awk -F: -v nr=N '{ for (x=nr; x<=NF; x++) {printf $x " "};print " ";}'

An example usage, find the first disk re-inited by MapRFS this month.

find . -name mfs.log -exec grep -H ^2012\-10\-* {} \; | grep spinit.cc:1002 | awk -F: -v nr=2 '{ for (x=nr; x<=NF; x++) \
{printf $x " "};print " ";}' | awk -v nr=2 '{ for (x=nr; x<=NF; x++) {printf $x " "};print " ";}' | sort -n | head -n 5

UPDATE:

Just realized cut could do this in a much simpler wa

cut -d : -f N-

where N is the column from where you need to display the lines.

Finding the culprit in df disk utilization different from du

Sometimes df and du show different outputs due to the open file descriptor issue. That's explained in detail all over google, so I'm not touching that here. Here's a quick command to find the culprit that makes the difference. This shows the top 5 files that consume around 1GB of space. You can adjust the number after the digit regex to change that. Keep in mind, occasionally temp files have names that can run into digits, so that will return in the grep results, but the sort should be accurately on the filesize columns.

lsof -n -P | grep -E [[:digit:]]{10} | sort -k 7 | head -n 5

"java.io.IOException: Could not create FileClient"

"java.io.IOException: Could not create FileClient"

If you see this error in the jobtracker logs for a failed job, you are most likely looking at an irrelevant error. To be clearer, your job failed before this point and was subsequently killed by the jobtracker. Some processes that linger in memory after the job was killed keeps attempting to access some of the resources assigned to them and this continues till they time out and eventually get killed. The resources, for eg. the file or parent directory etc. would have been removed as part of killing the job by the job tracker, hence the stray process would not be able to create any new files.

HBase fixing region missing Region //regionname// on HDFS but not listed in .META or deployed on any server

If you see this error on HBase

Region //regionname// on HDFS but not listed in .META or deployed on any server

then use the add_regions.rb script to have this fixed. 

Normally hbase errors can be fixed by hbase hbck -fix
But this approach works mostly in cases where the region is listed in .META but not assigned etc.

In such cases, the script add_regions.rb comes to our rescue. It can be used as follows:

hbase org.jruby.Main add_regions.rb //full-path-region-on-HDFS/.regioninfo

This will add the region to the .META. table. Next assign the region to a region-server. To do this , launch hbase shell and issue the command

assign 'full-region-name'

on the hbase shell.

The full region-name includes the standard format tablename.startrowkey.timestamp.md5sum.

The dot at the end is important and should be included while passing to the assign command.

Now run hbase hbck again and look at status 'simple' on the shell. Your region numbers should have increased according to the regions you have added now. The errors related to not listed in .META should be gone now.

Python copy absolute value of array elements into another array

In the array below, some values are positive, while others are negative.

>>> x = [-538,-181,-145,552,-847,6,141,-58,-122,314,-816,245,594,-613,-287,1232,-1479,-326,-197,715,4,-677,95,308,-1224,953,-81,-189,341,-654,242,-948,1088,-533,-328,123,552,-855,49,-443,-37,-57,199,56,-459,-47,-167,13,-521,476,-161,440,-540,180,43,-57,-236,-29,-830,265,-2,-379,-9,198,12,79,-257,113]
>>>


I want a sum total of the array after conversion to their absolute values.

>>> values = [abs(x[n]) for n in range(1,len(x))]
>>> values
[181, 145, 552, 847, 6, 141, 58, 122, 314, 816, 245, 594, 613, 287, 1232, 1479, 326, 197, 715, 4, 677, 95, 308, 1224, 953, 81, 189, 341, 654, 242, 948, 1088, 533, 328, 123, 552, 855, 49, 443, 37, 57, 199, 56, 459, 47, 167, 13, 521, 476, 161, 440, 540, 180, 43, 57, 236, 29, 830, 265, 2, 379, 9, 198, 12, 79, 257, 113]
>>> print sum (values)
24419

Extract each archive into its own directory

Assuming the archives are all bzipped archives with .tbz2

for i in `ls`; do a=`basename $i .tbz2`; mkdir $a; cd $a;tar -xvf ../$i;cd ..; done

Identify user from python mapper in Hadoop

If you are running a mapreduce job using hadoop streaming with python, and want to know the user the job runs as or other parameters in the OS environment on the task tracker node, use

if(envdict.has_key('user_name')):
           user = os.environ['user_name']
            sys.stdout.write('Job runs as username: '+user)

To make sure what variables you have, use this

       envdict = os.environ
       keylist = envdict.keys()
       sys.stdout.write('Variables available'+str(keylist)+'\n')

Moving Hbase heap dump locations

Occasionally HBase can generate heap dumps on OOM and put it in the bin folder (which is default). This can cause quite a lot of issues, if the system partitioning is not prepared for huge gigabyte files in the bin folder for hbase. To move the dump cores to another folder ,

Do this in the file  hbase-ver-conf/hbase-env.sh, add  -XX:HeapDumpPath=/path/to/dump to the line

>> export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError $HBASE_GC"

Restart HBase.

Now you can sleep well without fears of heap dumps filling up the hbase binary partitions :)

Compare list of files to identify files and directories

Say xyz is a file with a list of files and directories. (The files and directories are all on the working system itself)
You need to print only the directories amongst them.

for i in `cat xyz`; do if [ -d $i ] ; then echo $i ;fi; done

will print out only directories among them.

Hp Openview Errors

Last week, I had a few issues with bringing up the HP Openview Operations Manager. The snap-in wouldn't load .It just gave this error.

"The Console is unable to connect to the server. Please make sure that all services including the database are up and running."

Now on the services.msc in Windows, the database services and related services were all up and running. I could also connect to the db using the SQL studio. ( My DB is MS SQL 2005)

Then i tried to run this
C:\Program Files\HP\HP BTO Software\support\OVMigration\Binn\osql -E openview -S DBserverName\InstanceName

That too showed that things were running fine.

Next I tried running the command on the OVO server
vpstat -8

It showed that all services except OvEpMsgActSrv was running fine.

You can also use ovc -status to check the services.

Now this turned out to be failing due to a huge message queue file at this location.
"\Program Files\HP Openview\Data\Shared\tmp\queues"

Once that was dumped, I restarted the services with
vpstat -8 -r

and things were fine again. This is how a good configuration should look.


D:\HP\OM\support\OVMigration\Binn>osql.exe -E openview -s SERVERNAME\DBNAME
1>
2>
3>
4>
5> quit

D:\HP\OM\support\OVMigration\Binn>
D:\HP\OM\support\OVMigration\Binn>ovc -status
ovcd OV Control CORE (5624) Running
opcmsga OVO Message Agent AGENT,EA (2148) Running
opcmsgi OVO Message Interceptor AGENT,EA (6480) Running
opcle OVO Logfile Encapsulator AGENT,EA (428) Running
opcacta OVO Action Agent AGENT,EA (5580) Running
agtrep OV Discovery Agent AGENT,AgtRep (5776) Running
coda OV Performance Core COREXT (6456) Running
opcmona OVO Monitor Agent AGENT,EA (6576) Running
ovconfd OV Config and Deploy COREXT (9720) Running
opcwbemi OVO WMI Interceptor AGENT,EA (5572) Running
ovbbccb OV Communication Broker CORE (4424) Running
ovcs OV Certificate Server SERVER (1524) Running
ovtomcatB OV Tomcat(B) Servlet Container WEB (6128) Running

D:\HP\OM\support\OVMigration\Binn>
D:\HP\OM\support\OVMigration\Binn>vpstat -3
============================================================
running: D:\HP\OM\bin\vpstat.exe
version: A.22.0.56.30 on SERVERNAME
now: Sun May 01 11:01:25 2011
============================================================

============================================================
==================== Services Information ==================
============================================================

Configuration file: D:\HP\OM\data\conf\vpstat\vpstat.conf
Ver: 0.8.10.074
Date: Thu Jun 19 15:40:36 2008
Remark: HP Operation Manager - Performance.

Services on : LOCAL MACHINE

(NT Services) :

OK: All services : SERVICE_ACTIVE


D:\HP\OM\support\OVMigration\Binn>vpstat -8
============================================================
running: D:\HP\OM\bin\vpstat.exe
version: A.22.0.56.30 on UP-OVO01
now: Sun May 01 11:01:28 2011
============================================================

============================================================
==================== Processes Information =================
============================================================

Configuration file: D:\HP\OM\data\conf\vpstat\vpstat.conf
Ver: 0.8.10.074
Date: Thu Jun 19 15:40:36 2008
Remark: HP Operation Manager - Performance.


OK:ALL CRITICAL PROCESSES ARE RUNNING

D:\HP\OM\support\OVMigration\Binn>

PIX and SMTP troubles

Twice in our current environment, we've had to face issues with PIX interfering with very specific SMTP communications. The first time, one of our users could not receive a particular type of pdf attachments from one source. After a hell of troubleshooting, it finally turned out to be that PIX ESMTP filtering was the source of the problem.

This time, there was another similar issue. Some domains with a big list of entries in the "ADDITIONAL SECTION" of the DNS response could not me sent emails. The error being

Delivery Failure
Attempted Delivery to: Unknown  Error: 554 5.4.4 [internal] domain lookup failed

This error though was not correct, since the DNS lookups were successful in the nslookup utility on the messaging gateway.

Digging further in pointed to the issue being related to large DNS query responses which were being rejected by PIX DNS restrictions. DNS UDP Packets of size greater than 512 bytes were not allowed by PIX. So Once that was removed, things started working well.

Autocad network license manager error

I was having a tough time with the following error in many systems on a trusted domain within our network. The Autocad licensing server was on our domain and could be accessed without issues from the client desktop which was on the trusted domain, but AutoCAD wouldn't start. It just kept popping up this error

The network license manager was not able to get a valid license. If this problem continues, please contact your system administrator. Error [1.5.-18]

Turned out that, in the AutoCAD server license file, the server name provided was not accessible from the client desktop since it was not in a FQDN format. When the hosts file was modified to do the name mapping, it worked fine. I then created a host record in the trusted domain with the same name to point to the licensing server and it worked.

Anonymous users in proxy logs

Ever since we installed a new Microsoft TMG server for proxy authentication and web filtering, we've seen random users being denied access for no good reason. The rules have been set to allow access to these users and it works for most of the other users.

The logs seem to show that the users, who are logged in as domain users still show up as anonymous accounts at the TMG server.

In one such instance, this step fixed it.
In IE-> tools-> Internet Options-> Advanced-> Security
Toggle the "Enable integrated windows authentication."

Kernel versions to RHEL versions mapping

2.6.18-53.el5 (RHEL5.1)
2.6.18-92.el5 (RHEL5.2)
2.6.18-128.el5 (RHEL5.3)
2.6.18-164.el5 (RHEL5.4)
2.6.18-194.el5 (RHEL 5.5)

2.6.9-11.EL (RHEL4.1)
2.6.9-42.EL (RHEL4.4)
2.6.9-55.EL (RHEL4.5)
2.6.9-67.EL (RHEL4.6)
2.6.9-78-0.1.EL (RHEL4.7)
2.6.9-89.EL(RHEL4.8)

Update error on Websphere

I was trying to update a WAS 6.0 server on linux with
6.0-WS-WAS-LinuxX32-RP0000002.pak

It completed successfully.

Then with the same IBM Update Installer for WebSphere package(6.0.2.0) I tried to apply the fix pack 6.0.2-WS-WAS-LinuxX32-FP00000013.pak to the server but it kept popping up the message

"Unable to find IBM WebSphere Application Server at a version that is greater than or equal to 6.0.0.0 and less than or equal to 6.0.2.0.".

The issue was that it had to be done with IBM Update Installer for WebSphere 6.0.2.7. So I moved the earlier updateinstaller directory to updateinstaller.old and then untarred the new version, moved the fixpack to the maintenance directory and ran updateInstaller and it worked.

Websphere uninstallation error

Often when we remove the /opt/IBM folder manually to remove a Websphere installation (for reinstalling) this error pops up during the second attempt

Specify a different directory or perform a manual uninstall to remove all packages before reinstalling to the same directory

This happens because the WAS installation is stored in more locations than just the /opt folder. To clean up the system after you accidentally remove the /opt folder,(kill all java processes first)

1) rpm -qa |grep WS
and remove the rpms

2)remove the ~/.WASRegistry file.

3) remove vpd.properties.

Oracle Tablespace Check

Here's a script that checks the tablespace utilization on a database, and sends a mail if it crosses a threshold

#!/bin/bash
# Tablespace check script 
# Author: Nabeel Moidu
# Website: nmkuttiady.blogspot.com
# Set threshold here
THRESHOLD=70
# Set the Oracle user here
ORACLE_USER=ora10g
# Set the mail id of the DBA
DBA="dba@organization.tld"

# Get output of tablespace usage on the database
su - $ORACLE_USER -c "sqlplus -s \"/ as sysdba\" < /backup/script/tablespace_usage.sql > /tmp/output.txt "

# Remove blank lines from file 

cat /tmp/output.txt | grep -v ^$ | grep -v "no rows selected" > tablespaces.txt

# If no tablespace crosses threshold, the file will be empty. So send mail only if file not empty

if test -s "tablespaces.txt"

then

        # Prepare the mail
        echo "The following tablespaces have exceeded the 60 percent threshold" > mail.txt

        #Send the mail with the hostname in subject
        HOST=`hostname`
        SUBJECT="Daily Tablespace Check - "$HOST
        echo $SUBJECT
        mail -s "$SUBJECT" $DBA < mail.txt

fi

This is the SQL query to get the tablespace output

set pages 700
set linesize 1000
select t.tablespace_name,total_MB,round((total_MB-free_MB),2) used_MB, round(free_MB,2) free_MB,round(((total_MB-free_MB)/total_MB)*100,2) per_used,round((free_MB/total_MB)*100,2) per_free from (select tablespace_name,sum(bytes/1024/1024) free_MB from dba_free_space group by tablespace_name) f,(select tablespace_name,sum(bytes/1024/1024) total_MB from dba_data_files group by tablespace_name) t where f.tablespace_name(+)=t.tablespace_name order by 5;

VBScript for Telephone Extension verification by users

This VBScript application identifies a logged on User, retrieves their login name, checks up their telephone number and asks them for a confirmation if its the correct number. If its wrong, it mails the corrected name and number to a particular user and also uploads it to a particular php script. I used this to migrate our users from Cisco Call Manager phone directory to an integrated Active Directory Address book.

Dim objADSysInfo,strUser,objUser,inputPhone,strEngInfo,strEngMsg,strAraInfo,strAraMsg,Result
Dim xmlhttp,httpURL,strDate,strMsg

' Function to display the retrieved information to a user and accept his modification
Function EnterExtension
'Collect AD information from system where the application is executed
Set objADSysInfo = CreateObject("ADSystemInfo")
'Retrieve username of the logged in user
strUser = objADSysInfo.UserName
'Lookup LDAP for other properties of the user
Set objUser = GetObject("LDAP://" & strUser)
'Display the retrieved information to the user
strEngInfo = "Your Name is " & objUser.CN & VbCrLf & VbCrLf & "Telephone Extension is " & objUser.ipPhone & "." & VbCrLf
strEngMsg = " If this is incorrect, enter your extension (only the last four digits Eg. 5159) and click OK. Otherwise click Cancel"
'Accept User's modified extension number
inputPhone=InputBox( strEngInfo & VbCrLf & strEngMsg & VbCrLf & VbCrLf )

End Function

'Function to Process the extension number entered by the user
Function ProcessExtension

'Upload the modified number and useraccount to a php application which can process it
httpURL = "http://servername.domain/TelExtension.php?UName=" + objUser.sAMAccountName + "&UExt=" + inputPhone
Set xmlhttp = CreateObject("Microsoft.XMLHTTP")
xmlhttp.Open "GET", httpURL, false
xmlhttp.Send
strDate = formatDateTime(Now(), vbLongDate)
strMsg = "Your request will be processed by end of " & strDate
MsgBox strMsg,64,"Information Systems Department"

End Function

'Function to trigger an email
Function SendMail

'Launch Outlook
 Set MyApp = CreateObject("Outlook.Application")
 Set MyItem = MyApp.CreateItem(0) 'olMailItem
 With MyItem
 .To = "admin@domain.tld"
' Properly set subject so as to filter out mail at the mail client
 .Subject = "Request for Extension change :" + objUser.sAMAccountName + "," + inputPhone
 .ReadReceiptRequested = False
' Compose body of the Email
 .HTMLBody = objUser.sAMAccountName + " was previously " + objUser.ipPhone + " and needs to be modified to " + inputPhone
' Send the mail
 .Send
 End With


End Function

'Function to Check the validity of extension entered
Function CheckExtension

If IsEmpty(inputPhone) Then
MsgBox strEngInfo & VbCrLf & "Thank You for your cooperation",64,"Information Systems Department"
'This is expected if the user clicks Cancel, i.e. his extension in our records is correct. So Set Flag to 1
Result=1
 If (inputPhone > 4000 And inputPhone < 4199) OR (inputPhone > 5000 And inputPhone < 5999) Then
   ProcessExtension()
   Result=1
  Else
   strMsg = "Please enter your extension again. Enter only the last four digits.For Eg. 5678 " 
   MsgBox strMsg,48,"UPDA Information Systems Department"
   Result=0
  End If
   
 End If

End Function

' Main code Begins here

Result=0

'Run till the user enters a valid number or cancels 
 Do While Result=0
  EnterExtension()
  CheckExtension()
 Loop 

Compare data in two CSV files

[root@nabeelmoidu html]# cat compare.php

 
 
// Input of the format
// field1 in file1 (eg, 2446)
// field2,field1 in file2  (Nabeel, 2446)
 
$filename='file1.csv';
$handle = fopen("$filename", "r");
$ctr=0;
 
while (($read = fgetcsv($handle, 1000, ",")) !== FALSE){
    $num[$ctr]= (int)$read[0];
//    echo $num[$ctr]."
";
        $ctr = $ctr + 1;
}
 
$filename1='file2.csv';
$handle1 = fopen("$filename1", "r");
$ctr1=0;
     while (($rawname = fgetcsv($handle1, 1000, ",")) !== FALSE)
{    $ext[$ctr1]= (int)$rawname[0];
    $name[$ctr1]= $rawname[1];
//    echo $ext[$ctr1]."
";
       $ctr1 = $ctr1 + 1;
}
 
for ($n=0; $n <= $ctr1; $n++) { $match[$n]=1; }
 
for ($x=0; $x <= $ctr; $x++)
    {
    for ($y=0; $y <= $ctr1; $y++) {
            if ($num[$x]==$ext[$y]){$match[$y]=0;}
        else {};
    }
    }
 
for ($y=0; $y <= $ctr1; $y++) {
                if ($match[$y])
                        { echo $name[$y]." |  ".$ext[$y]."
";}
                else {};
    }
 
fclose($handle);
fclose($handle1);
?>