After upgrade from cloudera 5.3.3 to 5.4.3 hive cli is throwing the below error

Error:

ERROR] Terminal initialization failed; falling back to unsupported

java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected

                at jline.TerminalFactory.create(TerminalFactory.java:101)

                at jline.TerminalFactory.get(TerminalFactory.java:158)

                at org.apache.hive.beeline.BeeLineOpts.<init>(BeeLineOpts.java:73)

                at org.apache.hive.beeline.BeeLine.<init>(BeeLine.java:117)

                at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:469)

                at org.apache.hive.beeline.BeeLine.main(BeeLine.java:453)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

                at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected

                at org.apache.hive.beeline.BeeLineOpts.<init>(BeeLineOpts.java:101)

                at org.apache.hive.beeline.BeeLine.<init>(BeeLine.java:117)

                at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:469)

                at org.apache.hive.beeline.BeeLine.main(BeeLine.java:453)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

                at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 

Reason:  Hive has upgraded to Jline2 but jline 0.94 exists in the Hadoop lib.

Resolution:  Delete jline from the Hadoop lib directory

Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.

Error Message :

15/06/18 16:00:29 ERROR source.SpoolDirectorySource: FATAL: Spool Directory source xxxxxx: { spoolDir: /root/Documents/FlumeWorkSpace/amsInput/ }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.

java.lang.NullPointerException

        at com.interceptor.AMSLogParserInterceptor.intercept(AMSLogParserInterceptor.java:34)

        at com.interceptor.AMSLogParserInterceptor.intercept(AMSLogParserInterceptor.java:52)

        at org.apache.flume.interceptor.InterceptorChain.intercept(InterceptorChain.java:62)

        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:146)

        at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:236)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Reason :

·         Because of corrupted file which you loading, problem with your custom interceptor written by you.

·         Duplicate files loading

RHEL 7 Installation

1. Select Install Red Hat Enterprise Linux 7.0

2. Choose Language

3. Select Time Zone

4. Select software as per the requirement

5. Device selection :  Here we have two options 1. automatic   2. Manual

Here i am going with automatic , select the drive and click on DONE

6. Provide Root password

Click on DONE.

Click on Reboot

Click on Finish

IP V6 Configuration in RHEL 6

Append the below lines in /etc/sysconfig/network

Similarly append the below lines in /etc/sysconfig/network-scripts/ifcfg-eth0 

Restart the network service using service network restart

Verify the IP address using ifconfig

Ping the new IP using ping6 command :

Njoy :)

device-eth0-does-not-seem-to-be-present,delaying initialization

Error:  "device-eth0-does-not-seem-to-be-present"

Cause : 

when I cloned one of my Linux virtual machine. So the cloned machine was not able to detect the NIC and every time I tried to restart the network it was throwing this error

Solution: 

The MAC id details for the NIC card is stored in the below mentioned file which is automatically created every time the machine boots.

                     /etc/udev/rules.d/70-persistent-net.rules

If you open this file the contents would be like this

Delete this file

           rm -f /etc/udev/rules.d/70-persistent-net.rules

Reboot the machine i.e  init 6

Now we can see difference in MAC id and NIC card name

Copy the new MAC id and NIC card name to your ifcfg-eth0 file

vi /etc/sysconfig/network-scripts/ifcfg-eth0

Restart the Network service using :  service network restart

check for IP details using " ifconfig"

openstack swift [ Object Storage]

• OpenStack Swift, also known as OpenStack Object Storage, is an open source object storage system that is licensed under the Apache 2.0 license and runs on standard server hardware.

• OpenStack Swift is best suited to backup and archive unstructured data, such as documents, images, audio and video files, email and virtual machine images.

• Objects and files are written to multiple drives, and the Swift software ensures the data is replicated across a server cluster. By default, Swift places three copies of every object in as unique-as-possible locations -- first by region, then by zone, server and drive. 

• If a server or hard drive fails, OpenStack Object Storage replicates its content from active nodes to new locations in the cluster. 

• The system, which is accessed through a REST HTTP application programming interface (API), can scale horizontally to store petabytes of data through the  addition of nodes, which typically equate to servers. OpenStack Swift software is based on Cloud Files technology developed by Rackspace Hosting Inc. 

• Rackspace and NASA initiated the project and co-founded the community that develops and maintains OpenStack software, which includes compute, storage and networking components for building cloud computing services.

 Swift Components:
 =================

 1.Proxy Server
Tie together the Swift architecture
Request routing
Exposes the public API

 2.Ring
Maps names to entities (accounts, containers, objects) on disk.
Stores data based on zones, devices, partitions, and replicas
Weights can be used to balance the distribution of partitions
Used by the Proxy Server for many background processes
 3.Object Server
Blob storage server
metadata kept in xattrs
data in binary format
Object location based on name & timestamp hash

Swift & Large Object Storage:
==============================

Default 5GB limit on the size of an uploaded object
Segmentation makes download size of a single object is virtually unlimited
Segments large object are uploaded and a special manifest file is created when downloaded, all segments are concatenated as a single object.
Greater upload speed
Possible parallel uploads of segments.

Swift Components:
=================

Replication
       Keep the system consistent, handle failures

Updaters
       Process failed or queued updates

Auditors
      Verify integrity of objects, containers, and accounts

Container Server:
      Handles listing of objects, stores as SQLite DB

Account Server:
       Handles listing of containers, stores as SQLite DB

Limitation:
===========
Search is limited to queries based on the object's name and to a single bucket. No metadata or content-based search capabilities are provided.

Kerberos Basics

Kerberos is a network authentication protocol and it is built on the assumption that network connections are unreliable. Kerberos uses secret-key cryptography to enable strong authentication by providing user-to-server authentication.

ü  A realm establishes an authentication administrative domain. Each realm has its own Kerberos database which contains the users and services for that particular administrative domain.

ü  Principals are the entries in the Kerberos database. Each user, host or service is given a principal.

ü  Tickets are issued by the authentication server. Clients present tickets to the application server to demonstrate the authenticity of their identity. Each ticket has an expiration and a renewal time. The Kerberos server has no control over the issued tickets, so even if we prevent a user from obtaining a ticket, if the user has already a valid ticket, he/she can use this to contact the service (until the ticket expires).

ü  Keytabs stores long-term keys for one or more principals.

A Kerberos server, usually called Key Distribution Center (KDC) resides on one physical host, but logically has multiple components incorporated:

ü  Database - contains the user and service entries (user's principal, maximum validity, maximum renewal time, password expiration, etc.)

ü  Authentication Server (AS) - replies to the authentication requests from the client, when the not yet authenticated user must insert a password. The AS sends a Ticket Granting Ticket (TGT) back which can be used further on by the user, without re-entering their password.

ü  Ticket Granting Server(TGS) - distributes service tickets based on the TGT

To access a service using Kerberos a client must do the following:

1.       Authenticate to the Kerberos Authentication Server and receive a Ticket Granting Ticket (TGT)

2.       Request a service ticket from the Ticket Granting Server

3.       Use the service ticket to authenticate to the server that is providing the service the client wants to use (in our case: HDFS, MapReduce, HBase, etc.)

The following picture shows the steps mentioned above.