Mani's Tech Blog

Hadoop Components

Node : A node is simply a computer, typically non-enterprise, commodity hardware for nodes that contain data.

Rack :A rack is a collection of 30 or 40 nodes that are physically stored close together and are all connected to the same network switch.
Network bandwidth between any two nodes in rack is greater than bandwidth between two nodes on different racks.

Hadoop has two major components:

- The distributed filesystem component, the main example of which is the Hadoop Distributed FileSystem, though other file systems are supported.

- The MapReduce component, which is a framework for performing calculations on the data in the distributed file system.

HDFS :

-HDFS runs on top of the existing file systems on each node in a Hadoop cluster.

-Hadoop works best with very large files. The larger the file, the less time Hadoop spends seeking for the next data location on disk and the more time Hadoop runs at the limit of the bandwidth of your disks. Seeks are generally expensive operations that are useful when you only need to analyze a small subset of your dataset. Since Hadoop is designed to run over your entire dataset, it is best to minimize seeks byusing largefiles. Hadoop is designed for streaming or sequential data access rather than random access. Sequential data access means fewer seeks, since Hadoop only seeks to the beginning of each block and begins reading sequentially from there. Hadoop uses blocks to store a file or parts of a file.

MapReduce :

A MapReduce program consists of two types of transformations that can be applied to data any number of times - a map transformation and a reduce transformation.
A MapReduce job is an executing MapReduce program that is divided into map tasks that run in parallel with each other and reduce tasks that run in parallel with each other.

Hadoop History

MDS-01330: unable to load MDS configuration document

My whole SOA env was down

Error Message in logs :

MDS-01330: unable to load MDS configuration document
MDS-01329: unable to load element "persistence-config"
MDS-01370: MetadataStore configuration for metadata-store-usage "OWSM_TargetRepos" is invalid.
MDS-00922: The ConnectionManager "oracle.mds.internal.persistence.db.JNDIConnectionManagerImpl" cannot be instantiated.
weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: ORA-28001: the password has expired

Cause :

The reason for this is that the passwords used by SOA (with its different users) to access the Database have expired

Solution:

1.Log in to SQLPLUS with system as sysdba [ If you dont have access ask your DBA guys to perform these tasks ]
2.Run the below Query to see the schemas that present in your repository

SET linesize 120
SET pagesize 1000
col owner FORMAT a20
col comp_name FORMAT a40
col version FORMAT a12

SELECT owner, comp_name, version, status, upgraded
FROM schema_version_registry
ORDER BY 1, 2, 3;

[You should change the passwords to all schemas that have the OWNER beginning with outdated "Prefix", Prefix refers to the related prefix used for your SOA schemas.]

3. change DB Pwd using

ALTER USER <user_name> identified BY <new_password>;

4. Log in to weblogic console and go to "Domain Structure/Services/Data Source" to change password to new one in Connection Pool tab.Do it for mds-owsm, mds-soa, OraSDPMDataSource, SOADataSource, and SOALocalTxDataSource.

5.Restart all the services in right order.

Datasource is found to be in suspended state in WebLogic 10.3

Issue : Datasource is found to be in suspended state in WebLogic 10.3

Sol:

If you see that any DataSource is in Suspended Mode then please first check the Server Logs...you may find one of the reason as "N/W disconnect from the Database". In this case you may try:

Look at the Server Logs...related to JDBC information.
Try to use the "utils.dbping" utility provided by WebLogic to test the connectivity between target Server Box and the Database.
If you see that the above step is able to establish the Connectivity with the database successfully then try to use the "weblogic.Admin" utility to Restore (Resume) the DataSource.

You can use “weblogic.Admin” utility to Enable and Disable the Pool (DataSource) just to confirm whether the dataSource is actually active or not

1.Run the configuration script "setWLSEnv.sh" first.

To suspend:

java weblogic.Admin -url t3://localhost:<port> -username weblogic -password <password> SUSPEND_POOL YourDataSourceName

To re-enable:

java weblogic.Admin -url t3://localhost:<port> -username weblogic -password <password> RESUME_POOL YourDataSourceName

If you get any exception or error while doing this it means there may be some database connectivity issue...

DBPing Utiity In WeblogicServer

DBPing :

The dbping command-line utility tests the connection between a DBMS and your client machine via a JDBC driver.

Syntax :

$ java utils.dbping ORACLE_THIN <USERNAME> <PASSWORD> <IP:PORT:SID>

Example

Source the environment using setDomainEnv.sh

[oracle@oracle basedomain]$ java utils.dbping ORACLE_THIN mani oracle 172.24.146.88:1581/mani

OUTPUT :

**** Success!!! ****
You can connect to the database in your app using:
java.util.Properties props = new java.util.Properties();
props.put("user", "mani");
props.put("password", "oracle");
java.sql.Driver d =
Class.forName("oracle.jdbc.OracleDriver").newInstance();
java.sql.Connection conn =
Driver.connect("jdbc:oracle:thin:@172.24.146.88:1581/mani", props);

Web services,WSDL,SOAP

Web Service :

Web services are application components
Web services communicate using open protocols
Web services are self-contained and self-describing
Web services can be discovered using UDDI
Web services can be used by other applications
HTTP and XML is the basis for Web services

Web service Life Cycle :

A web service has the below six life-cycle events:

1. Service Provider creates the web service and deploys it.
2. Then service provider, publishes the information about a web service in a service registry.
3. Service consumer search a service registry for available services.
4. Service consumer retrieves the information about the service.
5. Service consumer creates a web service client.
6. Web service client invokes the web service at the location specified in the service description.

NOTE:- Its not mandatory to publish the web service information in a service registry, if the service provider knows the entire list of service consumers. They can directly provide the WSDL URL / file to service consumers.

WSDL : Web Services Description Language