What is SDKMAN? In this article, I will explore how SDKMAN works and how it can be used to manage multiple versions of Java in Linux. Installing SDKMAN To use SDKMAN, you must first install… More
Configuring Oracle AQ with WSO2 Micro Integrator
In this blog post, I’ll guide you through the process of setting up Oracle AQ with the WSO2 MI server based on my own experiences.
Tested Products and versions
- WSO2 Product : WSO2 MI: 4.1.0 (U2 level 79)
- DB version: Oracle 12c
- OS : Ubuntu 22.04.2 LTS
- JDK: openjdk version “11.0.21”
- SQL developer: Version 23.1.0.097
Prerequisites:
- WSO2 MI Installation:
- Install and set up WSO2 MI on your server.
Configure Oracle 12c DB with docker
If you require you can refer to the below steps and commands to configure the Oracle 12c docker image.
To Search for Oracle-related images
sudo docker search oracle --filter=STARS=17
To pull this image
sudo docker pull truevoly/oracle-12c
To list all the locally available Docker images on your system
sudo docker images
To create a new directory (ora_data
) in the /var/local/
path
sudo mkdir -p /var/local/ora_data
To grant read, write, and execute (full) permissions to all users (owner, group, and others) for all files and directories under /var/local/ora_data/
sudo chmod -R 777 /var/local/ora_data/
To run a Docker container for Oracle Database 12c
sudo docker run \
-p 1521:1521 -p 5500:5500 \
-e ORACLE_SID=sys \
-e ORACLE_PWD=oracle \
-e ORACLE_MEM=4000 \
-v /opt/oracle/ora_data \
-d \
docker.io/truevoly/oracle-12c:latest
To open an interactive shell session (/bin/bash
) within the specified Docker container
sudo docker exec -it c5223891ab6e /bin/bash
Connect Oracle db to Queue
Create a connection and log in using the sys user
- Username: sys
- Password: oracle
- SID: xe
- Port: 1521
- Role: SYSDBA
Create a new user / schema using the sys
create user testuser identified by oracle;
grant connect to testuser;
grant all privileges to testuser;
Grant privileges to the user that we created
grant all on sys.dbms_aq to testuser;
grant all on dbms_aqadm to testuser;
grant all on dbms_aqin to testuser;
create a queue and queue table by executing the below in the schema we created previously
EXEC dbms_aqadm.create_queue_table('testqt', 'SYS.AQ$_JMS_TEXT_MESSAGE')
EXEC dbms_aqadm.create_queue('testq','testqt')
EXEC dbms_aqadm.start_queue('testq')
To enable the display of server output.
set serverout on
If needed you can simply open two SQL developer instances and directly push the messages to the queue and then receive the messages.
- For that, you can use the below to send a message to the queue directly.
DECLARE
enqueue_options DBMS_AQ.ENQUEUE_OPTIONS_T;
message_properties DBMS_AQ.MESSAGE_PROPERTIES_T;
message_handle RAW (16);
msg SYS.AQ$_JMS_TEXT_MESSAGE;
BEGIN
msg := SYS.AQ$_JMS_TEXT_MESSAGE.construct;
msg.set_text('This is a test message!');
DBMS_AQ.ENQUEUE (
queue_name => 'testq',
enqueue_options => enqueue_options,
message_properties => message_properties,
payload => msg,
msgid => message_handle);
COMMIT;
END;
/
YOU CAN USE THE BELOW, TO LISTEN TO THE QUEUE AND DEQUEUE MESSAGES IN THE QUEUE TABLE
DECLARE
dequeue_options DBMS_AQ.DEQUEUE_OPTIONS_T;
message_properties DBMS_AQ.MESSAGE_PROPERTIES_T;
message_handle RAW (16);
msg SYS.AQ$_JMS_TEXT_MESSAGE;
BEGIN
DBMS_AQ.dequeue (
queue_name => 'testq',
dequeue_options => dequeue_options,
message_properties => message_properties,
payload => msg,
msgid => message_handle);
DBMS_OUTPUT.PUT_LINE(msg.TEXT_VC);
COMMIT;
END;
/
Configure the JMS Transport Sender and Receiver in WSO2 MI
Add the below configurations into the deployment.toml file which resides in the <MI_HOME>/conf directory.
Configure the JMS Transport Sender
[[transport.jms.sender]]
name = "commonJmsSenderConnectionFactory"
parameter.db_url="jdbc:oracle:thin:@localhost:1521/xe" ### DB url with SID
parameter.initial_naming_factory = "oracle.jms.AQjmsInitialContextFactory"
parameter.connection_factory_name = "QueueConnectionFactory"
parameter.connection_factory_type = "queue"
parameter.naming_security_principal="testuser" ### user
parameter.naming_security_credential="oracle" ### password
[[transport.jms.sender]]
name = "commonJmsSenderConnectionFactory"
parameter.db_url="jdbc:oracle:thin:@localhost:1521/xe" ### DB url with SID
parameter.initial_naming_factory = "oracle.jms.AQjmsInitialContextFactory"
parameter.connection_factory_name = "TopicConnectionFactory"
parameter.connection_factory_type = "topic"
parameter.naming_security_principal="testuser" ### user
parameter.naming_security_credential="oracle" ### password
Configure the JMS Transport Listner
[[transport.jms.listener]]
name = "AqQueueConnectionFactory"
parameter.initial_naming_factory = "oracle.jms.AQjmsInitialContextFactory"
parameter.db_url="jdbc:oracle:thin:@localhost:1521/xe" ### DB url with SID
parameter.connection_factory_name = "QueueConnectionFactory"
parameter.connection_factory_type = "queue"
parameter.naming_security_principal="testuser" ### user
parameter.naming_security_credential="oracle" ### password
[[transport.jms.listener]]
name = "AqTopicConnectionFactory"
parameter.initial_naming_factory = "oracle.jms.AQjmsInitialContextFactory"
parameter.db_url="jdbc:oracle:thin:@localhost:1521/xe" ### DB url with SID
parameter.connection_factory_name = "TopicConnectionFactory"
parameter.connection_factory_type = "topic"
parameter.naming_security_principal="testuser" ### user
parameter.naming_security_credential="oracle" ### password
create a Proxy service to publish messages to a queue in Oracle AQ
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="TestProxy"
transports="http https"
startOnLoad="true">
<description/>
<target>
<inSequence>
<log>
<property name="in" value="==== IN ====="/>
</log>
<property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
<property name="OUT_ONLY" value="true"/>
<send>
<endpoint>
<address uri="jms:/Queues/testq?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&java.naming.factory.initial=oracle.jms.AQjmsInitialContextFactory&transport.jms.DestinationType=queue&java.naming.security.principal=testuser&java.naming.security.credentials=oracle&db_url=jdbc:oracle:thin:@localhost:1521/xe"/>
</endpoint>
</send>
</inSequence>
</target>
</proxy>
Create an Inbound endpoint to consume messages from the queue in Oracle AQ
<?xml version="1.0" encoding="UTF-8"?>
<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse" name="DurableTopicInboundListener" sequence="request" onError="fault" protocol="jms" suspend="false">
<parameters>
<parameter name="interval">1000</parameter>
<parameter name="transport.jms.Destination">Queues/testq</parameter>
<parameter name="transport.jms.CacheLevel">1</parameter>
<parameter name="transport.jms.ConnectionFactoryJNDIName">QueueConnectionFactory</parameter>
<parameter name="sequential">true</parameter>
<parameter name="java.naming.factory.initial">oracle.jms.AQjmsInitialContextFactory</parameter>
<parameter name="java.naming.provider.url">jdbc:oracle:thin:@localhost:1521/testdb</parameter>
<parameter name="java.naming.security.principal">testuser</parameter>
<parameter name="java.naming.security.credentials">oracle</parameter>
<parameter name="db_url">jdbc:oracle:thin:@localhost:1521/xe</parameter>
<parameter name="transport.jms.SessionAcknowledgement">AUTO_ACKNOWLEDGE</parameter>
<parameter name="transport.jms.SessionTransacted">false</parameter>
<parameter name="transport.jms.ConnectionFactoryType">queue</parameter>
</parameters>
</inboundEndpoint>
Deploy the sequence as below
- Then it will engage when it sends the correct message.
Eg:
<?xml version="1.0" encoding="UTF-8"?>
<sequence name="request" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<log level="full"/>
<drop/>
</sequence>
Fault sequence will consume when it sends the incorrect message
<sequence xmlns="http://ws.apache.org/ns/synapse" name="fault">
<!-- Log the message at the full log level with the ERROR_MESSAGE and the ERROR_CODE-->
<log level="full">
<property name="MESSAGE" value="Executing default 'fault' sequence"/>
<property name="ERROR_CODE" expression="get-property('ERROR_CODE')"/>
<property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')"/>
</log>
<!-- Drops the messages by default if there is a fault -->
<drop/>
</sequence>
- Additionally, you need to add the below jar files to the wso2mi-4.1.0-home/lib directory.
- If you need you can use jars which I used for the testings from here.
- aqapi.jar
- jmscommon.jar
- jta.jar
- ojdbc7.jar
Once you have done all the required configurations you can test the scenario.
Eg:
curl --location 'http://localhost:8290/services/TestProxy' \
--header 'Content-Type: application/xml' \
--data '<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://www.w3.org/2005/08/addressing">
<soapenv:Body>
<m0:getQuote xmlns:m0="http://services.samples">
<m0:request>
<m0:symbol>test</m0:symbol>
</m0:request>
</m0:getQuote>
</soapenv:Body>
</soapenv:Envelope>'
Hope you find this blog useful! Now, it’s your turn to dive in and explore further.
Cheers !!
How SSL Works
Risk management
- Risk management is part of information security .
- That means preventing or reducing the unauthorized access to data, or data deletion, or data modification or any inconvenient thing do for the information
- So this Information security’s primary focus on confidentiality, integrity, and availability of data
- We also known as the CIA triad
CIA Triad
- Confidentiality
- That means Computer-related assets are only available to authorized parties.
- Only those parties should have access to something.
- So Controlling the access based on the need is Confidentiality.
- We use Authorization, authentication, encryption concepts to do this.
- Integrity
- It Assure that the data or information systems can be trusted (hashing,signing)
- So the data should be Unmodified, Meaningful and usable
- Availability
- This means Data and information systems are available when required (monitoring, alerting,HA)
- So it needs to timely response to our requests
- So it should Control concurrency, support for simultaneous access with proper deadlock, and access management.
What is SSL Means?
- SSL which means Secure Sockets Layer
- That means a protocol which we can use to establish authenticated and encrypted link between network.
- This is a cryptographic protocol which used to provide security over internet communications
- So this SSL provides a secure channel between two machines or devices operating over the internet or an internal network.
- As this SSL is a security protocol, it describe how algorithms should be used.
Example
- One common example is SSL use to secure communication between a web browser and a web server.
- That means HTTPS
- Hyper text transfer protocol secure
- As like HTTP, HTTPS is application layer protocol
- So it is transferring web pages
- Using HTTP is not secure method.
- So we can use HTTPS which is secure protocol. So data will be encrypted and send.
- So if attacker trying to get information, they can also get encrypted data.
- Main thing is this HTTPS websites includes SSL/TLS certificate which is signed by a CA.
- we can use this method for online payments like credit card transactions
How Do I Know a Website is Secure with SSL?
Why we use SSL?
- Transferring sensitive information over a network can be risky
- Because You can’t always be sure the entity you are communicating is real one.
- Also there is possibility that an unauthorized third party is reading our data.
- If attacker access our data, he can modify the data, before sending it to the receiver.
- We can use this secure method to transfer sensitive data Like payments information.
- In online transactions need to be more secure. So we need to use this kind of secure method to transfer information.
- Protect the data of you and your users
- This will encrypt the data and transfer.
- That means any malicious trying to get this data, he will also get the encrypted data.
- Which is not usable.
- Our users can feel safe that their data is in safe
- Showing your users you can be trusted
- This method is providing a SSL certificate
- So users know they can trust you.
How does the SSL provide data security?
- Authenticating
- It ensure the identity of the other party which is authenticating the user.
- Two parties need to authenticate weather they have access to communicate
- Establish an encrypted connection
- Once these users are authenticated, SSL provides an encrypted connection between them for secure message transmission.
- Send encrypted data
- Here encryption algorithms are using with SSL
- So This ensure that data doesn’t modified during transit.
What is SSL Handshake?
- The SSL/TLS handshake enables client and server validate each other and start communicating through the secure SSL/TLS tunnel.
- That means this handshake is done when client and server trying to communicate with each other for the first time
- So before starting this SSL/TLS handshake that TCP connection need to be already established
Uses of SSL
- Online credit card transactions or other online payments.
- Office Communications Servers – Webmail servers like Outlook
- Cloud-based computing platforms – virtualization applications
- Hosting control panel login – like cPanel
1-Way and 2-Way SSL
1-way SSL
- This is the common way to verify the authenticity of the website that you are trying to access.
- In this authentication, the client is never verified.
- It only validate the server, to ensure that the data is sending from the valid server.
2-way SSL
- Here both the client and server exchange their certificates and verifies authenticity.
- After that Mutual trust is obtaining because CA is verifying both parties’ certificates.
What is a Certificate?
- Certificate is a digitally signed document that binds the identity of entity and its public key.
- SSL certificate we also known as a TLS or SSL/TLS certificate.
- Here TLS means -Transport Layer Security.
- This is also a protocol we use for authentication and encryption.
- Normally we say Certificates are data files that have encrypted cryptographic keys with additional information such as domain name, host name, and server details of organization
- Certificates can either be self-signed or issued by a Certification Authority (CA)
- Certification Authorities are entities that are trusted to issue valid certificates for other entities
Certificate contents
- Issuer
- The issuer is the CA that issued the certificate.
- If a user trusts the CA that issues a certificate, and if the certificate is valid, the user can trust the certificate.
- Period of validity
- A certificate has an expiration date, and this date is one piece of information that should be checked when verifying the validity of a certificate.
- Subject
- The subject field includes information about the entity that the certificate represents.
- Subject’s public key
- The primary piece of information that the certificate provides is the subject’s public key.
- All the other fields are provided to ensure the validity of this key.
- Signature
- The certificate is digitally signed by the CA that issued the certificate.
- The signature is created using the CA’s private key and ensures the validity of the certificate.
- Because only the certificate is signed, not the data sent in the SSL transaction, SSL does not provide for non-repudiation.
How to get a certificate?
- For the SSL certificate to be valid one, it need to obtain from a CA.
- CA is an outside organization and it is a trusted third party
- Client Generate the CSR which means certificate signing request
- Then Send CSR to the CA that means certificate authority
- The CA will also digitally sign the certificate with their own private key, allowing client devices to verify it.
- Once the certificate is issued, it needs to be installed and activated on the website’s server.
- So normally Web hosting services handle this for website operators.
- Once it’s activated on the origin server, the website will be able to load over HTTPS
- So then all traffic to and from the website will be encrypted and secure.
SSL CAs
- GoDaddy
- HubSpot
- Cloudflare
Self-signed SSL certificate
- Security certificate that is not signed by a certificate authority
- As the name says, this is a certificate that is generated for internal purposes and is not issued by a CA.
- website owner generates their own certificate,and this certificate that is not signed by a certificate authority
- These certificates are easy to make and do not need money
- We can create them using some tools
- OpenSSL,
- Java’s keytool
- Adobe Reader
- WolfSSL
- Apple’s Keychain
- But there is no outside authority to verify the origin server
- These self-signed SSL certificate are easy to customize
Difference Between Self-Signed and CA signed certificates
TCP and HTTP
What is a protocol?
- A Protocol is a collection of rules and procedures for two computers to exchange information.
- Rules are defined for each step and process during communication between two or more computers.
- Networks have to follow these rules to successfully transmit data.
What is IP?
- The Internet Protocol (IP) is the method for sending data from one device to another across the internet.
- Every device has an IP address that uniquely identifies it and enables it to communicate with and exchange data with other devices connected to the internet.
- IP is responsible for defining how applications and devices exchange packets of data with each other.
- It is the principal communications protocol responsible for the formats and rules for exchanging data and messages between computers on a single network or several internet-connected networks.
- IP is a network-layer protocol that provides a connection-less data transmission service that is used by TCP
- Main purpose is to deliver data packets between the source application or device and the destination using methods and structures that place tags, such as address information, within data packets.
- Data is transmitted link by link; an end-to-end connection is never set up during the call. The unit of data transmission is the datagram.
OSI Model
- OSI Model – Open Systems Interconnection Model
- It is a conceptual framework used to describe the functions of a networking system.
OSI Layers
- Layer 7 – application layer – enables the user to interact with the application or network when the user wants to read messages, transfer files or engage in other network-related activities.
- Layer 6 – presentation layer – translates or formats data for the application layer based on the semantics or syntax that the app accepts.
- Layer 5 – session layer – sets up, coordinates and terminates conversations between apps.
- Layer 4 – transport layer – handles transferring data across a network and providing error-checking mechanisms and data flow controls.
- Layer 3 – network layer – moves data into and through other networks.
- Layer 2 – data link layer – handles problems that occur as a result of bit transmission errors.
- Layer 1 – physical layer – transports data using electrical, mechanical or procedural interfaces.
What Is TCP?
- TCP/IP stands for Transmission Control Protocol/Internet Protocol
- TCP defines how applications can create channels of communication across a network.
- Data streaming from source to destination gets split up into chunks known as “packets” for more manageable transport.
- Whenever you send or receive a packet of data, a host of information about that data rides along.
- This includes information added by the TCP.
- TCP ensure that all data sent in a stream moves from Point A to Point B in correct order and intact.
TCP Three-Way Handshake Process
- TCP/IP uses a three-way handshake to establish a connection between a device and a server
- It ensures multiple TCP socket connections can be transferred in both directions concurrently.
- Both the device and server must synchronize and acknowledge packets before communication begins, then they can negotiate, separate, and transfer TCP socket connections.
Layers of the TCP/IP model
Common TCP/IP protocols and their ports
- Hypertext Transfer Protocol – HTTP – 80
- HTTP Secure – HTTPS – 443
- File Transfer Protocol – FTP – 21
- Secure Shell – SSH – 22
- Telnet -23
- Simple Mail Transfer Protocol – SMTP – 25
How does TCP/IP work?
Why is TCP/IP important?
- Can be modified easily.
- Compatible with all operating systems
- Compatible with all types of computer hardware and networks.
- Highly scalable
Key features of the TCP/IP model
- Supports flexible architecture
- End-node verification
- Dynamic Routing
Pros and cons of TCP/IP
Advantages | Disadvantages |
---|---|
TCP/IP model vs OSI model
Difference between TCP/IP model and OSI model
An overview of HTTP
What Is HTTP?
- The Hypertext Transfer Protocol is an application protocol that allows users to communicate data on the World Wide Web.
How does HTTP work?
Parts of the HTTP request
- Protocol – It can be HTTP/ HTTPS /FTP or any other protocol
- Domain (Host) – Name that used to identify one or more IP address where resource is located.
- Path – Specify the resource location on server
- Parameters – Additional data use to identify the resource
HTTP Request Methods
- GET – retrieve a specific resource or a collection of resources
- POST – create a new resource
- PUT – update a specific resource
- DELETE – remove a specific resource
HTTP Response
- 2xx success – indicates the action requested by the client was received, understood, and accepted.
- 3xx redirection – indicates the client must take additional action to complete the request.
- 4xx client errors – intended for situations in which the error seems to have been caused by the client.
- 5xx server errors – indicate cases in which the server is aware that it has encountered an error or is otherwise incapable of performing the request.
How HTTP and TCP use to establish connection and transfer data
HTTP and TCP
OSGI
What Is OSGi?
- OSGi is a Java framework for developing and deploying modular software programs and libraries.
- As a developer, you use the OSGi specification and tools to create one or more bundles.
- OSGi defines the lifecycle for these bundles.
- It also hosts them and supports their interactions in a container.
- You can think of an OSGi container as roughly analogous to a JVM, with additional powers. Likewise, think of bundles as Java applications with unique abilities. Bundles run inside the OSGi container as client and server components.
- In OSGi, multiple applications can exist within the same container: the OSGi bundle runtime environment. The container ensures each component is sufficiently isolated, and also has access to any dependencies it requires.
OSGi overview
- OSGi framework
- The framework is the runtime that implements and
provides OSGi functionality.
- The framework is the runtime that implements and
- OSGi standard services
- The standard services define reusable APIs for common tasks, such as Logging.
Why we need OSGi?
- It reduces complexity in almost all aspects of development
- Code is easier to write and test, reuse is increased, build systems become significantly simpler
- Deployment is more manageable
Advantages of OSGi
- You can install, uninstall, start, and stop different modules of your application dynamically without restarting the container.
- Your application can have more than one version of a particular module running at the same time.
- OSGi provides very good infrastructure for developing service-oriented applications, as well as embedded, mobile, and rich internet apps.
Usage
- Healthcare
- Media Control Systems
- Telecommunications
- Enterprise Software Platforms
- Robotics
- mobile phones to the open-source Eclipse IDE
OSGi containers
- It is a Component framework.
- Three popular open source OSGi containers:
- Equinox – By Eclipse
- Apache Felix – By Apache Software Foundation
- Knopflerfish – By Makewave
OSGI Architecture
- Module Layer
- The Bundle
- Packaging and sharing the code
- Lifecycle Layer
- The Container
- Managing an OSGi application
- Service Layer
- The system
- How OSGi bundles communicate
Module Layer (Bundle Layer)
- OSGi lets you divide your Java application into bundles
- Every bundle has a name and a version
- Bundles can declare dependency on each-other
- When you drop a bundle in an OSGi container, then the container handles the inter-dependencies automatically.
Sample Manifest.MF file
Manifest-Version: 1.0
Bnd-LastModified: 1618862826488
Build-Jdk: 1.8.0_281
Built-By: Maneesha
Bundle-Description: Simple User Registration Module with REST API.
Bundle-DocURL: http://www.wso2.org/
Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt
Bundle-ManifestVersion: 2
Bundle-Name: org.wso2.carbon.identity.user.registration.smu.core
Bundle-SymbolicName: org.wso2.carbon.identity.user.registration.smu.co
re
Bundle-Vendor: WSO2
Bundle-Version: 0.0.1
Created-By: Apache Maven Bundle Plugin
DynamicImport-Package: *
Export-Package: org.wso2.carbon.identity.user.registration.smu.core;Class=org.wso2.carbon.user.core.service.RealmService)";effective:=act
ive,osgi.service;filter:="(objectClass=org.wso2.carbon.utils.Configur
ationContextService)";effective:=active,osgi.ee;filter:="(&(osgi.ee=J
avaSE)(version=1.8))"
Service-Component: OSGI-INF/org.wso2.carbon.identity.framework.user.re
gistration.smu.core.component.xml
Tool: Bnd-3.2.0.201605172007
What is Modularity?
- Modularity is a software designing architecture.
- Modularity enables the program to separate the functionality of a large program into independent, interchangeable modules, in such a way that each contains everything necessary to execute only one aspect of the desired functionality.
What is a Module?
- Logically independent piece
- Each module in the system has their own boundary and in a single module
Advantages of modularity
- The module is small enough to understand and debug
- Easy to monitor, control and maintain
- Can be tested independently
Life-cycle Layer
- A Life-Cycle layer adds bundles that can be dynamically installed, started, stopped, updated and uninstalled.
- Bundles rely on the module layer for class loading but add an API to manage the modules in run time.
- The life cycle layer introduces dynamics that are normally not part of an application.
- Extensive dependency mechanisms are used to assure the correct operation of the environment.
- Life cycle operations are fully protected with the security architecture.
- INSTALLED
- BundleContex.installBundle() , the operation creates a bundle in this state.
- RESOLVED
- All Java classes that the bundle needs are available. This state indicates that the bundle is either ready to be started or has stopped.
- STARTING
- The bundle is being started, the BundleActivator.start() the method has been called but the start method has not yet returned. When the bundle has an activation policy, the bundle will remain in the STARTING state until the bundle is activated according to its activation policy.
- ACTIVE
- The bundle has been successfully activated and is running; its Bundle Activator start method has been called and returned. If there is an error in the ACTIVE state it will go to the RESOLVED state again, this jumps to RESOLVED state because in that state all the dependencies are resolved so no need to jump to the INSTALLED state again.
- STOPPING
- The bundle is being stopped. The BundleActivator.stop() method has been called but the stop method has not yet returned.
- UNINSTALLED
- The bundle has been uninstalled. It cannot move into another state.
Service Layer
- Thus to provide the interaction between bundles, services are used.
- Services are specified by the Java interface.
Parties in Service
- Service Provider
- Bundles can implement this interface and register the service with the Service Registry.
- Service Consumer
- Clients of the service can find it in the registry
Heap Dump
What is a heap?
- Heap space in Java is used for dynamic memory allocation for Java objects and JRE classes at the runtime.
- Whenever we create a Java object by creating an instance of a class, it is always placed in an area known as the heap.
- The heap gets created when the JVM starts up.
- It expands or shrinks during runtime to accommodate the objects created or destroyed in our application.
- When the heap becomes full, the garbage collection process is run to collect the objects that are not referenced anymore.
- SHALLOW HEAP:
- The amount of memory occupied by the object itself
- The actual memory consumed by one object.
- RETAINED HEAP:
- Amount of memory that will be freed when the particular object is garbage collected
- The amount of memory that an object can be reclaimed,including the memory occupied by direct and indirect references.
- normally we say it is a sum of shallow sizes of all objects in the retained set of that object.
Key Features of Java Heap Memory
- It’s accessed via complex memory management techniques that include,
- Young Generation
- Old or Tenured Generation
- Permanent Generation
- If heap space is full, Java throws java.lang.OutOfMemoryError
- Access to this memory is relatively slower than stack memory
- This memory, in contrast to stack, isn’t automatically deallocated.
- It needs Garbage Collector to free up unused objects so as to keep the efficiency of the memory usage
- Unlike stack, a heap isn’t threadsafe and needs to be guarded by properly synchronizing the code
Types of Applications where Heap Size Matters
- BigData analyzing
- NoSQL Databases
- Analytics
- eCommerce
Garbage Collection
- Garbage Collection is process of reclaiming the runtime unused memory automatically
- It is a form of automatic memory management
- Garbage collection will free up the memory of the object that doesn’t have any reference.
What is a heap dump?
- Heap dumps contain a snapshot of all the live objects that are being used by a running Java application on the Java heap.
- It contains information such as,
- what are the objects in memory?
- what values do they carry?
- what is their size?
- what other objects do they reference?
Heap dumps formats:
- Classic format
- human-readable format (ascii text)
- Portable Heap Dump (PHD) format
- default (binary)
Usage of heap dump analyzing
- The JVM software allocates memory for objects from the heap for all class instances and arrays.
- The garbage collector reclaims the heap memory when an object is no longer needed and there are no references to the object.
- By examining the heap you can locate where objects are created and find the references to those objects in the source.
- So this heap dumps are very useful to troubleshoot memory-leak problems and optimize memory usage in Java applications.
- Memory problems that can investigate using the heal dumps
- Memory leaks
- Garbage Collection problems
- java.lang.OutOfMemoryError
OutofMemory Error
- Can be occurred due to the memory size that we already allocated for the application is not enough.
- Because didn’t increase the heap size.
What we are looking for in a Heap dump is:
- Objects with high memory usage
- Object graph to identify objects of not releasing memory
- Reachable and unreachable objects
How to Take Heap Dump
- A JVM argument can be added to generate heap dump whenever an OutOfMemoryError occurs.
- The
-XX:+HeapDumpOnOutOfMemoryError
option can be added to generate a heap dump on OutOfMemoryError.
- The
- By default, the heap dump is created in a file called java_pid pid .hprof in the working directory of the VM, but we can set an alternative path using the JVM option
-XX:HeapDumpPath=path
. - Using a jmap tool available with JDK.
- The following command can be executed from the command line:
-
jmap -dump:format=b,file=heap.bin <pid>
“<pid>” can be replaced with the process id of the application
-
jmap -dump:live,format=b,file=heapDump2.hprof 102707
- live: if set it only prints objects which have active references and discards the ones that are ready to be garbage collected. This parameter is optional
- format=b: specifies that the dump file will be in binary format. If not set the result is the same
- file: the file where the dump will be written to
- pid: id of the Java process
jcmd 12587 GC.heap_dump /tmp/dump.hprof
As with jmap, the dump generated is in binary format.
Leak Suspects Report
Java Memory Model
What is a JVM ?
- JVM – Java Virtual Machine
- It is an abstract machine.
- It is a specification that provides runtime environment in which java bytecode can be executed.
- JVMs are available for many hardware and software platforms
- However JVM is platform dependent.
- Whenever we execute a Java program, a separate memory area is reserved for storing various parts of our application code which we roughly call JVM memory.
- Without having a good understanding of how JVM actually consumes the memory and how garbage collector uses different parts of this volatile memory, we may miss some important considerations for better memory management, thus better performance.
It is:
- A specification
- where working of Java Virtual Machine is specified.
- But implementation provider is independent to choose the algorithm.
- An implementation
- Its implementation is known as JRE (Java Runtime Environment).
- Runtime Instance
- Whenever you write java command on the command prompt to run the java class, an instance of JVM is created.
JVM,
- Loads code
- Verifies code
- Executes code
- Provides runtime environment
What is Java Garbage collection ?
- Garbage collection is the mechanism used in Java to de-allocate unused memory.
- In java, garbage means referenced objects.
- Java applications obtain objects in memory as needed.
- It is the task of garbage collection (GC) in the Java virtual machine (JVM) to automatically determine what memory is no longer being used by a Java application and to recycle this memory for other uses.
What is a Memory Model?
- The memory model describes possible behaviors of a program.
- It describes thread interaction with memory.
- It Changes in the ordering of reads and writes can cause race conditions.
- This memory model provides sequential consistency for data race free programs
What is Java Memory Model?
- Java Memory Model is the model that describes the behavior of memory in Java program.
- It is a set of rules all JVMs have to follow to ensure correct working of our concurrent programs.
- it is specification which guarantees visibility of fields when you have reordered of instructions.
The java memory model specification specifies how and when a thread can see the value of a shared variable modified by another thread and how to access the shared variable when necessary
eg:
If you run the program in one JVM then you pick that program and pick it into another JVM, the program output is exactly the same.
Java Memory Model Structure
- The Java Virtual Machine defines various run-time data areas that are used during execution of a program.
Cyber Security
Analyzing a Thread Dump
What is a thread dump?
- A thread Dump contains a snapshot of all the threads active at a particular point during the execution of a program.
- It contains all relevant information about the thread and its current state.
- Using thread dumps, we can detect the current state of execution of the threads in the process when the process is executing.
What is the importance of the thread dump analysis?
- Identifying problems in the Java platform is made easy by creating thread dumps.
- A thread dump contains a set of statements which help the system administrator to easily identify the problem in the program.
- Optimizing JVM performance
- Optimizing application performance
- Diagnosing problems
- e.g. Deadlock
Capturing the Java Thread Dump
- Jps
- Lists the instrumented Java Virtual Machines (JVMs) on the target system
- Jps -l
- Displays the full package name for the application’s main class or the full path name to the application’s JAR file.
- Jstack
- To get PID we can use jps command
- Prints Java thread stack traces for a Java process, core file, or remote debug server.
Analyze Thread Dump
Steps
01. Generate DeadLock by following below sample code.
public class DeadlockP {
public static void main(String[] args) throws Exception {
Object resourceA = new Object();
Object resourceB = new Object();
Thread threadLockingResourceAFirst = new Thread(new DeadlockRunnable(resourceA, resourceB));
Thread threadLockingResourceBFirst = new Thread(new DeadlockRunnable(resourceB, resourceA));
threadLockingResourceAFirst.start();
Thread.sleep(500);
threadLockingResourceBFirst.start();
}
private static class DeadlockRunnable implements Runnable {
private final Object firstResource;
private final Object secondResource;
public DeadlockRunnable(Object firstResource, Object secondResource) {
this.firstResource = firstResource;
this.secondResource = secondResource;
}
@Override
public void run() {
try {
synchronized(firstResource) {
printLockedResource(firstResource);
Thread.sleep(1000);
synchronized(secondResource) {
printLockedResource(secondResource);
}
}
} catch (InterruptedException e) {
System.out.println("Exception occurred: " + e);
}
}
private static void printLockedResource(Object resource) {
System.out.println(Thread.currentThread().getName() + ": locked resource -> " + resource);
}
}
}
layani@layani-ThinkPad-X1-Carbon-Gen-8:~/Documents/java$ javac DeadlockP.java
layani@layani-ThinkPad-X1-Carbon-Gen-8:~/Documents/java$ java DeadlockP
Thread-0: locked resource -> java.lang.Object@5a590c27
Thread-1: locked resource -> java.lang.Object@7ec5aaaa
02. Capturing the Java Thread Dump using JPS command
03. Get the PID for our application.
layani@layani-ThinkPad-X1-Carbon-Gen-8:~/Documents/java$ jps
3095 CShell.jar
727233 DeadlockP
727284 Jps
04. Capture the thread dump using jstack.
05. Store the result in a text file
layani@layani-ThinkPad-X1-Carbon-Gen-8:~/Documents/java$ jstack -l 727233 > DeadLock-thread-dump.txt
2021-08-07 18:22:08
Full thread dump OpenJDK 64-Bit Server VM (25.282-b08 mixed mode):
"Attach Listener" #13 daemon prio=9 os_prio=0 tid=0x00007f51b0001000 nid=0xb1e95 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"DestroyJavaVM" #12 prio=5 os_prio=0 tid=0x00007f520000a800 nid=0xb18c2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Thread-1" #11 prio=5 os_prio=0 tid=0x00007f5200117800 nid=0xb18d6 waiting for monitor entry [0x00007f51b70e5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at DeadlockP$DeadlockRunnable.run(DeadlockP.java:34)
- waiting to lock <0x000000076db5fea8> (a java.lang.Object)
- locked <0x000000076db5feb8> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Thread-0" #10 prio=5 os_prio=0 tid=0x00007f5200116800 nid=0xb18d5 waiting for monitor entry [0x00007f51b71e6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at DeadlockP$DeadlockRunnable.run(DeadlockP.java:34)
- waiting to lock <0x000000076db5feb8> (a java.lang.Object)
- locked <0x000000076db5fea8> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f52000f2000 nid=0xb18d3 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f52000cd000 nid=0xb18d2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f52000ca800 nid=0xb18d1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f52000c9000 nid=0xb18d0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f52000c6000 nid=0xb18cf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f52000c0000 nid=0xb18ce runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f520008f000 nid=0xb18cd in Object.wait() [0x00007f51ec2ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076db08ee0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x000000076db08ee0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f520008a800 nid=0xb18cc in Object.wait() [0x00007f51ec3ae000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076db06c00> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x000000076db06c00> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f5200081000 nid=0xb18cb runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f520001f800 nid=0xb18c3 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f5200021800 nid=0xb18c4 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f5200023000 nid=0xb18c5 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f5200025000 nid=0xb18c6 runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f5200026800 nid=0xb18c7 runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f5200028800 nid=0xb18c8 runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f520002a000 nid=0xb18c9 runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f520002c000 nid=0xb18ca runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f52000f4800 nid=0xb18d4 waiting on condition
JNI global references: 5
Found one Java-level deadlock:
=============================
"Thread-1":
waiting to lock monitor 0x00007f51b8006568 (object 0x000000076db5fea8, a java.lang.Object),
which is held by "Thread-0"
"Thread-0":
waiting to lock monitor 0x00007f51b8003ac8 (object 0x000000076db5feb8, a java.lang.Object),
which is held by "Thread-1"
Java stack information for the threads listed above:
===================================================
"Thread-1":
at DeadlockP$DeadlockRunnable.run(DeadlockP.java:34)
- waiting to lock <0x000000076db5fea8> (a java.lang.Object)
- locked <0x000000076db5feb8> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:748)
"Thread-0":
at DeadlockP$DeadlockRunnable.run(DeadlockP.java:34)
- waiting to lock <0x000000076db5feb8> (a java.lang.Object)
- locked <0x000000076db5fea8> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
Main Parts of THREAD Dump
Name
it can provide useful information if developers include a meaningful thread name
Priority (prior)
the priority of the thread
Java ID (tid)
the unique ID given by the JVM
Native ID (nid)
the unique ID given by the OS, useful to extract correlation with CPU or memory processing
State
the actual state of the thread
Stack trace
the most important source of information to decipher what is happening with our application
Manual Analysis of Thread Dumps
ps -mo pid,lwp,stime,time,cpu -C java
layani@layani-ThinkPad-X1-Carbon-Gen-8:~/Documents/java$ ps -mo pid,lwp,stime,time,cpu -C java
PID LWP STIME TIME CPU
727233 - 17:56 00:00:00 -
- 727233 17:56 00:00:00 -
- 727234 17:56 00:00:00 -
- 727235 17:56 00:00:00 -
Thread Dumps Analyzers
Fast thread
- Thread dump analysis tool to troubleshoot complex production problems.
- We can upload the thread dump as a file or we can directly copy and paste the thread dump
Spotify Thread Dump Analyzer
- It is an online tool and accepts the thread dump as a file or we can directly copy and paste the thread dump
Jstack review
- analyzes java thread dumps from within the browser
Swagger
What is Swagger?
- Swagger is a set of rules, specifications and tools that help us document our APIs.
- With Swagger, documentation can be used directly to automate API-dependent processes.
- By using Swagger UI to expose our API’s documentation, we can save significant time.
- Swagger UI uses an existing JSON or YAML document and makes it interactive.
- Swagger is a 100% open source, standard, language-agnostic specification and a complete framework for describing, producing, consuming, and visualizing RESTful APIs, without the need of a proxy or third-party services.
Why we need Swagger?
- It’s comprehensible for developers and non-developers.
- Product managers, partners, and even potential clients can have input into the design of your API, because they can see it clearly mapped out in the friendly UI.
- It’s human readable and machine readable.
- This means that not only can this be shared with your team internally, but the same documentation can be used to automate API-dependent processes.
- It’s easily adjustable.
- This makes it great for testing and debugging API problems.
Swagger Editor
API Management with WSO2 API Manager
- WSO2 API manager is a comprehensive open source API Management solution commercially supported by WSO2 Inc.
- We can create a REST API using a SWAGGER (Open API definition) for an existing API.
Sample API Definition of MockAPI
OAuth2
What is Authentication?
- Authentication is the process of proving your own identity to third party service.
- It verifies whether a person’s identity is the same as he or she had declared himself or herself to be.
- So when we are trying to log in to Facebook or Google, we are required to first enter the email and password to verify our identity.
- Validating that users are whom they claim to be.
- Username/password
- Tokens
- Bio-metrics
What is Authorization?
- Authorization is the process of giving someone permission to do something or have something.
- In this process, a person grants another person to have some of its resources or use some of its resources.
- It is done after successful Authentication.
- Here an Authorization server is providing access to third-party services to use some of its resources.
- It is a process of granting the user permission to access a specific resource.
- Eg:
- Customer should be able to view their own payment.
What is OAuth2 means?
- OAuth 2.0 means “Open Authorization.
- It’s an open standard for authorization
- It allows an application to access resources hosted by another application on behalf of a user.
- It enables applications to obtain limited access to user accounts on an HTTP service
- It works by delegating user authentication to the service that hosts the user account, and authorizing third-party applications to access the user account.
- OAuth 2 provides authorization flows for web and desktop applications, and mobile devices.
OAuth2 Roles
- Resource Owner:
- The user or system that owns the protected resources and can grant access to them.
- An entity capable of granting access to a protected resource.
- When the resource owner is a person, it is referred to as an end-user.
- eg: the user who is going to use that application.
- The user or system that owns the protected resources and can grant access to them.
- Client:
- The client is the system that requires access to the protected resources. To access resources, the Client must hold the appropriate Access Token.
- An application making protected resource requests on behalf of the resource owner and with his authorization.
- The term “client” does not imply any particular implementation characteristics
- e.g: whether the application executes on a server, a desktop, or other devices
- The client is the system that requires access to the protected resources. To access resources, the Client must hold the appropriate Access Token.
- Authorization Server:
- This server receives requests from the Client for Access Tokens and issues them upon successful authentication and consent by the Resource Owner. The authorization server exposes two endpoints: the Authorization endpoint, which handles the interactive authentication and consent of the user, and the Token endpoint, which is involved in a machine to machine interaction.
- The server issuing access tokens to the client after successfully authenticating the resource owner and obtaining authorization.
- eg: Google, Facebook etc.
- The server issuing access tokens to the client after successfully authenticating the resource owner and obtaining authorization.
- This server receives requests from the Client for Access Tokens and issues them upon successful authentication and consent by the Resource Owner. The authorization server exposes two endpoints: the Authorization endpoint, which handles the interactive authentication and consent of the user, and the Token endpoint, which is involved in a machine to machine interaction.
- Resource Server:
- A server that protects the user’s resources and receives access requests from the Client. It accepts and validates an Access Token from the Client and returns the appropriate resources to it.
- eg: Dropbox, GoogleDrive etc.
- A server that protects the user’s resources and receives access requests from the Client. It accepts and validates an Access Token from the Client and returns the appropriate resources to it.
Flow
- The application requests authorization to access service resources from the user
- If the user authorized the request, the application receives an authorization grant
- The application requests an access token from the authorization server (API) by presenting authentication of its own identity, and the authorization grant
- If the application identity is authenticated and the authorization grant is valid, the authorization server (API) issues an access token to the application. Authorization is complete.
- The application requests the resource from the resource server (API) and presents the access token for authentication
- If the access token is valid, the resource server (API) serves the resource to the application
Authorization Grant
- Authorization Code: used with server-side Applications
- Implicit: used with Mobile Apps or Web Applications (applications that run on the user’s device)
- Resource Owner Password Credentials: used with trusted Applications, such as those owned by the service itself
- Client Credentials: used with Applications API access
OAuth Tokens
- Access tokens
- The client uses to access the Resource Server (API).
- They have very short lifetime i.e they expire within some minutes or hours.
- Refresh Tokens
- The client uses to get a new Access token.
- Their lifetime is much longer than access tokens i.e days, month and years.
Refresh token
The access token provided by any other grant flow has an expiration time. The client can use the refresh token as a grant to fetch the access token after the access token is expired.
A refresh token generally has a longer expiry duration. With the refresh token, the client can get the access token without reauthorization from the resource owner.
The client needs to provide refresh_token, client_id and client_secret while using this flow.
URL: https://auth-server.com/token
request params:
grant_type: refresh_token
client_id: 29352915982374239857
client_secret: uf1ie98f2kda2vu25uye3k1h
The response of API is the same as the access token.
Client Credential grant
This auth flow is used to grant the token outside the context of a user. The trusted client can use this to access resources that are not related to a user.
URL: https://auth-server.com/token
request params:
grant_type: client_credential
client_id: 29352915982374239857
client_secret: uf1ie98f2kda2vu25uye3k1h
The response of API is the same as the access token
Get Authorization Code
URL: https://auth-server.com/auth
request params:
response_type=code
client_id=29352915982374239857
redirect_uri=https://client.com/redirect
scope=create+delete
state=xcoiv98y2kd22vusuye3kch
With the above request, code is returned in response to the provided redirect URL.
Exchange Code for a Token
URL: https://auth-server.com/token
request params:
code: nao1n3f
grant_type: authorization_code
client_id: 29352915982374239857
client_secret: uf1ie98f2kda2vu25uye3k1h
The response of the API will provide a token:
{
"access_token":"K1Z04lLdZ1S0pIMF09DI1FTZjJ1jmOO06",
"token_type":"bearer",
"expires_in":3600,
"refresh_token":"IzH6bFh6laf6gQ6TgA3sj19mfgA5egZx9a",
"scope":"create delete"
}
The client can use the “ access_token” to access the resource.
Advantages of OAuth 2.0
- This flexible protocol relies on SSL (Secure Sockets Layer) to ensure data between the web server and browsers remain private to save user access token.
- SSL uses cryptography industry protocols to keep data safe.
- It allows limited access to the user’s data and allows accessing when authorization tokens expire.
- It is easy to implement and provides strong authentication.
- In addition to the two-factor authentication, tokens can be revoked if necessary
- Uses single sign on
- It has ability to share data for users without having to release personal information.
Disadvantages of OAuth 2.0
- If you are adding more extension at the ends in the specification, it will produce a wide range of non-interoperable implementations.
- you have to write separate pieces of code for Facebook, Google
- There is no common format, as a result, each service requires its own implementation.
- In the process of user verification, sometimes you have to make additional requests to get minimal user information.
- It can be solved with the help of JWT token, but not all services support it.
- When a token is stolen, an attacker gains access to the secure data for a while.
- To minimize this risk a token with signature can be used.
- If your favorite sites are connected to the central hub and the central account is hacked, then it will lead to serious effects across several sites instead of just one.