Glassfish

Apr 1st, 2018

References

Prerequisites

  1. Debian
  2. Java

GlassFish is an open-source enterprise level application server with high reliability and performance, which can run any Java EE project.

Clustering

A cluster is a collection of GlassFish Server instances that work together as one logical entity. A cluster provides a runtime environment for one or more Java Platform, Enterprise Edition (Java EE) applications. A cluster provides high availability through failure protection, scalability, and load balancing.

GlassFish Clustering

Install Glassfish

To download and install Glassfish application server

# Download Glassfish on Maven Central
$ wget https://repo1.maven.org/maven2/org/glassfish/main/distributions/glassfish/5.1.0/glassfish-5.1.0.zip
# Install unzip and then unzip Glassfish
$ apt-get install unzip
$ unzip glassfish-5.1.0.zip -d /opt

In order to visit admin page (your_server_IP:4848) remotely, you need to enable secure admin not to get “Secure Admin must be enabled to access the DAS remotely” error

cd /opt/glassfish5/bin
# Make the asadmin script executable
chmod +x asadmin
$ asadmin --host your_server_IP --port 4848 enable-secure-admin

You will receive the below error:

remote failure: At least one admin user has an empty password, which secure admin does not permit. Use the change-admin-password command or the admin console to create non-empty passwords for admin accounts.
Command enable-secure-admin failed.

If it is failed as above, change admin password first (default: username admin, password empty)

$ asadmin --port 15123 change-admin-password

Now run this command again:

$ asadmin --host your_server_IP --port 4848 enable-secure-admin

You must restart all running servers for the change in secure admin to take effect.

$ asadmin restart-domain domain1

To change port from 8080 to 80, edit

nano /opt/glassfish5/glassfish/domains/domain1/config/domain.xml

change port to 80

<network-listener protocol="http-listener-1" port="8080"

To change application context root, edit

nano /opt/glassfish5/glassfish/domains/domain1/config/domain.xml

change context-root to /

<application context-root="/application"

Fine-tuning

Read more

Display a list of command-line options that are passed to the Java application launcher when GlassFish Server is started.

asadmin list-jvm-options

To change jvm options, edit

nano /opt/glassfish5/glassfish/domains/domain1/config/domain.xml

Lists the instrumented HotSpot Java Virtual Machines. Get pid just like ps in linux.

jps

Prints a histogram of the heap.

jmap -histo:live <pid>

Prints a heap summary.

jmap -heap <pid>

Guidelines for Calculating Java Heap Sizing

Space Command Line Option Occupancy Factor
Java heap -Xms and -Xmx 3x to 4x old generation space occupancy after full garbage collection
Permanent Generation -XX:PermSize -XX:MaxPermSize 1.2x to 1.5x permanent generation space occupancy after full garbage collection
Young Generation -Xmn 1x to 1.5x old generation space occupancy after full garbage collection
Old Generation Implied from overall Java heap size minus the young generation size 2x to 3x old generation space occupancy after full garbage collection

As a sample

  • Having 2GB RAM => -Xms512m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=512m
  • Having 8GB RAM => -Xms2048m -Xmx4096m -XX:CompileThreshold=8000 -XX:PermSize=1024m -XX:MaxPermSize=2048m

Commands

change-master-password

Each asadmin command accepts argument –passwordfile to instruct it to read all the necessary passwords from it to avoid asking for passwords interactively. But it’s a bit tricky to find out how to define passwords in this password file, because it’s used for multiple types of passwords.

  • admin password with prefix AS_ADMIN_PASSWORD, default is empty password
  • master password with prefix AS_ADMIN_MASTERPASSWORD, default is “changeit”
  • user password with prefix AS_ADMIN_USERPASSWORD
  • alias password with prefix AS_ADMIN_ALIASPASSWORD

The default master password is changeit. You may want to change it by following command:

asadmin change-master-password --savemasterpassword=true mydomain

Using --savemasterpassword=true will save the password to the disk so that asadmin start-domain command doesn’t prompt for password. If you don’t want to store the password, omit this option.

Starting, Stopping, Restarting

$ asadmin start-domain
$ asadmin stop-domain
$ asadmin restart-domain

Adding Resources

$ asadmin add-resources "/path/to/glassfish-resources.xml"

Creating AuthRealm

$ asadmin create-auth-realm --classname realm_class [--help] [--property(name=value)[:name=value]*][--target target_name] auth_realm_name

During the creation of AuthRealm, if you get “remote failure: Creation of Authrealm jdbcbid failed.”, it turns out that the problem was locale language of JVM. After adding the following properties to JVM Options in Glassfish, everything will begin to work properly.

$ asadmin create-jvm-options -Duser.language=en
$ asadmin create-jvm-options -Duser.region=US

Then restart server to apply changes.

Deploying, Undeploying

$ asadmin deploy "/path/to/[application-name].war"
$ asadmin undeploy [application-name]

To prevent password entry each time

In order to save typing “admin username” and “password” every time you deploy or undeploy an application, create a password file pwdfile with content:

AS_ADMIN_PASSWORD=your_admin_password

Add –passwordfile in command:

$ asadmin --passwordfile pwdfile deploy /path/to/[application-name].war

Now the prompt for username/password won’t appear.

Recover Glassfish admin password

changeit

admin;{SSHA256}dvCEGFNHGtSyXIhJvwR5FnviH+u8fCadrUIqp6uJc1tP9Bv10CGT7A==;asadmin

empty

admin;{SSHA256}2GiTMM0n4ulpMfRaMPMImYvVF522XY6nW2cKSXoY1EoM2QlHrHUkDQ==;asadmin

Glassfish mail-resource for domain, gmail and yandex using no-ssl, ssl, tls

Domain settings with no-ssl

<mail-resource jndi-name="mail/myMailSession"
               host="mail.domain.com"
               user="user@domain.com"
               from="user@domain.com"
               object-type="user"
               store-protocol="imap"
               store-protocol-class="com.sun.mail.imap.IMAPStore"
               transport-protocol="smtp"
               transport-protocol-class="com.sun.mail.smtp.SMTPTransport"
               debug="true"
               enabled="true">
    <property name="mail.smtp.auth" value="true"/>
    <property name="mail.smtp.port" value="587"/>
    <property name="mail.smtp.password" value="PASSWORD"/>
    <property name="mail.smtp.starttls.enable" value="false"/>
</mail-resource>

Gmail settings with ssl

<mail-resource jndi-name="mail/myMailSession"
               host="smtp.gmail.com"
               user="user@gmail.com"
               from="user@gmail.com"
               object-type="user"
               store-protocol="imaps"
               store-protocol-class="com.sun.mail.imap.IMAPSSLStore"
               transport-protocol="smtps"
               transport-protocol-class="com.sun.mail.smtp.SMTPSSLTransport"
               debug="true"
               enabled="true">
    <property name="mail.smtps.auth" value="true"/>
    <property name="mail.smtps.password" value="PASSWORD"/>
</mail-resource>

Yandex settings with ssl

<mail-resource jndi-name="mail/myMailSession"
               host="smtp.yandex.com.tr"
               user="user@domain.com.tr"
               from="user@domain.com.tr"
               object-type="user"
               store-protocol="imaps"
               store-protocol-class="com.sun.mail.imap.IMAPSSLStore"
               transport-protocol="smtps"
               transport-protocol-class="com.sun.mail.smtp.SMTPSSLTransport"
               debug="true"
               enabled="true">
    <property name="mail.smtps.auth" value="true"/>
    <property name="mail.smtps.password" value="PASSWORD"/>
</mail-resource>

alternatedocroot example

glassfish-web.xml

<property description="Uploaded images" name="alternatedocroot_1" value="from=/images/* dir=/Users/[username]/Developments/backup"/>

xhtml file

<p:graphicImage value="/images/logo.png"/>

Note: Be sure that there is “/Users/[username]/Developments/backup/images” directory Note: use “value” attribute, not “name” attribute

JDBC connection pool and datasource

cp $HOME/mysql-connector-java-5.1.5-bin.jar $GLASSFISH_HOME/domains/domain1/lib/ext

$GLASSFISH_HOME/bin/asadmin create-jdbc-connection-pool --datasourceclassname com.mysql.jdbc.jdbc2.optional.MysqlDataSource --property user=root:password=test:DatabaseName=test:ServerName=localhost:port=3306 test-pool

$GLASSFISH_HOME/bin/asadmin create-jdbc-resource --connectionpoolid test-pool jdbc/test

To create a connection pool that supports distributed transaction, use com.mysql.jdbc.jdbc2.optional.MysqlXADataSource as datasourceclassname, and set --restype javax.sql.XADataSource option:

$GLASSFISH_HOME/bin/asadmin create-jdbc-connection-pool --restype javax.sql.XADataSource --datasourceclassname com.mysql.jdbc.jdbc2.optional.MysqlXADataSource --property user=root:password=test:DatabaseName=test:ServerName=localhost:port=3306 test-pool

Run asadmin ping-connection-pool test-pool to verify whether the created connection pool can connect to the database. The database server needs to be running.

Configurations

According to the Glassfish Documentation, you can basically use server-config.

The default-config configuration is a special configuration that acts as a template for creating named configurations. Clusters and instances cannot refer to the default-config configuration. The default-config configuration can only be copied to create configurations.

So, below commands makes the same changes.

$ [GLASSFISH_HOME]/bin/asadmin create-jvm-options -Xmx=4g
$ [GLASSFISH_HOME]/bin/asadmin create-jvm-options --target server-config -- '-Xmx4096m'

Performance

Refer the documentation Optimize GlassFish Performance in a Production Environment

HTTP Compression

Edit GLASSFISH_HOME/glassfish/domains/domain1/config/domain.xml and change http-listener-1 and http-listener-2 under server-config configuration.

<configs>
  <config name="server-config">
...
    <network-config>
      <protocols>
        <protocol name="http-listener-1">
          <http max-connections="250" default-virtual-server="server"
                compression="on" 
                compression-min-size-bytes="128"
                compressable-mime-type="text/html,text/css,text/plain,text/xml,application/javascript,application/json">
            <file-cache></file-cache>
          </http>
          <ssl></ssl>
        </protocol>
        <protocol name="http-listener-2" security-enabled="true">
          <http max-connections="250" default-virtual-server="server"
                compression="on" 
                compression-min-size-bytes="128"
                compressable-mime-type="text/html,text/css,text/plain,text/xml,application/javascript,application/json">
            <file-cache></file-cache>
          </http>
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
        </protocol>

Set/configure the EJB Timer Service’s DataSource

Database setup

Create the table EJB__TIMER__TBL from {GLASSFISH_HOME}/glassfish/lib/install/databases/ejbtimer_{DB_ENGINE}.sql. DB_ENGINE is MySQL for example.

CREATE TABLE EJB__TIMER__TBL (
    `CREATIONTIMERAW`      BIGINT        NOT NULL,
    `BLOB`                 BLOB,
    `TIMERID`              VARCHAR(255)  NOT NULL,
    `CONTAINERID`          BIGINT        NOT NULL,
    `OWNERID`              VARCHAR(255)  NULL,
    `STATE`                INTEGER       NOT NULL,
    `PKHASHCODE`           INTEGER       NOT NULL,
    `INTERVALDURATION`     BIGINT        NOT NULL,
    `INITIALEXPIRATIONRAW` BIGINT        NOT NULL,
    `LASTEXPIRATIONRAW`    BIGINT        NOT NULL,
    `SCHEDULE`             VARCHAR(255)  NULL,
    `APPLICATIONID`        BIGINT        NOT NULL,
    CONSTRAINT `PK_EJB__TIMER__TBL` PRIMARY KEY (`TIMERID`)
);

Server configuration

  1. Start your GlassFish server admin console: usually http://localhost:4848
  2. On your left navigate to [Configurations] > [server-config] > [EJB Container]
  3. Then click the TAB up top [EJB Timer Service]
  4. Then fill out Timer Datasource: with your JDBC Resource eg. [mysql-pu]. ( Note: Defaults to jdbc/__TimerPool)
  5. Restart the Server

The result

...
INFO: [TimerBeanContainer] Created  TimerBeanContainer: TimerBean
INFO: EJB5181:Portable JNDI names for EJB TimerBean: [java:global/ejb-timer-service-app/TimerBean, java:global/ejb-timer-service-app/TimerBean!com.sun.ejb.containers.TimerLocal]
INFO: WEB0671: Loading application [ejb-timer-service-app] at [/ejb-timer-service-app]
INFO: EJB5109:EJB Timer Service started successfully for data source [mysql-pu]
INFO: Setting DBReadBeforeTimeout to false
INFO: ==> Restoring Timers ... 
INFO: There are no EJB Timers owned by this server
INFO: <== ... Timers Restored.
...

Still Broken? TimerService just stopped working (it used to work)?

So this happened to me after a day full of “deploy on save”. The TimerService was unavailable all of a sudden.

Severe:   Exception while loading the app
Severe:   Undeployment failed for context /ejb-timer-service-app
Warning:   Cannot deploy or load EJBTimerService: org.glassfish.deployment.common.DeploymentException: Error in linking security policy for ejb-timer-service-app -- Inconsistent Module State

Solution:

  • Stop the Glassfish server
  • Delete all the content from {GLASSFISH_HOME}/glassfish/domains/yourdomainname/generated
  • Restart Glassfish

It works, but everytime I need to deploy I need to follow this procedure again.

Errors

Out-of-Memory exception

By changing the -Xmx JVM options in GLASSFISH_HOME/domains/domain1/config/domain.xml will solve the problem. Default value is <jvm-options>-Xmx512m</jvm-options>, set a higher value like <jvm-options>-Xmx1024m</jvm-options> instead.

If you have already done that or if it doesn’t resolve the OutOfMemoryError, then try and get a heap dump when the OOME occurs, then analyze it. To get a heap dump, add in domain.xml the following option: <jvm-options>-XX:+HeapDumpOnOutOfMemoryError</jvm-options>. This will cause Glassfish to generate a heap dump file with a .hprof extension. Once you have this file, you can analyze it with a tool such as Eclipse Memory Analyzer Tool to find out which object allocation is causing the JVM to throw an OOME.

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Solution:

  1. openssl s_client -connect server:port -showcerts
  2. copy the text from cert you want into a file
  3. keytool -import -trustcacerts -file [-keystore mystore]

Example:

Grab the server certificate and save it to yourcert.pem file

openssl s_client -showcerts -connect sns.eu-west-1.amazonaws.com:443

or directly save it to yourcert.pem file

echo | openssl s_client -connect yoursever:port 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > yourcert.pem

Copy the text from cert you want into a file. In this example name it as yourcert.pem

Import root and intermediate certificates to the trusted root certificate

For JAVA

sudo keytool -import -alias <server_name> -keystore $JAVA_HOME/lib/security/cacerts -file yourcert.pem -storepass changeit

For GlassFish

keytool -import -alias <server_name> -keystore ${GLASSFISH_HOME}/glassfish/domains/domain1/config/cacerts.jks -file <server_name>.cert -storepass changeit

And restart the GlassFish server.

Other useful commands

#To Delete a Certificate by Using keytool
sudo keytool -delete -noprompt -alias aws -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit
#To get list of certificates
keytool -list -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit > list.txt
keytool -v -list -keystore ${GLASSFISH_HOME}/glassfish/domains/domain1/config/keystore.jks > list.txt

java.lang.NoSuchMethodError: sun.security.ssl.Handshaker.setApplicationProtocols([Ljava/lang/String;)V

The problem is that Glassfish embeds an old version of the SUN implementation (sun.security.ssl.SSLSessionImpl). Removing it will resolve the problem.

$ cd ${GLASSFISH_HOME}/glassfish/modules/endorsed
#View the content of grizzly-npn-bootstrap.jar
$ jar -tf grizzly-npn-bootstrap.jar
#Remove "sun" directory and all directories under it from grizzly-npn-bootstrap.jar
$ zip -d grizzly-npn-bootstrap.jar sun\*

other commands to remove a file or a folder

#Remove a file from a jar
$ zip -d file.jar unwanted_file.txt
#Remove a directory from a jar
$ zip -d file.jar unwanted_folder/

TLS ALPN (Application-Layer Protocol Negotiation) support is not available. HTTP/2 support will not be enabled.

Error generating artifacts for the following WSDL []. Possible causes can be invoking https when the application is not configured for security.

When try to go to the local SOAP URL both ?WSDL and ?Tester links work fine.

When go to the remote SOAP URL (JAX-WS) ?WSDL link works, but ?Tester is not due to this error.

The problem is caused by the default setting restricting access to External Schema. By default in IDE we use -Djavax.xml.accessExternalSchema=all as vmArg in IDE’s settings or in maven pom file.

If you are deploying to the GlassFish Server you need to modify the configuration file of the GlassFish Server (domain.xml) to enable the server to access external schemas to parse the wsdl file and generate the test client. To enable access to external schemas, open the GlassFish configuration file (GLASSFISH_INSTALL/glassfish/domains/domain1/config/domain.xml) and add the following JVM option element. You will need to restart the server for the change to take effect.

</java-config>
  ...
  <jvm-options>-Djavax.xml.accessExternalSchema=all</jvm-options>
</java-config>

Connection could not be allocated because: Cannot open file:GLASSFISH_HOME/glassfish/domains/domain1/config/keystore.jks

Add allowPublicKeyRetrieval=true and useSSL=false in GlassFish Admin Console JDBC > JDBC Connection Pools > Additional Properties.

or add them to glassfish-resources.xml

<jdbc-connection-pool 
  ...
  <property name="allowPublicKeyRetrieval" value="true"/>
  <property name="useSSL" value="false"/>
</jdbc-connection-pool>

or import certificate into glassfish and change the master password. In domain.xml, the following 2 lines needed to be added in the jvm-options area.

<jvm-options>-Djavax.net.ssl.keyStorePassword=[password]</jvm-options>
<jvm-options>-Djavax.net.ssl.trustStorePassword=[password]</jvm-options>

The Master Password is required to start the domain. No console, no prompting possible. You should either create the domain with –savemasterpassword=true or provide a password file with the –passwordfile option. Command start-domain failed.

This error occurs after running glassfish image on docker compose.