For the SGD gateways and servers to communicate, each server stores self-signed certificates in a java keystore to ensure communication originates from the correct host. When a gateway is added to an SGD array the information is synchronized to all other array members, whereas each gateway has its own configuration.

These instructions are for troubleshooting only and are supposed to provide some background information. They are not meant to be run on a daily basis. Output may vary, based on configuration

Check the SSL fingerprints for gateways on the array

In order for the SGD array to accept incoming connections from a SGD gateway, the gateways internal (self-signed) certificate needs to be added to the array. This is usually accomplished by first exporting the certificate on the gateway

Gateway

# /opt/SUNWsgdg/bin/gateway cert export --certfile /tmp/gw.pem && chmod +r /tmp/gw.pem

and then import that certificate on the SGD array

Array

# /opt/tarantella/bin/tarantella gateway add --name "gateway" --certfile /tmp/gw.pem

On the SGD array all the gateway certs are stored in a Java keystore at /opt/tarantella/var/info/gatwaycerts

Check if certificates match

The following command lists all configured gateways on the array, but only shows when the certificate will expire and not the fingerprint of any other information that would allow us to compare

tarantella gateway list

# tarantella gateway listInstalled gateway: sgdgw550-2Issuer: CN=sgdgw550-2.sub02112115000.sgd.oraclevcn.com, O=Oracle Corporation Inc, ST=CA, C=USSerial Number: 717042666Subject: CN=sgdgw550-2.sub02112115000.sgd.oraclevcn.com, O=Oracle Corporation Inc, ST=CA, C=USValid from Wed Feb 05 20:06:54 GMT 2020 to Fri Dec 14 20:06:54 GMT 2029Installed gateway: sgdgw550Issuer: CN=sgdgw550.sub02112115000.sgd.oraclevcn.com, O=Oracle Corporation Inc, ST=CA, C=USSerial Number: 724823471Subject: CN=sgdgw550.sub02112115000.sgd.oraclevcn.com, O=Oracle Corporation Inc, ST=CA, C=USValid from Thu Apr 04 20:15:51 GMT 2019 to Sat Feb 10 20:15:51 GMT 2029

To get the fingerprint of the certificates we can list the keystore on the array

keytool on the array

# true|keytool -list -keystore /opt/tarantella/var/info/gatewaycerts                 Enter keystore password: *****************  WARNING WARNING WARNING  ****************** The integrity of the information stored in your keystore  ** has NOT been verified!  In order to verify its integrity, ** you must provide your keystore password.              ******************  WARNING WARNING WARNING  *****************Keystore type: jksKeystore provider: SUNYour keystore contains 2 entriessgdgw550, Jan 16, 2020, trustedCertEntry,Certificate fingerprint (SHA1): A3:79:62:62:1D:67:E7:8A:B8:BD:DE:B3:45:F6:CE:D6:63:D4:7A:18sgdgw550-2, Feb 5, 2020, trustedCertEntry,Certificate fingerprint (SHA1): 88:58:AA:BB:A5:93:AF:A0:AB:66:31:4E:4E:8B:8F:3D:32:74:3F:5B

In order to compare the certificate from the gateway with what the array has stored we need to get the fingerprint on the gateways

keytool on the gateways

[root@sgdgw550 ~]# keytool -list -keystore /opt/SUNWsgdg/proxy/etc/keystore -storepass $(cat /opt/SUNWsgdg/etc/password) -alias osgd_gatewayosgd_gateway, Apr 4, 2019, PrivateKeyEntry,Certificate fingerprint (SHA1): A3:79:62:62:1D:67:E7:8A:B8:BD:DE:B3:45:F6:CE:D6:63:D4:7A:18[root@sgdgw550-2 ~]# keytool -list -keystore /opt/SUNWsgdg/proxy/etc/keystore -storepass $(cat /opt/SUNWsgdg/etc/password) -alias osgd_gatewayosgd_gateway, Feb 5, 2020, PrivateKeyEntry,Certificate fingerprint (SHA1): 88:58:AA:BB:A5:93:AF:A0:AB:66:31:4E:4E:8B:8F:3D:32:74:3F:5B

Check SGD server certificates on the gateway

When using SGD gateways with an array, there is a possible scenario that can happen which is particularly difficult to troubleshoot: the self-signed SSL certificate of an SGD array member can change or expire, leaving the gateway unable to communicate.

Problem

When first setting up a gateway, every single SGD server in the array has to be added.

# /opt/SUNWsgdg/bin/gateway server add \ --server <server name> \ --url <how the gateway can reach the server> \ --certfile <server name:/opt/tarantella/var/tsp/cert.pm> \ --ssl-certfile <server name:/opt/tarantella/var/info/certs/PeerCAcert.pem>

We can always list all the array members configured on a gateway with the following command

# /opt/SUNWsgdg/bin/gateway server listSGD Server: primary550.sub02112115000.sgd.oraclevcn.comURL: https://primary550.sub02112115000.sgd.oraclevcn.comCA Certificate:   Owner: CN=primary550.sub02112115000.sgd.oraclevcn.com CA Cert   Issuer: CN=primary550.sub02112115000.sgd.oraclevcn.com CA Cert   Serial number: 8d1cd3afce6290f3   Valid from: Thu Jan 16 22:28:19 GMT 2020 until: Sun Jan 13 22:28:19 GMT 2030SSL Certificate:   Owner: CN=primary550.sub02112115000.sgd.oraclevcn.com, O="Oracle Corporation, Inc.", ST=CA, C=US   Issuer: CN=primary550.sub02112115000.sgd.oraclevcn.com, O="Oracle Corporation, Inc.", ST=CA, C=US   Serial number: d00f5ded14639c5f   Valid from: Thu Jan 16 22:28:23 GMT 2020 until: Fri Jan 15 22:28:23 GMT 2021SGD Server: secondary550.sub02112115000.sgd.oraclevcn.comURL: https://secondary550.sub02112115000.sgd.oraclevcn.comCA Certificate:   Owner: CN=secondary550.sub02112115000.sgd.oraclevcn.com CA Cert   Issuer: CN=secondary550.sub02112115000.sgd.oraclevcn.com CA Cert   Serial number: e75274ffec4f45da   Valid from: Wed Jan 22 18:09:16 GMT 2020 until: Sat Jan 19 18:09:16 GMT 2030SSL Certificate:   Owner: CN=secondary550.sub02112115000.sgd.oraclevcn.com, O="Oracle Corporation, Inc.", ST=CA, C=US   Issuer: CN=secondary550.sub02112115000.sgd.oraclevcn.com, O="Oracle Corporation, Inc.", ST=CA, C=US   Serial number: 92d9a17132af1ca9   Valid from: Wed Jan 22 18:09:20 GMT 2020 until: Thu Jan 21 18:09:20 GMT 2021

The output of these commands only show what is, and not what should be.

In case an existing server in the array is being upgraded or changed or the /opt/tarantella/bin/tarantella security command is being used and the specific SGD servers certificates change, a gateway with stale information will not be able to launch emulator sessions on that SGD server in the array.

With multiple gateways load balancing between multiple array members this becomes difficult to troubleshoot because every application launch can be on a different server and thus randomly fail or work.

Solution

The following script can be run on a SGD gateway to check if the information is in fact correct.

#!/bin/bash## use openssl to get the certificate for CA and SSL from SGD server#export INSTDIR=/opt/SUNWsgdgexport KEYSTORE=${INSTDIR}/proxy/etc/keystoreexport STOREPASS=$(cat ${INSTDIR}/etc/password)export OPENSSL=/usr/bin/openssl;[ -x ${INSTDIR}/bin/bin/openssl ] && OPENSSL=${INSTDIR}/bin/bin/opensslexport KEYTOOL=/usr/bin/keytool;[ -x ${INSTDIR}/bin/jre/bin/keytool ] && KEYTOOL=${INSTDIR}/bin/jre/bin/keytool[ "$(uname -s)" == "SunOS" ] && PATH=/usr/ucb:$PATHfunction die() {    echo $1    exit -1}function get_gw_ssl_fingerprint() {    server=$1    true ""|${KEYTOOL} -list -v -storepass "${STOREPASS}" -keystore ${KEYSTORE} -alias ${server}-ssl 2>/dev/null > /tmp/gw.${server}.ssl    cat /tmp/gw.${server}.ssl|awk '/SHA1:/{print $2}'}function get_gw_ca_fingerprint() {    server=$1    true ""|${KEYTOOL} -list -v -storepass "${STOREPASS}" -keystore ${KEYSTORE} -alias ${server} 2>/dev/null > /tmp/gw.${server}.ca    cat /tmp/gw.${server}.ca| awk '/SHA1:/{print $2}'}function extract_certificate() {    server=$1    port=$2    #awk 'BEGIN{printline=0}/^-----BEGIN CERTIFICATE-----/{printline=1}/^-----END CERTIFICATE-----/{printline=0;print $0}{if(printline) print $0}'    awk 'BEGIN{n=0}/-----BEGIN/{f=sprintf("/tmp/srv.%d.%d.%s", port, n++, server)} f{print>f} /-----END/{f=""}' port=${port} server=${server}}function get_server_cert(){    server=$1    port=$2    alg=$3    true | ${OPENSSL} s_client -connect ${server}:${port} 2>/dev/null > /tmp/srv.${port}.${server}    ${OPENSSL} x509 -noout -fingerprint -${alg} -inform pem 2>/dev/null -in /tmp/srv.${port}.${server} | awk -F= '{print $2}'}function get_srv_ca_fingerprint(){    server=$1    CERTTMP=/tmp/peerca.$server    curl -ks --head --fail https://${server}/peerca >/dev/null && curl -o $CERTTMP -sk https://${server}/peerca    if [ -f $CERTTMP ];then        ${OPENSSL} x509 -noout -fingerprint -${alg:-sha1} -inform pem -in $CERTTMP 2>/dev/null | awk -F= '{print $2}'    else        echo "N/A"        echo "----------------------------------------------------------------------------------">&2        echo "https://${server}/peerca does not exist" >&2        echo "login to $server and issue the following commands as root" >&2        echo "cd /opt/tarantella/var/docroot">&2        echo "chmod +r ../info/certs/PeerCAcert.pem && ln -s ../info/certs/PeerCAcert.pem peerca">&2        echo "----------------------------------------------------------------------------------">&2    fi    rm -f $CERTTMP}function check_server(){    server=$1    #echo "checking ${server}">&2    GW=$(get_gw_ssl_fingerprint $server)    SRV=$(get_server_cert $server 443 sha1)    [ "${GW}" == "${SRV}" ] || echo "$server GW=${GW} and SRV=${SRV} SSL fingerprint do not match"    GW=$(get_gw_ca_fingerprint $server)    SRV=$(get_srv_ca_fingerprint $server)    [ "${GW}" == "${SRV}" ] || echo "$server GW=${GW} and SRV=${SRV} CA fingerprint do not match"}[ "$(whoami)" == "root" ] || die "this must be run by root"[ -d /opt/SUNWsgdg ] || die "this script needs to run on the gateway(s)"SERVERS=$(awk -F= '/^[^#]+/{print $2}' $INSTDIR/etc/serveridmap.properties)pids=""servers=""for server in ${SERVERS};do    if [ "$(${INSTDIR}/bin/bin/ttahostprobe ${server}:443)" == "y" ];then        servers=(${servers[*]} $server)        check_server $server & pid=$!        pids=(${pids[*]} $pid)    else        echo "${server} not online">&2    fidoneecho "waiting for ${servers[*]}"wait ${pids[*]}rm -f /tmp/gw.* /tmp/srv.* /tmp/peerca.*