DataCore Swarm log4j Remediation (CVE-2021-44228)

Owned by John Bell
Last updated: Dec 29, 2022 by Kyle Miller (Deactivated)
3 min read

The DataCore Swarm Engineering Team has made Swarm 14 product updates (2021-12-16) for the log4j vulnerability CVE-2021-44228 and has verified workarounds that resolve the issue in all current and previous versions. The Gateway and Elasticsearch software are affected (CLOUD-3482, SWAR-9322).

The guidance per Gateway version is:

  • Gateway 5.4.1 (Swarm 9, 10, 11) and earlier doesn't use log4j2, it installs log4j-1.2.17.jar and as a result is unaffected

  • Gateway 6.0.0+ (Swarm 10, 11, 12) should apply the workaround below

  • Gateway 7.5+ (Swarm 14) can apply the workaround below or upgrade to Gateway 7.7.1-2 released 2021-12-16 (in the Swarm-v14.0-20211216 bundle) which updates log4j to 2.16.0, which resolves all known vulnerabilities.

The Gateway workaround involves two parts:

  1. You will need to upgrade your JRE on the Gateway servers. Under CentOS 7 you can upgrade to the latest JRE by issuing yum upgrade java-1.8.0-openjdk (which as of this writing should upgrade to 1.8.0_312). You can run java -version to confirm.

  2. Once your JRE is on the latest version, you will need to make the following change to the /etc/sysconfig/cloudgateway file:

    JAVA_EXTRA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

Once the above two changes are made on each of your Gateway servers, you must then restart the Gateway service with systemctl restart cloudgateway. After that, you can run some simple tests to make sure Gateway is functioning properly.

Confirmation of the options taking hold can be performed by looking at the Java process running in the process table (example):

# ps auxwwf | grep java root 22258 109 1.4 4345096 231080 ? Sl 09:57 0:14 _ /usr/bin/java -Xms1024m -Xmx1024m -Dpid.file=/var/run/cloudgateway.pid -Dnodaemon=true -Dgateway.config=/etc/caringo/cloudgateway/gateway.cfg -Dlogging.config=/etc/caringo/cloudgateway/logging.yaml -Dgateway.error=/var/log/caringo/cloudgateway_server.error -Djava.util.concurrent.ForkJoinPool.common.parallelism=200 -Dlog4j2.formatMsgNoLookups=true -cp /opt/caringo/cloudgateway/lib/* com.caringo.gateway.Main

Additional Notes:

  • It's best practice to periodically “yum update" to be sure you're using the latest Java 1.8.0 runtime

  • If your Java version is 2018's 8u191 or later you should be protected from the worst exploits, but reference best practice guidance per previous

The guidance per Elasticsearch version is:

Elasticsearch by default protects against remote code execution and information leakage. Also, it should never be exposed directly to users in Swarm deployments. But it is good to apply the property workaround Elastic mentions here. Do not upgrade elasticsearch, remain on 6.8.6 or 7.5.2.

https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

Simply add the property below to /etc/elasticsearch/jvm.options where the heap size is set. Do this on each of your elasticsearch nodes and systemctl restart elasticsearch. You can do a rolling restart (https://www.elastic.co/guide/en/elasticsearch/reference/7.5/restart-cluster.html#restart-cluster-rolling ) to avoid downtime.

# Xms represents the initial size of total heap space # Xmx represents the maximum size of total heap space -Xms31g -Xmx31g # Workaround CVE-2021-44228 -Dlog4j2.formatMsgNoLookups=true

Additional Notes:

  • The jvm.options workaround works with 7.5.2, 6.8.6 and 5.6.12.

  • Elasticsearch 7.5.2 bundles its own, recent, Java. It does not use the OS package “java-openjdk-1.8.0” so that package can be removed from elasticsearch 7.5.2 servers. Elasticsearch 6.8.6 and 5.6.12 do use the OS java, so it’s good to update it.

  • This workaround is unnecessary with the unsupported elasticsearch 2.3.3 since it does not use log4j2.

  • Recent elasticsearch documentation mentions a /etc/elasticsearch/jvm.options.d/ directory can be used. But the elasticsearch 6.8.6 and 7.5.2 that Swarm supports will not use that directory, so the jvm.options file must be updated.

  • Swarm 14.0 does not support versions later than 7.5.2, so do not upgrade elasticsearch to 7.16.1. If you accidentally upgrade elasticsearch to a release after 7.5.2 it cannot be downgraded, you will need to create a new elasticsearch cluster and wait for a new Swarm Search Feed to populate it. Support for later elasticsearch versions is coming in 2022.

  • If you originally used our configure_elasticsearch_with_swarm_search.py script to configure your elasticsearch 7.5.2 cluster you can use it to automatically make the jvm.options change and restart the service. Just upgrade to caringo-elasticsearch-search-7.0.3-1.noarch.rpm (released 2021-12-16) on each elasticsearch node and run configure_elasticsearch_with_swarm_search.py --upgrade . It will refresh your config files including jvm.options file, reinstall 7.5.2, and restart the service.

If you are still using the unsupported Elasticsearch 2.3.3 or 5.6.12 please file a support ticket and plan to upgrade in 2022. In-place upgrades of those old elasticsearch versions are not supported:

https://perifery.atlassian.net/wiki/spaces/public/pages/2443804878#HowtoUpgradeSwarm-UpgradingfromUnsupportedElasticsearch

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.