Table of Contents

style	none

Managing Log Levels in Swarm Cluster Using a Shell Script

Overview

This knowledge base entry provides detailed information about the set_swarm_log_level.sh script, which is used to set and manage the logging levels of Swarm API clusters. This script simplifies the process of changing log levels and allows users to revert changes after a specified duration if needed.

Purpose

The purpose of this script is to:

Change the log level of a Swarm cluster to a specified value.
Optionally revert the log level back to its original setting after a defined duration.
Provide an easy-to-use interface for managing log levels without needing extensive API knowledge.

Script Functionality

Input Parameters: The script accepts parameters for the Swarm API IP address, credentials, new log level, and optional duration for temporary changes.
Validation and Error Handling: It checks for required parameters and handles errors gracefully.
Logging Level Management: Retrieves the current log level, updates it if necessary, and optionally reverts it after a countdown.

Key Features

Dynamic Duration Input: If the duration parameter is provided without a value, the user is prompted to enter the duration in seconds.
User Feedback: The script provides informative output regarding the current state and actions being taken, including success and error messages.
Count Down Timer: If a duration is specified, the script displays a countdown in hours, minutes, and seconds.

Script Source Code

...

language	bash

...

The Swarm Log Level Management Script (castor-change-log-level.sh) is designed to manage and dynamically adjust log levels on a DataCore Swarm cluster, providing options to change the log level temporarily and automatically revert it after a specified duration. It supports background execution via screen or tmux, making it ideal for long-running operations that require detachment from the terminal.

Script Features

Set and Revert Log Levels: Temporarily change the log level and revert after a specified duration.
Flexible JSON Parsing: Uses jq for JSON parsing if available; defaults to jq otherwise.
Background Execution: Optionally runs in the background using screen or tmux.
Log Size Monitoring: Reports the log size generated during the temporary log level change.
Countdown Display: Shows a countdown for the specified duration.

Requirements

jq (optional): Used for parsing JSON responses; falls back to grep if unavailable.
screen or tmux (optional): Required for background execution.
Permissions: Ensure sufficient permissions to execute on the DataCore Swarm server and access required files.

Script Usage

Code Block
./castor-change-log-level.sh -d <node_ip> -p <admin:password> -i <new_log_level> [-t <duration_in_seconds>] [--background] [-v]

Parameter	Description
`-d, --swarm_ip`	IP address of the Swarm API endpoint (or set `SCSP_HOST` environment variable).
`-p, --credentials`	Admin credentials in the format `admin:password`.
`-i, --log.level`	New log level to set (values: `10, 15, 20, 30, 40, 50`).
`-t, --time`	Duration in seconds to keep the new log level (optional).
`--persistent`	Runs the script in a detached session using `screen` or `tmux`, allowing continued operation if terminal session ends.
`-v, --verbose`	Enables verbose mode to display debug information.

Instruction for Use

Example 1: Set log level to 20 and keep it for 10 seconds

Code Block
./castor-change-log-level.sh -d 192.168.8.84 -p admin:datacore -i 20 -t 10

Example 2: Run in background mode with verbose logging

Code Block
./castor-change-log-level.sh -d 192.168.8.84 -p admin:datacore -i 20 -t 30 --background -v

Behavior

Log Level Change: Sets the log level to the specified value. If the current log level matches the requested level, the script skips the update.
Countdown: During the specified duration, the script displays a countdown every second.
Revert Log Level: After the countdown, the log level reverts to the initial value.
Log Size Report: Provides approximately log size generated during the temporary log level change.
Debug Mode: When -v is specified, debug messages display the script's internal operations.

Output Messages

Message	Description
`Swarm IP:`	Displays the specified Swarm IP address.
`Credentials:`	Credentials are masked for security.
`Cluster Name:`	Displays the cluster name retrieved from the Swarm API.
`New log level:`	Shows the new log level requested.
`Current log level:`	Displays the current log level.
`Updating log level to X...`	Indicates the beginning of the log level update process.
`Log level changed successfully...`	Confirms that the log level was successfully updated.
`Keeping log level at X for Y...`	Shows the temporary period for which the new log level is retained, with a countdown.
`Time's up! Reverting log level...`	Indicates that the temporary period has ended and the script is reverting the log level.
`Approximate X new logs generated...`	Provides information on the amount of logging activity generated during the temporary log level.

Example Output

Code Block

[root@scs dist]# ./castor-change-log-level.sh -p admin:datacore -i 10 -t 300
Swarm IP: 192.168.1.84
Credentials: [hidden for security]
Cluster Name: gatewayadmindomain

New log level: 10
Current log level is 30.
Updating log level to 10...
Log level changed successfully from 30 → 10.
Keeping log level at 10 for 300 second(s)...
Countdown: 00:00:01 remaining...

Time's up! Reverting log level back to 30...
Approximate 69.4MB new logs were generated at log level 10. Current castor.log size is 371.3MB after 00:05:00.
Log level reverted successfully back to 30.

[root@scs dist]#

Error Handling

Missing Parameters: Missing parameters prompt a usage message.
Invalid Duration: If a non-numeric duration is provided, you’re prompted to enter a valid duration in seconds.
Connection Issues: If unable to connect to the Swarm API, check the IP, credentials, and network access.

Notes

Credentials are masked in the output for security.
Log file sizes are shown in human-readable format (GB, MB, KB, B).

This script provides administrators with an effective way to adjust and monitor Swarm logging, supporting both temporary and permanent log level changes for troubleshooting and performance monitoring.

Script Source Code

Latest version: castor-change-log-level.sh

Code Block

language	bash

#!/bin/bash
# Written by Milton Suen (milton.suen@datacore.com) Oct 31, 2024
# Revision: Update to support running the script in a persistent session using screen or tmux.

# Function to display usage information
usage() {
    echo "Usage: $0 -d swarm_ip -p admin:password -i new_log_level [-t duration_in_seconds] [--persistent] [-v]"
    echo "  -d, --swarm_ip           IP address of the Swarm API endpoint (or set SCSP_HOST environment variable)"
    echo "  -p, --credentials        Credentials in the format admin:password"
    echo "  -i, --log.level          New log level to set"
    echo "  -t, --time               Duration in seconds to keep the new log level (optional)"
    echo "  --persistent             Run the script in a detached session using screen or tmux"
    echo "  -v, --verbose            Enable verbose mode for debug messages"
    exit 1
}

# Default options
persistent=false
verbose=false
output_log="script_output.log"  # Log file for capturing persistent session output

# Function to display debug messages if verbose mode is enabled
debug() {
    if $verbose; then
        echo "[DEBUG] $1"
    fi
}

# Function to check if either 'screen' or 'tmux' is installed
check_screen_or_tmux() {
    if ! command -v screen &>/dev/null && ! command -v tmux &>/dev/null; then
        echo "Error: Neither 'screen' nor 'tmux' is installed. Cannot run in persistent mode."
        persistent=false  # Disable persistent session
    fi
}

# Function to format file size
format_size() {
    local size=$1
    if (( size >= 1073741824 )); then
        echo "$(awk "BEGIN {printf \"%.1fGB\", $size/1073741824}")"
    elif (( size >= 1048576 )); then
        echo "$(awk "BEGIN {printf \"%.1fMB\", $size/1048576}")"
    elif (( size >= 1024 )); then
        echo "$(awk "BEGIN {printf \"%.1fKB\", $size/1024}")"
    else
        echo "${size}B"
    fi
}

# Function to format duration
format_duration() {
    local duration=$1
    local hours=$((duration / 3600))
    local minutes=$(( (duration % 3600) / 60 ))
    local seconds=$((duration % 60))
    printf "%02d:%02d:%02d" $hours $minutes $seconds
}

# Function to check if jq is available and set up JSON parsing method
check_jq() {
    if [[ -x "/usr/local/bin/jq" ]]; then
        echo "/usr/local/bin/jq"
    elif [[ -x "$(pwd)/jq" ]]; then
        echo "$(pwd)/jq"
    elif command -v jq &>/dev/null; then
        echo "jq"
    else
        echo "grep"
    fi
}

jq_or_grep=$(check_jq)

# Parse input arguments
while [[ "$#" -gt 0 ]]; do
    case $1 in
        -d|--swarm_ip) swarm_ip="$2"; shift 2 ;;
        -p|--credentials) credentials="$2"; shift 2 ;;
        -i|--log.level) new_log_level="$2"; shift 2 ;;
        -t|--time)
            if [[ -n "$2" && "$2" != -* ]]; then
                duration="$2"
                shift 2
            else
                read -p "Enter duration in seconds: " duration
                shift
            fi
            ;;
        --persistent) persistent=true; shift ;;
        -v|--verbose) verbose=true; shift ;;
        *) usage ;;
    esac
done

# Check if 'screen' or 'tmux' is installed
check_screen_or_tmux

# If swarm_ip is not provided, try using SCSP_HOST environment variable
if [[ -z "$swarm_ip" ]]; then
    if [[ -n "$SCSP_HOST" ]]; then
        swarm_ip="$SCSP_HOST"
        debug "Using Swarm IP from SCSP_HOST: $swarm_ip"
    else
        echo "Error: swarm_ip not provided and SCSP_HOST is not set."
        usage
    fi
fi

# Check if required arguments are provided
if [[ -z "$credentials" || -z "$new_log_level" ]]; then
    usage
fi

# Retrieve cluster name and handle JSON parsing
debug "Retrieving the cluster name from Swarm API."
if [[ "$jq_or_grep" == "grep" ]]; then
    clusterName=$(curl --user "$credentials" -sS "http://$swarm_ip:91/api/storage/clusters" | grep -oP '"name":\s*"\K[^"]+')
else
    clusterName=$(curl --user "$credentials" -sS "http://$swarm_ip:91/api/storage/clusters" | "$jq_or_grep" -r '._embedded.clusters[0].name')
fi

if [[ -z "$clusterName" ]]; then
    echo "Failed to retrieve the cluster name. Please check your inputs."
    exit 1
fi
debug "Cluster Name: $clusterName"

# Main logic function to run the script tasks
main_script() {
    local swarm_ip="$1"
    local credentials="$2"
    local new_log_level="$3"
    local duration="$4"
    local clusterName="$5"
    local log_file="/var/log/datacore/castor.log"
    local initial_size=$(stat -c%s "$log_file" 2>/dev/null || echo 0)
    local current_log_level
    local jq_or_grep="$6"

    # Display initial information
    echo "Swarm IP: $swarm_ip"
    echo "Credentials: [hidden for security]"
    echo "Cluster Name: $clusterName"

    debug "Starting main_script function..."

    # Retrieve current log level
    if [[ "$jq_or_grep" == "grep" ]]; then
        current_log_level=$(curl --user "$credentials" -sS "http://$swarm_ip:91/api/storage/clusters/$clusterName/settings/log.level" | grep -oP '"value":\s*\K[0-9]+')
    else
        current_log_level=$(curl --user "$credentials" -sS "http://$swarm_ip:91/api/storage/clusters/$clusterName/settings/log.level" | "$jq_or_grep" -r '.value')
    fi
    echo ""
    echo "New log -i, --log.levellevel: $new_log_level"
    echo "Current log level is New $current_log level to set"
    echo "  -t, --time_level."

    # Skip update if new level matches the current level
 Duration in seconds toif keep the new log level (optional)"[[ "$current_log_level" -eq "$new_log_level" ]]; then
     exit 1 } echo #""
Parse input arguments while [[ "$#"  -gt 0 ]]; do
    case $1 in
echo "Log level is already set to $new_log_level. No changes made."
       -d|--swarm_ip) swarm_ip="$2"; shift ;; return
    fi

    -p|--credentials) credentials="$2"; shift ;;
  # Update the log level using PUT
    echo "Updating -i|--log. level) to new$new_log_level="$2"; shift ;;..."
    response=$(curl --user "$credentials" -sS -X PUT -t|--time)
     H "Content-Type: application/json" \
      if [[ -n "$2" && "$2" != -* ]]; then
    "http://$swarm_ip:91/api/storage/clusters/$clusterName/settings/log.level" \
        -d "{\"value\": $new_log_level}")

    if [[     duration="$2"
 "$jq_or_grep" == "grep" ]]; then
        updated_log_level=$(echo "$response" | grep   shift
     -oP '"value":\s*\K[0-9]+')
      else
        updated_log_level=$(echo "$response"       read -p "Enter duration in seconds: " duration
| "$jq_or_grep" -r '.value')
    fi

    if fi
        [[ "$updated_log_level" -eq "$new_log_level" ]]; then
   ;;     echo "Log level changed *)successfully usage ;;
 from $current_log_level → $new_log_level."
  esac  else
  shift done  # Check if requiredecho arguments"Failed areto providedupdate iflog [[ -z "$swarm_ip" || -z "$credentials" || -z "$new_log_level" ]]; then
level. Response: $response"
        exit 1
   usage fi

#
Retrieve the cluster name clusterName=$(curl -u admin:caringo -sS "<http://$swarm_ip:91/api/storage/clusters>" | grep -oP '"name":\s*"\K[^"]+')# Countdown and revert log level
    if [[ -zn "$duration" && "$clusterName$duration" -gt 0 ]]; then
        echo "FailedKeeping tolog retrievelevel the cluster name. Please check your inputsat $new_log_level for $duration second(s)..."
    exit 1 fi  # Convert duration to an integer if it is set
if [[ -n "$duration" ]]; then
    if ! [[ "$duration" =~ ^[0-9]+$ ]]; then
 for ((i=duration; i>0; i--)); do
            printf -v countdown "%02d:%02d:%02d" $((i/3600)) $(( (i%3600) / 60 )) $((i%60))
      echo "Error: Duration must be a positiveecho integer value in seconds."
  -ne "Countdown: $countdown remaining...\r"
     exit 1     fi fisleep 1
# Display input parameters echo "Swarm IP: $swarm_ip"
echo "Credentials: [hidden for security]"
echo "New log level: $new_log_level"
echo "Cluster Name: $clusterName"

# Get the current done
        echo -e "\n\nTime's up! Reverting log level echoback "Retrieving the current log to $current_log_level..."
current_log_level

        response=$(curl --uuser "$credentials" -sS -X "<httpPUT -H "Content-Type: application/json" \
            "http://$swarm_ip:91/api/storage/clusters/$clusterName/settings/log.level>level" |\
jq '.value')  # Check if the current log level was retrieved successfully if [[ -z "d "{\"value\": $current_log_level}")
]];
then     echo "Failed to retrieveif the current log level. Please check your inputs."[[ "$jq_or_grep" == "grep" ]]; then
    exit 1 fi echo "Current log level is $currentreverted_log_level.=$(echo "$response" | #grep Check if the new log level is the same as the current log level
if [[ "$current_log_level" -eq "$new_log_level" ]]; then-oP '"value":\s*\K[0-9]+')
        else
     echo "Log level is already set to $newreverted_log_level. No changes made."
 =$(echo "$response" | "$jq_or_grep" -r '.value')
  exit 0 fi  # Update thefi
log
level using PUT echo "Updating log level to $new_log_level..."
responsefinal_size=$(curlstat -u "$credentials" -sS -X PUT -H "Content-Type: application/json" \
    "<http://$swarm_ip:91/api/storage/clusters/$clusterName/settings/log.level>" \c%s "$log_file" 2>/dev/null || echo 0)
       -d "{\"value\": $new_log_level}")

# Verify if the log level was updated
updated_log_level=$(echo "$response" | jq '.value')
if [[ "$updated_log_level" -eq "$new_log_level" ]]; then size_diff=$(( final_size - initial_size ))
        size_diff_formatted=$(format_size "$size_diff")
        duration_formatted=$(format_duration "$duration")
    echo "Log level changed successfullyecho from"Approximate $current$size_logdiff_levelformatted → $new_log_level."
else
    echo "Failed to update log level. Response: $response"
    exit 1
fi

# If duration is specified, wait and revert after the specified timenew logs were generated at log level $new_log_level. Current castor.log size is $(format_size "$final_size") after $duration_formatted."

        if [[ -n "$duration$reverted_log_level" &&-eq "$duration" -gt 0 $current_log_level" ]]; then
            echo "Keeping log level at $newLog level reverted successfully back to $current_log_level for $duration second(s)..."
         # Countdownelse
loop     for ((i=duration; i>0; i--)); do   echo "Failed to revert log level. #Response: Calculate$response"
hours, minutes, and seconds         hours=$((i / 3600))exit 1
        fi
minutes=$(( (i % 3600) /else
60 ))       echo  seconds=$((i % 60))
   "Log level change is permanent until manually modified."
    fi
}

# Run in persistent or directly
#if Format$persistent; countdownthen
in hh:mm:ss   # Pass the main_script function to printfthe -vscreen countdown "%02d:%02d:%02d" $hours $minutes $seconds
  session and store the output in a file
    if echocommand -nev "Countdown: $countdown remaining...\r"screen &>/dev/null; then
        sleepscreen 1-dmS indexer_script bash -c  done
    echo -e "\nTime's up! Reverting log level back to $current_log_level..."
    
    # Revert to original log level
    response=$(curl -u "$credentials" -sS -X PUT -H "Content-Type: application/json" \
        "<http://$swarm_ip:91/api/storage/clusters/$clusterName/settings/log.level>" \"$(declare -f main_script format_size format_duration check_jq debug); main_script \"$swarm_ip\" \"$credentials\" \"$new_log_level\" \"$duration\" \"$clusterName\" \"$jq_or_grep\" | tee \"$output_log\""
        screen -r indexer_script
    elif command -v tmux &>/dev/null; then
        tmux new-d "{\"value\": $current_log_level}")

    reverted_log_level=$(echo "$response" | jq '.value')
    if [[ "$reverted_log_level" -eq "$current_log_level" ]]; then
        echo "Log level reverted successfully back to $current_log_level."session -d -s indexer_script "$(declare -f main_script format_size format_duration check_jq debug); main_script \"$swarm_ip\" \"$credentials\" \"$new_log_level\" \"$duration\" \"$clusterName\" \"$jq_or_grep\" | tee \"$output_log\""
        tmux attach-session -t indexer_script
    else
        echo "Failed to revert log level. Response: $responseError: Neither screen nor tmux available. Run without --persistent."
        exit 1
    fi
else
    echo# "LogWait levelfor change is permanent until manually modified."
fi

Usage Instructions

To use the script, run it from the command line with the appropriate parameters:

Code Block
./set_swarm_log_level.sh -d <swarm_ip> -p <admin:password> -i <new_log_level> [-t <duration_in_seconds>]

Examples

Change Log Level Permanently:

Code Block
. /set_swarm_log_level.sh -d 192.168.8.84 -p admin:datacore -i 30

Change Log Level Temporarily:

Code Block
./set_swarm_log_level.sh -d 192.168.8.84 -p admin:datacore -i 20 -t 300

Prompt for Duration:

Code Block
./set_swarm_log_level.sh -d 192.168.8.84 -p admin:datacore -i 10 -t

Troubleshooting

Ensure correct permissions and network access to the Swarm API.
Verify input parameters and check for typos.
Ensure that curl and jq are installed and available in the environment.

Conclusion

...

the screen session to complete and then display the output log
    sleep 1
    while screen -list | grep -q "indexer_script"; do
        sleep 1
    done

    echo ""
    cat "$output_log"
else
    main_script "$swarm_ip" "$credentials" "$new_log_level" "$duration" "$clusterName" "$jq_or_grep" | tee "$output_log"
fi

Version	Old Version 1	New Version Current
Changes made by	Milton Suen	Milton Suen
Saved on	Oct 25, 2024	Dec 04, 2024

Page Comparison

Versions Compared

Key

Managing Log Levels in Swarm Cluster Using a Shell Script

Overview

Purpose

Script Functionality

Key Features

Script Source Code

Script Features

Requirements

Script Usage

Instruction for Use

Behavior

Output Messages

Example Output

Error Handling

Notes

Script Source Code

Usage Instructions

Examples

Troubleshooting

Conclusion