If you get unexpected HTTP response codes while using or integrating with Gateway, use these tips to troubleshoot the cause of the responses. An example of an unexpected response is if permission is denied (HTTP 403) on an object believed to be accessible.

Tip

Enabling Swarm Storage audit logs (Configuring External Logging) allows tracking all requests the Gateway sends related to a client request.

Request ID in Responses

To aid with tracking transactions, all Gateway HTTP responses contain a header with the request ID. This request ID is also recorded in the Gateway's server log file and the audit log file. Searching the server log file for a given request ID shows the processing steps that took place during the handling of the request.

These are three examples where the request ID is found in a client response:

SCSP Response Header

Gateway-Request-Id: 375400C95338546F

S3 Response Header

x-amz-request-id: 375400C95338546F

S3 Error Response Body

<RequestId>375400C95338546F</RequestId>

Request ID in Logs

Log Level

You must enable debug-level logging on Gateway to see these log entries. See Gateway Logging

To diagnose authentication and authorization issues (e.g., S3 signature errors) in Gateway 8.0.0 and later, set the Root logLevel to debug. Additionally, update the com.caringo.gateway.auth level at the end of logging.yaml from info to debug:

        ## Extra volume knobs, as these can be quite noisy.
        - name: com.caringo.gateway.auth
          level: debug  # default is info

Search for the request ID in the Gateway server log:

grep "375400C95338546F" /var/log/cloudgateway/server.log

The search results from the log show:

Request URI and whether an Authorization or cookie header is on the request
Action being performed, such as CreateDomain, GetObject
Owner of the context for the request
Merged Policy document used to evaluate authorization
LDAP search filter used for user or group lookups
Reason for the HTTP response

The merged Policy is normally a combination of the root, domain, and bucket policies. An example log entry showing the context owner and a merged Policy document is:

2014-03-31 11:12:32,442 DEBUG [qtp1994043452-35|C66CF2A1D4DD4C8D] 
	Auth: AUTHENTICATING: 'ldap john@' 
	Action is GetObject, user idsys is ldap (root), context owner is john@ and merged policy is:
	[Sid=1 Allow [AllActions] "/ *" {group=[CloudAdmins]} {}]

When using the Policy conditions, such as the Referer header restrictions, the merged Policy that is logged is the one used to evaluate permissions for the request. Check the condition statements to see which is being used if the expected portion of the Policy is not displaying.

Additional error details are contained with the HTTP response header Gateway-ErrorDetails and are also logged in the Gateway server log. An example of this type of log message is:

2012-10-19 08:41:28,327 DEBUG [qtp596850781-35 - /reports?domain=example.com| F09B3F5FCA0A477F] 
	Auth: Request failed: 403 User is not allowed and is not owner. 
	owner: john, user: george, dn: uid=george,ou=people,dc=example,dc=com

In the previous example, the user george, the full LDAP DN is given, is not allowed to perform the requested action within the /reports bucket because he is not the owner, john, and because there is no Policy that grants him permission.

If necessary for application debugging, the Gateway can dump the HTTP request headers received with each request. To enable request header logging, add the following setting to the gateway.cfg file and restart the Gateway process:

[debug]
showRequest = true

Tip

Use debug sparingly! It can produce a significant amount of extra information in the server log, including security-related Authorization and Cookie headers.

An example of the request headers in the log output is:

2013-08-30 15:23:24,804 DEBUG [qtp1872474714-40|F69277697D792B98]
	Auth: REQUEST: POST /?domain=john.example.com AUTHORIZATION 
	ContentType: application/castorcontext 
	workaround-content-type: application/ castorcontext 
	Host: 172.10.8.5:8084 
	Content-Length: 0 
	Accept: */ * 
	User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8x zlib/1.2.5 
	Authorization: Basic am9objpwYXNzd29yZA==

Unexpected HTTP Responses