Troubleshooting: reviewing ORDS connections, your application server, and response times

by

Symptom/Issue

In an internal Slack thread today, a user was trying to diagnose browser latency while attempting to connect to the ORDS landing page.

Peter suggested what I thought was a pretty neat heuristic for checking connections to ORDS and your database, as well as latency.

Methodology

Let’s say your symptoms are either slow loading or intermittent loss of service when attempting to reach an ORDS landing page. Your immediate thought might be to refresh your browser or bounce the ORDS server (if running in Standalone mode). But this isn’t practical in a production environment, nor is it practical if you have ORDS deployed on a Weblogic Server or Apache Tomcat (you can deploy using any of those three servers).

Quick Heuristic

Use cURL to test two different endpoints, compare their response times to each other, and compare to the response time you’ve observed historically1.

URL one

This first URL points to where your ORDS instance “lives.” 

https://[my host]/ords

“Lives,” what does that mean? If you’re deploying to Weblogic Server ORDS would “live” in the $WEBLOGIC_HOME/application directory. In Apache Tomcat it is the $CATALINA/webapps directory, and on Standalone (using the ORDS embedded Jetty server) that might be $HOME/[your ords product folder].

So if you execute a cURL command to that URL, you’re issuing a request to your application server. With that single command, you can determine:

  1. if it’s even up
  2. what the response is, and
  3. how fast that response is

Honestly, this sounds silly and obvious at first. But it costs you nothing to just rule out the application server as a culprit. The command2 I tested (with the response included): 

choina@MacBook-Pro-2 ~ % curl -v -s -w "\nTotal time: %{time_total}\n" http://localhost:8080/ords                               
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET /ords HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 301 Moved Permanently
< Location: /ords/
< Content-Length: 0
< 
* Connection #0 to host localhost left intact

Total time: 0.001882

NOTE: Make sure you issue this cURL command WITHOUT a trailing slash after /ords. So like this: http://localhost:8080/ords, and NOT like this: http://localhost:8080/ords/. If you include the trailing slash, you'll force a redirect to the http://localhost:8080/ords/_/landing page (this just adds unnecessary time to your result).

URL two

The second cURL command you can issue is to the Metadata catalog for your/a target schema. In this example, I’m working with my ordsdemo user. He’s been REST-enabled, and I’ve already set up some Resource Modules (aka ORDS APIs). So I know there is “stuff” at that endpoint.

https://[my host]/ords/[target schema]/metadata-catalog/

Here is the cURL command I used (with the response included):

choina@MacBook-Pro-2 ~ % curl -v -s -w "\n\nTotal time: %{time_total}\n" http://localhost:8080/ords/ordsdemo/metadata-catalog/

* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET /ords/ordsdemo/metadata-catalog/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-ORDS_DEBUG: true
< X-Frame-Options: SAMEORIGIN
< Transfer-Encoding: chunked
< 
* Connection #0 to host localhost left intact
{"items":[{"name":"MOVIE","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/movie/"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/movie/","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/movie/","mediaType":"application/openapi+json"}]},{"name":"moviestream","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/mymovies/movie-all"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/mymovies/movie-all","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/mymovies/","mediaType":"application/openapi+json"}]},{"name":"moviestream","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/mymovies/movie-genre"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/mymovies/movie-genre","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/mymovies/","mediaType":"application/openapi+json"}]},{"name":"test_endpoints","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/no_objects/hello_auth"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/no_objects/hello_auth","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/no_objects/","mediaType":"application/openapi+json"}]},{"name":"test_endpoints","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/no_objects/hello_client_cred"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/no_objects/hello_client_cred","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/no_objects/","mediaType":"application/openapi+json"}]}],"hasMore":false,"limit":25,"offset":0,"count":5,"links":[{"rel":"self","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/"},{"rel":"first","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/"}]}

Total time: 0.030173

In the above example, we are hitting the database server (because we are grabbing the metadata catalog for the target schema). Consequently, the Total time for the round trip is slightly higher. Now, I can compare the two times against each other and against what I’ve seen historically. Are they reasonable? Do they pass the “sniff test”? Or do the results merit a deeper investigation?

Fin

I cannot stress this enough; this is just a simple heuristic to get you started. It takes 20 seconds to execute these commands. You’re really just “slicing the pie” at this stage. I honestly don’t think that simile applies here (unless you’ve ever cleared a room before), but you get the idea – thin slicing. Inching your way ever closer to the solution. This is just a part of the route-cause analysis.

That’s all for now. Keep this in your back pocket. If you have your own heuristics or troubleshooting tips, leave them in a comment below. I’d love to hear how you diagnose issues related to networking, connectivity, latency, etc.

Follow

And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

  1. I was hesitant to include this last part, “you’ve observed historically,” because I don’t give a metric. Honestly, it depends on your workload. But if you are doing any sort of performance testing you probably have at least a good sense of what response times should be like for your application (I’m talking directly to developers here). Even if you don’t know these expected response times, you can still compare the individual results against each other: the first URL vs the second URL, in this example. ↩︎
  2. What do the optional -v -w -s variables mean? Good question -v is shorthand for “Verbose;” which means print out as much info as is available. The -s is shorthand for “Silent;” in other words, don’t show any progress bars. And the -w means “Write Out,” in this case, “explicitly write out the Total time for me please.” ↩︎

Leave a Comment