Tag: troubleshooting

  • Troubleshooting: reviewing ORDS connections, your application server, and response times

    Troubleshooting: reviewing ORDS connections, your application server, and response times

    Symptom/Issue

    In an internal Slack thread today, a user was trying to diagnose browser latency while attempting to connect to the ORDS landing page.

    Peter suggested what I thought was a pretty neat heuristic for checking connections to ORDS and your database, as well as latency.

    Methodology

    Let’s say your symptoms are either slow loading or intermittent loss of service when attempting to reach an ORDS landing page. Your immediate thought might be to refresh your browser or bounce the ORDS server (if running in Standalone mode). But this isn’t practical in a production environment, nor is it practical if you have ORDS deployed on a Weblogic Server or Apache Tomcat (you can deploy using any of those three servers).

    Quick Heuristic

    Use cURL to test two different endpoints, compare their response times to each other, and compare to the response time you’ve observed historically1.

    URL one

    This first URL points to where your ORDS instance “lives.”

    https://[my host]/ords

    “Lives,” what does that mean? If you’re deploying to Weblogic Server ORDS would “live” in the $WEBLOGIC_HOME/application directory. In Apache Tomcat it is the $CATALINA/webapps directory, and on Standalone (using the ORDS embedded Jetty server) that might be $HOME/[your ords product folder].

    So if you execute a cURL command to that URL, you’re issuing a request to your application server. With that single command, you can determine:

    1. if it’s even up
    2. what the response is, and
    3. how fast that response is

    Honestly, this sounds silly and obvious at first. But it costs you nothing to just rule out the application server as a culprit. The command2 I tested (with the response included):

    choina@MacBook-Pro-2 ~ % curl -v -s -w "\nTotal time: %{time_total}\n" http://localhost:8080/ords                               
    * Host localhost:8080 was resolved.
    * IPv6: ::1
    * IPv4: 127.0.0.1
    *   Trying [::1]:8080...
    * Connected to localhost (::1) port 8080
    > GET /ords HTTP/1.1
    > Host: localhost:8080
    > User-Agent: curl/8.7.1
    > Accept: */*
    > 
    * Request completely sent off
    < HTTP/1.1 301 Moved Permanently
    < Location: /ords/
    < Content-Length: 0
    < 
    * Connection #0 to host localhost left intact
    
    Total time: 0.001882

    URL two

    The second cURL command you can issue is to the Metadata catalog for your/a target schema. In this example, I’m working with my ordsdemo user. He’s been REST-enabled, and I’ve already set up some Resource Modules (aka ORDS APIs). So I know there is “stuff” at that endpoint.

    https://[my host]/ords/[target schema]/metadata-catalog/

    Here is the cURL command I used (with the response included):

    choina@MacBook-Pro-2 ~ % curl -v -s -w "\n\nTotal time: %{time_total}\n" http://localhost:8080/ords/ordsdemo/metadata-catalog/
    
    * Host localhost:8080 was resolved.
    * IPv6: ::1
    * IPv4: 127.0.0.1
    *   Trying [::1]:8080...
    * Connected to localhost (::1) port 8080
    > GET /ords/ordsdemo/metadata-catalog/ HTTP/1.1
    > Host: localhost:8080
    > User-Agent: curl/8.7.1
    > Accept: */*
    > 
    * Request completely sent off
    < HTTP/1.1 200 OK
    < Content-Type: application/json
    < X-ORDS_DEBUG: true
    < X-Frame-Options: SAMEORIGIN
    < Transfer-Encoding: chunked
    < 
    * Connection #0 to host localhost left intact
    {"items":[{"name":"MOVIE","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/movie/"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/movie/","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/movie/","mediaType":"application/openapi+json"}]},{"name":"moviestream","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/mymovies/movie-all"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/mymovies/movie-all","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/mymovies/","mediaType":"application/openapi+json"}]},{"name":"moviestream","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/mymovies/movie-genre"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/mymovies/movie-genre","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/mymovies/","mediaType":"application/openapi+json"}]},{"name":"test_endpoints","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/no_objects/hello_auth"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/no_objects/hello_auth","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/no_objects/","mediaType":"application/openapi+json"}]},{"name":"test_endpoints","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/no_objects/hello_client_cred"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/no_objects/hello_client_cred","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/no_objects/","mediaType":"application/openapi+json"}]}],"hasMore":false,"limit":25,"offset":0,"count":5,"links":[{"rel":"self","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/"},{"rel":"first","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/"}]}
    
    Total time: 0.030173

    In the above example, we are hitting the database server (because we are grabbing the metadata catalog for the target schema). Consequently, the Total time for the round trip is slightly higher. Now, I can compare the two times against each other and against what I’ve seen historically. Are they reasonable? Do they pass the “sniff test”? Or do the results merit a deeper investigation?

    Fin

    I cannot stress this enough; this is just a simple heuristic to get you started. It takes 20 seconds to execute these commands. You’re really just “slicing the pie” at this stage. I honestly don’t think that simile applies here (unless you’ve ever cleared a room before), but you get the idea – thin slicing. Inching your way ever closer to the solution. This is just a part of the route-cause analysis.

    That’s all for now. Keep this in your back pocket. If you have your own heuristics or troubleshooting tips, leave them in a comment below. I’d love to hear how you diagnose issues related to networking, connectivity, latency, etc.

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

    1. I was hesitant to include this last part, “you’ve observed historically,” because I don’t give a metric. Honestly, it depends on your workload. But if you are doing any sort of performance testing you probably have at least a good sense of what response times should be like for your application (I’m talking directly to developers here). Even if you don’t know these expected response times, you can still compare the individual results against each other: the first URL vs the second URL, in this example. ↩︎
    2. What do the optional -v -w -s variables mean? Good question -v is shorthand for “Verbose;” which means print out as much info as is available. The -s is shorthand for “Silent;” in other words, don’t show any progress bars. And the -w means “Write Out,” in this case, “explicitly write out the Total time for me please.” ↩︎
  • 401 Unauthorized invalid_token – troubleshooting Oracle Cloud IAM JWTs with ORDS

    401 Unauthorized invalid_token – troubleshooting Oracle Cloud IAM JWTs with ORDS

    Symptoms

    You are able to request an access token from OCI IAM. Yet, when you issue the subsequent request to your target resource (an ORDS endpoint), you receive the following message (error="invalid_token"): 

    WWW-Authenticate: Bearer realm="Oracle REST Data Services", error="invalid_token"

    Actions you’ve taken

    1. You’ve done the following in OCI:
      • Registered an Integrated Application with Oracle Identity and Access Management (IAM)
      • Created a Primary Audience & Scope
      • Obtained your Client ID and Client Secret
      • Configured your networking correctly (or at least have high confidence it’s configured correctly)
      • Acquired all of the correct, relevant URIs:
        • Authorization
        • Access Token
        • JWK
    2. You’ve configured the following in ORDS:
      • Created your JWT Profile according to the docs (or this tutorial)
      • Created the JWT Role and Privilege (which should be the same as the OCI Scope name)
      • And protected your target resource (aka ORDS API)
    3. You’ve placed everything where it should be in your choice of API testing tool (cURL, Postman, Insomnia, etc.).

    YET…you still receive this error="invalid_token" message, it is quite possible that you have not made the JWK URL publically accessible in OCI IAM. 

    Solution

    Here is how you can verify and resolve this issue. First, navigate to your domain, then select Settings.

    If this Configure client access box is unchecked, it is likely the culprit. Check it, then select Save Changes (the button at the bottom of the screen).

    Then, once you re-attempt your HTTP request, ORDS will be able to:

    1. Access the JWK URL (which you’ve included when you created your JWT Profile)
    2. Verify the authenticity of your JWT, and 
    3. Respond with the results from your resource (ORDS endpoint)

    Et voilà! And that’s it, you’re back in business! 

    To-do list

    I think we have some action items, too: 

    • Investigate this error message and see if we can improve the message to the user (we’ve already filed an enhancement request on this)
    • Update the docs to be more specific on this setting and where to find it (a documentation bug has already been filed for this)
    • Determine if this is a good candidate for adding to the troubleshooting section of our guide
    • Update my JWT tutorial

    And that’s all for now. ORDS version 25.1 is right around the corner, so look out for the official release notes and my accompanying blog too!

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!