Category: REST APIs

  • From one API to another: Using ORDS Auto-REST to update a table

    From one API to another: Using ORDS Auto-REST to update a table

    It turns out the National Highway Traffic Safety Administration (NHTSA; pronounced ‘nit-sə) has a Vehicle Identification Number (VIN) decoder (it’s freely available to the public). But even cooler are the set of APIs they have. The one I found most interesting, and useful (for my use case) was the /vehicles/DecodeVinValues/[your VIN]?format=[format]l&modelyear=[year] endpoint. Here is an example, using my vehicle:

    And as a cURL command (using my 1999 Chevy Silverado 🤠), with the subsequent server response:

    curl --request GET \
      --url 'https://vpic.nhtsa.dot.gov/api/vehicles/DecodeVinValues/1GCEK14V0XZ138493?format=json&modelyear=1999' \
    
    {
    	"Count": 1,
    	"Message": "Results returned successfully. NOTE: Any missing decoded values should be interpreted as NHTSA does not have data on the specific variable. Missing value should NOT be interpreted as an indication that a feature or technology is unavailable for a vehicle.",
    	"SearchCriteria": "VIN(s): 1GCEK14V0XZ138493",
    	"Results": [
    		{
    			"ABS": "",
    			"ActiveSafetySysNote": "",
    			"AdaptiveCruiseControl": "",
    			"AdaptiveDrivingBeam": "",
    			"AdaptiveHeadlights": "",
    			"AdditionalErrorText": "",
    			"AirBagLocCurtain": "",
    			"AirBagLocFront": "",
    			"AirBagLocKnee": "",
    			"AirBagLocSeatCushion": "",
    			"AirBagLocSide": "",
    			"AutoReverseSystem": "",
    			"AutomaticPedestrianAlertingSound": "",
    			"AxleConfiguration": "",
    			"Axles": "",
    			"BasePrice": "",
    			"BatteryA": "",
    			"BatteryA_to": "",
    			"BatteryCells": "",
    			"BatteryInfo": "",
    			"BatteryKWh": "",
    			"BatteryKWh_to": "",
    			"BatteryModules": "",
    			"BatteryPacks": "",
    			"BatteryType": "",
    			"BatteryV": "",
    			"BatteryV_to": "",
    			"BedLengthIN": "",
    			"BedType": "",
    			"BlindSpotIntervention": "",
    			"BlindSpotMon": "",
    			"BodyCabType": "",
    			"BodyClass": "Pickup",
    			"BrakeSystemDesc": "",
    			"BrakeSystemType": "Hydraulic",
    			"BusFloorConfigType": "Not Applicable",
    			"BusLength": "",
    			"BusType": "Not Applicable",
    			"CAN_AACN": "",
    			"CIB": "",
    			"CashForClunkers": "",
    			"ChargerLevel": "",
    			"ChargerPowerKW": "",
    			"CombinedBrakingSystem": "",
    			"CoolingType": "",
    			"CurbWeightLB": "",
    			"CustomMotorcycleType": "Not Applicable",
    			"DaytimeRunningLight": "",
    			"DestinationMarket": "",
    			"DisplacementCC": "4800.0",
    			"DisplacementCI": "292.91397165471",
    			"DisplacementL": "4.8",
    			"Doors": "2",
    			"DriveType": "4WD/4-Wheel Drive/4x4",
    			"DriverAssist": "",
    			"DynamicBrakeSupport": "",
    			"EDR": "",
    			"ESC": "",
    			"EVDriveUnit": "",
    			"ElectrificationLevel": "",
    			"EngineConfiguration": "V-Shaped",
    			"EngineCycles": "",
    			"EngineCylinders": "8",
    			"EngineHP": "",
    			"EngineHP_to": "",
    			"EngineKW": "",
    			"EngineManufacturer": "",
    			"EngineModel": "",
    			"EntertainmentSystem": "",
    			"ErrorCode": "0",
    			"ErrorText": "0 - VIN decoded clean. Check Digit (9th position) is correct",
    			"ForwardCollisionWarning": "",
    			"FuelInjectionType": "Multipoint Fuel Injection (MPFI)",
    			"FuelTankMaterial": "",
    			"FuelTankType": "",
    			"FuelTypePrimary": "",
    			"FuelTypeSecondary": "",
    			"GCWR": "",
    			"GCWR_to": "",
    			"GVWR": "Class 2E: 6,001 - 7,000 lb (2,722 - 3,175 kg)",
    			"GVWR_to": "",
    			"KeylessIgnition": "",
    			"LaneCenteringAssistance": "",
    			"LaneDepartureWarning": "",
    			"LaneKeepSystem": "",
    			"LowerBeamHeadlampLightSource": "",
    			"Make": "CHEVROLET",
    			"MakeID": "467",
    			"Manufacturer": "GENERAL MOTORS LLC",
    			"ManufacturerId": "984",
    			"Model": "Silverado",
    			"ModelID": "1850",
    			"ModelYear": "1999",
    			"MotorcycleChassisType": "Not Applicable",
    			"MotorcycleSuspensionType": "Not Applicable",
    			"NCSABodyType": "",
    			"NCSAMake": "",
    			"NCSAMapExcApprovedBy": "",
    			"NCSAMapExcApprovedOn": "",
    			"NCSAMappingException": "",
    			"NCSAModel": "",
    			"NCSANote": "",
    			"NonLandUse": "",
    			"Note": "",
    			"OtherBusInfo": "",
    			"OtherEngineInfo": "Iron",
    			"OtherMotorcycleInfo": "",
    			"OtherRestraintSystemInfo": "",
    			"OtherTrailerInfo": "",
    			"ParkAssist": "",
    			"PedestrianAutomaticEmergencyBraking": "",
    			"PlantCity": "ROANOKE",
    			"PlantCompanyName": "Fort Wayne Assembly",
    			"PlantCountry": "UNITED STATES (USA)",
    			"PlantState": "INDIANA",
    			"PossibleValues": "",
    			"Pretensioner": "",
    			"RearAutomaticEmergencyBraking": "",
    			"RearCrossTrafficAlert": "",
    			"RearVisibilitySystem": "",
    			"SAEAutomationLevel": "",
    			"SAEAutomationLevel_to": "",
    			"SeatBeltsAll": "",
    			"SeatRows": "",
    			"Seats": "",
    			"SemiautomaticHeadlampBeamSwitching": "",
    			"Series": "1500 (1/2 Ton)",
    			"Series2": "Conventional Cab",
    			"SteeringLocation": "",
    			"SuggestedVIN": "",
    			"TPMS": "",
    			"TopSpeedMPH": "",
    			"TrackWidth": "",
    			"TractionControl": "",
    			"TrailerBodyType": "Not Applicable",
    			"TrailerLength": "",
    			"TrailerType": "Not Applicable",
    			"TransmissionSpeeds": "",
    			"TransmissionStyle": "",
    			"Trim": "",
    			"Trim2": "",
    			"Turbo": "",
    			"VIN": "1GCEK14V0XZ138493",
    			"ValveTrainDesign": "",
    			"VehicleDescriptor": "1GCEK14V*XZ",
    			"VehicleType": "TRUCK",
    			"WheelBaseLong": "",
    			"WheelBaseShort": "",
    			"WheelBaseType": "",
    			"WheelSizeFront": "",
    			"WheelSizeRear": "",
    			"WheelieMitigation": "",
    			"Wheels": "",
    			"Windows": ""
    		}
    	]
    }

    On its own, this is pretty cool. But I wanted to do something more with this information. I wanted to store it, so I could use it later, for a Cloud World-related project.

    A GET and POST Python script

    So, with the help of one of the many popular LLMs I settled on a Python script that helps me take what you see above and get it “database ready.” Here is what that Python code looks like:

    import argparse
    import requests
    import json
    import re
    
    def sanitize_key(key: str) -> str:
        # Convert keys to lowercase, replace non-alphanumeric characters with underscores.
        key = key.strip()
        key = re.sub(r'[^a-zA-Z0-9]+', '_', key)
        return key.lower()
    
    def decode_vin(vin: str, model_year: int):
        url = f"https://vpic.nhtsa.dot.gov/api/vehicles/DecodeVinValues/{vin}?format=json&modelyear={model_year}"
        response = requests.get(url)
        response.raise_for_status()
    
        data = response.json()
    
        if "Results" not in data or not data["Results"]:
            raise ValueError("Invalid API response: No Results found.")
    
        raw_result = data["Results"][0]
    
        # Sanitize keys and skip empty values
        result = {
            sanitize_key(k): v
            for k, v in raw_result.items()
            if v not in (None, "", "Not Applicable")
        }
    
        return result
    
    def post_to_endpoint(payload: dict, endpoint: str):
        headers = {'Content-Type': 'application/json'}
        response = requests.post(endpoint, json=payload, headers=headers)
        response.raise_for_status()
        return response.json()
    
    def main():
        parser = argparse.ArgumentParser(description="A tool to decode VINs and send them to an Oracle REST endpoint")
        parser.add_argument("vin", type=str, help="Vehicle Identification Number")
        parser.add_argument("model_year", type=int, help="Model year of the vehicle")
        parser.add_argument("--pretty", action="store_true", help="Pretty-print output JSON")
        args = parser.parse_args()
    
        # Decode and sanitize
        result = decode_vin(args.vin, args.model_year)
    
        # Print to console
        if args.pretty:
            print(json.dumps(result, indent=2))
        else:
            print(result)
    
        # POST to local Oracle ORDS endpoint
        endpoint_url = "http://localhost:8080/ords/ordsdemo/vin_table/"
        try:
            post_response = post_to_endpoint(result, endpoint_url)
            print("\nPOST response from server:")
            print(json.dumps(post_response, indent=2))
        except requests.HTTPError as e:
            print(f"Error posting to endpoint: {e}")
    
    if __name__ == "__main__":
        main()

    👈🏼 Click here to learn what all different parts of this script do.

    I’ll explain briefly according to the code blocks:

    • Lines 6-10 take the results, or rather the “keys” (which will later become the columns in a database table), and standardize them. The keys end up being all lower case, no whitespaces.
    • Lines 12-31 are the bulk of the script. That part issues a GET request to the VIN decoder endpoint. If you can picture in your head, what I’m doing effectively is creating a table of keys and values. Those will then be sent over to my ORDS endpoint (a table I separately created and Auto-REST enabled with ORDS).
    • Lines 33-37 and 56-62 kind of work in tandem. That first block, lines 33-57, basically expects all the “stuff” you see in line 58. Line 58 sends the results (after we’ve processed it), using the ORDS endpoint, to the function in lines 33-57.
    • Lines 39-53 are my entry and exit points for this script. So on the command line I’d execute the following command (I’m using the same VIN and model year): python3 vindecode.py 1GCEK14V0XZ138493 1999

    Target 23ai Database

    So this whole exercise doesn’t work unless I have a target, in my database. And for this project I’m working on, a simple table that has been Auto-REST enabled is fine. I just need it to store these values. And I can use the VIN as a Primary Key (VINs are unique, and no two are alike).

    And if I’m thinking ahead, I’ll probably never update any single row. I just need to be able to insert a new row with a POST request, and then later issue a GET request by VIN. So an Auto-REST enabled table, with the VIN as a Primary Key will work just fine. As things scale, this might change, but with minimal technical debt incurred, this approach works for now.

    As an aside, when I’m working through projects like these, where I’m building out a new object, I usually have a general idea of what I want to achieve, but I will go through tons of iterations1. This time I ended up creating and dropping like four table versions before I ended on this one (four is on the low-end actually). The final DDL for this table:

    CREATE TABLE ORDSDEMO.VIN_TABLE 
        ( 
         VIN                         VARCHAR2 (17) , 
         MAKE                        VARCHAR2 (50) , 
         MAKEID                      NUMBER (5) , 
         MANUFACTURER                VARCHAR2 (100) , 
         MANUFACTURERID              NUMBER (10) , 
         MODEL                       VARCHAR2 (100) , 
         MODELID                     NUMBER (10) , 
         MODELYEAR                   NUMBER (4) , 
         SERIES                      VARCHAR2 (100) , 
         SERIES2                     VARCHAR2 (100) , 
         TRIM                        VARCHAR2 (100) , 
         TRIM2                       VARCHAR2 (100) , 
         VEHICLEDESCRIPTOR           VARCHAR2 (20) , 
         VEHICLETYPE                 VARCHAR2 (50) , 
         BODYCLASS                   VARCHAR2 (50) , 
         BODYCABTYPE                 VARCHAR2 (50) , 
         DRIVETYPE                   VARCHAR2 (50) , 
         BRAKESYSTEMTYPE             VARCHAR2 (50) , 
         BRAKESYSTEMDESC             VARCHAR2 (100) , 
         FUELINJECTIONTYPE           VARCHAR2 (100) , 
         FUELTYPEPRIMARY             VARCHAR2 (50) , 
         FUELTYPESECONDARY           VARCHAR2 (50) , 
         ENGINECONFIGURATION         VARCHAR2 (50) , 
         ENGINECYLINDERS             NUMBER (2) , 
         ENGINEHP                    NUMBER , 
         ENGINEKW                    NUMBER , 
         ENGINEMANUFACTURER          VARCHAR2 (100) , 
         ENGINEMODEL                 VARCHAR2 (100) , 
         DISPLACEMENTCC              NUMBER (6,1) , 
         DISPLACEMENTCI              NUMBER (8,5) , 
         DISPLACEMENTL               NUMBER (4,1) , 
         DOORS                       NUMBER (2) , 
         GVWR                        VARCHAR2 (100) , 
         PLANTCITY                   VARCHAR2 (50) , 
         PLANTSTATE                  VARCHAR2 (50) , 
         PLANTCOUNTRY                VARCHAR2 (50) , 
         PLANTCOMPANYNAME            VARCHAR2 (100) , 
         OTHERENGINEINFO             VARCHAR2 (255) , 
         ERRORTEXT                   VARCHAR2 (255) , 
         ERRORCODE                   NUMBER (2) , 
         AIRBAGSIDE                  VARCHAR2 (50) , 
         AIRBAGFRONT                 VARCHAR2 (50) , 
         AIRBAGCURTAIN               VARCHAR2 (50) , 
         TPMS                        VARCHAR2 (100) , 
         SEATBELTSALL                VARCHAR2 (50) , 
         ENTERTAINMENTSYSTEM         VARCHAR2 (100) , 
         CUSTOMMOTORCYCLETYPE        VARCHAR2 (100) , 
         TRAILERTYPE                 VARCHAR2 (100) , 
         TRAILERBODYTYPE             VARCHAR2 (100) , 
         TRANSMISSIONSTYLE           VARCHAR2 (100) , 
         TRANSMISSIONSPEEDS          VARCHAR2 (50) , 
         WINDOWS                     VARCHAR2 (50) , 
         WHEELS                      VARCHAR2 (50) , 
         WHEELBASESHORT              VARCHAR2 (50) , 
         WHEELBASELONG               VARCHAR2 (50) , 
         WHEELBASETYPE               VARCHAR2 (100) , 
         NOTE                        VARCHAR2 (255) , 
         ABS                         VARCHAR2 (10) , 
         ADAPTIVECRUISECONTROL       VARCHAR2 (50) , 
         ADAPTIVEHEADLIGHTS          VARCHAR2 (50) , 
         FORWARDCOLLISIONWARNING     VARCHAR2 (50) , 
         PEDESTRIANAUTOMATICEBRAKING VARCHAR2 (50) , 
         REARCROSSTRAFFICALERT       VARCHAR2 (50) , 
         REARAUTOMATICEBRAKING       VARCHAR2 (50) , 
         PARKASSIST                  VARCHAR2 (50) , 
         TRACTIONCONTROL             VARCHAR2 (50) , 
         ESC                         VARCHAR2 (50) , 
         DYNAMICBRAKESUPPORT         VARCHAR2 (50) , 
         LANEDEPARTUREWARNING        VARCHAR2 (50) , 
         LANEKEEPSYSTEM              VARCHAR2 (50) , 
         ACTIVESAFETYSYSNOTE         VARCHAR2 (255) , 
         ADDITIONALERRORTEXT         VARCHAR2 (255) 
        ) 
        TABLESPACE USERS 
        LOGGING 
    ;
    
    ALTER TABLE ORDSDEMO.VIN_TABLE 
        ADD PRIMARY KEY ( VIN ) 
        USING INDEX LOGGING ;

    Some background

    Oh, please know that I used an LLM for this DDL too. I could have reduced the number of columns since this particular vehicle has a lot of null or “n/a” fields. But other, future vehicles might actually have values for some of those columns. They can stay for now. You might not like the stylistic choice of condensing all my values, but I think this is inconsequential. You’d probably transform them again, in your application layer, when you pull them down from the database. Also, I’m familiar with these values. I could have abbreviated them even further, but for readability, I think this is good (for future me, and others too).

    ORDS Auto-REST

    The last thing I needed to do was Auto-REST enable my table. Which, if you’ve used the SQL Developer Web UI, you know how easy this is. Two clicks and you’re done. From there I have access to several HTTP methods; I only care about GET [all], GET{by id}, and POST.

    Oh, and if you look at the URI that is provided, it’s the same URI that you saw in line 56 of my Python script. So that sort of brings us full circle.

    If you recall, I used the VIN as a Primary Key. This image shows how that VIN can now be used in the URI (instead of an arbitrary Row ID).

    Review/Checkpoint

    So, at this point, I have two target endpoints:

    1. One where I retrieve a vehicle’s details by its VIN and model year (I don’t own this one – this is the one provided by NHTSA).
    2. Another (that I own) where I am updating and retrieving vehicle records.

    I’ve created a VIN_TABLE table, with the VIN as the Primary Key, and I’ve Auto-REST enabled it with ORDS. I’ve then taken the endpoint for issuing a POST request and added it to the Python script (saved as vindecode.py). Now I can issue the following command to add a record to my table:

    python3 vindecode.py 1GCEK14V0XZ138493 1999

    And just so you know I’m being honest, I’ll drop everything from my target table, and run through this workflow again.

    Back in VS Code, I’ll issue the command, and receive the response from my database (via ORDS).

    And finally, I’ll review the results (I already did this in VS Code, but I’ll do it in the browser and SQL Worksheet too):

    Review

    And that’s it! One of countless examples of how you might work through a pretty common pattern such as this (using LLMs too). The nice thing here is I don’t have to worry about any ORMs (never have, never will). You have to understand, my reality is such that ORMs do not exist. All I’ve ever known is ORDS; a bit of middleware that sends/receives HTTP/HTTPS requests on your behalf, without ever having to map a thing.

    And we’ve made it much easier to take ORDS for a test drive. We’ve been working behind the scenes to improve our Docker/Podman documentation; we just released the latest refresh to our docs here. Hopefully the docs are much more approachable for the container newbie.

    OH! And if you ever have any issues, open a ticket here. Our team now “owns” the ORDS content on that GitHub repository; so, we’ll be a lot more responsive too!

    That is all for now. If you haven’t downloaded the latest ORDS, check it out here. Bye 👋🏼.

    P.S. Yes, I realize this post’s image is of a GMT-400, and not a GMT-800. But it was a cool pic, and I really liked it. Credit to The Autopian.

    1. I’m including a link to the prompting I used for creating most of what you see here. I think in this new age of AI/LLMs it is important to understand prompting. Eventually, whether we like it or not, we’ll all need to become “prompt engineers.” And as you’ll see in that thread, you actually have to know what you are doing to be able to ask the right questions and to challenge an LLM’s assumptions.

      A lot of times, these things are just flat out wrong, and overly verbose too. As you’ll see, at one point the LLM added a mapping function in one of the iterations, which in my mind was completely unnecessary. Additionally, I wouldn’t have constructed this script as a pseudo-command line interface. BUT…I learned a new library (argparse), and I think I’ll explore that more in the future. ↩︎
  • Troubleshooting: reviewing ORDS connections, your application server, and response times

    Troubleshooting: reviewing ORDS connections, your application server, and response times

    Symptom/Issue

    In an internal Slack thread today, a user was trying to diagnose browser latency while attempting to connect to the ORDS landing page.

    Peter suggested what I thought was a pretty neat heuristic for checking connections to ORDS and your database, as well as latency.

    Methodology

    Let’s say your symptoms are either slow loading or intermittent loss of service when attempting to reach an ORDS landing page. Your immediate thought might be to refresh your browser or bounce the ORDS server (if running in Standalone mode). But this isn’t practical in a production environment, nor is it practical if you have ORDS deployed on a Weblogic Server or Apache Tomcat (you can deploy using any of those three servers).

    Quick Heuristic

    Use cURL to test two different endpoints, compare their response times to each other, and compare to the response time you’ve observed historically1.

    URL one

    This first URL points to where your ORDS instance “lives.”

    https://[my host]/ords

    “Lives,” what does that mean? If you’re deploying to Weblogic Server ORDS would “live” in the $WEBLOGIC_HOME/application directory. In Apache Tomcat it is the $CATALINA/webapps directory, and on Standalone (using the ORDS embedded Jetty server) that might be $HOME/[your ords product folder].

    So if you execute a cURL command to that URL, you’re issuing a request to your application server. With that single command, you can determine:

    1. if it’s even up
    2. what the response is, and
    3. how fast that response is

    Honestly, this sounds silly and obvious at first. But it costs you nothing to just rule out the application server as a culprit. The command2 I tested (with the response included):

    choina@MacBook-Pro-2 ~ % curl -v -s -w "\nTotal time: %{time_total}\n" http://localhost:8080/ords                               
    * Host localhost:8080 was resolved.
    * IPv6: ::1
    * IPv4: 127.0.0.1
    *   Trying [::1]:8080...
    * Connected to localhost (::1) port 8080
    > GET /ords HTTP/1.1
    > Host: localhost:8080
    > User-Agent: curl/8.7.1
    > Accept: */*
    > 
    * Request completely sent off
    < HTTP/1.1 301 Moved Permanently
    < Location: /ords/
    < Content-Length: 0
    < 
    * Connection #0 to host localhost left intact
    
    Total time: 0.001882

    URL two

    The second cURL command you can issue is to the Metadata catalog for your/a target schema. In this example, I’m working with my ordsdemo user. He’s been REST-enabled, and I’ve already set up some Resource Modules (aka ORDS APIs). So I know there is “stuff” at that endpoint.

    https://[my host]/ords/[target schema]/metadata-catalog/

    Here is the cURL command I used (with the response included):

    choina@MacBook-Pro-2 ~ % curl -v -s -w "\n\nTotal time: %{time_total}\n" http://localhost:8080/ords/ordsdemo/metadata-catalog/
    
    * Host localhost:8080 was resolved.
    * IPv6: ::1
    * IPv4: 127.0.0.1
    *   Trying [::1]:8080...
    * Connected to localhost (::1) port 8080
    > GET /ords/ordsdemo/metadata-catalog/ HTTP/1.1
    > Host: localhost:8080
    > User-Agent: curl/8.7.1
    > Accept: */*
    > 
    * Request completely sent off
    < HTTP/1.1 200 OK
    < Content-Type: application/json
    < X-ORDS_DEBUG: true
    < X-Frame-Options: SAMEORIGIN
    < Transfer-Encoding: chunked
    < 
    * Connection #0 to host localhost left intact
    {"items":[{"name":"MOVIE","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/movie/"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/movie/","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/movie/","mediaType":"application/openapi+json"}]},{"name":"moviestream","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/mymovies/movie-all"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/mymovies/movie-all","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/mymovies/","mediaType":"application/openapi+json"}]},{"name":"moviestream","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/mymovies/movie-genre"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/mymovies/movie-genre","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/mymovies/","mediaType":"application/openapi+json"}]},{"name":"test_endpoints","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/no_objects/hello_auth"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/no_objects/hello_auth","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/no_objects/","mediaType":"application/openapi+json"}]},{"name":"test_endpoints","links":[{"rel":"describes","href":"http://localhost:8080/ords/ordsdemo/no_objects/hello_client_cred"},{"rel":"canonical","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/no_objects/hello_client_cred","mediaType":"application/json"},{"rel":"alternate","href":"http://localhost:8080/ords/ordsdemo/open-api-catalog/no_objects/","mediaType":"application/openapi+json"}]}],"hasMore":false,"limit":25,"offset":0,"count":5,"links":[{"rel":"self","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/"},{"rel":"first","href":"http://localhost:8080/ords/ordsdemo/metadata-catalog/"}]}
    
    Total time: 0.030173

    In the above example, we are hitting the database server (because we are grabbing the metadata catalog for the target schema). Consequently, the Total time for the round trip is slightly higher. Now, I can compare the two times against each other and against what I’ve seen historically. Are they reasonable? Do they pass the “sniff test”? Or do the results merit a deeper investigation?

    Fin

    I cannot stress this enough; this is just a simple heuristic to get you started. It takes 20 seconds to execute these commands. You’re really just “slicing the pie” at this stage. I honestly don’t think that simile applies here (unless you’ve ever cleared a room before), but you get the idea – thin slicing. Inching your way ever closer to the solution. This is just a part of the route-cause analysis.

    That’s all for now. Keep this in your back pocket. If you have your own heuristics or troubleshooting tips, leave them in a comment below. I’d love to hear how you diagnose issues related to networking, connectivity, latency, etc.

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

    1. I was hesitant to include this last part, “you’ve observed historically,” because I don’t give a metric. Honestly, it depends on your workload. But if you are doing any sort of performance testing you probably have at least a good sense of what response times should be like for your application (I’m talking directly to developers here). Even if you don’t know these expected response times, you can still compare the individual results against each other: the first URL vs the second URL, in this example. ↩︎
    2. What do the optional -v -w -s variables mean? Good question -v is shorthand for “Verbose;” which means print out as much info as is available. The -s is shorthand for “Silent;” in other words, don’t show any progress bars. And the -w means “Write Out,” in this case, “explicitly write out the Total time for me please.” ↩︎
  • Example: an ORDS API that returns a user-configured JSON object

    Example: an ORDS API that returns a user-configured JSON object

    This actually came about from a Support Request. We were troubleshooting a JWT issue, that had ORDS in the mix. Ultimately, this is looking more and more like a networking issue (maybe something to do with routing rules between the load balancer and the backend server).

    But in the course of our troubleshooting, we wanted to rule out some variables. One of those variables was the JWK URL we were using for the ORDS_SECURITY.CREATE_JWT_PROFILE procedure. So we had this idea to kind of fool ORDS into thinking it was visiting the IDCS (the predecessor to Oracle IAM) JWK URL when, in fact, it was just visiting another ORDS endpoint.

    If you’ve seen my JWT + ORDS tutorials (one for OCI IAM, another for Microsoft Azure), then you know when you create an ORDS JWT Profile, it will look something like this:

    BEGIN
      ORDS_SECURITY.CREATE_JWT_PROFILE(
          p_issuer => 'https://identity.oraclecloud.com/',
          p_audience => 'my_primary_audience' ,
          p_jwk_url =>'https://idcs-10a10a10a10a10a10a10a10a.identity.oraclecloud.com/admin/v1/SigningCert/jwk'
      );
      COMMIT;
    END;
    /

    In the above code block, just know that p_issuer is your Identity Provider (e.g., IAM, IDCS, Microsoft Entra, etc.), p_audience is the Primary Audience (which you created when you registered your client application with your Identity Provider), and the p_jwk_url is where ORDS goes to look up the keys for decoding a JWT.

    The eventual JWT that ORDS receives would include the Issuer and Audience, too. But it would also include other values such as the Scope. That Scope name would be associated with the ORDS Privilege–the privilege you created specifically to use for protecting/associating with your ORDS Resource Module. And ORDS would use all these values (and the JWK “keys” found at that JWK URL to decode the JWT) together to:

    1. Verify you are who you say you are, and
    2. You have the proper Scope (or as ORDS interprets it…the proper privilege) to access this protected resource

    Back to the use case. So, for troubleshooting reasons, we took that JSON object located at that JWK URL and recreated it as an ORDS endpoint. Let me point out that this is cool, but what you should take away from this is that there are tons of things you can do with ORDS, and you have a lot of control over how ORDS will send back information to you, your application, or your web client.

    Here is how we structured the Handler code:

    BEGIN
        HTP.P('{"
        keys": [
            {
                "kty": "RSA",
                "x5t#S256": "Lorem ipsum dolor sit amet__pellentesque elementum",
                "e": "AQAB",
                "x5t": "xwSmccaQZDvAZPOpZPHOiQDlLgo",
                "kid": "SIGNING_KEY",
                "x5c": [
                    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce lorem dui, elementum et odio ut, sollicitudin gravida dui. Ut nec dapibus elit. Morbi sit amet blandit justo. Quisque accumsan turpis metus, vel sodales nisl porta vel. Nunc placerat, mi id eleifend pharetra, massa urna imperdiet arcu, vitae dignissim nulla ex sed felis. Maecenas auctor risus ac nisl efficitur, sit amet euismod odio finibus. Praesent lacinia nunc id ex tempor, sed tempor nisl porta. Integer semper efficitur arcu, a dictum quam vulputate tempus. Nulla congue dapibus tortor vel volutpat. Curabitur et sollicitudin purus. Mauris quis aliquam augue. Nulla ac leo tristique, ultricies mauris sit amet, ultrices quam. Praesent placerat a lectus sit amet dapibus. Donec rhoncus felis velit, sed placerat nunc pellentesque elementum==",
    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce lorem dui, elementum et odio ut, sollicitudin gravida dui. Ut nec dapibus elit. Morbi sit amet blandit justo. Quisque accumsan turpis metus, vel sodales nisl porta vel. Nunc placerat, mi id eleifend pharetra, massa urna imperdiet arcu, vitae dignissim nulla ex sed felis. Maecenas auctor risus ac nisl efficitur, sit amet euismod odio finibus. Praesent lacinia nunc id ex tempor, sed tempor nisl porta. Integer semper efficitur arcu, a dictum quam vulputate tempus. Nulla congue dapibus tortor vel volutpat. Curabitur et sollicitudin purus. Mauris quis aliquam augue. Nulla ac leo tristique, ultricies mauris sit amet, ultrices quam. Praesent placerat a lectus sit amet dapibus. Donec rhoncus felis velit, sed placerat nunc pellentesque elementum"
                ],
                "key_ops": [
                    "verify",
                    "wrapKey",
                    "encrypt"
                ],
                "alg": "RS256",
                "n": 
    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce lorem dui, elementum et odio ut, sollicitudin gravida dui. Ut nec dapibus elit. Morbi sit amet blandit justo. Quisque accumsan turpis metus, vel sodales nisl porta vel. Nunc placerat, mi id eleifend pharetra, massa urna imperdiet arcu, vitae dignissim nulla"
            }
        ]
    }');
    END;

    With that new ORDS endpoint, we (temporarily) recreated the ORDS JWT Profile with the new ORDS API as a proxy for our true JWK URL. Like this (this JWK URL is entirely fake, BTW):

    BEGIN
      ORDS_SECURITY.CREATE_JWT_PROFILE(
          p_issuer => 'https://identity.oraclecloud.com/',
          p_audience => 'my_primary_audience' ,
          p_jwk_url => 'https://abcdefg-myadb.region-01.oraclecloudapps.com/ords/admin/jwk/jwk'
      );
      COMMIT;
    END;
    /

    Hypothetically if you were to navigate to that ORDS URI, you’d see a facsimile of the actual JWK information:

    So now, we’ve temporarily removed the JWK URL from the troubleshooting equation. However, we can still access that protected endpoint since we have a valid token and keys to decode it (courtesy of the ORDS endpoint we created)!

    And this my friends, is the type of mad scientist stuff that I live for. I had to share our experience to underscore two main points:

    1. You have a lot of flexibility with these ORDS APIs, and
    2. When it comes to troubleshooting/root cause analysis, you must break things down into individual parts!

    And that’s it for today. Class dismissed 🤓

    BTW, I took some inspiration from here too. And of course, a link to the JWT section of our docs.

    If you liked this, consider sharing. And if you have any cool or clever ORDS tricks you use, comment below!

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

  • A simple ORDS GET request using the Go language

    A simple ORDS GET request using the Go language

    Venkata this one’s for you 😀

    It took me the rest of the afternoon, but here is a pretty simple GET request example, using a user-defined ORDS API. This API is derived from one of our LiveLabs, a Python + JavaScript application. But this particular scenario pretty well exemplifies what we discussed in our presentation yesterday—that the RESTful framework makes your backend code much more language-agnostic. With a few tweaks, I was able to “port” this over to Go.

    ORDS example API

    The ORDS API I’m working with looks like this on the back end (this is our REST Workshop; where you build ORDS APIs):

    More formally, it looks like this:

    SELECT DISTINCT
        GENRE
    FROM
        MOVIE,
        JSON_TABLE ( GENRES, '$[*]'
                COLUMNS (
                    GENRE PATH '$'
                )
            )
    ORDER BY
        1 ASC

    The MOVIE table I’m calling upon consists of various data types, but the target GENRE In this case, the column is a JSON object that actually contains an array. So, all I’m doing is returning the unique genres (i.e., removing duplicates) from a table of thousands of movies. What is nice about this approach is that all my SQL and PL/SQL stay in the database. All my application’s code needs now is an endpoint.

    If I don’t want to keep pinging that endpoint, I can always just use a JSON document as a proxy for the endpoint. And if I just point my browser to that URI (i.e. ORDS API), here is what the results look like (I had to take two screenshots to capture everything):

    When you are looking at these results, it’s best to focus on the “root” key:value pairs of the ORDS response. For a fairly standard ORDS GET request, the response body will consist of:

    • items
    • hasMore
    • limit
    • offset
    • count, and
    • links

    When I collapse my example, I am left with the following:

    {
        "items": [
        ],
        "hasMore": false,
        "limit": 25,
        "offset": 0,
        "count": 24,
        "links": [
        ]
    }
    Note: The items will be an array, as will the links.

    A caveat to this is when/if you choose to Auto-REST enable an object, the result will look mostly the same. However, ORDS adds a self-referential link to each result. But those links are an array, too.

    Regardless, it shouldn’t affect how you approach this in Go. The struct of those types are the same: Items []Items json:"items" and Links []Links json:"links", respectively. Compare this auto-REST-enabled result to the one above:

    Worth noting: The image below is using a different endpoint. Its basically doing a "select * from movies;" on the MOVIES table. Hence, why it looks a little different. But you should still be able to see what I am talking about. If you want to learn more about ORDS Auto-REST, then check this out. 
    If you squint, you can see how each result has its own “rel” link. That is how Auto-REST would look, this is expected.

    Go time

    Coming from Python and PL/SQL, along with some JavaScript, I found Go to be extremely enjoyable. I get the gist of how you define the types and structures of “things.” I haven’t even bothered to look at the documentation yet; I didn’t even need to. It was easy to follow along using the two tutorials I found (here and here). After a few minutes of debugging (which is quite intuitive in Go, actually), I had my code working.

    Actually, once I figured out the syntax and Go idiosyncracies, creating this simple example wasn’t much of an effort.

    Here is the Go code that I came up with:

    package main
    
    import (
    	"encoding/json"
    	"fmt"
    	"io/ioutil"
    	"log"
    	"net/http"
    	"os"
    )
    
    type Response struct {
    	Items   []Items `json:"items"`
    	Hasmore bool    `json:"hasMore"`
    	Limit   int     `json:"limit"`
    	Offset  int     `json:"offset"`
    	Count   int     `json:"count"`
    	Links   []Links `json:"links"`
    }
    
    type Items struct {
    	Genre string `json:"genre"`
    }
    
    type Links struct {
    	Rel  string `json:"rel"`
    	Href string `json:"href"`
    }
    
    func main() {
    
    	response, err := http.Get("http://localhost:8081/ords/ordsdemo/mymovies/movie-genre")
    
    	if err != nil {
    		fmt.Print(err.Error())
    		os.Exit(1)
    	}
    
    	responseData, err := ioutil.ReadAll(response.Body)
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	var responseObject Response
    	json.Unmarshal(responseData, &responseObject)
    
    	for i := 0; i < len(responseObject.Items); i++ {
    		fmt.Println(responseObject.Items[i].Genre)
    	}
    
    	fmt.Println(responseObject.Hasmore)
    	fmt.Println(responseObject.Limit)
    	fmt.Println(responseObject.Offset)
    	fmt.Println(responseObject.Count)
    
    	for i := 0; i < len(responseObject.Links); i++ {
    		fmt.Println(responseObject.Links[i].Rel + responseObject.Links[i].Href)
    	}
    
    }

    If you know Go, then I’m not teaching you anything that you don’t already know. You probably know the libraries I’m using. However, I think it is worth pointing out a few ORDS-specific things.

    When it comes to the items and the links they are always going to be an array. So you’ll need to account for that when you are defining the structures (aka struct) for your user-defined types (let me know if this terminology is correct, it just seems so intuitive, so I’m going with it).

    Here are some other things to consider:

    • hasMore is a Boolean; or bool in Go. It’s also going to be either true or false.
      • True means there are more results (so you can page forward if you want).
      • False means there are no more results.
    • limit just means how many results you should expect. In my case, 25 results (that is our default, but you can adjust this when you build your APIs).
      • Another way to look at this is that my pagination is set to 25 results per page.
    • offset and limit are kind of interdependent. If you are on the first page then your offset will be = 0. But if you wanted to view your second “page” or set of results, then you’d set your offset = 25.
      • And because your limit = 25 then you’d see results 26-50.
    • The count is just the total results per page, or 25 in this case (hypothetically, if you were to make it to the end of your results set, it would probably not be exactly 25, something to keep in mind).

    Here is a complete JSON object of my first results (if you want to play around with it until you get ORDS up and running):

    {
        "items": [
            {
                "genre": "Action"
            },
            {
                "genre": "Adventure"
            },
            {
                "genre": "Animation"
            },
            {
                "genre": "Biography"
            },
            {
                "genre": "Comedy"
            },
            {
                "genre": "Crime"
            },
            {
                "genre": "Documentary"
            },
            {
                "genre": "Drama"
            },
            {
                "genre": "Family"
            },
            {
                "genre": "Fantasy"
            },
            {
                "genre": "Film-Noir"
            },
            {
                "genre": "History"
            },
            {
                "genre": "Horror"
            },
            {
                "genre": "Lifestyle"
            },
            {
                "genre": "Musical"
            },
            {
                "genre": "Mystery"
            },
            {
                "genre": "Reality-TV"
            },
            {
                "genre": "Romance"
            },
            {
                "genre": "Sci-Fi"
            },
            {
                "genre": "Sport"
            },
            {
                "genre": "Thriller"
            },
            {
                "genre": "Unknown"
            },
            {
                "genre": "War"
            },
            {
                "genre": "Western"
            }
        ],
        "hasMore": false,
        "limit": 25,
        "offset": 0,
        "count": 24,
        "links": [
            {
                "rel": "self",
                "href": "http://localhost:8081/ords/ordsdemo/mymovies/movie-genre"
            },
            {
                "rel": "describedby",
                "href": "http://localhost:8081/ords/ordsdemo/metadata-catalog/mymovies/item"
            },
            {
                "rel": "first",
                "href": "http://localhost:8081/ords/ordsdemo/mymovies/movie-genre"
            }
        ]
    }

    All the code you see here is also in my blog post code repo. There are other examples there, too!

    Also, huge thanks for asking about Go. It gave me an excuse to start learning it. I think I might be a Go fanboi now 🫣!

    And for all others reading:

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

  • Configuring OCI IAM Domain JWTs to use with ORDS OAuth2.0 protected APIs

    Configuring OCI IAM Domain JWTs to use with ORDS OAuth2.0 protected APIs

    Table of Contents

    Intended Audience/Purpose

    Does this describe you?

    If this describes you, then keep reading!


    Creating a Domain

    Assuming you are at the central OCI Console, click the navigation icon at the top of your screen. Then, navigate to Identity & Security and click Create domain.

    NOTE: Make sure you are in your correct Compartment (this will be up to you to decide)! Everything I show here is done within the same /ords Compartment. 

    In this example, I’ve chosen to create an “External User” Domain type. I haven’t tested with any other Domain types, but I assume they all work similarly.

    Once you’ve entered all the information (e.g., Domain administrator values and Compartment), click Create Domain. Then, click on the Domain you just created. Next, you’ll need to create and add an “Integrated application.”

    Mini-tutorial: Adding a user to your Domain

    In this mini-tutorial, I demonstrate how to create a new user. Notice how I am using a personal email account, you can totally do this!

    Once you’ve added this user, they recieve a password reset email for this Domain. They must create a new password before logging in.

    Configure client access

    First, navigate to your domain, then select Settings.

    If this Configure client access box is unchecked, check it, then select Save Changes (the button at the bottom of the screen).

    That’s it. You’re done!

    Adding an Integrated Application to the Domain

    Adding the application

    From within your Domain dashboard, click the Integrated Applications menu item. The “Add application” modal will appear. Read through the application types and choose the most appropriate one. For this demo, I selected “Confidential Application.”

    Continue filling in the modal fields. Make sure you choose a unique name!

    I’ve left the URLs section empty for this demo and kept the Display settings checked. The Display settings are options for users when logging into their Domain (i.e., they don’t impact this workflow, but they are nice to have set for later).

    Click Next to continue to the Resource server configuration section.

    Configuring OAuth2.0

    In this section, you’ll configure your Resouce server and Client. First, we’ll create a Scope1.

    Resource server configuration

    When you configure your Scope, you must include a Primary audience field. Here, I’m just choosing the Domain’s name. The Primary audience needs to be structured exactly like this (with the trailing backslash)! Next, create a Scope (I’m using dot notation, since this will mirror the privilege I’ll create in ORDS, later).

    When finished, click Add.

    Client Configuration

    Next, you’ll include details about your client. In this demo, I use Postman as a client application stand-in. For this example I’m using the Authorization code grant type2, along with a Redirect URL.

    FYI: Postman has a /callback URL that you can use as a Redirect URL.
    Mini-tutorial: How do I find Postman’s Callback URL?

    From the Authorization tab, select OAuth 2.0. Then scroll down to the Configure New Token section. There you will find the Callback URL. Copy it. You’ll need it!

    I’ve chosen “All” for the Token issuance policy. Next, click Finish.

    Issuing a POST request to obtain a JWT

    You can send off that initial POST request to obtain the JWT from IAM with your Domain set. You’ll then use the details contained in that JWT to configure ORDS.

    Gathering the required values for the request

    If your application hasn’t already been activated, do so now.

    Next, click the Edit OAuth configuration button and retrieve your Client ID and Client secret. Save it to a clipboard or your environment files (if using this for an application).

    You will also need the Domain URL. Navigate to your Domain’s main dashboard (❗️make sure you are still in the correct Compartment). Then copy the Domain URL to your clipboard, or environment file (aka .env file).

    Setting up Postman

    In Postman, make sure you have selected “Request Headers” in the Add authorization data to field. Also, add the word “Bearer” to the Header Prefix field.

    You’ll want to select Authorization Code as the Grant Type. For the Authorization URL and Access Token URL, you will use your Domain URL followed by:

    • Auth URL
      [Your Domain URL]/oauth2/v1/authorize
    • Access Token URL
      [Your Domain URL]/oauth2/v1/token

    Next, add your Client ID, Client Secret, and Scope. Notice how the Scope in Postman uses the Primary audience and the Scope (if you don’t remember where these came from, review the Resource server configuration section). Select “Send as Basic Auth header” for the Client Authentication field.

    Requesting the JWT

    This next part is easy. From Postman, scroll down to the bottom of the Authorization window until you see the Get New Access Token button. Click it; a new browser window will appear. Enter the user credentials (I’m using the one I created specifically for this demo), and if any pop-up blocker notifications appear, make sure you “allow pop-ups.”

    Once authenticated with IAM, you’ll be redirected back to Postman (remember, this is the Redirect URL you added in the Client Configuration section).

    Copy the entire JWT, and then click Use Token. You’ll now see the token in the Current Token field in Postman.

    HELP ME! Okay, if the Sign-in and/or Redirect didn't work for you, it might be that you are still signed into another OCI session, with a different user (a user that doesn't belong to this Domain). So, what I've had to do in the past is make sure I'm logged out of all OCI sessions, clear my cookies, close out my browser, and then restart with a new one. This has nothing to do with ORDS, but its what worked for me.

    ORDS configuration

    In these following sections, you will configure your ORDS JWT Profile and ORDS Privilege (associated with the Scope you’ve created). But first, you’ll need to decode your JWT.

    Decoding the JWT

    In the past, I’ve used two different tools for decoding these JWTs: JWT.io and JWT.ms. In this demonstration I’m using JWT.io.

    Paste your JWT into one of the decoders and copy the following values to your clipboard:

    • iss
    • scope
    • aud

    The values you’ll need:

    Creating the ORDS JWT profile

    Step one: Create your JWT Profile. This process “registers” a JWT profile with ORDS, so when an incoming request comes in, ORDS can “cross-reference” information related to Issuers, Audiences, and Scopes.

    The easiest way to do all this is to first navigate to an SQL Worksheet and then select the schema from the Navigator tab. Then, under the database objects drop-down menu, choose Packages.

    Click the OAUTH package to expand it, scroll to the bottom of that list, right click the CREATE_JWT_PROFILE PL/SQL procedure, and click Run. A slider will appear.

    Enter your details exactly like you see below:

    • P_ISSUER is https://identity.oraclecloud.com
    • P_AUDIENCE should be whatever your Primary audience is, with the trailing slash included!
    • P_JWK_URL
    FYI: The P_JWK_URL is a combination of [Your Domain URL] + /admin/v1/SigningCert/jwk (this is the endpoint identified in the OCI IAM Domains API doc). And since this is a /v1 endpoint, I assume there may be more. So, double-check the docs in case of a different/updated version. 

    Then, execute the PL/SQL procedure.

    Next, you’ll create an ORDS Privilege (to associate with our protected ORDS Resources).

    Creating an ORDS privilege

    Aka, the “thing” we are protecting and making accessible with a valid JWT. Navigate to your REST Workshop. From the Security tab, select Privileges.

    Create a new Privilege and name it exactly the same as the Scope you created in IAM. DO NOT INCLUDE the Primary audience in this scope name (In fact, you can’t; we won’t let you anyway 🤣)! In the Protected Modules tab, select the Module you want to protect. Either drag it from the Available Modules tab to the Selected Modules tab or use the arrows to move it over. Then click Create.

    A peek at the ORDS API I’m using:

    This just prints out the Current User, using the ORDS :current_user Implicit parameter.

    Accessing your protected Resource (aka ORDS API) with the JWT

    Now, you may access this protected resource. Unless you’ve taken an hour to set ORDS up, your token will still be valid (it’s valid for an hour). Otherwise, you’ll need to obtain a new, fresh JWT. Using the target ORDS API (mine is the “Hello :current_user” endpoint) in your Postman request field, click Send. You’ll see two things happen (if you are viewing your Postman Console):

    1. An initial POST request to the Token server, and
    2. A subsequent GET request to your target ORDS API

    I should clarify…you might not see that initial POST request because you would have accomplished this before you opened the Postman Console. So, if you’d like to see these requests in real time, do this:

    • Clear out your cookies
    • Delete your existing token
    • Clear out the Console
    • Open the Console back up
    • Request a new JWT
    • Request your ORDS API (your protected Resource)

    The End

    And that’s it, finished! If you’ve made it this far, congratulations. That was a lot of work. Time for a coffee break!

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

    1. Read up on Scopes here. ↩︎
    2. There are four different Grant Types in the OAuth 2.0 framework: Authorization Code, Implicit, Resource Owner Password Credentials, and Client Credentials. You can read more about them here. ↩︎

  • ORDS as a JotForm webhook: An actual customer use case

    Recently, we (one of our Support engineers and I) worked with a customer who had integrated various ORDS POST APIs with a web-based form application called JotForm. JotForm offers integrations for numerous workflows, some official and others you can create independently. That’s where we come in.

    In this example, a user would enter information into an online JotForm form, and when the user hits Submit, that would trigger a Webhook to an ORDS endpoint. But in this customer’s scenario, some of the data he was expecting wasn’t arriving in his database (he was on a 21c, but I tested these in a 23ai Autonomous database—the same still applies to both). For our customer, some of the fields in his table showed NULL.

    A Webhook example

    We wanted to understand better what was happening, so I created an account in Jotform and set up a Webhook similar to his configuration. This Webhook essentially forwards the JotForm form data to a target endpoint (an ORDS POST handler in our case).

    Webhooks are weird

    Here is what I don’t like about Webhooks (perhaps this is just my ignorance on full display):

    1. What if the request doesn’t work or fails? Are reattempts automatic, or is that user information lost forever?
    2. In this case, we have no real idea what is being sent to ORDS (at least not without some creative interrogation).
    3. Without spending too much time with Webhooks, would an APEX front end not be more straightforward (i.e., same ecosystem and it is a direct POST as opposed to a Webhook that forwards a POST request to an POST endpoint…say that five times fast! More on this later…).

    Where our problem starts

    Here, we have an online form in JotForm. On the front end, I’ve set up three fields (these are all customizable in their web application):

    1. Full Name, which consists of First Name and Last Name
    2. Contact number
    3. And a “Radio” button selection 1

    The Workflow

    A user enters their information and clicks Submit. However, after inspecting the Console and Network tabs (in your Developer tools), you’ll notice two things:

    1. No POST requests
    2. No available request body (What does JotForm send to us?)

    The absence of information is unsurprising since I’m working with a third-party tool (it’s a business, man!). There are enumerable reasons why you’d obscure information. Regardless of the cause, that obfuscation complicates matters when you want to ingest the form data into your database. You can follow the debugging advice to view the PHP output sent internally to JotForm. But I don’t know PHP, and I wouldn’t even know how to map PHP to JSON or SQL datatypes, for that matter.

    ORDS Implicit Parameters to the Rescue

    Here’s where you have to get creative. To better understand what was in that POST request, I relied on ORDS’ Implicit Bind parameters to understand what data was sent to us by the Webhook. Figuring this out required some trial and error on my part. But you can start with the :content_type bind parameter to see what kind of POST it is and go from there. In my example, I kept it simple and relied on the :content_type parameter and whatever was included in the POST body (up to this point, I wasn’t sure what to expect). I named that second value incoming_payload (which maps to the “PAYLOAD” column in the JOTFORM_DATA table)

    INFO: I used the :body_json bind parameter, for the sole fact that neither the :body nor the :body_text parameters seemed to work for me. 

    Resource Handler code

    A look at the ORDS Resource Handler code:

    -- The :content_type implicit bind parameter would "map" to the CONTENT_TYPE column in the 
    -- JOTFORM_DATA table, while incoming_payload would map to the PAYLOAD column. 
    
    DECLARE
        incoming_payload CLOB;
    BEGIN
        incoming_payload := :body_json;
        INSERT INTO JOTFORM_DATA VALUES ( :content_type,
                                          incoming_payload );
    
    END;

    Table DDL

    You’ll notice a table in the handler code called JOTFORM_DATA. Well, I ended up creating a table of two columns, as seen here:

    1. Content_type
    2. Payload
    CREATE TABLE JOTFORM_DATA 
        ( 
         CONTENT_TYPE VARCHAR2 (4000) , 
         PAYLOAD      VARCHAR2 (4000) 
        ) 
    ;
    Why VARCHAR2(4000)?  Two reasons. First, its just that it seems like VARCHAR2 is catch-all/the most flexible data type for testing like this. Secondly, since the PAYLOAD might be JSON or possibly malformed JSON, if I decide, I can always use an "IS JSON" check constraint. But that can come later.2

    Moving through the workflow

    With the Resource Handler code in place and the table defined, we can test this Webhook and see what it sends across HTTP.

    Next, a look at what my Autonomous database received:

    Okay! So now we have something to work with. And we know at least two things:

    1. The POST request is a multipart/form-data Content-Type, and3
    2. The payload we received appears to be JSON

    Next, we can copy the contents from the PAYLOAD column. And throw it into any number of online JSON Formatting tools, to see what we are working with:

    Final thoughts

    Well, I can already see why somebody might struggle with API integrations (in general, not just here). There is a lot to learn and A LOT of unknowns. Take this actual use case, for instance. Of course, you’d assume this payload to be multipart/form-data. You’d also expect the payload contents to be the JotForm form’s data. However, in this case, the form is pre-processed before it even reaches ORDS or the Autonomous database. So it is a good thing we did some investigation because there is way more “stuff” than we originally anticipated!

    Next, we’ll have to parse through the payload within the payload. For that, we can also use the ORDS Implicit Bind parameters, Automatic Binding, and JSON_VALUE functions. But we’ll save that for the next post. Stay tuned!

    Complete code

    Click for code

    You can find the most up-to-date code here. The code used in this post:

    -- Table DDL
    
    CREATE TABLE JOTFORM_DATA 
        ( 
         CONTENT_TYPE VARCHAR2 (4000) , 
         PAYLOAD      VARCHAR2 (4000) 
        ) 
    ;
    
    -- ORDS Resource Module PL/SQL
    
    BEGIN
      ORDS.ENABLE_SCHEMA(
          p_enabled             => TRUE,
          p_schema              => '[Your username/schema]',
          p_url_mapping_type    => 'BASE_PATH',
          p_url_mapping_pattern => '[same as p_schema in all lowercase]',
          p_auto_rest_auth      => FALSE);
        
      ORDS.DEFINE_MODULE(
          p_module_name    => 'ords_and_jotform',
          p_base_path      => '/oj/',
          p_items_per_page => 25,
          p_status         => 'PUBLISHED',
          p_comments       => NULL);
    
      ORDS.DEFINE_TEMPLATE(
          p_module_name    => 'ords_and_jotform',
          p_pattern        => 'jotform_data',
          p_priority       => 0,
          p_etag_type      => 'HASH',
          p_etag_query     => NULL,
          p_comments       => NULL);
    
      ORDS.DEFINE_HANDLER(
          p_module_name    => 'ords_and_jotform',
          p_pattern        => 'jotform_data',
          p_method         => 'POST',
          p_source_type    => 'plsql/block',
          p_items_per_page => 25,
          p_mimes_allowed  => NULL,
          p_comments       => NULL,
          p_source         => 
    'DECLARE
        incoming_payload CLOB;
    BEGIN
        incoming_payload := :body_json;
        INSERT INTO JOTFORM_DATA VALUES ( :content_type,
                                          incoming_payload );
    
    END;');
    
            
    COMMIT;
    
    END;

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

    1. What is a Radio button? ↩︎
    2. Directly from the JSON Developer’s Guide Release 23, “You can store JSON data in Oracle Database using columns whose data types are JSON, VARCHAR2, CLOB, or BLOB.” Although, the guide then continues to state that, “Oracle recommends that you use JSON data type, which stores JSON data in a native binary format.” This is an easy change to make early on though. Should I choose to, I can adjust future table columns to accommodate the JSON data type. ↩︎
    3. About the multipart/form-data Content-Type. This section discusses the multipart/form-data Content-Type, but also ExampleBoundaryString why it is used/included. You’ll see that in some of the screenshots throughout. ↩︎

  • Microsoft Entra OAuth2.0 JWTs and ORDS secure APIs Tutorial: Configuration and Testing

    Microsoft Entra OAuth2.0 JWTs and ORDS secure APIs Tutorial: Configuration and Testing

    Since ORDS first introduced support for authenticating with JWTs, there have been many questions surrounding configuration and testing. In this example, I cover, from start to finish:

    1. Registering an application in Microsoft Entra
    2. Creating a Client ID, Secret, and related Scope
    3. Use Postman and OAuth2.0 to request a JWT from Microsoft
    4. Decode the JWT (token) so you can use parts of it to create a JWT profile in ORDS
    5. Create an ORDS privilege with relevant roles, which you’ll use later for testing a protected ORDS endpoint (in Postman)

    If this sounds like what you are looking for, then read on.

    There’s a lot to cover here, so there’s no faffing about on this one. Let’s go!

    Configuring Microsoft Entra

    I assume you know how to locate Microsoft Entra (formerly Microsoft AD; details here).

    App registration

    Locate the Applications category, and from the drop-down menu, select App registrations.

    Fill out the information that best fits your scenario. Make sure you understand which Supported account type to use. I chose option three: Accounts in any organizational directory and personal Microsoft accounts.

    NOTE: If you want to test this OAuth code flow with a personal Microsoft email, GitHub, Skype, or another Microsoft "property" then you'll want to make the same choice as me.

    Retrieve the Callback URL from Postman

    You’ll notice in the Redirect URL section that there is another drop-down menu. You’ll want to select “Web.” You’ll need the Postman-provided “Callback” URL for this to work.

    NOTE: This walkthrough assumes you are using Postman, but Insomnia, cURL, and language-specific libraries will have similar, but distinct procedures for the "URL redirect."

    The Callback URL looks disabled in Postman, but you can still highlight and copy it. Copy and save it, then return to the App registration page (in Microsoft Entra).

    Complete app registration

    Click Register. A message will appear, notifying you of the successful app registration. Copy the Application (Client) ID and save it to your clipboard or a text editor. I’m using TextEdit, which you’ll see later.

    Create a new Client Secret ID and Value

    Next, you must create a Client Secret ID and Secret Value. Navigate to the Certificates & Secrets category. Then click the New client secret button. Add a description of the client’s secret along with an expiration date. 180 days is the recommendation.

    Once you’ve completed all the fields, click the Add button. Another confirmation message will appear. You should see your Client Secret Value and Client Secret ID in plain text. COPY your Client Secret Value now, this will be the only time it is visible! Copy your Client Secret ID, too.

    Paste them to your clipboard or text editor. We’ll be using them shortly and in more than one place!

    Add a scope pt 1

    Next, you’ll need to add a scope. Click the Expose an API category and the Add a scope button.

    Application ID URI

    If you haven’t added an Application ID URI, this wizard will prompt you to set one up before you can proceed. You’ll see this screen if this is your first time doing this.

    Microsoft Entra will have prepopulated the Application ID URI field for you. Click the Save and continue button.

    Add a scope pt 2

    The scope name appends to the Application ID URI as you fill in the first field. In this example, I’ve used a “dot notation” for the scope name. Since this is the naming convention we use in ORDS for privileges, I decided to keep everything standardized.

    Once you’ve filled in all the fields, click the Add scope button. Copy the scope name and save it to your clipboard or text editor. This scope name should combine your Application ID URI + the scope you just created. In my case, it looks like this:

    api://367ad5fd-a417-49f9-aed2-7b2290bd4ce3/my.ords.app.scope

    Obtain required OAuth endpoints

    The final step in Microsoft Entra is to retrieve all relevant OAuth2.0 and JWT endpoints. You’ll find 2/3rds of these from the Overview. Click Overview, then the Endpoints tab. You’ll want to copy and save the following endpoints:

    • OAuth 2.0 authorization endpoint(v2)
    • OAuth 2.0 token endpoint (v2)

    The final endpoint identifies the public keys used for decoding JWTs. For some reason, it’s not found in Microsoft Entra, so save this link and copy it to your clipboard or text editor.

    That’s it for Microsoft Entra configuration! Two more steps, and then we can test our secure ORDS endpoint.

    Request a JWT

    With any luck, you’ve been following along and saved all the required information for Postman. Here is what my clipboard looks like, hopefully yours does too:

    Configuring Postman

    Open Postman and enter the following information:

    • Authorization URL
    • Access Token URL
    • Client ID
    • Client Secret
    • Scope
    • Client Authentication

    Once you’ve filled everything in, click the Get New Access Token button.

    Granting access

    If you’ve kept everything identical to what I have, you should see the following prompts asking you to authorize/grant permission to this app so that you may acquire a JWT (Access Token).

    Redirecting back to Postman

    After the browser redirects you back to Postman, copy the token value. You’ll need to decode it next. You’ll use some of the properties (referred to as “claims” in this context) in the JWT when configuring ORDS.

    Decode the JWT

    There are many language libraries available to decode these JWTs. However, for a quick test scenario, these browser-based tools work well:

    Copy and paste the JWT string into one of the decode boxes to decode the “claims” contained in the JWT. Whichever one you choose, it doesn’t matter, they’ll look identical. Copy the “iss”, or issuer value, and the “aud”, or audience value. Save both of these values to your clipboard or text editor.

    Database Actions

    Finally, we move to Database Actions! There are two areas of focus here:

    1. The SQL Worksheet and
    2. The REST Workshop

    Navigate to the SQL Worksheet first.

    SQL Worksheet

    Regardless of whether you are an ADMIN user, a user with the ORDS ADMINISTRATOR role, or a regular database user, you can find the relevant CREATE_JWT_PROFILE PL/SQL procedure in the ORDS_SECURITY or ORDS_SECURITY_ADMIN packages. You can find them in the ORDS_METADATA schema. I could use either as the ADMIN.

    ORDS_SECURITY_ADMIN & ORDS_SECURITY

    After you’ve located the procedure, select and copy it.

    Obtaining the CREATE_JWT_PROFILE procedure

    Open a new SQL Worksheet and paste the procedure. It is missing a lot; hopefully, you’ve been taking notes.

    Editing the CREATE_JWT_PROFILE procedure

    To “fix” this procedure, you’ll need most of the things you’ve been saving along the way:

    • Issuer value
    • Audience value
    • Public keys (also known as JWKs or JSON Web Keys)

    Once your PL/SQL procedure looks the same (with your unique details, obviously), click the Run Script button. You’re done with the SQL Worksheet and can move on to the REST Workshop!

    REST Workshop

    You’ll need to create a privilege next. Remember, this privilege will be identical to the application’s scope name (which you created in Microsoft Entra). Navigate to Privileges, then click the Create Privilege button.

    ORDS privilege setup

    You can choose whatever you like for the Label and Description, but the name MUST match your application’s scope name.

    You’ll need to select SQL Developer for the Role and

    CORRECTION: Role is not necessary for this implementation. The association between the ORDS privilege name and the Application's scope takes care of this for you.

    choose the “target” resource you want to protect. In my case, it’s just an SQL statement that I’ve REST-enabled (it performs a simple query on a database table).

    When complete, you should see a privilege tile similar to the one in my example. The name of that privilege tile should match the name of your application’s scope.

    Test a secure ORDS resource with JWTs

    Believe it or not, you’ve finished the configuration. Onto testing!

    Test an ORDS endpoint

    Here, I select an endpoint I’ve protected (with the privilege I just created). I’ll use that in my Postman example.

    Executing in Postman

    I return to Postman, ensuring that none of my configurations has changed. This time, I’m adding my ORDS URI to the GET address bar. To be safe, I request a new Access Token and add that to my configuration. You’ll see that in the “Current Token” Field in the image below.

    Next, I click the Send button, and volià, it works like a charm!

    Success

    You’ll see a 200 OK Successful Response Status Code and my ORDS payload. And just like that, you’ve done it. You’ve just learned how to access a secure ORDS resource using JWTs!

    And what exactly are we looking at here? Well, these are baby names from 2023 US Social Security Card applications filtered using the following SQL:

    SELECT DISTINCT
        NAME
    FROM
        NAMES2023
    WHERE
        NAME LIKE '%eigh%'

    Forcing a 401 Response Status Code

    And in case you think I’m lying, set Auth Type to “No Auth” and reattempt the GET request. You should see a 401 Unauthorized Response Status Code. If you don’t, “Houston, we have a problem!

    The end

    I think this could help many people, so if you have made it this far, please share and bookmark it for later use. I’m also including the PDF version of the images. I didn’t include notes but placed the arrows and boxes with the utmost precision.

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

  • New ORDS feature: handling multiple files from a multipart/form-data POST request

    New ORDS feature: handling multiple files from a multipart/form-data POST request

    A new feature

    An ORDS user (or application) can now upload multiple files as part of a multipart/form-data POST request under various conditions. How can this be achieved with an ORDS endpoint?

    First, you must become acquainted with some newly introduced PL/SQL procedures and functions. You can find these in your ORDS Metadata.

    The single function and procedures used in this example.

    The code

    Now that you know where to look for these new procedures and functions, I’ll walk through my code.

    Resource Handler

    Here is the code I’m using for my Resource Handler (i.e., the ORDS API). It is a POST request that accepts the following parameters:

    • l_body_json
    • l_parameter_name
    • l_file_name
    • l_content_type
    • l_file_body

    Also, pay special attention to these three pieces of Resource Handler code:

    Line 2L_BODY_JSON CLOB := :BODY_JSON
    Lines 9 and 10ORDS.BODY_FILE_COUNT
    Line 11ORDS.GET_BODY_FILE
    These are new additions to ORDS; we’ll be revisiting them shortly.
    DECLARE
        L_BODY_JSON      CLOB := :BODY_JSON;
        L_PARAMETER_NAME VARCHAR2(4000);
        L_FILE_NAME      VARCHAR2(4000);
        L_CONTENT_TYPE   VARCHAR2(200);
        L_FILE_BODY      BLOB;
    
    BEGIN
        HTP.P( 'Number of files that were received: ' || ORDS.BODY_FILE_COUNT);
        FOR i IN 1..ORDS.BODY_FILE_COUNT LOOP
            ORDS.GET_BODY_FILE(
                P_FILE_INDEX     => i,
                P_PARAMETER_NAME => L_PARAMETER_NAME,
                P_FILE_NAME      => L_FILE_NAME,
                P_CONTENT_TYPE   => L_CONTENT_TYPE,
                P_FILE_BLOB      => L_FILE_BODY
            );
    
            INSERT INTO BODYFILESDEMO (
                FILENAME,
                CONTENTTYPE,
                FILEBODY
            ) VALUES ( L_FILE_NAME,
                       L_CONTENT_TYPE,
                       L_FILE_BODY );
            HTP.P('Inserted File: ' || L_FILE_NAME );
        END LOOP;
    
    END;

    Expand for this example’s complete PL/SQL ORDS Resource Module definition.
    -- Generated by ORDS REST Data Services 24.3.0.r2620924
    -- Schema: ORDSDEMO  Date: Fri Oct 04 07:28:00 2024 
    --
            
    BEGIN
      ORDS.DEFINE_MODULE(
          p_module_name    => 'body.files.demo',
          p_base_path      => '/v1/',
          p_items_per_page => 25,
          p_status         => 'PUBLISHED',
          p_comments       => NULL);
    
      ORDS.DEFINE_TEMPLATE(
          p_module_name    => 'body.files.demo',
          p_pattern        => 'example',
          p_priority       => 0,
          p_etag_type      => 'HASH',
          p_etag_query     => NULL,
          p_comments       => NULL);
    
      ORDS.DEFINE_HANDLER(
          p_module_name    => 'body.files.demo',
          p_pattern        => 'example',
          p_method         => 'POST',
          p_source_type    => 'plsql/block',
          p_mimes_allowed  => NULL,
          p_comments       => NULL,
          p_source         => 
    'DECLARE
        L_BODY_JSON      CLOB := :BODY_JSON;
        L_PARAMETER_NAME VARCHAR2(4000);
        L_FILE_NAME      VARCHAR2(4000);
        L_CONTENT_TYPE   VARCHAR2(200);
        L_FILE_BODY      BLOB;
    BEGIN
        HTP.P( ''Number of files that were received: '' || ORDS.BODY_FILE_COUNT);
        FOR i IN 1..ORDS.BODY_FILE_COUNT LOOP
            ORDS.GET_BODY_FILE(
                P_FILE_INDEX     => i,
                P_PARAMETER_NAME => L_PARAMETER_NAME,
                P_FILE_NAME      => L_FILE_NAME,
                P_CONTENT_TYPE   => L_CONTENT_TYPE,
                P_FILE_BLOB      => L_FILE_BODY
            );
    
            INSERT INTO BODYFILESDEMO (
                FILENAME,
                CONTENTTYPE,
                FILEBODY
            ) VALUES ( L_FILE_NAME,
                       L_CONTENT_TYPE,
                       L_FILE_BODY );
            HTP.P(''Inserted File: '' || L_FILE_NAME );
        END LOOP;
    
    END;');
    
        
            
    COMMIT;
    
    END;

    And here is the table I’m using:

    And the DDL, should you choose to recreate the table:

    CREATE TABLE BODYFILESDEMO 
        ( 
         ID          NUMBER (*,0) GENERATED BY DEFAULT AS IDENTITY 
            ( START WITH 1 CACHE 20 )  NOT NULL , 
         FILENAME    VARCHAR2 (200) , 
         CONTENTTYPE VARCHAR2 (200) , 
         FILEBODY    BLOB 
        ) 
    ;

    Practical example

    In practice, here is how everything works. I’m using Postman as a proxy for my application (i.e., client); I have it set up like this:

    Next, if I were to issue a POST request to my ORDS endpoint (using Postman), here are the results:

    In short, where before, I couldn’t INSERT multiple files via an ORDS POST request, now I can.

    Dissecting the POST

    Using the :body_json implicit parameter (you can review all the ORDS implicit parameters here), in concert with the ORDS.body_file_count function and the ORDS.get_body_file procedure, you can now send multipart/form-data with files (as seen in this POST request). You might also include JSON form data in the POST request body in these requests, but it is not compulsory.

    There are considerations though; read on.

    A closer look

    1. When POSTing a multipart/form-data request (such as the one in this example), I must bind this :BODY_JSON implicit parameter to something, even if that something will be an empty property (i.e., even if my POST request doesn’t have form data in JSON format).
    2. ORDS interprets this handler code as such:
      • ORDS knows to receive form data in JSON format; however, even if no form data is present, it knows that there may be multiple files in the POST request.
    3. And if there are numerous files, ORDS can use the ORDS.BODY_FILE_COUNT function to count how many files there are (using the FOR LOOP) and then with the ORDS.GET_BODY_FILE function perform the next operation, which in this case is an INSERT INTO table operation.
    DECLARE
        L_BODY_JSON      CLOB := :BODY_JSON;
        L_PARAMETER_NAME VARCHAR2(4000);
        L_FILE_NAME      VARCHAR2(4000);
        L_CONTENT_TYPE   VARCHAR2(200);
        L_FILE_BODY      BLOB;
    
    BEGIN
        HTP.P( 'Number of files that were received: ' || ORDS.BODY_FILE_COUNT);
        FOR i IN 1..ORDS.BODY_FILE_COUNT LOOP
            ORDS.GET_BODY_FILE(
                P_FILE_INDEX     => i,
                P_PARAMETER_NAME => L_PARAMETER_NAME,
                P_FILE_NAME      => L_FILE_NAME,
                P_CONTENT_TYPE   => L_CONTENT_TYPE,
                P_FILE_BLOB      => L_FILE_BODY
            );
    
            INSERT INTO BODYFILESDEMO (
                FILENAME,
                CONTENTTYPE,
                FILEBODY
            ) VALUES ( L_FILE_NAME,
                       L_CONTENT_TYPE,
                       L_FILE_BODY );
            HTP.P('Inserted File: ' || L_FILE_NAME );
        END LOOP;
    
    END;

    TL;DR implicit bind parameter review

    As far as a multipart/form-data with files POST request goes, ORDS “:BODY_" bind parameters now include and support the following:

    Impact Bind ParameterAbout files…About form data…About JSON…
    :BODYAccepts only one file (to be used with BLOB files).Multiple files are accessible using the new ORDS.BODY_FILE function and procedures.n/a
    :BODY_TEXTAccepts only one file (to be used with CLOB files).ORDS will automatically recognize form data and handle it accordingly.n/a
    :BODY_JSONORDS will automatically recognize form data and handle it accordingly.Form data will NOT be automatically recognized.Will be treated accordingly where form data, in JSON format, exists in the POST body.
    How ORDS now treats multipart/form-data with files POST requests under various conditions.

    One final note

    These changes only relate to multipart/form-data with files POST requests! Media types such as the following are not the focus of this article:

    • application/x-url-urlencoded
    • application/x-www-form-urlencoded
    • application/json, and
    • multipart/form-data without files

    The end

    That’s it for now. I’m sure there will be questions. I hope to put together a working example in a Flask app (a remix of our LiveLab workshop) soon. So stay tuned. I’ll update you as I learn more. Leave a comment if anything is unclear, and share if you think this is helpful.

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

  • Python script to retrieve objects from Oracle Cloud Bucket

    Python script to retrieve objects from Oracle Cloud Bucket

    For…reasons, I needed a way to retrieve all the .CSV files in a regional bucket in Oracle Cloud Object Storage, located at this address:

    https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsprod/b/moviestream_data_load_workshop_20210709/o

    You can visit it; we use it for one of our LiveLabs (this one), so I’m sure it will stay live for a while 😘. Once there, you’ll see all the available files in that bucket:

    FYI: A bucket is one of the three core components of Oracle Cloud's Object Storage. The other two are Objects and Namespaces (more details at Object Storage FAQs).

    In this case, there were more than just .CSV files; the script I created will also download those. But, of course, your specific situation may vary. Onto the Python script!

    The code

    Allow me to first address a few considerations and limitations:

    1. There is probably a way to create a function that accepts a parameter to make this even more elegant and compact. I’m not “there” yet. So, if you have a better solution, please comment, and I’ll amend my code!
    2. I still haven’t learned error and exception handling in Python. So anytime you see me “code in” print(...) that is basically me attempting to check to see if what I think will happen actually happens.
    3. I’m not sure if my range() and len() practices are appropriate, but it works, so I’m going with it!
    4. Don’t ask me how I even found out about the webbrowser library. I must have found it on a forum, or StackOverflow (hate that I can’t attribute credit to my savior).

    The code for real

    import requests
    import json
    import webbrowser
    
    url = 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsprod/b/moviestream_data_load_workshop_20210709/o/'
    
    r = requests.get(url)
    
    data = r.json()
    # print(data)
    
    newlist = []
    for name in data['objects']:
        newlist.append((name['name']))
    
    # print(newlist)
    length = len(newlist)
    newurl = []
    
    for i in range(length):
        newurl = url + newlist[i]
        webbrowser.open(newurl, new=0, autoraise=True)

    As you can see, with spacing, it’s less than 25 lines of code. I’m using a total of three libraries: requests, json, and webbrowser. The first two libraries you are probably familiar with, the third maybe not so much. Webbrowser() is great, when used correctly, the code will automatically open a new browser tab and execute whatever it is told to do (in this case, go to a URL and issue a GET request). Make sense?

    Not to worry, I’ll break this down into smaller blocks, to better understand what is happening.

    Lines 1-10

    import requests
    import json
    import webbrowser
    
    url = 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsprod/b/moviestream_data_load_workshop_20210709/o/'
    
    r = requests.get(url)
    data = r.json()
    # print(data)

    I’ve imported the three relevant libraries (remember if you don’t have these libraries, you can use pip and perform a pip install to grab them). And I assign URL equal to the target bucket. And from here I perform my first GET request. I do this for two main reasons:

    1. to remind me of the structure of the JSON, and
    2. because I am about to loop through all the available files in this bucket (so I need to capture them somehow)

    Since I’ve essentially assigned the value of r.json() equal to data, I can now print(data) to visually inspect the….um….data. After executing lines 1-10, I can see the corresponding output in my terminal:

    Now that I know this is working (because of my manual check), I can proceed to the next step.

    Lines 12-16

    Executing this code…

    import requests
    import json
    import webbrowser
    
    url = 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsprod/b/moviestream_data_load_workshop_20210709/o/'
    
    r = requests.get(url)
    
    data = r.json()
    print(data)
    
    newlist = []
    for name in data['objects']:
        newlist.append((name['name']))
    
    print(newlist)

    Will yield a new list:

    But how you ask? Good question, lots of trial and error. In short, I created an empty list, named newList. And later populated it with “stuff”. In normal people speak, lines 13-14 say the following,

    “There is an item, actually many items in the JSON payload we received from our GET request. For simplicity, lets call each of those items name, since ultimately what we want are file names. And we’ll use bracket notation to ‘drill’ down into more specific sections of the original JSON payload. And since we want just the file name (values), we want all the stuff that is in objects (if that doesn’t make sense review the lines 1-10 section again). We are then going to add to newList only the names (aka values) of the files. And these lines of code are what help us to populate the newList list.

    Maybe by this stage, you’ve caught on to what I’m trying to do. If not, I won’t spoil it for you 🙃.

    Lines 18-19

    And if you don’t quite “get it” yet, not to worry, it’ll click here in a second. Next, I need to see how many items are in this newlist I just created. Here is the code that is responsible for making that happen:

    import requests
    import json
    import webbrowser
    
    url = 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsprod/b/moviestream_data_load_workshop_20210709/o/'
    
    r = requests.get(url)
    
    data = r.json()
    # print(data)
    
    newlist = []
    for name in data['objects']:
        newlist.append((name['name']))
    
    # print(newlist)
    
    length = len(newlist)
    print(length)

    Here is the result in my terminal:

    It looks like I have 85 elements, things, stuffs in my list. I’ll use that knowledge for the final steps in my script.

    Lines 21-26

    import requests
    import json
    import webbrowser
    
    url = 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsprod/b/moviestream_data_load_workshop_20210709/o/'
    
    r = requests.get(url)
    
    data = r.json()
    # print(data)
    
    newlist = []
    for name in data['objects']:
        newlist.append((name['name']))
    
    # print(newlist)
    
    length = len(newlist)
    # print(length)
    
    newurl = []
    
    for i in range(length):
        newurl = url + newlist[i]
        webbrowser.open(newurl, new=0, autoraise=True)
        print(newurl)

    Line 21, you’ve seen this before (like 2 mins ago).

    However, lines 23-25 are new. If you recall my “considerations and limitations” section. I bet there is a better way to do this (or all of this actually), but this worked for me, so I’ll keep it. In short, we know we have 85 “iterables” (that’s the for i seen in line 23) and the range(length) portion simply says, “Begin at 0 and then keep going till you reach the entire length of…length()…which in this case is 85).”

    Next, I take the original URL (from line 5 in the code) and add each element of the newList to the end, making a newurl. From there, we open a web browser (new tabs actually) and visit that new amended URL aka newurl (I think more appropriately this is a URI, no?).

    And for the visual, a video of the entire process (no audio):

    And finally, for my own gratification, I’ll print out the newurl list just so I can marvel at my work. The new URLs:

    Now that I have these individual files, I can simply drag and drop them into my tables (in Database Actions obviously).

    As always, this and other code can be found in my blog repo. I hope this comes in handy someday! That’s all for now 😀!

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!

  • Build an ORDS API Resource Module, GET request with JavaScript fetch, display in HTML

    Recap

    This post is a continuation of a previous one, which can be found here. In this post, I’ll:

    If you are coming from the previous related post, then you’ll recall I used the following SQL query:

    My next step is to take this SQL and bring it to the REST Workshop, where I’ll turn it into an API.

    REST Workshop

    There are several ways you can navigate to the REST Workshop. Typically, I return to the Database Actions LaunchPad. From there, I select REST.

    The Handler code

    I've already created my Resource Module, Template, and Handler. I kept everything default, with no authentication enabled.

    The only thing I changed was the SQL query. I removed the final line, fetching the first 10 only. I want to be able to control the pagination of the API. If I were to keep that last line, this eventual endpoint would always only return the first 10 rows. And what if I want the next ten rows thereafter? Well, if I hard-code this, then I can’t really make that work. So, I chose to leave it open-ended.

    Technically, it is NOT open-ended because I retained the default pagination of 25. But, by removing that fetch first 10 rows condition, I can now fetch ALL rows that fit those parameters (in increments of 25).

    If I visit this new endpoint, it will appear like this:

    And if I collapse the items, you’ll see something that is EXTREMELY confusing. If I removed that fetch first 10 rows condition in the original SQL query, then why do we see a limit and offset of 10?

    The answer is because I actually set the Items Per Page equal to 10 (in the Resource Handler). This is the REST equivalent of a dirty joke. Consider yourself roasted…

    JavaScript

    With that endpoint live, I can take the API and drop it into some sample JavaScript and HTML code.

    JavaScript and HTML

    I learned a great deal about this JavaScript by reviewing this YouTube video. That is where I learned how to map through the items of my ORDS payload. And there was a refresher on JavaScript string interpolation (with template literals) too!

    PAUSE: Don't be too intimidated by string interpolation and template literals! Read the link I included, and take your time. If you are coming from Python, its similar to Jinja (when using Flask) and f-string literals 🙃.

    You can see that I’m using the map() constructor to iterate through all the data in my JSON payload. Remember, this was the payload in the items portion of my endpoint!

    I believe the item in list.map((item) is a reference to an individual item inline 4’s data.items. The reason why I think this is because if I change the items in lines 7-10 in my JavaScript to something random, like the name bobby, things start to break:

    However, if I change everything back to item, and start the live server in VS Code, I’ll be met with the following rendering:

    That’s it, though. Combining the ORDS API, the Fetch API, simple JavaScript, and HTML will allow you to create this straightforward web page.

    Reviewing Inspector, Console, Network

    I also have a few more screenshots, one each for the HTML Inspector, Console Log, and Client/Server Network. All of these show what is happening under the covers but in different contexts.

    Inspector

    In the Inspector, you can see how the JavaScript map() constructor plus the document.querySelector() in line 18 of the JavaScript code work in tandem with line 12 of the HTML script to display contents on the page:

    Console

    Here, you can see the items in the Console. This is because we added console.log(item)in line 19 of the JavaScript code.

    Network

    Finally, you can see the 200 GET request from our ORDS API. Then, on the far right of the screen, you can see the JSON payload coming from that same ORDS endpoint.

    Admittedly, the way the “Cast” is displayed is not correct. That is yet another array of cast members. And I’ve yet to learn how to structure that correctly. So, if you are reading this, and you know, let me know!

    Finally, all the code you’ve seen in this post can be found in my moviestreamjs github repository.

    That’s all for now!

    Follow

    And don’t forget to follow, like, subscribe, share, taunt, troll, or stalk me!