MacLochlainns Weblog

Michael McLaughlin's Technical Blog

Site Admin

Archive for the ‘sql’ Category

SQL Statement Management

without comments

It’s very difficult explaining to students new to relational databases how SQL works. There are many parts that seem intuitive and then there are others that confuse and confound.

For beginners, the idea that a SQL statement is simply a text string that you must dispatch to a SQL statement processing engine is new. That’s because they use an Integrated Development Environment (IDE) that hides, or abstracts the complexity, of how SQL executes.

I start my core SQL class by demonstrating how to run a text literal query without a FROM clause in MySQL Workbench, like this:

SELECT 'Hello World!' AS "Output";

After writing the query, I highlight everything except the semicolon and click the lightening bolt that dispatches the static string to the SQL statement engine. They see this result:

Then, I launch a mysql Monitor session and write the query with a semicolon to dispatch the SQL static string to the SQL statement engine:

SELECT 'Hello World!' AS "Output";

and, with a \g to dispatch the SQL static string to the SQL statement engine:

SELECT 'Hello World!' AS "Output"\g

Both queries return the same output, as shown below:

+--------------+
| output       |
+--------------+
| Hello World! |
+--------------+
1 row in set (0.00 sec)

Rewriting the query with a \G to dispatch the SQL static string to the SQL statement engine:

SELECT 'Hello World!' AS "Output"\G

Both queries return the following output:

*************************** 1. row ***************************
output: Hello World!
1 row in set (0.00 sec)

The next step requires removing the MySQL Workbench and MySQL Monitor from the demonstration. Without either of those tools, a Python program can demonstrate how to run a static SQL string.

The query is now a string literal into a query.sql file. The Python program reads the query.sql file, dispatches the embedded query, and displays the query results.

This is the query.sql file is:

SELECT 'Hello World!' AS "output";

This is the query.py file is:

#!/usr/bin/python
 
# Import libraries.
import sys
import mysql.connector
from mysql.connector import errorcode
 
# ============================================================
#  Use a try-catch block to read and parse a query from a
#  a file found in the same local directory as the Python
#  program.
# ============================================================
try:
  file = open('query.sql','r')
  query = file.read().replace('\n',' ').replace(';','')
  file.close()
 
except IOError:
  print("Could not read file:", fileName)
 
# ============================================================
#  Attempt connection in a try-catch block.
# ============================================================
# --------------------------------------------------------
#  Open connection, bind variable in query and format
#  query output before closing the cursor.
# --------------------------------------------------------
try:
  # Open connection.
  cnx = mysql.connector.connect(user='student', password='student',
                                host='127.0.0.1',
                                database='studentdb')
 
  # Create cursor.
  cursor = cnx.cursor()
 
  # Execute cursor, and coerce string to tuple.
  cursor.execute(query)
 
  # Display the rows returned by the query.
  for row in cursor:
    print(row[0])
 
  # Close cursor.
  cursor.close()
 
# --------------------------------------------------------
#  Handle MySQL exception 
# --------------------------------------------------------
except mysql.connector.Error as e:
  if e.errno == errorcode.ER_ACCESS_DENIED_ERROR:
    print("Something is wrong with your user name or password")
  elif e.errno == errorcode.ER_BAD_DB_ERROR:
    print("Database does not exist")
  else:
    print("Error code:", e.errno)        # error number
    print("SQLSTATE value:", e.sqlstate) # SQLSTATE value
    print("Error message:", e.msg)       # error message
 
# --------------------------------------------------------
#  Close connection after try-catch completes.
# --------------------------------------------------------
# Close the connection when the try block completes.
else:
  cnx.close()

In Linux or Unix from the relative directory where both the query.sql and query.py files are located:

./query.py

It returns:

Hello World!

These examples demonstrate that a query without variable substitution is only a static string. In all the cases, the static SQL strings are dispatched to the SQL engine by a terminator like a semicolon or through an ODBC library call that executes the static SQL string.

Written by maclochlainn

October 20th, 2024 at 1:38 pm

Troubleshoot Oracle Errors

without comments

It’s always a bit difficult to trap errors in SQL*Developer when you’re running scripts that do multiple things. As old as it is, using the SQL*Plus utility and spooling to log files is generally the fastest way to localize errors across multiple elements of scripts. Unfortunately, you must break up you components into local components, like a when you create a type, procedure, function, or package.

This is part of my solution to leverage in-depth testing of the Oracle Database 23ai Free container from an Ubuntu native platform. You can find this prior post shows you how to setup Oracle*Client for Ubuntu and connect to the Oracle Database 23ai Free container.

After you’ve done that, put the following oracle_errors Bash shell function into your testing context, or into your .bashrc file:

# Troubleshooting errors utility function.
oracle_errors ()
{
  #  Oracle Error prefixes qualify groups of error types, like
  #  this subset of error prefixes used in the Bash function.
  # ============================================================
  #  JMS - Java Messaging Errors
  #  JZN - JSON Errors
  #  KUP - External Table Access Errors
  #  LGI - File I/O Errors
  #  OCI - Oracle Call Interface Errors
  #  ORA - Oracle Database Errors
  #  PCC - Oracle Precompiler Errors
  #  PLS - Oracle PL/SQL Errors
  #  PLW - Oracle PL/SQL Warnings
  #  SP2 - Oracle SQL*Plus Errors
  #  SQL - SQL Library Errors
  #  TNS - SQL*Net (networking) Errors
  # ============================================================
 
  # Define a array of Oracle error prefixes.
  prefixes=("jms" "jzn" "kup" "lgi" "oci" "ora" "pcc" "pls" "plw" "sp2" "sql" "tns")
 
  # Prepend the -e for the grep utility to use regular expression pattern matching; and
  # use the ^before the Oracle error prefixes to avoid returning lines that may
  # contain the prefix in a comment, like the word lookup contains the prefix kup.
  for str in ${prefixes[@]}; do
    patterns+=" -e ^${str}"
  done
 
  # Display output from a SQL*Plus show errors command written to a log file when
  # a procedure, function, object type, or package body fails to compile. This
  # prints the warning message followed by the line number displayed.
  patterns+=" -e ^warning"
  patterns+=" -e ^[0-9]/[0-9]"
 
  # Assign any file filter to the ext variable.
  ext=${1}
 
  # Assign the extension or simply use a wildcard for all files.
  if [ ! -z ${ext} ]; then
    ext="*.${ext}"
  else
    ext="*"
  fi
 
  # Assign the number of qualifying files to a variable.
  fileNum=$(ls -l ${ext} 2>/dev/null | grep -v ^l | wc -l)
 
  # Evaluate the number of qualifying files and process.
  if [ ${fileNum} -eq "0" ]; then
    echo "[0] files exist."
  elif [ ${fileNum} -eq "1" ]; then
    fileName=$(ls ${ext})
    find `pwd` -type f | grep -in ${ext} ${patterns}  |
    while IFS='\n' read list; do
      echo "${fileName}:${list}"
    done
  else
    find `pwd` -type f | grep -in ${ext} ${patterns}  |
    while IFS='\n' read list; do
      echo "${list}"
    done
  fi
 
  # Clear ${patterns} variable.
  patterns=""
}

Now, let’s create a debug.txt test file to demonstrate how to use the oracle_errors, like:

ORA-12704: character SET mismatch
PLS-00124: name OF EXCEPTION expected FOR FIRST arg IN exception_init PRAGMA
SP2-00200: Environment error
JMS-00402: Class NOT found
JZN-00001: END OF input

You can navigate to your logging directory and call the oracle_errors function, like:

oracle_errors txt

It’ll return the following, which is file number, line number, and error code:

debug.txt:1:ORA-12704: character set mismatch
debug.txt:2:PLS-00124: name of exception expected for first arg in exception_init pragma
debug.txt:3:SP2-00200: Environment error
debug.txt:4:JMS-00402: Class not found
debug.txt:5:JZN-00001: End of input

There are other Oracle error prefixes but the ones I’ve selected are the more common errors for Java, JavaScript, PL/SQL, Python, and SQL testing. You can add others if your use cases require them to the prefixes array. Just a note for those new to Bash shell scripting the “${variable_name}” is required for arrays.

For a more complete example, I created the following files for a trivial example of procedure overloading in PL/SQL:

  1. tables.sql – that creates two tables.
  2. spec.sql – that creates a package specification.
  3. body.sql – that implements a package specification.
  4. test.sql – that implements a test case using the package.
  5. integration.sql – that calls the the scripts in proper order.

The tables.sql, spec.sql, body.sql, and test.sql use the SQL*Plus spool command to write log files, like:

SPOOL spec.txt
 
-- Insert code here ...
 
SPOOL OFF

The body.sql file includes SQL*Plus list and show errors commands, like:

SPOOL spec.txt
 
-- Insert code here ...
 
LIST
SHOW ERRORS
 
SPOOL OFF

The integration.sql script calls the tables.sql, spec.sql, body.sql, and test.sql in order. Corrupting the spec.sql file by adding a stray “x” to one of the parameter names causes a cascade of errors. After running the integration.sql file with the introduced error, the Bash oracle_errors function returns:

body.txt:2:Warning: Package Body created with compilation errors.
body.txt:148:4/13     PLS-00323: subprogram or cursor 'WARNER_BROTHER' is declared in a      
test.txt:4:ORA-06550: line 2, column 3: 
test.txt:5:PLS-00306: wrong number or types of arguments in call to 'WARNER_BROTHER' 
test.txt:6:ORA-06550: line 2, column 3:

I hope that helps those learning how to program and perform integration testing in an Oracle Database.

Written by maclochlainn

July 9th, 2024 at 4:37 pm

sqlplus on Ubuntu

without comments

With the release of Oracle Database 23c Free came the ability to update components of the container’s base operating system. Naturally, I took full advantage of that to build my development machine on an Ubuntu 22.0.4 VMware instance with a Docker implementation of the Oracle Database 23c Free container.

Unfortunately, there were changes from that release to the release of Oracle Database 23ai Free. Specifically, Oracle disallows direct patching of their published container’s native Unbreakable Linux 8. It appears the restriction lies in licensing but I haven’t been able to get a clear answer. Oracle’s instructions also shifted from using Docker to using Podman, which reduces the development platform to a limited type of Database as a Service (DaaS) environment. Moreover, that means it requires more skill to leverage the Oracle Database 23ai Free container as a real developer environment by installing and configuring Oracle’s Client software on the host Ubuntu operating system. Then, you must create a host of shared directories to the container to use external files or test external libraries.

While Oracle’s invocation of proprietary control of their native OS is annoying, it’s not nearly as onerous as Apple’s decision to not offer an Intel chip for their MacBook Pro machines. I’ve a hunch Oracle will grant access to their Oracle 23ai Free container in the future but for now this article shows you how to get native SQL*Plus access working.

As to Apple, while I’ve fixed my older machines by upgrading my Intel-based MacBook Pro (i7) to native Ubuntu, it still annoying. Yes, Tim Cooke, I’d rather run Ubuntu than sell back a wonderful piece of hardware on the cheap to Apple. I also did the same upgrade to my iMac 5K with 32 GB of RAM but swapped the cheap hybrid drive for a 2TB SSD.

Now to the technical content that lets you natively develop using Oracle’s SQL*Plus on Ubuntu against the Oracle Database 23ai Free container. While I love SQL*Developer, it has significant limits when testing large blocks of code. Whereas, good techniques, sqlplus, and Bash shell can simplify code development and integration testing.

Here are the steps to get sqlplus working on Ubuntu for your Oracle Database 23ai Free container:

  1. You need to download the following two zip files from the Oracle Instant Client Downloads for Linux x86-64 (64-bit) website, which assumes an Intel x86 Chip Architecture:

  2. Open a terminal as your default Ubuntu user and do the following to assume the root superuser responsibility:

    sudo sh

    As the root user, create the following directory for the Oracle Client software:

    mkdir /opt/oracle/instantclient_23_4

    As the root user, copy the previously downloaded files to the /opt/oracle directory (this assumes your default user is name as the student user:

    cp ~student/Downloads/instantclient*.zip  /opt/oracle/.

    As the root user, change directory with the cd command to the /opt/oracle directory and verify with the ls -al command that you have the following two files:

    total 120968
    drwxr-xr-x 4 root root      4096 Jul  3 14:29 .
    drwxr-xr-x 6 root root      4096 Jul  3 09:09 ..
    drwxr-xr-x 4 root root      4096 Jul  3 10:11 instantclient_23_4
    -rw-r--r-- 1 root root 118377607 Jul  3 14:29 instantclient-basic-linux.x64-23.4.0.24.05.zip
    -rw-r--r-- 1 root root   5471693 Jul  3 14:29 instantclient-sqlplus-linux.x64-23.4.0.24.05.zip

    As the root user, unzip the two zip files in the following order with the unzip command:

    unzip instantclient-basic-linux.x64-23.4.0.24.05.zip

    and, then

    unzip instantclient-sqlplus-linux.x64-23.4.0.24.05.zip

  3. As the root user, run these two commands:

    sudo sh -c "echo /opt/oracle/instantclient_23_4 > \
    /etc/ld.so.conf.d/oracle-instantclient.conf"
    sudo ldconfig

    Next, you’ll test the installation. As the root user, run these three commands, which you’ll later add to your standard Ubuntu user’s .bashrc file:

    export ORACLE_HOME=/opt/oracle/instantclient_23_4
    export LD_LIBRARY_PATH=$ORACLE_HOME
    export PATH=$PATH:$ORACLE_HOME

    As the root user, you can now test whether you can start the Oracle SQL*Plus client with the following command:

    sqlplus /nolog

    It should connect and return this:

    SQL*Plus: RELEASE 23.0.0.0.0 - Production ON Wed Jul 3 10:12:33 2024
    Version 23.4.0.24.05
     
    Copyright (c) 1982, 2024, Oracle.  ALL rights reserved.
     
    SQL>

    If you get this type of error, either you didn’t install the Oracle instant client basic libraries or you installed an incompatible version:

    sqlplus: error while loading shared libraries: libclntsh.so.23.1: cannot open shared object file: No such file or directory

    If you got the error, you’ll need to readdress the installation of the Oracle instant client typically.

    Another type of error can occur if you get ahead of these instructions and try to connect to the Oracle Database 23ai Free container with syntax like this:

    sql> connect c##student/student@free

    because you’ll most likely get an error like this:

    ERROR:
    ORA-12162: TNS:net service name is incorrectly specified
    Help: https://docs.oracle.com/error-help/db/ora-12162/

    The error occurs because you haven’t setup the Oracle Net Services, which is level 5 in the OSI (Open System Interconnection) Model. In Oracle-speak, that means you haven’t setup a tnsnames.ora file, failed to put the tnsnames.ora file in the right place, or failed to set the $TNS_ADMIN environment variable correctly.

  4. While there are many ways to setup a tnsnames.ora file, the best way is to follow Oracle’s recommended approaches. In the Oracle client approach you should put the tnsnames.ora file in the $ORACLE_HOME/network/admin directory and use the $TNS_ADMIN environment variable to point to it. Unfortunately, that approach doesn’t work when you’re installing the Oracle client software unless you want to play with mount points. It’s easiest to create a hidden directory in your sandbox user, which is student in this example.

    As the root user, use the mkdir command to create the .oracle directory in your student user directory:

    mkdir /home/student/.oracle

    As the student user, navigate to the /home/student/.oracle directory and create the tnsnames.ora file with the following text:

    # tnsnames.ora Network Configuration FILE:
     
    FREE =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.0)(PORT = 1521))
        (CONNECT_DATA =
          (SERVER = DEDICATED)
          (SERVICE_NAME = FREE)
        )
      )
     
    LISTENER_FREE =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.0)(PORT = 1521))
     
    FREEPDB1 =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.0)(PORT = 1521))
        (CONNECT_DATA =
          (SERVER = DEDICATED)
          (SERVICE_NAME = FREEPDB1)
        )
      )
     
    EXTPROC_CONNECTION_DATA =
      (DESCRIPTION =
         (ADDRESS_LIST =
           (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC_FOR_FREE))
         )
         (CONNECT_DATA =
           (SID = PLSExtProc)
           (PRESENTATION = RO)
         )
      )

    Exit the root user to your student user. As the student user set the $TNS_ADMIN environment variable like:

    export TNS_ADMIN=$HOME/.oracle

    Assuming you’ve already created a container user, like c##student, connect to sqlplus with the following syntax:

    sqlplus c##student/student@free

    You should see the following when connection to an Oracle 23c Container:

    SQL*Plus: Release 23.0.0.0.0 - Production on Wed Jul 3 15:05:10 2024
    Version 23.4.0.24.05
     
    Copyright (c) 1982, 2024, Oracle.  All rights reserved.
     
    Last Successful login time: Wed Jul 03 2024 10:52:13 -06:00
     
    Connected to:
    Oracle Database 23c Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
    Version 23.3.0.23.09
     
    SQL>

    You should see the following when connection to an Oracle 23ai Container:

    SQL*Plus: Release 23.0.0.0.0 - Production on Sat Jul 20 11:05:08 2024
    Version 23.4.0.24.05
     
    Copyright (c) 1982, 2024, Oracle.  All rights reserved.
     
    Last Successful login time: Sat Jul 20 2024 10:41:38 -06:00
     
    Connected to:
    Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
    Version 23.4.0.24.05
     
    SQL>
  5. The last step adds all of the configuration settings into the .bashrc file. Before we do that, you may want to add the rlwrap utility library so you can use the up-arrow to navigate the sqlplus history. You install it as the root or sudo user on Ubuntu, like

    apt install -y rlwrap

    If you want to manually check what you’re removing, use the following command as the root user:

    apt autoremove

  6. The last step requires that you put the environment variables into the student user’s .bashrc shell script, and add a sqlplus function to take advantage of the new libraries added to read your prior history inside the SQL*Plus command line.

    You should edit the .bashrc file and add the following environment variables and sqlplus() function:

    # Configure Oracle Client software.
    export ORACLE_HOME=/opt/oracle/instantclient_23_4
    export LD_LIBRARY_PATH=$ORACLE_HOME
    export PATH=$PATH:$ORACLE_HOME
    export TNS_ADMIN=$HOME/.oracle
     
    # A user-defined function to wrap the sqlplus history.
    sqlplus () 
    {
        # Discover the fully qualified program name. 
        path=`which rlwrap 2>/dev/null`
        file=''
     
        # Parse the program name from the path.
        if [ -n ${path} ]; then
            file=${path##/*/}
        fi;
     
        # Wrap when there is a file and it is rewrap.
        if [ -n ${file} ] && [[ ${file} = "rlwrap" ]]; then
            rlwrap $ORACLE_HOME/sqlplus "${@}"
        else
            echo "Command-line history unavailable: Install the rlwrap package."
            $ORACLE_HOME/sqlplus "${@}"
        fi
    }

    You should remember that when you access sqlplus from the Ubuntu environment the TNS net service name is required. If you should forget to include it like this:

    sqlplus c##student/student

    You’ll get the following error:

    ERROR:
    ORA-12162: TNS:net service name is incorrectly specified
    Help: https://docs.oracle.com/error-help/db/ora-12162/

    The correct way is:

    sqlplus c##student/student@free

As always, I hope this helps those looking for a solution.

Written by maclochlainn

July 3rd, 2024 at 1:58 pm

Updating Nested Tables

without comments

This two-part series covers how you update User-Defined Types (UDTs) and Attribute Data Types (ADTs). There are two varieties of UDTs. One is a column of a UDT object type and the other a UDT collection of a UDT object type.

You update nested UDT columns by leveraging the TABLE function. The TABLE function lets you create a result set, and access a UDT object or collection column. You need to combine the TABLE function and a CROSS JOIN to update elements of a UDT collection column.

ADTs are collections of a scalar data types. Oracle’s scalar data types are DATE, NUMBER, CHAR and VARCHAR2 (or, variable length strings). ADTs are unique and from some developer’s perspective difficult to work with.

The first article in this series shows you how to work with a UDT object type column and a UDT collection type. The second article will show you how to work with an ADT collection type.

PL/SQL uses ADT collections all the time. PL/SQL also uses User-Defined Types (UDTs) collections all the time. UDTs can be record or object types, or collections of records and objects. Record types are limited, and only work inside a PL/SQL scope. Object types are less limited and you can use them in a SQL or PL/SQL scope.

Object types come in two flavors. One acts as a typical record structure and has no methods and the other acts like an object type in any object-oriented programming language (OOPL). This article refers only to object types like typical record structures. That means when you read ADTs you should think of a SQL collection of a scalar data type, and when you read UDTs you should think of a SQL collection of an object type without methods.

You can create tables that hold nested tables. Nested tables can use a SQL ADT or UDT data type. Inserting data into nested tables is straightforward when you understand the syntax, but updating nested tables can be complex. The complexity exists because Oracle treats nested tables of ADTs differently than UDTs. My article series will show you how to simplify updating ADT columns.

That’s why it has two parts:

  • How you insert and update rows with UDT columns and collection columns
  • How you insert and update rows with ADT collection columns

If you’re asking yourself why there isn’t a section for deleting rows, that’s simple. You delete them the same way as you would any other row, using the DELETE statement.

How you insert and update rows with UDT columns and collection columns

This section shows you how to create a table with a UDT column and a UDT collection column. It also shows you how to insert and update the embedded columns.

You insert into any ordinary UDT column by prefacing the data with a constructor name. A constructor name is the same as a UDT name. The following creates an address_type UDT that you will use inside a customer table:

SQL> CREATE OR REPLACE
  2    TYPE address_type IS OBJECT
  3    ( street  VARCHAR2(20)
  4    , city    VARCHAR2(30)
  5    , state   VARCHAR2(2)
  6    , zip     VARCHAR2(5));
  7  /

You should take note that the address_type UDT doesn’t have any methods. All object types without methods have a default constructor. The default constructor follows the same rules as tables in the database.

Create the sample customer table with an address column that uses the address_type UDT as its data type; for instance:

SQL> CREATE TABLE customer
  2  ( customer_id  NUMBER
  3  , first_name   VARCHAR2(20)
  4  , last_name    VARCHAR2(20)
  5  , address      ADDRESS_TYPE
  6  , CONSTRAINT pk_customer PRIMARY KEY (customer_id));

Line 5 defines the address column with the address_type UDT. You insert a row with an embedded address_type data record as follows:

SQL> INSERT
  2  INTO   customer
  3  VALUES
  4  ( customer_s.NEXTVAL
  5  ,'Oliver'
  6  ,'Queen'
  7  , address_type( street => '1 Park Place'
  8                , city   => 'Starling City'
  9                , state  => 'NY'
 10                , zip    => '10001'));

Lines 7 through 10 includes the constructor call to the address_type UDT. The address_type constructor uses named notation rather than positional notation. You should always try to use named notation for object type constructor calls.

Updating an element of a UDT object structure is straightforward, because you simply refer to the column and a member of the UDT object structure. The syntax for that type of UPDATE statement follows:

SQL> UPDATE customer c
  2  SET    c.address.state = 'NJ'
  3  WHERE  c.first_name = 'Oliver'
  4  AND    c.last_name = 'Queen';

The address_type UDT works for an object structure but not for a UDT collection. You need to add a column to differentiate between rows of the nested collection. You can redefine the address_type UDT as follows:

SQL> CREATE OR REPLACE
  2    TYPE address_type IS OBJECT
  3    ( status  VARCHAR2(8)
  4    , street  VARCHAR2(20)
  5    , city    VARCHAR2(30)
  6    , state   VARCHAR2(2)
  7    , zip     VARCHAR2(5));
  8  /

After creating the UDT object type, you need to create an address_table UDT collection of the address_type UDT object type. You use the following syntax to create the SQL collection:

SQL> CREATE OR REPLACE
  2    TYPE address_table IS TABLE OF address_type;
  3  /

Having both the UDT object and collection types, you can drop and create the customer table with the following syntax:

SQL> CREATE TABLE customer
  2  ( customer_id  NUMBER
  3  , first_name   VARCHAR2(20)
  4  , last_name    VARCHAR2(20)
  5  , address      ADDRESS_TABLE
  6  , CONSTRAINT pk_customer PRIMARY KEY (customer_id))
  7  NESTED TABLE address STORE AS address_tab;

Line 5 defines the address column as a UDT collection. Line 7 instructs how to store the UDT collection as a nested table. You designate the address column as the nested table and store it as an address_tab table. You can access the nested table only through its container, which is the customer table.

You can insert rows into the customer table with the following syntax. This example stores a single row with two elements of the address_type in the nested table:

SQL> INSERT
  2  INTO   customer
  3  VALUES
  4  ( customer_s.NEXTVAL
  5  ,'Oliver'
  6  ,'Queen'
  7  , address_table(
  8        address_type( status   => 'Obsolete'
  9                    , street => '1 Park Place'
 10                    , city => 'Starling City'
 11                    , state => 'NY'
 12                    , zip => '10001')
 13      , address_type( status   => 'Current'
 14                    , street => '1 Dockland Street'
 15                    , city => 'Starling City'
  16                    , state => 'NY'
 17                    , zip => '10001')));

Lines 7 through 17 have two constructor calls for the address_type UDT object type inside the address_table UDT collection. After you insert an address_table UDT collection, you can query an element by using the SQL built-in TABLE function and a CROSS JOIN. The TABLE function returns a SQL result set. The CROSS JOIN lets you create cross product that you can filter inside the WHERE clause.

A CROSS JOIN between two tables or a table and result set from a nested table matches every row in the customer table with every row in the nested table. A best practice would include a WHERE clause that filters the nested table to a single row in the result set.

The syntax for such a query is complex, and follows below:

SQL> COL first_name  FORMAT A8  HEADING "First|Name"
SQL> COL last_name   FORMAT A8  HEADING "Last|Name"
SQL> COL street      FORMAT A20 HEADING "Street"
SQL> COL city        FORMAT A14 HEADING "City"
SQL> COL state       FORMAT A5  HEADING "State"
SQL> SELECT c.first_name
  2  ,      c.last_name
  3  ,      a.street
  4  ,      a.city
  5  ,      a.state
  6  FROM   customer c CROSS JOIN TABLE(c.address) a
  7  WHERE  a.status = 'Current';

As mentioned, the TABLE function on line 6 translates the UDT collection into a SQL result set, which acts as a temporary table. The alias a becomes the name of the temporary table. Lines 3, 4, 5, and 7 all reference the temporary table.

The query should return the following for the customer and their current address value:

First    Last
Name     Name     Street               City           State
-------- -------- -------------------- -------------- -----
Oliver   Queen    1 Dockland Street    Starling City  NY

Oracle thought through the fact that you should be able to update UDT collections. The same TABLE function lets you update elements in the nested table. You can update the elements in nested UDT tables provided you create a unique key, such as a natural key or primary key. Oracle’s syntax doesn’t support constraints on nested tables, which means you need to implement it by design and protect by carefully controlling inserts and updates to the nested table.

You can update the state value of the current address with the following UPDATE statement:

SQL> UPDATE TABLE(SELECT c.address
  2               FROM   customer c
  3               WHERE  c.first_name = 'Oliver'
  4               AND    c.last_name = 'Queen') a
  5  SET    a.state = 'NJ'
  6  WHERE  a.status = 'Current';

Line 5 sets the current state value in the address_table UDT nested table. Line 6 filters the nested table to the current address element. You need to ensure that any UDT object type holds a member attribute or set of member attributes that holds a unique value. That’s because you need to ensure that there’s a way to find a unique element within a UDT collection. If you require the table, you should see the change inside the nested table.

Oracle does not provide equivalent syntax for such a change in an ADT collection type. The second article in this series show you how to implement PL/SQL functions to solve that problem.

Written by maclochlainn

May 9th, 2024 at 9:38 pm

Oracle23ai Ubuntu Install

without comments

What to do with a Late 2015 iMac with an i7 Quad CPU running at 3.4 GHz, 32 GB or RAM, a 5K Display and an almost warn out hybrid 1 TB hard disk? You could sell it to Apple for pennies, but why enrich them. I opted to upgrade it with an OWC kit that had a 2 TB SSD Disk. Then, I installed Ubuntu 22.0.4 and built a DaaS (Database as a Service) machine with Oracle Database 23ai in a Docker container, and MySQL 8 and PostgreSQL 14 natively.

I’ve posted on installing MySQL 8 and PostgreSQL 14 on Ubuntu before when I repurposed my late 2014 MacBook Pro. This post covers the installation of Docker and Oracle Database 23ai.

Install Docker

Contrary to the instructions, you should do the following as a sudoer user:

sudo apt install -y docker.io

Install all dependency packages using the following command:

sudo snap install docker

You should see the following:

docker 20.10.24 from Canonical✓ installed

You can verify the Docker install with the following command:

sudo docker --version

It should show something like this:

Docker version 24.0.5, build 24.0.5-0ubuntu1~22.04.1

You can check the pulled containers with the following command but at this point there should be no pulled containers.

sudo docker images

At this point, a docker group already exists but you need to add your user to the docker group with the following command:

sudo usermod -aG docker $USER

Using the Docker Commands:

  • To activate the logging, utilize the -f parameter.
  • To divide JSON, use Docker by default; to extract individual keys, use JQ.
  • In your Container file, there are quite a few areas where commands may be specified.
  • Posting to the volumes could be more effective while the picture is being built.
  • Docker offers a highly efficient way to create an alias for its own built-in commands. This makes it easier to set up and handle lengthy and enormous orders. These alias values are stored in the directories /.bashrc or and /.bash_aliases.
  • Docker offers further assistance to remove unused code fragments from the installation of the container.
  • Docker always favors reading statements from the container file that have not changed. Therefore, time savings may be realized by arranging what is shown in the container file in a way that ensures the elements that are susceptible to change are shown towards the end of the document and those that are most likely to undergo change are shown at the top.

Install Oracle Database 23ai Free in a Docker container

Use the following command to pull and install the Oracle Database 23ai container:

sudo docker run --name oracle23ai -p 1521:1521 -p 5500:5500 -e ORACLE_PWD=cangetin container-registry.oracle.com/database/free:latest

After installing the Oracle Database 23ai Free container, you can access it as the root user by default with this syntax:

docker exec -it -u root oracle23ai bash

At the root prompt, you can connect to the system schema with the following command:

sqlplus system/cangetin@FREE

You should see the following:

SQL*Plus: RELEASE 23.0.0.0.0 - Production ON Thu May 9 03:56:57 2024
Version 23.4.0.24.05
 
Copyright (c) 1982, 2024, Oracle.  ALL rights reserved.
 
LAST SUCCESSFUL login TIME: Wed Apr 24 2024 21:23:00 +00:00
 
Connected TO:
Oracle DATABASE 23ai Free RELEASE 23.0.0.0.0 - Develop, Learn, AND Run FOR Free
Version 23.4.0.24.05
 
SQL>

Create a c##student as a sandbox user:

After you create and provision the Oracle Database 21ai Free, you can create a c##student sand-boxed user with the following two step process.

  1. Create a c##student Oracle user account with the following command as the system user:

    CREATE USER c##student IDENTIFIED BY student
    DEFAULT TABLESPACE users QUOTA 200M ON users
    TEMPORARY TABLESPACE temp;

  2. Grant necessary privileges to the newly created c##student user:

    GRANT CREATE CLUSTER, CREATE INDEXTYPE, CREATE OPERATOR
    ,     CREATE PROCEDURE, CREATE SEQUENCE, CREATE SESSION
    ,     CREATE TABLE, CREATE TRIGGER, CREATE TYPE
    ,     CREATE VIEW TO c##student;

  3. Connect to the sandboxed user with the following syntax (by the way it’s a pluggable user account as qualified in Oracle Database 12c forward):

    SQL> CONNECT c##student/student@FREE

    or, disconnect and reconnect with this syntax:

    sqlplus system/cangetin@FREE

Set Docker Oracle 23ai to start always

Assuming that your container name was oracle23ai, as qualified above, you can run the following command to automatically restart the Docker container:

docker update --restart=always `docker ps -aqf "name=oracle23ai"`

The docker command inside the backquotes uses the Docker instance’s name to return the Docker container_id value, which can also be seen when you run the following command:

docker ps

which returns:

CONTAINER ID   IMAGE                                                COMMAND                  CREATED       STATUS                    PORTS                                                                                  NAMES
b211f494e692   container-registry.oracle.com/database/free:latest   "/bin/bash -c $ORACL…"   13 days ago   Up 18 minutes (healthy)   0.0.0.0:1521->1521/tcp, :::1521->1521/tcp, 0.0.0.0:5500->5500/tcp, :::5500->5500/tcp   oracle23ai

The Docker container_id value is required when you perform a Docker update operation.

Configuring your Docker Oracle 23ai environment

Unless you like memorizing the Docker command-line, you may automate connecting as the root user or add a sand boxed user. The root user typically has more power than you need to perform ordinary development and use-case testing tasks.

A sand boxed user has narrow access, can’t start and stop the database instance or perform Oracle Datasbase 23ai administration. In this segment, you’ll learn how to create a couple local Bash functions to simplify your use of the Oracle Database 23ai container; and how to extend the configuration of Oracle’s Docker container:

  • Adding a student user to the Docker container and configuring it to access the Oracle Database 23ai locally from within the Docker container using a direct sqlplus connection.
  • Configuring the Docker container to support external files and leverage a shared directory with your base operating system.

Automating Docker instance connections:

The following shows you how to add a local Bash function to automate access to the Docker container from the Linux command-line. You put the following Bash function in your base Linux operating system’s user .bashrc file:

  1. Create the following Bash function:

    # User defined function to launch Oracle 23 ai container
    # as the root user.
    admin () 
    {
        # Discover the fully qualified program name. 
        path=`which docker 2>/dev/null`
        file=''
     
        # Parse the program name from the path.
        if [ -n ${path} ]; then
            file=${path##/*/}
        fi
     
        # Wrap when there is a file and it is rewrap.
        if [ -n ${file} ] && [[ ${file} = "docker" ]]; then
            python -c "import subprocess; subprocess.run(['docker exec -it --user root oracle23ai bash'], shell=True)" 
        else
            echo "Docker is unavailable: Install the docker package."
        fi
    }

  2. After you source the .bashrc file or simply reconnect as to the terminal as your user, which resources the .bashrc file, you can access the oracle23ai Docker instance with this command:

    admin

    It will display a new prompt with the root user and the Docker container_id value, like:

    [root@b211f494e692 oracle]#

    You can exit the Docker container by typing exit at the Linux command line. If you curious what version of Linux you’re using inside the Docker instance, you can’t use the uname command because it returns the hosting Linux distribution (distro). You must use the following when inside the Docker instance:

    cat /etc/os-release

    or, outside the Docker instance you can use the following docker command:

    docker exec oracle23ai cat /etc/os-release

    Either way, for an Oracle Database 23ai container, it should return:

    NAME="Oracle Linux Server"
    VERSION="8.9"
    ID="ol"
    ID_LIKE="fedora"
    VARIANT="Server"
    VARIANT_ID="server"
    VERSION_ID="8.9"
    PLATFORM_ID="platform:el8"
    PRETTY_NAME="Oracle Linux Server 8.9"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:oracle:linux:8:9:server"
    HOME_URL="https://linux.oracle.com/"
    BUG_REPORT_URL="https://github.com/oracle/oracle-linux"
     
    ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
    ORACLE_BUGZILLA_PRODUCT_VERSION=8.9
    ORACLE_SUPPORT_PRODUCT="Oracle Linux"
    ORACLE_SUPPORT_PRODUCT_VERSION=8.9

    Unfortunately, Oracle has appeared to block updates to the Oracle Unbreakable Linux 8 instance inside the container, which makes native SQL*Plus use more difficult. That’s because you’ll need to install the Oracle SQL*Plus client in the hosting Operating System.

    I’ve written a separate blog post that instructs you on how to install and use Oracle SQL*Plus client on Ubuntu.

Install SQL Developer in the base Linux operating system

The first steps are installing the Java Runtime Environment and Java Development Kit, and then downloading, installing and configuring SQL Developer. These are the required steps:

  1. Install the Java Runtime Environment:

    sudo apt install default-jre

    The log file for this is:

  2. Install the Java Runtime Environment:

    sudo apt install -y default-idk

    The log file for this is:

  3. Download SQL Developer from here; and then install SQL Developer to the /opt directory on your Ubuntu local instance:

    Use the following command to unzip the SQL Developer files to the /opt directory:

    sudo unzip ~/Downloads/sqldeveloper-23.1.0.097.1607-no-jre.zip
  4. Create the following /usr/local/bin/sqldeveloper symbolic link:

    sudo ln -s /opt/sqldeveloper/sqldeveloper.sh /usr/local/bin/sqldeveloper
  5. Edit the /opt/sqldeveloper/sqldeveloper.sh file by replacing the following line:

    cd "`dirname $0`"/sqldeveloper/bin && bash sqldeveloper $*

    with this version:

    /opt/sqldeveloper/sqldeveloper/bin/sqldeveloper $*
  6. Now, you can launch SQL Developer from any location on your local Ubuntu operating system, like:

    sqldeveloper
  7. You can now connect as the system user through SQL Developer to the Oracle Database 23ai Free Docker instance with the following connection information:

    (Excuse recycling the version from 21c but I didn’t see any utility in making a new screen shot.)

  8. You can also create a Desktop shortcut by creating the sqldeveloper.desktop file in the /usr/share/applications directory. The SQL Developer icon is provided in the sqldeveloper base directory.

    You should create the following sqldeveloper.desktop file to use a Desktop shortcut:

    [Desktop Entry]
    Name=Oracle SQL Developer
    Comment=SQL Developer from Oracle
    GenericName=SQL Tool
    Exec=/usr/local/bin/sqldeveloper
    Icon=/opt/sqldeveloper/icon.png
    Type=Application
    StartupNotify=true
    Categories=Utility;Oracle;Development;SQL;

As always, I hope this helps those trying to accomplish this task.

Written by maclochlainn

May 8th, 2024 at 10:12 pm

Parametric Queries

without comments

In 2021, I wrote a MySQL example for my class on the usefulness of Common Table Expressions (CTEs). When discussing the original post, I would comment on how you could extend the last example to build a parametric reporting table.

Somebody finally asked for a concrete example. So, this explains how to build a sample MySQL parametric query by leveraging a filter cross join and tests the parameter use with a Python script.

You can build this in any database you prefer but I used a studentdb database with the sakila sample database installed. I’ve granted privileges to both databases to the student user. The following SQL is required for the example:

-- Conditionally drop the levels table.
DROP TABLE IF EXISTS levels;
 
-- Create the levels list.
CREATE TABLE levels
( level_id       int unsigned primary key auto_increment
, parameter_set  enum('Three','Five')
, description    varchar(20)
, min_roles      int
, max_roles      int );
 
-- Insert values into the list table.
INSERT INTO levels
( parameter_set
, description
, min_roles
, max_roles )
VALUES
 ('Three','Hollywood Star', 30, 99999)
,('Three','Prolific Actor', 20, 29)
,('Three','Newcommer',1,19)
,('Five','Newcommer',1,9)
,('Five','Junior Actor',10,19)
,('Five','Professional Actor',20,29)
,('Five','Major Actor',30,39)
,('Five','Hollywood Star',40,99999);

The sample lets you use the three or five value labels while filtering on any partial full_name value as the result of the query below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
-- Query the data.
WITH actors AS
 (SELECT   a.actor_id
  ,        a.first_name
  ,        a.last_name
  ,        COUNT(*) AS num_roles
  FROM     sakila.actor a INNER JOIN sakila.film_actor fa
  ON       a.actor_id = fa.actor_id
  GROUP BY actor_id)
SELECT   CONCAT(a.last_name,', ',a.first_name) full_name
,        l.description
,        a.num_roles
FROM     actors a CROSS JOIN levels l
WHERE    a.num_roles BETWEEN l.min_roles AND l.max_roles
AND      l.parameter_set = 'Five'
AND      a.last_name LIKE CONCAT('H','%')
ORDER BY a.last_name
,        a.first_name;

They extends a concept exercise found in Chapter 9 on subqueries in Alan Beaulieu’s Learning SQL book.

This is the parametric Python program, which embeds the function locally (to make it easier for those who don’t write a lot of Python). You could set the PYTHONPATH to a relative src directory and import your function if you prefer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
#!/usr/bin/python
 
# Import the libraries.
import sys
import mysql.connector
from mysql.connector import errorcode
 
# ============================================================
 
# Define function to check and replace arguments.
def check_replace(argv):
 
  # Set defaults for incorrect parameter values.
  defaults = ("Three","_")
 
  # Declare empty list variables.
  inputs = []
  args = ()
 
  # Check whether or not parameters exist after file name.
  if isinstance(argv,list) and len(argv) != 0:
 
    # Check whether there are at least two parameters.
    if len(argv) >= 2:
 
      # Loop through available command-line arguments.
      for element in argv:
 
        # Check first of two parameter values and substitute
        # default value if input value is an invalid option.
        if len(inputs) == 0 and (element in ('Three','Five')) or \
           len(inputs) == 1 and (isinstance(element,str)):
          inputs.append(element)
        elif len(inputs) == 0:
          inputs.append(defaults[0])
        elif len(inputs) == 1:
          inputs.append(defaults[1])
 
      # Assign arguments to parameters.
      args = (inputs)
 
    # Check whether only one parameter value exists.
    elif len(argv) == 1 and (argv[0] in ('Three','Five')):
      args = (argv[0],"_")
 
    # Assume only one parameter is valid and substitute an 
    # empty string as the second parameter.
    else:
      args = (defaults[0],"_")
 
    # Substitute defaults when missing parameters.
  else:
    args = defaults
 
  # Return parameters as a tuple.
  return args
 
# ============================================================
 
# Assign command-line argument list to variable by removing
# the program file name.
# ============================================================
params = check_replace(sys.argv[1:])
# ============================================================
 
#  Attempt the query.
# ============================================================
#  Use a try-catch block to manage the connection.
# ============================================================
try:
  # Open connection.
  cnx = mysql.connector.connect(user='student', password='student',
                                host='127.0.0.1',
                                database='studentdb')
  # Create cursor.
  cursor = cnx.cursor()
 
  # Set the query statement.
  query = ("WITH actors AS "
           "(SELECT   a.first_name "
           " ,        a.last_name "
           " ,        COUNT(*) AS num_roles "
           " FROM     sakila.actor a INNER JOIN sakila.film_actor fa "
           " ON       a.actor_id = fa.actor_id "
           " GROUP BY a.first_name "
           " ,        a.last_name ) "
           " SELECT   CONCAT(a.last_name,', ',a.first_name) AS full_name "
           " ,        l.description "
           " ,        a.num_roles "
           " FROM     actors a CROSS JOIN levels l "
           " WHERE    a.num_roles BETWEEN l.min_roles AND l.max_roles "
           " AND      l.parameter_set = %s "
           " AND      a.last_name LIKE CONCAT(%s,'%') "
           " ORDER BY a.last_name "
           " ,        a.first_name")
 
  # Execute cursor.
  cursor.execute(query, params)
 
  # Display the rows returned by the query.
  for (full_name, description, num_roles) in cursor:
    print('{0} is a {1} with {2} films.'.format( full_name.title()
                                               , description.title()
                                               , num_roles))
 
  # Close cursor.
  cursor.close()
 
# ------------------------------------------------------------
# Handle exception and close connection.
except mysql.connector.Error as e:
  if e.errno == errorcode.ER_ACCESS_DENIED_ERROR:
    print("Something is wrong with your user name or password")
  elif e.errno == errorcode.ER_BAD_DB_ERROR:
    print("Database does not exist")
  else:
    print("Error code:", e.errno)        # error number
    print("SQLSTATE value:", e.sqlstate) # SQLSTATE value
    print("Error message:", e.msg)       # error message
 
# Close the connection when the try block completes.
else:
  cnx.close()

As always, I hope this helps those trying to understand how CTEs can solve problems that would otherwise be coded in external imperative languages like Python.

Written by maclochlainn

March 1st, 2024 at 12:30 am

Oracle 23c Free Ext Files

without comments

This is an example of how you would upload data from a flat file, or Comma Separated Value (CSV) file inside Docker Oracle Database 23c Free. It’s important to note that in the file upload you are transferring information that doesn’t have surrogate key values by leveraing joins inside a MERGE statement.

Step #1 : Create a virtual directory

You can create a virtual directory without a physical directory but it won’t work when you try to access it. Therefore, you should create the physical directory first. Assuming you’ve created the Docker Oracle Database 23c Free instance, you should put the code in subdirectories of the /opt/oracle file directory.

  • Connect as the root user with the following Docker command:

    docker exec -it --user root oracle23c bash

    Issue the following commands as the oracle user inside the Docker container to create the necessary physical directories. You may need to refer to my earlier blog post if you haven’t setup the oracle user inside the Docker instance. While this blog post will only use the /opt/oracle/upload/text and /opt/oracle/upload/log directories, a subsequent post will demonstrate the preprocessing module for the external tables.

    mkdir /opt/oracle/upload
    mkdir /opt/oracle/upload/text
    mkdir /opt/oracle/upload/log
    mkdir /opt/oracle/upload/preproc
  • Connect to the Oracle Database 23c Free inside the container as the system user to create a c##studentrole, and do the following three things:

    • Grant privileges to the c##studentrole, and grant the c##studentrole to the c##student user.

      -- Create the role.
      CREATE ROLE c##studentrole;
       
      -- Grant privileges to the role.
      GRANT CREATE CLUSTER, CREATE INDEXTYPE, CREATE PROCEDURE,
      CREATE SEQUENCE, CREATE SESSION, CREATE TABLE, CREATE TRIGGER,
      CREATE TYPE, CREATE VIEW TO c##studentrole;
       
      -- Grant privileges to the user.
      GRANT c##studentrole TO c##student;
    • As the system user, create the necessary virtual directories that map to the physical directories inside the Docker container:

      CREATE DIRECTORY upload AS '/opt/oracle/upload/text';
      CREATE DIRECTORY preproc AS '/opt/oracle/upload/preproc';
      CREATE DIRECTORY LOG AS '/opt/oracle/upload/log';
    • As the system user, grant the necessary privileges on the virtual directories to the c##studentrole role:

      GRANT read ON DIRECTORY upload TO c##studentrole;
      GRANT read, WRITE ON DIRECTORY LOG TO c##studentrole;
      GRANT read, EXECUTE ON DIRECTORY preproc TO c##studentrole;

Step #2 : Position your CSV file in the physical directory

After creating the virtual directory, copy the following contents into a file named kingdom_import.csv in the /opt/oracle/upload/texgt directory or folder. If you attempt to do this in Windows, you need to disable Windows UAC before performing this step.

Place the following in the kingdom_import.csv file. The trailing commas aren’t too meaningful in Oracle but they’re very helpful if you use the file in MySQL. A key element in creating this files requires that you avoid trailing line returns at the bottom of the file because they’re inserted as null values. There should be no lines after the last row of data.

'Narnia',77600,'Peter the Magnificent','20-MAR-1272','19-JUN-1292','The Lion, The Witch and The Wardrobe',
'Narnia',77600,'Edmund the Just','20-MAR-1272','19-JUN-1292','The Lion, The Witch and The Wardrobe',
'Narnia',77600,'Susan the Gentle','20-MAR-1272','19-JUN-1292','The Lion, The Witch and The Wardrobe',
'Narnia',77600,'Lucy the Valiant','20-MAR-1272','19-JUN-1292','The Lion, The Witch and The Wardrobe',
'Narnia',42100,'Peter the Magnificent','12-APR-1531','31-MAY-1328','Prince Caspian',
'Narnia',42100,'Edmund the Just','12-APR-1531','31-MAY-1328','Prince Caspian',
'Narnia',42100,'Susan the Gentle','12-APR-1531','31-MAY-1328','Prince Caspian',
'Narnia',42100,'Lucy the Valiant','12-APR-1531','31-MAY-1328','Prince Caspian',
'Camelot',15200,'King Arthur','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Lionel','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Bors','10-MAR-0631','12-DEC-0635','The Once and Future King',
'Camelot',15200,'Sir Bors','10-MAR-0640','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Galahad','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Gawain','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Tristram','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Percival','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Lancelot','30-SEP-0670','12-DEC-0686','The Once and Future King',

Step #3 : Reconnect as the student user

Disconnect and connect as the c##student user, or reconnect as the c##student user. The reconnect syntax that protects your password is:

CONNECT c##student@free

Step #4 : Run the script that creates tables and sequences

Copy the following into a create_kingdom_upload.sql file within a directory of your choice. I use varchar as the data type because it’s an alias for varchar2 and highlights appropriately with the GeSHi formatting. Then, run it as the student account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
-- Conditionally drop tables.
DROP TABLE IF EXISTS kingdom;
DROP TABLE IF EXISTS knight;
DROP TABLE IF EXISTS kingdom_knight_import;
 
-- Conditionally drop sequences.
DROP SEQUENCE IF EXISTS kingdom_s1;
DROP SEQUENCE IF EXISTS knight_s1;
 
-- Create normalized kingdom table.
CREATE TABLE kingdom
( kingdom_id    NUMBER
, kingdom_name  VARCHAR(20)
, population    NUMBER
, book          VARCHAR(40));
 
-- Create a sequence for the kingdom table.
CREATE SEQUENCE kingdom_s1;
 
-- Create normalized knight table.
CREATE TABLE knight
( knight_id             NUMBER
, knight_name           VARCHAR(22)
, kingdom_allegiance_id NUMBER
, allegiance_start_date DATE
, allegiance_end_date   DATE
, book                  VARCHAR(40));
 
-- Create a sequence for the knight table.
CREATE SEQUENCE knight_s1;
 
-- Create external import table.
CREATE TABLE kingdom_knight_import
( kingdom_name           VARCHAR(20)
, population             NUMBER
, knight_name            VARCHAR(22)
, allegiance_start_date  DATE
, allegiance_end_date    DATE
, book                   VARCHAR(40))
  ORGANIZATION EXTERNAL
  ( TYPE oracle_loader
    DEFAULT DIRECTORY upload
    ACCESS PARAMETERS
    ( RECORDS DELIMITED BY NEWLINE CHARACTERSET US7ASCII
      BADFILE     'LOG':'kingdom_import.bad'
      DISCARDFILE 'LOG':'kingdom_import.dis'
      LOGFILE     'LOG':'kingdom_import.log'
      FIELDS TERMINATED BY ','
      OPTIONALLY ENCLOSED BY "'"
      MISSING FIELD VALUES ARE NULL )
    LOCATION ('kingdom_import.csv'))
REJECT LIMIT UNLIMITED;

Step #5 : Test your access to the external table

There a number of things that could go wrong with setting up an external table, such as file permissions. Before moving on to the balance of the steps, you should test what you’ve done. Run the following query from the student account to check whether or not you can access the kingdom_import.csv file.

1
2
3
4
5
6
7
8
9
10
11
12
SET PAGESIZE 999
COL kingdom_name  FORMAT A7     HEADING "Kingdom|Name"
COL folks         FORMAT 99999  HEADING "Folks"
COL knight_name   FORMAT A21    HEADING "Knight Name"
COL dates         FORMAT A11    HEADING "Start Date"
COL source_book   FORMAT A38    HEADING "Book"
SELECT   kingdom_name
,        knight_name
,        TO_CHAR(allegiance_start_date,'DD-MON-YYYY')
||       TO_CHAR(allegiance_end_date,'DD-MON-YYYY') AS dates
,        book
FROM     kingdom_knight_import;

Step #6 : Create the upload procedure

Copy the following into a create_upload_procedure.sql file within a virtual directory of your choice. As noted above in the external table definition writes only occur in the log virtual directory. This is important because there are articles out there on the Internet that could misdirect you when you get the following error message on the upload virtual directory.

ORA-06564: Object UPLOAD does not exist or is not accessible to the user.

By the way, you’ll only see that error if you fail to:

  • Designate the procedure as AUTH_ID CURRENT, and
  • Enabled SERVEROUTPUT inside the SQL*Plus command-line interface (CLI) session or inside the glogin.sql file for the Oracle Database 23c Free Docker instance.

Then, run it as the student account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
-- Create a procedure to wrap the transaction.
CREATE OR REPLACE
  PROCEDURE upload_kingdom AUTHID CURRENT_USER IS 
BEGIN
  -- Set save point for an all or nothing transaction.
  SAVEPOINT starting_point;
 
  -- Insert or update the table, which makes this rerunnable when the file hasn't been updated.  
  MERGE INTO kingdom target
  USING (SELECT   DISTINCT
                  k.kingdom_id
         ,        kki.kingdom_name
         ,        kki.population
         ,        kki.book
         FROM     kingdom_knight_import kki LEFT JOIN kingdom k
         ON       kki.kingdom_name = k.kingdom_name
         AND      kki.population = k.population
         AND      kki.book = k.book) SOURCE
  ON (target.kingdom_id = SOURCE.kingdom_id)
  WHEN MATCHED THEN
  UPDATE SET kingdom_name = SOURCE.kingdom_name
  WHEN NOT MATCHED THEN
  INSERT VALUES
  ( kingdom_s1.nextval
  , SOURCE.kingdom_name
  , SOURCE.population
  , SOURCE.book);
 
  -- Insert or update the table, which makes this rerunnable when the file hasn't been updated.  
  MERGE INTO knight target
  USING (SELECT   kn.knight_id
         ,        kki.knight_name
         ,        k.kingdom_id
         ,        kki.allegiance_start_date AS start_date
         ,        kki.allegiance_end_date AS end_date
         ,        kki.book
         FROM     kingdom_knight_import kki INNER JOIN kingdom k
         ON       kki.kingdom_name = k.kingdom_name
         AND      kki.population = k.population LEFT JOIN knight kn 
         ON       k.kingdom_id = kn.kingdom_allegiance_id
         AND      kki.knight_name = kn.knight_name
         AND      kki.allegiance_start_date = kn.allegiance_start_date
         AND      kki.allegiance_end_date = kn.allegiance_end_date
         AND      kki.book = kn.book) SOURCE
  ON (target.kingdom_allegiance_id = SOURCE.kingdom_id)
  WHEN MATCHED THEN
  UPDATE SET allegiance_start_date = SOURCE.start_date
  ,          allegiance_end_date = SOURCE.end_date
  ,          book = SOURCE.book
  WHEN NOT MATCHED THEN
  INSERT VALUES
  ( knight_s1.nextval
  , SOURCE.knight_name
  , SOURCE.kingdom_id
  , SOURCE.start_date
  , SOURCE.end_date
  , SOURCE.book);
 
  -- Save the changes.
  COMMIT;
 
EXCEPTION
  WHEN OTHERS THEN
    dbms_output.put_line(SQLERRM);
    ROLLBACK TO starting_point;
    RETURN;
END;
/

Step #7 : Run the upload procedure

You can run the file by calling the script above. The procedure ensures that records are inserted or updated into their respective tables.

EXECUTE upload_kingdom;

Step #8 : Test the results of the upload procedure

You can test whether or not it worked by running the following queries.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
-- Format Oracle output.
COLUMN kingdom_id    FORMAT 999      HEADING "Kingdom|ID #"
COLUMN kingdom_name  FORMAT A14      HEADING "Kingdom|Name"
COLUMN population    FORMAT 999,999  HEADING "Population"
COLUMN book          FORMAT A40      HEADING "Source Book"
 
-- Check the kingdom table.
SELECT * FROM kingdom;
 
-- Format Oracle output.
SET PAGESIZE 999
COLUMN knight_id              FORMAT 999  HEADING "Knight|ID #"
COLUMN knight_name            FORMAT A23  HEADING "Knight|Name"
COLUMN kingdom_allegiance_id  FORMAT 999  HEADING "Kingdom|ID #"
COLUMN allegiance_start_date  FORMAT A11 HEADING "Allegiance|Start Date"
COLUMN allegiance_end_date    FORMAT A11 HEADING "Allegiance|End Date"
 
-- Check the knight table.
SELECT   knight_id
,        knight_name
,        kingdom_allegiance_id
,        TO_CHAR(allegiance_start_date,'DD-MON-YYYY') AS allegiance_start_date
,        TO_CHAR(allegiance_end_date,'DD-MON-YYYY') AS allegiance_end_date
FROM     knight;

It should display the following information:

Kingdom Kingdom
   ID # Name	       Population Source Book
------- -------------- ---------- ----------------------------------------
      1 Narnia		   42,100 Prince Caspian
      2 Narnia		   77,600 The Lion, The Witch and The Wardrobe
      3 Camelot 	   15,200 The Once and Future King
 
 
Knight Knight		       Kingdom Allegiance  Allegiance
  ID # Name			  ID # Start Date  End Date
------ ----------------------- ------- ----------- -----------
     1 Peter the Magnificent	     2 20-MAR-1272 19-JUN-1292
     2 Edmund the Just		     2 20-MAR-1272 19-JUN-1292
     3 Susan the Gentle 	     2 20-MAR-1272 19-JUN-1292
     4 Lucy the Valiant 	     2 20-MAR-1272 19-JUN-1292
     5 Peter the Magnificent	     1 12-APR-1531 31-MAY-1328
     6 Edmund the Just		     1 12-APR-1531 31-MAY-1328
     7 Susan the Gentle 	     1 12-APR-1531 31-MAY-1328
     8 Lucy the Valiant 	     1 12-APR-1531 31-MAY-1328
     9 King Arthur		     3 10-MAR-0631 12-DEC-0686
    10 Sir Lionel		     3 10-MAR-0631 12-DEC-0686
    11 Sir Bors 		     3 10-MAR-0631 12-DEC-0635
    12 Sir Bors 		     3 10-MAR-0640 12-DEC-0686
    13 Sir Galahad		     3 10-MAR-0631 12-DEC-0686
    14 Sir Gawain		     3 10-MAR-0631 12-DEC-0686
    15 Sir Tristram		     3 10-MAR-0631 12-DEC-0686
    16 Sir Percival		     3 10-MAR-0631 12-DEC-0686
    17 Sir Lancelot		     3 30-SEP-0670 12-DEC-0686

You can rerun the procedure to check that it doesn’t alter any information, then you could add a new knight to test the insertion portion.

Written by maclochlainn

January 6th, 2024 at 11:53 pm

Oracle 23c Free SQL*Plus

without comments

It’s always frustrated me when using the sqlplus command-line interface (CLI) that you can’t just “up arrow” to through the history. At least, that’s the default case unless you wrap the sqlplus executable.

I like to do my development work as close to the database as possible. The delay from SQL Developer to the database or VSCode to the database is just too long. Therefore, I like the native sqlplus to be as efficient as possible. This post shows you how to install the rlwarp utility to wrap sqlplus and create a sandboxed student user for a local development account inside the Oracle 23c Free container. You should note that the Docker or Podman Container is using Oracle Unbreakable Linux 8 as it’s native OS.

You can connect to your Docker version of Oracle Database 23c Free with the following command:

docker exec -it -u root oracle23c bash

You can’t just use dnf to install rlwrap and get it to magically install all the dependencies. That would be too easy, eh?

Attempting to do so will lock your base OS and eventually force you to kill with prejudice the hung dnf process (at least it forced me to do so). You need to determine the rlwrap dependencies and then install them first. In that process, I noticed that the which utility program wasn’t installed in the container.

Naturally, I installed the which utility first with this command:

dnf install -y which

The rlwrap dependencies are: glibc, ncurses, perl, readline, python, and git. Only the perl, python, and git are missing from the list of formal dependencies but there’s another dependency the epel-release package.

If you want to verify whether a package is installed, you can use the rpm command like this:

rpm -qa | grep package_name

I installed the perl programming environment (a big install) with this command:

dnf install -y perl

I installed the python3 with this command:

dnf install -y python3

I installed the git module with this command:

dnf install -y git

I installed the epel-release container with this command:

dnf install -y epel-release

After installing all of these, you’re now ready to install the core rlwrap utility program. Like the other installations, you use:

dnf install -y rlwrap

At this point, you need to create a sandboxed user account for the Docker instance because as a developer using the root user for simple tasks is a bad idea. While you could do this with a Docker command, the Oracle 23c Free edition raised a lock on the /etc/group file when I tried it. Naturally, that’s not a problem because you can connect as the root user with this syntax:

docker exec -it -u root oracle23c bash

As the root user, create a student account as a developer account in the Oracle 23c Free container:

useradd -u 501 -g dba -G users -d /home/student -s /bin/bash/ -c "Student" -n student

You’ll be unable to leverage the tnsnames.ora file unless you alter the prior command to replace dba with oinstall or add the following command:

usermod -a -G oinstall student

Exit the Oracle 23c Free container as the root user and reconnect as the student user with this syntax:

docker exec -it --user student oracle23c bash

While you’re connected as the root user, you should create an upload directory as a subdirectory of the $ORACLE_BASE directory. The $ORACLE_BASE directory in the Oracle Database 23c Free Docker image is the /opt/oracle directory.

You should use the following syntax to create the upload directory and change its permission to that of the Oracle Database 23c Free installation (for a future blog post on developing external table deployment on the Docker image):

mkdir /opt/oracle
chown -R oracle:install /opt/oracle/upload

You also can add the following student function to the Ubuntu student user’s .bashrc file. It means all you need to type to connect to the Oracle Database 23c Free Docker instance is “student“. I like shortcuts like this one, which let you leverage one-line Python commands.

student () 
{
    # Discover the fully qualified program name. 
    path=`which docker 2>/dev/null`
    file=''
 
    # Parse the program name from the path.
    if [ -n ${path} ]; then
        file=${path##/*/}
    fi
 
    # Wrap when there is a file and it is rewrap.
    if [ -n ${file} ] && [[ ${file} = "docker" ]]; then
        python -c "import subprocess; subprocess.run(['docker exec -it --user student oracle23c bash'], shell=True)" 
    else
        echo "Docker is unavailable: Install the docker package."
    fi
}

Open a Ubuntu Terminal shell and type a student function name to connect to the Docker Oracle Database 23c Free instance where you can now test things like external tables with the SQL*Plus command line without installing it on the Ubuntu local operating system.

student@student-virtual-machine:~$ student
[student@d28375f0c43f ~]$ sqlplus c##student/student@free
 
SQL*Plus: Release 23.0.0.0.0 - Production on Wed Jan 3 02:14:22 2024
Version 23.3.0.23.09
 
Copyright (c) 1982, 2023, Oracle.  All rights reserved.
 
Last Successful login time: Wed Jan 03 2024 01:56:44 +00:00
 
Connected to:
Oracle Database 23c Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.3.0.23.09
 
SQL>

Then, I added this sqlplus function to the /home/student/.bashrc file, which is owned by the student user. However, I also added the instruction to change to the student user’s home directory because the Oracle 23c Free container will connect you to the /home/oracle directory by default. I also added the default long list (ll) alias to the .bashrc file.

sqlplus () 
{
    # Discover the fully qualified program name. 
    path=`which rlwrap 2>/dev/null`
    file=''
 
    # Parse the program name from the path.
    if [ -n ${path} ]; then
        file=${path##/*/}
    fi;
 
    # Wrap when there is a file and it is rewrap.
    if [ -n ${file} ] && [[ ${file} = "rlwrap" ]]; then
        rlwrap sqlplus "${@}"
    else
        echo "Command-line history unavailable: Install the rlwrap package."
        $ORACLE_HOME/bin/sqlplus "${@}"
    fi
}
 
# Change to the user's home directory.
cd ${HOME}
 
# Create a long list alias:
alias ll='ls -l --color=auto'

After you’ve configured your student user, you can configure the oracle user account to work like a regular server. Exit the Docker Oracle Database 23c Free as the student user, then connect as the root user with this command:

docker exec -it -u root oracle23c bash

As the root user you can become the oracle user with the following command:

su - oracle

Now, add the following .bashrc shell in the /home/oracle directory:

# The oracle user's .bashrc
 
# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi
 
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
    PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
 
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
 
# User specific aliases and functions
export ORACLE_SID=FREE
export ORACLE_BASE=/opt/oracle
export ORACLE_HOME=/opt/oracle/product/23c/dbhomeFree
export PATH=$PATH:/$ORACLE_HOME/bin
 
# Change to the user's home directory.
cd ${HOME}
 
# Create a long list alias:
alias ll='ls -l --color=auto'
 
sqlplus () 
{
    # Discover the fully qualified program name. 
    path=`which rlwrap 2>/dev/null`
    file=''
 
    # Parse the program name from the path.
    if [ -n ${path} ]; then
        file=${path##/*/}
    fi;
 
    # Wrap when there is a file and it is rewrap.
    if [ -n ${file} ] && [[ ${file} = "rlwrap" ]]; then
        rlwrap sqlplus "${@}"
    else
        echo "Command-line history unavailable: Install the rlwrap package."
        $ORACLE_HOME/bin/sqlplus "${@}"
    fi
}

You need to manually source the .bashrc for the oracle user because it’s not an externally available user. Use this syntax to connect as the internal user:

sqlplus / as sysdba

It’ll display:

SQL*Plus: RELEASE 23.0.0.0.0 - Production ON Wed Jan 3 07:08:11 2024
Version 23.3.0.23.09
 
Copyright (c) 1982, 2023, Oracle.  ALL rights reserved.
 
 
Connected TO:
Oracle DATABASE 23c Free RELEASE 23.0.0.0.0 - Develop, Learn, AND Run FOR Free
Version 23.3.0.23.09
 
SQL>

After all this, I can now click the “up arrow” to edit any of the sqlplus command history. If you like to work inside sqlplus natively, this should help you.

Written by maclochlainn

December 20th, 2023 at 11:11 pm

SQL 1 v.s. SQL 3

without comments

In Alan Beaulieu’s wonderful book Learning SQL: Generate, Manipulate, and Retrieve Data, he uses a SQL 1 to demonstrates a CROSS JOIN in MySQL counting from 1 to 100. My students always find it difficult to read because the subqueries take so much space it makes it difficult to see the process, for example he gives this SQL 1 solution:

SELECT ones.x + tens.x + 1 AS counter
FROM
 (SELECT 0 AS x UNION ALL
  SELECT 1 AS x UNION ALL
  SELECT 2 AS x UNION ALL
  SELECT 3 AS x UNION ALL
  SELECT 4 AS x UNION ALL
  SELECT 5 AS x UNION ALL
  SELECT 6 AS x UNION ALL
  SELECT 7 AS x UNION ALL
  SELECT 8 AS x UNION ALL
  SELECT 9 AS x ) ones CROSS JOIN
 (SELECT 0 AS x UNION ALL
  SELECT 10 AS x UNION ALL
  SELECT 20 AS x UNION ALL
  SELECT 30 AS x UNION ALL
  SELECT 40 AS x UNION ALL
  SELECT 50 AS x UNION ALL
  SELECT 60 AS x UNION ALL
  SELECT 70 AS x UNION ALL
  SELECT 80 AS x UNION ALL
  SELECT 90 AS x ) tens
ORDER BY counter;

While anybody with a command of SQL should be able to see how it works, for those new to SQL it’s difficult. It’s more effective to use a Common Table Expression with the WITH clause because the derived tables become variables in the scope of the WITH clause and the final query works more like a CROSS JOIN between two tables:

WITH ones AS
 (SELECT 0 AS x UNION ALL
  SELECT 1 AS x UNION ALL
  SELECT 2 AS x UNION ALL
  SELECT 3 AS x UNION ALL
  SELECT 4 AS x UNION ALL
  SELECT 5 AS x UNION ALL
  SELECT 6 AS x UNION ALL
  SELECT 7 AS x UNION ALL
  SELECT 8 AS x UNION ALL
  SELECT 9 AS x )
, tens AS
 (SELECT 0 AS x UNION ALL
  SELECT 10 AS x UNION ALL
  SELECT 20 AS x UNION ALL
  SELECT 30 AS x UNION ALL
  SELECT 40 AS x UNION ALL
  SELECT 50 AS x UNION ALL
  SELECT 60 AS x UNION ALL
  SELECT 70 AS x UNION ALL
  SELECT 80 AS x UNION ALL
  SELECT 90 AS x )
SELECT ones.x + tens.x + 1 AS counter
FROM ones CROSS JOIN tens
ORDER BY counter;

As always, I hope this helps somebody trying to sort out the syntax and workflow.

Written by maclochlainn

November 7th, 2023 at 10:10 pm

Posted in MySQL,MySQL 8,sql

Tagged with

AWS EC2 TNS Listener

without comments

Having configured an AlmaLinux 8.6 with Oracle Database 11g XE, MySQL 8.0.30, and PostgreSQL 15, we migrated it to AWS EC2 and provisioned it. We used the older and de-supported Oracle Database 11g XE because it didn’t require any kernel modifications and had a much smaller footprint.

I had to address why attempting to connect with the sqlplus utility raised the following error after provisioning a copy with a new static IP address:

ERROR:
ORA-12514: TNS:listener does NOT currently know OF service requested IN CONNECT descriptor

A connection from SQL Developer raises a more addressable error, like:

ORA-17069

I immediately tried to check the connection with the tnsping utility and found that tnsping worked fine. However, when I tried to connect with the sqlplus utility it raised an ORA-12514 connection error.

There were no diagnostic steps beyond checking the tnsping utility. So, I had to experiment with what might block communication.

I changed the host name from ip-172-58-65-82.us-west-2.compute.internal to a localhost string in both the listener.ora and tnsnames.ora. The listener.ora file:

# listener.ora Network Configuration FILE:
 
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/xe)
      (PROGRAM = extproc)
    )
  )
 
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC_FOR_XE))
      (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
    )
  )
 
DEFAULT_SERVICE_LISTENER = (XE)

The tnsnames.ora file:

# tnsnames.ora Network Configuration FILE:
 
XE =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = XE)
    )
  )
 
EXTPROC_CONNECTION_DATA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC_FOR_XE))
    )
    (CONNECT_DATA =
      (SID = PLSExtProc)
      (PRESENTATION = RO)
    )
  )

I suspected that it might be related to the localhost value. So, I checked the /etc/hostname and /etc/hosts files.

Then, I modified /etc/hostname file by removing the AWS EC2 damain address. I did it on a memory that Oracle’s TNS raises errors for dots or periods in some addresses.

The /etc/hostname file:

ip-172-58-65-82

The /etc/hosts file:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 ip-172-58-65-82
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 ip-172-58-65-82

Now, we can connect to the Oracle Database 11g XE instance with the sqlplus utility. I believe this type of solution will work for other AWS EC2 provisioned Oracle databases.

Written by maclochlainn

March 22nd, 2023 at 10:09 pm