MacLochlainns Weblog

Michael McLaughlin's Technical Blog

Site Admin

Archive for the ‘Oracle’ Category

Oracle 23c Free Ext Files

without comments

This is an example of how you would upload data from a flat file, or Comma Separated Value (CSV) file inside Docker Oracle Database 23c Free. It’s important to note that in the file upload you are transferring information that doesn’t have surrogate key values by leveraing joins inside a MERGE statement.

Step #1 : Create a virtual directory

You can create a virtual directory without a physical directory but it won’t work when you try to access it. Therefore, you should create the physical directory first. Assuming you’ve created the Docker Oracle Database 23c Free instance, you should put the code in subdirectories of the /opt/oracle file directory.

  • Connect as the root user with the following Docker command:

    docker exec -it --user root oracle23c bash

    Issue the following commands as the oracle user inside the Docker container to create the necessary physical directories. You may need to refer to my earlier blog post if you haven’t setup the oracle user inside the Docker instance. While this blog post will only use the /opt/oracle/upload/text and /opt/oracle/upload/log directories, a subsequent post will demonstrate the preprocessing module for the external tables.

    mkdir /opt/oracle/upload
    mkdir /opt/oracle/upload/text
    mkdir /opt/oracle/upload/log
    mkdir /opt/oracle/upload/preproc
  • Connect to the Oracle Database 23c Free inside the container as the system user to create a c##studentrole, and do the following three things:

    • Grant privileges to the c##studentrole, and grant the c##studentrole to the c##student user.

      -- Create the role.
      CREATE ROLE c##studentrole;
       
      -- Grant privileges to the role.
      GRANT CREATE CLUSTER, CREATE INDEXTYPE, CREATE PROCEDURE,
      CREATE SEQUENCE, CREATE SESSION, CREATE TABLE, CREATE TRIGGER,
      CREATE TYPE, CREATE VIEW TO c##studentrole;
       
      -- Grant privileges to the user.
      GRANT c##studentrole TO c##student;
    • As the system user, create the necessary virtual directories that map to the physical directories inside the Docker container:

      CREATE DIRECTORY upload AS '/opt/oracle/upload/text';
      CREATE DIRECTORY preproc AS '/opt/oracle/upload/preproc';
      CREATE DIRECTORY LOG AS '/opt/oracle/upload/log';
    • As the system user, grant the necessary privileges on the virtual directories to the c##studentrole role:

      GRANT read ON DIRECTORY upload TO c##studentrole;
      GRANT read, WRITE ON DIRECTORY LOG TO c##studentrole;
      GRANT read, EXECUTE ON DIRECTORY preproc TO c##studentrole;

Step #2 : Position your CSV file in the physical directory

After creating the virtual directory, copy the following contents into a file named kingdom_import.csv in the /opt/oracle/upload/texgt directory or folder. If you attempt to do this in Windows, you need to disable Windows UAC before performing this step.

Place the following in the kingdom_import.csv file. The trailing commas aren’t too meaningful in Oracle but they’re very helpful if you use the file in MySQL. A key element in creating this files requires that you avoid trailing line returns at the bottom of the file because they’re inserted as null values. There should be no lines after the last row of data.

'Narnia',77600,'Peter the Magnificent','20-MAR-1272','19-JUN-1292','The Lion, The Witch and The Wardrobe',
'Narnia',77600,'Edmund the Just','20-MAR-1272','19-JUN-1292','The Lion, The Witch and The Wardrobe',
'Narnia',77600,'Susan the Gentle','20-MAR-1272','19-JUN-1292','The Lion, The Witch and The Wardrobe',
'Narnia',77600,'Lucy the Valiant','20-MAR-1272','19-JUN-1292','The Lion, The Witch and The Wardrobe',
'Narnia',42100,'Peter the Magnificent','12-APR-1531','31-MAY-1328','Prince Caspian',
'Narnia',42100,'Edmund the Just','12-APR-1531','31-MAY-1328','Prince Caspian',
'Narnia',42100,'Susan the Gentle','12-APR-1531','31-MAY-1328','Prince Caspian',
'Narnia',42100,'Lucy the Valiant','12-APR-1531','31-MAY-1328','Prince Caspian',
'Camelot',15200,'King Arthur','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Lionel','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Bors','10-MAR-0631','12-DEC-0635','The Once and Future King',
'Camelot',15200,'Sir Bors','10-MAR-0640','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Galahad','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Gawain','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Tristram','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Percival','10-MAR-0631','12-DEC-0686','The Once and Future King',
'Camelot',15200,'Sir Lancelot','30-SEP-0670','12-DEC-0686','The Once and Future King',

Step #3 : Reconnect as the student user

Disconnect and connect as the c##student user, or reconnect as the c##student user. The reconnect syntax that protects your password is:

CONNECT c##student@free

Step #4 : Run the script that creates tables and sequences

Copy the following into a create_kingdom_upload.sql file within a directory of your choice. I use varchar as the data type because it’s an alias for varchar2 and highlights appropriately with the GeSHi formatting. Then, run it as the student account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
-- Conditionally drop tables.
DROP TABLE IF EXISTS kingdom;
DROP TABLE IF EXISTS knight;
DROP TABLE IF EXISTS kingdom_knight_import;
 
-- Conditionally drop sequences.
DROP SEQUENCE IF EXISTS kingdom_s1;
DROP SEQUENCE IF EXISTS knight_s1;
 
-- Create normalized kingdom table.
CREATE TABLE kingdom
( kingdom_id    NUMBER
, kingdom_name  VARCHAR(20)
, population    NUMBER
, book          VARCHAR(40));
 
-- Create a sequence for the kingdom table.
CREATE SEQUENCE kingdom_s1;
 
-- Create normalized knight table.
CREATE TABLE knight
( knight_id             NUMBER
, knight_name           VARCHAR(22)
, kingdom_allegiance_id NUMBER
, allegiance_start_date DATE
, allegiance_end_date   DATE
, book                  VARCHAR(40));
 
-- Create a sequence for the knight table.
CREATE SEQUENCE knight_s1;
 
-- Create external import table.
CREATE TABLE kingdom_knight_import
( kingdom_name           VARCHAR(20)
, population             NUMBER
, knight_name            VARCHAR(22)
, allegiance_start_date  DATE
, allegiance_end_date    DATE
, book                   VARCHAR(40))
  ORGANIZATION EXTERNAL
  ( TYPE oracle_loader
    DEFAULT DIRECTORY upload
    ACCESS PARAMETERS
    ( RECORDS DELIMITED BY NEWLINE CHARACTERSET US7ASCII
      BADFILE     'LOG':'kingdom_import.bad'
      DISCARDFILE 'LOG':'kingdom_import.dis'
      LOGFILE     'LOG':'kingdom_import.log'
      FIELDS TERMINATED BY ','
      OPTIONALLY ENCLOSED BY "'"
      MISSING FIELD VALUES ARE NULL )
    LOCATION ('kingdom_import.csv'))
REJECT LIMIT UNLIMITED;

Step #5 : Test your access to the external table

There a number of things that could go wrong with setting up an external table, such as file permissions. Before moving on to the balance of the steps, you should test what you’ve done. Run the following query from the student account to check whether or not you can access the kingdom_import.csv file.

1
2
3
4
5
6
7
8
9
10
11
12
SET PAGESIZE 999
COL kingdom_name  FORMAT A7     HEADING "Kingdom|Name"
COL folks         FORMAT 99999  HEADING "Folks"
COL knight_name   FORMAT A21    HEADING "Knight Name"
COL dates         FORMAT A11    HEADING "Start Date"
COL source_book   FORMAT A38    HEADING "Book"
SELECT   kingdom_name
,        knight_name
,        TO_CHAR(allegiance_start_date,'DD-MON-YYYY')
||       TO_CHAR(allegiance_end_date,'DD-MON-YYYY') AS dates
,        book
FROM     kingdom_knight_import;

Step #6 : Create the upload procedure

Copy the following into a create_upload_procedure.sql file within a virtual directory of your choice. As noted above in the external table definition writes only occur in the log virtual directory. This is important because there are articles out there on the Internet that could misdirect you when you get the following error message on the upload virtual directory.

ORA-06564: Object UPLOAD does not exist or is not accessible to the user.

By the way, you’ll only see that error if you fail to:

  • Designate the procedure as AUTH_ID CURRENT, and
  • Enabled SERVEROUTPUT inside the SQL*Plus command-line interface (CLI) session or inside the glogin.sql file for the Oracle Database 23c Free Docker instance.

Then, run it as the student account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
-- Create a procedure to wrap the transaction.
CREATE OR REPLACE
  PROCEDURE upload_kingdom AUTHID CURRENT_USER IS 
BEGIN
  -- Set save point for an all or nothing transaction.
  SAVEPOINT starting_point;
 
  -- Insert or update the table, which makes this rerunnable when the file hasn't been updated.  
  MERGE INTO kingdom target
  USING (SELECT   DISTINCT
                  k.kingdom_id
         ,        kki.kingdom_name
         ,        kki.population
         ,        kki.book
         FROM     kingdom_knight_import kki LEFT JOIN kingdom k
         ON       kki.kingdom_name = k.kingdom_name
         AND      kki.population = k.population
         AND      kki.book = k.book) SOURCE
  ON (target.kingdom_id = SOURCE.kingdom_id)
  WHEN MATCHED THEN
  UPDATE SET kingdom_name = SOURCE.kingdom_name
  WHEN NOT MATCHED THEN
  INSERT VALUES
  ( kingdom_s1.nextval
  , SOURCE.kingdom_name
  , SOURCE.population
  , SOURCE.book);
 
  -- Insert or update the table, which makes this rerunnable when the file hasn't been updated.  
  MERGE INTO knight target
  USING (SELECT   kn.knight_id
         ,        kki.knight_name
         ,        k.kingdom_id
         ,        kki.allegiance_start_date AS start_date
         ,        kki.allegiance_end_date AS end_date
         ,        kki.book
         FROM     kingdom_knight_import kki INNER JOIN kingdom k
         ON       kki.kingdom_name = k.kingdom_name
         AND      kki.population = k.population LEFT JOIN knight kn 
         ON       k.kingdom_id = kn.kingdom_allegiance_id
         AND      kki.knight_name = kn.knight_name
         AND      kki.allegiance_start_date = kn.allegiance_start_date
         AND      kki.allegiance_end_date = kn.allegiance_end_date
         AND      kki.book = kn.book) SOURCE
  ON (target.kingdom_allegiance_id = SOURCE.kingdom_id)
  WHEN MATCHED THEN
  UPDATE SET allegiance_start_date = SOURCE.start_date
  ,          allegiance_end_date = SOURCE.end_date
  ,          book = SOURCE.book
  WHEN NOT MATCHED THEN
  INSERT VALUES
  ( knight_s1.nextval
  , SOURCE.knight_name
  , SOURCE.kingdom_id
  , SOURCE.start_date
  , SOURCE.end_date
  , SOURCE.book);
 
  -- Save the changes.
  COMMIT;
 
EXCEPTION
  WHEN OTHERS THEN
    dbms_output.put_line(SQLERRM);
    ROLLBACK TO starting_point;
    RETURN;
END;
/

Step #7 : Run the upload procedure

You can run the file by calling the script above. The procedure ensures that records are inserted or updated into their respective tables.

EXECUTE upload_kingdom;

Step #8 : Test the results of the upload procedure

You can test whether or not it worked by running the following queries.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
-- Format Oracle output.
COLUMN kingdom_id    FORMAT 999      HEADING "Kingdom|ID #"
COLUMN kingdom_name  FORMAT A14      HEADING "Kingdom|Name"
COLUMN population    FORMAT 999,999  HEADING "Population"
COLUMN book          FORMAT A40      HEADING "Source Book"
 
-- Check the kingdom table.
SELECT * FROM kingdom;
 
-- Format Oracle output.
SET PAGESIZE 999
COLUMN knight_id              FORMAT 999  HEADING "Knight|ID #"
COLUMN knight_name            FORMAT A23  HEADING "Knight|Name"
COLUMN kingdom_allegiance_id  FORMAT 999  HEADING "Kingdom|ID #"
COLUMN allegiance_start_date  FORMAT A11 HEADING "Allegiance|Start Date"
COLUMN allegiance_end_date    FORMAT A11 HEADING "Allegiance|End Date"
 
-- Check the knight table.
SELECT   knight_id
,        knight_name
,        kingdom_allegiance_id
,        TO_CHAR(allegiance_start_date,'DD-MON-YYYY') AS allegiance_start_date
,        TO_CHAR(allegiance_end_date,'DD-MON-YYYY') AS allegiance_end_date
FROM     knight;

It should display the following information:

Kingdom Kingdom
   ID # Name	       Population Source Book
------- -------------- ---------- ----------------------------------------
      1 Narnia		   42,100 Prince Caspian
      2 Narnia		   77,600 The Lion, The Witch and The Wardrobe
      3 Camelot 	   15,200 The Once and Future King
 
 
Knight Knight		       Kingdom Allegiance  Allegiance
  ID # Name			  ID # Start Date  End Date
------ ----------------------- ------- ----------- -----------
     1 Peter the Magnificent	     2 20-MAR-1272 19-JUN-1292
     2 Edmund the Just		     2 20-MAR-1272 19-JUN-1292
     3 Susan the Gentle 	     2 20-MAR-1272 19-JUN-1292
     4 Lucy the Valiant 	     2 20-MAR-1272 19-JUN-1292
     5 Peter the Magnificent	     1 12-APR-1531 31-MAY-1328
     6 Edmund the Just		     1 12-APR-1531 31-MAY-1328
     7 Susan the Gentle 	     1 12-APR-1531 31-MAY-1328
     8 Lucy the Valiant 	     1 12-APR-1531 31-MAY-1328
     9 King Arthur		     3 10-MAR-0631 12-DEC-0686
    10 Sir Lionel		     3 10-MAR-0631 12-DEC-0686
    11 Sir Bors 		     3 10-MAR-0631 12-DEC-0635
    12 Sir Bors 		     3 10-MAR-0640 12-DEC-0686
    13 Sir Galahad		     3 10-MAR-0631 12-DEC-0686
    14 Sir Gawain		     3 10-MAR-0631 12-DEC-0686
    15 Sir Tristram		     3 10-MAR-0631 12-DEC-0686
    16 Sir Percival		     3 10-MAR-0631 12-DEC-0686
    17 Sir Lancelot		     3 30-SEP-0670 12-DEC-0686

You can rerun the procedure to check that it doesn’t alter any information, then you could add a new knight to test the insertion portion.

Written by maclochlainn

January 6th, 2024 at 11:53 pm

Native sqlplus editing

without comments

I have to remind myself from time to time that Ubuntu is a Desktop or Workstation and by default can go missing key server software, like ssh. This became evident when I wanted to check whether I could run sqlplus from my Mac OS terminal through my Ubuntu VM and internally embedded Oracle Database 23c Free docker instance.

If like me you forgot to add it, you can add the ssh service with the following commands to your Ubuntu VM:

sudo apt update
sudo apt install -y openssh-server
sudo systemctl start ssh.service

Then, you can test the installation with an ssh call to localhost, like:

ssh localhost

You should see the following, where you need to enter the sudoer’s password to continue. Your localhost target causes an authenticity check, like:

The authenticity of host 'localhost (127.0.0.1)' can't be established.
ED25519 key fingerprint is SHA256:js8knEf/lOE1rSss3u8lP4Ii634Y0CkUz+oJM5dt3w4.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Enter yes to continue:

Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

It will now add localhost to the list of known hosts provide standard messages, as shown below.

Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
student@localhost's password: 
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.2.0-39-generic x86_64)
 
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
 
Expanded Security Maintenance for Applications is not enabled.
 
9 updates can be applied immediately.
5 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable
 
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
 
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
 
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Having verified the installation and functionality of ssh in the Ubuntu VM. Then, I launched a Terminal session on my MacBookPro base operating system. Using the Ubuntu instance ssh and a customized Bash function, I discovered its IP address.

The following is the get_ip() user-defined function in the Ubuntu instance’s student user’s customized .bashrc file:

# Return the local instance's IP address.
get_ip ()
{
  echo `hostname -I | cut -f1 -d' '`
}

In this instance, it returned:

192.168.195.155

With the IP address, I secured shelled into my Ubuntu sudoer student user like this:

ssh student@192.168.195.155

It’ll prompt you for the remote server’s student password, like:

student@192.168.195.155's password:

After entering the correct password, I got the standard reply of a valid connection:

Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.2.0-39-generic x86_64)
 
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
 
Expanded Security Maintenance for Applications is not enabled.
 
9 updates can be applied immediately.
5 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable
 
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
 
Last login: Fri Jan  5 18:13:21 2024 from 127.0.0.1

Next, I connected to the Ubuntu Docker Oracle Database 23c Free instance with this syntax:

docker exec -it --user student oracle23c bash

At the prompt for the Docker instance of Oracle Database 23c Free, you can type sqlplus to work directly against the Oracle Database 23c Free instance with a pluggable c##student database user.

sqlplus c##student/student
 
SQL*Plus: Release 23.0.0.0.0 - Production on Sat Jan 6 01:38:06 2024
Version 23.3.0.23.09
 
Copyright (c) 1982, 2023, Oracle.  All rights reserved.
 
Last Successful login time: Sat Dec 23 2023 04:30:00 +00:00
 
Connected to:
Oracle Database 23c Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.3.0.23.09

Now, I can interactively edit my files with vi in the Docker Oracle Database 23c Free directory. The following demonstrates using the sandboxed student() function from my earlier Oracle 23c Free SQL*Plus blog post and connects as a sandboxed student user in the Docker Oracle 23c Free container. The image uses a different Mac OS and different Ubuntu VM from the earlier entries in this blog post from the earlier examples.

You can edit and test the files in the Docker Oracle 23c Free instance through the command-line interface (CLI). You can further automate the ssh connection by making the Ubuntu instance’s IP address a static address instead of a DCHP-assigned address; and then you can put it in the Mac OS’s /etc/hosts file which lets you resolve it by name (through file versus DNS resolution).

As always, I hope this helps those looking for a solution.

Written by maclochlainn

January 5th, 2024 at 6:26 pm

Oracle 23c Free SQL*Plus

without comments

It’s always frustrated me when using the sqlplus command-line interface (CLI) that you can’t just “up arrow” to through the history. At least, that’s the default case unless you wrap the sqlplus executable.

I like to do my development work as close to the database as possible. The delay from SQL Developer to the database or VSCode to the database is just too long. Therefore, I like the native sqlplus to be as efficient as possible. This post shows you how to install the rlwarp utility to wrap sqlplus and create a sandboxed student user for a local development account inside the Oracle 23c Free container. You should note that the Docker or Podman Container is using Oracle Unbreakable Linux 8 as it’s native OS.

You can connect to your Docker version of Oracle Database 23c Free with the following command:

docker exec -it -u root oracle23c bash

You can’t just use dnf to install rlwrap and get it to magically install all the dependencies. That would be too easy, eh?

Attempting to do so will lock your base OS and eventually force you to kill with prejudice the hung dnf process (at least it forced me to do so). You need to determine the rlwrap dependencies and then install them first. In that process, I noticed that the which utility program wasn’t installed in the container.

Naturally, I installed the which utility first with this command:

dnf install -y which

The rlwrap dependencies are: glibc, ncurses, perl, readline, python, and git. Only the perl, python, and git are missing from the list of formal dependencies but there’s another dependency the epel-release package.

If you want to verify whether a package is installed, you can use the rpm command like this:

rpm -qa | grep package_name

I installed the perl programming environment (a big install) with this command:

dnf install -y perl

I installed the python3 with this command:

dnf install -y python3

I installed the git module with this command:

dnf install -y git

I installed the epel-release container with this command:

dnf install -y epel-release

After installing all of these, you’re now ready to install the core rlwrap utility program. Like the other installations, you use:

dnf install -y rlwrap

At this point, you need to create a sandboxed user account for the Docker instance because as a developer using the root user for simple tasks is a bad idea. While you could do this with a Docker command, the Oracle 23c Free edition raised a lock on the /etc/group file when I tried it. Naturally, that’s not a problem because you can connect as the root user with this syntax:

docker exec -it -u root oracle23c bash

As the root user, create a student account as a developer account in the Oracle 23c Free container:

useradd -u 501 -g dba -G users -d /home/student -s /bin/bash/ -c "Student" -n student

You’ll be unable to leverage the tnsnames.ora file unless you alter the prior command to replace dba with oinstall or add the following command:

usermod -a -G oinstall student

Exit the Oracle 23c Free container as the root user and reconnect as the student user with this syntax:

docker exec -it --user student oracle23c bash

While you’re connected as the root user, you should create an upload directory as a subdirectory of the $ORACLE_BASE directory. The $ORACLE_BASE directory in the Oracle Database 23c Free Docker image is the /opt/oracle directory.

You should use the following syntax to create the upload directory and change its permission to that of the Oracle Database 23c Free installation (for a future blog post on developing external table deployment on the Docker image):

mkdir /opt/oracle
chown -R oracle:install /opt/oracle/upload

You also can add the following student function to the Ubuntu student user’s .bashrc file. It means all you need to type to connect to the Oracle Database 23c Free Docker instance is “student“. I like shortcuts like this one, which let you leverage one-line Python commands.

student () 
{
    # Discover the fully qualified program name. 
    path=`which docker 2>/dev/null`
    file=''
 
    # Parse the program name from the path.
    if [ -n ${path} ]; then
        file=${path##/*/}
    fi
 
    # Wrap when there is a file and it is rewrap.
    if [ -n ${file} ] && [[ ${file} = "docker" ]]; then
        python -c "import subprocess; subprocess.run(['docker exec -it --user student oracle23c bash'], shell=True)" 
    else
        echo "Docker is unavailable: Install the docker package."
    fi
}

Open a Ubuntu Terminal shell and type a student function name to connect to the Docker Oracle Database 23c Free instance where you can now test things like external tables with the SQL*Plus command line without installing it on the Ubuntu local operating system.

student@student-virtual-machine:~$ student
[student@d28375f0c43f ~]$ sqlplus c##student/student@free
 
SQL*Plus: Release 23.0.0.0.0 - Production on Wed Jan 3 02:14:22 2024
Version 23.3.0.23.09
 
Copyright (c) 1982, 2023, Oracle.  All rights reserved.
 
Last Successful login time: Wed Jan 03 2024 01:56:44 +00:00
 
Connected to:
Oracle Database 23c Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.3.0.23.09
 
SQL>

Then, I added this sqlplus function to the /home/student/.bashrc file, which is owned by the student user. However, I also added the instruction to change to the student user’s home directory because the Oracle 23c Free container will connect you to the /home/oracle directory by default. I also added the default long list (ll) alias to the .bashrc file.

sqlplus () 
{
    # Discover the fully qualified program name. 
    path=`which rlwrap 2>/dev/null`
    file=''
 
    # Parse the program name from the path.
    if [ -n ${path} ]; then
        file=${path##/*/}
    fi;
 
    # Wrap when there is a file and it is rewrap.
    if [ -n ${file} ] && [[ ${file} = "rlwrap" ]]; then
        rlwrap sqlplus "${@}"
    else
        echo "Command-line history unavailable: Install the rlwrap package."
        $ORACLE_HOME/bin/sqlplus "${@}"
    fi
}
 
# Change to the user's home directory.
cd ${HOME}
 
# Create a long list alias:
alias ll='ls -l --color=auto'

After you’ve configured your student user, you can configure the oracle user account to work like a regular server. Exit the Docker Oracle Database 23c Free as the student user, then connect as the root user with this command:

docker exec -it -u root oracle23c bash

As the root user you can become the oracle user with the following command:

su - oracle

Now, add the following .bashrc shell in the /home/oracle directory:

# The oracle user's .bashrc
 
# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi
 
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
    PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
 
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
 
# User specific aliases and functions
export ORACLE_SID=FREE
export ORACLE_BASE=/opt/oracle
export ORACLE_HOME=/opt/oracle/product/23c/dbhomeFree
export PATH=$PATH:/$ORACLE_HOME/bin
 
# Change to the user's home directory.
cd ${HOME}
 
# Create a long list alias:
alias ll='ls -l --color=auto'
 
sqlplus () 
{
    # Discover the fully qualified program name. 
    path=`which rlwrap 2>/dev/null`
    file=''
 
    # Parse the program name from the path.
    if [ -n ${path} ]; then
        file=${path##/*/}
    fi;
 
    # Wrap when there is a file and it is rewrap.
    if [ -n ${file} ] && [[ ${file} = "rlwrap" ]]; then
        rlwrap sqlplus "${@}"
    else
        echo "Command-line history unavailable: Install the rlwrap package."
        $ORACLE_HOME/bin/sqlplus "${@}"
    fi
}

You need to manually source the .bashrc for the oracle user because it’s not an externally available user. Use this syntax to connect as the internal user:

sqlplus / as sysdba

It’ll display:

SQL*Plus: RELEASE 23.0.0.0.0 - Production ON Wed Jan 3 07:08:11 2024
Version 23.3.0.23.09
 
Copyright (c) 1982, 2023, Oracle.  ALL rights reserved.
 
 
Connected TO:
Oracle DATABASE 23c Free RELEASE 23.0.0.0.0 - Develop, Learn, AND Run FOR Free
Version 23.3.0.23.09
 
SQL>

After all this, I can now click the “up arrow” to edit any of the sqlplus command history. If you like to work inside sqlplus natively, this should help you.

Written by maclochlainn

December 20th, 2023 at 11:11 pm

OracleDB Python Tutorial 1

without comments

This shows you how to get Python working with the Oracle Database 23c in Docker or Podman on Ubuntu. You can find useful connection strings for this in Oracle Database Free Get Started.

  1. First step requires you to install the pip3/span> utility on Ubuntu.

    sudo apt install -y python3-pip

  2. Second step requires that you pip3 install the oracledb library:

    sudo pip3 install oracledb --upgrade

  3. Third step requires you write a Python program to test your connection to Oracle Database 23c Free, like:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    
    #!/usr/bin/python
     
    # Import the Oracle library.
    import oracledb
     
    try:
      # Create a connection to local Docker or Podman installation.
      db = oracledb.connect(user='c##student', password='student', dsn='localhost:51521/FREE')
     
      # Print a connection message.
      print("Connected to the Oracle", db.version, "database.")
     
    except oracledb.DatabaseError as e:
      error, = e.args
      print(sys.stderr, "Oracle-Error-Code:", error.code)
      print(sys.stderr, "Oracle-Error-Message:", error.message)
     
    finally:
      # Close connection. 
      db.close()

    The 51521 port is the recommended port when setting up Docker or Podman services, however, it can be set to any port above 1024.

    It should print:

    Connected to the Oracle 23.3.0.23.9 database.
  4. Fourth step requires you write a Python program to test querying data from an Oracle Database 23c Free instance. I created the following avenger table and seeded it with six Avengers.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    
    /* Conditionally drop the table. */
    DROP TABLE IF EXISTS avenger;
     
    /* Create the table. */
    CREATE TABLE avenger
    ( avenger_id      NUMBER
    , first_name      VARCHAR2(20)
    , last_name       VARCHAR2(20)
    , character_name  VARCHAR2(20));
     
    /* Seed the table with data. */
    INSERT INTO avenger VALUES (1,'Anthony','Stark','Iron Man');
    INSERT INTO avenger VALUES (2,'Thor','Odinson','God of Thunder');
    INSERT INTO avenger VALUES (3,'Steven','Rogers','Captain America');
    INSERT INTO avenger VALUES (4,'Bruce','Banner','Hulk');
    INSERT INTO avenger VALUES (5,'Clinton','Barton','Hawkeye');
    INSERT INTO avenger VALUES (6,'Natasha','Romanoff','Black Widow');

    Then, I extended the program logic to include a cursor and for loop to read the values from the avenger table:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    
    #!/usr/bin/python
     
    # Import the Oracle library.
    import oracledb
     
    try:
      # Create a connection to local Docker or Podman installation.
      db = oracledb.connect(user='c##student', password='student', dsn='localhost:51521/FREE')
     
      # Create a cursor.
      cursor = db.cursor()
     
      # Execute a query.
      cursor.execute("SELECT   character_name " +
                     ",        first_name " +
                     ",        last_name " +
                     "FROM     avenger " +
                     "ORDER BY character_name")
     
      # Read the contents of the cursor.
      for row in cursor:
        print(row[0] + ':',row[2] + ',',row[1])
     
    except oracledb.DatabaseError as e:
      error, = e.args
      print(sys.stderr, "Oracle-Error-Code:", error.code)
      print(sys.stderr, "Oracle-Error-Message:", error.message)
     
    finally:
      # Close cursor and connection.
      cursor.close() 
      db.close()

    The 51521 port is the recommended port when setting up Docker or Podman services, however, it can be set to any port above 1024.

    It should print:

    Black Widow: Romanoff, Natasha
    Captain America: Rogers, Steven
    God of Thunder: Odinson, Thor
    Hawkeye: Barton, Clinton
    Hulk: Banner, Bruce
    Iron Man: Stark, Anthony
  5. Fifth step requires you write a Python program to test querying data filtered by a local variable from an Oracle Database 23c Free instance. This example looks only for the Hulk among the six Avengers.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    
    #!/usr/bin/python
     
    # Import the Oracle library.
    import oracledb
     
    try:
      # Create a connection to local Docker or Podman installation.
      db = oracledb.connect(user='c##student', password='student', dsn='localhost:51521/FREE')
     
      # Create a cursor.
      cursor = db.cursor()
     
      # Execute a query.
      stmt = "SELECT   character_name "            \
             ",        first_name "                \
             ",        last_name "                 \
             "FROM     avenger "                   \
             "WHERE    character_name = :avenger " \
             "ORDER BY character_name"
     
      # Execute with bind variable.
      cursor.execute(stmt, avenger = "Hulk")
     
      # Read the contents of the cursor.
      for row in cursor:
        print(row[0] + ':',row[2] + ',',row[1])
     
    except oracledb.DatabaseError as e:
      error, = e.args
      print(sys.stderr, "Oracle-Error-Code:", error.code)
      print(sys.stderr, "Oracle-Error-Message:", error.message)
     
    finally:
      # Close cursor and connection. 
      cursor.close() 
      db.close()

    It should print:

    Hulk: Banner, Bruce

As always, I hope this puts everything together for setting up Python with Oracle Database 23c Free.

Written by maclochlainn

December 10th, 2023 at 12:27 am

Parallels + Ubuntu

without comments

Installing Parallels on my iStudio (M2) was straightforward because I let it install Windows 11. Then, when I wanted to install Ubuntu it wasn’t quite that easy. You just need to understand that you click the Parallels’ Window menu option and Control Center option.

The Control Center option provides the following dialog.

Click the + symbol to create a new virtualization and you get the following dialog; and choose the Download Ubuntu with x86_64 emulation if you want to install a Docker image that’s not ported to ARM, like the Oracle Database 19c.

It’ll then explain in this dialog that is uses Apple’s Rosetta 2 technology, which means you should be able to install an Intel architecture Docker image with Oracle Database 23c.

As always, I hope this helps those trying to sort out how to leverage a new stack.

Written by maclochlainn

October 19th, 2023 at 3:41 pm

SQL Developer & PostgreSQL

without comments

I had a request from one of the adjunct professors to connect SQL Developer to the PostgreSQL database. This is in support of our database programming class that teaches students how to write PL/SQL against the Oracle database and pgPL/SQL against the PostgreSQL database. We also demonstrate transactional management through Node.js, Python and Java.

Naturally, this is also a frequent step taken by those required to migrate PostgreSQL data models to an Oracle database. While my final solution requires mimicking Oracle’s database user to schema, it does work for migration purposes. I’ll update this post when I determine how to populate the database drop-down list.

The first step was figuring out where to put the PostgreSQL JDBC Java ARchive (.jar) file on a Linux distribution. You navigate to the end-user student account in a Terminal and change to the .sqldeveloper directory. Then, create a jdbc subdirectory as the student user with the following command:

mkdir /home/student/.sqldeveloper/jdbc

Then, download the most current PostgreSQL JDBC Java ARchive (.jar) file and copy it into the /home/student/.sqldeveloper/jdbc, which you can see afterward with the following command:

ll /home/student/.sqldeveloper/jdbc

It should display:

-rw-r--r--. 1 student student 1041081 Aug  9 13:46 postgresql-42.3.7.jar

The next series of steps are done within SQL Developer. Launch SQL Developer and navigate to Tools and Preferences, like this:

Inside the Preferences dialog, navigate to Database and Third Party JDBC Drivers like shown and click the Add Entry button to proceed:

Inside the Select Path Entry dialog, select the current PostgreSQL JDBC Java ARchive (.jar) file, which is postgresql-42-3.7.jar in this example. Then, click the Select button.

You are returned to the Preferences dialog as shown below. Click the OK button to continue.

After completing the 3rd Party Java Driver setup, you attempt to create a new connection to the PostgreSQL database. You should see that you now have two available Database Type values: Oracle and PostgreSQL, as shown below:

When you click on the PostgreSQL Database Type, the dialog updates to the following view. Unfortunately, I couldn’t discover how to set the values in the list for the Choose Database drop down. Naturally, a sandboxed user can’t connect to the PostgreSQL database without qualifying the database name.

Unless you qualify the PostgreSQL database or connect as the postgres user with a privileged password, SQL Developer translates the absence of a database selection to a database name equivalent to the user’s name. That’s the default behavior for the Oracle database but differs from the behavior for MySQL, PostgreSQL, and Microsoft SQL Server. It returns the following

Status: Failure - Test failed: FATAL: database "student" does not exist

As seen in the diaglog’s result when testing the connection:

Based on my hunch and not knowing how to populate the database field for the connection, I did the following:

  • Created a Linux OS videodb user.
  • Copied the .bashrc file with all the standard Oracle environment variables.
  • Created the /home/videodb/.sqldeveloper/jdbc directory.
  • Copied the postgresql-42.3.7.jar into the new jdbc directory.
  • Connected as the postgres super user and created the PostgreSQL videodb user with this syntax:

    CREATE USER videodb
      WITH ROLE dba
           ENCRYPTED PASSWORD 'cangetin';
  • As the postgres super user, granted the following privileges:

    -- Grant privileges on videodb database videodb user.
    GRANT ALL ON DATABASE "videodb" TO "videodb";
     
    -- Connect to the videodb database.
    \c
     
    -- Grant privileges.
    GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO videodb;
    GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO videodb;
  • Added the following line to the pg_hba.conf file in the /var/lib/pgsql/15/data directory as the postgres user:

    local   all             videodb                                 peer
  • Connected as the switched from the student to videodb Linux user, and launched SQL Developer. Then, I used the Tools menu to create the 3rd party PostgreSQL JDBC Java ARchive (.jar) file in context of the SQL Developer program. Everything completed correctly.
  • Created a new PostgreSQL connection in SQL Developer and tested it with success as shown:

  • Saving the new PostgreSQL connection, I opened the connection and could run SQL statements and display the catalog information, as shown:

    Connected as the videodb user to the videodb database I can display tables owned by student and videodb users:

    -- List tables.
    \d
     
                       List of relations
     Schema |           Name           |   Type   |  Owner
    --------+--------------------------+----------+---------
     public | new_hire                 | table    | student
     public | new_hire_new_hire_id_seq | sequence | student
     public | oracle_test              | table    | videodb
    (3 rows)

    In SQL Developer, you can also inspect the tables, as shown:

At this point, I’m working on trying to figure out how to populate the database drop-down table. However, I’ve either missed a key document or it’s unfortunate that SQL Developer isn’t as friendly as MySQL Workbench in working with 3rd Party drivers.

Written by maclochlainn

August 8th, 2023 at 11:29 pm

A tkprof Korn Shell

without comments

Reviewing old files, I thought posting my tkprof.ksh would be helpful. So, here’s the script that assumes you’re using Oracle e-Business Suite (Demo database, hence the APPS/APPS connection); and if I get a chance this summer I’ll convert it to Bash shell.

#!/bin/ksh
# -------------------------------------------------------------------------
# Author:   Michael McLaughlin
# Name:     tkprof.ksh
 
# Purpose:  The program takes the following arguments:
#           1. A directory
#           2. A search string
#           3. A target directory
#           It assumes raw trace files have an extension of ".trc".
#           The output file name follows this pattern (because it is
#           possible for multiple tracefiles to be written during the
#           same minute).
# -------------------------------------------------------------------------
 
# Function to find minimum field delimiter.
function min
{
  # Find the whitespace that preceeds the file date.
  until [[ $(ls -al $i | cut -c$minv-$minv) == " " ]]; do
    let minv=minv+1
  done
}
 
# Function to find maximum field delimiter.
function max
{
  # Find the whitespace that succeeds the file date.
  until [[ $(ls -al $i | cut -c$maxv-$maxv) == " " ]]; do
    let maxv=maxv+1
  done
}
 
# Debugging enabled by unremarking the "set -x"
# set -x
 
# Print header information
print =================================================================
print Running [tkprof.ksh] script ...
 
# Evaluate whether an argument is provide and if no argument
# is provided, then substitute the present working directory.
if   [[ $# == 0 ]]; then
  dir=${PWD}  
  str="*"
  des=${PWD}  
elif [[ $# == 1 ]]; then
  dir=${1}
  str="*"
  des=${1}
elif [[ $# == 2 ]]; then
  dir=${1}
  str=${2}
  des=${1}
elif [[ $# == 3 ]]; then
  dir=${1}
  str=${2}
  des=${3}
fi
 
# Evaluate whether the argument is a directory file.
if [[ -d ${dir} ]] && [[ -d ${des} ]]; then
 
  # Print what directory and search string are targets.
  print =================================================================
  print Run in tkprof from [${dir}] directory ...
  print The files contain a string of [${str}] ...
  print =================================================================
 
  # Evaluate whether the argument is the present working
  # directory and if not change directory to that target
  # directory so file type evaluation will work.
  if [[ ${dir} != ${PWD} ]]; then
    cd ${dir} 
  fi
 
  # Set file counter.
  let fcnt=0
 
  # Submit compression to the background as a job.
  for i in $(grep -li "${str}" *.trc); do
 
    # Evaluate whether file is an ordinary file.
    if [[ -f ${i} ]]; then
 
      # Set default values each iteration.
      let minv=40
      let maxv=53
 
      # Increment counter.
      let fcnt=fcnt+1
 
      # Call functions to reset min and max values where necessary.
      min ${i}
      max ${i}
 
      # Parse date stamp from trace file without multiple IO calls.
      # Assumption that the file is from the current year.
      date=$(ls -al ${i} | cut -c${minv}-${maxv}) 
      mon=$(echo ${date} | cut -c1-3)
      yr=$(date          | cut -c25-28)
 
      # Validate month is 10 or greater to pad for reduced whitespace.
      if (( $(echo ${date} | cut -c5-6) < 10 )); then
        day=0$(echo ${date}| cut -c5-5)
        hr=$(echo ${date}  | cut -c7-8)
        min=$(echo ${date} | cut -c10-11)
      else
        day=$(echo ${date} | cut -c5-6)
        hr=$(echo ${date}  | cut -c8-9)
        min=$(echo ${date} | cut -c11-12)
      fi
 
      fn=file${fcnt}_${day}-${mon}-${yr}_${hr}:${min}:${day}
 
      print Old [$i] and new [$des/$fn]
      tkprof ${i} ${des}/${fn}.prf explain=APPS/APPS sort='(prsela,exeela,fchela)'
 
      # Print what directory and search string are targets.
      print =================================================================
    fi
  done
 
else
  # Print message that a directory argument was not provided.
  print You failed to provie a single valid directory argument.
fi

I hope this helps those looking for a solution.

Written by maclochlainn

May 21st, 2023 at 2:25 am

Listener for APEX

without comments

Unless dbca lets us build the listener.ora file, we often leave off some component. For example, running listener control program the following status indicates an incorrectly configured listener.ora file.

lsnrctl status

It returns the following, which displays an endpoint for the XDB Server (I’m using Oracle Database 11g XE because it’s pre-containerized and has a small testing footprint):

LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 24-MAR-2023 00:59:06
 
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_XE)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.2.0 - Production
Start Date                21-MAR-2023 21:17:37
Uptime                    2 days 3 hr. 41 min. 29 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Default Service           XE
Listener Parameter File   /u01/app/oracle/product/11.2.0/xe/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/localhost/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_XE)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=8080))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "XE" has 1 instance(s).
  Instance "XE", status READY, has 1 handler(s) for this service...
Service "XEXDB" has 1 instance(s).
  Instance "XE", status READY, has 1 handler(s) for this service...
The command completed successfully

The listener is missing the second SID_LIST_LISTENER value of CLRExtProc value. A complete listener.ora file should be as follows for the Oracle Database XE:

# listener.ora Network Configuration FILE:
 
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/xe)
      (PROGRAM = extproc)
    )
    (SID_DESC =
      (SID_NAME = CLRExtProc)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/xe)
      (PROGRAM = extproc)
    )
  )
 
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC_FOR_XE))
      (ADDRESS = (PROTOCOL = TCP)(HOST = localhost.localdomain)(PORT = 1521))
    )
  )
 
DEFAULT_SERVICE_LISTENER = (XE)

With this listener.ora file, the Oracle listener control utility will return the following correct status, which hides the XDB Server’s endpoint:

LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 24-MAR-2023 02:38:57
 
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_XE)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.2.0 - Production
Start Date                24-MAR-2023 02:38:15
Uptime                    0 days 0 hr. 0 min. 42 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Default Service           XE
Listener Parameter File   /u01/app/oracle/product/11.2.0/xe/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/11.2.0/xe/log/diag/tnslsnr/localhost/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_XE)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521)))
Services Summary...
Service "CLRExtProc" has 1 instance(s).
  Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

It seems a number of examples on the web left the SID_LIST_LISTENER value of CLRExtProc value out of the listener.ora file. As always, I hope this helps those looking for a complete solution rather than generic instructions without a concrete example.

Written by maclochlainn

March 24th, 2023 at 1:00 am

AWS EC2 TNS Listener

without comments

Having configured an AlmaLinux 8.6 with Oracle Database 11g XE, MySQL 8.0.30, and PostgreSQL 15, we migrated it to AWS EC2 and provisioned it. We used the older and de-supported Oracle Database 11g XE because it didn’t require any kernel modifications and had a much smaller footprint.

I had to address why attempting to connect with the sqlplus utility raised the following error after provisioning a copy with a new static IP address:

ERROR:
ORA-12514: TNS:listener does NOT currently know OF service requested IN CONNECT descriptor

A connection from SQL Developer raises a more addressable error, like:

ORA-17069

I immediately tried to check the connection with the tnsping utility and found that tnsping worked fine. However, when I tried to connect with the sqlplus utility it raised an ORA-12514 connection error.

There were no diagnostic steps beyond checking the tnsping utility. So, I had to experiment with what might block communication.

I changed the host name from ip-172-58-65-82.us-west-2.compute.internal to a localhost string in both the listener.ora and tnsnames.ora. The listener.ora file:

# listener.ora Network Configuration FILE:
 
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/xe)
      (PROGRAM = extproc)
    )
  )
 
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC_FOR_XE))
      (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
    )
  )
 
DEFAULT_SERVICE_LISTENER = (XE)

The tnsnames.ora file:

# tnsnames.ora Network Configuration FILE:
 
XE =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = XE)
    )
  )
 
EXTPROC_CONNECTION_DATA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC_FOR_XE))
    )
    (CONNECT_DATA =
      (SID = PLSExtProc)
      (PRESENTATION = RO)
    )
  )

I suspected that it might be related to the localhost value. So, I checked the /etc/hostname and /etc/hosts files.

Then, I modified /etc/hostname file by removing the AWS EC2 damain address. I did it on a memory that Oracle’s TNS raises errors for dots or periods in some addresses.

The /etc/hostname file:

ip-172-58-65-82

The /etc/hosts file:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 ip-172-58-65-82
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 ip-172-58-65-82

Now, we can connect to the Oracle Database 11g XE instance with the sqlplus utility. I believe this type of solution will work for other AWS EC2 provisioned Oracle databases.

Written by maclochlainn

March 22nd, 2023 at 10:09 pm

GROUP BY Quirk

without comments

It’s always interesting to see how others teach SQL courses. It can be revealing as to whether they understand SQL or only understand a dialect of SQL. In this case, one of my old students was taking a graduate course in SQL and the teacher was using MySQL. The teacher made an issue of using ANSI SQL:1999 or SQL3 and asked the following question, which I suspect is a quiz bank question from a textbook:

“How would you get all students’ names and for each student the number of courses that the
student has registered for?”

They referenced the MySQL 5.7 documentation for the GROUP BY and SQL:1999 as if MySQL implemented the ANSI SQL:1999 specification defined the standard. I didn’t know whether to laugh or cry because they were referring to MySQL 5.7 when we’re all using MySQL 8 and anybody who’s worked in more than MySQL knows that the behavior for a GROUP BY in MySQL can work without listing the necessary non-aggregated columns in the SELECT-list.

For example, their working solution, which is from the instructor and the author of their MySQL textbook the correct perspective of ANSI:1999 behavior. It doesn’t matter that their solution is actually based on ANSI:1992 not ANSI:1999 because it will only succeed because of a quirk of MySQL:

SELECT   a.studentname
,        COUNT(b.courseid)
FROM     students a INNER JOIN registeredcourses b
ON       a.studentid = b.studentid
GROUP BY a.studentid;

While it works in MySQL, it doesn’t work because it conforms to an ANSI standard. It works in MySQL, notwithstanding that standard because it violates the standard.

In Oracle, PostgreSQL, and SQL Server, it raises an exception. For example, Oracle raises the following exception:

SELECT   a.studentname
         *
ERROR at line 1:
ORA-00979: not a GROUP BY expression

The correct way to write the GROUP BY is:

SELECT   a.studentname
,        COUNT(b.courseid)
FROM     students a INNER JOIN registeredcourses b
ON       a.studentid = b.studentid INNER JOIN courses c
ON       b.courseid = c.courseid
GROUP BY a.studentname;

Then, it would return:

Student Name                   Course IDs
------------------------------ ----------
Montgomery Scott                        1
Leonard McCoy                           2
James Tiberus Kirk                      3

For reference, here’s a complete test case for MySQL:

/* Drop table conditionally. */
DROP TABLE IF EXISTS students;
 
/* Create table. */
CREATE TABLE students
( studentID    int unsigned primary key auto_increment
, studentName  varchar(30));
 
/* Drop table conditionally. */
DROP TABLE IF EXISTS courses;
 
/* Create table. */
CREATE TABLE courses
( courseid    int unsigned primary key auto_increment
, coursename  varchar(40));
 
/* Drop table conditionally. */
DROP TABLE IF EXISTS registeredcourses;
 
/* Create table. */
CREATE TABLE registeredcourses
( courseid    int unsigned
, studentid   int unsigned );
 
/* Insert into students. */
INSERT INTO students
( studentName )
VALUES
 ('James Tiberus Kirk')
,('Leonard McCoy')
,('Montgomery Scott');
 
/* Insert into courses. */
INSERT INTO courses
( coursename )
VALUES
 ('English Literature')
,('Physics')
,('English Composition')
,('Botany')
,('Mechanical Engineering');
 
/* Insert into registeredcourses. */
INSERT INTO registeredcourses
( studentid
, courseid )
VALUES
 (1,1)
,(1,3)
,(1,4)
,(2,2)
,(2,5)
,(3,4); 
 
/* Check global sql_mode to ensure only_full_group_by is set. */
SELECT @@GLOBAL.SQL_MODE;
 
/* Query with a column not found in the SELECT-list. */
SELECT   a.studentname
,        COUNT(b.courseid)
FROM     students a INNER JOIN registeredcourses b
ON       a.studentid = b.studentid
GROUP BY a.studentid;
 
/* Query consistent with ANSI SQL:1992 */
SELECT   a.studentname
,        COUNT(b.courseid)
FROM     students a INNER JOIN registeredcourses b
ON       a.studentid = b.studentid INNER JOIN courses c
ON       b.courseid = c.courseid
GROUP BY a.studentname;

and, another complete test case for Oracle:

/* Drop tabhe unconditionallly. */
DROP TABLE students;
 
/* Create table. */
CREATE TABLE students
( studentID    NUMBER PRIMARY KEY
, studentName  VARCHAR(30));
 
/* Drop table unconditionally. */
DROP TABLE courses;
 
/* Create table. */
CREATE TABLE courses
( courseid    NUMBER PRIMARY KEY
, coursename  VARCHAR(40));
 
/* Drop table unconditionally. */
DROP TABLE registeredcourses;
 
/* Create table. */
CREATE TABLE registeredcourses
( courseid    NUMBER
, studentid   NUMBER );
 
/* Insert values in student. */
INSERT INTO students ( studentid, studentName ) VALUES (1,'James Tiberus Kirk');
INSERT INTO students ( studentid, studentName ) VALUES (2,'Leonard McCoy');
INSERT INTO students ( studentid, studentName ) VALUES (3,'Montgomery Scott');
 
/* Insert values in courses. */
INSERT INTO courses ( courseid, coursename ) VALUES (1,'English Literature');
INSERT INTO courses ( courseid, coursename ) VALUES (2,'Physics');
INSERT INTO courses ( courseid, coursename ) VALUES (3,'English Composition');
INSERT INTO courses ( courseid, coursename ) VALUES (4,'Botany');
INSERT INTO courses ( courseid, coursename ) VALUES (5,'Mechanical Engineering');
 
/* Insert values into registeredcourses. */
INSERT INTO registeredcourses ( studentid, courseid ) VALUES (1,1);
INSERT INTO registeredcourses ( studentid, courseid ) VALUES (1,3);
INSERT INTO registeredcourses ( studentid, courseid ) VALUES (1,4);
INSERT INTO registeredcourses ( studentid, courseid ) VALUES (2,2);
INSERT INTO registeredcourses ( studentid, courseid ) VALUES (2,5);
INSERT INTO registeredcourses ( studentid, courseid ) VALUES (3,4); 
 
/* Non-ANSI SQL GROUP BY statement. */
SELECT   a.studentname
,        COUNT(b.courseid)
FROM     students a INNER JOIN registeredcourses b
ON       a.studentid = b.studentid
GROUP BY a.studentid;
 
/* ANSI SQL GROUP BY statement. */
SELECT   a.studentname AS "Student Name"
,        COUNT(b.courseid) AS "Course IDs"
FROM     students a INNER JOIN registeredcourses b
ON       a.studentid = b.studentid INNER JOIN courses c
ON       b.courseid = c.courseid
GROUP BY a.studentname;

I hope this helps those learning the correct way to write SQL.

Written by maclochlainn

January 12th, 2023 at 11:30 pm