MacLochlainns Weblog

Oracle Database 9i introduced external tables. You can create external tables to load plain text files by using Oracle SQL*Loader. Alternatively, you can create external tables that load and unload files by using Oracle Data Pump. This article demonstrates both techniques.

You choose external tables that use Oracle SQL*Loader when you want to import plain text files. There are three types of plain text files. They are comma-separated value (CSV), tab-separated value (TSV), and position specific text files.

External tables that use Oracle Data Pump don’t work with plain text files. They work with an Oracle proprietary format. That means you load source files previously created by an Oracle Data Pump export. You typically create external tables with Oracle Data Pump when you’re moving large data sets between database instances.

External tables use Oracle’s virtual directories. An Oracle virtual directory is an internal reference in the data dictionary. A virtual directory maps a unique directory name to a physical directory on the local operating system. Virtual directories were simple before Oracle Database 12c gave us the multitenant architecture. In a multitenant database there are two types of virtual directories. One services the schemas of the Container Database (CDB) and it’s in the CDB’s SYS schema. The other services the schemas of a Pluggable Database (PDB) and it’s in the ADMIN schema for the PDB.

You can create a CDB virtual directory as SYSTEM user with the following syntax in Windows:

SQL> CREATE DIRECTORY upload AS 'C:\Data\Upload';

or, like this in Linux or Unix:

SQL> CREATE DIRECTORY upload AS '/u01/app/oracle';

There are some subtle differences between these two statements. Windows directories or folders start with a logical drive letter, like C:\, D:\, and so forth. Linux and Unix directories start with a mount point like /u01.

One of the subtle differences is directory and file ownership. You can change ownership for a directory in Windows as the Administrator account. The change makes the directory publically accessible, and that’s probably fine for a test database. After such a change, the Oracle user can find the external file even when parent directories aren’t navigable. Although, a production database on Windows would requires more skill at setting and restricting file permissions.

Linux and Unix directories require that the oracle user can navigate the tree from the mount point to the target physical directory. Also, you must designate the ownership of external files as the same as the Oracle Database user. Assuming a standard install of the Oracle Database 11g XE instance, you would issue the following shell command as the root user to change file ownership and access privileges:

# chown –R oracle:dba /u01/app/oracle/upload
# chmod –R 755 /u01/app/oracle/upload

After you create the virtual directory, you must grant privileges or a role to the user that defines the external table. While data and log files should be separated, this example assumes they co-exist in the same directory.

The following statement grants read privilege for the data file and write privileges for the log files to a CDB user. You should run this statement as the system user.

SQL> GRANT read, WRITE ON DIRECTORY upload TO c##importer;

or, like this in non-multitenant database or PDB user:

SQL> GRANT read, WRITE ON DIRECTORY upload TO importer;

The last preparation steps require a plain text file in the physical directory. Let’s create a CSV file of key Avenger characters, and name it the avenger.csv file.

The avenger.csv file holds the following values:

1,'Anthony','Stark','Iron Man'
2,'Thor','Odinson','God of Thunder'
3,'Steven','Rogers','Captain America'
4,'Bruce','Banner','Hulk'
5,'Clinton','Barton','Hawkeye'
6,'Natasha','Romanoff','Black Widow'

You create the external table after creating the virtual directory, granting read and write privileges on the virtual directory, and creating an external physical file. The syntax for the CREATE TABLE statement of an external table is very similar to the syntax of an ordinary table. The difference between the two types of tables is a clause. An internal table has a STORAGE clause, while an external table has an ORGANIZATION EXTERNAL clause.

The following creates the avenger table as an external table:

SQL> CREATE TABLE avenger
  2  ( avenger_id      NUMBER
  3  , first_name      VARCHAR2(20)
  4  , last_name       VARCHAR2(20)
  5  , character_name  VARCHAR2(20))
  6    ORGANIZATION EXTERNAL
  7    ( TYPE oracle_loader
  8      DEFAULT DIRECTORY upload
  9      ACCESS PARAMETERS
 10      ( RECORDS DELIMITED BY NEWLINE CHARACTERSET US7ASCII
 11        BADFILE     'UPLOAD':'avenger.bad'
 12        DISCARDFILE 'UPLOAD':'avenger.dis'
 13        LOGFILE     'UPLOAD':'avenger.log'
 14        FIELDS TERMINATED BY ','
 15        OPTIONALLY ENCLOSED BY "'"
 16        MISSING FIELD VALUES ARE NULL)
 17      LOCATION ('avenger.csv'))
 18  REJECT LIMIT UNLIMITED;

Lines 1 through 5 create the columns of the avenger table. Lines 6 through 17 contain the ORGANIZATION EXTERNAL clause. Line 7 designates the external table as managed by the Oracle SQL*Loader utility. Line 8 sets the default virtual directory. Lines 11 through 12 set the bad, discard, and log file location. The bad and discard files keep all that can’t be read. The log file keeps all rows read by a query against the avenger table.

You also have the option of making all reads automatic parallel. You simply add a PARALLEL clause, like this:

19  PARALLEL;

A simple query with SQL*Plus formatting lets us test whether the avenger table works. The query to display all columns of all rows is:

SQL> COLUMN first_name FORMAT A10
SQL> COLUMN last_name  FORMAT A10
SQL> COLUMN character_name FORMAT A15
SQL> SELECT * FROM avenger;

Yields the following formatted output:

AVENGER_ID FIRST_NAME LAST_NAME  CHARACTER_NAME
---------- ---------- ---------- ---------------
         1 Anthony    Stark      Iron Man
         2 Thor       Odinson    God of Thunder
         3 Steven     Rogers     Captain America
         4 Bruce      Banner     Hulk
         5 Clinton    Barton     Hawkeye
         6 Natasha    Romanoff   Black Widow
 
6 rows selected.

It’s possible to redefine the avenger table to use either relative or fixed positional columns. You change the ACCESS PARAMETERS clause on lines 9 through 16 to make this change.
The following ACCESS PARAMETERS clause runs across lines 9 through 19 and creates relative position definition:

  9      ACCESS PARAMETERS
 10      ( RECORDS DELIMITED BY NEWLINE CHARACTERSET US7ASCII
 11        BADFILE     'UPLOAD':'avenger.bad'
 12        DISCARDFILE 'UPLOAD':'avenger.dis'
 13        LOGFILE     'UPLOAD':'avenger.log'
 14        FIELDS
 15        MISSING FIELD VALUES ARE NULL
 16        ( avenger_id      CHAR(4)
 17        , first_name      CHAR(20)
 18        , last_name       CHAR(20)
 19        , character_name  CHAR(4)))

You can change from the relative position, to a fixed position by changing lines 16 through 19. The change for fixed length strings is:

 16        ( avenger_id      POSITION 1:4
 17        , first_name      POSITION 5:24
 18        , last_name       POSITION 25:44
 19        , character_name  POSITION 45:64))

Having worked with the Oracle SQL*Loader version of external tables, lets create one that uses Oracle Data Pump. Assuming we keep the same data structure, drop the avenger table, and create a catalog managed avenger_internal table.

This statement creates the avenger_internal table:

SQL> CREATE TABLE avenger_internal
  2  ( avenger_id      NUMBER
  3  , first_name      VARCHAR2(20)
  4  , last_name       VARCHAR2(20)
  5  , character_name  VARCHAR2(20));

To avoid writing six INSERT statements, you can write one INSERT statement with a query against the SQL*Loader avenger table. The syntax for that INSERT statement is:

SQL> INSERT INTO avenger_internal
  2  SELECT * FROM avenger;

With an internally managed table, you create an avenger_export table that uses Oracle Data Pump like this:

SQL> CREATE TABLE avenger_export
  2  ORGANIZATION EXTERNAL
  3  ( TYPE oracle_datapump
  4    DEFAULT DIRECTORY upload
  5    LOCATION ('avenger_export.dmp')) AS
  6  SELECT   avenger_id
  7  ,        first_name
  8  ,        last_name
  9  ,        character_name
 10  FROM     avenger_internal;

The CREATE TABLE statement exports data to the avenger_export.dmp file immediately. You must drop and recreate the avenger_export table to get a fresh extract of the avenger_internal table’s data. You must also remove the previous avenger_export.dmp file before you try to recreate the avenger_export table.

You raise the following error when you fail to remove the previous export file:

CREATE TABLE avenger_export
*
ERROR AT line 1:
ORA-29913: error IN executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-11012: FILE avenger_export.dmp IN /u01/... already EXISTS

This is a simple example with only four columns. You might think you can use the SELECT * as the SELECT-list of the query on lines 6 through 10. If you’re running Oracle Database 12c, you can use the shorter syntax, but if you’re running Oracle Database 11g you can’t. If you attempt it in an Oracle Database 11g instance, the CREATE TABLE statement returns the following error:
ERROR at line 6:

ORA-30656: COLUMN TYPE NOT supported ON external organized TABLE

You create an avenger_import table with another twist on this now familiar Oracle SQL syntax. The CREATE TABLE statement is:

SQL> CREATE TABLE avenger_import
  2  ( avenger_id      NUMBER
  3  , first_name      VARCHAR2(20)
  4  , last_name       VARCHAR2(20)
  5  , character_name  VARCHAR2(20))
  6    ORGANIZATION EXTERNAL
  7    ( TYPE oracle_datapump
  8      DEFAULT DIRECTORY up2load
  9      LOCATION ('avenger_export.dmp'));

Like the export process, the import process happens immediately when the CREATE TABLE statement runs. A query against the avenger_import table would show you the original six rows we started with in the plain text files.

This article has introduced Oracle external tables. It has shown you how to import plain text files with SQL*Loader. It has also shown you how to export files from tables.

This is an example of how you would upload data from a flat file, or Comma Separated Value (CSV) file. It’s important to note that in the file upload you are transferring information that doesn’t have surrogate key values by leveraing joins inside a MERGE statement.

Step #1 : Create a virtual directory

You can create a virtual directory without a physical directory but it won’t work when you try to access it. Therefore, you should create the physical directory first. Assuming you’ve created a /u01/app/oracle/upload file directory on the Windows platform, you can then create a virtual directory and grant permissions to the student user as the SYS privileged user.

The syntax for these steps is:

CREATE DIRECTORY upload AS '/u01/app/oracle/upload';
GRANT READ, WRITE ON DIRECTORY upload TO student;

Step #2 : Position your CSV file in the physical directory

After creating the virtual directory, copy the following contents into a file named kingdom_import.csv in the /u01/app/oracle/upload directory or folder. If you attempt to do this in Windows, you need to disable Windows UAC before performing this step.

Place the following in the kingdom_import.csv file. The trailing commas aren’t too meaningful in Oracle but they’re very helpful if you use the file in MySQL. A key element in creating this files requires that you avoid trailing line returns at the bottom of the file because they’re inserted as null values. There should be no lines after the last row of data.

'Narnia',77600,'Peter the Magnificent','20-MAR-1272','19-JUN-1292',
'Narnia',77600,'Edmund the Just','20-MAR-1272','19-JUN-1292',
'Narnia',77600,'Susan the Gentle','20-MAR-1272','19-JUN-1292',
'Narnia',77600,'Lucy the Valiant','20-MAR-1272','19-JUN-1292',
'Narnia',42100,'Peter the Magnificent','12-APR-1531','31-MAY-1531',
'Narnia',42100,'Edmund the Just','12-APR-1531','31-MAY-1531',
'Narnia',42100,'Susan the Gentle','12-APR-1531','31-MAY-1531',
'Narnia',42100,'Lucy the Valiant','12-APR-1531','31-MAY-1531',
'Camelot',15200,'King Arthur','10-MAR-0631','12-DEC-0686',
'Camelot',15200,'Sir Lionel','10-MAR-0631','12-DEC-0686',
'Camelot',15200,'Sir Bors','10-MAR-0631','12-DEC-0635',
'Camelot',15200,'Sir Bors','10-MAR-0640','12-DEC-0686',
'Camelot',15200,'Sir Galahad','10-MAR-0631','12-DEC-0686',
'Camelot',15200,'Sir Gawain','10-MAR-0631','12-DEC-0686',
'Camelot',15200,'Sir Tristram','10-MAR-0631','12-DEC-0686',
'Camelot',15200,'Sir Percival','10-MAR-0631','12-DEC-0686',
'Camelot',15200,'Sir Lancelot','30-SEP-0670','12-DEC-0682',

Step #3 : Reconnect as the student user

Disconnect and connect as the student user, or reconnect as the student user. The reconnect syntax that protects your password is:

CONNECT student@xe

Step #4 : Run the script that creates tables and sequences

Copy the following into a create_kingdom_upload.sql file within a directory of your choice. Then, run it as the student account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

-- Conditionally drop tables and sequences.
BEGIN
  FOR i IN (SELECT TABLE_NAME
            FROM   user_tables
            WHERE  TABLE_NAME IN ('KINGDOM','KNIGHT','KINGDOM_KNIGHT_IMPORT')) LOOP 
    EXECUTE IMMEDIATE 'DROP TABLE '||i.table_name||' CASCADE CONSTRAINTS';
  END LOOP;
  FOR i IN (SELECT sequence_name
            FROM   user_sequences
            WHERE  sequence_name IN ('KINGDOM_S1','KNIGHT_S1')) LOOP 
    EXECUTE IMMEDIATE 'DROP SEQUENCE '||i.sequence_name;
  END LOOP;
END;
/
 
-- Create normalized kingdom table.
CREATE TABLE kingdom
( kingdom_id    NUMBER
, kingdom_name  VARCHAR2(20)
, population    NUMBER);
 
-- Create a sequence for the kingdom table.
CREATE SEQUENCE kingdom_s1;
 
-- Create normalized knight table.
CREATE TABLE knight
( knight_id             NUMBER
, knight_name           VARCHAR2(24)
, kingdom_allegiance_id NUMBER
, allegiance_start_date DATE
, allegiance_end_date   DATE);
 
-- Create a sequence for the knight table.
CREATE SEQUENCE knight_s1;
 
-- Create external import table.
CREATE TABLE kingdom_knight_import
( kingdom_name          VARCHAR2(20)
, population            NUMBER
, knight_name           VARCHAR2(24)
, allegiance_start_date DATE
, allegiance_end_date   DATE)
  ORGANIZATION EXTERNAL
  ( TYPE oracle_loader
    DEFAULT DIRECTORY upload
    ACCESS PARAMETERS
    ( RECORDS DELIMITED BY NEWLINE CHARACTERSET US7ASCII
      BAFFLE      'UPLOAD':'kingdom_import.bad'
      DISCARDFILE 'UPLOAD':'kingdom_import.dis'
      LOGFILE     'UPLOAD':'kingdom_import.log'
      FIELDS TERMINATED BY ','
      OPTIONALLY ENCLOSED BY "'"
      MISSING FIELD VALUES ARE NULL )
    LOCATION ('kingdom_import.csv'))
REJECT LIMIT UNLIMITED;

Step #5 : Test your access to the external table

There a number of things that could go wrong with setting up an external table, such as file permissions. Before moving on to the balance of the steps, you should test what you’ve done. Run the following query from the student account to check whether or not you can access the kingdom_import.csv file.

1
2
3
4
5
6
7
8
9

COL kingdom_name FORMAT A8 HEADING "Kingdom|Name"
COL population   FORMAT 99999999 HEADING "Population"
COL knight_name  FORMAT A30 HEADING "Knight Name"
SELECT   kingdom_name
,        population
,        knight_name
,        TO_CHAR(allegiance_start_date,'DD-MON-YYYY') AS allegiance_start_date
,        TO_CHAR(allegiance_end_date,'DD-MON-YYYY') AS allegiance_end_date
FROM     kingdom_knight_import;

Step #6 : Create the upload procedure

Copy the following into a create_upload_procedure.sql file within a directory of your choice. Then, run it as the student account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

-- Create a procedure to wrap the transaction.
CREATE OR REPLACE PROCEDURE upload_kingdom IS 
BEGIN
  -- Set save point for an all or nothing transaction.
  SAVEPOINT starting_point;
 
  -- Insert or update the table, which makes this rerunnable when the file hasn't been updated.  
  MERGE INTO kingdom target
  USING (SELECT   DISTINCT
                  k.kingdom_id
         ,        kki.kingdom_name
         ,        kki.population
         FROM     kingdom_knight_import kki LEFT JOIN kingdom k
         ON       kki.kingdom_name = k.kingdom_name
         AND      kki.population = k.population) SOURCE
  ON (target.kingdom_id = SOURCE.kingdom_id)
  WHEN MATCHED THEN
  UPDATE SET kingdom_name = SOURCE.kingdom_name
  WHEN NOT MATCHED THEN
  INSERT VALUES
  ( kingdom_s1.nextval
  , SOURCE.kingdom_name
  , SOURCE.population);
 
  -- Insert or update the table, which makes this rerunnable when the file hasn't been updated.  
  MERGE INTO knight target
  USING (SELECT   kn.knight_id
         ,        kki.knight_name
         ,        k.kingdom_id
         ,        kki.allegiance_start_date AS start_date
         ,        kki.allegiance_end_date AS end_date
         FROM     kingdom_knight_import kki INNER JOIN kingdom k
         ON       kki.kingdom_name = k.kingdom_name
         AND      kki.population = k.population LEFT JOIN knight kn 
         ON       k.kingdom_id = kn.kingdom_allegiance_id
         AND      kki.knight_name = kn.knight_name
         AND      kki.allegiance_start_date = kn.allegiance_start_date
         AND      kki.allegiance_end_date = kn.allegiance_end_date) SOURCE
  ON (target.kingdom_allegiance_id = SOURCE.kingdom_id)
  WHEN MATCHED THEN
  UPDATE SET allegiance_start_date = SOURCE.start_date
  ,          allegiance_end_date = SOURCE.end_date
  WHEN NOT MATCHED THEN
  INSERT VALUES
  ( knight_s1.nextval
  , SOURCE.knight_name
  , SOURCE.kingdom_id
  , SOURCE.start_date
  , SOURCE.end_date);
 
  -- Save the changes.
  COMMIT;
 
EXCEPTION
  WHEN OTHERS THEN
    ROLLBACK TO starting_point;
    RETURN;
END;
/

Step #7 : Run the upload procedure

You can run the file by calling the stored procedure built by the script. The procedure ensures that records are inserted or updated into their respective tables.

EXECUTE upload_kingdom;

Step #8 : Test the results of the upload procedure

You can test whether or not it worked by running the following queries.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

-- Check the kingdom table.
SELECT * FROM kingdom;
 
-- Format Oracle output.
COLUMN knight_id             FORMAT 999 HEADING "Knight|ID #"
COLUMN knight_name           FORMAT A23 HEADING "Knight Name"
COLUMN kingdom_allegiance_id FORMAT 999 HEADING "Kingdom|Allegiance|ID #"
COLUMN allegiance_start_date FORMAT A11 HEADING "Allegiance|Start Date"
COLUMN allegiance_end_date   FORMAT A11 HEADING "Allegiance|End Date"
SET PAGESIZE 999
 
-- Check the knight table.
SELECT   knight_id
,        knight_name
,        kingdom_allegiance_id
,        TO_CHAR(allegiance_start_date,'DD-MON-YYYY') AS allegiance_start_date
,        TO_CHAR(allegiance_end_date,'DD-MON-YYYY') AS allegiance_end_date
FROM     knight;

It should display the following information:

KINGDOM_ID KINGDOM_NAME         POPULATION
---------- -------------------- ----------
         1 Narnia                    42100
         2 Narnia                    77600
         3 Camelot                   15200
 
                                  Kingdom
Knight                         Allegiance Allegiance  Allegiance
  ID # Knight Name                   ID # Start Date  End Date
------ ----------------------- ---------- ----------- -----------
     1 Peter the Magnificent            2 20-MAR-1272 19-JUN-1292
     2 Edmund the Just                  2 20-MAR-1272 19-JUN-1292
     3 Susan the Gentle                 2 20-MAR-1272 19-JUN-1292
     4 Lucy the Valiant                 2 20-MAR-1272 19-JUN-1292
     5 Peter the Magnificent            1 12-APR-1531 31-MAY-1531
     6 Edmund the Just                  1 12-APR-1531 31-MAY-1531
     7 Susan the Gentle                 1 12-APR-1531 31-MAY-1531
     8 Lucy the Valiant                 1 12-APR-1531 31-MAY-1531
     9 King Arthur                      3 10-MAR-0631 12-DEC-0686
    10 Sir Lionel                       3 10-MAR-0631 12-DEC-0686
    11 Sir Bors                         3 10-MAR-0631 12-DEC-0635
    12 Sir Bors                         3 10-MAR-0640 12-DEC-0686
    13 Sir Galahad                      3 10-MAR-0631 12-DEC-0686
    14 Sir Gawain                       3 10-MAR-0631 12-DEC-0686
    15 Sir Tristram                     3 10-MAR-0631 12-DEC-0686
    16 Sir Percival                     3 10-MAR-0631 12-DEC-0686
    17 Sir Lancelot                     3 30-SEP-0670 12-DEC-0682

You can rerun the procedure to check that it doesn’t alter any information, then you could add a new knight to test the insertion portion.

While trying to explain a student question about Oracle object types, it seemed necessary to show how to write a dependency tree. I did some poking around and found there wasn’t a convenient script at hand. So, I decided to write one.

This assumes the following Oracle object types, which don’t have any formal methods (methods are always provided by PL/SQL or Java language implementations):

CREATE OR REPLACE TYPE base_t AS OBJECT
( base_id  NUMBER ) NOT FINAL;
/
 
CREATE OR REPLACE TYPE person_t UNDER base_t
( first_name   VARCHAR2(20)
, middle_name  VARCHAR2(20)
, last_name    VARCHAR2(20)) NOT FINAL;
/
 
CREATE OR REPLACE TYPE driver_t UNDER person_t
( license VARCHAR2(20));
/

Here’s a query to show the hierarchy of object types and attributes by object-level in the hierarchy:

COL type_name  FORMAT A20  HEADING TYPE_NAME
COL attr_no    FORMAT 999  HEADING ATTR_NO
COL attr_name  FORMAT A20  HEADING ATTR_NAME
COL TYPE       FORMAT A12  HEADING TYPE
SELECT   DISTINCT
         LPAD(' ',2*(LEVEL-1)) || ut.type_name AS type_name
,        uta.attr_no
,        uta.attr_name
,        CASE
           WHEN uta.attr_type_name = 'NUMBER' THEN
             uta.attr_type_name
           WHEN uta.attr_type_name = 'VARCHAR2' THEN
             uta.attr_type_name || '(' || uta.LENGTH || ')'
         END AS TYPE
FROM     user_types ut
,        user_type_attrs uta
WHERE    ut.typecode = 'OBJECT'
AND      ut.type_name = uta.type_name
AND      uta.inherited = 'NO'
START
WITH     ut.type_name = 'BASE_T'
CONNECT
BY PRIOR ut.type_name = ut.supertype_name
ORDER BY uta.attr_no;

It should return the following:

TYPE_NAME	     ATTR_NO ATTR_NAME		  TYPE
-------------------- ------- -------------------- ------------
BASE_T			   1 BASE_ID		  NUMBER
  PERSON_T		   2 FIRST_NAME 	  VARCHAR2(20)
  PERSON_T		   3 MIDDLE_NAME	  VARCHAR2(20)
  PERSON_T		   4 LAST_NAME		  VARCHAR2(20)
    DRIVER_T		   5 LICENSE		  VARCHAR2(20)

As always, I hope this helps those looking to discover an Oracle object type hierarchy without examining each object type in turn.

It’s always a challenge when you want to build your own Oracle SQL Tools. I was asked how you could synchronize multiple cursors into a single source. The answer is quite simple, you write an Oracle object type to represent a record structure, an Oracle list of the record structure, and a stored function to return the list of the record structure.

For this example, you create the following table_struct object type and a table_list collection type:

/* Drop the types from most to least dependent. */
DROP TYPE table_list;
DROP TYPE table_struct;
 
/* Create the record type structure. */
CREATE OR REPLACE
  TYPE table_struct IS OBJECT
  ( table_name    VARCHAR2(30)
  , column_cnt    NUMBER
  , row_cnt       NUMBER );
/
 
/* Create the collection of a record type structure. */
CREATE OR REPLACE
  TYPE table_list IS TABLE OF table_struct;
/

The following listing function now reads all table names from the user_tables view. A subordinate cursor reads the user_tab_columns view for the number of columns in a table. A Native Dynamic SQL (NDS) cursor counts the number of rows in each tables found in the .

/* Create the listing function. */
CREATE OR REPLACE
FUNCTION listing RETURN table_list IS
 
  /* Variable list. */
  lv_column_cnt  NUMBER;
  lv_row_cnt     NUMBER;
 
  /* Declare a statement variable. */
  stmt  VARCHAR2(200);
 
  /* Declare a system reference cursor variable. */
  lv_refcursor  SYS_REFCURSOR;
  lv_table_cnt  NUMBER;
 
  /* Declare an output variable.  */
  lv_list  TABLE_LIST := table_list();
 
  /* Declare a table list cursor that excludes APEX tables. */
  CURSOR c IS
    SELECT table_name
    FROM   user_tables
    WHERE  table_name NOT IN
            ('DEPT','EMP','APEX$_ACL','APEX$_WS_WEBPG_SECTIONS','APEX$_WS_ROWS'
            ,'APEX$_WS_HISTORY','APEX$_WS_NOTES','APEX$_WS_LINKS'
            ,'APEX$_WS_TAGS','APEX$_WS_FILES','APEX$_WS_WEBPG_SECTION_HISTORY'
            ,'DEMO_USERS','DEMO_CUSTOMERS','DEMO_ORDERS','DEMO_PRODUCT_INFO'
            ,'DEMO_ORDER_ITEMS','DEMO_STATES');
 
  /* Declare a column count. */
  CURSOR cnt
  ( cv_table_name  VARCHAR2 ) IS
    SELECT   table_name
    ,        COUNT(column_id) AS cnt_columns
    FROM     user_tab_columns
    WHERE    table_name = cv_table_name
    GROUP BY table_name;
 
BEGIN
  /* Read through the data set of non-environment variables. */
  FOR i IN c LOOP
 
    /* Count the columns of a table. */
    FOR j IN cnt(i.table_name) LOOP
      lv_column_cnt := j.cnt_columns;
    END LOOP;
 
    /* Declare a statement. */
    stmt := 'SELECT COUNT(*) AS column_cnt FROM '||i.table_name;
 
    /* Open the cursor and write set to collection. */
    OPEN lv_refcursor FOR stmt;
    LOOP
      FETCH lv_refcursor INTO lv_table_cnt;
      EXIT WHEN lv_refcursor%NOTFOUND; 
      lv_list.EXTEND;
      lv_list(lv_list.COUNT) := table_struct(
                                    table_name => i.table_name
                                  , column_cnt => lv_column_cnt
                                  , row_cnt    => lv_table_cnt );
    END LOOP;
  END LOOP;
 
  RETURN lv_list;
END;
/

The following query pulls the processed data set as the function’s result:

COL table_name   FORMAT A20     HEADING "Table Name"
COL column_cnt   FORMAT 9,999  HEADING "Column #"
COL row_cnt      FORMAT 9,999  HEADING "Row #"
SELECT table_name
,      column_cnt
,      row_cnt
FROM   TABLE(listing);

It returns the following result set:

Table Name	     Column #  Row #
-------------------- -------- ------
SYSTEM_USER		   11	   5
COMMON_LOOKUP		   10	  49
MEMBER			    9	  10
CONTACT 		   10	  18
ADDRESS 		   10	  18
STREET_ADDRESS		    8	  28
TELEPHONE		   11	  18
RENTAL			    8  4,694
ITEM			   14	  93
RENTAL_ITEM		    9  4,703
PRICE			   11	 558
TRANSACTION		   12  4,694
CALENDAR		    9	 300
AIRPORT 		    9	   6
ACCOUNT_LIST		    8	 200
 
15 rows selected.

As always, I hope this helps those trying to work with the Oracle database.

What happens when you can’t get a PL/SQL Web Toolkit to work because it only prints to a web page? That’s more tedious because any dbms_output.put_line command you embed only prints to a SQL*Plus session. The answer is quite simple, you create a test case and test it inside a SQL*Plus environment.

Here’s a sample web page that fails to run successfully …

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

CREATE OR REPLACE
  PROCEDURE html_table_values
  ( name_array   OWA_UTIL.VC_ARR
  , value_array  OWA_UTIL.VC_ARR ) IS
  BEGIN
    /* Print debug to SQL*Plus session. */
    FOR i IN 1..name_array.COUNT LOOP
      DBMS_OUTPUT.put_line('Value ['||name_array(i)||'='||value_array(i)||']');
    END LOOP;
 
    /* Open HTML page with the PL/SQL toolkit. */
    htp.print('<!DOCTYPE html>');
    htp.htmlopen;
    htp.headopen;
    htp.htitle('Test');
    htp.headclose;
    htp.bodyopen;
    htp.line;
    htp.print('Test');
    htp.line;
    htp.bodyclose;
    htp.htmlclose;
END;
/

You can test the program with the following anonymous block as the SYSTEM user, which is equivalent to the following URL:

http://localhost:8080/db/html_table_values?begin=1004&end=1012

The following test program lets you work:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

DECLARE
  x  OWA_UTIL.VC_ARR;
  y  OWA_UTIL.VC_ARR;
BEGIN
  /* Insert first row element. */
  x(1) := 'begin';
  y(1) := '1004';
 
  /* Insert second row element. */
  x(2) := 'end';
  y(2) := '1012';
 
  /* Call the anonymous schema's web page. */
  anonymous.html_table_values(x,y);
END;
/

It should print:

Value [begin=1004]
Value [end=1012]

I hope this helps those looking for a solution.

My vacation from my blog is officially over. The question that I’m answering today is: How can you pass a set of non-sequential ID values to a function and return a result set? You can solve the problem by passing an ADT (Attribute Data Type) or UDT (User Defined Type) variable into a subquery of a cursor. The subquery leverages the TABLE function to translate the ADT or UDT into SQL result set, which is equivalent to a comma-delimited list of values.

You can also solve this problem with Native Dynamic SQL (NDS). However, the person who posed the question didn’t want to use NDS to build out a variable length list of comma-delimited numbers.

You need to create three object types for this example. They are:

• a list of numbers
• a record structure, declared as an object type without methods
• a list of the record structure

These are the SQL commands to create the required data types:

CREATE OR REPLACE
  TYPE list_ids IS TABLE OF NUMBER;
/

CREATE OR REPLACE
  TYPE item_struct IS OBJECT
  ( item_id       NUMBER
  , item_title    VARCHAR2(80)
  , release_date  DATE );
/

CREATE OR REPLACE
  TYPE item_struct_list IS TABLE OF item_struct;
/

Next, you create a nonsynchronous function. It takes a sparsely populated list of values that map to the surrogate key of the column, which is typically the table’s primary key column. It returns a collection of the item_struct object type. This type of function is an object-table function.

The code follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

CREATE OR REPLACE
  FUNCTION nonsynchronous 
  ( pv_list_ids  LIST_IDS ) RETURN item_struct_list IS
    /* Declare a record data structure list. */
    lv_struct_list  ITEM_STRUCT_LIST := item_struct_list();
 
    /* Declare a sparsely indexed list of film items. */
    CURSOR get_items
    ( cv_list_ids  LIST_IDS ) IS
      SELECT   item_id AS item_id
      ,        item_title
      ||       CASE
                 WHEN item_subtitle IS NOT NULL THEN
                   ': '|| item_subtitle
               END AS item_title
      ,        release_date AS release_date
      FROM     item
      WHERE    item_id IN (SELECT *
                           FROM   TABLE(cv_list_ids))
      ORDER BY item_id;
BEGIN
  /* Lood through the sparsely populated list of numbers. */
  FOR i IN get_items(pv_list_ids) LOOP
    lv_struct_list.EXTEND;
    lv_struct_list(lv_struct_list.COUNT) := item_struct( item_id      => i.item_id
                                                       , item_title   => i.item_title
                                                       , release_date => i.release_date );
  END LOOP;
 
  /* Return the record structure list. */
  RETURN lv_struct_list;
END;
/

The foregoing nonsynchronous function uses a nested query that transforms to a result set on lines 18 and 19. In the execution block of the program, it uses a call to the item_struct structure to capture and assign row values to an element of the lv_struct_list variable.

You can now test the nonsynchronous function with the following query:

COL item_id      FORMAT 9999  HEADING "Item|ID #"
COL item_title   FORMAT A40   HEADING "Item Title"
COL release_date FORMAT A11   HEADING "Release|Date"
SELECT   *
FROM     TABLE(nonsynchronous(list_ids(1002, 1013, 1007)));

The query returns the record set as an ordered list in the result set, like:

Item					       Release
 ID # Item Title			       Date
----- ---------------------------------------- -----------
 1002 Star Wars I: Phantom Menace	       04-MAY-99
 1007 RoboCop				       24-JUL-03
 1013 The DaVinci Code			       19-MAY-06

I hope this answers the question about how to get results sets with sparsely populated ID values.

Often, the biggest problem with regular expressions is that those who use them sometimes don’t use them correctly. A great example occurs in the Oracle Database with the REGEXP_LIKE function. For example, some developer use the following to validate whether a string is a number but it only validates whether the first character is a number.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

DECLARE
  lv_input  VARCHAR2(100);
BEGIN
  /* Assign input value. */
  lv_input := '&input';
 
  /* Check for numeric string. */
  IF REGEXP_LIKE(lv_input,'[[:digit:]]') THEN
    dbms_output.put_line('It''s a number.');
  ELSE
    dbms_output.put_line('It''s a string.');
  END IF;
END;
/

When they test numbers it appears to works, it even appears to work when the test string start with number, but it fails with any string that starts with a character. That’s because the REGEXP_LIKE function on line 8 only checks the first character, but the following checks all the characters in the string.

8

  IF REGEXP_LIKE(lv_inputs(i),'[[:digit:]]{'||LENGTH(lv_inputs(i))||'}') THEN

You can also fix it with the following non-Posix solution:

8

  IF REGEXP_LIKE(lv_input,'[[0-9]]') THEN

You can add a collection to the program and use it to test single-digit, double-digit, and string with a leading integer. Save the program as test.sql and you can test three conditions with one call.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

DECLARE
  /* Declare the local collection type. */
  TYPE inputs IS TABLE OF VARCHAR2(100);
 
  /* Declare a local variable of the collection type. */
  lv_inputs  INPUTS;
BEGIN
  /* Assign the inputs to the collection variable. */
  lv_inputs := inputs('&1','&2','&3');
 
  /* Read through the collection and print whether it's an number or string. */
  FOR i IN 1..lv_inputs.COUNT LOOP
    IF REGEXP_LIKE(lv_inputs(i),'[[:digit:]]{'||LENGTH(lv_inputs(i))||',}') THEN
      dbms_output.put_line('It''s a number.');
    ELSE
      dbms_output.put_line('It''s a string.');
    END IF;
  END LOOP;
END;
/

You can run the test.sql program like this:

SQL> @test.sql 1 12 1a

It prints:

It's a number.
It's a number.
It's a string.

As always, I hope this helps those looking for a solution.

Using Oracle’s Resource Manager requires you to understand the IO dynamics. The first step requires you to run the CALIBRATE_IO procedure from the DBMS_RESOURCE_MANAGER package.

Oracle provides some great examples about how to use the CALIBRATE_IO procedure of the DBMS_RESOURCE_MANAGER package in the Oracle Database Database PL/SQL Packages and Types Reference. The CALIBRATE_IO procedure returns the best answer when you provide a valid number of files, which you can capture by querying the V$ASM_DISK view.

The following code queries the view and assigns the value to a session level variable:

CLEAR BREAKS
CLEAR COLUMNS
CLEAR COMPUTES
 
VARIABLE files NUMBER
 
BEGIN
  SELECT COUNT(DISTINCT name) disks
  INTO :files
  FROM v$asm_disk;
END;
/

When you have the number of files, you can calibrate the IO with the following anonymous block. The query should always work but just in case the NVL function on line 9 assigns the default number of files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

DECLARE
  lv_num_physical_disks BINARY_INTEGER; — v$asm_disk
  lv_max_latency BINARY_INTEGER := 10;
  lv_max_iops BINARY_INTEGER;
  lv_max_mbps BINARY_INTEGER;
  lv_actual_latency BINARY_INTEGER;
BEGIN
  /* Assign actual files to anonymous block variable. */
  lv_num_physical_disks := NVL(:files,2);
 
  /* Run the calibrate_io procedure. */
  DBMS_RESOURCE_MANAGER.CALIBRATE_IO(
      num_physical_disks => lv_num_physical_disks
    , max_latency => lv_max_latency
    , max_iops => lv_max_iops
    , max_mbps => lv_max_mbps
    , actual_latency => lv_actual_latency);
END;
/

You can query the results like this:

SELECT max_iops
,      max_mbps
,      max_pmbps
,      latency
,      num_physical_disks
FROM   dba_rsrc_io_calibrate;

It should show results like these:

MAX_IOPS MAX_MBPS MAX_PMBPS LATENCY NUM_PHYSICAL_DISKS
-------- -------- --------- ------- ------------------
    8894      443       294       9                 18

Hope this helps those using the CALIBRATE_IO procedure of the DBMS_RESOURCE_MANAGER package.

Using an Oracle object type’s instance in a query is a powerful capability. Unfortunately, Oracle’s SQL syntax doesn’t make it immediately obvious how to do it. Most get far enough to put it in a runtime view (a subquery in the FROM clause), but then they get errors like this:

SELECT	 instance.get_type()
         *
ERROR AT line 4:
ORA-00904: "INSTANCE"."GET_TYPE": invalid identifier

The problem is how Oracle treats runtime views, which appears to me as a casting error. Somewhat like the ORDER BY clause irregularity that I noted in July, the trick is complete versus incomplete syntax. The following query fails and generates the foregoing error:

1
2
3
4

SELECT instance.get_type() AS object_type
,      instance.to_string() AS object_content
FROM  (SELECT dependent()AS instance
       FROM   dual);

If you add a table alias, or name, to the runtime view on line 4, it works fine:

1
2
3
4

SELECT cte.instance.get_type() AS object_type
,      cte.instance.to_string() AS object_content
FROM  (SELECT dependent() AS instance
       FROM   dual) cte;

That is the trick. You use an alias for the query, which assigns the alias like a table reference. The reference lets you access instance methods in the scope of a query. Different columns in the query’s SELECT-list may return different results from different methods from the same instance of the object type.

You can also raise an exception if you forget the open and close parentheses for a method call to a UDT, which differs from how Oracle treats no argument functions and procedures. That type of error would look like this:

SELECT cte.instance.get_type AS object_type
       *
ERROR AT line 1:
ORA-00904: : invalid identifier

It is an invalid identifier because there’s no public variable get_type, and a method is only found by using the parenthesis and a list of parameters where they’re required.

The object source code is visible by clicking on the expandable label below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

DROP TYPE dependent_t;
DROP TYPE base_t;
 
SET SERVEROUTPUT ON SIZE UNLIMITED
 
-- Create an object.
CREATE OR REPLACE TYPE base_t IS OBJECT
( TYPE       VARCHAR2(20)
, CONSTRUCTOR FUNCTION base_t RETURN SELF AS RESULT
, MEMBER FUNCTION get_type RETURN VARCHAR2
, MEMBER FUNCTION to_string RETURN VARCHAR2) 
INSTANTIABLE NOT FINAL;
/
 
-- Create an object body.
CREATE OR REPLACE TYPE BODY base_t IS
CONSTRUCTOR FUNCTION base_t RETURN SELF AS RESULT IS
BEGIN
  RETURN;
END base_t;
MEMBER FUNCTION get_type RETURN VARCHAR2 IS
BEGIN
  RETURN self.TYPE;
END;
MEMBER FUNCTION to_string RETURN VARCHAR2 IS
BEGIN
  RETURN self.TYPE;
END to_string;
END;
/
 
-- Create a subtype.
CREATE OR REPLACE TYPE dependent UNDER base_t
( child  VARCHAR2(40)
, CONSTRUCTOR FUNCTION dependent RETURN SELF AS RESULT
, CONSTRUCTOR FUNCTION dependent
  ( child  VARCHAR2 ) RETURN SELF AS RESULT
, OVERRIDING MEMBER FUNCTION get_type RETURN VARCHAR2
, OVERRIDING MEMBER FUNCTION to_string RETURN VARCHAR2) 
INSTANTIABLE NOT FINAL;
/
 
-- Create a subtype body.
CREATE OR REPLACE TYPE BODY dependent IS
  CONSTRUCTOR FUNCTION dependent RETURN SELF AS RESULT IS
  BEGIN
    /* Assign subtype name to type. */
    self.TYPE := 'DEPENDENT';
    RETURN;
  END dependent;
  CONSTRUCTOR FUNCTION dependent
  ( child  VARCHAR2 ) RETURN SELF AS RESULT IS
    lv_dependent DEPENDENT := dependent();
  BEGIN
    /* Assign default constructor to self instance. */
    self := lv_dependent;
    /* Assign parameters to object instance. */
    self.TYPE  := 'DEPENDENT';
    self.child := child;
    RETURN;
  END dependent;
  OVERRIDING MEMBER FUNCTION get_type RETURN VARCHAR2 IS
  BEGIN
    RETURN self.TYPE;
  END;
  OVERRIDING MEMBER FUNCTION to_string RETURN VARCHAR2 IS
  BEGIN
    RETURN self.TYPE;
  END to_string;
END;
/

As always, I hope this helps those solving problems.

Back in January 2014, I wrote a script to cleanup an Oracle student schema. It worked well until I started using APEX 4 in my student schema. You create the following 75 objects when you create an APEX 4 schema.

OBJECT TYPE    TOTAL
------------ -------
TABLE		  17
INDEX		  28
SEQUENCE	   5
TRIGGER 	  14
LOB		   9
FUNCTION	   2

Here’s the modified script that ignores the objects created automatically by Oracle APEX when you create a student workspace:

BEGIN
  FOR i IN (SELECT    object_name
            ,         object_type
            ,         last_ddl_time
            FROM      user_objects
            WHERE     object_name NOT IN
                       ('APEX$_WS_WEBPG_SECTION_HISTORY','APEX$_WS_WEBPG_SECTIONS_T1'
                       ,'APEX$_WS_WEBPG_SECTIONS_PK','APEX$_WS_WEBPG_SECTIONS'
                       ,'APEX$_WS_WEBPG_SECHIST_IDX1','APEX$_WS_TAGS_T1'
                       ,'APEX$_WS_TAGS_PK','APEX$_WS_TAGS_IDX2','APEX$_WS_TAGS_IDX1'
                       ,'APEX$_WS_TAGS','APEX$_WS_ROWS_T1','APEX$_WS_ROWS_PK'
                       ,'APEX$_WS_ROWS_IDX','APEX$_WS_ROWS','APEX$_WS_NOTES_T1'
                       ,'APEX$_WS_NOTES_PK','APEX$_WS_NOTES_IDX2','APEX$_WS_NOTES_IDX1'
                       ,'APEX$_WS_NOTES','APEX$_WS_LINKS_T1','APEX$_WS_LINKS_PK'
                       ,'APEX$_WS_LINKS_IDX2','APEX$_WS_LINKS_IDX1','APEX$_WS_LINKS'
                       ,'APEX$_WS_HISTORY_IDX','APEX$_WS_HISTORY','APEX$_WS_FILES_T1'
                       ,'APEX$_WS_FILES_PK','APEX$_WS_FILES_IDX2','APEX$_WS_FILES_IDX1'
                       ,'APEX$_WS_FILES','APEX$_ACL_T1','APEX$_ACL_PK','APEX$_ACL_IDX1'
                       ,'APEX$_ACL','CUSTOM_AUTH','CUSTOM_HASH','DEPT','EMP'
                       ,'UPDATE_ORDER_TOTAL')
            AND NOT ((object_name LIKE 'DEMO%' OR
                      object_name LIKE 'INSERT_DEMO%' OR
                      object_name LIKE 'BI_DEMO%') AND
                      object_type IN ('TABLE','INDEX','SEQUENCE','TRIGGER'))
            AND NOT (object_name LIKE 'SYS_LOB%' AND object_type = 'LOB')
            AND NOT (object_name LIKE 'SYS_C%' AND object_type = 'INDEX')
            ORDER BY object_type DESC) LOOP
 
    /* Drop types in descending order. */
    IF i.object_type = 'TYPE' THEN
 
      /* Drop type and force operation because dependencies may exist. Oracle 12c
         also fails to remove object types with dependents in pluggable databases
         (at least in release 12.1). Type evolution works in container database
         schemas. */
      EXECUTE IMMEDIATE 'DROP '||i.object_type||' '||i.object_name||' FORCE';
 
    /* Drop table tables in descending order. */
    ELSIF i.object_type = 'TABLE' THEN
 
      /* Drop table with cascading constraints to ensure foreign key constraints
         don't prevent the action. */
      EXECUTE IMMEDIATE 'DROP '||i.object_type||' '||i.object_name||' CASCADE CONSTRAINTS PURGE';
 
      /* Oracle 12c ONLY: Purge the recyclebin to dispose of system-generated
         sequence values because dropping the table doesn't automatically 
         remove them from the active session.
         CRITICAL: Remark out the following when working in Oracle Database 11g. */
      EXECUTE IMMEDIATE 'PURGE RECYCLEBIN';
 
    ELSIF i.object_type = 'LOB' OR i.object_type = 'INDEX' THEN
 
      /* A system generated LOB column or INDEX will cause a failure in a
         generic drop of a table because it is listed in the cursor but removed
         by the drop of its table. This NULL block ensures there is no attempt
         to drop an implicit LOB data type or index because the dropping the
         table takes care of it. */
      NULL;
 
    ELSE
 
      dbms_output.put_line('DROP '||i.object_type||' '||i.object_name||';');
      /* Drop any other objects, like sequences, functions, procedures, and packages. */
      EXECUTE IMMEDIATE 'DROP '||i.object_type||' '||i.object_name;
 
    END IF;
  END LOOP;
END;
/

As always, I hope this helps others.