Archive for the ‘Java’ Category
SQL Developer JVM Fix
It’s amazing the moving parts in Oracle Database 11g, and the ones that I don’t use after installation for a while often reach out to teach me something new. That’s what happened when I went to launch SQL Developer on my Windows 7 Professional 64-bit installation.
I got the message Unable to find a Java Virtual Machine, as shown in the image below. It really means unable to find a 32-bit Java 5 or Java 6 (1.6.0_4+) SDK home. Although the installation appeared fine, it pointed the SQL Developer configuration file to an incompatible 64-bit Java 7 SDK.
You fix this error by following these steps:
- Open the
sqldeveloper.conf
file and check theSetJavaHome
parameter value. You find thesqldeveloper.conf
file in the following directory:
%ORACLE_HOME\sqldeveloper\sqldeveloper\bin |
- Install the 32-bit Java 6 (1.6.0_4+) SDK on the operating system if not installed already. You can check whether it’s installed by looking for it in the Program Files (x86) folder.
- Change the value of the
SetJavaHome
parameter to point to the new 32-bit Java 6 home directory (or folder). The following change to line 18 in thesqldeveloper.conf
file should fix it on your installation (provided that’s your version of the JVM).
18 | SetJavaHome C:\Program Files (x86)\Java\jdk1.6.0_34 |
Hope this helps those who run into the same issue.
Oracle and Java Tutorial
I’m posting this because of a question raised against this older post on how to configure the %CLASSPATH%
to find the ojdbc6.jar
file. This is the lab file I use in my Database 1 class to expose students to the moving parts of writing Java programs against the Oracle database. That’s why I choose to use a CLOB
data type, which requires Oracle’s DBMS_LOB
package and wrapping stored procedures.
If you want the same content for MySQL, here’s the link. The full program in either blog entry is available by clicking on the fold/unfold Java Source Code Program widget at the bottom of the respective posts.
This demonstrates how to create an Java infrastructure for reading and writing large text files to an Oracle database. The example provides:
- A
FileIO.jar
library that lets you enter Oracle connection parameters through aJOptionPane
, and a customizedJFileChooser
to filter and read source files from the file system. - A
ojdbc6.jar
file, which is Oracle’s library for JDBC communication with the Oracle Databases.
The steps to compiling and testing this code are qualified below:
- Download and install the Java Software Development Kit (JSDK) for Java 6.
- Create a
C:\JavaTest
folder on Windows, or a/JavaTest
directory from some mount point of your choice. - Download and position the
ojdbc6.jar
andFileIO.jar
files in theJavaTest
directory. - Create a batch file to source your environment path (%PATH% on Windows and $PATH on Linux or Mac OS X) and the two Java Archive (JAR) files. A sample batch file is noted below:
set PATH=C:\Program Files\Java\jdk1.6.0_07\bin;%PATH% set CLASSPATH=C:\JavaDev\Java6\ojdbc6.jar;C:\JavaDev\Java6\FileIO.jar;. |
You can run this file by simply typing the files first name. On Linux or Mac OS X, you first need to grant it privileges with the chmod
command as 755
.
- Copy the
WriteReadCLOB.java
code from the bottom of this posting and also put it into theJavaTest
directory. - Compile the
WriteReadCLOB.java
source code with thejavac
utility, as shown below:
javac WriteReadCLOB.java |
After you compile it, you should run it as follows:
java WriteReadCLOB |
- Before running the code, you’ll need to seed (
INSERT
) a row that meets the desired hard coded criteria. It requires anITEM_TITLE
value of'The Lord of the Rings - Fellowship of the Ring'
and anITEM_SUBTITLE
of'Widescreen Edition'
in theITEM
table. - When it runs, you’ll see the following tabbed
JOptionPane
.
You need to enter the following values before clicking the OK button:
- Host: The
hostname
of your machine. - Port: The
port
that the Oracle Listener is running on (the default value is1521
). - Database: The Oracle TNS Alias, which is
orcl
for the full database sample database orxe
for the Oracle Database 10g Express Edition. - UserID: The
user
(schema) name where you’ve created anITEM
table. - Password: The
password
for the user’s account.
In the JFileChooser
, select a file to upload to the database.
You should see what you uploaded displayed in a JFrame
.
Java Source Code Program ↓
The drop down unfolds the WriteReadCLOB.java
source code.
The following program has dependencies on the FileIO.jar file. You need to download it and put it in your $CLASSPATH
for Linux or Mac OS X or %CLASSPATH%
for Windows.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 | // -------------------------------------------------------------------- // WriteReadCLOB.java // by Michael McLaughlin // // This code demonstrates reading a large text file and displaying // the text stream in a JLabel in a JFrame. // // The UPDATE and SELECT statements have dependencies on the // create_store.sql script. // -------------------------------------------------------------------- // Java Application class imports. import java.awt.Dimension; import java.awt.Font; import java.awt.GridLayout; import java.io.Reader; import javax.swing.JFrame; import javax.swing.JLabel; import javax.swing.JOptionPane; import javax.swing.JPanel; import javax.swing.JScrollPane; import javax.swing.JTextArea; // Generic JDBC imports. import java.sql.CallableStatement; import java.sql.Clob; import java.sql.Connection; import java.sql.DatabaseMetaData; import java.sql.ResultSet; import java.sql.ResultSetMetaData; import java.sql.SQLException; import java.sql.Statement; // Oracle JDBC import. import oracle.jdbc.driver.OracleDriver; import oracle.jdbc.pool.OracleDataSource; // Include book libraries (available at publisher website). import plsql.jdbc.DataConnectionPane; import plsql.fileio.FileIO; // -------------------------------------------------------------------/ public class WriteReadCLOB extends JFrame { // Define database connections. private String host; private String port; private String dbname; private String userid; private String passwd; // Define data connection pane. private DataConnectionPane message = new DataConnectionPane(); // Construct the class. public WriteReadCLOB (String s) { super(s); // Get database connection values or exit. if (JOptionPane.showConfirmDialog(this,message ,"Set Oracle Connection String Values" ,JOptionPane.OK_CANCEL_OPTION) == 0) { // Set class connection variables. host = message.getHost(); port = message.getPort(); dbname = message.getDatabase(); userid = message.getUserID(); passwd = message.getPassword(); // Print connection to console (debugging tool). message.getConnection(); // Create a JPanel for data display. ManageCLOB panel = new ManageCLOB(); // Configure the JPanel. panel.setOpaque(true); setContentPane(panel); // Configure the JFrame. setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE); setLocation(100,100); pack(); setVisible(true); } else System.exit(1); } // -------------------------------------------------------------------/ private class ManageCLOB extends JPanel { // Define display variables. private String clobText; private JScrollPane scrollPane; private JTextArea textArea; // -----------------------------------------------------------------/ public ManageCLOB () { // Set layout manager. super(new GridLayout(1,0)); // Assign file read to String. clobText = FileIO.openFile(FileIO.findFile(this)); // Insert record before querying it. if (clobText.length() > 0) { if (insertClob(host,port,dbname,userid,passwd,clobText)) clobText = getQuery(host,port,dbname,userid,passwd); else clobText = null; } else System.exit(2); // Construct text area and format it. textArea = new JTextArea(clobText); textArea.setEditable(false); textArea.setFont(new Font(Font.SANS_SERIF,Font.PLAIN,14)); textArea.setLineWrap(true); textArea.setRows(10); textArea.setSize(400,100); textArea.setWrapStyleWord(true); // Put the image in container, and add label to panel. scrollPane = new JScrollPane(textArea); add(scrollPane); } // ---------------------------------------------------------------/ private Boolean insertClob(String host,String port,String dbname ,String user,String pswd,String fileString) { try { // Set the Pooled Connection Source OracleDataSource ods = new OracleDataSource(); String url = "jdbc:oracle:thin:@//"+host+":"+port+"/"+dbname; ods.setURL(url); ods.setUser(userid); ods.setPassword(passwd); // Define connection. Connection conn = ods.getConnection(); // Create statement. CallableStatement stmt = conn.prepareCall("UPDATE item "+ "SET item_desc = ? "+ "WHERE item_title = "+ "'The Lord of the Rings - Fellowship of the Ring'"+ "AND item_subtitle = 'Widescreen Edition'"); // Set string into statement. stmt.setString(1,fileString); // Execute query. if (stmt.execute()) conn.commit(); // Close resources. stmt.close(); conn.close(); // Return CLOB as a String data type. return true; } // End of connection try-block. catch (SQLException e) { if (e.getSQLState() == null) { System.out.println( new SQLException("Oracle Thin Client Net8 Connection Error.", "ORA-" + e.getErrorCode() + ": Incorrect Net8 thin client arguments:\n\n" + " host name [" + host + "]\n" + " port number [" + port + "]\n" + " database name [" + dbname + "]\n" , e.getErrorCode()).getSQLState()); // Return an empty String on error. return false; } else { System.out.println(e.getMessage()); // Return an empty String on error. return false; }}} // -----------------------------------------------------------------/ private String getQuery(String host,String port,String dbname ,String user,String pswd) { // Define method variables. char[] buffer; int count = 0; int length = 0; String data = null; String[] type; StringBuffer sb; try { // Set the Pooled Connection Source OracleDataSource ods = new OracleDataSource(); String url = "jdbc:oracle:thin:@//"+host+":"+port+"/"+dbname; ods.setURL(url); ods.setUser(userid); ods.setPassword(passwd); // Define connection. Connection conn = ods.getConnection(); // Define metadata object. DatabaseMetaData dmd = conn.getMetaData(); // Create statement. Statement stmt = conn.createStatement(); // Execute query. ResultSet rset = stmt.executeQuery( "SELECT item_desc " + "FROM item " + "WHERE item_title = " + "'The Lord of the Rings - Fellowship of the Ring'"+ "AND item_subtitle = 'Widescreen Edition'"); // Get the query metadata, size array and assign column values. ResultSetMetaData rsmd = rset.getMetaData(); type = new String[rsmd.getColumnCount()]; for (int col = 0;col < rsmd.getColumnCount();col++) type[col] = rsmd.getColumnTypeName(col + 1); // Read rows and only CLOB data type columns. while (rset.next()) { for (int col = 0;col < rsmd.getColumnCount();col++) { if (type[col] == "CLOB") { // Assign result set to CLOB variable. Clob clob = rset.getClob(col + 1); // Check that it is not null and read the character stream. if (clob != null) { Reader is = clob.getCharacterStream(); // Initialize local variables. sb = new StringBuffer(); length = (int) clob.length(); // Check CLOB is not empty. if (length > 0) { // Initialize control structures to read stream. buffer = new char[length]; count = 0; // Read stream and append to StringBuffer. try { while ((count = is.read(buffer)) != -1) sb.append(buffer); // Assign StringBuffer to String. data = new String(sb); } catch (Exception e) {} } else data = (String) null; } else data = (String) null; } else { data = (String) rset.getObject(col + 1); }}} // Close resources. rset.close(); stmt.close(); conn.close(); // Return CLOB as a String data type. return data; } catch (SQLException e) { if (e.getSQLState() == null) { System.out.println( new SQLException("Oracle Thin Client Net8 Connection Error.", "ORA-" + e.getErrorCode() + ": Incorrect Net8 thin client arguments:\n\n" + " host name [" + host + "]\n" + " port number [" + port + "]\n" + " database name [" + dbname + "]\n" , e.getErrorCode()).getSQLState()); // Return an empty String on error. return data; } else { System.out.println(e.getMessage()); return data; }} finally { if (data == null) System.exit(1); }}} // -----------------------------------------------------------------/ public static void main(String[] args) { // Define window. WriteReadCLOB frame = new WriteReadCLOB("Write & Read CLOB Text"); }} |
Delay or synchronize it?
A couple students in one of my classes ran into a problem when competing Java threads tried to insert new rows in a table. They raised an error when they tried the DELAY
keyword to avoid the race (collision) condition in an INSERT
statement. It was simple to explain to them that the DELAY
keyword doesn’t work with an InnoDB table. Any attempt throws the following error:
ERROR 1616 (HY000): DELAYED OPTION NOT supported FOR TABLE 'message' |
Important Update: INSERT DELAYED
is gone in MySQL 5.6.6 (announcement) and the whole issue comes down to synchronizing threads (some dislike the solution) or using the ON DUPLICATE KEY
clause.
They retested their Java application after redefining the target table using the MyISAM engine. They found it worked but that’s a bad fix in Java (a brief Java/MySQL tutorial post). They really needed to synchronize the Java thread (line #22), leave out the DELAY
keyword, and manage the table with the InnoDB engine. Here’s the modified Java code (by the way, they named their project VulcanTech if you’re wondering about the packages in the import statement):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | package vulcantech.vth.server.commands; import java.io.IOException; import java.net.Socket; import vulcantech.vth.server.combeans.MessageBean; public class MessageHandler implements Handler { @Override public void handleIt(Object... args) { MessageBean message = (MessageBean) args[0]; Socket sock = (Socket) args[1]; DatabaseConnection dbconnection = new DatabaseConnection(); String update = new String("INSERT INTO message(message_timestamp, sender, recipient, message, checked) VALUES(\'" + message.getTimeStamp() + "\', \'" + message.getSender() + "\', \'" + message.getRecipient() + "\', \'" + message.getMessage() + "\', b\'0\')"); synchronized (this) { dbconnection.executeUpdate(update); } try { sock.getOutputStream().write(1); } catch (IOException e) { e.printStackTrace(); } dbconnection.close(); } } |
Hope this helps those who encounter race conditions against MySQL when you’re writing Enterprise Java Beans (EJBs).
Zend CE has a Worm?
After updating the AVGFree virus definitions, I was surprised to find that Zend CE (Community Edition) 4.0.6 had a reported worm in the JavaServer.exe
file. There was greater surprise when Zend CE 5.3.9 (5.6.0-SP1) also had the same reported worm.
This is the message identifying the worm (click on it to see a full size image), and you can read about this particular worm on the Mcafee site or the AVG threat labs site:
If you check AVGFree’s page, the actual infection isn’t a stated variant, but it appears the heuristics are a bit aggressive.
File Name: C:\Program Files (x86)\Zend\ZendServer\bin\JavaServer.exe Infection: Win32/DH.FF860061{00000000-00080000-00000000} |
Unless you have the full version of AVGFree or another security program to try and fix the file, you can only quarantine the file. Quarantine or removal disables Zend CE from working. It begs the question: “How does Zend release a core file with a worm?” or “Is AVGFree reporting a false positive?”
Update: AVGFree was providing a false positive. In addition to the checks by Zeev at Zend, I created a new test instance with Norton 360 and it likewise found no virus/worm in Zend’s JavaServer.exe
file. Hopefully the post will prevent others from spending more than a Google search to sort it out.
Since I use AVGFree on all my Windows 7 VM test instances, it seemed logical to illustrate how to work around this current false positive and annoying quarantining of the core JavaServer.exe
file from the Zend Server. There are two sets of tasks, the first requires removing the file from quarantine and the second eliminates future scans from quarantining the file again.
Remove the file from the Virus Vault
- Launch AVGFree and navigate to the History menu option and choose the Virus Vault option, as shown below.
- Click the Virus Vault option in the list of the History, which displays the following screen. Click the Infection row and then click the Restore button to remove the file from the virus vault.
- A confirmation dialog opens and you click the Yes button to proceed.
- The Infection row is gone When you’re returned to the History dialog. Click the Close button to complete this task.
Exclude the file from future scans
- Select the Tools menu option and choose the Advanced settings … option, as shown below.
- Click the Excluded files option in the list of the History, which displays the following screen. Click the Add button to select the file for exclusion. Click the Apply button to effect the change and the OK button to complete the change.
All I can say, one the AVGFree false positive was annoying and it’s dark at 3 a.m. and light the next day. 😉
Thanks to those who knew or surmised it was AVGFree’s heuristics and took the time to add a comment.
Oracle CSV Imports
The first step in creating an effective import plan for comma-separated value (CSV) files is recognizing your options in a database. There are several options in an Oracle database. You can read the file with Java, C/C++, C#, PL/SQL (through the UTL_FILE package), PHP, Perl, or any other C-callable programming language; or you can use SQL*Loader as a standalone utility or through externally managed tables (known as external tables). The most convenient and non-programming solution is using external tables.
Adopting external tables as your import solution should drive you to consider how to manage the security surrounding this type of methodology. Host hardening is a critical security step because it shuts down most, hopefully all, unauthorized use of the operating system where the database and external files reside. Next, you need to manage the access to the external tables and ensure that exposure of business sensitive information in CSV files is minimized.
This post explains how to manage access and police (cleanup external files) once they’re read into the database. It assumes you have root-level permissions to the operating system and database. The SYS and SYSTEM accounts have the equivalent of root permissions for database configuration. The rule of thumb with these accounts is simple, manage as much as possible with the SYSTEM account before you use the SYS account.
Setting up the Import File System
While you can do all the setup of virtual directories in Oracle regardless of whether you’ve set them up in the operating system, it’s a good idea to set them up in the OS first. The example is using a Windows 7 OS, so you’ll need to change the directories when working in Linux or Unix. Here are the directories:
C:\Imports\ImportFiles C:\Imports\ImportLogs |
You may take note that there are two directories. That’s because you don’t want to grant write privileges to the Oracle virtual directory where you put the files. You can grant read-only privileges to the virtual directory and read-write privileges to the log directory.
Setting up the Import User/Schema
This step lets you create an IMPORT
user/schema in the Oracle database. You need to connect as the SYSTEM user to perform these steps (or another authorized DBA account with adequate privileges):
CREATE USER import IDENTIFIED BY import DEFAULT TABLESPACE users QUOTA 1000M ON users TEMPORARY TABLESPACE temp; |
After creating the user, grant the privileges like this as the SYSTEM user:
GRANT CREATE CLUSTER, CREATE INDEXTYPE, CREATE OPERATOR , CREATE PROCEDURE, CREATE SEQUENCE, CREATE SESSION , CREATE SYNONYM, CREATE TABLE, CREATE TRIGGER , CREATE TYPE, CREATE VIEW TO import; |
Setting up Virtual Directories
A virtual directory in Oracle acts maps an internal database directory name (known as a virtual directory) to a physical directory of the operating system. You create two virtual directories in this example, one holds read-only permissions to the directory where you’re putting the data file, and the other holds read-write permissions to the directory where you’re writing any log files from the external file process.
Log files are generated from this process when you query the data from the external file. Any error in the files conformity is written to a log file.
CREATE DIRECTORY upload_files AS 'C:\Imports\ImportFiles'; CREATE DIRECTORY upload_logs AS 'C:\Imports\ImportLogs'; |
After creating the virtual directories in the database, you must grant appropriate access to the user account that will access the data. This grants those permissions to the IMPORT user:
GRANT READ ON DIRECTORY upload_files TO import; GRANT READ, WRITE ON DIRECTORY upload_logs TO import; |
Setting up an External Table
An external table references both the UPLOAD_FILES and UPLOAD_LOGS virtual directories, and the virtual directories must map to physical directories that allow read and write privileges to the Oracle user. Here’s the external table for this example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | CREATE TABLE item_import_ext_table ( asin_number VARCHAR2(10) , item_type VARCHAR2(15) , item_title VARCHAR2(60) , item_subtitle VARCHAR2(60) , item_rating VARCHAR2(8) , item_rating_agency VARCHAR2(4) , item_release_date DATE) ORGANIZATION EXTERNAL ( TYPE oracle_loader DEFAULT DIRECTORY upload_files ACCESS PARAMETERS ( RECORDS DELIMITED BY NEWLINE CHARACTERSET US7ASCII BADFILE 'UPLOAD_LOGS':'item_import_ext_table.bad' DISCARDFILE 'UPLOAD_LOGS':'item_import_ext_table.dis' LOGFILE 'UPLOAD_LOGS':'item_import_ext_table.log' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY "'" MISSING FIELD VALUES ARE NULL ) LOCATION ('item_import.csv')) REJECT LIMIT UNLIMITED; |
Setting up a Physical File
You should put the following in a item_import.csv physical file (case sensitivity won’t matter on the Windows 7 platform but will matter on the Linux or Unix platforms):
'B000W74EQC','DVD_WIDE_SCREEN','Harry Potter and the Sorcerer''s Stone',,'PG','MPAA','11-DEC-2007' 'B000W746GK','DVD_WIDE_SCREEN','Harry Potter and the Chamber of Secrets',,'PG','MPAA','11-DEC-2007' 'B000W796OM','DVD_WIDE_SCREEN','Harry Potter and the Prisoner of Azkaban',,'PG','MPAA','11-DEC-2007' 'B000E6EK2Y','DVD_WIDE_SCREEN','Harry Potter and the Goblet of Fire',,'PG-13','MPAA','07-MAR-2006' 'B000W7F5SS','DVD_WIDE_SCREEN','Harry Potter and the Order of the Phoenix',,'PG-13','MPAA','11-DEC-2007' 'B002PMV9FG','DVD_WIDE_SCREEN','Harry Potter and the Half-Blood Prince',,'PG','MPAA','08-DEC-2009' 'B001UV4XHY','DVD_WIDE_SCREEN','Harry Potter and the Deathly Hallows, Part 1',,'PG-13','MPAA','15-APR-2011' 'B001UV4XIS','DVD_WIDE_SCREEN','Harry Potter and the Deathly Hallows, Part 2',,'PG-13','MPAA','11-NOV-2011' |
Testing the External Table
After putting the item_import.csv file in the C:\Imports\ImportFiles directory, you can test the process at this point by running the following query:
SET PAGESIZE 99 COLUMN asin_number FORMAT A11 HEADING "ASIN #" COLUMN item_title FORMAT A46 HEADING "ITEM TITLE" COLUMN item_rating FORMAT A6 HEADING "RATING" COLUMN item_release_date FORMAT A11 HEADING "RELEASE|DATE" SELECT asin_number , item_title , item_rating , TO_CHAR(item_release_date,'DD-MON-YYYY') AS item_release_date FROM item_import_ext_table; |
It should return eight rows.
Extending Access to the Data Dictionary
The physical directory names of virtual directories are hidden from generic users. They’re available in the ALL_DIRECTORIES and DBA_DIRECTORIES administrative view for queries by the SYS, SYSTEM, and any DBA role privileged users.
While a privileged user can query the view, placing the view inside a function or procedure deployed in the privileged user’s schema would raise an ORA-00942 error. That error signals that the table or view does not exist.
This example deploys the view in the SYSTEM schema. That means it requires you make the following grant as the SYS user:
GRANT SELECT ON sys.dba_directories TO system; |
After making the grant from the SYS schema to the SYSTEM schema, connect to the SYSTEM schema. Then, create the following GET_DIRECTORY_PATH function:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | CREATE OR REPLACE FUNCTION get_directory_path ( virtual_directory IN VARCHAR2 ) RETURN VARCHAR2 IS -- Define RETURN variable. directory_path VARCHAR2(256) := ''; --Define dynamic cursor. CURSOR get_directory (virtual_directory VARCHAR2) IS SELECT directory_path FROM sys.dba_directories WHERE directory_name = virtual_directory; -- Define a LOCAL exception FOR name violation. directory_name EXCEPTION; PRAGMA EXCEPTION_INIT(directory_name,-22284); BEGIN OPEN get_directory (virtual_directory); FETCH get_directory INTO directory_path; CLOSE get_directory; -- RETURN file name. RETURN directory_path; EXCEPTION WHEN directory_name THEN RETURN NULL; END get_directory_path; / |
It’s tempting to make the grant on this function to PUBLIC user but that would expose information that any DBA should try and limit. That means you grant EXECUTE privilege only to the IMPORT schema.
This grant should be made as the SYSTEM user:
GRANT EXECUTE ON get_directory_path TO import; |
After granting the EXECUTE privilege to the IMPORT user, connect to the IMPORT schema and create a synonym to the GET_DIRECTORY_PATH function. The syntax for that command is:
CREATE SYNONYM get_directory_path FOR system.get_directory_path; |
You can now test your access to the function with the following query from the IMPORT schema:
SELECT get_directory_path('UPLOAD_FILES') FROM dual; |
You should return the following if you’ve got everything working at this point:
GET_DIRECTORY_PATH('UPLOAD_FILES') ------------------------------------ C:\Imports\ImportFiles |
At this point, you’ve completed the second major configuration component. You now need the ability to read files outside the database, which can be done with Java in Oracle 10g or Oracle 11g (that’s not possible in Oracle 10g XE or Oracle 11g XE because they don’t support an internal JVM). The
Reading Virtual Directory Files
The GET_DIRECTORY_PATH function provides you with the ability to read the Oracle data catalog and find the absolute directory path of a virtual directory. In this framework, you need this value to find whether the item_import.csv physical file is present in the file system before you read the file.
There doesn’t appear to be a neat little function to read an external directory. At least, there’s not one in the UTL_FILE or DBMS_LOB packages where you’d think it should be found. Unfortunately, that leaves us with two alternatives. One is to write an external library in C, C++, or C#. Another is to write an internal Java library that reads the file system. You accomplish this by granting permissions to a target directory or directories.
The first step is to create a scalar array of VARCHAR2 variables, like
CREATE OR REPLACE TYPE file_list AS TABLE OF VARCHAR2(255); / |
The second step is to write the Java library file. You can write it three ways. One accepts default error handling and the others override the default exception handling. If you’re new to Java, you should take the basic library with default handling. If you’ve more experience, you may want to override the helpful message with something that causes the developer to check with the DBA or simply suppress the message to enhance security.
You should note that the database connection is an Oracle Database 11g internal database connection. The connection only does one thing. It allows you to map the ArrayDescriptor to a schema-level SQL collection type. The element types of these collections should be scalar variables, like DATE, NUMBER, or VARCHAR2 data types.
The more advanced method overrides exception handling by suppressing information about the java.properties settings. You can do it by catching the natively thrown exception and re-throw it or ignore it. The example ignores it because handling it in Java reports an unhandled exception at the PL/SQL or SQL layer, which leads end users to think you have a major design problem.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "ListVirtualDirectory" AS // Import required classes. import java.io.*; import java.security.AccessControlException; import java.sql.*; import java.util.Arrays; import oracle.sql.driver.*; import oracle.sql.ArrayDescriptor; import oracle.sql.ARRAY; // Define the class. public class ListVirtualDirectory { // Define the method. public static ARRAY getList(String path) throws SQLException { // DECLARE variable AS a NULL, required because OF try-catch block. ARRAY listed = NULL; // Define a connection (this IS FOR Oracle 11g). Connection conn = DriverManager.getConnection("jdbc:default:connection:"); // USE a try-catch block TO trap a Java permission error ON the directory. try { // DECLARE a class WITH the file list. File directory = NEW File(path); // DECLARE a mapping TO the schema-level SQL collection TYPE. ArrayDescriptor arrayDescriptor = NEW ArrayDescriptor("FILE_LIST",conn); // Translate the Java String[] TO the Oracle SQL collection TYPE. listed = NEW ARRAY(arrayDescriptor,conn,((Object[]) directory.list())); } catch (AccessControlException e) {} RETURN listed; }} / |
You can’t call an internal Java library without a PL/SQL wrapper function. Here’s the wrapper function for this Java library:
CREATE OR REPLACE FUNCTION list_files(path VARCHAR2) RETURN FILE_LIST IS LANGUAGE JAVA NAME 'ListVirtualDirectory.getList(java.lang.String) return oracle.sql.ARRAY'; / |
You MUST grant the Oracle Database’s internal JVM authority to read the external directory before you can return the directory contents. Any attempt to read a directory without the proper permissions raises an ORA-29532 exception.
The following is an anonymous block to grant permissions to a directory. You must grant a minimum of read permissions but since you’ll also delete this file later in the post you should grant read, write, and delete. You must run it from the SYSDBA role as the SYS user.
1 2 3 4 5 6 7 | BEGIN DBMS_JAVA.GRANT_PERMISSION('IMPORT' ,'SYS:java.io.FilePermission' ,'C:\Imports\ImportFiles' ,'read,write,delete'); END; / |
While you’re connected, it’s a good idea to grant the same privileges to your log directory:
1 2 3 4 5 6 7 | BEGIN DBMS_JAVA.GRANT_PERMISSION('IMPORT' ,'SYS:java.io.FilePermission' ,'C:\Imports\ImportLogs' ,'read,write,delete'); END; / |
You should now be able to read the contents of an external file from another PL/SQL block or from a SQL statement. Here’s an example of the SQL statement call that uses everything developed to this point:
SELECT column_value AS "File Names" FROM TABLE(list_files(get_directory_path('UPLOAD_FILES'))); |
It should return the item_import.csv physical file as the only file in the physical directory, like:
File Names ----------------- item_import.csv |
Mapping an External Table to a source File
The next step leverages the user segment of the Oracle Database’s data catalog and all the components developed above to find and display the external table and external file. This query returns the results:
COLUMN TABLE_NAME FORMAT A30 COLUMN file_name FORMAT A30 SELECT xt.table_name , xt.file_name FROM (SELECT uxt.TABLE_NAME , ixt.column_value AS file_name FROM user_external_tables uxt CROSS JOIN TABLE(list_files(get_directory_path(uxt.default_directory_name))) ixt) xt JOIN user_external_locations xl ON xt.table_name = xl.table_name AND xt.file_name = xl.location; |
It should return the following:
TABLE_NAME FILE_NAME ------------------------------ ------------------------------ ITEM_IMPORT_EXT_TABLE item_import.csv |
You can migrate the query into the following function. It returns a zero when the file isn’t found and a one when it is found.
CREATE OR REPLACE FUNCTION external_file_found ( table_in VARCHAR2 ) RETURN NUMBER IS -- Define a default return value. retval NUMBER := 0; -- Decalre a cursor to find external tables. CURSOR c (cv_table VARCHAR2) IS SELECT xt.table_name , xt.file_name FROM (SELECT uxt.TABLE_NAME , ixt.column_value AS file_name FROM user_external_tables uxt CROSS JOIN TABLE(list_files(get_directory_path(uxt.default_directory_name))) ixt) xt JOIN user_external_locations xl ON xt.table_name = xl.table_name AND xt.file_name = xl.location AND xt.table_name = UPPER(cv_table); BEGIN FOR i IN c(table_in) LOOP retval := 1; END LOOP; RETURN retval; END; / |
With the EXTERNAL_FILE_FOUND function, you can create a function that returns rows when the external file is found and no rows when the external file isn’t found. The following view hides the logic required to make that work:
CREATE OR REPLACE VIEW item_import AS SELECT * FROM item_import_ext_table WHERE external_file_found('ITEM_IMPORT_EXT_TABLE') = 1; |
Conveniently, you can now query the ITEM_IMPORT view without the risk of raising the following error when the file is missing:
SELECT * FROM item_import_ext_table * ERROR at line 1: ORA-29913: error IN executing ODCIEXTTABLEOPEN callout ORA-29400: DATA cartridge error KUP-04040: file item_import.csv IN UPLOAD_FILES NOT found |
You can now grant the SELECT privilege on the ITEM_IMPORT view to your application schema, like:
GRANT SELECT ON item_import TO application; |
After granting the SELECT privilege on the ITEM_IMPORT view to the APPLICATION schema, you can create a synonym to hide the IMPORT schema.
CREATE SYNONYM item_import FOR item_import; |
At this point, many developers feel they’re done. Enclosing the results in a schema-level function provides more utility than a view. The next section shows you how to replace the view with a schema-level function.
Replacing the View with an Object Table Function
Inside a schema-level function, you can assign the results from the query to a SQL collection of an object type. The object type should mirror the structure of the table, like the following:
1 2 3 4 5 6 7 8 9 | CREATE OR REPLACE TYPE item_import_object IS OBJECT ( asin_number VARCHAR2(10) , item_type VARCHAR2(15) , item_title VARCHAR2(60) , item_subtitle VARCHAR2(60) , item_rating VARCHAR2(8) , item_rating_agency VARCHAR2(4) , item_release_date DATE); / |
After creating the object type that mirrors the structure of the ITEM_IMPORT_EXT_TABLE table, you need to create a list like collection of the object type. The nested table collection type acts like a list in Oracle:
1 2 3 | CREATE OR REPLACE TYPE item_import_object_table IS TABLE OF item_import_object; / |
After defining the object type and collection, you can access them in the following type of function:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | CREATE OR REPLACE FUNCTION external_file_contents ( table_in VARCHAR2 ) RETURN item_import_object_table IS -- Define a local counter. lv_counter NUMBER := 1; -- Construct an empty collection of ITEM_IMPORT_OBJECT data types. lv_item_import_table ITEM_IMPORT_OBJECT_TABLE := item_import_object_table(); -- Decalre a cursor to find external tables. CURSOR c (cv_table VARCHAR2) IS SELECT * FROM item_import_ext_table WHERE external_file_found(cv_table) = 1; BEGIN FOR i IN c(table_in) LOOP lv_item_import_table.EXTEND; lv_item_import_table(lv_counter) := item_import_object(i.asin_number ,i.item_type ,i.item_title ,i.item_subtitle ,i.item_rating ,i.item_rating_agency ,i.item_release_date); lv_counter := lv_counter + 1; END LOOP; /* * This is where you can place autonomous function calls: * ====================================================== * - These can read source and log files, and write them * to CLOB attributes for later inspection or review. * - These can call Java libraries to delete files, but * you should note that Java deletes any file rather * than moving it to the trash bin (where you might * recover it. */ RETURN lv_item_import_table; END; / |
Between the assignment to the collection and the return statement of the function, you have the ability of calling any number of autonomous functions. Any schema-level function can call autonomous functions that read and write tables with DML statements, like the INSERT, UPDATE, and DELETE statements. You can also call schema-functions that wrap Java libraries that delete external files.
You can confirm that the steps work by running the following query with or without the SQL*Plus formatting:
/* * SQL*Plus formatting. */ SET PAGESIZE 99 COLUMN asin_number FORMAT A11 HEADING "ASIN #" COLUMN item_title FORMAT A46 HEADING "ITEM TITLE" COLUMN item_rating FORMAT A6 HEADING "RATING" COLUMN item_release_date FORMAT A11 HEADING "RELEASE|DATE" /* * Query works only when item_import.csv file is present. */ SELECT asin_number , item_title , item_rating , TO_CHAR(item_release_date,'DD-MON-YYYY') AS item_release_date FROM TABLE(external_file_contents('ITEM_IMPORT_EXT_TABLE')); |
It should return the following from SQL*Plus:
RELEASE ASIN # ITEM TITLE RATING DATE ----------- ---------------------------------------------- ------ ----------- B000W74EQC Harry Potter and the Sorcerer's Stone PG 11-DEC-2007 B000W746GK Harry Potter and the Chamber of Secrets PG 11-DEC-2007 B000W796OM Harry Potter and the Prisoner of Azkaban PG 11-DEC-2007 B000E6EK2Y Harry Potter and the Goblet of Fire PG-13 07-MAR-2006 B000W7F5SS Harry Potter and the Order of the Phoenix PG-13 11-DEC-2007 B002PMV9FG Harry Potter and the Half-Blood Prince PG 08-DEC-2009 B001UV4XHY Harry Potter and the Deathly Hallows, Part 1 PG-13 15-APR-2011 B001UV4XIS Harry Potter and the Deathly Hallows, Part 2 PG-13 11-NOV-2011 |
The creation of the schema-level function lets you recreate the ITEM_IMPORT view. The following view would encapsulate (or hide) the presence of the function, which hides all the infrastructure components developed before this section (see line 14 in the function):
1 2 3 | CREATE OR REPLACE VIEW item_import AS SELECT * FROM TABLE(external_file_contents('ITEM_IMPORT_EXT_TABLE')); |
Implementing a Managed Import Process
During any import the information from the import process is exposed and one or more items may fail during the import process. That means the source file and loading log files must be preserved immediately after reading the data successfully. This is done by loading the data source file and log, discard, and bad import files into database tables. Only the source and log files exist when all rows are well formed, but the log files are reused for any subsequent load and require human inspection to isolate a specific upload.
The best way to implement this requires creating individual tables to hold each of the four potential large objects. The ITEM_MASTER table holds a transactional primary key and a table name for the import table. The primary key of the ITEM_MASTER table is the base key for imports and the ITEM_DATA, ITEM_LOG, ITEM_DISCARD, and ITEM_BAD tables hold foreign keys that point back to the ITEM_MASTER table’s primary key. These tables also hold a character large object column (CLOB), which will hold the respective source data file or log, discard, or bad files.
The following create the tables for the logging framework:
CREATE TABLE import_master ( import_master_id NUMBER CONSTRAINT pk_import_master PRIMARY KEY , import_table VARCHAR2(30)); -- Create sequence for import master. CREATE SEQUENCE import_master_s; -- Create import table. CREATE TABLE import_data ( import_data_id NUMBER CONSTRAINT pk_import_data PRIMARY KEY , import_master_id NUMBER , import_data CLOB , CONSTRAINT fk_import_data FOREIGN KEY (import_data_id) REFERENCES import_master (import_master_id)) LOB (import_data) STORE AS BASICFILE item_import_clob (TABLESPACE users ENABLE STORAGE IN ROW CHUNK 32768 PCTVERSION 10 NOCACHE LOGGING STORAGE (INITIAL 1048576 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645)); -- Create sequence for import master. CREATE SEQUENCE import_data_s; -- Create import table. CREATE TABLE import_log ( import_log_id NUMBER CONSTRAINT pk_import_log PRIMARY KEY , import_master_id NUMBER , import_log CLOB , CONSTRAINT fk_import_log FOREIGN KEY (import_log_id) REFERENCES import_master (import_master_id)) LOB (import_log) STORE AS BASICFILE item_import_log_clob (TABLESPACE users ENABLE STORAGE IN ROW CHUNK 32768 PCTVERSION 10 NOCACHE LOGGING STORAGE (INITIAL 1048576 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645)); -- Create sequence for import master. CREATE SEQUENCE import_log_s; -- Create import table. CREATE TABLE import_discard ( import_discard_id NUMBER CONSTRAINT pk_import_discard PRIMARY KEY , import_master_id NUMBER , import_discard CLOB , CONSTRAINT fk_import_discard FOREIGN KEY (import_discard_id) REFERENCES import_master (import_master_id)) LOB (import_discard) STORE AS BASICFILE item_import_discard_clob (TABLESPACE users ENABLE STORAGE IN ROW CHUNK 32768 PCTVERSION 10 NOCACHE LOGGING STORAGE (INITIAL 1048576 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645)); -- Create sequence for import master. CREATE SEQUENCE import_discard_s; -- Create import table. CREATE TABLE import_bad ( import_bad_id NUMBER CONSTRAINT pk_import_bad PRIMARY KEY , import_master_id NUMBER , import_bad CLOB , CONSTRAINT fk_import_bad FOREIGN KEY (import_bad_id) REFERENCES import_master (import_master_id)) LOB (import_bad) STORE AS BASICFILE item_import_bad_clob (TABLESPACE users ENABLE STORAGE IN ROW CHUNK 32768 PCTVERSION 10 NOCACHE LOGGING STORAGE (INITIAL 1048576 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645)); -- Create sequence for import master. CREATE SEQUENCE import_bad_s; |
The tables set the targets for uploading the source and log files. You should note that the table name is also the column name for the CLOB column, this becomes convenient when supporting a Native Dynamic SQL (NDS) statement in a single autonomous function. The LOAD_CLOB_FROM_FILE function supports reading the external source and log files and writing them their respective tables.
There is a DEADLOCK possibility with this type of architecture. It requires that the base row in the IMPORT_MASTER table is committed before attempting inserts into one of the dependent tables. A call to the function raises an error when the primary key column hasn’t been committed before hand.
You already set the access privileges for the DBMS_LOB package when you granted them to the UPLOAD_FILES and UPLOAD_LOGS virtual directories. This function only requires read permissions, which were granted to both virtual directories.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | CREATE OR REPLACE FUNCTION load_clob_from_file ( pv_src_file_name IN VARCHAR2 , pv_virtual_dir IN VARCHAR2 , pv_table_name IN VARCHAR2 , pv_column_name IN VARCHAR2 , pv_foreign_key IN NUMBER ) RETURN NUMBER IS -- Declare placeholder for sequence generated primary key. lv_primary_key NUMBER; -- Declare default return value. lv_retval NUMBER := 0; -- Declare local variables for DBMS_LOB.LOADCLOBFROMFILE procedure. des_clob CLOB; src_clob BFILE := BFILENAME(pv_virtual_dir,pv_src_file_name); des_offset NUMBER := 1; src_offset NUMBER := 1; ctx_lang NUMBER := dbms_lob.default_lang_ctx; warning NUMBER; -- Declare pre-reading size. src_clob_size NUMBER; -- Declare variables for handling NDS sequence value. lv_sequence VARCHAR2(30); lv_sequence_output NUMBER; lv_sequence_tagline VARCHAR2(10) := '_s.nextval'; -- Define local variable for Native Dynamic SQL (NDS) Statement. stmt VARCHAR2(2000); -- Declare the function as an autonomous transaction. PRAGMA AUTONOMOUS_TRANSACTION; BEGIN -- Open file only when found. IF dbms_lob.fileexists(src_clob) = 1 AND NOT dbms_lob.isopen(src_clob) = 1 THEN src_clob_size := dbms_lob.getlength(src_clob); dbms_lob.open(src_clob,dbms_lob.lob_readonly); END IF; -- Concatenate the sequence name with the tagline. lv_sequence := pv_table_name || lv_sequence_tagline; -- Assign the sequence through an anonymous block. stmt := 'BEGIN ' || ' :output := '||lv_sequence||';' || 'END;'; -- Run the statement to extract a sequence value through NDS. EXECUTE IMMEDIATE stmt USING IN OUT lv_sequence_output; -- Create a dynamic statement that works for all source and log files. -- ---------------------------------------------------------------------- -- NOTE: This statement requires that the row holding the primary key -- has been committed because otherwise it raises the following -- error because it can't verify the integrity of the foreign -- key constraint. -- ---------------------------------------------------------------------- -- DECLARE -- * -- ERROR at line 1: -- ORA-00060: deadlock detected while waiting for resource -- ORA-06512: at "IMPORT.LOAD_CLOB_FROM_FILE", line 50 -- ORA-06512: at line 20 -- ---------------------------------------------------------------------- stmt := 'INSERT INTO '||pv_table_name||' '||CHR(10)|| 'VALUES '||CHR(10)|| '('||lv_sequence_output||CHR(10)|| ','||pv_foreign_key||CHR(10)|| ', empty_clob())'||CHR(10)|| 'RETURNING '||pv_column_name||' INTO :locator'; -- Run dynamic statement. EXECUTE IMMEDIATE stmt USING OUT des_clob; -- Read and write file to CLOB, close source file and commit. dbms_lob.loadclobfromfile( dest_lob => des_clob , src_bfile => src_clob , amount => dbms_lob.getlength(src_clob) , dest_offset => des_offset , src_offset => src_offset , bfile_csid => dbms_lob.default_csid , lang_context => ctx_lang , warning => warning ); -- Close open source file. dbms_lob.close(src_clob); -- Commit write and conditionally acknowledge it. IF src_clob_size = dbms_lob.getlength(des_clob) THEN COMMIT; lv_retval := 1; ELSE RAISE dbms_lob.operation_failed; END IF; RETURN lv_retval; END load_clob_from_file; / |
You can test this procedure against the data source file with the following script file:
-- Insert a sample row in the master table. INSERT INTO import_master VALUES (import_master_s.nextval,'ITEM_IMPORT_EXT_TABLE'); -- Record the row value to avoid deadlock on uncommitted master record. COMMIT; -- Test program for loading CLOB files. DECLARE -- Declare testing variables. lv_file_name VARCHAR2(255) := 'item_import.csv'; lv_virtual_dir VARCHAR2(255) := 'UPLOAD_FILES'; lv_table_name VARCHAR2(30) := 'IMPORT_DATA'; lv_column_name VARCHAR2(30) := 'IMPORT_DATA'; lv_foreign_key NUMBER; BEGIN -- Assign the current value of the sequence to a local variable. lv_foreign_key := import_master_s.currval; -- Check if you can read and insert a CLOB column. IF load_clob_from_file(lv_file_name ,lv_virtual_dir ,lv_table_name ,lv_table_name ,lv_foreign_key) = 1 THEN -- Display a successful subordinate routine. dbms_output.put_line('Subordinate routine succeeds.'); ELSE -- Display a failed subordinate routine. dbms_output.put_line('Subordinate routine fails.'); END IF; END load_clob_from_file; / |
You can test this procedure against the log file with the following script file:
DECLARE -- Declare testing variables. lv_file_name VARCHAR2(255) := 'item_import_ext_table.log'; lv_virtual_dir VARCHAR2(255) := 'UPLOAD_LOGS'; lv_table_name VARCHAR2(30) := 'IMPORT_LOG'; lv_column_name VARCHAR2(30) := 'IMPORT_LOG'; lv_foreign_key NUMBER; BEGIN -- Assign the current value of the sequence to a local variable. lv_foreign_key := import_master_s.currval; dbms_output.put_line('Foreign key ['||lv_foreign_key||']'); -- Check if you can read and insert a CLOB column. IF load_clob_from_file(lv_file_name ,lv_virtual_dir ,lv_table_name ,lv_table_name ,lv_foreign_key) = 1 THEN -- Display a successful subordinate routine. dbms_output.put_line('Subordinate routine succeeds.'); ELSE -- Display a failed subordinate routine. dbms_output.put_line('Subordinate routine fails.'); END IF; END; / |
You now have the ability to read and store the source and log files in CLOB columns. The next step is to write a master function that writes the master row and calls the LOAD_CLOB_FROM_FILE function for the source file and each of the log files. That’s what the CLEANUP_EXTERNAL_FILES function provides.
Unfortunately, the Java logic requires using the logical and operation, which is two ampersands (&&). This requires that you turn off substitution variables in SQL*Plus. You do that by disabling DEFINE, like this:
SET DEFINE OFF |
You can compile this Java library file after you’ve disabled LOAD_CLOB_FROM_FILE:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | CREATE OR REPLACE FUNCTION cleanup_external_files ( table_in VARCHAR2 , data_directory_in VARCHAR2 , log_directory_in VARCHAR2 ) RETURN NUMBER IS -- Declare a local Attribute Data Type (ADT). TYPE list IS TABLE OF VARCHAR2(3); -- Declare a collection. lv_extension LIST := list('csv','log','bad','dis'); -- Define a default return value. retval NUMBER := 0; -- Declare base target table name. lv_target_table VARCHAR2(30) := 'IMPORT'; lv_foreign_key NUMBER; -- Decalre a cursor to find external tables. CURSOR check_source (cv_table_name VARCHAR2) IS SELECT xt.file_name FROM (SELECT uxt.TABLE_NAME , ixt.column_value AS file_name FROM user_external_tables uxt CROSS JOIN TABLE(list_files(get_directory_path(uxt.default_directory_name))) ixt) xt JOIN user_external_locations xl ON xt.TABLE_NAME = xl.TABLE_NAME AND xt.file_name = xl.location AND xt.TABLE_NAME = UPPER(cv_table_name); -- Declare a cursor to find files and compare for one input file name. CURSOR check_logs (cv_file_name VARCHAR2) IS SELECT list.column_value FROM TABLE(list_files(get_directory_path('UPLOAD_LOGS'))) list JOIN (SELECT cv_file_name AS file_name FROM dual) FILTER ON list.column_value = FILTER.file_name; -- Declare the function as autonomous. PRAGMA AUTONOMOUS_TRANSACTION; BEGIN -- Master loop to check for source and log files. FOR i IN check_source (table_in) LOOP -- Assign next sequence value to local variable. lv_foreign_key := import_master_s.nextval; -- Write the master record and commit it for the autonomous threads. INSERT INTO import_master VALUES (lv_foreign_key,'ITEM_IMPORT_EXT_TABLE'); COMMIT; -- Process all file extensions. FOR j IN 1..lv_extension.COUNT LOOP -- The source data file is confirmed by the CHECK_SOURCE cursor. IF lv_extension(j) = 'csv' THEN -- Load the source data file. -- ---------------------------------------------------------- -- The RETVAL holds success or failure, this approach -- suppresses an error when the file can't be loaded. -- It should only occur when there's no space available -- in the target table. retval := load_clob_from_file(i.file_name ,data_directory_in ,lv_target_table||'_DATA' ,lv_target_table||'_DATA' ,lv_foreign_key); lv_foreign_key := lv_foreign_key + 1; ELSE -- Verify that log file exists before attempting to load it. FOR k IN check_logs (LOWER(table_in)||'.'||lv_extension(j)) LOOP -- Load the log, bad, or dis(card) file. -- ---------------------------------------------------------- -- The RETVAL holds success or failure, as mentioned above. retval := load_clob_from_file(LOWER(table_in)||'.'||lv_extension(j) ,log_directory_in ,lv_target_table||'_'||lv_extension(j) ,lv_target_table||'_'||lv_extension(j) ,lv_foreign_key); END LOOP; END IF; END LOOP; retval := 1; END LOOP; RETURN retval; END; / |
Deleting Files from Virtual Directories
After you’ve read the files through a query and uploaded the source and log files to the database, you need to cleanup the files. This can be done by using another Java library function, provided you granted read, write, and delete privileges to the internal Java permissions file.
The DeleteFile Java library deletes files from the file system. It doesn’t put them in the trash can for final delete, it removes them completely.
Now you can build the Java library that lets you delete a file. A quick caveat, this code includes an AND logical operator that is two ampersands (&&). SQL uses an ampersand (&) for substitution variables. You’ll need to suppress that behavior when you run this code.
You do that by issuing the following command to disable substitution variables in SQL*Plus:
1 | SET DEFINE OFF |
You create the DeleteFile library like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "DeleteFile" AS // Java import statements import java.io.File; import java.security.AccessControlException; // Class definition. public class DeleteFile { // Define variable(s). private static File file; // Define copyTextFile() method. public static void deleteFile(String fileName) throws AccessControlException { // CREATE files FROM canonical file names. file = NEW File(fileName); // DELETE file(s). IF (file.isFile() && file.delete()) {}}} / |
You need a PL/SQL Wrapper to call the library, and here it is:
1 2 3 4 | CREATE OR REPLACE PROCEDURE delete_file (dfile VARCHAR2) IS LANGUAGE JAVA NAME 'DeleteFile.deleteFile(java.lang.String)'; / |
You can call this separately or embed it inside the UPLOAD_LOGS function, which saves re-writing the logic to find any source or log files.
This has provided you with an external table import framework. You can extend the framework by wrapping the query in an object table function. Such a function would afford you the opportunity to cleanup the source and log files after the query operation.
Java Generics in Oracle
Somebody posed the question about using a Comparator in the sorting examples provided in this earlier post on Updating Table View Columns (columns using a Varray or Nested Table of a single scalar data type). It seems the individual thought that you can’t use Java Generics inside an Oracle Database 11g’s Java libraries. It’s seems odd since they’ve been around since Java 5.
You can use Generics like those shown in the following example. It builds on explanation from the prior post. If you want to get the whole set of facts click the link above but you should have all the code you need in this post.
An example like this requires you first define a collection of strings in the database. This one uses the following definition:
1 2 | CREATE OR REPLACE TYPE stringlist IS TABLE OF VARCHAR2(4000); / |
This creates the Java library source, and line 21 shows the use of Generics in the instantiation of a anonymous Comparator class:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "SortList" AS // Import required classes. import java.io.*; import java.security.AccessControlException; import java.sql.*; import java.util.Arrays; import java.util.Comparator; import oracle.sql.driver.*; import oracle.sql.ArrayDescriptor; import oracle.sql.ARRAY; // Define class. public class Sorting { public static ARRAY sortTitleCaseList(oracle.sql.ARRAY list) throws SQLException, AccessControlException { // Convert Oracle data type to Java data type. String[] unsorted = (String[])list.getArray(); // Sort elements. Arrays.sort(unsorted, new Comparator<String>() { public int compare(String s1, String s2) { // Declare a sorting key integer for the return value. int sortKey; // Check if lowercase words match and sort on first letter only. if (s1.toLowerCase().compareTo(s2.toLowerCase()) == 0) sortKey = s1.substring(0,1).compareTo(s2.substring(0,1)); else sortKey = s1.toLowerCase().compareTo(s2.toLowerCase()); // Return the sorting index. return sortKey; }}); // Define a connection (this is for Oracle 11g). Connection conn = DriverManager.getConnection("jdbc:default:connection:"); // Declare a mapping to the schema-level SQL collection type. ArrayDescriptor arrayDescriptor = new ArrayDescriptor("STRINGLIST",conn); // Translate the Java String{} to the Oracle SQL collection type. ARRAY sorted = new ARRAY(arrayDescriptor,conn,((Object[])unsorted)); // Return the sorted list. return sorted; } } / |
The PL/SQL wrapper for this class would be:
1 2 3 4 | CREATE OR REPLACE FUNCTION sortTitleCaseList(list STRINGLIST) RETURN STRINGLIST IS LANGUAGE JAVA NAME 'Sorting.sortNaturalCaseList(oracle.sql.ARRAY) return oracle.sql.ARRAY'; / |
You can test the code with the following query:
1 2 | SELECT column_value FROM TABLE(sortTitleCaseList(stringlist('Oranges','apples','Apples','Bananas','Apricots','apricots'))); |
It sorts the strings based on a title case sort, like:
COLUMN_VALUE ------------------------ Apples apples Apricots apricots Bananas Oranges 6 rows selected. |
If you want a quick example of a Generic Collection sort operation outside of the database, here a sample file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | // Import required classes. import java.util.Arrays; import java.util.Comparator; /** * An example of using a Comparator sort. */ public class SortStaticString { /* * Sort the instance array. */ public static String[] sortTitleCaseList(String[] list) { // Sort elements by title case. Arrays.sort(list, new Comparator<String>() { public int compare(String s1, String s2) { // Declare a sorting key integer for the return value. int sortKey; // Check if lowercase words match and sort on first letter only. if (s1.toLowerCase().compareTo(s2.toLowerCase()) == 0) sortKey = s1.substring(0,1).compareTo(s2.substring(0,1)); else sortKey = s1.toLowerCase().compareTo(s2.toLowerCase()); // Return the sorting index. return sortKey; }}); // Return the sorted Index. return list; } /* * Test case. */ public static void main(String[] args) { // Construct and instance and apply sort method. args = SortStaticString.sortTitleCaseList(args); // Print the title case sorted list. for (int i = 0; i < args.length; i++) { System.out.println(args[i]); } } } |
You would call the SortStaticString class as follows:
java SortStaticString apples Oranges Pears Apples orange Grapefruit |
I hope this helps the interested party and any others looking for a sample file. 😉
Understanding Java Enum
Somebody wanted an example of how to write an Enum
class in Java 7 (a bit late since its introduced in Java 5) because they found the Enum
tutorial unhelpful (not as helpful to their purpose at hand). They wanted to understand how to use an Enum
type in another class. Here’s an example set of files to do that and here’s the link to the jazzed up Java 7 API online docs).
First, you need to understand that while the equals()
, toString()
, and hashCode()
override methods should always be provided in your classes. The exception is when they’re designated final
, like the toString()
and hashCode()
methods of the Enum
class. Second, you can write an Enum class with or without private variables. The inclusion of private instance variables makes the Enum
a complex Enum
(that’s just the vocabulary for the Java certification tests.
Sample Enum
Class
The AppleComputer class is a complex Enum
, and you should note that the constructor is private, and must always define the values of instance variables. The instance variables are defined within the parentheses after the name in the enumeration list.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | /** * Enumeration class AppleComputer - write a description of the enum class here * * @author Michael McLaughlin * @version 1.0 */ public enum AppleComputer { /** * This is an entry-level desktop computer. */ IMAC("Entry-level Desktop",1199), /** * This is an manager-level laptop computer. */ MACBOOKAIR("Manager-level Laptop",999), /** * This is an manager-level laptop computer. */ MACBOOKPRO("Developer-level Laptop",1199), /** * This is an developer-level laptop computer. */ MACMINI("Mini-Desktop",599), /** * This is a mini-desktop computer. */ MACPRO("Desktop",2499); /** * Private variable definitions. */ private double cost; private String description; /** * Constructs an instance with a cost and description. */ private AppleComputer(String description, double cost) { this.description = description; this.cost = cost; } /** * Returns the cost field of an Apple Computer. */ public double getCost() { return this.cost; } /** * Returns the description field of an Apple Computer. */ public String getDescription() { return description; } /** * Returns the description field of an Apple Computer. */ public String getDescription(String name) { return this.description; } /** * Returns the equality of between two AppleComputer Enum types. */ public boolean equals(AppleComputer ac) { // First comparision on primitives and second on String instances. if ((this.cost == ac.getCost()) && (this.description.equals(ac.getDescription()))) return true; else return false; } /** * Method to test class integrity. */ public static void main(String[] args) { if (args.length == 1) { System.out.printf("Apple Computer : %s is %s\n", AppleComputer.valueOf(args[0]).toString(), AppleComputer.IMAC.getDescription(args[0])); System.exit(0); } else { for (AppleComputer ac : AppleComputer.values()) System.out.printf("Apple Computer : %s is %s\n", ac, ac.description); }} } |
Sample Class that uses the Enum
Class
The EnumTextUse class demonstrates how to use and identify the instance of a complex Enum
class in another class. The setState()
method has two approaches, one where you pass the name and discover viable enumeration possibilities, and two when you pass an Enum instance.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | /** * The EnumTextUse class demonstrates how to identify and use an Enum * class that contains a text value. * * @author Michael McLaughlin * @version 1.0 */ /** * Import classes. */ import java.text.NumberFormat; import java.text.DecimalFormat; public class EnumTextUse { /** * Declare class level variables. */ private AppleComputer ac; private String desc; /** * Constructor for objects of class TestEnum */ public EnumTextUse() {} /** * An example of a method that takes a text string to find the ENUM * element value. */ public int setState(String name) { // Declare local variable for false. int returnValue = 0; // put your code here try { // Verify that it's a valid instance of AppleComputer. if (AppleComputer.valueOf(name) instanceof AppleComputer) { if (AppleComputer.IMAC == AppleComputer.valueOf(name)) { this.ac = AppleComputer.IMAC; } else if (AppleComputer.MACBOOKAIR == AppleComputer.valueOf(name)) { this.ac = AppleComputer.MACBOOKAIR; } else if (AppleComputer.MACBOOKPRO == AppleComputer.valueOf(name)) { this.ac = AppleComputer.MACBOOKPRO; } else if (AppleComputer.MACPRO == AppleComputer.valueOf(name)) { this.ac = AppleComputer.MACPRO; } else if (AppleComputer.MACMINI == AppleComputer.valueOf(name)) { this.ac = AppleComputer.MACMINI; }} // Return -1 as the truth indicator state was set or unnecessary. returnValue = -1; } catch (Exception e) { System.out.println(name + " is no longer sold."); } // Return the int value for true or false. return returnValue; } /** * An example of a method that takes a text string to find the ENUM * element value. */ public int setState(AppleComputer ac) { // Assign the AppleComputer fields to a local variable. double costSavings; double localCost = this.ac.getCost(); String localDesc = this.ac.toString() + " : " + this.ac.getDescription(); // Define format mask for output. NumberFormat f = new DecimalFormat("##,###,##0.00"); // Declare local variable for false. int returnValue = 0; // put your code here try { // Check for an instance of the Enum. if (this.ac instanceof AppleComputer) { // Find different (unequal) instances and update with the new one. if (this.ac.equals(ac)) { // Print message on match between prior and set value of AppleComputer. this.desc = this.ac + " is the authorized platform and no cost difference."; } else { // Assign the new Enum value, calculate and display cost savings message. this.ac = ac; costSavings = this.ac.getCost() - localCost; // Determine the message based on a reduced or increased cost of replacement. if (costSavings > 0) { this.desc = this.ac + " is substituted for " + localDesc + " at $" + f.format(costSavings) + " more than planned."; } else { this.desc = this.ac + " is substituted for " + localDesc + " at $" + f.format(Math.abs(costSavings)) + " less than planned."; }}} // Return -1 as the truth indicator state was set or unnecessary. returnValue = -1; } catch (Exception e) { System.out.println(e.getMessage()); } // Return the int value for true or false. return returnValue; } /** * Return the current description value. */ public String getState() { return this.desc; } /** * Allows testing the program. */ public static void main(String [] args) { // Declare a string of possible enumeration types. String [] list = {"IMAC","MACBOOK","MACBOOKAIR","MACBOOKPRO","MACMINI","MACPRO"}; // Construct a test instance. EnumTextUse etu = new EnumTextUse(); /** * Read through the list of enumeration types, printing output from * the inherited or overridden toString() method. */ for (int i = 0; i < list.length; i++) { if (etu.setState(list[i]) != 0) { // The company standard must apply in all cases. if (etu.setState(AppleComputer.IMAC) != 0) System.out.println(etu.getState()); } } } } |
You can run the EnumTextUse
class from the command-line or tool of your choice, like:
$ java EnumTextUse |
It’ll print the following text:
IMAC is the authorized platform and no cost difference. MACBOOK is no longer sold. IMAC is substituted for MACBOOKAIR : Manager-level Laptop at $200.00 more than planned. IMAC is substituted for MACBOOKPRO : Developer-level Laptop at $0.00 less than planned. IMAC is substituted for MACMINI : Mini-Desktop at $600.00 more than planned. IMAC is substituted for MACPRO : Desktop at $1,300.00 less than planned. |
You can check this Java Community Process page for the nuts and bolts of the Enum class. As always, I hoped this helped. Let me know if anything requires more clarity or any correction.
Updating Table View Columns
Answering a reader’s question: How can you sort data inside an Oracle table view column? This blog post shows you how to perform the trick, but for the record I’m not a fan of nested tables. A table view column is an Oracle specific user-defined type (UDT), and is nested table or varray of a scalar data type.
Oracle’s assigned a formal name to this type of UDT. It’s now labeled an Attribute Data Type (ADT). The ADT doesn’t allow you to update nested elements outside of PL/SQL program units.
This blog post reviews table view columns, and extends concepts from Oracle Database 11g & MySQL 5.6 Developer Handbook (by the way virtually everything in the book is relevant from MySQL 5.1 forward). It demonstrates how you can use PL/SQL user-defined functions (UDFs) to supplement the SQL semantics for updating nested tables, and then it shows how you can reshuffle (sort) data store the sorted data in table view columns.
Before you implement table view columns, you should answer two design questions and one relational modeling principal. You should also understand that this direction isn’t portable across database implementations. It currently supported fully by the Oracle database and mostly by PostgreSQL database. You can find how to join nested tables helpful in understanding the UPDATE
statements used in this posting, and this earlier post on UPDATE
and DELETE
statements.
Design Questions:
- Should you implement full object types with access methods in PL/SQL? The object type solution says there is no value in the nested data outside of the complete object. While choosing the table view column solution says that there is value to just implementing a nested list without element handling methods.
- Should you embed the elements in an XML_TYPE? An XML solution supports hierarchical node structures more naturally, like when you only access child nodes through the parent node. While choosing the table view column solution says that you want to avoid the XML Software Development Kit and that the data set is small and more manageable in a table view column.
Design Principle:
- Should you implement an ID-dependent relational modeling concept? An ID-dependent model replaces the primary and foreign keys with the relative position of parent and child elements. This is the design adopted when you choose a table view column, and it is more complex than single subject relational tables.
You should note that table view columns are inherently static at creation. You must also update the entire nested table view column when using Oracle SQL. Oracle SQL does let you modified attributes of object types in nested tables, as qualified in my new book (page 252).
Any attempt to modify a table view column element in SQL raises an ORA-25015 error. The error message states that (you) cannot perform DML on this nested TABLE VIEW COLUMN
.
You can update the table view column value by replacing it with a new collection, and that’s done with a PL/SQL function. This type of function preserves the ordered list in the table view column by finding and replacing an element in the collection.
Unfortunately, developers who use nested tables typically design table view columns with an internal ordering scheme. That means the collection is ordered during insert or update. This type of design relies on the fact that you can’t change the order without re-writing the stored structure.
While common for those you use these, it is a bad practice to rely on the ordering of elements in a collection. At least, it’s a bad practice when we’re trying to work within the relational model. All that aside, here’s how you ensure element updates while preserving element position:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | CREATE OR REPLACE FUNCTION update_collection ( old_element_collection STREET_LIST , old_element_value VARCHAR2 , new_element_value VARCHAR2 ) RETURN STREET_LIST IS -- Declare and initial a new counter. lv_counter NUMBER := 1; -- Declare local return collection variable. lv_element_collection STREET_LIST := street_list(); BEGIN FOR i IN 1..old_element_collection.COUNT LOOP IF NOT old_element_collection(i) = old_element_value THEN lv_element_collection.EXTEND; lv_element_collection(lv_counter) := old_element_collection(i); ELSE lv_element_collection.EXTEND; lv_element_collection(lv_counter) := new_element_value; END IF; lv_counter := lv_counter + 1; END LOOP; RETURN lv_element_collection; END update_collection; / |
Then, you can use the user-defined function (UDF) inside a SQL UPDATE
statement, like this:
1 2 3 4 5 6 | UPDATE TABLE (SELECT e.home_address FROM employee e WHERE e.employee_id = 1) e SET e.street_address = update_collection(e.street_address, 'Suite 525','Suite 522') , e.city = 'Oakland' WHERE e.address_id = 1; |
The UPDATE_COLLECTION
function replaces Suite 525 with Suite 522, and preserves the sequence of elements in a new nested table. The UPDATE
statement assigns the modified nested table to the table view column. You can find the code to create the employee table in Chapter 6 (pages 148-149), and the code to insert the default data in Chapter 8 (page 229) of Oracle Database 11g & MySQL 5.6.
The lv_counter
variable could be replaced with a reference to the for loop’s iterator (i
) because the counts of both collections are the same. I opted for the local variable to make the code easier to read.
While common for those you use these, it is a bad practice to rely on the ordering of elements in a collection. At least, it’s a bad practice when we’re trying to work within the relational model. Along the same line of thought, you also have the ability of removing elements from a table view column with a similar PL/SQL function. You could write the function like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | CREATE OR REPLACE FUNCTION delete_from_collection ( old_element_collection STREET_LIST , old_element_value VARCHAR2 ) RETURN STREET_LIST IS -- Declare and initial a new counter. lv_counter NUMBER := 1; -- Declare local return collection variable. lv_element_collection STREET_LIST := street_list(); BEGIN FOR i IN 1..old_element_collection.COUNT LOOP IF NOT old_element_collection(i) = old_element_value THEN lv_element_collection.EXTEND; lv_element_collection(lv_counter) := old_element_collection(i); lv_counter := lv_counter + 1; END IF; END LOOP; RETURN lv_element_collection; END delete_from_collection; / |
Then, you can use the user-defined function (UDF) to delete an element from the collection inside a SQL UPDATE
statement, like this:
1 2 3 4 5 6 | UPDATE TABLE (SELECT e.home_address FROM employee1 e WHERE e.employee_id = 1) e SET e.street_address = delete_from_collection(e.street_address,'Suite 522') , e.city = 'Oakland' WHERE e.address_id = 1; |
After understanding all that, let’s examine how you sort data in a nested table or varray of a scalar data type (the basis of a table view column). The easiest way is a BULK COLLECT INTO
statement nested inside a function, like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | CREATE OR REPLACE FUNCTION sort_collection ( old_element_collection STREET_LIST) RETURN STREET_LIST IS -- Declare and initial a new counter. lv_counter NUMBER := 1; -- Declare local return collection variable. lv_element_collection STREET_LIST := street_list(); BEGIN -- Sort a collection alphabetically based on case sensitivity. SELECT column_value BULK COLLECT INTO lv_element_collection FROM TABLE(old_element_collection) ORDER BY column_value; RETURN lv_element_collection; END sort_collection; / |
You could test it with this:
1 2 | SELECT column_value FROM TABLE(sort_collection(street_list('Adams', 'Lewis', 'Clark', 'Fallon'))); |
Then, you can use the user-defined function (UDF) to update a table view column like this:
1 2 3 4 5 6 | UPDATE TABLE (SELECT e.home_address FROM employee1 e WHERE e.employee_id = 1) e SET e.street_address = sort_collection(e.street_address) , e.city = 'Oakland' WHERE e.address_id = 1; |
The funny thing about database solutions these days is that some Java developers don’t appreciate the simplicity of SQL and PL/SQL and would solve the problem with Java. Especially, if it was an case insensitive sort operation. That’s the hard way (easy way at the bottom), but I figured it should be thrown in because some folks think everything is generic if written in Java. Though, I thought making it proprietary would increase the irony and wrote it as a Java library for Oracle.
Here’s the Java library, which you can run from the SQL*Plus command line, SQL Developer, or that pricey Toad:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "SortOracleList" AS // Import required classes. import java.io.*; import java.security.AccessControlException; import java.sql.*; import java.util.Arrays; import oracle.sql.driver.*; import oracle.sql.ArrayDescriptor; import oracle.sql.ARRAY; // Define class. public class DemoSort { public static ARRAY getList(oracle.sql.ARRAY list) throws SQLException, AccessControlException { // Convert Oracle data type to Java data type. String[] unsorted = (String[])list.getArray(); // Sort elements. Arrays.sort(unsorted, String.CASE_INSENSITIVE_ORDER); // Define a connection (this is for Oracle 11g). Connection conn = DriverManager.getConnection("jdbc:default:connection:"); // Declare a mapping to the schema-level SQL collection type. ArrayDescriptor arrayDescriptor = new ArrayDescriptor("STRINGLIST",conn); // Translate the Java String{} to the Oracle SQL collection type. ARRAY sorted = new ARRAY(arrayDescriptor,conn,((Object[])unsorted)); return sorted; }} / |
Then, you write the PL/SQL wrapper like this:
1 2 3 4 | CREATE OR REPLACE FUNCTION sortTable(list STRINGLIST) RETURN STRINGLIST IS LANGUAGE JAVA NAME 'DemoSort.getList(oracle.sql.ARRAY) return oracle.sql.ARRAY'; / |
You could test the case insensitive sort with this:
1 2 | SELECT column_value FROM TABLE(sort_collection(street_list('Adams', 'adams', 'Lewis', 'Clark', 'Fallon'))); |
Naturally, it ignores the fact you could do it like this without Java by using the UPPER
function in the purely PL/SQL SORT_COLLECTION
function shown earlier in this post:
12 13 14 15 | -- Sort a collection alphabetically based on case insensitive comparison. SELECT column_value BULK COLLECT INTO lv_element_collection FROM TABLE(old_element_collection) ORDER BY UPPER(column_value); |
Anyway, it’s a bunch of thoughts about writing solutions for table view columns. Hope it helps those interested in nested tables.
MySQL and Java Tutorial
This demonstrates how to create an Java infrastructure for reading and writing large text files to a MySQL database. The example provides:
- A
FileIO.jar
library that lets you enter MySQL connection parameters through aJOptionPane
, and a customizedJFileChooser
to filter and read source files from the file system. - A
mysql-connector-java-3.1.14-bin.jar
file, which is MySQL’s library for JDBC communication with the MySQL Databases.
The steps to compiling and testing this code are qualified below:
- Download and install the Java Software Development Kit (JSDK) for Java 6.
- Create a
C:\JavaTest
folder on Windows, or a/JavaTest
directory from some mount point of your choice. - Download and position the
mysql-connector-java-3.1.14-bin.jar
andFileIO.jar
files in theJavaTest
directory. - Create a batch file to source your environment path (%PATH% on Windows and $PATH on Linux or Mac OS X) and the two Java Archive (JAR) files. A sample batch file is noted below:
set PATH=C:\Program Files\Java\jdk1.6.0_07\bin;%PATH% set CLASSPATH=C:\JavaDev\Java6\mysql-connector-java-3.1.14-bin.jar;C:\JavaDev\Java6\FileIO.jar;. |
You can run this file by simply typing the files first name. On Linux or Mac OS X, you first need to grant it privileges with the chmod
command as 755
.
- Copy the
WriteReadCLOBMysql.java
code from the bottom of this posting and also put it into theJavaTest
directory. - Compile the
WriteReadCLOBMysql.java
source code with thejavac
utility, as shown below:
javac WriteReadCLOBMysql.java |
After you compile it, you should run it as follows:
java WriteReadCLOBMysql |
- Before running the code, you’ll need to seed (
INSERT
) a row that meets the desired hard coded criteria. It requires anITEM_TITLE
value of'The Lord of the Rings - Fellowship of the Ring'
and anITEM_SUBTITLE
of'Widescreen Edition'
in theITEM
table. - When it runs, you’ll see the following tabbed
JOptionPane
.
You need to enter the following values before clicking the OK button:
- Host: The
localhost
key word, orhostname
of your physical machine running the database. - Port: The
port
that the MySQL Listener is running on (the default value is3306
). - Database: The Oracle TNS Alias, which is
sampledb
for the full database sample database. - UserID: The
user
name with permissions to the database entered that can access anITEM
table. - Password: The
password
for the user’s account.
In the JFileChooser
, select a file to upload to the database.
You should see what you uploaded displayed in a JFrame
.
Java Source Code Program ↓
The drop down unfolds the WriteReadCLOB.java
source code.
The following program has dependencies on the FileIO.jar file. You need to download it and put it in your $CLASSPATH
for Linux or Mac OS X or %CLASSPATH%
for Windows.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 | // -------------------------------------------------------------------- // WriteReadCLOBMysql.java // by Michael McLaughlin // // This code demonstrates reading an image file and displaying // the image in a JLabel in a JFrame. // // The UPDATE and SELECT statements have dependencies on the // create_store.sql script. // -------------------------------------------------------------------- // Java Application class imports. import java.awt.Dimension; import java.awt.Font; import java.awt.GridLayout; import java.io.Reader; import javax.swing.JFrame; import javax.swing.JLabel; import javax.swing.JOptionPane; import javax.swing.JPanel; import javax.swing.JScrollPane; import javax.swing.JTextArea; // Generic JDBC imports. import java.sql.*; // Mysql JDBC import. import com.mysql.jdbc.Driver.*; import java.io.File; import java.sql.Connection; import java.sql.DriverManager; // Include book libraries (available at publisher website). import plsql.jdbc.DataConnectionPane; import plsql.fileio.FileIO; // -------------------------------------------------------------------/ public class WriteReadCLOBMysql extends JFrame { // Define database connections. private String host; private String port; private String dbname; private String userid; private String passwd; // Define data connection pane. private DataConnectionPane message = new DataConnectionPane(); // Construct the class. public WriteReadCLOBMysql (String s) { super(s); // Get database connection values or exit. if (JOptionPane.showConfirmDialog(this,message ,"Set Oracle Connection String Values" ,JOptionPane.OK_CANCEL_OPTION) == 0) { // Set class connection variables. host = message.getHost(); port = message.getPort(); dbname = message.getDatabase(); userid = message.getUserID(); passwd = message.getPassword(); // Print connection to console (debugging tool). message.getConnection(); // Create a JPanel for data display. ManageCLOB panel = new ManageCLOB(); // Configure the JPanel. panel.setOpaque(true); setContentPane(panel); // Configure the JFrame. setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE); setLocation(100,100); pack(); setVisible(true); } else System.exit(1); } // -------------------------------------------------------------------/ private class ManageCLOB extends JPanel { // Define display variables. private String clobText; private JScrollPane scrollPane; private JTextArea textArea; // -----------------------------------------------------------------/ public ManageCLOB () { // Set layout manager. super(new GridLayout(1,0)); // Assign file read to String. clobText = FileIO.openFile(FileIO.findFile(this)); // Insert record before querying it. if (clobText.length() > 0) { if (insertClob(host,port,dbname,userid,passwd,clobText)) clobText = getQuery(host,port,dbname,userid,passwd); else clobText = null; } else System.exit(2); // Construct text area and format it. textArea = new JTextArea(clobText); textArea.setEditable(false); textArea.setFont(new Font(Font.SANS_SERIF,Font.PLAIN,14)); textArea.setLineWrap(true); textArea.setRows(10); textArea.setSize(400,100); textArea.setWrapStyleWord(true); // Put the image in container, and add label to panel. scrollPane = new JScrollPane(textArea); add(scrollPane); } // ---------------------------------------------------------------/ private Boolean insertClob(String host,String port,String dbname ,String user,String pswd,String fileString) { try { // Define connection. DriverManager.registerDriver(new com.mysql.jdbc.Driver()); Connection conn = DriverManager.getConnection("jdbc:mysql://" + host + ":" + port + "/" + dbname, user, pswd); // Create statement. PreparedStatement prest; String sql = "UPDATE item SET item_desc = ? WHERE item_title = 'The Lord of the Rings - Fellowship of the Ring' AND item_subtitle = 'Widescreen Edition'"; prest = conn.prepareStatement(sql); prest.setString(1,fileString); // Execute query. if (prest.execute()) conn.commit(); // Close resources. prest.close(); conn.close(); // Return CLOB as a String data type. return true; } // End of connection try-block. catch (SQLException e) { if (e.getSQLState() == null) { System.out.println( new SQLException("mysql Client Net8 Connection Error.", "mysql-" + e.getErrorCode() + ": Incorrect Net8 thin client arguments:\n\n" + " database name [" + dbname + "]\n", e.getErrorCode()).getSQLState()); // Return an empty String on error. return false; } else { System.out.println(e.getMessage()); // Return an empty String on error. return false; }}} // -----------------------------------------------------------------/ private String getQuery(String host,String port,String dbname ,String user,String pswd) { // Define method variables. char[] buffer; int count = 0; int length = 0; String data = null; String[] type; StringBuffer sb; try { // Define connection. DriverManager.registerDriver(new com.mysql.jdbc.Driver()); Connection conn = DriverManager.getConnection("jdbc:mysql://" + host + ":" + port + "/" + dbname, user, pswd); // Define metadata object. DatabaseMetaData dmd = conn.getMetaData(); // Create statement. Statement stmt = conn.createStatement(); // Execute query. ResultSet rset = stmt.executeQuery( "SELECT item_desc " + "FROM item " + "WHERE item_title = " + "'The Lord of the Rings - Fellowship of the Ring'"+ "AND item_subtitle = 'Widescreen Edition'"); // Get the query metadata, size array and assign column values. ResultSetMetaData rsmd = rset.getMetaData(); type = new String[rsmd.getColumnCount()]; for (int col = 0;col < rsmd.getColumnCount();col++) type[col] = rsmd.getColumnTypeName(col + 1); // Read rows and only CLOB data type columns. while (rset.next()) { for (int col = 0;col < rsmd.getColumnCount();col++) { if (type[col] == "CLOB") { // Assign result set to CLOB variable. Clob clob = rset.getClob(col + 1); // Check that it is not null and read the character stream. if (clob != null) { Reader is = clob.getCharacterStream(); // Initialize local variables. sb = new StringBuffer(); length = (int) clob.length(); // Check CLOB is not empty. if (length > 0) { // Initialize control structures to read stream. buffer = new char[length]; count = 0; // Read stream and append to StringBuffer. try { while ((count = is.read(buffer)) != -1) sb.append(buffer); // Assign StringBuffer to String. data = new String(sb); } catch (Exception e) {} } else data = (String) null; } else data = (String) null; } else { data = (String) rset.getObject(col + 1); }}} // Close resources. rset.close(); stmt.close(); conn.close(); // Return CLOB as a String data type. return data; } catch (SQLException e) { if (e.getSQLState() == null) { System.out.println( new SQLException("mysql Client Net8 Connection Error.", "mysql-" + e.getErrorCode() + ": Incorrect Net8 thin client arguments:\n\n" + " database name [" + dbname + "]\n", e.getErrorCode()).getSQLState()); // Return an empty String on error. return data; } else { System.out.println(e.getMessage()); return data; }} finally { if (data == null) System.exit(1); }}} // -----------------------------------------------------------------/ public static void main(String[] args) { // Define window. WriteReadCLOBMysql frame = new WriteReadCLOBMysql("Write & Read CLOB Text"); }} |
OOW2010 – Day 3
Oracle Press authors met at the bookstore to sign books for an hour this morning and visit with our editors (shown at left). Then, we went off to the OTN Lounge and JavaOne at the Hilton.
It’s interesting to see different Java solution sets, especially on embedded devices. I also got a good look at the Vaadin server-side AJAX framework. It was interesting. The QuickConnect AJAX framework isn’t represented at the conference, but it’s a free open source project that may interest some.
Java’s dominance in the Oracle product stack is clear. Java is the crux of the SOA architecture for Oracle’s middleware solutions. Interestingly, I’ve met a number who are new to Java. A number of the vendors here are looking for skilled Java programmers, which leads me to see a lot of opportunity for developers.
I snagged a copy of Oracle Essbase and Oracle OLAP today. Oracle Essbase is part of Oracle Fusion Middleware 11g. I’ve began reading the book this evening. It appears a good starting place for those exploring Oracle Essbase and Oracle OLAP solution space.