MacLochlainns Weblog

Michael McLaughlin's Technical Blog

Site Admin

Archive for the ‘Linux’ Category

Fedora 30 Missing Library

without comments

Having run into an obsolete library issue installing Oracle Database 18c XE on Fedora, Version 30, I opted to revert my student image to Oracle Database 11g XE. The installation went without issue but when I tried to log into SQL*Plus as the oracle user, I got the following error message:

sqlplus: error while loading shared libraries: libnsl.so.1: cannot open shared object file: No such file or directory

The libnsl.so.1 library is no longer installed as part of the distribution for Fedora 28 forward but you can install it with the yum tool, like:

yum install -y libnsl

If you attempted to run the oracle-xe utility to configure the database prior to adding this library, it fails to provision the instance without a message. You won’t get the message until you manually try to connect as the sysdba privileged user. At that point, you’ll determine the instance wasn’t provisioned.

You can see that the installation failed when the oracle-xe utility fails to print the following lines to the console after the options are entered:

Starting Oracle Net Listener...Done
Configuring database...Done
Starting Oracle Database 11g Express Edition instance...Done
Installation completed successfully.

After installing the missing library, the oracle-xe utility works correctly. Alas, it looks like I’ll never bother to sort the Oracle Database 18c XE issues because after this version of the image we are moving the courses to a PostgreSQL database. PostgreSQL offers the smaller footprint that supports the core learning objectives of the courses.

As always, I hope this helps those looking for a solution.

Written by maclochlainn

August 11th, 2019 at 9:29 pm

Fedora 30 Install Chrome

without comments

While building the new Fedora 30 Linux instance for my students, I needed to install the Google Chrome browser. Here are the instructions for installing the Chrome browser.

  1. As the root user, create the google-chrome.repo file in the /etc/yum.repos.d directory with the following information:

    [chrome]
    name=google-chrome
    baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64
    enabled=1
    gpgcheck=1
    gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub
  2. Next, use the yum utility to install the Chrome browser:

    yum install -y google-chrome

    You should see the following log information:

  3. Click the following “f” icon in the lower left corner to bring up the Application Launcer:

    It raises the Application Launcher dialog:

    Enter Chrome inside the search field and it launches the Chrome browser:

As always, I hope this helps those trying to do the same thing.

Written by maclochlainn

August 9th, 2019 at 11:56 pm

Posted in Chrome,Fedora,Linux

Add user as sudoer

without comments

Somebody asked why adding a user to the wheel group in didn’t enable them as a sudoer, as qualified in my earlier Fedora post. The reason is that you also need to modify the primary group in the /etc/passwd file to specify the Group ID value for the wheel group as the primary group of the designated student user.

You can identify the Group ID with the following command:

cat /etc/group | grep wheel

It should return the following for the wheel group:

wheel:x:10:student

You need to check the target student user in the /etc/passwd file, which you can do with the following command:

cat /etc/passwd | grep student

It should return the following for the student user, which has a default group value equal to the user of the same name:

student:x:1000:1000:Student:/home/student:/bin/bash

As the root user, edit the /etc/passwd file to correct the student user’s primary group ID, as follows:

student:x:1000:10:Student:/home/student:/bin/bash

You should see the following two lines. If you want authorized sudoers to provide a password (recommended), then modify the first line by removing the # comment. If you don’t want authorized sudoers to provide a password, modify the second line by removing the # comment. Open the /etc/sudoers file with vi or gedit if you’d like a GUI editor.

# %wheel     ALL=(ALL)      ALL
# %wheel     ALL=(ALL)      NOPASSWD: ALL

Hope this helps. It’s a quick update for Fedora 30, you su to root and add your user to the sudoers list with the following syntax:

usermod someusername -a -G wheel

By the way, don’t forget to log off and then back on to the account.

Written by maclochlainn

June 12th, 2019 at 1:03 am

Posted in Fedora,Linux,Unix

Tagged with ,

Find files with errors

without comments

My students wanted a quick solution on how to find the log files that contain errors. That’s a simple line of code in Linux if you want any Oracle errors that start with ORA-:

find $HOME/lab2 -type f | xargs grep -i ora\-

It takes only a moment more to look for errors starting with ORA- or PLS-, like:

find $HOME/lab2 -type f | xargs grep -i -e ora\- -e pls\-

The latter might return something like this:

contact_lab.txt:ORA-00904: "MEMBER_LAB_ID": invalid identifier 
contact_lab.txt:ORA-00942: table or view does not exist 
contact_lab.txt:ORA-00942: table or view does not exist 
member_lab.txt:ORA-02264: name already used by an existing constraint 
member_lab.txt:ORA-00955: name is already used by an existing object

You can improve the error identification by identifying line numbers by adding -n option, like:

find $HOME/lab2 -type f | xargs grep -in -e ora\- -e pls\-

The latter might return something like this when there are two or more files:

contact_lab.txt:76:ORA-00904: "MEMBER_LAB_ID": invalid identifier 
contact_lab.txt:150:ORA-00942: table or view does not exist 
contact_lab.txt:157:ORA-00942: table or view does not exist 
member_lab.txt:75:ORA-02264: name already used by an existing constraint 
member_lab.txt:149:ORA-00955: name is already used by an existing object

Unfortunately, the command raises an error when there aren’t any files found of with a qualified extension. It also fails to prepend the file name when there’s only one qualified file name. As a result of these deficiencies, I’ve written the following Bash shell script. I’ve opted to call it the .findErrors.bashrc file name and deploy it in the user’s $HOME directory.

#!/bin/bash
 
  # Assign any file filter to the ext variable.
  ext=${1}
 
  # Assign the extension or simply use a wildcard for all files.
  if [ ! -z ${ext} ]; then
    ext="*.${ext}"
  else
    ext="*"
  fi
 
  # Assign the number of qualifying files to a variable.
  fileNum=$(ls -l ${ext} 2>/dev/null | grep -v ^l | wc -l)
 
  # Evaluate the number of qualifying files and process.
  if [ ${fileNum} -eq "0" ]; then
    echo "[0] files exist."
  elif [ ${fileNum} -eq "1" ]; then
    fileName=$(ls ${ext})
    find `pwd` -type f | grep -in ${ext} -e ora\- -e pls\- |
    while IFS='\n' read list; do
      echo "${fileName}:${list}"
    done
  else
    find `pwd` -type f | grep -in ${ext} -e ora\- -e pls\- |
    while IFS='\n' read list; do
      echo "${list}"
    done                                                                                                                                                   
  fi

You can modify the errors() function with or without a file extension to identify errors beginning with ORA- or PLS- in their log files. As always, I hope this helps those looking for a solution.

Written by maclochlainn

May 21st, 2019 at 8:04 pm

Chrome on Fedora 27

without comments

Installing Chrome wasn’t as easy just running the yum utility. You need to download it from the following website.

https://www.google.com/chrome/browser/desktop/index.html

When you try to use the rpm utility to run it, you’ll get the following errors:

warning: google-chrome-stable_current_x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 7fac5991: NOKEY
error: Failed dependencies:
        /usr/bin/lsb_release is needed by google-chrome-stable-74.0.3729.131-1.x86_64
        libappindicator3.so.1()(64bit) is needed by google-chrome-stable-74.0.3729.131-1.x86_64
        liberation-fonts is needed by google-chrome-stable-74.0.3729.131-1.x86_64

The failure points to three missing libraries:

  • lab_release
  • libappindicator3.so.1
  • liberation-fonts

While the error message indicates you’re missing the libappindicator3.so.1 library, the required library is actually the libappindicator-gtk3 library. You can use the yum utility to install the required library from the repository as the root user.

The library you need is actually in the repository as libappindicator-gtk3. You can use the yum utility to install the required library.

yum install -y libappindicator-gtk3
Last metadata expiration check: 1:04:24 ago on Sat 04 May 2019 05:34:34 PM MDT.                                       
Dependencies resolved.                                                                                                
======================================================================================================================
 Package                             Arch                  Version                        Repository             Size 
======================================================================================================================
Installing:                                                                                                           
 libappindicator-gtk3                x86_64                12.10.0-16.fc27                fedora                 41 k 
Installing dependencies:
 libdbusmenu-gtk3                    x86_64                16.04.0-4.fc27                 fedora                 38 k
 libindicator-gtk3                   x86_64                12.10.1-11.fc27                fedora                 67 k
 
Transaction Summary
======================================================================================================================
Install  3 Packages
 
Total download size: 147 k
Installed size: 382 k
Downloading Packages:
(1/3): libdbusmenu-gtk3-16.04.0-4.fc27.x86_64.rpm                                      63 kB/s |  38 kB     00:00    
(2/3): libappindicator-gtk3-12.10.0-16.fc27.x86_64.rpm                                 65 kB/s |  41 kB     00:00    
(3/3): libindicator-gtk3-12.10.1-11.fc27.x86_64.rpm                                    93 kB/s |  67 kB     00:00    
----------------------------------------------------------------------------------------------------------------------
Total                                                                                 145 kB/s | 147 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                              1/1 
  Installing       : libindicator-gtk3-12.10.1-11.fc27.x86_64                                                     1/3 
  Running scriptlet: libindicator-gtk3-12.10.1-11.fc27.x86_64                                                     1/3 
  Installing       : libdbusmenu-gtk3-16.04.0-4.fc27.x86_64                                                       2/3 
  Running scriptlet: libdbusmenu-gtk3-16.04.0-4.fc27.x86_64                                                       2/3 
  Installing       : libappindicator-gtk3-12.10.0-16.fc27.x86_64                                                  3/3 
  Running scriptlet: libappindicator-gtk3-12.10.0-16.fc27.x86_64                                                  3/3 
  Verifying        : libappindicator-gtk3-12.10.0-16.fc27.x86_64                                                  1/3 
  Verifying        : libdbusmenu-gtk3-16.04.0-4.fc27.x86_64                                                       2/3 
  Verifying        : libindicator-gtk3-12.10.1-11.fc27.x86_64                                                     3/3 
 
Installed:
  libappindicator-gtk3.x86_64 12.10.0-16.fc27                  libdbusmenu-gtk3.x86_64 16.04.0-4.fc27                 
  libindicator-gtk3.x86_64 12.10.1-11.fc27                    
 
Complete!

While the error message indicates you’re missing the lab_release library, the required library is actually the redhat-lsb-core library. You can use the yum utility to install the required library from the repository as the root user.

yum install -y redhat-lsb-core
Last metadata expiration check: 1:30:39 ago on Sat 04 May 2019 05:34:34 PM MDT.
Dependencies resolved.
======================================================================================================================
 Package                                Arch               Version                          Repository           Size
======================================================================================================================
Installing:
 redhat-lsb-core                        x86_64             4.1-36.fc27                      fedora               42 k
Installing dependencies:
 at                                     x86_64             3.1.20-6.fc27                    fedora               79 k
 mailx                                  x86_64             12.5-25.fc27                     updates             258 k
 ncurses-compat-libs                    x86_64             6.0-14.20170722.fc27             updates             321 k
 redhat-lsb-submod-security             x86_64             4.1-36.fc27                      fedora               20 k
 spax                                   x86_64             1.5.3-10.fc27                    fedora              212 k
 
Transaction Summary
======================================================================================================================
Install  6 Packages
 
Total download size: 932 k
Installed size: 1.9 M
Downloading Packages:
(1/6): redhat-lsb-submod-security-4.1-36.fc27.x86_64.rpm                               24 kB/s |  20 kB     00:00    
(2/6): redhat-lsb-core-4.1-36.fc27.x86_64.rpm                                          48 kB/s |  42 kB     00:00    
(3/6): at-3.1.20-6.fc27.x86_64.rpm                                                     80 kB/s |  79 kB     00:00    
(4/6): spax-1.5.3-10.fc27.x86_64.rpm                                                  554 kB/s | 212 kB     00:00    
(5/6): mailx-12.5-25.fc27.x86_64.rpm                                                  1.1 MB/s | 258 kB     00:00    
(6/6): ncurses-compat-libs-6.0-14.20170722.fc27.x86_64.rpm                            903 kB/s | 321 kB     00:00    
----------------------------------------------------------------------------------------------------------------------
Total                                                                                 479 kB/s | 932 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                              1/1 
  Installing       : mailx-12.5-25.fc27.x86_64                                                                    1/6 
  Installing       : ncurses-compat-libs-6.0-14.20170722.fc27.x86_64                                              2/6 
  Running scriptlet: ncurses-compat-libs-6.0-14.20170722.fc27.x86_64                                              2/6 
  Installing       : spax-1.5.3-10.fc27.x86_64                                                                    3/6 
  Running scriptlet: spax-1.5.3-10.fc27.x86_64                                                                    3/6 
  Installing       : redhat-lsb-submod-security-4.1-36.fc27.x86_64                                                4/6 
  Installing       : at-3.1.20-6.fc27.x86_64                                                                      5/6 
  Running scriptlet: at-3.1.20-6.fc27.x86_64                                                                      5/6 
  Installing       : redhat-lsb-core-4.1-36.fc27.x86_64                                                           6/6 
  Running scriptlet: redhat-lsb-core-4.1-36.fc27.x86_64                                                           6/6 
Running as unit: run-rca25c70a677d4866bb6056b31e1034c3.service
  Verifying        : redhat-lsb-core-4.1-36.fc27.x86_64                                                           1/6 
  Verifying        : at-3.1.20-6.fc27.x86_64                                                                      2/6 
  Verifying        : redhat-lsb-submod-security-4.1-36.fc27.x86_64                                                3/6 
  Verifying        : spax-1.5.3-10.fc27.x86_64                                                                    4/6 
  Verifying        : ncurses-compat-libs-6.0-14.20170722.fc27.x86_64                                              5/6 
  Verifying        : mailx-12.5-25.fc27.x86_64                                                                    6/6 
 
Installed:
  redhat-lsb-core.x86_64 4.1-36.fc27                       at.x86_64 3.1.20-6.fc27                                   
  mailx.x86_64 12.5-25.fc27                                ncurses-compat-libs.x86_64 6.0-14.20170722.fc27           
  redhat-lsb-submod-security.x86_64 4.1-36.fc27            spax.x86_64 1.5.3-10.fc27                                 
 
Complete!

The last missing library is liberation-fonts, which you can also install from the repository with the yum utility, like so as the root user:

yum install -y liberation-fonts
Last metadata expiration check: 2:01:27 ago on Sat 04 May 2019 05:34:34 PM MDT.
Dependencies resolved.
======================================================================================================================
 Package                              Arch                Version                          Repository            Size
======================================================================================================================
Installing:
 liberation-fonts                     noarch              1:1.07.4-10.fc27                 updates               17 k
Installing dependencies:
 liberation-narrow-fonts              noarch              1:1.07.4-10.fc27                 updates              208 k
 
Transaction Summary
======================================================================================================================
Install  2 Packages
 
Total download size: 225 k
Installed size: 476 k
Downloading Packages:
(1/2): liberation-fonts-1.07.4-10.fc27.noarch.rpm                                      49 kB/s |  17 kB     00:00    
(2/2): liberation-narrow-fonts-1.07.4-10.fc27.noarch.rpm                              336 kB/s | 208 kB     00:00    
----------------------------------------------------------------------------------------------------------------------
Total                                                                                 191 kB/s | 225 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                              1/1 
  Installing       : liberation-narrow-fonts-1:1.07.4-10.fc27.noarch                                              1/2 
  Installing       : liberation-fonts-1:1.07.4-10.fc27.noarch                                                     2/2 
  Running scriptlet: liberation-fonts-1:1.07.4-10.fc27.noarch                                                     2/2 
  Verifying        : liberation-fonts-1:1.07.4-10.fc27.noarch                                                     1/2 
  Verifying        : liberation-narrow-fonts-1:1.07.4-10.fc27.noarch                                              2/2 
 
Installed:
  liberation-fonts.noarch 1:1.07.4-10.fc27               liberation-narrow-fonts.noarch 1:1.07.4-10.fc27              
 
Complete!

You can install the Chrome browser with the following command as the root user:

yum install -y google-chrome-stable_current_x86_64.rpm
Last metadata expiration check: 2:08:09 ago on Sat 04 May 2019 05:34:34 PM MDT.
Dependencies resolved.
======================================================================================================================
 Package                           Arch                Version                        Repository                 Size
======================================================================================================================
Installing:
 google-chrome-stable              x86_64              74.0.3729.131-1                @commandline               56 M
 
Transaction Summary
======================================================================================================================
Install  1 Package
 
Total size: 56 M
Installed size: 196 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                              1/1 
  Running scriptlet: google-chrome-stable-74.0.3729.131-1.x86_64                                                  1/1 
  Installing       : google-chrome-stable-74.0.3729.131-1.x86_64                                                  1/1 
  Running scriptlet: google-chrome-stable-74.0.3729.131-1.x86_64                                                  1/1 
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Resource temporarily unavailable)
error: /tmp/google.sig.KSJ5OI: key 1 import failed.
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Resource temporarily unavailable)
error: /tmp/google.sig.KSJ5OI: key 2 import failed.
Redirecting to /bin/systemctl start atd.service
Running as unit: run-rcb32925ea21c401da74f536a222563ef.service
  Verifying        : google-chrome-stable-74.0.3729.131-1.x86_64                                                  1/1 
 
Installed:
  google-chrome-stable.x86_64 74.0.3729.131-1                                                                         
 
Complete!

When you launch the Chrome browser, it will write the following error message to standard out:

[11346:11357:0504/212207.077855:ERROR:ssl_client_socket_impl.cc(946)] handshake failed; returned -1, SSL error code 1, net_error -200
context mismatch in svga_sampler_view_destroy

The issue is corrected in new distributions of Chrome. The actual bug is shown in the screen capture below.

I upgraded the code in Fedora 27 because the Firefox browser wasn’t processing HTML Select tags correctly. After the upgrade of all fixes, the Firefox browser fails to let you use tabs to query other web pages. I shutdown Firefox, restarted it successfully, and took note that the liberation-fonts package was the problem. The tab processing issue occurred because I applied liberation-fonts package without restarting Firefox. Both Firefox and Chrome use the liberation-fonts package for displaying text.

You should run Chrome with the following command-line syntax to avoid the warning errors triggered by an existing bug:

google-chrome 2>/dev/null &

It would display a web page, like:

I hope this helps those looking for a solution.

Written by maclochlainn

May 4th, 2019 at 7:56 pm

Posted in Linux

Python & Oracle 1

without comments

While Python is an interpreted language, Python is a very popular programming language. You may ask yourself why it is so popular? The consensus answers to why it’s so popular points to several factors. For example, Python is a robust high-level programming language that lets you:

  • Get complex things done quickly
  • Automate system and data integration tasks
  • Solve complex analytical problems

You find Python developers throughout the enterprise. Development, release engineering, IT operations, and support teams all use Python to solve problems. Business intelligence and data scientists use Python because it’s easy to use.

Developers don’t generally write end-user applications in Python. They mostly use Python as scripting language. Python provides a simple syntax that lets developers get complex things done quickly. Python also provides you with a vast set of libraries that let you can leverage to solve problems. Those libraries often simplify how you analyze complex data or automate repetitive tasks.

This article explains how to use the Python programming language with the Oracle database. It shows you how to install and use the cx_Oracle library to query data. Part 2 will cover how you insert, update, and delete data in the Oracle database, and how you call and use PL/SQL stored functions and procedures.

The article has two parts:

  • How you install and use cx_Oracle with the Oracle database
  • How you query data statically or dynamically

This audience for this article should know the basics of writing a Python program. If you’re completely new to Python, you may want to get a copy of Eric Matthes’ Python Crash Course: A Hands-On, Project-Based Introduction to Programming. More experienced developers with shell scripting backgrounds may prefer Al Sweigart’s Automate the Boring Stuff with Python.

This article uses Python 2.7, which appears to be the primary commercial version of Python in most organizations. At least, it’s what most vendors ship with Linux distros. It also happens to be the Python distro on Fedora Linux.

How you install and use cx_Oracle with the Oracle database

The first step requires that you test the current version of Python on your Operating System (OS). For the purpose of this paper, you use the student user account. The student user is in the sudoer list, which gives the account super user privileges.

You can find the Python version by opening a Terminal session and running the following command:

[student@localhost ~]$ python -V

It displays:

Python 2.7.5

You can download the current version of the cx_Oracle library at the Python Software Foundation’s web site. At the time of writing, the current version of the cx_Oracle is the cx_Oracle 5.2.1 version. The cx_Oracle library is available for download as a Red Hat Package Manager (RPM) module.

You download the cx_Oracle-5.2.1-11g-py26-1.x86_64.rpm to the /tmp directory or to a sudoer-enabled user’s downloads directory. Let’s assume you download the RPM into the /tmp directory. After you download the RPM, you can install it with the yum utility with this syntax:

yum install -y /tmp/cx_Oracle-5.2.1-11g-py27-1.x86_64.rpm

However, the most current version is now 7.0. You want the following file on Fedora 64-bit Linux, which can be found at the Python Package Index web site:

cx_Oracle-7.0.0-cp27-cp27mu-manylinux1_x86_64.whl

A wheel file requires that you use the pip utility (make sure to upgrade to the current version), like:

sudo pip install cx_Oracle-7.0.0-cp27-cp27mu*.whl

It should print the following to the console:

Processing ./cx_Oracle-7.0.0-cp27-cp27mu-manylinux1_x86_64.whl                                            
Installing collected packages: cx-Oracle                                                                  
Successfully installed cx-Oracle-7.0.0

The cx_Oracle library depends on the Oracle Client software, which may or may not be installed. It installs without a problem but would raise a runtime error when using the Python software. You can check whether cx_Oracle is installed with the following syntax:

rpm –qa oracle-instantclient11.2-basic

If the oracle-instantclient11.2-basic library isn’t installed, the command returns nothing. If the oracle-instantclient11.2-basic library is installed it returns the following:

oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64

Assuming you don’t have the Oracle Client software installed, you should download it from Oracle’s Instant Client Downloads web page. After you download the RPM, you install the Oracle 11g Release 2 Client software with the following syntax:

yum install -y /tmp/oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64.rpm

You now have the necessary software installed and configured to run and test Python programs that work with the Oracle database. Python uses a standard path configuration to look for Python modules or libraries. You can see that set of path values by connecting to the Python IDLE environment, which is the runtime environment. The IDLE environment is very much like the SQL*Plus environment.

You connect to the Python IDLE environment by typing the following:

python

It opens the Python IDLE environment. It should display the following:

Python 2.7.5 (default, Apr 10 2015, 08:09:05) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

You import the sys library and then you can print the path elements with the following command:

>>> import sys
print sys.path

It should print the following for Python 2.7 in Fedora Linux:

['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

You can now test whether the environment works by typing the following commands in the IDLE environment:

>>> import cx_Oracle
db = cx_Oracle.connect("student/student@xe")
print db.version

It prints:

11.2.0.2.0

The other two sections require you to test components inside Python files. That means you need to supplement the default Python path variable. You do that by adding values to the Python environment variable, which is $PYTHONPATH.

The following adds the /home/student/Code/python directory to the Python path variable:

export set PYTHONPATH=/home/student/Code/python

Next, we create an connection.py file, which holds the following:

# Import the Oracle library.
import cx_Oracle
 
try:
  # Create a connection.
  db = cx_Oracle.connect("student/student@xe")
 
  # Print a message.
  print "Connected to the Oracle " + db.version + " database."
 
except cx_Oracle.DatabaseError, e:
  error, = e.args
  print >> sys.stderr, "Oracle-Error-Code:", error.code
  print >> sys.stderr, "Oracle-Error-Message:", error.message
 
finally
  # Close connection. 
  db.close()

The import statement adds the cx_Oracle library to the program scope. The cx_Oracle library’s connect function takes either the user name and password, or the user name, password, and TNS alias.

The except block differs from what you typically see. The code value maps to the SQLCODE value and the message value maps to the SQLERRM value.

You can test the connection.py file as follows in the /home/student/Code/python directory:

python connection.py

It prints the following:

Connected to the Oracle 11.2.0.2.0 database.

This section has shown you how to setup the cx_Oracle library, and how you can test the cx_Oracle library with Python programs.

How you query data statically or dynamically

The prior section shows you how to connect to an Oracle instance and how to verify the driver version of the cx_Oracle library. Like most ODBC and JDBC software, Python first creates a connection. Then, you need to create a cursor inside a connection.

The basicCursor.py program creates a connection and a cursor. The cursor holds a static SQL SELECT statement. The SELECT statement queries a string literal from the pseudo dual table.

# Import the Oracle library.
import sys
import cx_Oracle
 
try:
  # Create a connection.
  db = cx_Oracle.connect("student/student@xe")
 
  # Create a cursor.
  cursor = db.cursor()
 
  # Execute a query.
  cursor.execute("SELECT 'Hello world!' FROM dual")
 
  # Read the contents of the cursor.
  for row in cursor:
    print (row[0]) 
 
except cx_Oracle.DatabaseError, e:
  error, = e.args
  print >> sys.stderr, "Oracle-Error-Code:", error.code
  print >> sys.stderr, "Oracle-Error-Message:", error.message
 
finally:
  # Close cursor and connection. 
  cursor.close()
}  db.close()

The connect function assigns a database connection to the local db variable. The cursor function returns a cursor and assigns it to the local cursor variable. The execute function dispatches the query to Oracle’s SQL*Plus and returns the result set into a row element of the local cursor variable. The for-each loop reads the row element from the cursor variable and prints one row at a time. Since the cursor only returns a string literal, there’s only one row to return.

You test the program with this syntax:

python basicConnection.py

It prints:

Hello world!

The next basicTable.py program queries the item table. The item table holds a number of rows of data. The code returns each row inside a set of parentheses.

# Import the Oracle library.
import cx_Oracle
 
try:
  # Create a connection.
  db = cx_Oracle.connect("student/student@xe")
 
  # Create a cursor.
  cursor = db.cursor()
 
  # Execute a query.
  cursor.execute("SELECT item_title " +
                 ",      item_rating " +
                 "FROM   item " +
                 "WHERE  item_type = "
                 "        (SELECT common_lookup_id " +
                 "         FROM   common_lookup " +
                 "         WHERE  common_lookup_type = 'DVD_WIDE_SCREEN')")
 
  # Read the contents of the cursor.
  for row in cursor:
    print (row[0], row[1]) 
 
except cx_Oracle.DatabaseError, e:
  error, = e.args
  print >> sys.stderr, "Oracle-Error-Code:", error.code
  print >> sys.stderr, "Oracle-Error-Message:", error.message
 
finally:
  # Close cursor and connection. 
  cursor.close()
  db.close()

The SQL query is split across several lines by using the + operator. The + operator concatenates strings, and it lets you format a long query statement. The range for loop returns tuples from the cursor. The tuples are determined by the SELECT-list of the query.

The query returns the following type of results:

('Casino Royale', 'PG-13')
...
('Star Wars - Episode I', 'PG')
('Star Wars - Episode II', 'PG')
('Star Wars - Episode III', 'PG-13')
('Star Wars - Episode IV', 'PG')
('Star Wars - Episode V', 'PG')
('Star Wars - Episode VI', 'PG')

At this point, you know how to work with static queries. The next example shows you how to work with dynamic queries. The difference between a static and dynamic query is that an element of the string changes.

You have two options for creating dynamic strings. The first lets you glue a string inside a query. The second lets you embed one or more bind variables in a string. As a rule, you should use bind variables because they avoid SQL injection risks.

The following is the basicDynamicTable.py script

# Import the Oracle library.
import cx_Oracle
 
sRate = 'PG-13'
 
try:
  # Create a connection.
  db = cx_Oracle.connect("student/student@xe")
 
  # Define a dynamic statment.
  stmt = "SELECT item_title, item_rating FROM item WHERE item_rating = :rating"
 
  # Create a cursor.
  cursor = db.cursor()
 
  # Execute a statement with a bind variable.
  cursor.execute(stmt, rating = sRate)
 
  # Read the contents of the cursor.
  for row in cursor:
    print (row[0], row[1]) 
 
except cx_Oracle.DatabaseError, e:
  error, = e.args
  print >> sys.stderr, "Oracle-Error-Code:", error.code
  print >> sys.stderr, "Oracle-Error-Message:", error.message
 
finally:
  # Close cursor and connection. 
  cursor.close()
  db.close()

You need to assign a dynamic SQL statement to a local string variable. The bind variable is preceded with a colon (:). The execute function takes a string variable with the dynamic SQL statement. Then, you provide a name and value pair. The name needs to match the bind variable in the dynamic SQL statement. The value needs to map to a local Python variable.

The query should return a full list from the item table for the two item_title and item_rating columns:

('Casino Royale', 'PG-13')
...
('Harry Potter and the Goblet of Fire', 'PG-13')
('Harry Potter and the Order of the Phoenix', 'PG-13')
('The Lord of the Rings - Fellowship of the Ring', 'PG-13')
('The Lord of the Rings - Two Towers', 'PG-13')
('The Lord of the Rings - The Return of the King', 'PG-13')
('The Lord of the Rings - The Return of the King', 'PG-13')

This article should have shown you how to effectively work static and dynamic queries. You can find the scripts on the github.com server.

Written by maclochlainn

December 6th, 2018 at 11:40 pm

Lab Correction

without comments

Anyone using the August 2018 Fedora image should note that I neglected to put the right transaction_upload2.csv file in the /u01/app/oracle/upload directory. You can fix that by navigating to the Lab 12 Instructions web page and click on the zip file link to download the correct file. You will see the following dialog asking whether you want to open the file with the Ark utility, click OK to continue:

After clicking OK to open in Ark, you will see the following Ark dialog:

Click on the Home option in the Places dialog to the left, then click the Downloads option. You should see the following dialog before you click the Extract button.

Open a Konsole session and become the root superuser with the following command:

su - root

Change directory to Lab8_Final_CSV_Files directory where you extracted the transaction_upload2.csv file, like this:

cd /home/student/Downloads/Lab8_Final_CSV_Files

Copy the transaction_upload2.csv file to the /u01/app/oracle/upload directory with the following command:

cp /home/student/Downloads/Lab8_Final_CSV_Files/transaction_upload2.csv /u01/app/oracle/upload/.

Change directory to the /u01/app/oracle/upload directory and run the following long list (ll) command:

ll

You should see the following:

-rw-r--r--. 1 oracle dba      80 Aug 23 22:13 character.csv
drwxr-xr-x. 2 oracle dba    4096 Aug 23 20:44 preproc
drwxr-xr-x. 2 oracle dba    4096 Aug 23 23:35 textfile
-rw-r--r--. 1 oracle dba  128700 Dec  4 15:46 transaction_upload2.csv
-rw-r--r--. 1 oracle dba 1739520 Aug 23 22:04 transaction_upload.csv

The transaction_upload2.csv file contains a value of 3 for the created_by and last_updated_by user values. There shouldn’t be a value of 3 in the system_user_id column of the system_user table. The transaction_upload2.csv file should contain a value of 1002 for the created_by and last_updated_by user values.

You can modify the transaction_upload2.csv file once you’ve put it in the correct directory as the root user with the following command:

cat transaction_upload2.csv | sed -e 's/\,3\,/\,1002\,/g' > x; cp x transaction_upload2.csv; rm x

The new image will correct this problem.

Written by maclochlainn

December 4th, 2018 at 4:34 pm

Preprocessing External Tables

without comments

A question that comes up now and again is there a way in Oracle Database 11g Express Edition to mimic some behavior in the Oracle Standard or Enterprise editions. Many of these questions arise because developers want to migrate a behavior they’ve implemented in Java to the Express Edition. Sometimes the answer is no but many times the answer is yes. The yes answers come with a how.

This article answers the question: “How can I read an operating systems’ file directory with out an embedded Java Virtual Machine (JVM)?” These developers have read or implemented logic like that found in my earlier “Using DBMS_JAVA to Read External Files” article. The answer is simple. You need to use a preprocessing script inside an external table. That’s what you will learn in this article, but if you’re not familiar with external tables you should read this other “External Tables” article.

External tables let you access plain text files with SQL*Loader or Oracle’s proprietary Data Pump files. You typically create external tables with Oracle Data Pump when you’re moving large data sets between database instances.

External tables use Oracle’s virtual directories. An Oracle virtual directory is an internal reference in the data dictionary. A virtual directory maps a unique directory name to a physical directory on the local operating system. Virtual directories were simple before Oracle Database 12c gave us the multitenant architecture. In a multitenant database there are two types of virtual directories. One services the schemas of the Container Database (CDB) and it’s in the CDB’s SYS schema. The other services the schemas of a Pluggable Database (PDB) and it’s in the ADMIN schema for the PDB.

You can create a CDB virtual database as SYSTEM user with the following syntax in Windows:

SQL> CREATE DIRECTORY upload AS 'C:\Data\Upload';

or, like this in Linux or Unix:

SQL> CREATE DIRECTORY upload AS '/u01/app/oracle';

There are some subtle differences between these two statements. Windows directories or folders start with a logical drive letter, like C:\, D:\, and so forth. Linux and Unix directories start with a mount point like /u01.

As you can read in the “External Tables” article, you need to change the ownership of external files and directories to the oracle user and, default, oracle user’s default dba group. Likewise, you should change the privilege of the containing directory to 755 (owner has read, write, and execute privileges; and group and others have read and execute privileges.

The balance of this article is broken into two pieces configuring a working external table with preprocessing and troubleshooting cartridge errors.

External Tables with Preprocessing Example

There are xxx database steps to creating this example. The first database step requires you create three virtual directories. The syntax for the three statements is:

SQL> CREATE DIRECTORY upload AS '/u01/app/oracle/upload';
SQL> CREATE DIRECTORY LOG AS '/u01/app/oracle/log';
SQL> CREATE DIRECTORY preproc AS '/u01/app/oracle/preproc';

The upload directory hosts the files you want to discover for upload. The log directory hosts the log files for the external tables. The preproc directory hosts the executable program, which generates a list of files currently in the upload directory.

After creating the virtual directories or before creating them, you should create the physical directories in the Linux operating system. The virtual directories can only point to something when it actually exists. Moreover, they work like Oracle’s synonyms that point to other objects in the database. The physical files need to be in a directory tree that is navigable by the oracle user and the oracle user and it’s default primary dba group needs to own them.

You can use the following command to change ownership when you’re the root user:

# chown –R oracle:dba /u01/app/oracle

The second database step requires that you grant privileges on the virtual directories to the student user. You can do that with the following syntax:

SQL> GRANT read ON DIRECTORY upload;
SQL> GRANT read, WRITE ON DIRECTORY LOG;
SQL> GRANT read, EXECUTE ON DIRECTORY preproc;

The upload directory requires read-only privileges. The log directory requires read and write privileges. The read privileges let it find files and the write privilege lets it append to log files when they already exist. The preproc directory requires read and execute privileges. The read privilege is the same as that explained earlier. The execute privilege lets you run the preprocessing program file.

The third database step requires creating an external file with preprocessing. The following script creates the sample table:

SQL> CREATE TABLE directory_list
  2  ( file_name  VARCHAR2(60))
  3  ORGANIZATION EXTERNAL
  4  ( TYPE oracle_loader
  5    DEFAULT DIRECTORY preproc
  6    ACCESS PARAMETERS
  7    ( RECORDS DELIMITED BY NEWLINE CHARACTERSET US7ASCII
  8	 PREPROCESSOR preproc:'list2dir.sh'
  9	 BADFILE     'LOG':'dir.bad'
 10	 DISCARDFILE 'LOG':'dir.dis'
 11	 LOGFILE     'LOG':'dir.log'
 12	 FIELDS TERMINATED BY ','
 13	 OPTIONALLY ENCLOSED BY "'"
 14	 MISSING FIELD VALUES ARE NULL)
 15    LOCATION ('list2dir.sh'))
 16 REJECT LIMIT UNLIMITED;

Line 5 designates the default directory as preproc because the location of the executable file should be in the preproc directory. Line 8 designates that there is a preprocessing step, and it identifies the virtual directory and physical file name inside single quotes. Line 15 identifies the source file for the external table, which is an executable program.

Next, you need to create the bash file to get and return a directory list. Before you write that file, you need to understand that preprocessing script files don’t inherit a $PATH environment variable from Oracle.

That probably means you might have tried to create a simple bash shell command like the following in a list2dir.sh file.

ls /u01/app/oracle/upload | find . -type f | ls *csv | sed -e 's/\.\///'

When you test this file by calling it from SQL, like this:

SQL> SELECT * FROM directory_list;

It raises the following exception stack:

SELECT * FROM directory_list
*
ERROR AT line 1:
ORA-29913: error IN executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04095: preprocessor command /u01/app/oracle/preprocess/list2dir.sh
encountered error "/u01/app/oracle/preprocess/list2dir.sh: line 1: ls: No such file or directory

The reason isn’t immediately clear to some developers. The significant error is:

ls: No such file or directory

The error message indicates that a call through Oracle’s OCI call interface cannot find the location of the ls program. That occurs because there is no $PATH variable set a list of values that points to the /usr/bin directory where you find the ls program. You need to prepend /usr/bin before the ls, find, and sed programs.

/usr/bin/ls /u01/app/oracle/upload | /usr/bin/find . -type f | /usr/bin/ls *csv | /usr/bin/sed -e 's/\.\///'

Create a list2dir.sh file in the /u01/app/oracle/preproc directory with the preceding command line. Then, make sure oracle is the owner with a primary dba group and the privileges are 755 on the file. The command to set the privileges is:

# chmod –R 755 /u01/app/oracle/preproc.sh

Having completed that Linux operating system step you should probably put some files in the upload directory. You can create empty files with the touch command at the linux command line for this example.

The fourth database step lets you query the external table, which runs the preprocessing program and returns its results as values in the table:

SQL> CREATE * FROM directory_list;

It should return something like this:

FILE_NAME
------------------------------
character.csv
transaction_upload2.csv
transaction_upload.csv

As always, this is written to help those solve problems.

Written by maclochlainn

November 11th, 2018 at 10:54 pm

MongoDB Update Statement

without comments

While discussing the pros and cons of MongoDB, my students wanted to know how to update a specific element in a collection. Collections are like tables in relational databases.

You create the users collection by inserting rows like this:

db.users.insert(
[
  { contact_account: "CA_20170321_0001"
  , first_name: "Jonathan"
  , middle_name: "Eoin"
  , last_name: "MacGregor"
  , addresses:
    {
      street_address: ["1111 Broadway","Suite 101"]
    , city: "Oakland"
    , state: "CA"
    , zip: "94607" 
    }
  }
, { contact_account: "CA_20170328_0001"
  , first_name: "Remington"
  , middle_name: "Alain" 
  , last_name: "Dennison"
  , addresses:
    {
      street_address: ["2222 Broadway","Suite 121"]
    , city: "Oakland"
    , state: "CA"
    , zip: "94607" 
    }
  }
])

You can query the results with the db.users.find({}) command, or you can query the formatted results with the following command:

db.users.find({}).pretty()

You can provide a simple update of middle_name element of a given collection element with the findAndModify() function. The following queries the users collection to find the JSON middle_name element where the contact_account value is equal to the “CA_20170330_0001” string.

db.users.findAndModify(
  { query: { contact_account: "CA_20170328_0001" }
  , update: { $set: { middle_name: "Alan" }}
  , upsert: false })

After changing the middle_name value from “Alain” to “Alan”, you can query the single element of the collection with the following:

db.users.find({ contact_account: "CA_20170328_0001" }).pretty()

It should return the following:

{
        "_id" : ObjectId("5bd7f69ba135dda917665de7"),
        "contact_account" : "CA_20170328_0001",
        "first_name" : "Remington",
        "middle_name" : "Alan",
        "last_name" : "Dennison",
        "addresses" : {
                "street_address" : [
                        "2222 Broadway",
                        "Suite 121"
                ],
                "city" : "Oakland",
                "state" : "CA",
                "zip" : "94607"
        }
}

You can replace the addresses string element value a collection of elements with the following findAndModify() function:

db.users.findAndModify(
  { query: { contact_account: "CA_20170328_0001" }
  , update:
    { $set:
      { addresses:
        [
          {
            active_status: true
          , start_date : new Date("2018-10-30")
          , street_address: ["2222 Broadway","Suite 121"]
          , city: "Oakland"
          , state: "CA"
          , zip: "94607" 
          }
        , {
            active_status: false
          , start_date: new Date("2017-10-01")
          , end_date : new Date("2018-10-29")
          , street_address: ["2222 Broadway","Suite 121"]
          , city: "Oakland"
          , state: "CA"
          , zip: "94607" 
          }
        ]
      }
    }
  , upsert: false })

You can re-query the modified result set with find() function with the same query syntax as used previously. This looks for a specific member element in the collection by matching the contact_account name’s value pair. It is the same as the one used earlier in this blog post.

db.users.find({ contact_account: "CA_20170328_0001" }).pretty()

It should return the following:

{
        "_id" : ObjectId("5bd7f69ba135dda917665de7"),
        "contact_account" : "CA_20170328_0001",
        "first_name" : "Remington",
        "middle_name" : "Alan",
        "last_name" : "Dennison",
        "addresses" : [
                {
                        "active_status" : true,
                        "start_date" : ISODate("2018-10-30T00:00:00Z"),
                        "street_address" : [
                                "2222 Broadway",
                                "Suite 121"
                        ],
                        "city" : "Oakland",
                        "state" : "CA",
                        "zip" : "94607"
                },
                {
                        "active_status" : false,
                        "start_date" : ISODate("2017-10-01T00:00:00Z"),
                        "end_date" : ISODate("2018-10-29T00:00:00Z"),
                        "street_address" : [
                                "2222 Broadway",
                                "Suite 121"
                        ],
                        "city" : "Oakland",
                        "state" : "CA",
                        "zip" : "94607"
                }
        ]
}

As always, I hope this helps someone.

Written by maclochlainn

October 30th, 2018 at 12:22 am

Posted in Linux,MongoDB,Unix

Tagged with

Linux mongod Service

with one comment

The installation of MongoDB doesn’t do everything for you. In fact, the first time you start the mongod service, like this as the root user or sudoer user with the command:

service mongod start

A sudoer user will be prompted for their password, like

A typical MongoDB instance raises the following errors:

Redirecting to /bin/systemctl start mongod.service
[student@localhost cit425]$ mongo
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.11
Server has startup warnings: 
2018-10-29T10:51:57.515-0600 I STORAGE  [initandlisten] 
2018-10-29T10:51:57.515-0600 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-10-29T10:51:57.515-0600 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-10-29T10:51:58.264-0600 I CONTROL  [initandlisten] 
2018-10-29T10:51:58.264-0600 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.                                                                                               
2018-10-29T10:51:58.264-0600 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.                                                                              
2018-10-29T10:51:58.264-0600 I CONTROL  [initandlisten]                                                  
2018-10-29T10:51:58.265-0600 I CONTROL  [initandlisten]                                                  
2018-10-29T10:51:58.265-0600 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 15580 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.

You can fix this by following the MongoDB instructions for the Unix ulimit Settings, which will tell you to create a mongod file in the /etc/systemd/system directory. You should create this file as the root superuser. This is what you should put in the file:

[Unit]
Description=MongoDB
Documentation=man:mongo
 
[Service]
# Other directives omitted
# (file size)
LimitFSIZE=infinity
# (cpu time)
LimitCPU=infinity
# (virtual memory size)
LimitAS=infinity
# (locked-in-memory size)
LimitMEMLOCK=infinity
# (open files)
LimitNOFILE=64000
# (processes/threads)
LimitNPROC=64000

Then, you should be able to restart the mongod service without any warnings with this command:

service mongod restart

As always, I hope this helps somebody.

Written by maclochlainn

October 29th, 2018 at 11:39 am