MacLochlainns Weblog

Michael McLaughlin's Technical Blog

Site Admin

Archive for the ‘Fedora’ Category

MongoDB Two Step

without comments

Sometimes a bit of humor helps with a new topic. Creating a database in MongoDB is a two-step process, like the Texas Two-Step (a nick name for a country/western two-step danced in common time). A Texas Two Step is a one-two, one-two shuffle which is like the two-step process for how you create a MongoDB database.

While databases in MongoDB are a multiuser sandbox like a relational database, you can’t simply create them and grant them privileges. You must first USE the database and then put a collection (a.k.a., equivalent to a table in MySQL) in it. This blog post shows you how to create a MongoDB play database with an actor collection. It shows you the commands to create the a database with one collection in it and how to verify its existence.

You connect through the mongo shell, like:

mongo

The first-step requires you to connect to the play database even though it doesn’t exist. You do that by typing:

> use play

You can verify you’re in the play database by typing the db command, which the mongo shell treats as an expression. It returns play, as the name of the current database.

However, when you call the shell show dbs helper:

> show dbs

It displays:

admin   0.000GB
config  0.000GB
local   0.000GB

You also should note that the play database still doesn’t exist when you run the JavaScript equivalent to the shell show dbs helper:

> db.getMongo().getDBs()

It returns the following list of databases, which excludes the as yet not created play database:

{
  "databases" : [
    {
      "name" : "admin",
      "sizeOnDisk" : 32768,
      "empty" : false
    },
    {
      "name" : "config",
      "sizeOnDisk" : 61440,
      "empty" : false
    },
    {
      "name" : "local",
      "sizeOnDisk" : 81920,
      "empty" : false
    }
  ],
  "totalSize" : 176128,
  "ok" : 1
}

You create an actor collection (a.k.a. the equivalent of a relational table) with the following syntax, where db maps to the play database that you’re using. Note that you must create or insert one document to begin a document collection. While you can use the insert method, you should use either the newer insertOne() or insertMany() methods.

> db.actors.insertOne({"actor" : {"first_name" : "Chris", "last_name" : "Pine"}, "age" : 40 })

A new call to the shell show dbs helper:

> show dbs

displays your new play database:

admin   0.000GB
config  0.000GB
local   0.000GB
play    0.000GB

You can add two more documents (a.k.a. for rows in a relational database) with the insertMany() method:

> db.actors.insertMany([{"actor" : {"first_name" : "Chris", "last_name" : "Evans"}, "age" : 39 }
                       ,{"actor" : {"first_name" : "Chris", "last_name" : "Pratt"}, "age" : 41 }])

A quick word to the JavaScript novices out there. Don’t forget the square brackets ([{},{},...]) in the insertMany() method call or you’ll get an error like this:

2021-04-12T16:20:13.237-0600 E QUERY    [js] TypeError: documents.map is not a function :
DBCollection.prototype.insertMany@src/mongo/shell/crud_api.js:295:1
@(shell):1:1

You call the shell show collections helper or db.getMongo().getCollectionNames() JavaScript function to display the collections in the play database. The show collections displays a list of collections, and the db.getMongo().getCollectionNames() displays an JavaScript array of collections.

If you’re like me, Mongo’s db convention is a bit risky that I could do something in the wrong database. So, I put the following function into my .mongorc.js (a.k.a., MongoDB Resource file):

prompt = function() { var dbName = db; return dbName + "> " }

It ensures you will see the current database name to the left of the prompt (“>”), like:

play>

You can query the documents from the actors collection with the following:

play> db.actors.find().pretty()

It returns:

{
  "_id" : ObjectId("6074c692813c5a85db9cc9df"),
  "actor" : {
    "first_name" : "Chris",
    "last_name" : "Pine"
  },
  "age" : 40
}
{
  "_id" : ObjectId("6074c7ea813c5a85db9cc9e0"),
  "actor" : {
    "first_name" : "Chris",
    "last_name" : "Evans"
  },
  "age" : 39
}
{
  "_id" : ObjectId("6074c7ea813c5a85db9cc9e1"),
  "actor" : {
    "first_name" : "Chris",
    "last_name" : "Pratt"
  },
  "age" : 41
}

As always, I hope this helps those looking for a solution.

Written by maclochlainn

April 12th, 2021 at 5:57 pm

PL/pgSQL OUT Mode

without comments

A friend asked me a question about using the OUT mode parameter versus INOUT mode parameters in functions. He formed an opinion that they didn’t work in PostgreSQL PL/pgSQL.

Unfortunately, there’s not a lot of material written about how to use the OUT mode parameter in functions. I thought an article showing the standard example with a call to the function might help. The standard example function from the PostgreSQL documentation is:

1
2
3
4
5
6
7
8
9
CREATE OR REPLACE 
  FUNCTION sales_tax( IN  amount  real
                    , OUT tax     real )
  AS $$
BEGIN
  /* Calculate the tax at 6%. */
  tax := amount * 0.06;
END;
$$ LANGUAGE plpgsql;

The RETURNS clause is optional but here’s how you can include it. The following example works exactly like the former.

1
2
3
4
5
6
7
8
9
10
CREATE OR REPLACE 
  FUNCTION sales_tax( IN  amount  real
                    , OUT tax     real )
  RETURNS real AS
$$
BEGIN
  /* Calculate the tax at 6%. */
  tax := amount * 0.06;
END;
$$ LANGUAGE plpgsql;

You call the PostgreSQL sales_tax() function like this:

SELECT 'Sales Tax ['|| sales_tax(200) ||']' AS "Return Value";

It should return the following:

  Return Value  
----------------
 Sales Tax [12]
(1 row)

You can also call it in an inline code block (e.g., what Oracle documentation calls an anonymous block), like:

1
2
3
4
5
6
7
8
9
10
11
12
13
DO
$$
DECLARE
  /* Declare a local variable. */
  tax_paid  real := 0;
BEGIN
  /* Query the results into a local variable. */
  SELECT sales_tax(100) INTO tax_paid;
 
  /* Print the taxes paid. */
  RAISE NOTICE 'Tax Paid [%]',tax_paid;
END;
$$;

You can replace the sales_tax function with its OUT mode tax parameter with the following classic sales_tax function, which adds a tax_rate parameter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
CREATE OR REPLACE
  FUNCTION sales_tax( IN  amount   real
                    , IN  tax_rate real )
  RETURNS real AS $$
DECLARE
  /* Declare a local variable. */
  tax real;
BEGIN
  /* Calculate the tax at 6%. */
  tax := amount * (tax_rate / 100);
 
  /* Return the tax. */
  RETURN tax;
END;
$$ LANGUAGE plpgsql;

Let’s return the original approach with the OUT parameter. Then, let’s expand the list of parameters to include an INOUT mode state variable, like:

1
2
3
4
5
6
7
8
9
10
11
12
CREATE OR REPLACE
  FUNCTION sales_tax( IN     amount   real
                    , IN     tax_rate real
                    , INOUT  state    VARCHAR(14)
                    , OUT    tax      real )
  RETURNS RECORD AS
$$
BEGIN
  /* Calculate the tax at 6%. */
  tax := amount * (tax_rate / 100);
END;
$$ LANGUAGE plpgsql;

You don’t need to include the RETURNS RECORD phrase because PL/pgSQL implements a very mature adapter pattern and it adjusts the return type automatically to the parameter list. On the other hand, many beginning programmers and support staff won’t know that. That’s why I recommend you include it for clarity.

You can call this in a query with a column alias, like:

SELECT 'Sales Tax ['|| sales_tax(100,8.25,'California') ||']' AS "Return Value";

It will return a tuple:

         Return Value          
-------------------------------
 Sales Tax [(California,8.25)]
(1 row)

You can implement it inside an inline block by adding a local variable of the RECORD data type, like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
DO
$$
DECLARE
  /* Declare a local variable. */
  state     varchar(14) := 'California';
  tax_paid  real        := 0;
  result    RECORD;
BEGIN
  /* Query the results into a local variable. */
  SELECT sales_tax(100,8.25,state) INTO result;
 
  /* Print the taxes paid. */
  RAISE NOTICE 'Tax Paid [%]', result;
END;
$$;

It prints the following:

NOTICE:  Tax Paid [("(California,8.25)")]

You can actually return the individual members of the tuple by putting the function call inside the FROM clause, like:

SELECT * FROM sales_tax(100,8.25,'California');

It now returns the members of the tuple in separate columns:

   state    | tax  
------------+------
 California | 8.25
(1 row)

Alternatively, you can call it from inside an inline block, like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
DO
$$
DECLARE
  /* Declare a local variable. */
  state     varchar(14) := 'California';
  tax_paid  real        := 0;
BEGIN
  /* Query the results into a local variable. */
  SELECT * INTO state, tax_paid FROM sales_tax(100,8.25,state);
 
  /* Print the taxes paid. */
  RAISE NOTICE 'Tax Paid [%] [%]', state, tax_paid;
END;
$$;

It prints:

NOTICE:  Tax Paid [California] [8.25]

Now, let’s rewrite the function into a traditional function with all IN mode variables that returns a RECORD structure with additional values. Just one quick caveat (the big but), you can only assign values to dynamically constructed RECORD structures by using the SELECT-INTO or FOR statements. Below is the refactored sales_tax() function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
CREATE OR REPLACE
  FUNCTION sales_tax( IN  subtotal REAL
                    , IN  tax_rate REAL
                    , IN  state    VARCHAR(14))
  RETURNS RECORD AS
$$
DECLARE
  /* Declare a local variable. */
  tax        REAL;
  tax_record RECORD;
BEGIN
  /* Calculate the tax at 6%. */
  tax := subtotal * (tax_rate / 100);
 
  /* Assign state to record. */
  SELECT state, tax INTO tax_record;
 
  /* Return the tax. */
  RETURN tax_record;
END;
$$ LANGUAGE plpgsql;

It returns the same set of values as the early version with the four parameter example above but you only need three IN-only mode variables to get the result. Other than the parameter lists, the biggest change appears to be the assignment line, which is required in the explicit and traditional function that has only IN mode parameters:

16
  SELECT state, tax INTO tax_record;

Given you can return any RECORD structure you want, why use INOUT and OUT mode parameters? Don’t you loose clarity about what your stored function does? Or, at least, don’t you make understanding the program logic more difficult when you use INOUT and OUT mode variables? The only benefit appears to be when you shift your input variables from the SELECT clause to the INTO clause.

Hopefully, this shows folks how to use the OUT mode parameter; and how closely related it is to a classic function.

Written by maclochlainn

November 25th, 2020 at 12:36 am

Installing PL/Python Extension

without comments

While PL/Python is an untrusted language inside PostgreSQL, I was installing it to test some of its features. First, we check to see if PL/Python is installed by attempting to create a PL/Python function:

CREATE FUNCTION pima(a integer, b integer)
RETURNS integer AS
$$
if a > b:
  return a
return b
$$ LANGUAGE plpython3u;

It likely should raise an error like this because PL/Python is an untrusted language. It’s untrusted because it runs with root privileges rather than a restricted user’s privilege. More or less, with PL/Python you can access the entire database. This makes PL/Python more of a threat than tool beyond experimentation in a test database.

ERROR:  language "plpython3u" does not exist
HINT:  Use CREATE EXTENSION to load the language into the database.

A quick query as the postgres user tells you whether or not PL/Python is installed:

SELECT * FROM pg_language;

It returns the following:

 lanname  | lanowner | lanispl | lanpltrusted | lanplcallfoid | laninline | lanvalidator | lanacl 
----------+----------+---------+--------------+---------------+-----------+--------------+--------
 internal |       10 | f       | f            |             0 |         0 |         2246 | 
 c        |       10 | f       | f            |             0 |         0 |         2247 | 
 sql      |       10 | f       | t            |             0 |         0 |         2248 | 
 plpgsql  |       10 | t       | t            |         14088 |     14089 |        14090 | 
(4 rows)

I attempted to add PL/Python with the following command:

CREATE EXTENSION plpython3u;

It raised the following error:

ERROR:  could not open extension control file "/usr/pgsql-11/share/extension/plpython3u.control": No such file or directory

The error basically appeared to occur because there’s a missing Python 3 package from what I could find on the web. I installed the missing postgresql-plpython3 package as the root superuser, which you also could install as a member of the sudoer list. The following shows how to install it as a sudoer member:

sudo yum install -y postgresql-plpython3

You should see something close to the following console output:

Last metadata expiration check: 0:43:53 ago on Fri 06 Nov 2020 10:42:28 AM MST.
Dependencies resolved.
============================================================================================================
 Package                           Architecture        Version                   Repository            Size
============================================================================================================
Installing:
 postgresql-plpython3              x86_64              11.7-2.fc30               updates               86 k
Installing dependencies:
 postgresql-server                 x86_64              11.7-2.fc30               updates              5.3 M
 
Transaction Summary
============================================================================================================
Install  2 Packages
 
Total download size: 5.3 M
Installed size: 23 M
Downloading Packages:
(1/2): postgresql-plpython3-11.7-2.fc30.x86_64.rpm                          218 kB/s |  86 kB     00:00    
(2/2): postgresql-server-11.7-2.fc30.x86_64.rpm                             3.0 MB/s | 5.3 MB     00:01    
------------------------------------------------------------------------------------------------------------
Total                                                                       2.1 MB/s | 5.3 MB     00:02     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                    1/1 
  Running scriptlet: postgresql-server-11.7-2.fc30.x86_64                                               1/2 
  Installing       : postgresql-server-11.7-2.fc30.x86_64                                               1/2 
warning: /var/lib/pgsql/.bash_profile created as /var/lib/pgsql/.bash_profile.rpmnew
 
  Running scriptlet: postgresql-server-11.7-2.fc30.x86_64                                               1/2 
  Installing       : postgresql-plpython3-11.7-2.fc30.x86_64                                            2/2 
  Running scriptlet: postgresql-plpython3-11.7-2.fc30.x86_64                                            2/2 
  Verifying        : postgresql-plpython3-11.7-2.fc30.x86_64                                            1/2 
  Verifying        : postgresql-server-11.7-2.fc30.x86_64                                               2/2 
 
Installed:
  postgresql-plpython3-11.7-2.fc30.x86_64                postgresql-server-11.7-2.fc30.x86_64               
 
Complete!

While I thought this might fix the problem, it didn’t and raised the following error:

ERROR:  could not open extension control file "/usr/pgsql-11/share/extension/plpython3u.control": No such file or directory

I found the plpython3u.control file on GitHub and put the following plpython3u.control file, after comparing it against the plpgsql.control file, into the /usr/pgsql-11/share/extension directory. At this point, I began wondering why it’s looking in the /usr/pgsql-11/share/extension directory instead of a /usr/plpython3u/share/extension directory (does not exist).

# plpython3u extension
comment = 'PL/Python3U untrusted procedural language'
default_version = '1.0'
module_pathname = '$libdir/plpython3'
relocatable = false
schema = pg_catalog
superuser = true

I retried creating the plpython3u extension:

CREATE EXTENSION plpython3u;

It raised the following error:

ERROR:  extension "plpython3u" has no installation script nor update path for version "1.0"

At this point, I could find no further help on the Internet. I did notice that there were these two *.sql files in the /usr/pgsql-11/share/extension directory:

  • plpgsql–1.0.sql
  • plpgsql–unpackaged–1.0.sql

I found this plpython3u--1.0.sql file on GitHub:

/* src/pl/plpython/plpython3u--1.0.sql */
 
CREATE FUNCTION plpython3_call_handler() RETURNS language_handler
  LANGUAGE c AS 'MODULE_PATHNAME';
 
CREATE FUNCTION plpython3_inline_handler(internal) RETURNS void
  STRICT LANGUAGE c AS 'MODULE_PATHNAME';
 
CREATE FUNCTION plpython3_validator(oid) RETURNS void
  STRICT LANGUAGE c AS 'MODULE_PATHNAME';
 
CREATE LANGUAGE plpython3u
  HANDLER plpython3_call_handler
  INLINE plpython3_inline_handler
  VALIDATOR plpython3_validator;
 
COMMENT ON LANGUAGE plpython3u IS 'PL/Python3U untrusted procedural language';

I retried creating the plpython3u extension, as a member of the sudoer list and got a new error:

ERROR:  permission denied to create extension "plpython3u"
HINT:  Must be superuser to create this extension.

I retried creating the plpython3u extension as the postgres user, who is the owning user:

CREATE EXTENSION plpython3u;

It raised the following error:

ERROR:  could not access file "$libdir/plpython3": No such file or directory

While I intend to finish this, that’s as far as I got. It appears from some of the things I’ve read I need to recompile or configure items that would destabilize what I have working at the moment. Finishing this will need to wait for me to build another test environment from scratch. If you catch this post and know the remaining steps, I invite you to add them in the comments.

Written by maclochlainn

November 8th, 2020 at 1:54 pm

Conditional Updates

without comments

While I’m switching labs next term after more than a decade with more comprehensive lab set, I’m hoping the new exercises build the students’ core SQL skill set. Next term, I hope to see whether the change is successful. I’ve opted for using Alan Beaulieu’s Learning SQL: Generate, Manipulate, and Retrieve Data, 3rd Edition, because it’s a great book and uses the MySQL database.

One exercise that the students will lose is a data migration exercise from a badly designed common_lookup table to a well designed common_lookup table. The starting point is shown below on the left and the fixed version is on the right.

        

There are several problems with the original common_lookup table’s design. The first problem is that the common_lookup_context column does not uniquely identify a location within the data model for at least one list of lookup values. While it uses table names generally, it has no way to support two or more lists within the same table. It also uses a 'MULTIPLE' string for a list of values that supports two tables. The two tables supported by 'MULTIPLE' string are the address and telephone tables.

The lab instructions have the students add the following three columns to the table:

  • common_lookup_table
  • common_lookup_column
  • common_lookup_code

Together the combination of the common_lookup_table and common_lookup_column columns create a non-unique super key. The super key identifies micro subtables. The combination of the common_lookup_table, common_lookup_column, and common_lookup_type columns creates a unique natural key that defines all possible values for a lookup list based on a column in a table.

The lab asked the students to use the existing data, string literal values, and some simple rules to populate the new common_lookup_table and common_lookup_column columns with data. The rules or steps were:

  1. Migrate valid table names from the common_lookup_context column to the common_lookup_table column.
  2. Migrate a literal 'ADDRESS' value into the common_lookup_table column when the common_lookup_context column holds a 'MULTIPLE' string value.
  3. Migrate valid table names from the common_lookup_context column to the common_lookup_column column by appending a '_TYPE' string literal to the common_lookup_context column values, except for those three rows that have a ‘VISA_CARD’, ‘MASTER_CARD’, or ‘DISCOVER_CARD’. The three exempted rows should update the common_lookup_column column with a 'CREDIT_CARD_TYPE' string literal.
  4. Migrate a literal 'ADDRESS_TYPE' value into the common_lookup_column column when the common_lookup_context column holds a 'MULTIPLE' string value.
  5. After these changes, insert two new rows in the common_lookup table. They should contain 'TELEPHONE' and 'TELEPHONE_TYPE' string literal values. One of the rows should contain 'HOME' for the common_lookup_type column value and the other 'WORK' for the same column.
  6. Then, the students were asked to update the foreign key column value in the telephone_type column of the telephone table.

It was a tremendous learning experience for those who did it because there were so many steps required to migrate the structure and data. Years ago, I would offer students half their final grade if they could complete the first four steps in a single UPDATE statement. Occasionally, I had students complete it. We worked through the problem with a small group of students today in one of my two weekly tutoring sessions. I thought it might be nice to document the solution, which use CASE operators in the SET clause of the UPDATE statement.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
UPDATE common_lookup
SET    common_lookup_table =
         CASE
           WHEN NOT common_lookup_context = 'MULTIPLE' THEN
             common_lookup_context
           ELSE
             'ADDRESS'
         END
,      common_lookup_column =
         CASE
           WHEN common_lookup_table = 'MEMBER' AND
                common_lookup_type IN ('VISA_CARD','MASTER_CARD','DISCOVER_CARD') THEN
             'CREDIT_CARD_TyPE'
           WHEN NOT common_lookup_context = 'MULTIPLE' THEN
             CONCAT(common_lookup_context,'_TYPE')
           ELSE
             'ADDRESS_TYPE'
         END;

As a rule, students would solve Step #6, which migrates the foreign key values of the telephone table’s telephone_type column to the new rows inserted into the common_lookup table. Most would accomplish that step with two UPDATE statements. Very few could see how to create a single UPDATE statement for both conditions and migrate from a now obsolete foreign key value that pointed to the rows of the address table’s rows in the common_lookup table to a valid foreign key value pointed to the telephone table’s rows in the common_lookup table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
UPDATE   telephone
SET      telephone_type =
           CASE
             WHEN common_lookup_type = 'HOME' THEN
               (SELECT common_lookup_id
                FROM common_lookup
                WHERE common_lookup_table = 'TELEPHONE'
                AND common_lookup_type = 'HOME')
             ELSE
               (SELECT common_lookup_id
                FROM common_lookup
                WHERE common_lookup_table = 'TELEPHONE'
                AND common_lookup_type = 'WORK')
             END
WHERE    telephone_type = 
           (SELECT common_lookup_id
            FROM common_lookup
            WHERE common_lookup_table = 'ADDRESS'
            AND common_lookup_type IN ('HOME','WORK');

As always, I hope this helps those looking for new ideas in SQL.

Written by maclochlainn

November 5th, 2020 at 11:03 pm

Quick Python Flask

without comments

A quick example of installing and running a sample hello.py file with the Flask application with the Command Line Interface (CLI) documentation on the Flask website. The first thing you need to do is install the flask module with the pip3 utility on Fedora Linux (or most other platforms):

pip3 install flask --user student

You should see a successful log like this for student user:

Requirement already satisfied: flask in /usr/local/lib64/python3.7/site-packages (1.1.2)
Collecting student
  Downloading https://files.pythonhosted.org/packages/b5/af/be416c18e4fe63a582e06fb0d47bf059bd0f4f413e5a6cfe893747ebaf79/Student-0.0.1-py3-none-any.whl
Requirement already satisfied: click>=5.1 in /usr/lib/python3.7/site-packages (from flask) (7.1.1)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.7/site-packages (from flask) (1.1.0)
Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.7/site-packages (from flask) (1.0.1)
Requirement already satisfied: Jinja2>=2.10.1 in /usr/lib/python3.7/site-packages (from flask) (2.10.1)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib64/python3.7/site-packages (from Jinja2>=2.10.1->flask) (1.1.1)
Installing collected packages: student
Successfully installed student-0.0.1

The smallest footprint hello.py program is:

# Import statement.
from flask import Flask
 
# Application defintion.
app = Flask(__name__)
 
# Basic URI rotuing.
@app.route("/")
 
# Define a hello function.
def hello():
  return "Hello World!"
 
# Run the program.
if __name__ == "__main__":
  app.run()

You can run the program from the command line with two commands:

export FLASK_APP=hello.py
flask run

It will show you the following in the terminal session:

 * Serving Flask app "hello.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Type the following URL in a local browser:

http://localhost:5000/

Flask will add the following to the console log:

127.0.0.1 - - [26/Oct/2020 00:37:49] "GET / HTTP/1.1" 200 -

You can jazz the hello.py program up with a an argument list, like this modified example based on this comparison article on Python frameworks:

# Import libraries.
from flask import Flask, escape, request
 
# Define the application.
app = Flask(__name__)
 
# Define a base URI route and function.
@app.route('/')
def index():
  return "Index Page"
 
# Define an application URI route and function.
@app.route("/hello")
def hello():
  name = request.args.get("name","Simon")
  return f'Hello {escape(name)}!'
 
# Define an about URI route and function.
@app.route("/about")
def about():
  return "About Page"
 
# Run the file.
if __name__ == "__main__":
  app.run()

It prints 'Hello Simon!' in a web page. If you try to stop your Flask server with a Ctrl+Z instead of a Ctrl+C, the next time you go to start it you will receive an error message like:

OSError: [Errno 98] Address already in use

You’ll need to find the Linux process ID and kill the process with prejudice. You can find the process with the following Linux command:

sudo netstat -nlp | grep 5000
[sudo] password for student:

It prompts you for your sudoer password, and then returns a line like:

tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      76802/python3

You kill the process with the following Linux command:

kill -9 76802

As always, I hope this helps those looking for the missing pieces.

Written by maclochlainn

October 26th, 2020 at 12:54 am

Wrap Oracle’s tnsping

without comments

If you’ve worked with the Oracle database a while, you probably noticed that some utilities write to stdout for both standard output and what should be standard error (stderr). One of those commands is the tnsping utility.

You can wrap the tnsping command to send the TNS-03505 error to stdout with the following code. I put Bash functions like these in a library.sh script, which I can source when automating tasks.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#!/usr/bin/bash
 
tnsping()
{
  if [ ! -z ${1} ]; then
    # Set default return value.
    stdout=`$ORACLE_HOME/bin/tnsping ${1} | tail -1`
 
    # Check stdout to return 0 for success and 1 for failure.
    if [[ `echo ${stdout} | cut -c1-9` = 'TNS-03505' ]]; then
      python -c 'import os, sys; arg = sys.argv[1]; os.write(2,arg + "\n")' "${stdout}"
    else
      echo "${1}"
    fi
  fi
}

You should notice that the script uses a Python call to redirect the error message to standard out (stdout) but you can redirect in Bash shell with the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#!/usr/bin/bash
 
tnsping()
{
  if [ ! -z ${1} ]; then
    # Set default return value.
    stdout=`$ORACLE_HOME/bin/tnsping ${1} | tail -1`
 
    # Check stdout to return 0 for success and 1 for failure.
    if [[ `echo ${stdout} | cut -c1-9` = 'TNS-03505' ]]; then
      echo ${stdout} 1>&2
    else
      echo "${1}"
    fi
  fi
}

Interactively, we can now test a non-existent service name like wrong with this syntax:

tnsping wrong

It’ll print the standard error to console, like:

TNS-03505: Failed to resolve name

or, you can suppress standard error (stderr) by redirecting it to the traditional black hole, like:

tnsping wrong 2>/dev/null

After redirecting standard error (stderr), you simply receive nothing back. That lets you evaluate in another script whether or not the utility raises an error.

In an automating Bash shell script, you use the source command to put the Bash function in scope, like this:

source library.sh

As always, I hope this helps those looking for a solution.

Written by maclochlainn

September 23rd, 2020 at 11:43 pm

SQL Developer JDK

without comments

In my classes, we use a VMware Linux install with SQL Developer. One of my students called me in a panic after an upgrade of packages when SQL Developer failed to launch. The student was astute enough to try running it from the command line where it generates an error like:

 Oracle SQL Developer
 Copyright (c) 2005, 2018, Oracle and/or its affiliates. All rights reserved.
 
/opt/sqldeveloper/sqldeveloper/bin/../../ide/bin/launcher.sh: line 954: [: : integer expression expected
The JDK (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.fc30.x86_64/) is not a valid JDK.
The JDK was specified by a SetJavaHome directive in a .conf file or by a --setjavahome option.
Type the full pathname of a JDK installation (or Ctrl-C to quit), the path will be stored in /home/student/.sqldeveloper/19.2.0/product.conf
 
Error:  Unable to get APP_JAVA_HOME input from stdin after 10 tries

The error is simple, the SQL Developer package update wipe clean the configuration of the SetJavaHome variable in the user’s ~/.sqldeveloper/19.2.0/product.conf file. The fix is three steps because its very likely that the Java packages were also updated. Here’s how to fix it:

  1. Navigate to the directory where you’ve installed the Java Virtual Machine (JVM) and find the current version of the JVM installed:

    cd /usr/lib/jvm
    ls java*

    It will return a set of files, like:

    java
    java-1.8.0
    java-1.8.0-openjdk
    java-1.8.0-openjdk-1.8.0.252.b09-0.fc30.x86_64
    java-openjdk
    jre
    jre-1.8.0
    jre-1.8.0-openjdk
    jre-1.8.0-openjdk-1.8.0.252.b09-0.fc30.x86_64
    jre-openjdk

  2. Navigate to your user’s product configuration file with this command:

    cd ~/.sqldeveloper/19.2.0
  3. Add the following line to the product.conf file:

    # SetJavaHome /path/jdk
    SetJavaHome /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-0.fc30.x86_64/

Now, you should be able to run it from the command line. The shortcut icon should also work if one was installed. Also, don’t forget to update your $JAVA_HOME variable in the master Bash resource file, or your local user’s .bashrc files.

As always, I hope this helps those looking for a quick solution.

Written by maclochlainn

September 22nd, 2020 at 12:08 pm

MySQL JSON Server

with one comment

A student question: Does JavaScript make context switching for web-based applications obsolete? Wow! I asked what that meant. He said, it means JavaScript replaces all other server-side programming languages, like PHP, C#, or Python. I asked the student why he believed that. His answer was that’s what two interviewing managers told him.

I thought it would be interesting to put the idea to a test. Below is a Node.js script that acts as a utility that queries the MySQL database with substitution variables in query. It also returns a standard out (stdout) stream of the MySQL query’s results. It also supports three flag and value pairs as arguments, and optionally writes the results of the MySQL query to a log file while still returning result as the stdout value. All errors are written to the standard error (stderr) stream.

The Node.js solution is completely portable between Windows and Linux. You can deploy it to either platform without any edits for Windows case insensitive Command-Line Interface (CLI). Clearly, Node.js offers a replacement for direct interaction with the .NET components in PowerShell. This appears to mean basic Linux shell or PowerShell knowledge is all that’s required to write and deploy JavaScript programs as server-side programming solutions. It means anything that you would have done with the .NET you can do with JavaScript. Likewise, you can replace PHP, C#, Python, or Ruby server-side scripts with JavaScript programs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
// Declare constants. 
const fs = require('fs') 
const util = require('util') 
const express = require('express') 
const mysql = require('mysql') 
const connection = mysql.createConnection({ 
  host: 'localhost', 
  user: 'student', 
  password: 'student', 
  database: 'studentdb' 
}) 
 
// Declare local variables for case insensitive use. 
var data = '' 
var buffer = Buffer.alloc(0) 
var path = '' 
 
// Declare default query variables dates. 
var startDate = new Date('1980-01-01') 
var endDate = new Date() 
 
// Set default endDate value as tomorrow. 
endDate.setDate(new Date().getDate() + 1) 
 
// Define a regular expression for valid file names. 
var regexp = /^([0-9a-zA-Z]+|[0-9a-zA-Z]+\.+[0-9a-zA-Z]{3})$/ 
 
// Assign dynamic variables from arguments. 
var argv = process.argv.slice(2) 
 
// Check for paired values, evaluate and assign them to local variables. 
if ((argv.length % 2) == 0) { 
 for (let i = 0; i < argv.length; i += 2) { 
   // Assign a file name to write to the output path. 
   if ((argv[i].toLowerCase() == '-f') && (regexp.test(argv[i+1]))) { 
     // Assign present working for Windows or Linux. 
     if (process.platform == 'win32') 
       path = '.\\' + argv[1] 
     else 
       path = './' + argv[1] 
   } 
   // Assign a start date from the input string. 
   else if (argv[i].toLowerCase() == '-b') { 
     startDate = new Date(argv[i+1]) 
   } 
   // Assign a end date from the input string. 
   else if (argv[i].toLowerCase() == '-e') { 
     endDate = new Date(argv[i+1]) 
   } 
 } 
} 
else { 
 console.error('Arguments must be in pairs: flag and value.') 
} 
 
// Define and run MySQL query. 
connection.query("SELECT   i.item_title " + 
                 ",        date_format(i.release_date,'%d-%M-%Y') AS release_date " + 
                 "FROM     item i JOIN common_lookup cl " + 
                 "ON       i.item_type = cl.common_lookup_id " + 
                 "WHERE    cl.common_lookup_type = 'BLU-RAY' " + 
                 "AND      i.release_date BETWEEN ? AND ? " + 
                 "ORDER BY i.release_date" 
                ,[startDate, endDate], function (err, result) { 
 if (err) { 
   console.error('Query contains error ...') 
   console.error('-> ' + err) 
 } 
 else { 
   // Prints the index value in the RowDataPacket. 
   for(let element in result) { 
     data += result[element].item_title + ', ' + result[element].release_date + '\n' 
   } 
   // Write file when data string is not empty. 
   if (data.length > 0 ) { 
     buffer = Buffer.alloc(data.length,data) 
 
     // Check for a defined path before writing a file. 
     if (path.length > 0) { 
       // Open the file. 
       fs.open(path, 'w', function(err, fd) { 
         if (err) { 
           console.error('Could not open [' + path + '] file [' + err + ']') 
         } 
         else { 
           // Write the file. 
           fs.write(fd, buffer, 0, buffer.length, null, function(err) { 
             if (err) 
               console.error('Error writing [' + path + '] file [' + err + ']') 
             fs.close(fd, function() { 
               if (fs.existsSync(path)) { 
                 process.exit(0) 
               } 
             }) 
           }) 
         }   
       }) 
     } 
     // Set standard out (stdout). 
     console.log(data) 
   } 
   else { 
     console.error('Query returned no rows.') 
   } 
 } 
}) 
 
// Close MySQL connection. 
connection.end()

You can call this code with the default values, like

node app.js

You can call this code with a user defined file name, and a custom start and end date values, like

node app.js -f output.csv -b '2001-01-01' -e '2004-12-31'

The latter command returns the following by querying my MySQL studentdb video store:

Star Wars II, 16-May-2002 
Harry Potter and the Chamber of Secrets, 28-May-2002 
Harry Potter and the Sorcerer's Stone, 28-May-2002 
Die Another Day, 03-June-2003 
Harry Potter and the Prisoner of Azkaban, 23-October-2004

As always, I hope this helps somebody trying to sort it out.

Written by maclochlainn

August 1st, 2020 at 1:05 am

Node.js Fedora Install

with 2 comments

I want to add the MEAN (MongoDB, Express.js, Angular.js, and Node.js) stack to my backend server development course. This post documents the installation and configuration of components on Fedora 30.

The first step requires installing the Node package. The Node package also contains the Node package manager (npm). You install the Node packages as the root user or as a sudoer user with the following command.

yum install -y npm

It should produce the following installation log:

Last metadata expiration check: 1:10:42 ago on Wed 08 Jul 2020 06:57:52 PM MDT.
Dependencies resolved.
================================================================================================================================
 Package                         Architecture          Version                                     Repository              Size
================================================================================================================================
Installing:
 npm                             x86_64                1:6.13.4-1.10.19.0.1.fc30                   updates                3.8 M
Installing dependencies:
 nodejs                          x86_64                1:10.19.0-1.fc30                            updates                 88 k
 nodejs-libs                     x86_64                1:10.19.0-1.fc30                            updates                9.1 M
Installing weak dependencies:
 nodejs-full-i18n                x86_64                1:10.19.0-1.fc30                            updates                7.3 M
 
Transaction Summary
================================================================================================================================
Install  4 Packages
 
Total download size: 20 M
Installed size: 91 M
Downloading Packages:
(1/4): nodejs-10.19.0-1.fc30.x86_64.rpm                                                         173 kB/s |  88 kB     00:00    
(2/4): nodejs-full-i18n-10.19.0-1.fc30.x86_64.rpm                                               2.8 MB/s | 7.3 MB     00:02    
(3/4): nodejs-libs-10.19.0-1.fc30.x86_64.rpm                                                    2.7 MB/s | 9.1 MB     00:03    
(4/4): npm-6.13.4-1.10.19.0.1.fc30.x86_64.rpm                                                   1.3 MB/s | 3.8 MB     00:02    
--------------------------------------------------------------------------------------------------------------------------------
Total                                                                                           4.9 MB/s |  20 MB     00:04     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: npm-1:6.13.4-1.10.19.0.1.fc30.x86_64                                                                   1/1 
  Preparing        :                                                                                                        1/1 
  Installing       : nodejs-libs-1:10.19.0-1.fc30.x86_64                                                                    1/4 
  Installing       : nodejs-full-i18n-1:10.19.0-1.fc30.x86_64                                                               2/4 
  Installing       : npm-1:6.13.4-1.10.19.0.1.fc30.x86_64                                                                   3/4 
  Installing       : nodejs-1:10.19.0-1.fc30.x86_64                                                                         4/4 
  Running scriptlet: nodejs-1:10.19.0-1.fc30.x86_64                                                                         4/4 
  Verifying        : nodejs-1:10.19.0-1.fc30.x86_64                                                                         1/4 
  Verifying        : nodejs-full-i18n-1:10.19.0-1.fc30.x86_64                                                               2/4 
  Verifying        : nodejs-libs-1:10.19.0-1.fc30.x86_64                                                                    3/4 
  Verifying        : npm-1:6.13.4-1.10.19.0.1.fc30.x86_64                                                                   4/4 
 
Installed:
  nodejs-1:10.19.0-1.fc30.x86_64           nodejs-full-i18n-1:10.19.0-1.fc30.x86_64     nodejs-libs-1:10.19.0-1.fc30.x86_64    
  npm-1:6.13.4-1.10.19.0.1.fc30.x86_64    
 
Complete!

After installing the Node package, you should use the Node package manager (npm) to install the Node Monitor nodemon. nodemon is a popular utility that automatically lets you restart Node programs when you make changes to the source code.

While npm is installed as part of the Node package, you must use npm to install the Node Monitor. The following command installs the nodemon globally on your Fedora system. The -g flag lets you install it globally, which is important when you manage package.json files.

npm install -g nodemon

You install nodemon globally but most of your web app or project files will be installed locally. Node is a different paradigm than building an Apache or IIS web application because Node provides a framework for you to build a web server.

Here’s a quick Hello World! example that I borrowed a JavaScript helloworld.js file from an excellent Web Development with Node & Express: Leveraging the JavaScript Stack by Ethan Brown. For those who haven’t worked with JavaScript in years, semicolons are optional now.

/* Construct a web server. */
const http = require('http')
const port = process.env.PORT || 3000
 
const server = http.createServer((req, res) => {
  res.writeHead(200, { 'Content-Type': 'text/plain' })
  res.end('Hello world!')
})
 
server.listen(port, () => console.log(`server started on port ${port}); ` +
                                      'press Ctrl-C to terminate...'))

I put this in /var/www/html/node directory, which is owned by the superuser, root. You need to start the server before accessing it from a browser. You can start the program with the following syntax as a privileged user:

node /var/www/html/node/helloworld.js

Then, you can use the localhost to access it with the following URL:

http://localhost:3000

It will display the following:

Next, you need to use the Node Package Manager (npm) to install the Express.js packages. You do that with the following syntax:

npm install -g express express-generator

It should produce a console out put like the following:

npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
/usr/local/bin/express -> /usr/local/lib/node_modules/express-generator/bin/express-cli.js
+ express@4.17.1
+ express-generator@4.16.1
added 60 packages from 42 contributors in 4.798s

After you install all the packages, you can inspect them with the following command. The packages are found in the /usr/local/lib/node_modules/express directory. The listing is generated from the package.json file on Fedora and Ubuntu Linux.

npm list -g

It should display something like this:

/usr/local/lib
├─┬ express@4.17.1
│ ├─┬ accepts@1.3.7
│ │ ├─┬ mime-types@2.1.27
│ │ │ └── mime-db@1.44.0
│ │ └── negotiator@0.6.2
│ ├── array-flatten@1.1.1
│ ├─┬ body-parser@1.19.0
│ │ ├── bytes@3.1.0
│ │ ├── content-type@1.0.4 deduped
│ │ ├── debug@2.6.9 deduped
│ │ ├── depd@1.1.2 deduped
│ │ ├─┬ http-errors@1.7.2
│ │ │ ├── depd@1.1.2 deduped
│ │ │ ├── inherits@2.0.3
│ │ │ ├── setprototypeof@1.1.1 deduped
│ │ │ ├── statuses@1.5.0 deduped
│ │ │ └── toidentifier@1.0.0
│ │ ├─┬ iconv-lite@0.4.24
│ │ │ └── safer-buffer@2.1.2
│ │ ├── on-finished@2.3.0 deduped
│ │ ├── qs@6.7.0 deduped
│ │ ├─┬ raw-body@2.4.0
│ │ │ ├── bytes@3.1.0 deduped
│ │ │ ├── http-errors@1.7.2 deduped
│ │ │ ├── iconv-lite@0.4.24 deduped
│ │ │ └── unpipe@1.0.0 deduped
│ │ └── type-is@1.6.18 deduped
│ ├─┬ content-disposition@0.5.3
│ │ └── safe-buffer@5.1.2 deduped
│ ├── content-type@1.0.4
│ ├── cookie@0.4.0
│ ├── cookie-signature@1.0.6
│ ├─┬ debug@2.6.9
│ │ └── ms@2.0.0
│ ├── depd@1.1.2
│ ├── encodeurl@1.0.2
│ ├── escape-html@1.0.3
│ ├── etag@1.8.1
│ ├─┬ finalhandler@1.1.2
│ │ ├── debug@2.6.9 deduped
│ │ ├── encodeurl@1.0.2 deduped
│ │ ├── escape-html@1.0.3 deduped
│ │ ├── on-finished@2.3.0 deduped
│ │ ├── parseurl@1.3.3 deduped
│ │ ├── statuses@1.5.0 deduped
│ │ └── unpipe@1.0.0
│ ├── fresh@0.5.2
│ ├── merge-descriptors@1.0.1
│ ├── methods@1.1.2
│ ├─┬ on-finished@2.3.0
│ │ └── ee-first@1.1.1
│ ├── parseurl@1.3.3
│ ├── path-to-regexp@0.1.7
│ ├─┬ proxy-addr@2.0.6
│ │ ├── forwarded@0.1.2
│ │ └── ipaddr.js@1.9.1
│ ├── qs@6.7.0
│ ├── range-parser@1.2.1
│ ├── safe-buffer@5.1.2
│ ├─┬ send@0.17.1
│ │ ├── debug@2.6.9 deduped
│ │ ├── depd@1.1.2 deduped
│ │ ├── destroy@1.0.4
│ │ ├── encodeurl@1.0.2 deduped
│ │ ├── escape-html@1.0.3 deduped
│ │ ├── etag@1.8.1 deduped
│ │ ├── fresh@0.5.2 deduped
│ │ ├── http-errors@1.7.2 deduped
│ │ ├── mime@1.6.0
│ │ ├── ms@2.1.1
│ │ ├── on-finished@2.3.0 deduped
│ │ ├── range-parser@1.2.1 deduped
│ │ └── statuses@1.5.0 deduped
│ ├─┬ serve-static@1.14.1
│ │ ├── encodeurl@1.0.2 deduped
│ │ ├── escape-html@1.0.3 deduped
│ │ ├── parseurl@1.3.3 deduped
│ │ └── send@0.17.1 deduped
│ ├── setprototypeof@1.1.1
│ ├── statuses@1.5.0
│ ├─┬ type-is@1.6.18
│ │ ├── media-typer@0.3.0
│ │ └── mime-types@2.1.27 deduped
│ ├── utils-merge@1.0.1
│ └── vary@1.1.2
└─┬ express-generator@4.16.1
  ├── commander@2.15.1
  ├── ejs@2.6.1
  ├─┬ minimatch@3.0.4
  │ └─┬ brace-expansion@1.1.11
  │   ├── balanced-match@1.0.0
  │   └── concat-map@0.0.1
  ├─┬ mkdirp@0.5.1
  │ └── minimist@0.0.8
  └── sorted-object@2.0.1

You can also create a secure node site (HTTPS) with the following additional steps. They include creating a self-signed secure public and private key. This creates the public key:

openssl genrsa -out key.pem

The openssl command will generate a private key key.pem file. It generates something like the following text message to console:

Generating RSA private key, 2048 bit long modulus (2 primes)
...........+++++
.............................................+++++
e is 65537 (0x010001)

Next, you need to generate a self-signed certificate. You do this in two steps.

  1. Create a Distinguished Name (DN) file. The csr.pem file is the DN file. You need it to create a self-signed certificate:

    openssl req -new -key key.pem -out csr.pem

    It will prompt you for values, like the following:

    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [XX]:
    State or Province Name (full name) []:MiddleEarth
    Locality Name (eg, city) [Default City]:Rivendell
    Organization Name (eg, company) [Default Company Ltd]:Fellowship    
    Organizational Unit Name (eg, section) []:Self
    Common Name (eg, your name or your server's hostname) []:Localhost
    Email Address []:bilbo@loth.org
     
    Please enter the following 'extra' attributes
    to be sent with your certificate request
    A challenge password []:friend
    An optional company name []:Bilbo
  2. Use the DN file to create your secure certificate. The following openssl command creates the certificate file by using your private key key.pem and DN csr.pem files.

    openssl x509 -req -days 9999 -in csr.pem -signkey key.pem -out cert.pem

    It should generate a Secure certificate cert.pem file and return something like the following to the console.

    Signature ok
    subject=C = XX, ST = MiddleEarth, L = Rivendell, O = Fellowship, OU = Self, CN = Localhost, emailAddress = bilbo@loth.org
    Getting Private key

You can put these private key (key.pem) and certificate (cert.pem) files in an ssl subdirectory of the directory where you put the JavaScript program. The following creates a secure server page with the following code.

/* Construct a secure web server. */ 
const https = require('https') 
const fs = require('fs') 
const port = process.env.PORT || 3000 
 
const options = { 
 key: fs.readFileSync('ssl/key.pem'), 
 cert: fs.readFileSync('ssl/cert.pem') 
} 
 
const server = https.createServer((options, res) => { 
 res.writeHead(200, { 'Content-Type': 'text/plain' }) 
 res.end('Hello world!') 
}) 
 
server.listen(port, () => console.log(`server started on port ${port}); ` + 
                                     'press Ctrl-C to terminate...'))

If you try launch your browser using the localhost instead of a DNS or file resolved network name on the designated port, it will raise the following security error:

This site can't be reached
https's server IP address could not be found.
DNS_PROBE_FINISHED_NXDOMAIN

An alternate approach to writing a secure server includes using Express.js library. The syntax changes somewhat and you include two new libraries, as shown below:

/* Construct a secure web server. */ 
const https = require('https') 
const express = require('express') 
const fs = require('fs') 
const app = express() 
const port = process.env.PORT || 3001 
 
const options = { 
 key:  fs.readFileSync('ssl/key.pem'), 
 cert: fs.readFileSync('ssl/cert.pem') 
} 
 
https.createServer(options, app).listen(port, () => { 
 console.log(`Express started in ${app.get('env')} mode ` + 
             `on port + ${port}.`) 
})

This will fail with the following error message if you’re running it with a global installation unless you set the $NODE_PATH environment variable correctly. Without setting the variable you may get the following error message:

internal/modules/cjs/loader.js:638 
   throw err; 
   ^ 
 
Error: Cannot find module 'express' 
   at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) 
   at Function.Module._load (internal/modules/cjs/loader.js:562:25) 
   at Module.require (internal/modules/cjs/loader.js:692:17) 
   at require (internal/modules/cjs/helpers.js:25:18) 
   at Object.<anonymous> (/var/www/html/node/helloworldsecure.js:3:17) 
   at Module._compile (internal/modules/cjs/loader.js:778:30) 
   at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10) 
   at Module.load (internal/modules/cjs/loader.js:653:32) 
   at tryModuleLoad (internal/modules/cjs/loader.js:593:12) 
   at Function.Module._load (internal/modules/cjs/loader.js:585:3)

Sometimes they’ll advise you to do an individual user installation of Express.js to get past this error but that’s not necessary. You just need to set the $NODE_PATH environment variable as follows:

export NODE_PATH=/usr/local/lib/node_modules

This will enable the JavaScript to work without error and without a specific user installation. Assuming you name either of these programs as helloworldsecure.js, you run them with the following command:

node helloworldsecure.js

You can terminate the program with a Ctrl+c or if use the kill -15 pid command if you started it as a background process. You can find the process ID (pid) with the jobs command.

As always, I hope this is helpful to those starting out with this cool technology stack.

Written by maclochlainn

July 8th, 2020 at 9:10 pm

Recursive bash function

without comments

While teaching a class on the Linux Command-Line (CLI), the book gave an example of generating a list of random US telephone numbers into a file. The book uses the RANDOM function to generate segments of the telephone number, and then the grep command to identify malformed telephone numbers.

My students wanted me to explain why the numbers were malformed. I had to explain that the RANDOM function returns a random number between 1 and 99,999. The RANDOM function may return a 1 to 5 digit random number, which means you may get a 1-digit or 2-digit number when you request a 3-digit random number or a 1- to 3-digit number when you request a 4-digit random number.

The author’s example is:

for i in {1..10}; do
  echo "(${RANDOM:0:3}) ${RANDOM:0:3}-${RANDOM:0:4}" >> list.txt
done

They asked if there was a way to write a shell script that guaranteed random but well-formed US telephone numbers. I said yes, however, you need to write a recursive bash shell function and assign the result to a global variable set in the shell script.

They seemed doubtful, so I wrote it for them. Here’s the script if you’re interested in learning more about bash shell scripting. While I implemented it with an bash array, that’s optional.

#!/usr/bin/bash
 
# ============================================================
#  Name:   telephone.sh
#  Author: Michael McLaughlin
#  Date:   05-May-2020
# ------------------------------------------------------------
#  Purpose: Demonstrate how to generate random telehpone
#           numbers. The RANDOM function returns a random
#           number between 1 and 99999; and while you can
#           easily shave off a extra digit guarnteeing a
#           value above 100 is impossible without logic.
# ============================================================
 
targetLength()
{
  # Declare variable in function-level scope.
  randomString=''
 
  # Check the number of parameters to process.
  if [[ ${#} = 2 ]]; then
    # Assign value to function-level and local variables.
    randomString=${1}
    formatLength=${2}
 
    # Get the length of the telephone number as integer.
    length=`echo -n ${randomString} | wc -c`
 
    # Calculate any shortfall.
    short=$((${formatLength}-${length}))
 
    # Check if the telephone number is too short.
    if [[ ${short} > 0 ]]; then
      randomString=`echo "${randomString}${RANDOM:0:${short}}"`
    fi
  fi
 
  # Check if the combination of random numbers equals the target length
  # and assign the value to the global variable, or repeat processing
  # by making a recursive function call.
  if [[ `echo -n ${randomString} | wc -c` = ${formatLength} ]]; then
    result=${randomString}
  else
    targetLength ${randomString} ${formatLength}
  fi
}
 
# Declare global variable to support targetLength().
result=''
 
# Declare an array of strings.
declare -A telephone_parts
 
# Generate one hundred random telephone numbers.
for i in {1..100}; do
  # Create random three digit area code.
  targetLength ${RANDOM:0:3} 3
  telephone_parts[1]=${result}
 
  # Create random three digit prefix code.
  targetLength ${RANDOM:0:3} 3
  telephone_parts[2]=${result}
 
  # Create random four digit number code.
  targetLength ${RANDOM:0:4} 4
  telephone_parts[3]=${result}
 
  # Print the telephone numbers.
  echo "[${i}] (${telephone_parts[1]}) ${telephone_parts[2]}-${telephone_parts[3]}"
done

For reference, a recursive function call isn’t required here. It could be done more effectively with the following while loop:

targetLength()
{
  # Declare variable in function-level scope.
  randomString=''
  short=1
 
  # Check the number of parameters to process.
  if [[ ${#} = 2 ]]; then
    # Assign value to function-level and local variables.
    randomString=${1}
    formatLength=${2}
 
    # Check if the telephone number is too short.
    while [[ ${short} > 0 ]]; do
      # Get the length of the telephone number as integer.
      length=`echo -n ${randomString} | wc -c`
 
      # Calculate any shortfall.
      short=$((${formatLength}-${length}))
 
      # Assign new value to randomString.
      randomString=`echo "${randomString}${RANDOM:0:${short}}"`
    done
 
    # Assign randomString to global result variable.
    result=${randomString}
  fi
}

As always, I hope this helps those you want to learn or solve a problem.

Written by maclochlainn

July 1st, 2020 at 11:55 pm