Back in April 2008 I tried to get Django 0.96.1 working on Oracle and soon ran into some issues that clearly indicated to me it was a bit too early to invest into this framework. In fact, that experience, is what turned me heavily onto PHP/OCI8. But I kept Python in mind all this time because I really liked how structured the language was. PHP’s flexibility for WEB development is awesome, but it falls short as a scripting language when it comes to working with OS, networks and stuff we DBAs like to automate.
Few weeks ago I finally dived into python trying to adapt it as a replacement for some of the CORE shell scripts in my monitoring framework. I wanted to switch to SOA model where remote agents would talk to the management server via HTTP instead of SQL*Net. I quickly wrote the client side but then got stuck making a decision as to which language to write the server side component of SOA. Option a) was to continue using Oracle Application Server and mod_plsql, b) switch to PHP/Apache, c) adapt python.
After carefully weighting my options I decided to give python a chance since I wanted to learn what it’s capabilities were for the WEB applications. Django is the up and coming python framework, some even call it python application server. It recently went into production with v1, but it’s actually been around for over 4 years used as a private framework “by a fast-moving online-news operation“. It’s clean, easy to install and best of all it has great documentation and following.
I completed Django install in one day, which included compiling Python, cx_Oracle and mod_wsgi from source. All in all the process went fairly well but I kept wishing there was a good step by step writeup available online for getting it all setup with Oracle. There are couple of key compiler flags that make your life with python and Oracle a lot easier, but it was nowhere to be found and I learned about them by trial and error. I documented the whole process and attaching it here — hope this helps!
NOTE: This writeup assumes you have access to current Oracle Linux Support via ULN (linux.oracle.com) and your up2date was setup with ULN. It also assumes you are running 32bit Oracle Enterprise Linux or Red HAT equivalent:
[pyweb@oraemgc ~]$ cat /etc/issue
Enterprise Linux Enterprise Linux AS release 4 (October Update 5)
Kernel \r on an \m
[pyweb@oraemgc ~]$ uname -a
Linux oraemgc 2.6.9-55.0.0.0.2.ELsmp #1 SMP Wed May 2 14:59:56 PDT 2007 i686 i686 i386 GNU/Linux
Upgrade python to 2.6
First up2date tk and tcl pkgs to solve “INFO: Can’t locate Tcl/Tk libs and/or headers” error while compiling python:
up2date tk
up2date tcl
Download python source code:
wget http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tgz
gunzip Python-2.6.4.tgz
tar xvf Python-2.6.4.tar
Compile Python:
NOTE: see problems trying to build python 2.6 as a shared library and Shared Library Search Paths for explanation of LD_RUN_PATH.
cd Python-2.6.4
LD_RUN_PATH=/usr/local/lib; export LD_RUN_PATH
make clean
## shared is needed by mod_wsgi
./configure --enable-shared
make
make install
## FYI I ignored the following warning/error during make:
##
## Failed to find the necessary bits to build these modules:
## _sqlite3 _tkinter bsddb185
## sunaudiodev
## To find the necessary bits, look in setup.py in detect_modules() for the module's name.
##
## BEFORE:
##
[root@oraemgc Python-2.6.4]# python
Python 2.3.4 (#1, Jul 27 2009, 07:32:09)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-11.0.1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
[root@oraemgc Python-2.6.4]#
## AFTER UPGRADE:
##
[root@oraemgc Python-2.6.4]# python
Python 2.6.4 (r264:75706, Nov 24 2009, 19:31:16)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-8.0.1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
[root@oraemgc Python-2.6.4]#
Install Oracle Instant client RPMs (needed for cx_Oracle)
up2date oracle-instantclient11.1-basic
up2date oracle-instantclient11.1-devel
up2date oracle-instantclient11.1-sqlplus
install cx_Oracle
Download cx_Oracle-5.0.2.tar.gz from http://cx-oracle.sourceforge.net/ (click on “Source Code only” link once on that page and it’ll redirect you to that file).
NOTE: In order to avoid runtime errors such as “python: error while loading shared libraries: libpython2.6.so.1.0: cannot open shared object file: No such file or directory” I am setting LD_RUN_PATH during compilation of cx_Oracle library.
gunzip cx_Oracle-5.0.2.tar.gz
tar xvf 10826.cx_Oracle-5.0.2.tar
cd ./cx_Oracle-5.0.2
## LD_RUN_PATH will hardwire the path to libclntsh.so.11.1 into cx_Oracle.so
## at compile time similar to what I used with Python
##
export LD_RUN_PATH=/usr/lib/oracle/11.1/client/lib
rm -rf build
python setup.py build
python setup.py install
Quick test of cx_Oracle:
export ORACLE_HOME=/usr/lib/oracle/11.1/client
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH:.
export TNS_ADMIN=/u01/app/oracle/admin/scripts/mon/TNSADMIN
oraemgc.oracle-> sqlplus webproc/welcome@lmon
SQL*Plus: Release 11.1.0.7.0 - Production on Tue Nov 24 17:44:36 2009
Copyright (c) 1982, 2008, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>
oraemgc.oracle-> python
Python 2.6.4 (r264:75706, Nov 24 2009, 17:36:09)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-8.0.1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import cx_Oracle
>>>
>>> connection = cx_Oracle.connect("webproc", "welcome", "lmon")
>>>
>>> cursor = connection.cursor()
>>> cursor.execute("""
... select * from dual""")
<__builtin__.OracleCursor on >
>>> for column_1 in cursor:
... print "Values:", column_1
...
Values: ('X',)
>>>
Install Django
Create user that will own django software:
useradd pyweb
mkdir -p /u01/app/pyweb
chown pyweb:pyweb /u01/app/pyweb
## [root@oraemgc cx_Oracle-5.0.2]# grep pyweb /etc/passwd
## pyweb:x:502:504::/u01/app/pyweb:/bin/bash
##
su - pyweb
vi .profile
------ add the following -------
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
export ORACLE_HOME=/usr/lib/oracle/11.1/client
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH:.
export TNS_ADMIN=/u01/app/oracle/admin/scripts/mon/TNSADMIN
## create django directory
##
cd
mkdir -p product
cd product
## download latest django code from their SVN
##
svn co http://code.djangoproject.com/svn/django/trunk/
Configure django with Python. The steps below are largely based on http://docs.djangoproject.com/en/dev/topics/install/#database-installation
## verify where site pkgs are located
##
cd /u01/app/pyweb/product/django-trunk
[pyweb@oraemgc django-trunk]$ python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"
/usr/local/lib/python2.6/site-packages
## link django with python interpreter
## and
## create a symbolic link to the file django-trunk/django/bin/django-admin.py
##
## as root
##
ln -s /u01/app/pyweb/product/django-trunk/django /usr/local/lib/python2.6/site-packages/django
ln -s /u01/app/pyweb/product/django-trunk/django/bin/django-admin.py /usr/local/bin
## [root@oraemgc cx_Oracle-5.0.2]# ls -l /usr/local/lib/python2.6/site-packages/
## total 304
## -rw-r--r-- 1 root root 898 Nov 24 17:41 cx_Oracle-5.0.2-py2.6.egg-info
## -rwxr-xr-x 1 root root 297255 Nov 24 17:41 cx_Oracle.so
## lrwxrwxrwx 1 root root 42 Nov 24 18:07 django -> /u01/app/pyweb/product/django-trunk/django
## -rw-r--r-- 1 root root 119 Nov 24 17:40 README
## [root@oraemgc cx_Oracle-5.0.2]#
##
## [root@oraemgc cx_Oracle-5.0.2]# ls -l /usr/local/bin
## total 8772
## -rwxr-xr-x 1 root root 111 Nov 24 17:36 2to3
## -rwxr-xr-x 1 oracle root 2854 May 13 2008 coraenv
## -rwxr-xr-x 1 oracle root 2417 May 13 2008 dbhome
## lrwxrwxrwx 1 root root 62 Nov 24 18:09 django-admin.py -> /u01/app/pyweb/product/django-trunk/django/bin/django-admin.py
## -rwxr-xr-x 1 root root 99 Nov 24 17:36 idle
## -rwxr-xr-x 1 oracle root 2947 May 13 2008 oraenv
## -rwxr-xr-x 1 root root 84 Nov 24 17:36 pydoc
## -rwxr-xr-x 2 root root 4451815 Nov 24 17:39 python
## -rwxr-xr-x 2 root root 4451815 Nov 24 17:39 python2.6
## -rwxr-xr-x 1 root root 1424 Nov 24 17:40 python2.6-config
## lrwxrwxrwx 1 root root 16 Nov 24 17:40 python-config -> python2.6-config
## -rwxr-xr-x 1 root root 18054 Nov 24 17:36 smtpd.py
## [root@oraemgc cx_Oracle-5.0.2]#
##
NOTE: Whenever you want to update your copy of the Django source code, just run the command svn update from within the django-trunk directory. When you do this, Subversion will automatically download any changes.
Install mod_wsgi
based on http://code.google.com/p/modwsgi/
Note that ‘daemon’ mode of mod_wsgi is only available if Apache 2.0 or 2.2 is being used on a UNIX platform and where the Apache run time library has been compiled with support for multithreading. Generally multithreading support is standard except for on some BSD variants.
up2date httpd
[root@oraemgc mod_wsgi-3.0]# /usr/sbin/httpd -version
Server version: Apache/2.0.52
Server built: Nov 11 2009 19:32:25
[root@oraemgc mod_wsgi-3.0]#
Whatever version of Python is used, it must also have been compiled with
support for multithreading. To avoid a measure of memory bloat with your
Apache processes, Python should also have been compiled with shared library
support enabled. A number of Python binary packages for Linux systems are
not compiled with shared library support enabled. You should therefore
consider recompiling Python from source code with shared library support
enabled. If a shared library is not used, you will have problems trying
to use mod_wsgi on a server where mod_python is also being loaded.
FYI: I compiled python with “./configure –enable-shared” flag to support shared library and used LD_RUN_PATH=/usr/local/lib; export LD_RUN_PATH during compilation to avoid setting LD_LIBRARY_PATH during runtime (see “Upgrade python to 2.6” step above)
If using a Linux system and you do not have the appropriate Apache “dev”
package installed, then “apxs” will not be able to be found when
“configure” is run. Also ensure that you have installed the correct “dev”
package for the version of Apache being used. For most Linux distributions,
the “dev” package for Apache 2 is “apache2-dev” where the corresponding
Apache package was “apache2”. Some systems however distinguish the “dev”
package based on which MPM is used by Apache. As such, it may also be
called “apache2-worker-dev” or “apache2-prefork-dev”. If using Apache 2,
do not mix things up and install “apache-dev” by mistake, which is the
“dev” package for Apache 1.3 called just “apache”.
[root@oraemgc mod_wsgi-3.0]# up2date --whatprovides /usr/sbin/apxs
httpd-devel-2.0.52-41.ent.6.0.1.i386
up2date httpd-devel
Get mod_wsgi-3.0.tar.gz from http://code.google.com/p/modwsgi/:
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.0.tar.gz
gunzip mod_wsgi-3.0.tar.gz
tar xvf mod_wsgi-3.0.tar
cd mod_wsgi-3.0
make clean
./configure
make
make install
Apache Configuration:
[root@oraemgc mod_wsgi-3.0]# ls -l /etc/httpd/modules/mod_wsgi.so
-rwxr-xr-x 1 root root 290621 Nov 24 19:54 /etc/httpd/modules/mod_wsgi.so
vi /etc/httpd/conf/httpd.conf
--------- edit/add the following -----
LoadModule wsgi_module modules/mod_wsgi.so
LogLevel info
/etc/init.d/httpd stop
/etc/init.d/httpd start
Configure mod_wsgi for Django
NOTE: based on http://docs.djangoproject.com/en/dev/howto/deployment/modwsgi/#howto-deployment-modwsgi and http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango
NOTE: my project is called “evnt” you can use any name you desire just change it below:
su - pyweb
mkdir -p www_top
cd www_top
django-admin.py startproject evnt
## this created evnt directory and the following files:
##
## [pyweb@oraemgc www_top]$ ls -lta evnt/
## total 20
## drwxrwxr-x 2 pyweb pyweb 4096 Nov 24 20:20 .
## drwxrwxr-x 3 pyweb pyweb 4096 Nov 24 20:20 ..
## -rw-r--r-- 1 pyweb pyweb 0 Nov 24 20:20 __init__.py
## -rwxr-xr-x 1 pyweb pyweb 546 Nov 24 20:20 manage.py
## -rw-r--r-- 1 pyweb pyweb 2818 Nov 24 20:20 settings.py
## -rw-r--r-- 1 pyweb pyweb 538 Nov 24 20:20 urls.py
## [pyweb@oraemgc www_top]$
##
Setup custom directories to hold django/mod_wsgi configuration files
mkdir /u01/app/pyweb/www_top/evnt/apache
mkdir /u01/app/pyweb/www_top/evnt/media
vi /u01/app/pyweb/www_top/evnt/apache/django.wsgi
-------- add the following -----
import os, sys
sys.path.append('/u01/app/pyweb/www_top')
sys.path.append('/u01/app/pyweb/www_top/evnt')
os.environ['DJANGO_SETTINGS_MODULE'] = 'evnt.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
As root edit httpd.conf to configure your new django project with wsgi:
vi /etc/httpd/conf/httpd.conf
------------ add the following ----------
## EVNT django project
##
WSGIDaemonProcess oradba user=pyweb group=pyweb processes=2 threads=25 maximum-requests=10000
WSGISocketPrefix run/wsgi
Alias /evnt-media/ /u01/app/pyweb/www_top/evnt/media/
Order deny,allow
Allow from all
WSGIScriptAlias /evnt /u01/app/pyweb/www_top/evnt/apache/django.wsgi
Order deny,allow
Allow from all
WSGIProcessGroup oradba
Setup TNS_ADMIN for httpd:
vi /etc/sysconfig/httpd
--- add the following ---
TNS_ADMIN=/u01/app/oracle/admin/scripts/mon/TNSADMIN; export TNS_ADMIN
Bounce httpd
/etc/init.d/httpd restart
[root@oraemgc logs]# ps -ef | grep pyweb
pyweb 939 936 0 20:40 ? 00:00:00 /usr/sbin/httpd
pyweb 940 936 0 20:40 ? 00:00:00 /usr/sbin/httpd
pyweb 941 936 0 20:40 ? 00:00:00 /usr/sbin/httpd
root 1099 17430 0 20:40 pts/4 00:00:00 grep pyweb
[root@oraemgc logs]#
Test setup by going to http://yourhostname.com/evnt which should produce the following page:
End.
What’s next? Keep configuring your project by writing code to access Oracle, here’s a link to some examples which I am currently working on: Using the Django Framework with Oracle 11g.
One Comment
Vitaliy Mogilevskiy December 3, 2009
Posted In: python
Nice job. Especially the bit about: vi /etc/sysconfig/httpd
I was stuck and kept getting: Error loading cx_Oracle module: libclntsh.so.11.1: cannot open shared object file: No such file or directory
By adding the necessary value for LD_LIBRARY_PATH I was able to resolve my problem and get django working with Oracle 11g and Apache.
Thanks.