Friday, November 17, 2006

SBCL + UCW Installation Roadmap


The following is a rundown of the procedure we use to get our appliance up and running with SBCL and UCW on a stock minimal install of Fedora Core 6 x86. You can see the actual application in action if you go to Paragent.com and click on demo.

Preping the OS


After getting the initial system up and running, be sure to get the latest updates by running:

# yum -y install emacs
# yum -y upgrade

This will download and install any updates since the ISO images were created.

Next, you will want to make sure that the firewall is set up correctly

# system-config-securitylevel-tui

choose ssh, http, https if need be, and any other ports you might need. You will also want to make sure that SELinux is disabled. Otherwise, it interferes with the apache/mod_lisp communication with the lisp process. You can do this by going to /etc/selinux/config and setting

SELINUX=disabled

There may be a safer way to accomplish this, but I didn't feel like mucking about with SELinux. If anyone knows the proper way to set up the SELinux permissions, let me know.

SBCL


Next, you need to get sbcl installed. we typically use the latest version of sbcl from cvs. However, in order to build sbcl, you will need an initial copy that we can use to bootstrap the build process. As usual, yum is the easiest way to go here.

# yum -y install sbcl

Now, you can go about getting sbcl and building it. In this example, we are putting all of our lisp related items into /lisp

# mkdir /lisp
# cd /lisp
# cvs -d:pserver:anonymous@sbcl.cvs.sourceforge.net:/cvsroot/sbcl login
Cvs password: [return]
# cvs -z3 -d:pserver:anonymous@sbcl.cvs.sourceforge.net:/cvsroot/sbcl co -P sbcl
# cd sbcl
# emacs -nw customize-target-features.lisp

This last command is to create a file called customize-target-features.lisp that will tell the sbcl build process what extra features we would like. In this case, it is only threading support we are interested in. Put the following code into the file and save:

;; add threading support
(lambda (list) (cons :sb-thread list))

Now, we are ready to build sbcl:

# sh make.sh

and install it (assuming you have the priviledges here to do so):

# sh install.sh

now that we have a local version, get rid of the sbcl with which we bootstrapped the whole operation.

# yum remove sbcl

NOTE: for some reason, even though the new one is in the /usr/local/bin dir, which is in our path, it is still looking for it in /usr/bin. if you just type sbcl at the prompt without the absolute path, it won't find it. must be caching it somewhere.

[UPDATE] Zach pointed out in the comments that

hash -r

Will clear the path cache

ASDF-BINARY-LOCATIONS


Next, you will want to set up asdf-binary-locations. You can use asdf-install to get this package. In order to install it system-wide, you will want to be running as a user that has permissions to write to the sbcl site-systems directory (i.e. root)

# /usr/local/bin/sbcl
* (require 'asdf)
* (require 'asdf-install)
* (asdf-install:install 'asdf-binary-locations)

select 1 'System-wide' for where to install, and hit zero when the GPG error pops up

SPLIT-SEQUENCE


One handy library that gets used a bit is split sequence, so we go ahead and install it at this point as well.

* (asdf-install:install 'split-sequence)

again, select 1 for system-wide install and then 0 to skip gpg check.

you need to make sure that sbcl stuff doesn't get caught up in the asdf-binary-locations tom-foolery, so create a ~/.sbclrc file and put the following into it:

;;; -*- Lisp -*-
(require :asdf)
(require :asdf-binary-locations)
(defvar asdf::*source-to-target-mappings*
'(("/usr/local/lib/sbcl/" nil)))

SLIME


The next thing you will want to install is slime:

# cd /lisp
# cvs -d :pserver:anonymous@common-lisp.net:/project/slime/cvsroot login
Cvs password: anonymous
# cvs -z3 -d :pserver:anonymous@common-lisp.net:/project/slime/cvsroot co -P slime

To use slime, you will need to edit your .emacs file:

; .emacs setup for slime
(setq inferior-lisp-program "/usr/local/bin/sbcl")
(add-to-list 'load-path "/lisp/slime")
(require 'slime)
(slime-setup)

MySQL


We happen to use mysql as our back-end database, so we install it now if it isn't already on the system.

# yum install mysql
# yum install mysql-server
# yum install mysql-devel
# /etc/init.d/mysqld start

to get mysqld to restart on reboot properly, issue a

# chkconfig mysqld on

CLSQL


Now we want to install clsql. It has an additional dependency of uffi that we will also install at this time.

# cd /lisp
# wget http://files.b9.com/uffi/uffi-1.5.17.tar.gz
# wget http://www.paragent.com/downloads/clsql-3.6.3.tar.gz
# tar zxf uffi-1.5.17.tar.gz
# tar zxf clsql-3.6.3.tar.gz
# cd /usr/local/lib/sbcl/site-systems
# ln -s /lisp/uffi-1.5.17/uffi.asd .
# ln -s /lisp/clsql-3.6.3/clsql.asd .
# ln -s /lisp/clsql-3.6.3/clsql-mysql.asd .
# ln -s /lisp/clsql-3.6.3/clsql-uffi.asd .
# cd /lisp/clsql-3.6.3/db-mysql/
# make clean
# make

Additional Dependencies


Depending on what you are going to do, you probably don't need all of these, but I have included them here to show some extra things that you might want.

CL-BASE64


This is a dependency of cl-smtp, the email library we use.

# cd /lisp
# wget http://files.b9.com/cl-base64/cl-base64-latest.tar.gz
# tar zxf cl-base64-latest.tar.gz
# cd /usr/local/lib/sbcl/site-systems
# ln -s /lisp/cl-base64-3.3.2/cl-base64.asd .

CL-SMTP


cl-smtp allows us to send out email using normal mail servers. It even supports authentication to the server, so if your mail provider requires you to log in to send mail, this still works. Very handy for sending out automated announcements, etc.

# cd /lisp
# wget http://common-lisp.net/project/cl-smtp/cl-smtp.tar.gz
# tar zxf cl-smtp.tar.gz
# cd /usr/local/lib/sbcl/site-systems
# ln -s /lisp/cl-smtp/cl-smtp.asd .

Ironclad


If you are going to be doing any encryption type stuff, ironclad is a useful library. It is available via asdf-install, so from slime, you can:

CL-USER> (require 'asdf-install)
CL-USER> (asdf-install:install 'ironclad)

Remember that you need to have permission to write to your sbcl site-systems directory when you start up emacs if you want to be able to do a system-wide install.

UCW


This is the biggie. Luckily, they have packaged up everything you need in a handy boxset.

# cd /lisp
# wget http://common-lisp.net/project/ucw/ucw-boxset.tar.gz
# tar zxf ucw-boxset.tar.gz

Your Code Here


This is where you would put your code. Hopefully this will turn into a link to a ucw tutorial at some point.

Create MySQL Database


You might want to make some changes to you mysql configuration. A couple changes we make are in /etc/my.cnf. Add the following two lines to [mysqld] config:

lower_case_table_names=1
set-variable=max_connections=500

mod_lisp


You will next want to get mod_lisp2 and set it up (this is assuming apache 2.x). It does take a little bit of fiddling.

# cd /lisp
# mkdir mod_lisp
# cd mod_lisp
# wget http://www.fractalconcept.com:8000/public/open-source/mod_lisp/mod_lisp2.c

At the moment this file needs to be edited to take into account newer version of apache. The code you need to change is line 152-3 of mod_lisp2.c:

#if (APR_MAJOR_VERSION==1 && APR_MINOR_VERSION==2 && APR_PATCH_VERSION==2)

needs to be changed to accept anything over 1.2.2:

#if (APR_MAJOR_VERSION==1 && APR_MINOR_VERSION==2 && APR_PATCH_VERSION>=2)

Now you can install the apache development tools if you don't have them, and build the library:

# yum install httpd-devel
# apxs -ica mod_lisp2.c

Add the following in /etc/httpd/conf/httpd.conf

#
# mod_lisp2 configuration
#

<LocationMatch "/.*\.ucw">
LispServer 127.0.0.1 3001 ucw
SetHandler lisp-handler
</LocationMatch>

<LocationMatch "/.*\.csv">
LispServer 127.0.0.1 3001 ucw
SetHandler lisp-handler
</LocationMatch>

Now, restart the apache server

# apachectl restart
# chkconfig httpd on

You will now want to make sure ucw is using mod_lisp. edit /lisp/ucw-boxset/ucw_dev/src/vars.lisp

;;; `ucw:create-server' variables
(defvar *ucw-backend-type* :mod-lisp)
(defvar *ucw-backend-host* "127.0.0.1")
(defvar *ucw-backend-port* 3001)
(defvar *ucw-server-class* 'standard-server)

detachtty


One useful program included in the ucw-boxset is detachtty, which allows you to spin off your lisp process and then come back to it later.

# cd /lisp/ucw-boxset/dependencies/detachtty
# make
# make install

Labels: , , , ,

Wednesday, November 15, 2006

quote of the day



Overheard on c.l.l:
>> Live anywhere else in the world and the internet is a useful resource. You can for instance find things about where you live by using a search engine. When I lived in cambridge I could type things like "fish and chip shops cambridge" or "curtains cambridge" and get useful results. If I do the same in Limerick all I get is ribald poetry. The only way of finding anything quickly is the phone book.

>try googling: limerick curtains -"there -once -was" should eliminate most of the ribald poetry ;^)

Wow, that mostly works, thanks.

Labels: ,

Tuesday, November 14, 2006

at your service



We had a couple off site trips today. One was a planned install, the other was to go check out a server that was having problems, and didn't yet have remote access setup. We quickly figured out what was causing the web server grief, which we happened to have a fix for, in addition to a couple additional fixes I had brought with me on a thumb drive. That is when we decided to do some extreme programming on-the-spot. The customer had a feature request to collect some information about the services that were installed on their monitored machines. We reloaded the web and back-end services in slime, and started hacking away with the customer watching over our shoulders. Jasko and I tag teamed on the keyboard, and in about 30 minutes had it knocked out and running.

There are a couple takeaways for me from the experience.

First: emacs. despite the fact that Jasko prefers Eclipse to emacs, we had a fully functioning development environment running on a crappy VGA terminal window with only a keyboard. emacs, M-x slime and we were off and running. Sure, having more code visible on screen would have been very nice, but it was perfectly usable.

Second: macros. A major aspect of why it took us so little time to add the new functionality was that the hard work had been done once with the creation of macros, and we just drafted behind. The entire effort in adding the data collection to the back-end was three lines:

(defpropquery query-services
"service" ("name" "description")
'services "service" (:name :description))

What you don't see is the 20-line macro that does all the magic in the background. It takes those three lines, and creates the code to query the remote agent, collect up the results, update the database, see if there were any changes to the properties, and log any events that might have occurred. The same thing was true in the database code, as well as the ucw components for the front-end.

Labels: ,