bspace notes

[This installation was so easy it may not seem worth the notes and output below. But over the years I've found it surprisingly useful to be able to refer back to notes such as this, and helpful to read others' detailed installation reports.]

---

On this page...

• Goal
• Sources of info
• Starting setup
• Plan
• Install
• Configure & test

---

Goal & motivation

• I'll be developing more and more web-apps that call services, and unless I call them all via ajax -- which might actually be a good idea -- I'll bump into the development server's single-threaded limitation. And even if I do call the services via ajax I'll likely run into problems if I'm calling more than one service simultaneously -- although the ability of an ajax call to fail gracefully via a timeout and try its resource again would be a cool thing to learn how to implement.

• Thinking about this off and on for the last few months, I recently came across a posting by the god of apache-python integration, Graham Dumpleton, about using mod_wsgi for development. This, on top of a slow solaris installation using mod_python (the slowness probably has more to do with the box than mod_python) give me the impetus to delve into this.

• Finally, gsf's encouragement to check out mod_wsgi at code4lib 2009 made me bite the bullet.

Sources of info

• General info
• Installation instructions
• OS X potential issues

Starting setup

• python version 2.5.1, via:

  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ python
  Python 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) 
  [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>>
  

• apache version 2.2.9, via phpinfo():

  apache2handler
  
  Apache Version  Apache/2.2.9 (Unix) mod_ssl/2.2.9 OpenSSL/0.9.7l DAV/2 PHP/5.2.6
  Apache API Version  20051115
  Server Administrator  you@example.com
  Hostname:Port ::1:0
  User/Group  www(70)/70
  Max Requests  Per Child: 0 - Keep Alive: on - Max Per Connection: 100
  Timeouts  Connection: 300 - Keep-Alive: 5
  Virtual Server  No
  Server Root /usr
  Loaded Modules  core prefork http_core mod_so mod_authn_file mod_authn_dbm mod_authn_anon mod_authn_dbd mod_authn_default mod_authz_host mod_authz_groupfile mod_authz_user mod_authz_dbm mod_authz_owner mod_authz_default mod_auth_basic mod_auth_digest mod_cache mod_disk_cache mod_mem_cache mod_dbd mod_dumpio mod_ext_filter mod_include mod_filter mod_deflate mod_log_config mod_log_forensic mod_logio mod_env mod_mime_magic mod_cern_meta mod_expires mod_headers mod_ident mod_usertrack mod_setenvif mod_version mod_proxy mod_proxy_connect mod_proxy_ftp mod_proxy_http mod_proxy_ajp mod_proxy_balancer mod_ssl mod_mime mod_dav mod_status mod_autoindex mod_asis mod_info mod_cgi mod_dav_fs mod_vhost_alias mod_negotiation mod_dir mod_imagemap mod_actions mod_speling mod_userdir mod_alias mod_rewrite mod_bonjour2 mod_php5
  

• Requirement: Docs say "The GNU C compiler from the MacOS X Developer Toolkit bundle is required." -- I should be fine; developer stuff is installed.

Plan

• Follow Quick installation guide, 'cause I've been living a good life recently.

Install

• Got source code from the specified download page.

• Selected single version listed (2.3)

• Unstuff

  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ mv /Users/birkin/Downloads/mod_wsgi-2.3.tar.gz /Developer_3rd/mod_wsgi-2.3.tar.gz
  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ cd /Developer_3rd/
  birkinbox:Developer_3rd birkin$ 
  birkinbox:Developer_3rd birkin$ /usr/bin/tar xvfz ./mod_wsgi-2.3.tar.gz
  mod_wsgi-2.3/
  mod_wsgi-2.3/configure
  mod_wsgi-2.3/configure.ac
  mod_wsgi-2.3/LICENCE
  mod_wsgi-2.3/Makefile-1.X.in
  mod_wsgi-2.3/Makefile-2.X.in
  mod_wsgi-2.3/mod_wsgi.c
  mod_wsgi-2.3/README
  birkinbox:Developer_3rd birkin$
  

• Configure

  birkinbox:Developer_3rd birkin$ 
  birkinbox:Developer_3rd birkin$ cd ./mod_wsgi-2.3/
  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ ls -alF
  total 952
  drwxr-xr-x@  9 birkin  admin     306 Aug 23  2008 ./
  drwxr-xr-x  20 birkin  admin     680 Mar  2 08:16 ../
  -rw-r--r--@  1 birkin  admin   11358 Jun 23  2007 LICENCE
  -rw-r--r--@  1 birkin  admin    1195 Dec 13  2007 Makefile-1.X.in
  -rw-r--r--@  1 birkin  admin    1247 Dec 13  2007 Makefile-2.X.in
  -rw-r--r--@  1 birkin  admin   16440 Mar 13  2008 README
  -rwxr-xr-x@  1 birkin  admin   78314 Dec 21  2007 configure*
  -rw-r--r--@  1 birkin  admin    4151 Jan 24  2008 configure.ac
  -rw-r--r--@  1 birkin  admin  352904 Aug 23  2008 mod_wsgi.c
  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ ./configure 
  checking for apxs2... no
  checking for apxs... /usr/sbin/apxs
  checking Apache version... 2.2.9
  checking for python... /usr/bin/python
  configure: creating ./config.status
  config.status: creating Makefile
  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ ls -alF
  total 1032
  drwxr-xr-x@ 13 birkin  admin     442 Mar  2 10:38 ./
  drwxr-xr-x  20 birkin  admin     680 Mar  2 08:16 ../
  -rw-r--r--@  1 birkin  admin   11358 Jun 23  2007 LICENCE
  -rw-r--r--   1 birkin  admin    1559 Mar  2 10:38 Makefile
  -rw-r--r--@  1 birkin  admin    1195 Dec 13  2007 Makefile-1.X.in
  -rw-r--r--@  1 birkin  admin    1247 Dec 13  2007 Makefile-2.X.in
  lrwxr-xr-x   1 birkin  admin      15 Mar  2 10:38 Makefile.in@ -> Makefile-2.X.in
  -rw-r--r--@  1 birkin  admin   16440 Mar 13  2008 README
  -rw-r--r--   1 birkin  admin    4474 Mar  2 10:38 config.log
  -rwxr-xr-x   1 birkin  admin   20621 Mar  2 10:38 config.status*
  -rwxr-xr-x@  1 birkin  admin   78314 Dec 21  2007 configure*
  -rw-r--r--@  1 birkin  admin    4151 Jan 24  2008 configure.ac
  -rw-r--r--@  1 birkin  admin  352904 Aug 23  2008 mod_wsgi.c
  birkinbox:mod_wsgi-2.3 birkin$
  

  So far so good.

• Build

  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ make
  /usr/sbin/apxs -c -I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -DNDEBUG -DMACOSX -DENABLE_DTRACE  -Wc,'-arch ppc7400' -Wc,'-arch ppc64' -Wc,'-arch i386' -Wc,'-arch x86_64' mod_wsgi.c -arch ppc7400 -arch ppc64 -arch i386 -arch x86_64 -Wl,-F/System/Library/Frameworks -framework Python -u _PyMac_Error -framework Python -ldl
  /usr/share/apr-1/build-1/libtool --silent --mode=compile gcc    -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -no-cpp-precomp  -I/usr/include/apache2  -I/usr/include/apr-1   -I/usr/include/apr-1  -arch ppc7400 -arch ppc64 -arch i386 -arch x86_64 -I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -DNDEBUG -DMACOSX -DENABLE_DTRACE  -c -o mod_wsgi.lo mod_wsgi.c && touch mod_wsgi.slo
  /usr/share/apr-1/build-1/libtool --silent --mode=link gcc -o mod_wsgi.la  -rpath /usr/libexec/apache2 -module -avoid-version    mod_wsgi.lo -arch ppc7400 -arch ppc64 -arch i386 -arch x86_64 -Wl,-F/System/Library/Frameworks -framework Python -u _PyMac_Error -framework Python -ldl
  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ ls -alF
  total 2648
  drwxr-xr-x@ 18 birkin  admin     612 Mar  2 11:26 ./
  drwxr-xr-x  20 birkin  admin     680 Mar  2 08:16 ../
  drwxr-xr-x   7 birkin  admin     238 Mar  2 11:26 .libs/
  -rw-r--r--@  1 birkin  admin   11358 Jun 23  2007 LICENCE
  -rw-r--r--   1 birkin  admin    1559 Mar  2 11:26 Makefile
  -rw-r--r--@  1 birkin  admin    1195 Dec 13  2007 Makefile-1.X.in
  -rw-r--r--@  1 birkin  admin    1247 Dec 13  2007 Makefile-2.X.in
  lrwxr-xr-x   1 birkin  admin      15 Mar  2 11:26 Makefile.in@ -> Makefile-2.X.in
  -rw-r--r--@  1 birkin  admin   16440 Mar 13  2008 README
  -rw-r--r--   1 birkin  admin    4474 Mar  2 11:26 config.log
  -rwxr-xr-x   1 birkin  admin   20621 Mar  2 11:26 config.status*
  -rwxr-xr-x@  1 birkin  admin   78314 Dec 21  2007 configure*
  -rw-r--r--@  1 birkin  admin    4151 Jan 24  2008 configure.ac
  -rw-r--r--@  1 birkin  admin  352904 Aug 23  2008 mod_wsgi.c
  -rw-r--r--   1 birkin  admin     800 Mar  2 11:26 mod_wsgi.la
  -rw-r--r--   1 birkin  admin     315 Mar  2 11:26 mod_wsgi.lo
  -rw-r--r--   1 birkin  admin  817180 Mar  2 11:26 mod_wsgi.o
  -rw-r--r--   1 birkin  admin       0 Mar  2 11:26 mod_wsgi.slo
  birkinbox:mod_wsgi-2.3 birkin$
  

  Smooth! Confirm that mod_wsgi.so file exists (the point of the build):

  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ ls -alF ./.libs/
  total 4432
  drwxr-xr-x   7 birkin  admin     238 Mar  2 11:26 ./
  drwxr-xr-x@ 18 birkin  admin     612 Mar  2 11:26 ../
  -rw-r--r--   1 birkin  admin  803768 Mar  2 11:26 mod_wsgi.a
  lrwxr-xr-x   1 birkin  admin      14 Mar  2 11:26 mod_wsgi.la@ -> ../mod_wsgi.la
  -rw-r--r--   1 birkin  admin     801 Mar  2 11:26 mod_wsgi.lai
  -rw-r--r--   1 birkin  admin  817100 Mar  2 11:26 mod_wsgi.o
  -rwxr-xr-x   1 birkin  admin  633872 Mar  2 11:26 mod_wsgi.so*
  birkinbox:mod_wsgi-2.3 birkin$
  

  Looks good.

• Install

  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ sudo make install
  Password:
  /usr/sbin/apxs -i -S LIBEXECDIR=/usr/libexec/apache2 -n 'mod_wsgi' mod_wsgi.la
  /usr/share/httpd/build/instdso.sh SH_LIBTOOL='/usr/share/apr-1/build-1/libtool' mod_wsgi.la /usr/libexec/apache2
  /usr/share/apr-1/build-1/libtool --mode=install cp mod_wsgi.la /usr/libexec/apache2/
  cp .libs/mod_wsgi.so /usr/libexec/apache2/mod_wsgi.so
  cp .libs/mod_wsgi.lai /usr/libexec/apache2/mod_wsgi.la
  cp .libs/mod_wsgi.a /usr/libexec/apache2/mod_wsgi.a
  ranlib /usr/libexec/apache2/mod_wsgi.a
  chmod 644 /usr/libexec/apache2/mod_wsgi.a
  ----------------------------------------------------------------------
  Libraries have been installed in:
     /usr/libexec/apache2
  
  If you ever happen to want to link against installed libraries
  in a given directory, LIBDIR, you must either use libtool, and
  specify the full pathname of the library, or use the `-LLIBDIR'
  flag during linking and do at least one of the following:
     - add LIBDIR to the `DYLD_LIBRARY_PATH' environment variable
       during execution
  
  See any operating system documentation about shared libraries for
  more information, such as the ld(1) and ld.so(8) manual pages.
  ----------------------------------------------------------------------
  chmod 755 /usr/libexec/apache2/mod_wsgi.so
  birkinbox:mod_wsgi-2.3 birkin$
  

  Confirm the mod_wsgi.so file has been installed in '/usr/libexec/apache2', as specified in this line of my Makefile:

  LIBEXECDIR = /usr/libexec/apache2
  
  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ ls -alF /usr/libexec/apache2
  total 81488
  drwxr-xr-x  72 root  wheel      2448 Mar  2 11:35 ./
  drwxr-xr-x  93 root  wheel      3162 Feb 12 16:44 ../
  (...)
  -rwxr-xr-x   1 root  wheel    633872 Mar  2 11:35 mod_wsgi.so*
  birkinbox:mod_wsgi-2.3 birkin$
  

  Nice, there it is at the bottom.

• Load module into apache

  I can never remember exactly where the httpd.conf file is.

  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ locate 'httpd.conf'
  (...)
  /private/etc/apache2/httpd.conf
  (...)
  birkinbox:mod_wsgi-2.3 birkin$
  

  First a backup:

  birkinbox:apache2 birkin$ sudo cp ./httpd.conf ./2009-03-02_httpd.conf
  

  Added to the LoadModule section:

  # Added 2009-03-02
  LoadModule wsgi_module libexec/apache2/mod_wsgi.so
  

• Restart

  birkinbox:apache2 birkin$ sudo apachectl restart
  

• Clean up

  birkinbox:apache2 birkin$ 
  birkinbox:apache2 birkin$ cd /Developer_3rd/mod_wsgi-2.3
  birkinbox:mod_wsgi-2.3 birkin$ 
  birkinbox:mod_wsgi-2.3 birkin$ make clean
  rm -rf .libs
  rm -f mod_wsgi.o mod_wsgi.la mod_wsgi.lo mod_wsgi.slo mod_wsgi.loT
  rm -f config.log config.status
  rm -rf autom4te.cache
  birkinbox:mod_wsgi-2.3 birkin$
  

Configure & test

• Reference: 'Configuring An Application'. Docs recommend following this to verify that mod_wsgi is actually working properly.

• Note: phpinfo() does indicate the module is loaded.

• Note: docs state to follow these QuickConfiguration instructions before delving into the more thorough configuration docs.

• Test app function & directories

  Created, per instructions, test function in a file and enclosing directory:

  birkinbox:repository birkin$ 
  birkinbox:repository birkin$ cd /Users/birkin/Documents/Brown_Library/ModWsgiTest 
  birkinbox:ModWsgiTest birkin$ 
  birkinbox:ModWsgiTest birkin$ ls -alF
  total 8
  drwxr-xr-x   3 birkin  staff   102 Mar  2 15:54 ./
  drwxr-xr-x  87 birkin  staff  2958 Mar  2 15:54 ../
  -rw-r--r--@  1 birkin  staff   277 Mar  2 15:43 mod_wsgi_test.wsgi
  birkinbox:ModWsgiTest birkin$ 
  birkinbox:ModWsgiTest birkin$ cat ./mod_wsgi_test.wsgi 
  def application(environ, start_response):
      status = '200 OK'
      output = 'Hello World!'
      response_headers = [('Content-type', 'text/plain'),
                          ('Content-Length', str(len(output)))]
      start_response(status, response_headers)
      return [output]
  birkinbox:ModWsgiTest birkin$
  

• httpd.conf file changes

  The docs give some sample configuration:

  <VirtualHost *:80>
  
    ServerName www.example.com
    ServerAlias example.com
    ServerAdmin webmaster@example.com
  
    DocumentRoot /usr/local/www/documents
  
    <Directory /usr/local/www/documents>
      Order allow,deny
      Allow from all
    </Directory>
  
    WSGIScriptAlias /myapp /usr/local/www/wsgi-scripts/myapp.wsgi
  
    <Directory /usr/local/www/wsgi-scripts>
      Order allow,deny
      Allow from all
    </Directory>
  
  </VirtualHost>
  

  My try will be:

  <VirtualHost *:80>
  
    ServerName 127.0.0.1
    ServerAlias 127.0.0.1
    ServerAdmin birkin_diana@brown.edu
  
    DocumentRoot /Users/birkin/Sites
  
    <Directory /Users/birkin/Sites>
      Order allow,deny
      Allow from all
    </Directory>
  
    WSGIScriptAlias /myapp /Users/birkin/Documents/Brown_Library/ModWsgiTest/mod_wsgi_test.wsgi
  
    <Directory /Users/birkin/Documents/Brown_Library/ModWsgiTest>
      Order allow,deny
      Allow from all
    </Directory>
  
  </VirtualHost>
  

• Backup

  birkinbox:~ birkin$ sudo cp /private/etc/apache2/httpd.conf /private/etc/apache2/2009-03-02b_httpd.conf
  

• Make change and restart

  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ sudo apachectl restart
  Password:
  birkinbox:~ birkin$
  

  Well, no errors on restart.

  Plain old 127.0.0.1 still yields the usual default page, which is good.

  Trying 'myapp'...

  birkinbox:~ birkin$ python
  Python 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) 
  [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> 
  >>> import urllib
  >>> 
  >>> urllib.urlopen( 'http://127.0.0.1/myapp/' ).read()
  '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don\'t have permission to access /myapp/\non this server.</p>\n</body></html>\n'
  >>>
  

  Ok... no permission, but it seems to recognize it as a valid web-address at least -- that's a start!

  Got it; problem is permissions on an enclosing folder...

  drwx------@  81 birkin  staff     2754 Feb  4 16:03 Documents/
  

• Test app function & directories #2

  Follow instructions this time and create a fully-accessible directory:

  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ mv /Users/birkin/Documents/Brown_Library/ModWsgiTest /Users/birkin/ModWsgiTest
  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ ls -alF /Users/birkin/
  total 1472
  (...)
  drwxr-xr-x    3 birkin  staff      102 Mar  2 17:03 ModWsgiTest/
  (...)
  birkinbox:~ birkin$
  

  Backup httpd.conf:

  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ ls -alF /private/etc/apache2/
  total 256
  (...)
  -rw-r--r--    1 root  wheel  17614 Mar  1  2008 2008-03-02_httpd.conf
  -rw-r--r--    1 root  wheel  17613 Mar  2 11:55 2009-03-02_httpd.conf
  -rw-r--r--    1 root  wheel  17685 Mar  2 16:59 2009-03-02b_httpd.conf
  (...)
  -rw-r--r--    1 root  wheel  18156 Mar  2 17:08 httpd.conf
  (...)
  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ sudo cp /private/etc/apache2/httpd.conf /private/etc/apache2/2009-03-02c_httpd.conf
  Password:
  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ ls -alF /private/etc/apache2/
  total 296
  (...)
  -rw-r--r--    1 root  wheel  17614 Mar  1  2008 2008-03-02_httpd.conf
  -rw-r--r--    1 root  wheel  17613 Mar  2 11:55 2009-03-02_httpd.conf
  -rw-r--r--    1 root  wheel  17685 Mar  2 16:59 2009-03-02b_httpd.conf
  -rw-r--r--    1 root  wheel  18156 Mar  2 18:40 2009-03-02c_httpd.conf
  (...)
  -rw-r--r--    1 root  wheel  18156 Mar  2 17:08 httpd.conf
  (...)
  birkinbox:~ birkin$
  

  Change httpd.conf; section now is:

  <VirtualHost *:80>
  
      ServerName 127.0.0.1
      ServerAlias 127.0.0.1
      ServerAdmin birkin_diana@brown.edu
  
      DocumentRoot /Users/birkin/Sites
  
      <Directory /Users/birkin/Sites>
          Order allow,deny
          Allow from all
      </Directory>
  
      WSGIScriptAlias /myapp /Users/birkin/ModWsgiTest/mod_wsgi_test.wsgi
  
      <Directory /Users/birkin/ModWsgiTest>
          Order allow,deny
          Allow from all
      </Directory>
  
  </VirtualHost>
  

• Restart & test

  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ sudo apachectl restart
  birkinbox:~ birkin$ 
  birkinbox:~ birkin$ python
  Python 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) 
  [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> 
  >>> import urllib
  >>> 
  >>> urllib.urlopen( 'http://127.0.0.1/myapp/' ).read()
  'Hello World!'
  >>>
  

  Success!!!

[These are notes from a project I worked on in grad-school in 2003-2004. As part of a 'voting' project, I wanted to automate the backup of a postgres database to an offsite location via a dump and rsync. In order to script the backup, my server needed to be able to automatically login to the backup server. A fellow student, J.E., and I worked on this piece together.

Recently a co-worker described a need to do something different, but similar in some ways, so I dug up these notes and pasted 'em in here, fairly raw. Note to hackers: the servers mentioned are long offline.]

---

Instructions

• Generate the key...

  [toolbox:~/Desktop] birkin% 
  [toolbox:~/Desktop] birkin% ssh-keygen -t rsa
  Generating public/private rsa key pair.
  Enter file in which to save the key (/Users/birkin/.ssh/id_rsa): 
  /Users/birkin/.ssh/id_rsa already exists.
  Overwrite (y/n)? y
  Enter passphrase (empty for no passphrase): 
  Enter same passphrase again: 
  Your identification has been saved in /Users/birkin/.ssh/id_rsa.
  Your public key has been saved in /Users/birkin/.ssh/id_rsa.pub.
  The key fingerprint is:
  71:04:1a:69:d4:ee:4a:d5:a8:b6:77:65:20:68:12:df birkin@toolbox.local
  [toolbox:~/Desktop] birkin%
  

  The '-t rsa' flag specifies ssh 2 protocol

• Examine the created keys...

  [toolbox:~/.ssh] birkin% 
  [toolbox:~/.ssh] birkin% ls -alF
  total 48
  drwx------ 7 birkin staff 238 16 Jun 21:15 ./
  drwxr-xr-x 57 birkin staff 1938 16 Jun 17:06 ../
  -rw------- 1 birkin staff 883 17 Jun 08:05 id_rsa
  -rw-r--r-- 1 birkin staff 230 17 Jun 08:05 id_rsa.pub
  -rw------- 1 birkin staff 535 16 Jun 20:39 identity
  -rw-r--r-- 1 birkin staff 339 16 Jun 20:39 identity.pub
  -rw-r--r-- 1 birkin staff 5351 16 Jun 20:56 known_hosts
  [toolbox:~/.ssh] birkin%
  

  The 'identity' files listed were generated when I was initially trying 'ssh -t rsa1', the ssh 1 protocol, and I believe can be ignored.

  [toolbox:~/.ssh] birkin% 
  [toolbox:~/.ssh] birkin% cat id_rsa.pub 
  ssh-rsa
  AAAAB3NzaC1yc2EAAAABIwAAAIEA0xmINQ6w3KGgxEexNJeb5bRDhOyp3R5zWfL6L5ghb8TqWDoF/x1e4KxoVp3NEMd594QISQzb4w74ZNkdGKnIqOEHs1Uy3zbutijsPQhWqXvZ40AMbOpOjawLAcrTWUfqmBcC7MW54cOiu2FIzvlHJhYVOBCyy1nBVduGJUPF5s=
  birkin@toolbox.local
  [toolbox:~/.ssh] birkin%
  

• Copy the public key to a file titled 'authorized_keys' which will be transferred to the remote computer(s) that I want to connect to.

  [toolbox:~/.ssh] birkin% 
  [toolbox:~/.ssh] birkin% cat id_rsa.pub > ~/Desktop/authorized_keys
  [toolbox:~/.ssh] birkin%
  

• Let's take a look to make sure it looks right...

  [toolbox:~/.ssh] birkin% 
  [toolbox:~/.ssh] birkin% cd ~/Desktop/
  [toolbox:~/Desktop] birkin% 
  [toolbox:~/Desktop] birkin% ls -alF
  total 64
  drwxr-xr-x 7 birkin staff 238 17 Jun 08:21 ./
  drwxr-xr-x 57 birkin staff 1938 16 Jun 17:06 ../
  -rwxr-xr-x 1 birkin staff 21508 17 Jun 08:20 .DS_Store*
  -rw-r--r-- 1 birkin staff 253 2 Nov 2003 .bash_profile
  -rw-r--r-- 1 birkin staff 0 20 Apr 2003 .localized
  -rw-r--r-- 1 birkin staff 230 17 Jun 08:21 authorized_keys
  drwxr-xr-x 42 birkin staff 1428 17 Jun 08:20 envelope/
  [toolbox:~/Desktop] birkin%         
  [toolbox:~/Desktop] birkin% cat authorized_keys 
  ssh-rsa
  AAAAB3NzaC1yc2EAAAABIwAAAIEA0xmINQ6w3KGgxEexNJeb5bRDhOyp3R5zWfL6L5ghb8TqWDoF/x1e4KxoVp3NEMd594QISQzb4w74ZNkdGKnIqOEHs1Uy3zbutijsPQhWqXvZ40AMbOpOjawLAcrTWUfqmBcC7MW54cOiu2FIzvlHJhYVOBCyy1nBVduGJUPF5s=
  birkin@toolbox.local
  [toolbox:~/Desktop] birkin%
  

  Looks good.

• Transfer the 'authorized keys' file from my OS X laptop to the remote computer...

  [toolbox:~/Desktop] birkin% 
  [toolbox:~/Desktop] birkin% rsync -v -e /usr/bin/ssh ~/Desktop/authorized_keys birkinbackup@harmonicas.msie.marlboro.edu:/home/birkinbackup/authorized_keys
  birkinbackup@harmonicas.msie.marlboro.edu's password: 
  authorized_keys
  wrote 316 bytes read 42 bytes 31.13 bytes/sec
  total size is 230 speedup is 0.64
  [toolbox:~/Desktop] birkin%
  

• Make sure it looks right on the remote computer...

  [toolbox:~/Desktop] birkin% 
  [toolbox:~/Desktop] birkin% ssh birkinbackup@harmonicas.msie.marlboro.edu
  birkinbackup@harmonicas.msie.marlboro.edu's password: 
  [birkinbackup@harmonicas birkinbackup]$ 
  [birkinbackup@harmonicas birkinbackup]$ ls -alF
  total 64
  drwx------ 3 birkinbackup birkinbackup 4096 Jun 17 08:27 ./
  drwxr-xr-x 6 root root 4096 Jun 12 15:01 ../
  -rw-r--r-- 1 birkinbackup birkinbackup 230 Jun 17 08:27 authorized_keys
  -rw------- 1 birkinbackup birkinbackup 6306 Jun 17 08:22 .bash_history
  -rw-r--r-- 1 birkinbackup birkinbackup 24 Jun 12 15:01 .bash_logout
  -rw-r--r-- 1 birkinbackup birkinbackup 191 Jun 12 15:01 .bash_profile
  -rw-r--r-- 1 birkinbackup birkinbackup 124 Jun 12 15:01 .bashrc
  -rw-r--r-- 1 birkinbackup birkinbackup 29 Jun 17 08:29 datecrontest
  -rw-r--r-- 1 birkinbackup birkinbackup 847 Jun 12 15:01 .emacs
  -rw-r--r-- 1 birkinbackup birkinbackup 120 Jun 12 15:01 .gtkrc
  drwx------ 2 birkinbackup birkinbackup 4096 Jun 17 00:20 .ssh/
  -rw-rw-r-- 1 birkinbackup birkinbackup 14220 Jun 12 18:59 testdump
  [birkinbackup@harmonicas birkinbackup]$ 
  [birkinbackup@harmonicas birkinbackup]$ cat authorized_keys 
  ssh-rsa
  AAAAB3NzaC1yc2EAAAABIwAAAIEA0xmINQ6w3KGgxEexNJeb5bRDhOyp3R5zWfL6L5ghb8TqWDoF/x1e4KxoVp3NEMd594QISQzb4w74ZNkdGKnIqOEHs1Uy3zbutijs+PQhWqXvZ40AMbOpOjawLAcrTWUfqmBcC7MW54cOiu2FIzvlHJhYVOBCyy1nBVduGJUPF5s=
  birkin@toolbox.local
  [birkinbackup@harmonicas birkinbackup]$
  

  Looks good.

• Move the file to the right place on the remote computer...

  [birkinbackup@harmonicas birkinbackup]$ 
  [birkinbackup@harmonicas birkinbackup]$ cat authorized_keys >> .ssh/authorized_keys 
  [birkinbackup@harmonicas birkinbackup]$
  

  The double brackets 'append' instead of overwrite. Also, I've checked this out -- the append is correct for our purposes in that it appends the new string on the following line. Actually, what would be nicer for inspection is this...

  [birkinbackup@harmonicas birkinbackup]$ 
  [birkinbackup@harmonicas birkinbackup]$ echo "" >> .ssh/authorized_keys 
  [birkinbackup@harmonicas birkinbackup]$ 
  [birkinbackup@harmonicas birkinbackup]$ cat authorized_keys >> .ssh/authorized_keys 
  [birkinbackup@harmonicas birkinbackup]$
  

  Let's check out the 'real' authorized_keys file (I should name the transfer file something else in the future to avoid any confusion)...

  [birkinbackup@harmonicas birkinbackup]$ 
  [birkinbackup@harmonicas birkinbackup]$ cd .ssh/
  [birkinbackup@harmonicas .ssh]$ 
  [birkinbackup@harmonicas .ssh]$ ls -alF
  total 24
  drwx------ 2 birkinbackup birkinbackup 4096 Jun 17 00:20 ./
  drwx------ 3 birkinbackup birkinbackup 4096 Jun 17 08:27 ../
  -rw-r--r-- 1 birkinbackup birkinbackup 975 Jun 17 08:42 authorized_keys
  -rw------- 1 birkinbackup birkinbackup 887 Jun 17 07:24 id_rsa
  -rw-r--r-- 1 birkinbackup birkinbackup 251 Jun 17 07:24 id_rsa.pub
  -rw-r--r-- 1 birkinbackup birkinbackup 603 Jun 16 17:34 known_hosts
  [birkinbackup@harmonicas .ssh]$ 
  [birkinbackup@harmonicas .ssh]$ cat authorized_keys 
  ssh-rsa
  AAAAB3NzaC1yc2EAAAABIwAAAIEAu4tdcJlZldiAAnfviR3vXWGjwWa4For/kbi/FvBTeTEtctxsS72/ppn5vFydv4V5iLDVdfWKrnTIwfn8BHinq2yvdX9OLsEyjzBqbu+ZIZCi7UefJxEWCdOGtDd0YWiJbQJkyuoHs4ShwF5YcuMcnmiEjOUWJ7B5N9QkXeD3wc0= birkinbackup@harmonicas.msie.marlboro.edu
  
  authorized_keys
  ssh-rsa
  AAAAB3NzaC1yc2EAAAABIwAAAIEA3+PWa9l6hu6sY43u5FASYr26AhRrUQDqcjT5VO+wePg2OaQyTedcNkRIGG6tVquFC+AXH5BOkI+EJAfSCJG2AE0YxSrM16rMgPM1wADJBlmhumiY5wuX5ROOc0azPpvLyjZwwFsSxgqpdtNtvwUCQEl94y3H5qqOvXtR+IVtp30= birkin@toolbox.local
  authorized_keys
  ssh-rsa
  AAAAB3NzaC1yc2EAAAABIwAAAIEA0xmINQ6w3KGgxEexNJeb5bRDhOyp3R5zWfL6L5ghb8TqWDoF/x1e4KxoVp3NEMd594QISQzb4w74ZNkdGKnIqOEHs1Uy3zbutijs+PQhWqXvZ40AMbOpOjawLAcrTWUfqmBcC7MW54cOiu2FIzvlHJhYVOBCyy1nBVduGJUPF5s= birkin@toolbox.local
  
  ssh-rsa
  AAAAB3NzaC1yc2EAAAABIwAAAIEA0xmINQ6w3KGgxEexNJeb5bRDhOyp3R5zWfL6L5ghb8TqWDoF/x1e4KxoVp3NEMd594QISQzb4w74ZNkdGKnIqOEHs1Uy3zbutijs+PQhWqXvZ40AMbOpOjawLAcrTWUfqmBcC7MW54cOiu2FIzvlHJhYVOBCyy1nBVduGJUPF5s= birkin@toolbox.local
  [birkinbackup@harmonicas .ssh]$
  

  The last line is the one we most recently created; the space preceding it is the result of the echo command; the lines 'authorized_keys' are mistakes from issuing echo in my experimentation instead of cat. I'm leaving these in to illustrate that there is tolerance for non-matching entries.

• Try connecting...

  [birkinbackup@harmonicas .ssh]$ 
  [birkinbackup@harmonicas .ssh]$ exit
  logout
  Connection to harmonicas.msie.marlboro.edu closed.
  [toolbox:~/Desktop] birkin% 
  [toolbox:~/Desktop] birkin% ssh birkinbackup@harmonicas.msie.marlboro.edu
  [birkinbackup@harmonicas birkinbackup]$
  

  No password-prompt: success!

Possible 'gotchas'

• Before actually trying a connection-script, run a manual ssh first; you may have to once manually ok that key-exchange message you normally see on a first-time ssh.

• If things aren't working, it could be a permissions issue...

  J.E. sent me a link [2008 note: this was in 2004] to http://kimmo.suominen.com/ssh/ and pointed out the caution to check file and directory permissions if connections still aren't working right after configuring everything.

  This site shows permissions to the ~/.ssh/ directory that allow writing by 'group', even though the text says only the 'owner' should have write permissions to that directory. On my account on the remote-computer, my ~/.ssh/ directory initially allowed group-write permissions, and passwordless login was not working. Changing those to...

  drwx------ 2 birkinbackup birkinbackup 4096 Jun 17 16:18 .ssh/
  

  ...allowed passwordless login to work. The beauteous text...

  [toolbox:~] birkin% 
  [toolbox:~] birkin% ssh birkinbackup@harmonicas.msie.marlboro.edu
  [birkinbackup@harmonicas birkinbackup]$ 
  [birkinbackup@harmonicas birkinbackup]$ ssh birkinbackup@play.msie.marlboro.edu
  [birkinbackup@play birkinbackup]$
  

  No password required. Sweet.

[This was first posted in 2006, to a no-longer-accessible wiki of mine, to accompany a Brown Internet Programming Group talk I gave. I'm slowly consolidating some of my posts and notes to this site.]

---

The problem

The situation: My preferred way of working is to program, on my laptop, code that often must communicate with a database running on a remote server. What are good ways of handling this? In the past I used two different approaches.

• Run a parallel database.
  • Pros: good when lots of database development/reconfiguration is required. No separate connection file is needed.
  • Cons: testing may lead to need to spend effort keeping database structures and sometimes data in sync.
• Access the remote database by programming the connection-code to determine which host its running on. If running on my laptop, the connection-code would locate the database at an internet address; if running on the same server as the database, the connection-code would locate the database at the localhost address.
  • Pros: Only need to deal with one database.
  • Cons: Non-localhost connectivity may be disabled for security reasons. If others work on the same code, the differing connection-code to detect multiple hosts can be a hassle and can reveal internet passwords. Care must be taken since the password may be transmitted over a non-secure connection.

Solution: Another programmer showed me how he solves this issue via ssh-tunneling. It's a wonderful solution.

Overview

Background info to keep in mind: Common client-server internet connections generally generally do not require specification of originating ports (the computers can pick a port), but do require specification of destination ports.

Example: my browser wants to access a web-page. My browser may send out the http request from any of a range of ports, but will specifically access the server's IP address at port 80, where the web-server is listening.

In ssh-tunneling, the client computer is set up to 'listen' for incoming data on a specified port at the 127.0.0.1 localhost IP address -- and to 'forward' that data, via a pre-established ssh connection, to a specific port at the server's IP address. This terminology may be a bit confusing, because the 'client' -- say, the local development laptop -- is 'listening', which a 'server' normally does. In this case, think of the server as a remote database-server.

What this means for my database situation is that I set up my laptop to listen for incoming data at '127.0.0.1:3306' and to forward that data to 'somehost.services.brown.edu:3306'. All I have to do in my connection code is specify that it attempt to connect to the database at '127.0.0.1:3306'.

The beauty of this is two-fold...

First, the same connection code can run on my laptop and on the server with no modification at all. Example of php connection code...

<?php
    mysql_connect("127.0.0.1:3306", "username", "password") or die ("Sorry, cannot connect to server");
    mysql_select_db("databasename") or die ("Sorry, cannnot connect to database");
?>

Second, because the 'set up' is using SSH as the fowarding mechanism, all data is transferred securely.

Note that there are many different ways of tunneling; this page focuses on one: 'local client' to 'remote service' (in this case, a remote database server).

Setting up the tunnel

Unices

On Linux, Unix, and the Mac, setting up a tunnel is as easy as issuing one command in a terminal window:

ssh -N -L 3306:somehost.services.brown.edu:3306 myaccount@somehost.services.brown.edu

Even if you're going to use a GUI client to set up the tunnel, examine the details of this command to get an understanding of what's going on:

• The ssh part is the normal secure-shell command.
• The -N flag specifies that commands flowing over this connection won't be executed on the remote computer, just forwarded.
• The -L flag specifies the details of the 'localPort:remoteHost:remotePort' section that follows this flag. It means that the local computer should listen for incoming connections on the specified localPort, and forward them over the ssh connection to the remote computer at the remotePort.
• The 'myaccount@somehost.services.brown.edu' sets up the ssh connection. This prompts me to enter my account-password on the remote computer 'somehost'.

Mac: Fugu

Fugu is an open-source ssh client that supports ssh tunneling.

To set up a tunnel in fugu:

• Select 'SSH' -> 'New SSH Tunnel'
• Enter 'somehost.services.brown.edu' in the 'Create Tunnel to' textbox'.
• Enter '3306' in the 'Service or Port' textbox (think of this as the 'Remote Port').
• Enter '3306' in the 'Local Port' textbox.
• Enter 'somehost.services.brown.edu' in the 'Remote Host' textbox.
  • I'm not sure of the distinction between the two 'host' textboxes, but entering info this way works.
• Enter your username in the 'Username' textbox.
• Enter '22' in the 'Port' textbox (think of this as the 'SSH Port').

Windows: Putty

Putty is a free Brown-offered ssh client that supports tunneling.

• Open 'putty.exe' file. The 'PuTTy Configuration' window appears.
• Click once on 'Category' -> 'Session'.
• On the right-side of the window enter 'somehost.services.brown.edu' in the 'Host Name (or IP address)' textbox.
• Enter '22' in the 'Port' textbox.
• Select 'SSH' for the 'Protocol'.
• Click once on 'Category' -> 'SSH' -> 'Tunnels'.
• Enter '3306' in the 'Source port' textbox.
• Enter 'somehost.services.brown.edu:3306' in the 'Destination' textbox.
• Select the 'Remote' radio-button under the 'Destination' textbox.
• Click the 'Add' button.
• Click the 'Open' button. The putty terminal window opens.
• When prompted by 'login as', enter your username and hit return.
• When prompted by for your password, enter it and hit return.
• That's it; the tunnel is established.

More tunnel fun

Web

The idea...

ssh -N -L 5005:123.123.123.123:80 myaccount@123.123.123.123

To implement this in Fugu or Putty, just switch the IP address, use '5005' for the 'Local Port' (Fugu) or 'Source Port' (Putty), and use '80' for the 'Service or Port' (Fugu) or the port following the host+colon in the 'Category' -> 'SSH' -> 'Tunnels' 'Destination' textbox (Putty).

You can then access a web page on the 123.123.123.123 server using an http://127.0.0.1:5005 address instead of the normal http://123.123.123.123 address.

Note that the 5005 'localPort' can really be any unused port above 1000. The only reason I keep the 'localPort' and 'remoteHostPort' the same in the database example is so my database connection code is the same and works the same on my development laptop and the actual database server.

Other

Note that these techniques can be applied in a wide variety of situations. * I've switched over to tunneled connections from Eclipse, my programming IDE, to my Subversion repositories. * Brown email is encrypted over the network, but if you have a home account that's not, you can check it from a coffee-shop unencrypted wireless network using ssh-tunneling.