If you use virtualenv to isolate your python project’s environment, and want your code tested automatically – read on!
virtualenv isolates your project’s python environment
virtualenv makes sure you lockdown your project’s main directory and all subdirectories of it. This ‘lockdown’ means that you never touches your global python binary, or any globally installed libraries (like “sudo pip install flask” ).
Once locked down, you install all packages again, even those you have globally installed. This enables you to have one version of flask globally installed, but another version in your project. All dependencies can be listed in a separate file and validate a precise environment for you to work with. Tightly controlled dependencies is key to a deployment without surprises.
Jenkins checks the health of your project for each change
Jenkins is a CI server which means it does a lot of repeating stuff so you can focus on doing more important stuff. More specifically, it listens for changes to your project’s version control system (like git).
When changes are detected, the project is built and the test suite is executed. If any step fails, the CI server tells you that it did.
Setup Jenkins, and make it use virtualenv
Jenkins needs some massaging before it handles the hijacked environment of virtualenv. This is how I did it for my local git repository:
Source Code Management: add the URI to your local repository, /Users/you/Sites/asdf in my case. Make sure the jenkins user can read this directory, otherwise the Jenkins GUI will tell you something random about invalid git repo, without a hint about a permissions error.
Build Triggers: Poll SCM (with an interval like 0 * * * *). This is needed because
you’re too lazy to build manually; and
you can not trigger builds with a git post-commit hook otherwise
Build > Execute shell. I’ve used two steps, one for setting up the environment and one for the actual tests:
# Setup a proper path, I call my virtualenv dir "venv" and# I've got the virtualenv command installed in /usr/local/binPATH=$WORKSPACE/venv/bin:/usr/local/bin:$PATHif[ ! -d "venv"]; thenvirtualenv venv
pip install -r requirements.txt --download-cache=/tmp/$JOB_NAME
Trigger Jenkins through git hook
You need to add a git hook which triggers a Jenkins build:
How to properly read command line arguments for a bash script and do it well even without getopt:
while :; do case"$1" in
-h|--help)echo"This is helping"exit 0
-s|--simple-flag)echo"You passed a flag"shift ;;
-p|--pass-option)shift echo"You set option to $1"option="$1"shift ;;
-*)echo"That's a weird option"exit 1
*)# No more args to readbreak ;;
I think the syntax is much cleaner than that of getopt’s as well – that is, if you’ve even got a bash version supporting getopt.
I have all my code stored in ~/Sites and every sub folder represents a project. Each folder is also a git repository. This means there’s history attached to it, which could be much more easily navigated than through the finder or a terminal. Some of the projects are mirrored on github but it’s too messy to keep them all there, although – that kind of overview is nice to have.
My solution to getting an overview was to setup gitweb locally, which makes for a very fast solution. This is how you do it on your OS X-machine:
Find where git’s installed at. I use homebrew, which means git (by default) resides in /usr/local/Cellar/git.
Grab the gitweb cgi, copy it to where your local Apache server can find it:
--- /usr/local/Cellar/git/18.104.22.168/share/gitweb/gitweb.cgi 2012-07-11 20:23:10.000000000 +0200+++ /Library/WebServer/CGI-Executables/gitweb.cgi 2012-07-11 20:50:36.000000000 +0200@@ -71,11 +71,11 @@ # absolute fs-path which will be prepended to the project path
#our $projectroot = "/pub/scm";
-our $projectroot = "/pub/git";+our $projectroot = "/Users/my-user/Sites"; # fs traversing limit for getting project list
# the number is relative to the projectroot
-our $project_maxdepth = 2007;+our $project_maxdepth = 5; # string of the home link on top of all pages
our $home_link_str = "projects";
@@ -95,15 +95,15 @@ our $site_footer = "";
# URI of stylesheets
-our @stylesheets = ("static/gitweb.css");+our @stylesheets = ("/gitweb/static/gitweb.css"); # URI of a single stylesheet, which can be overridden in GITWEB_CONFIG.
our $stylesheet = undef;
# URI of GIT logo (72x27 size)
-our $logo = "static/git-logo.png";+our $logo = "/gitweb/static/git-logo.png"; # URI of GIT favicon, assumed to be image/png type
#our $logo_url = "http://www.kernel.org/pub/software/scm/git/docs/";
Now, visit & bookmark http://localhost/cgi-bin/gitweb.cgi in your browser and you should be good to go.
A common pattern while editing is to yank anything (i.e. yy), visually select what you want to remove (i.e. vj) and paste (p). This will in turn place the overwritten test in the yank register; but we wanted to paste that same line again!
Type :reg to see all currently populated registers, they’re all preceded by ". This means that you can paste by visually selecting and typing "0p if “0 is the register’s content you want to paste.
Modify the theme a bit (far too many h1 used, for example)
Get a pattern from subtlepatterns.com
Upload it to a new /var/www/directory-here, add a virtual host section to apache’s config with ServerName test.iamnearlythere.com. Forgot to register test.iamnearlythere.com as an A record at my DNS registrar, 30 min wasted.
Install nginx and set the default mimetype to text/html since I was silly enough to use the URI form of url.com/post-title which made mynt generate posts as files without extensions which by default are served as application/octet-stream by nginx (“want to download this?”) and served as text/plain by apache.
The request dropped to 6 requests at 190 kb, totalling 1.47 s.
Since the machine now no longer needs apache or mysql:
Make sure apache doesn’t start on reload: $ sudo update-rc.d -f apache2 remove
Make sure mysqld doesn’t start on boot: http://superuser.com/a/139059/9539
Make sure nginx does start on boot: http://articles.slicehost.com/2007/10/17/ubuntu-lts-adding-an-nginx-init-script (I had to replace the nginx path with /usr/local/nginx/sbin/nginx)
Allright, I’m done.. ish. When using $ tail -f /usr/local/nginx/logs/error.log I notice a bunch of stuff I missed. Redirections here we go
Also, create a Makefile for generating, testing and deploying the blog (mynt generate; rsync)
Memory on vps could be turned down from 768 MB to 128, without tweaking nginx. Currently using about 20%.
$ ab -n 10000 -c 25 can be run without any problems what so ever
I’m pretty sure I suck at micromanaging apache & mysql. And that nginx is pretty fly with static files.
Also, a hiccup: I issued a 301 redirect in one case, and it went badly – my nginx non-ninja-config-skils made me issue a recursive redirect loop. 301 responses are heavily cached by browsers, I had to delete the cache after removing the direction out of the config file. Do not make this mistake during ‘live hours’ or with a heavily trafficked page.