If you use virtualenv to isolate your python project’s environment, and want your code tested automatically – read on!
virtualenv isolates your project’s python environment
virtualenv makes sure you lockdown your project’s main directory and all subdirectories of it. This ‘lockdown’ means that you never touches your global python binary, or any globally installed libraries (like “sudo pip install flask” ).
Once locked down, you install all packages again, even those you have globally installed. This enables you to have one version of flask globally installed, but another version in your project. All dependencies can be listed in a separate file and validate a precise environment for you to work with. Tightly controlled dependencies is key to a deployment without surprises.
Jenkins checks the health of your project for each change
Jenkins is a CI server which means it does a lot of repeating stuff so you can focus on doing more important stuff. More specifically, it listens for changes to your project’s version control system (like git).
When changes are detected, the project is built and the test suite is executed. If any step fails, the CI server tells you that it did.
Setup Jenkins, and make it use virtualenv
Jenkins needs some massaging before it handles the hijacked environment of virtualenv. This is how I did it for my local git repository:
Source Code Management: add the URI to your local repository, /Users/you/Sites/asdf in my case. Make sure the jenkins user can read this directory, otherwise the Jenkins GUI will tell you something random about invalid git repo, without a hint about a permissions error.
Build Triggers: Poll SCM (with an interval like 0 * * * *). This is needed because
you’re too lazy to build manually; and
you can not trigger builds with a git post-commit hook otherwise
Build > Execute shell. I’ve used two steps, one for setting up the environment and one for the actual tests:
12345678
# Setup a proper path, I call my virtualenv dir "venv" and# I've got the virtualenv command installed in /usr/local/binPATH=$WORKSPACE/venv/bin:/usr/local/bin:$PATHif[ ! -d "venv"]; thenvirtualenv venv
fi. venv/bin/activate
pip install -r requirements.txt --download-cache=/tmp/$JOB_NAME
and
12
. venv/bin/activate
python test_app.py
Trigger Jenkins through git hook
You need to add a git hook which triggers a Jenkins build:
Consider this a checklist for your API, should work for REST, CLI or source code!
What the interface is for
Have a scope and tell me what it is in a couple of sentences.
How to
Let me learn the general approach and edge cases via a concise and result oriented documentation.
Feedback
Scream at me when I did something wrong and, if I was close enough (and you have a lot of time), point me in the right direction.
Consistency
For example, when treating errors you shouldn’t alternate between simultaneous ways of telling your users something went wrong. Pick one and go with it.
How to properly read command line arguments for a bash script and do it well even without getopt:
12345678910111213141516171819202122232425
while :; do case"$1" in
-h|--help)echo"This is helping"exit 0
;;
-s|--simple-flag)echo"You passed a flag"shift ;;
-p|--pass-option)shift echo"You set option to $1"option="$1"shift ;;
-*)echo"That's a weird option"exit 1
*)# No more args to readbreak ;;
done
I think the syntax is much cleaner than that of getopt’s as well – that is, if you’ve even got a bash version supporting getopt.
Screen could be used as nohup with the additional possibility of attaching to the terminal session again, once your ssh session was aborted (or equivalent).
screen
Create a new screen
screen -S name
Create a new screen named “name”
screen -ls
List screens
screen -D
Detach from screen
screen -r
Attach to the single existing screen
screen -r 4262
Attach to the detached screen with pid 4262
screen -x 4262
Reattach to the attached screen with pid 4262
ctrl+a c
Create a new terminal in the current screen
ctrl+a H
Log all output of current screen to a log file
ctrl+a space
Change terminals within current screen
ctrl+a :sessionname name
Rename current screen to “name”
ctrl+a d
Detach from current screen, like exit but keeps the terminal for reattaching later on
I have all my code stored in ~/Sites and every sub folder represents a project. Each folder is also a git repository. This means there’s history attached to it, which could be much more easily navigated than through the finder or a terminal. Some of the projects are mirrored on github but it’s too messy to keep them all there, although – that kind of overview is nice to have.
My solution to getting an overview was to setup gitweb locally, which makes for a very fast solution. This is how you do it on your OS X-machine:
Find where git’s installed at. I use homebrew, which means git (by default) resides in /usr/local/Cellar/git.
Grab the gitweb cgi, copy it to where your local Apache server can find it:
Modify gitweb.cgi to let it find the static files, to find the git projects and to not choke on really large directories that I’m sure does not contain anything git’ anyway:
--- /usr/local/Cellar/git/1.7.11.1/share/gitweb/gitweb.cgi 2012-07-11 20:23:10.000000000 +0200+++ /Library/WebServer/CGI-Executables/gitweb.cgi 2012-07-11 20:50:36.000000000 +0200@@ -71,11 +71,11 @@ # absolute fs-path which will be prepended to the project path
#our $projectroot = "/pub/scm";
-our $projectroot = "/pub/git";+our $projectroot = "/Users/my-user/Sites"; # fs traversing limit for getting project list
# the number is relative to the projectroot
-our $project_maxdepth = 2007;+our $project_maxdepth = 5; # string of the home link on top of all pages
our $home_link_str = "projects";
@@ -95,15 +95,15 @@ our $site_footer = "";
# URI of stylesheets
-our @stylesheets = ("static/gitweb.css");+our @stylesheets = ("/gitweb/static/gitweb.css"); # URI of a single stylesheet, which can be overridden in GITWEB_CONFIG.
our $stylesheet = undef;
# URI of GIT logo (72x27 size)
-our $logo = "static/git-logo.png";+our $logo = "/gitweb/static/git-logo.png"; # URI of GIT favicon, assumed to be image/png type
-our $favicon = "static/git-favicon.png";+our $favicon = "/gitweb/static/git-favicon.png"; # URI of gitweb.js (JavaScript code for gitweb)
-our $javascript = "static/gitweb.js";+our $javascript = "/gitweb/static/gitweb.js"; # URI and label (title) of GIT logo link
#our $logo_url = "http://www.kernel.org/pub/software/scm/git/docs/";
Now, visit & bookmark http://localhost/cgi-bin/gitweb.cgi in your browser and you should be good to go.
A common pattern while editing is to yank anything (i.e. yy), visually select what you want to remove (i.e. vj) and paste (p). This will in turn place the overwritten test in the yank register; but we wanted to paste that same line again!
Type :reg to see all currently populated registers, they’re all preceded by ". This means that you can paste by visually selecting and typing "0p if “0 is the register’s content you want to paste.
Collection of intermediate but handy git tricks. This post will be updated now and then.
Apply patch from email
git am -3 --signoff <name of patch>.patch
Bisect
<object id> could be master, 35f1cab or v1.4, for example.
git bisect start
git bisect good <object id>
git bisect bad <object id>
# test the given revision
# say if it's bad or not with
git bisect bad
# continue this way and exit with
git bisect reset
Modify the theme a bit (far too many h1 used, for example)
Get a pattern from subtlepatterns.com
Upload it to a new /var/www/directory-here, add a virtual host section to apache’s config with ServerName test.iamnearlythere.com. Forgot to register test.iamnearlythere.com as an A record at my DNS registrar, 30 min wasted.
Install nginx and set the default mimetype to text/html since I was silly enough to use the URI form of url.com/post-title which made mynt generate posts as files without extensions which by default are served as application/octet-stream by nginx (“want to download this?”) and served as text/plain by apache.
The request dropped to 6 requests at 190 kb, totalling 1.47 s.
Since the machine now no longer needs apache or mysql:
Make sure apache doesn’t start on reload: $ sudo update-rc.d -f apache2 remove
Make sure mysqld doesn’t start on boot: http://superuser.com/a/139059/9539
Make sure nginx does start on boot: http://articles.slicehost.com/2007/10/17/ubuntu-lts-adding-an-nginx-init-script (I had to replace the nginx path with /usr/local/nginx/sbin/nginx)
Allright, I’m done.. ish. When using $ tail -f /usr/local/nginx/logs/error.log I notice a bunch of stuff I missed. Redirections here we go
Also, create a Makefile for generating, testing and deploying the blog (mynt generate; rsync)
Memory on vps could be turned down from 768 MB to 128, without tweaking nginx. Currently using about 20%.
$ ab -n 10000 -c 25 can be run without any problems what so ever
Conclusion:
I’m pretty sure I suck at micromanaging apache & mysql. And that nginx is pretty fly with static files.
Also, a hiccup: I issued a 301 redirect in one case, and it went badly – my nginx non-ninja-config-skils made me issue a recursive redirect loop. 301 responses are heavily cached by browsers, I had to delete the cache after removing the direction out of the config file. Do not make this mistake during ‘live hours’ or with a heavily trafficked page.