iamnearlythere

A Shell’s Must-have Features

I’ve been using zsh with oh-my-zsh for a long time now. If I were to replace it, this is what I’d look for in the future:

  • Case insensitive completion/globbing
  • Auto completion of filenames via tab, left + right, enter
  • Fuzzy matching/spelling correction of targets when cd-ing
  • Have sensible prompt looks
    • (aliased) directory
    • git status
      • branch name
      • dirty-index indicator
    • no username
  • ctrl+r history
  • !$ and !! expansion before execution
  • up + down to traverse history
  • globbing with **

Making Jenkins Work With Python’s Virtualenv

If you use virtualenv to isolate your python project’s environment, and want your code tested automatically – read on!

virtualenv isolates your project’s python environment

virtualenv makes sure you lockdown your project’s main directory and all subdirectories of it. This ‘lockdown’ means that you never touches your global python binary, or any globally installed libraries (like “sudo pip install flask” ).

Once locked down, you install all packages again, even those you have globally installed. This enables you to have one version of flask globally installed, but another version in your project. All dependencies can be listed in a separate file and validate a precise environment for you to work with. Tightly controlled dependencies is key to a deployment without surprises.

Jenkins checks the health of your project for each change

Jenkins is a CI server which means it does a lot of repeating stuff so you can focus on doing more important stuff. More specifically, it listens for changes to your project’s version control system (like git).

When changes are detected, the project is built and the test suite is executed. If any step fails, the CI server tells you that it did.

Setup Jenkins, and make it use virtualenv

Jenkins needs some massaging before it handles the hijacked environment of virtualenv. This is how I did it for my local git repository:

  • Download Jenkins
  • Start it, it should be up on http://localhost:8080
  • Add the Git Plugin
  • Setup a new project with these properties:
    • Source Code Management: add the URI to your local repository, /Users/you/Sites/asdf in my case. Make sure the jenkins user can read this directory, otherwise the Jenkins GUI will tell you something random about invalid git repo, without a hint about a permissions error.
    • Build Triggers: Poll SCM (with an interval like 0 * * * *). This is needed because
      • you’re too lazy to build manually; and
      • you can not trigger builds with a git post-commit hook otherwise
    • Build > Execute shell. I’ve used two steps, one for setting up the environment and one for the actual tests:
1
2
3
4
5
6
7
8
# Setup a proper path, I call my virtualenv dir "venv" and
# I've got the virtualenv command installed in /usr/local/bin
PATH=$WORKSPACE/venv/bin:/usr/local/bin:$PATH
if [ ! -d "venv" ]; then
        virtualenv venv
fi
. venv/bin/activate
pip install -r requirements.txt --download-cache=/tmp/$JOB_NAME

and

1
2
. venv/bin/activate
python test_app.py

Trigger Jenkins through git hook

You need to add a git hook which triggers a Jenkins build:

1
2
echo "curl http://localhost:8080/git/notifyCommit?url=/Users/you/Sites/asdf" >> .git/hooks/post-commit
chmod +x !$

Add an erroneous test, this will do:

1
2
def test_bad(self):
    assert False

You should see a new build being queued up in Jenkins within a minute. If that doesn’t work, execute the hook and watch the output for error messages:

1
./git/hooks/post-commit

Now, Jenkins should try to test your project but fail, and report the failure through the GUI. Tada.

From Youtube to Mp3

Because I always forget the exact commands:

1
2
youtube-dl https://youtube.com/?v=id_here
ffmpeg -i id_here.flv -acodec mp3 output.mp3

On OS X, you’ll find both youtube-dl and ffmpeg through homebrew.

What I Want From an Interface

Consider this a checklist for your API, should work for REST, CLI or source code!

What the interface is for

Have a scope and tell me what it is in a couple of sentences.

How to

Let me learn the general approach and edge cases via a concise and result oriented documentation.

Feedback

Scream at me when I did something wrong and, if I was close enough (and you have a lot of time), point me in the right direction.

Consistency

For example, when treating errors you shouldn’t alternate between simultaneous ways of telling your users something went wrong. Pick one and go with it.

  • error callbacks (JS)
  • separate errors from regular output (bash)
  • exceptions (python et al.)
  • magical return codes (C)

Getopt in Bash

How to properly read command line arguments for a bash script and do it well even without getopt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
while :; do
        case "$1" in
                -h|--help)
                        echo "This is helping"
                        exit 0
                        ;;

                -s|--simple-flag)
                        echo "You passed a flag"
                        shift
                        ;;
                -p|--pass-option)
                        shift
                        echo "You set option to $1"
                        option="$1"
                        shift
                        ;;
                -*)
                        echo "That's a weird option"
                        exit 1
                *)
                        # No more args to read
                        break
                        ;;
done

I think the syntax is much cleaner than that of getopt’s as well – that is, if you’ve even got a bash version supporting getopt.

Screen

Screen could be used as nohup with the additional possibility of attaching to the terminal session again, once your ssh session was aborted (or equivalent).

screenCreate a new screen
screen -S nameCreate a new screen named “name”
screen -lsList screens
screen -DDetach from screen
screen -rAttach to the single existing screen
screen -r 4262Attach to the detached screen with pid 4262
screen -x 4262Reattach to the attached screen with pid 4262
ctrl+a cCreate a new terminal in the current screen
ctrl+a HLog all output of current screen to a log file
ctrl+a spaceChange terminals within current screen
ctrl+a :sessionname nameRename current screen to “name”
ctrl+a dDetach from current screen, like exit but keeps the terminal for reattaching later on

Gitweb on OSX

I have all my code stored in ~/Sites and every sub folder represents a project. Each folder is also a git repository. This means there’s history attached to it, which could be much more easily navigated than through the finder or a terminal. Some of the projects are mirrored on github but it’s too messy to keep them all there, although – that kind of overview is nice to have.

My solution to getting an overview was to setup gitweb locally, which makes for a very fast solution. This is how you do it on your OS X-machine:

  • Find where git’s installed at. I use homebrew, which means git (by default) resides in /usr/local/Cellar/git.

  • Grab the gitweb cgi, copy it to where your local Apache server can find it:

1
2
# cp /usr/local/Cellar/git/1.7.11.1/share/gitweb/gitweb.cgi /Library/WebServer/CGI-Executables
# cp /usr/local/Cellar/git/1.7.11.1/share/gitweb/static /Library/WebServer/Documents/gitweb
  • Modify gitweb.cgi to let it find the static files, to find the git projects and to not choke on really large directories that I’m sure does not contain anything git’ anyway:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
--- /usr/local/Cellar/git/1.7.11.1/share/gitweb/gitweb.cgi  2012-07-11 20:23:10.000000000 +0200
+++ /Library/WebServer/CGI-Executables/gitweb.cgi    2012-07-11 20:50:36.000000000 +0200
@@ -71,11 +71,11 @@

 # absolute fs-path which will be prepended to the project path
 #our $projectroot = "/pub/scm";
-our $projectroot = "/pub/git";
+our $projectroot = "/Users/my-user/Sites";

 # fs traversing limit for getting project list
 # the number is relative to the projectroot
-our $project_maxdepth = 2007;
+our $project_maxdepth = 5;

 # string of the home link on top of all pages
 our $home_link_str = "projects";
@@ -95,15 +95,15 @@
 our $site_footer = "";

 # URI of stylesheets
-our @stylesheets = ("static/gitweb.css");
+our @stylesheets = ("/gitweb/static/gitweb.css");
 # URI of a single stylesheet, which can be overridden in GITWEB_CONFIG.
 our $stylesheet = undef;
 # URI of GIT logo (72x27 size)
-our $logo = "static/git-logo.png";
+our $logo = "/gitweb/static/git-logo.png";
 # URI of GIT favicon, assumed to be image/png type
-our $favicon = "static/git-favicon.png";
+our $favicon = "/gitweb/static/git-favicon.png";
 # URI of gitweb.js (JavaScript code for gitweb)
-our $javascript = "static/gitweb.js";
+our $javascript = "/gitweb/static/gitweb.js";

 # URI and label (title) of GIT logo link
 #our $logo_url = "http://www.kernel.org/pub/software/scm/git/docs/";

Now, visit & bookmark http://localhost/cgi-bin/gitweb.cgi in your browser and you should be good to go.

Yanking in Vim

A common pattern while editing is to yank anything (i.e. yy), visually select what you want to remove (i.e. vj) and paste (p). This will in turn place the overwritten test in the yank register; but we wanted to paste that same line again!

Type :reg to see all currently populated registers, they’re all preceded by ". This means that you can paste by visually selecting and typing "0p if “0 is the register’s content you want to paste.

Git

Collection of intermediate but handy git tricks. This post will be updated now and then.

Apply patch from email

    git am -3 --signoff <name of patch>.patch

Bisect

<object id> could be master, 35f1cab or v1.4, for example.

    git bisect start
    git bisect good <object id>
    git bisect bad <object id>
    # test the given revision
    # say if it's bad or not with
    git bisect bad
    # continue this way and exit with
    git bisect reset

Branch off commit

    git branch new_branch <ref>

Commit part of diff from a file

    git add -p

Create emailable patch

    # edit
    # commit
    git format patch origin/master --stdout > my_changes.patch

Where origin/master is the base for references.

Discard parts of a diff

    git checkout -p
    # is hunk too big? 's' to split it

Merge commits

If you want to add some changes that you’d want included in the previous un-pushed commit

    git commit -am "just a commit"
    # make changes
    git commit --amend -a

Move hunk of commit to another commit

    git rebase -i HEAD~2
    # p ...
    # e ... <- edit this
    #edit files
    # git add; git commit --amend; git rebase --continue

Remove remote branch

    git push <remote> :<branch>

Show log with patches

    git log -p
    # for a specific file
    git log -p <path>

From Wordpress on Apache to Mynt on Nginx

Going from Wordpress to mynt.

Purpose:

  • smaller footprint, tune down vps (mysql & apache + my bad tuning skills = ram gets eaten up)
  • write posts in markdown
  • use CLI for faster editing
  • posts in git
  • local preview for free

Bonuses:

  • getting to know a python project (mynt)
  • trying out nginx which is recommended for static content
    • before: mysql & apache processes hanging around, using 500 mb ram with near 0 load
    • response time (with wordpress cache plugin) was 2.49 s for 100.55 KB split on 17 requests (front page)
    • $ ab -n 1000 -c 5 http://iamnearlythere.com/ crashes
  • take the opportunity to learn less for a redesign, fits with the pregeneration. also, include responsive design (octopress = good influence)

Good to have:

  • rsync for deploying
  • log 404’s
  • nice source syntax (see examples on mynt’s web)

Procedure:

  • Install mynt, requires xcode on osx (for “watchdog” – an, in my eyes, unnecessary dependency)
  • Export all posts from wordpress (I got an XML file out of 3.3.1)
  • Run $ ./wp2mynt.php export-file.xml (wp2mynt.php is on github)
  • Modify the theme a bit (far too many h1 used, for example)
    • Get a pattern from subtlepatterns.com
  • Upload it to a new /var/www/directory-here, add a virtual host section to apache’s config with ServerName test.iamnearlythere.com. Forgot to register test.iamnearlythere.com as an A record at my DNS registrar, 30 min wasted.
  • Install nginx and set the default mimetype to text/html since I was silly enough to use the URI form of url.com/post-title which made mynt generate posts as files without extensions which by default are served as application/octet-stream by nginx (“want to download this?”) and served as text/plain by apache.
    • The request dropped to 6 requests at 190 kb, totalling 1.47 s.
  • Since the machine now no longer needs apache or mysql:
    • Make sure apache doesn’t start on reload: $ sudo update-rc.d -f apache2 remove
    • Make sure mysqld doesn’t start on boot: http://superuser.com/a/139059/9539
  • Make sure nginx does start on boot: http://articles.slicehost.com/2007/10/17/ubuntu-lts-adding-an-nginx-init-script (I had to replace the nginx path with /usr/local/nginx/sbin/nginx)
  • Allright, I’m done.. ish. When using $ tail -f /usr/local/nginx/logs/error.log I notice a bunch of stuff I missed. Redirections here we go
  • Also, create a Makefile for generating, testing and deploying the blog (mynt generate; rsync)

Result:

  • Memory on vps could be turned down from 768 MB to 128, without tweaking nginx. Currently using about 20%.
  • $ ab -n 10000 -c 25 can be run without any problems what so ever

Conclusion:

I’m pretty sure I suck at micromanaging apache & mysql. And that nginx is pretty fly with static files.

Also, a hiccup: I issued a 301 redirect in one case, and it went badly – my nginx non-ninja-config-skils made me issue a recursive redirect loop. 301 responses are heavily cached by browsers, I had to delete the cache after removing the direction out of the config file. Do not make this mistake during ‘live hours’ or with a heavily trafficked page.