Archive

thingsilearned

I’ve been a fanatic about finding the best ways to organize my life and stay “on top of things”.  There are a million articles online on how to go about this. I’ve read my fare share of these and tried a lot of the methods out.  Usually something ends up working decent for a while and then I either burn out or lose discipline in it.  So I pick up some other method or just deal with a more chaotic productivity schedule.

I never get too disappointed when one doesn’t work out as usually said method was proposed by some blogger who seemed to only have recently (and likely temporarily) gotten their shit together and decided to tell the world about it.  So there wasn’t too much of an expectation going in, but its often a bit frustrating to me to have the advice and tips given out so confidently.

The articles are usually written in a way that makes their method sound like the writer has been practicing this for years and achieved great success/peace with the workflow they deal with.  But I have never met a person who has found such success.

There are a thousand different TODO apps on the web and for download.  Each one will tell you that their tool is the best to keep organized and productive.  There are even more authors and bloggers.

In the end nothing has worked near as well for me as a blank notebook and a pen.  Sometimes a planner helps when my days are more about scheduled meetings and less about development time, but planners are just paper with a different set of lines.

So I guess the thing that I learned hear is that keeping organized and on top of things is a constant struggle and there’s no perfect fix for it.  Its just the nature of the game, and if you accept that it takes time and some effort then the whole process becomes a lot less frustrating and a lot less time is wasted trying out so many different tools.

Today I needed to write a wrapper around a program that would examine the stdio and respond immediately to certain results.  The task was handled nicely with a python script using subprocess.

There isn’t a lot of documentation examples on it so to figure it out I wrote this small test app and I thought I’d share it here.

To begin I wrote a simple script that prints out “Program has run for x seconds.” every x seconds to simulate a long running and noisy script that I might be observing.  Here is the code.

sample.py

import sys, time

class FlushFile(object):
   """Write-only flushing wrapper for file-type objects."""
   def __init__(self, f):
       self.f = f
   def write(self, x):
       self.f.write(x)
       self.f.flush()

# Replace stdout with an automatically flushing version
sys.stdout = FlushFile(sys.__stdout__)

for i in xrange(100):
   sys.stdout.write("Program has run for %d seconds.\n" % i)
   time.sleep(1)

The main part of the program is the last 3 lines where we write “Program has run for %d seconds” 100 times and pause for a second between each.

The FlushFile object is simply a nice hack to overwrite the default stdout object to ensure that the buffer is flushed every time it is written to.  Without the hack our log checker would simply be hung up until the task is fully completed instead of being able to read each line of the output as it is written.

The output looks like this

$ python sample.py
Program has run for 0 seconds.
Program has run for 1 seconds.
Program has run for 2 seconds.
.... # and on until 100 seconds

Next I wrote the wrapper which executes the above sample.py script as a python subprocess and watches the output.  For this example I check whether a ‘4’ exists in the output.  If it is I restart the script therefore creating a continual loop counting between 0 and 4 seconds.

observer.py


import subprocess, os, signal
cmnd = "python sample.py" #change this line to run your script
p, line = True, 'start'

while True:
   p = subprocess.Popen(cmnd, shell=True, stdout=subprocess.PIPE)

   while line:
       line = p.stdout.readline()
       print "line is:", line
       if line.count('4'):
           print "restarting the process"
           os.kill(p.pid, signal.SIGUSR1)
           line = True
           break

   del p

The output is continually checked in the ‘while line:’ loop of line 8.  In this example its printed out for our convenience.  Like 11 checks for the character ‘4’ somewhere in the output.  If it does exist it kills the process and the stdout reading loop.

That loop is wrapped in another however that simply restarts the process all over again.  The resulting output is

$ python observer.py
line is: Program has run for 0 seconds.
line is: Program has run for 1 seconds.
line is: Program has run for 2 seconds.
line is: Program has run for 3 seconds.
line is: Program has run for 4 seconds.
restarting the process
line is: Program has run for 0 seconds.
line is: Program has run for 1 seconds.
line is: Program has run for 2 seconds.
line is: Program has run for 3 seconds.
line is: Program has run for 4 seconds.
restarting the process
line is: Program has run for 0 seconds.
.....

So now we have a nice template for wrapping and responding to executables!

A few weeks ago I published a request for a generous designer to create a unique look for this blog.  Amanda Scharlemann was the first to respond and was a pleasure to work with!

Amanda is a design student at Bethany Lutheran College and shows great potential.  She had never worked with customizing WordPress Templates and the annoying restrictions that come with that, but was eager and quick to learn.  We went through several iterations to narrow down exactly what I was looking for and she spent a lot of time examining the example sites I gave her to get a feel for what I would like.  At each iteration she had significant improvements, helpful advice, and most importantly was open and attentive to feedback.

I dislike design processes where the designer is stubbornly biased in certain directions and completes 90% of the work before you’re allowed to review.  Amanda was not that way at all.  I’m not sure if they teach client-design relationships at Bethany or not, but she was a pleasure to work with.

Check out her resume.

The custom design was launched last week, and so far the results have been great.  I’ve seen a doubling in traffic, and several more comments than usual.  Perhaps  the most impressive statistic so far is that a friend I hadn’t talked to in a while came across my blog while goggling an issue he was having.  My article apparently helped him and he was excited to notice (thanks to the personalized sidebar) that the blog belonged to me!

Over time I will probably tweak things here and there as is my nature but I’m over all very pleased with the new look.

Thanks again Amanda!

In Django it is very straight forward to add extra commands to the manage.py and django-admin.py scripts.  The Django Documentation describes the process but gives no examples.  I like examples so I’m writing this for others.

In the past I’d add these  to make stand alone scripts that used the Django libraries.

from django.conf import settings
from django.core.management import setup_environ
setup_environ( settings )

The method works well but I find it cleaner to add commands to the manage and django-admin scripts instead of having several stand alone scripts.  The desired effect is to be able to run

$ ./manage.py your_command_name

instead of python your_command_name.py.  Big difference?  No…  But I think on a large project it adds organization and its some sort of a standard in the event that others will be using your code.

In the app which the command is used for make the following directory structure:

    management/
        __init__.py
        commands/
            __init__.py
            your_command_name.py

Then in your file (your_command_name.py) paste the following code, puting the functions to run in the handle_noargs function.

from django.core.management.base import NoArgsCommand

class Command(NoArgsCommand):
help = “Describe the Command Here”
def handle_noargs(self, **options):
< your code here >

That’s it!  You can also do some more complicated functionality with command options.  Take a look at commands listed in /django/core/management/commands for examples there.

Almost two years ago on a cranky afternoon I wrote this uninformed and poorly edited article:  Xcode Editor Kinda sux.

Since that day it has consistently been the top daily hit on my blog.  I did some Googlein’ and discovered that it is the top link (maybe just in my search context) when searching for “xcode editor”.

This probably makes the team who works on the editor unpleased… Sorry guys.  Let me know if you’ve made short cuts for that yet and I’ll write an update.

I looked at my stats and found this:

Xcode Editor – Kinda Sucks 1,330 More stats
Zope3/Plone to Django 1,215 More stats
The Dangers of a Startup 925 More stats

Once you’re done laughing at my piddly traffic you may notice that my top 2 blog highly trafficked blog posts of all time were also the most negatively critical.  The 3rd link isn’t negative by nature, but does have a negative title.  I will chalk that one up as a fluke as most of its traffic came from a single day that it was on Hacker News.

I’m personally quite proud that in my few years of blogging the crankiest opinion based things I’ve written about are those top two posts.  But, perhaps I would be more proud of greater traffic had I been less reserved.

My blog is a small sampleset.  Has anyone noticed similar trends?

I recently suspected the dreaded buffer overrun in a project I’m hacking for work. These can be a huge pain to find, especially if your code has a considerable amount of mallocs and fails irregularly. After dismissing the urge to just write it all in python (wasn’t practical for the application) I decided to learn a malloc debugging tool.

I was excited to learn that macs come with their own malloc debuger, entitled Malloc Debugger. How handy! Through several attempts to use it on my application however I kept getting the same error.

Unable to read malloc information from (null)

I’m probably just missing some libraries to include that would have been had I been using Xcode. Instead I was compiling with g++ -g.

Anyway, I eventually checked out Electric Fence (efence) and had decent results. It didn’t compile right away, you have to comment out line 33 of page.c as stated in this mailing list.

Copy the compiled library libefence.a to /usr/lib/ and re-compile your application with the flag -lefence.

Upon execution it will spit out problems you have in your malloc code and die if there was a buffer overrun. Its a great tool to turn on while you work as you’ll be informed of any overwrites as you’re developing. Not after the whole thing is done and quitting at odd times :).

Dave

I was going to write a brief summary/closer on our summer but ended up writing this lengthy article about our startup’s decision to switch to Django from Zope3/Plone as it is the major reason we’ve missed our previously projected release dates. Its difficult for me to post this as it may appear to cast some negativity toward Zope3/Plone, a community and CMS for which I have a great deal of respect. It is also difficult as it describes significant failure on my part.

I continue to post though as this startup is all about sharing the things you learn and this was a big one this summer. I wish I had found a post like this when I was first searching for Zope3/Plone experiences and hope others in my position find it in the future.A few months ago we made a huge team decision to scrap the month and a half we spent learning Zope3/Plone for our future content creation/collaboration site and switch to Django.

It was a really tough decision. Plone is an awesome content management system. Its built with python, has an amazing community, I had/have a Google Summer of Code in it, and the clincher: its already capable of doing 90% of what we want our site to do. It manages users great, handles blogs, wikis, profiles. It was mouth watering, and to be honest was my main source of confidence in the idea that we’d be able to pull off such a complicated site.

That confidence slowly deteriorated over our first month of learning Plone. I bought every book on Plone and Zope3 and in that first month in my parent’s basement Priyesh and I spent day and night learning the stuff. I read almost all of two Zope3 books and one on Plone. They made no sense to me. I kept re-reading them and going over examples. I could develop the examples just fine but I couldn’t figure out how to write my own.

I couldn’t get a feel for it. The concepts of interfaces, utilities, adapters, viewlets, ZCML, etc just weren’t meshing. This bothered me tremendously as I’ve been programming since childhood and consider myself (perhaps falsely so) at least halfway skilled in python and web development.

Note: I later found out that some of my problem was that, though only a few years old the Zope3 Developer’s Handbook has fairly outdated Zope3 methodology.

I looked around for classes as I decided its just the type of stuff you have to have a teacher for. The Plone community has fairly regular 2-3 day classes but none lined up with our location or timeline. I am almost certain that had I been able to take one of the Plone courses it would have come together fairly easy. As an alternative I found that a Plone sprint being hosted at the Googleplex! I went and tried helping out as I blogged about earlier. I wanted to get a feel for Zope3 and thought that it might inspire that “snap” in fluency where it’d suddenly be obvious how all of those components work in Zope3.

During the sprint I got to talk with Alexander Limi, one of the founders of Plone, who explained that Plone is in a very transitional state, attempting the crazy but so far successful task of slowly merging between Zope2 and Zope3. For those of you who aren’t in the know Zope3 is almost a completely different framework from Zope2. He explained that Plone will be incredibly difficult to learn for the next year or so as it completes its transition. New comers would have to learn both the old as well as the new and all the crazy ways in which they bridge the gap.

He was confident though, and rightfully so, that the Plone community was strong enough to make it through that year with very little new blood. He very humbly recommended going with a different framework for our startup; advice that we were already considering and took soon after. Plone may have already given us 90% of our site, but we decided that having to write the last 10% in Zope3 might take us longer than starting from zero.

Having lost so much time by judging a book by its cover we decided to do a great deal of reading and research before choosing our next framework. I had done a good deal of web development several years ago but a lot of fancy new and helpful frameworks have fortunately popped up since then!

We took a week, each of us diving into one or two frameworks, sharing what we liked and didn’t like about each. Our benchmark for each framework was the ease and accuracy with which we could replicate Jared’s One Blog. We’re still intending on posting our sometimes humorous results but haven’t had a public server up and running until recently.

As was already spoiled, we went with Django. I’d heard a lot of good things and they’ve so far been proven. For the most part we really enjoy developing in Django, though it can be frustrating to replicate features we already know are implemented in Plone :). There’s no on the shelves book on it (I like buying lots of books), the documentation doesn’t go very deep, and the community is significantly less active than Plone’s. The framework though, at least to me, just makes more sense. It was ridiculously easy to replicate the One Blog and problems with missing documentation can usually be solved with looking at a little source code or whining in IRC.

The change in game-plan has given us more work that we expected to do at the beginning of the summer. As I said, we thought we were 90% there and pretty much lost a month of development. Startups are all about obstacles though! Since the change we’ve been making tremendous progress and will be opening an Alpha to a few friends/advisors within the week.

Perhaps one day someone will open source a version of Plone in Django! Maybe that will be us. Djone? Glono?

zoPE <-> PlonE as djanGO <-> GlonO

Dave