Lorin Hochstein's Blog, page 30

Estimating confidence intervals, part 4

Here’s the latest installment in my continuing saga to estimate effort with 90% confidence intervals. Here’s the plot:

In this case, my estimate of the expected time to completion was fairly close to the actual time. The upper end of the 90% confidence interval is extremely high, largely because there was some work that I considered optional to complete the feature that decided to put off to some future data.

Here’s the plot of the error:

It takes a non-trivial amount of mental efforts to do these estimates each day. I may stop doing these soon.

 

Not apprenticeship!

Mark Guzdial points to an article by Nicholas Lemann in the Chronicle of Higher Ed entitled The Soul of the Research University. It’s a good essay about the schizophrenic nature of the modern research university. But Lemann takes some shots at the notion of teaching skills in the university. Here’s some devil’s advocacy from the piece:

Why would you want to be taught by professors who devote a substantial part of their time to writing projects, instead of working professionals whose only role at the university is to teach? Why shouldn’t the curriculum be devoted to imparting the most up-to-the-minute skills, the ones that will have most value in the employment market? Embedded in those questions is a view that a high-quality apprenticeship under an attentive mentor would represent no loss, and possibly an improvement, over a university education.

Later on, Lemann refutes that perspective, that students are better off being taught at research universities by professors engaged in research. He seems to miss the irony that this apprenticeship model is precisely how these research universities train PhD students. For bonus irony, here was the banner ad I saw atop the article:

Estimating confidence intervals, part 3

Another episode in our continuing series of effort estimation in the small with 90% confidence intervals. I recently finished implementing another feature after doing the effort estimates for each day. Here’s the plot:

 

Once again, I underestimated the effort even at the 90% level, although not as badly as last time. Here’s a plot of the error.

I also find it takes real mental energy to do these daily effort estimates.

 

 

 

Crossing the river with TLA+

Lately, I’ve been interested in approaches to software specifications that are amenable to model checking. A few weeks ago in this blog, I wrote about solving a logic puzzle with Alloy. Today’s post is about solving a different logic puzzle. I found this one from the Alloy online tutorial:

A farmer is on one shore of a river and has with him a fox, a chicken,

and a sack of grain. He has a boat that fits one object besides himself.

In the presence of the farmer nothing gets eaten, but if left without the

farmer, the fox will eat the chicken, and the chicken will eat the grain.

How can the farmer get all three possessions across the river safely?

To solve this, I used TLA+, a specification language developed by Leslie Lamport. It also uses PlusCal, which is an algorithm language that can be automated translated into TLA+ using the TLA toolbox.

Here’s my solution, which includes PlusCal but doesn’t show the automatically translated parts of the model.

-------------------------------- MODULE boat --------------------------------
EXTENDS Integers, FiniteSets
CONSTANTS Farmer, Fox, Chicken, Grain
CREATURES == {Farmer, Fox, Chicken, Grain}

alone(animals, side) == (animals \in SUBSET side) /\ ~ Farmer \in side

somebodyGetsEaten(l, r) == \/ alone({Fox, Chicken}, l)
\/ alone({Fox, Chicken}, r)
\/ alone({Chicken, Grain}, l)
\/ alone({Chicken, Grain}, r)

safe(l, r) == ~somebodyGetsEaten(l, r)

safeBoats(from, to) ==
{ boat \in SUBSET from : /\ Farmer \in boat
/\ Cardinality(boat)

============================================================

To solve the problem with the TLA toolbox, you’ll need to specify an invariant that will be violated when the puzzle is solved. I used right /= CREATURES.

Run the model, and it will produce a trace that violates the invariant:

 

(You’ll first need to translate the PlusCal into TLA+, and you’ll need to specify the value of the constants. I just chose “Model value” for each of them).

You can see the full model with the automatic PlusCal translation in one of my Github repos.

Estimating confidence intervals, part 2

Here is another data point from my attempt to estimate 90% confidence intervals. This plot shows my daily estimates for completing a feature I was working on.

 

The dashed line is the “truth”: it’s what my estimate would have been if I had estimated perfectly each day. The shaded region represents my 90% confidence estimate: I was 90% confident that the amount of time left fell into that region. The solid line is the traditional pointwise effort estimate: it was my best guess as to how many days I had left before the feature would be complete.

For this feature, I significantly underestimated the effort required to complete it. For the first four days, my estimates were so off that my 90% confidence interval didn’t include the true completion time: it was only correct 60% of the time.

This plot shows the error in my estimates for each day:

 

Apparently, I’m not yet a well-calibrated estimator. Hopefully, that will improve with further estimates.

Presentation as text

I gave a talk last week at Camp DevOps about Ansible and EC2. The talk is written in present format, which is a very lightly marked up text format, similar to Markdown. You can see the source file in a Github repo.

It was liberating to focus entirely on content and not worry too much about the exact appearance of the slide.

I also went for a minimalistic approach where I often didn’t even use titles. The slides won’t make much sense if you just look at them without me talking. Hopefully, they made some sense when I was talking in front of them.

“Who owns the fish” in Alloy

Hacker News linked to a logic puzzle with the following constraints:

There are five houses in five different colors starting from left to right. In each house lives a person of a different nationality. These owners all drink a certain type of beverage, smoke a certain brand of cigarette and keep a certain type of pet. No two owners have the same pet, smoke the same brand or drink the same beverage. The question is: WHO OWNS THE FISH??? Hints:

The Brit lives in the red house
The Swede keeps dogs as pets
The Dane drinks tea
The green house is on the left of the white house
The green house’s owner drinks coffee
The person who smokes Pall Mall rears birds
The owner of the yellow house smokes Dunhill
The man living in the centre house drinks milk
The Norwegian lives in the first house
The person who smokes Marlboro lives next to the one who keeps cats
The person who keeps horses lives next to the person who smokes Dunhill
The person who smokes Winfield drinks beer
The German smokes Rothmans
The Norwegian lives next to the blue house
The person who smokes Marlboro has a neighbor who drinks water

Alloy is very well-suited to solving this type of problem, so I gave it a go. Here’s what my Alloy model looks like:

open util/ordering[House]

sig House {
color: one Color
}

abstract sig Person {
occupies: one House,
drinks: one Beverage,
smokes: one Cigarette,
keeps: one Pet
}
one sig Brit, Swede, Dane, Norwegian, German extends Person {}

abstract sig Color {}
one sig White, Yellow, Blue, Red, Green extends Color {}

abstract sig Beverage {}
one sig Tea, Coffee, Milk, Beer, Water extends Beverage {}

abstract sig Pet {}
one sig Birds, Cats, Dogs, Horses, Fish extends Pet {}

abstract sig Cigarette {}
one sig PallMall, Dunhill, Marlboro, Winfield, Rothmans extends Cigarette {}

fact allRelationsAreOneToOne {
color.~color in iden
occupies.~occupies in iden
drinks.~drinks in iden
smokes.~smokes in iden
keeps.~keeps in iden
}

pred problemConstraints {

// The Brit lives in the red house
Red in Brit.occupies.color

//The Swede keeps dogs as pets
Dogs in Swede.keeps

// The Dane drinks tea
Tea in Dane.drinks

// The green house is on the left of the white house
Green in prev[color.White].color

// The green house's owner drinks coffee
Coffee in (occupies.(color.Green)).drinks

// The person who smokes Pall Mall rears birds
Birds in (smokes.PallMall).keeps

// The owner of the yellow house smokes Dunhill
Dunhill in (occupies.(color.Yellow)).smokes

// The man living in the centre house drinks milk
(drinks.Milk).occupies in first[].next.next

// The Norwegian lives in the first house
Norwegian in occupies.first[]

// The person who smokes Marlboro lives next to the one who keeps cats
(smokes.Marlboro).occupies in (keeps.Cats).occupies.(next + prev)

// The person who keeps horses lives next to the person who smokes Dunhill
(keeps.Horses).occupies in (smokes.Dunhill).occupies.(next + prev)

// The person who smokes Winfield drinks beer
Beer in (smokes.Winfield).drinks

// The German smokes Rothmans
German in smokes.Rothmans

// The Norwegian lives next to the blue house
Blue in Norwegian.occupies.(next+prev).color

// The person who smokes Marlboro has a neigbor who drinks water
(drinks.Water).occupies in (smokes.Marlboro).occupies.(next+prev)

}

run problemConstraints for exactly 5 House

Alloy’s “Magic Layout” did a surprisingly good job at displaying the results. I had to manually rearrange the output so the houses would be displayed in the correct order, but otherwise no fiddling was required. Here’s what it looks like:

I also put it up on Github.

Results from estimating confidence intervals

A few weeks ago, I decided to estimate 90% confidence intervals for each day that I worked on developing a feature.

Here are some results over 10 days from when I started estimating until when the feature was deployed into production.

The dashed line is the “truth”: it’s what my estimate would have been if I had estimated perfectly each day. The shaded region represents my 90% confidence estimate: I was 90% confident that the amount of time left fell into that region. The solid line is the traditional pointwise effort estimate: it was my best guess as to how many days I had left before the feature would be complete.

If we subtract out the “truth” from the other lines, we can see the error in my estimate for each day:

Some observations:

The 90% confidence interval always included the true value, which gives me hope that this an effective estimation approach.
My pointwise estimate underestimated the true time remaining for 9 out of 10 days.
My first pointwise estimate started off by a factor of two (estimate of 5 days versus an actual of 10 days), and got steadily better over time.

I generated these plots using IPython and the ggplot library. You can see my IPython notebook on my website with details on how these plots were made.