Friday, June 24, 2011

Empirical confidence intervals, redux

Edit:
Evidently I've re-invented the wheel (yet again!).




There's nothing quite like spending an afternoon trying to get R to Plot What I Mean. In hopes of getting a paper off my desk and out the door, I got to revisit the overplotting conundrum previously presented. I wanted to plot this:

but allow the viewer some concept of the relative density of points near 0% error . There's way too much overplotting here to tell by eye. With only a little reflection, it was obvious solution I offered before was jingus, chiefly because of the sparsity the data (and their non-rectangular hull) prevented the 2D kernel estimate from working very well. You can see this in action if you a command that almost Plots What I Mean--smoothScatter, which uses a 2D kernel estimate to plot most of the data, then overplots outliers:


A kernel density estimator is too fancy for this problem. All that you really need to do is divide the abscissa into a grid that has enough points at each slice that taking the pth percentile of all the points in the slice gives you a decent estimate of the true value of the percentile there. This is easy to do with quantcut (make the slices with equal number of observations) or cut2 (make the slices with at least n observations). Then you can connect the pth percentile at each slice with linear or loess interpolation. I even cooked up a panel function that you can drop straight into your favorite lattice xyplot:

> xyplot(y~x, data=comp3, cex=.5, xlim=c(-5, 1e3), ylab="Percentage error", xlab=expression(lambda), ylim=c(-100, 100), panel=panel.confbars, npt=15, conf=.05)
Make npt larger to get a denser interpolation grid, or smaller to get smoother interpolation. conf gives the 100-2*conf % confidence interval you wish to bound, so the default is a 95% confidence interval.

panel.confbars = function(x, y, npt=25, conf=.025, ...){
require(gtools)
panel.xyplot(x, y, ...)

xcut = quantcut(x, q=seq(0, 1, length.out=npt))
xval = quantile(x, p=seq(0, 1, length.out=npt))
yval = matrix(unlist(tapply(y, xcut, quantile, p=c(conf, 1-conf))), nrow=2)

lconf = approxfun(xval[-1], yval[1,])
uconf = approxfun(xval[-1], yval[2,])

panel.loess(xval, lconf(xval), type="l", col="red", lwd=2)
panel.loess(xval, uconf(xval), type="l", col="red", lwd=2)

}

Thursday, June 16, 2011

Amadou and Ami do Senegal

For this post, we will be known as Amadou and Ami...Andrew and Erin being a bit hard to pronounce if you are Senegalese.

The last few weeks of our trip were spent in Senegal visiting our friend Emily and seeing the country. Emily has been working in Ngaraff as a Peace Corps Volunteer for the past 14 months and graciously hosted our trip. And by graciously, I mean it...walking more slowly so that our sand-challenged legs could keep up, put up with our attempts at French while teaching us some basic Wolof, and didn't even get upset when our camel trip was ruined by a bout of violent gastroenteritis (Ami 2: Amadou 0).

Our first day was spent in Dakar, with a trip to Ile De Goree, important historically in the Atlantic slave trade. Although most references show that a very small percentage of slaves from Africa went through Goree, there are still signs of a disturbing past. The most dramatic being the intact holding chambers: boat houses and the Maison de Esclaves.

Boat Houses - Where captured men were kept in conditions similar to what they would encounter on boats to the new world while working on the island. The colonialists did not want to waste money on shipping the weak, so this was done to weed them out.


Dubbed the "Door to nowhere", boats would park on the outside of the Maison de Esclaves in order to load up slaves to take across the Atlantic.


Captured and recaptured several times by the French, Portuguese, Dutch, and British, signs of religious, cultural and architectural differences are seen on the island. Ile de Goree is now a haven for artists, who come to make and sell their paintings, carvings, and jewelry.

View of Dakar from Ile de Goree


cafe Touba stand


Resident Artist's housing on Goree


Next, we headed out down the coast of Senegal, amoung the delta and salt flats, to Palmarin. Joined by another volunteer who had lived in Palmarin for the last three years, we instantly felt at home in the village where we spent time playing in the ocean, building sand castles, taking a pirogue out camping, kayaking, and eating delicious freshly caught fish.

Taking the village kids swimming.


Crossing the lagoon to the beach.


Pirogues (brightly colored wooden boats) stopped to bring in fish outside of Palmarin.


Couldn't this be a Grant's ad? Convieniently comes in a triangular bottle that won't roll around when boating.


Oops...migratory bird island with a dog...birds are not too happy.


Campsite on a sandbar.


Kayaking in the mangroves.


Side trip to check out some Baobobs (which are, by the way, one of my most favorite trees).


Inside the Baobob.


After a great time on the coast we started the day-long journey inland to the Linguere region, to Emily's village. Emily, by the way, is a Wolof speaking badass, who negotiated her way through garages with style and fierce determination. A few sept place and minicar rides later, we landed in Ngaraff.

The people of Ngaraff greeted us with open arms, and lots of greetings:) Ever the gracious hosts, they tolerated my amateur millet pounding skills with a smile, allowed us to share dinner from their bowls in the evenings (even slaughtered a chicken), and were full of encouragement even if we couldn't communicate more than a hello and names in Wolof.

Ngaraff was also getting ready for a huge baptism party. This was to be the party of the year, with crazy amounts of food, women dressed to the nines, and even a generator to power the tunes. We spent the day making greeting rounds, watching the ladies dance-spar, holding cute children, and helping slice things to be cooked. And, I have to say, we brought the house down with our Toubab dancing skills. The crowd went wild when Amadou started his crazy-eyed booty shaking.

The ladies cooking up a huge Baptism feast. It is amazing how much food is generated in a few hours on wood burning fires.


Dressed to the nines, slicing some onions.


kidnapping...



Goat + Chicken + Cat = World Peace


We took a stroll into the bush after the Baptism party. The desert always strikes me as such harsh, hot, and challenging place to live that I often find myself wondering why people call it home, but sometimes it can catch you off guard with its beauty...especially around sunset.

Sunsets in the Djolof


Acacia thorns.


More pretty clouds at sunset.


Our last day in Linguere, we rose early and headed out into the bush on bikes to meet up with other PCVs. What was to be a hour bike ride turned into a 4 hour debacle as we were tested with flat tire after flat tire. Good thing Amadou and Bigue are pros at changing tires. The other volunteers patiently waited on us, and we had a great day in the village, watching a teaching lesson on making mosquito repellant lotion from local plants, and hanging out with a Senegalese-American-Canadian ex-pat who is setting up a homestead in the village of his birth.

Flat tire episode #1.


Somebody's been eating garbage...


Chariot ride back to the bikable trail


Waylaid by Ami's intestines, we spent a day lounging about before heading to St Louis for some beach time, jazz festival time, and debauchery. Our last few hours back in the capitol were spent shopping for and cooking a decidedly delicious meal of lasagna, broccoli, and ice cream.

Not wanting to leave, we trudged back to the airport. I felt extremely happy that I had been able to share this experience with such great people. We had the trip of a lifetime, and have Emily to thank for showing us the country, helping us communicate, and sharing her beautiful personality. I have already started to plan a trip back.