Instead of just googling the answer, we’re going to fetch some raw data and feed it into Hadoop Hive.

Just a short update if you’re unfamiliar with Hadoop and Hive. Hadoop is an ecosystem of tools for storing and analyzing Big Data. At its core Hadoop has its own distributed filesystem called HDFS that can be made to span over hundred of nodes and thousand of terabytes, even petabytes, of data. One layer above HDFS lives MapReduce – a simple yet effective method introduced by Google to distribute calculations on data. Hive lives on HDFS and MapReduce and brings SQL capabilities to Hadoop.

Think of your ordinary RDBMS as a sports car – a fast vehicle often built on fancy hardware. An RDBMS can yield answers to rather complex queries within milliseconds, at least if you keep your data sets below a couple of million rows. Hadoop is a big yellow elephant. It has traded speed for scalability and brute force  - it was conceived to move BIG chunks of data around. And it can live happily on commodity hardware. For the sake a brevity we’re going to use some rather small data sets – about 1 megabyte each. It won’t even fill a file block in HDFS (64 megabyte). A more realistic example of Hadoop’s capabilities would be something like querying 100 billion tweets. A RDBMS can’t do that.

You can run Hadoop on your local machine. Like me, on an old MacBook Pro using VMWare. Just download the latest image from Cloudera

The Swedish national weather service SMHI provides the data we need: daily temperature and precipitation data from 1961 to 1997, gathered at a weather station about 60 km from where I live.

Logon to your Hadoop instance and open the terminal to download the data:

wget http://data.smhi.se/met/climate/time_series/day/temperature/SMHI_day_temperature_clim_9720.txt

(sudo yum install wget – if wget is missing)

Trim off header information with a text editor

- – - – - – - -
9720
STOCKHOLM-BROMMA
1961 2010
0101 1231
593537. 179513.
DATUM TT1 TT2 TT3 TTN TTTM TTX
- – - – - – - -

Replace leading and trailing spaces, and replace spaces between fields with commas:

cat SMHI_day_temperature_clim_9720.txt | sed -e ‘s/^[ \t]*//’ | sed ‘s/[[:space:]]\+/,/g’ > temperature.txt

Now we have properly formatted raw data ready to import into Hive. Just type “hive” in the terminal to start up Hive. The columns in the temperature data set looks like this:

DATUM YearMonthDay YYYYMMDD
TT1 temperature at 06 UTC
TT2 temperature at 12 UTC
TT3 temperature at 18 UTC
TTX(1) daily max-temperature
TTN(1) daily min-temperature
TTTM(2) daily mean temperature
-999.0 missing value

CREATE TABLE temperature (
DATUM STRING,
TT1 DOUBLE,
TT2 DOUBLE,
TT3 DOUBLE,
TTN DOUBLE,
TTTM DOUBLE,
TTX DOUBLE)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘,’
STORED AS TEXTFILE;

It is not intuitive to me why the field separator is defined in the table definition, but that’s apparently how Hive works. Load the data into the Hive table:

LOAD DATA LOCAL INPATH ‘temperature.txt’
OVERWRITE INTO TABLE temperature;

Iterate over the precipitation data:

wget http://data.smhi.se/met/climate/time_series/day/precipitation/SMHI_day_precipitation_clim_9720.txt

Trim off header information

- – - – - – - -
9720
STOCKHOLM-BROMMA
1961 1997
0101 1231
593537. 179513.
DATUM PES PQRR PRR PRRC1 PRRC2 PRRC3 PSSS PWS
- – - – - – - -

Replace leading and trailing spaces, and replace spaces between fields with commas:

cat SMHI_day_precipitation_clim_9720.txt | sed -e ‘s/^[ \t]*//’ | sed ‘s/[[:space:]]\+/,/g’ > precipitation.txt

Columns for  the precipitation data set:

DATUM YearMonthDay YYYYMMDD
PES(1) ground snow/ice code
PRR(2) precipitation mm
PQRR(3) quality code
PRRC1(4) precipitation type
PRRC2(4) precipitation type
PRRC3(4) precipitation type
PSSS(5) total snow depth cm
PWS(3) thunder, fog or aurora borealis code
-999.0 missing value

Create the Hive precipitation table:

CREATE TABLE precipitation (
DATUM STRING,
PES DOUBLE,
QRR DOUBLE,
PRR DOUBLE,
PRRC1 DOUBLE,
PRRC2 DOUBLE,
PRRC3 DOUBLE,
PSSS DOUBLE,
PWS DOUBLE)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘,’
STORED AS TEXTFILE;

And load the precipitation data into the Hive table:

LOAD DATA LOCAL INPATH ‘precipitation.txt’
OVERWRITE INTO TABLE precipitation;

Let’s define a snowy day as a day that has a temperature below 0 degrees Celsius (freezing) with a precipitation of more than 3 mm (approximately 30 mm snow).

- Number of snow days grouped by year
TTTM Temperature < 0 degrees Celsius
PRR Percipitation > 3 mm (approximately 3 cm snow)

SELECT year(from_unixtime(unix_timestamp(precipitation.datum, ‘yyyyMMdd’))), count(*)
FROM precipitation join temperature on (precipitation.datum = temperature.datum)
AND temperature.TTTM < 0
AND precipitation.PRR > 3
GROUP BY year(from_unixtime(unix_timestamp(precipitation.datum, ‘yyyyMMdd’)));

Let’s execute the query:

hive> SELECT year(from_unixtime(unix_timestamp(precipitation.datum, ‘yyyyMMdd’))), count(*)
> FROM precipitation join temperature on (precipitation.datum = temperature.datum)
> AND temperature.TTTM < 0
> AND precipitation.PRR > 3
> GROUP BY year(from_unixtime(unix_timestamp(precipitation.datum, ‘yyyyMMdd’)));

Total MapReduce jobs = 2
Launching Job 1 out of 2
…

Starting Job = job_201201290156_0082, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201201290156_0082
Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201201290156_0082
2012-01-29 13:26:24,615 Stage-1 map = 0%, reduce = 0%
2012-01-29 13:26:33,272 Stage-1 map = 50%, reduce = 0%
2012-01-29 13:26:36,288 Stage-1 map = 100%, reduce = 0%
2012-01-29 13:26:47,395 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201201290156_0082

Launching Job 2 out of 2
…

Starting Job = job_201201290156_0083, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201201290156_0083
Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201201290156_0083
2012-01-29 13:26:54,675 Stage-2 map = 0%, reduce = 0%
2012-01-29 13:26:59,694 Stage-2 map = 100%, reduce = 0%
2012-01-29 13:27:10,791 Stage-2 map = 100%, reduce = 100%
Ended Job = job_201201290156_0083
OK
1961 6
1962 6
1963 7
1964 6
1965 7
1966 8
1967 7
1968 5
1969 7
1970 12
1971 8
1972 4
1973 8
1974 3
1975 3
1976 10
1977 13
1978 8
1979 6
1980 7
1981 17
1982 5
1983 8
1984 5
1985 19
1986 13
1987 4
1988 11
1989 4
1990 1
1991 3
1992 4
1993 6
1994 1
1995 6
1996 2
1997 6
Time taken: 52.914 seconds

Notice how Hive transform the SQL-query into MapReduce jobs. We could of course do this ourselves in Java, but we’d be swamped in code. Hive hides the underlying complexity of MapReduce in the form of more convenient and mainstream SQL.

Hive supports subqueries, let’s calculate the average number of snow days:

SELECT AVG(sum)
FROM (
SELECT year(from_unixtime(unix_timestamp(precipitation.datum, ‘yyyyMMdd’))), count(*) as sum
FROM precipitation join temperature on (precipitation.datum = temperature.datum)
AND temperature.TTTM < 0
AND precipitation.PRR > 3
GROUP BY year(from_unixtime(unix_timestamp(precipitation.datum, ‘yyyyMMdd’)))
) t;

From these calculations it seems like the worst case scenario for the snow removal service is 19 occasions per year with an average of 7.

Pretty sweet huh!

While hesitating to buy a new transmitter I stumbled upon the multiwii.com project, a project devoted to make multicopters (tricopters, quadcopters, hexacopters) fly with the help of Arduino (programmable microprocessor), Nintendo Wii Motion Plus (gyro), and Nintendo Nunchuck (accelerometer). It turns out that the gyro and accelerometer used in the Nintendo Wii hand controls are pretty competent. The Arduino is like a very small computer with enough processing power to make real time decisions based on the inputs from the Nintendo PCBs and the user. A tricopter is an inherently instable flying machine, and without the help of electronics (configured correctly) it will flip in a matter of milliseconds. The combination of the three main components makes for a very stable flying platform. Since these units are mass produced they come with a relatively low price tag.

Being a happy Arduino hobbyist I immediately fell in love with the idea to use Arduino as the tricopter’s brain, and as a bonus I could use my old transmitter. The mutilwii project comes with a two programs; an Arduino program that needs to be configured with your multicopters specific settings, such as number of rotors, min and max rpm, yaw direction and a few other parameters; and the MultiWiiConf program used to configure and calibrate the multicopter. Even though I followed the assembly instructions as close as I could, it took a while to sort things out. The soldering of the tiny Nintendo components and connecting all the wires to the Arduino proved to be quite a challenge. Finally when I had found the correct positions for the gyro (Wii Motion Plus) and the accelerometer (Nintendo Nunchuck), I could calibrate the thing and spin up the motors for the first time.

Equipped with three brushless motors, each capable of producining almost 1 kilo of thrust and mounted on the tip of long arms, the tricopter has enough punch to break lose from your hand if not  gripped firmly. With safety goggles on I did a lot of test runs, first holding the thing in my hand, and later on the tarmac outside my maker shed. Luckily nothing vital broke and no one got injured. After a few iterations of weeding out vibrations, trims, and other configurations; it finally took off!

“The solution you proposed last week did not work out. In fact it caused us to lose precious time and now we have to do it quick and dirty anyway.”

Pete is trying to make himself smaller to avoid being crushed under Tom.

“Oh, I’m so sorry, I shouldn’t have aired that stupid idea in the first place. I’m just awfully sorry. Stupid me!”

“Yeah, you’re right”, says Tom–now sitting in his chair again.

“May I add something?” Glen continues without waiting for an answer–”According to our development method, we should spend 15% of our time not doing project stuff. I read this very interesting article on creativity that said most inventions are made when we are doing non-productive things.” (Glen is winking his long finger and index finger on both hands when he says “non-productive”.)

Tom stares at Glen for a while and says: ”Will you get to the point?”

- – - – -

Let’s examine what’s going on here. According to (now late) family therapist Virginia Satir there are five styles of communications and four of them are dysfunctional (!). In a conversation between two people there are three components: me, you, and the context. Only by taking all three parts into account the conversation can be congruent and fully functional. If you remove any part (me, you, comtext), the conversation becomes dysfunctional. Let’s start with Tom. Tom is using the blaming style, leaving out the other person. He’s emphasizing the blaming style using body language. Pete is leaving himself out. He turn himself into a victim and swallow all of Tom’s accusations. Pete is placating. Glen on the other hand is leaving pretty much all out, being hyper-reasonable and irrelevant. No one is being congruent and I doubt this group is very effective.

What communication styles have you seen in action today?

For quite some time the brain was thought of as nothing more than a radiator for the heart–some kind of fancy heat pipe system to cool the blood. It’s true, without a hat in cold weather you loose body heat quickly; but now we know the brain plays a more important role than being a mere heat-exchanger. This doughy blob of fatty goo called brain, with its 100 gad-zillion or so grey and white brain cells, is the most amazing thing nature’s devised.

Most of the time me and my brain cooperates happily and we can be quite productive. We’re not perfect (who cares), but I know we’ve done great stuff together. I owe everything I know to my brain. My brain is dear to me, and I like it for what it is. But sometimes my brain paralyses me and descends me into a dark hole from where I have little or no access to my mind. In this hole I can’t think nor feel normally. In the darkness of this hole my mind finds joy only in figuring out different ways to end itself, and its host: ME. Sometimes my brain gives me limited access to its resources, leaving me in a somewhat sub-functional state, without access to vitality and creativity. Then I feel rightfully boring and dull. Sub-human. And people complain.

From a neuroscientific perspective there’s no distinction between brain and mind and self. Your brain is really YOU. From my personal therapeutic perspective, I find it useful to think about my brain as something separate from ME. That I can look at my mind from a meta level. After all; main lesson from Cognitive Behavior Therapy is “you’re not your thoughts, nor your feelings.” So I like to think of my brain as a devise. Something very useful, essential, and delicate. Something that requires care and attention to work properly. For me, as you’ll see, all this is a rather new way of thinking.

Perhaps my genes and/or my environment made me susceptible to this condition. Perhaps I simply didn’t treat my brain with as much care as it deserved. There was a time when I was determined to succeed with whatever I did; and for a while I was successful. I felt invincible. I absorbed problems around me like a sponge and I wanted to put things right. Fix things. I adopted them as my own. I didn’t notice my mindset was very close to what ancient Greeks would call “hubris.” When I had my first real setbacks, I blamed myself, and tried to work harder. Praise I got washed away quickly, but blame stuck more permanently. I exposed myself to stress without taking time to recover.  I stopped doing physical exercise. I gave my brain no slack. I thought I was just being ambitious. I thought I was doing the right thing. Not.

Then it happened, now five years ago: Whack! One morning I could not transport myself from bed to kitchen. Nor could I continue to work on the software I was supposed to build from scratch to production within seven weeks. Something was broken. Lost. Indeed, I saw it coming, but I didn’t expect the fall to be so hard. My brain’s way of saying: Stop man, gimme a break! So I met the doctor and the doctor prescribed rest, pills, and hooked me up with a therapist. I recovered and got back to work. Did my relationship with my brain change? Did I start to treat my brain any differently? No, not at all. I blamed my brain for being weak and inferior. A year later or so I found myself falling back into the hole, lying in bed incapacitated to think or feel–except feel bad about myself. Another therapist, more pills, and back to work. I was stumbling. And so a year ago, the big one hit me. What the hell. The problem solver in me just kept getting increasingly frustrated because it could not identify the point of failure. OK, I was under stress, yes, I f**ked up at work, yes, I got some not-so-encouraging-feedback, yes. Yikes. So what!? In a world where half of the population is starving and don’t have access to a toilet, how can I be obliged to feel the way I do? Why don’t I just shape up? I have more than enough don’t I… I was beating my brain. Constantly.

As far as I’ve understood, depression (like most mental illnesses) is the result of imbalances in the brain–signaling substances like serotonin, dopamine, noradrenaline … and physical connections between neurons, synapses, receptors, neurotransmitters, and other stuff that makes up the nervous system. Causes to these imbalances are more or less unknown, even to science. Although science know a whole lot more than it did just decades ago, we still know too little. It’s obvious when you start looking for treatments for depression; myriads of pills, therapies, things that could be labeled new age treatments and pure hocus-pocus–what an industry! So far my treatments have been a mix of pills and therapy. There are many pills out there and I’ve only tested a couple of them. The ones I’ve tested did help me when I was at my lowest points, but prolonged use made me drowsy and passive. Helpful but blunt tools. Like a gentle smack with a sledgehammer–great for disrupting a depression, but not without its side effects. Therapy can be great I think. I believe communication is a vital part of recovering from depression. What the depressed mind wishes the most, is to hide from the world, preferably in a dark hole. My therapists have been wise, and helped me see alternatives to what my depressed mind has suggested.

I was laid off last summer. Due to recession apparently. Being laid off like this made me angry and loathful and bitter for a while, but it slowly dawned on me that the only way out of my unhappiness was the unconditional surrender of my ego–to fully let go. Although painful, I think now the separation was all for the better–I had to break some bad old patterns. This pause has been precious to me. I’ve had time to contemplate, learn to bake sourdough bread, play harmonica, goof around with my wonderful girls, read, write, watch horror movies at noon on a Wednesday, feed birds, program my own programs, experiment with cooking, make my own blueberry ice-cream, and I’ve grown confident that I get better and better at controlling my condition. As of today I’m off pills, actively improving my physical condition (swimming mostly); learning more about the brain (a fascinating subject) and depression (out of necessity); trying to practice mindfulness (swimming again); eating, breathing and sleeping right; and reconnecting with the world.

I dare to say these things because I know I’m not alone–far from it. Over a lifetime, one of five falls into depression, all according to statistics. For good and bad, I’ve been straight forward with my depression and in return people have been straight forward with me, and let me in on their own personal experiences. I’m grateful for that, because when suspended in darkness, it helps an awful hell of a lot to know you’re not alone.

Much love.

Your grandchildren lying here looking beautiful at peace with their dreams. Your daughter just called in the message we’ve all been waiting for. I’ve braced myself for this moment, saved my tears for this. Now I let them start rolling down my cheeks.

Although your body was failing and was ready to give in, you cling on to life like you were climbing Mount Everest. But unlike any climber trying to reach the top of the world, your climb was no day-trip. Cancer had been doing its crippling work for many years. But you were strong, so strong–just like your mother. Your strength’s now passed on to your daughter and your grandchildren–my daughters.

We never said goodbye, just see you another day. Until that day, I’ll miss you dearly.

What do you think?