Iterating over joins in Pig

Apache Pig is a fantastic language for processing data. It is sometimes incredibly annoying, but it beats the hell out of writing a ton of map reduces and chaining them together. When iterating over joins, an issue that I know that I’m not the only one having ran into is referencing data after a join in pig.

Normally, you access fields using the dereference operators . or # depending on the data type. The period symbol, . is used for tuples and bags, i.e. tuple.field0, tuple.field1, bag.field0, bag.field1. Maps are dereference with a hash, #, i.e. map#’field0′, map#’field1′.

This does not work after a join. The expected iteration after a JOIN:


joined = JOIN list0 BY key, list1 BY key;
purified = FOREACH joined GENERATE list0.key;


This will fail with the obscure error: “scalar has more than one row in the output”. This error message is a known problem is and there is a . As can be seen from the ticket, the correct way to iterate over the join is by using the relation operator, :: instead of the dereferencing operators like this:


joined = JOIN list0 BY key, list1 BY key;
purified = FOREACH joined GENERATE list0::key;


If you fall for the temptation of skipping the name of the list to get the field from like this:

joined = JOIN list0 BY key, list1 BY key;
purified = FOREACH joined GENERATE key;


You will get the more informative message: “Found more than one match: list0::key, list1::key”.

What you are really doing after a join is addressing columns in relations. For users, addressing columns in a relation with a period would be easier, but using :: might make the underlying code easier to understand.

A fistful of links

Here’s a list of links I tweeted over the last week or so (as requested by @dmpetersson). I considered shamelessly removing the RT info, but came to my senses and carried them over. Clearly shows that the majority of my links are retweets, but then again more people might discover the interesting people who originally tweeted the links and follow them. I will start posting these digests weekly going forwards.

• Fedora 11 is here – looks like a nice upgrade from Fedora 10 and remains a good alternative to Ubuntu.
• Benchmarks: You are Doing it Wrong – interesting read about benchmarking web applications
• RT @Venture_Capital Do Young Venture Capitalists Have an Edge? – Experience is good, but so is openness to new ideas
• RT @DanekS How to write a tech press release – don’t show off. Journalists and editors may not understand geek
• RT @tferriss: Find out which interests Google associates with your cookie – who does Google think you are?
• RT @ruby_news: Rack 1.0 Released
• RT @tferriss: How to choose colors for your brand – great samples: via @gmc