MediaWiki - User contributions [en]

Topic:R8w8asd9f7fg0drz

2013-05-29T19:54:09Z

Dsc (talk | contribs) edited a post on "What I did in the hackathon"

Thread:Talk:Amsterdam Hackathon 2013/What I did in the hackathon/reply (5)

2013-05-29T19:54:09Z

Dsc: derf

In addition to helping with UserMetrics2 as [[User:DAndreescu|DAndreescu]] said above, I hammered out a MediaWiki extension enabling Limn graphs to be embedded via wikitext:

https://github.com/dsc/limn-mediawiki-ext

The embedding API is provided via a tag and a parser function which have identical functionality:

* Function: <code>{{#graph:<GRAPH_ID>|<OPTION_KEY>=<OPTION_VAL>|...}}</code>
* Tag: <code><graph graph-id="<GRAPH_ID>" <OPTION_KEY>="<OPTION_VAL>" ... /></code>

The graph definition and data will be pulled from an existing Limn server, configured via mw-config:

<pre>
require_once( "$IP/extensions/Limn/Limn.php" );
$wgLimnServerBase = 'http://dev-reportcard.wmflabs.org/';
$wgLimnServerRemoteMode = 'proxy';
</pre>

This is mostly a proof of concept, but I'm pretty excited about the future -- another step toward democratizing data exploration and visualization!

Topic:R8w8asd9f7fg0drz

2013-05-29T19:49:08Z

Dsc (talk | contribs) commented on "What I did in the hackathon" (In addition to helping with UserMetrics2 as DAndreescu said above, I hammered out a MediaWiki extension enabling Limn graphs to be embedd...)

Talk:Wikimedia Hackathon 2013

2013-05-29T19:49:08Z

Dsc: /* What I did in the hackathon */ Added reply

{{LiquidThreads imported revision}}

== Travel to Amsterdam ==

Our Philippine group is interested to join. However, we need your assistance on how we can ask for travel grant and the specific requirements needed like cost of accommodation, flight, etc. Maybe we can do what was done on GlamCamp Amsterdam 2011 where WMNL was the one who requested for travel scholarship. [[User:Exec8|Exec8]] ([[User talk:Exec8|talk]]) 17:33, 29 January 2013 (UTC)
:Hi Exec8, we have money for scholarships and we will publish information about how to apply for that as soon as possible. [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 14:27, 23 February 2013 (UTC)
::Ok. Kindly notify us the soonest so that we would have no issues with time when applying at the Dutch Embassy in Manila if qualified. [[User:Exec8|Exec8]] ([[User talk:Exec8|talk]]) 16:58, 26 February 2013 (UTC)
:::You can start by signing up for the hackathon as registration is now open [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 19:04, 7 March 2013 (UTC)
::::I updated [[Amsterdam Hackathon 2013/Scholarships]] [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 18:17, 8 March 2013 (UTC)

== Proposal for name and logo ==

[[File:MediaWiki Hackathon UNOFFICIAL DRAFT.svg|right|200px]]
Hi, today I got the talk to propose a name and a logo for the event. After [[Talk:Groups/Promotion#Naming_our_developer_events_23068|the discussion]] about naming of events and some additional thoughts, the proposal is simply:

'''MediaWiki Hackathon'''

It is a usual practice in conferences and events to have a name detached from location, year, edition, etc. Now the MediaWiki Hackathon is in Amsterdam, in the future it might be again there or somewhere else, but it will be our beloved MediaWiki Hackathon anyway.

Back to Amsterdam, the web page and other materials could use a standard formula like:

'''<big>MediaWiki Hackathon</big>'''

'''Amsterdam 24–26 May 2013'''

The logo proposed is also tied to the "MediaWiki Hackathon" and detached from a specific location, year or edition. It can be used in the future as such, unless we want to evolve it. There is also a [[media:MediaWiki Community UNOFFICIAL DRAFT.svg|version without text]].

The logo combines the sunflower (a reference to the technologies powering Wikimedia projects) inside the circle made out of 3 segments (representing the Wikimedia community). It follows the [//wikimediafoundation.org/wiki/Wikimedia visual identity guidelines Wikimedia visual identity guidelines] and you can find the sources at [[media:MediaWiki Hackathon UNOFFICIAL DRAFT.svg]]. [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 05:31, 21 February 2013 (UTC)
:Thanks Quim, I like the idea of having a standardised name, logo and design for all MediaWiki events. As for the logo itself, I prefer the more "techy" style that was for example used in [[:File:Hack-A-Ton-Berlin_2.jpg|Berlin in 2011]] or in [[:File:Hack-A-Thon-Brighton-Cropped.png|Brighton]] (as long as they follow the visual guidelines). Best, [[User:Nicole Ebber (WMDE)|Nicole Ebber (WMDE)]] ([[User talk:Nicole Ebber (WMDE)|talk]]) 12:39, 21 February 2013 (UTC)
::I'm happy to see that you agree with the naming.
::About the logo, I think it is better not to increase the iconic soup we already have at http://wikimedia.org - which is more than enough to confuse anybody out our communities (and many inside). The origin of those logos was [[media:Wikimedia Community Logo-Toolserver.svg|the Toolserver logo]]. That type of logo hasn't been used in past events and I would let it rest unless we really really want to push it forward.
::The sunflower has been our identity for tech since basically the beginning, and it is being used from https://mediawiki.org to https://wikitech.wikimedia.org/ . I think using the sunflower is the best that we can do to help strengthening the identity of our tech community at large and also to help keeping the soup of Wikimedia logos under control. [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 15:17, 21 February 2013 (UTC)
:::The WMF hackathon in Bangalore last year was called the [[Bangalore_DevCamp_November_2012|Wikipedia DevCamp]]. It was decided to use the already recognized wikipedia branding and call it 'Dev' because in India thee word 'hack' has a negative connotation .
:::Wearing a t shirt with 'Wikipedia' increases the value of it so much more than 'mediawiki' :) [[User:Planemad|Planemad]] ([[User talk:Planemad|talk]]) 16:46, 21 February 2013 (UTC)
::::Hi Arun! Is there an original SVG version of [[:File:DevCamp Bangalore header.png]] available?
::::Erik also thinks that promoting "Wikipedia" for outreach events is better. I don't know if the Amsterdam organizers have any designer around but in any case your logo could be tweaked to satisfy a "Wikipedia Hackathon" or "Wikipedia DevCamp" in Amsterdam as well. Even that orange would fit, if needed. :)
::::Erik is proposing to keep relying on [[media:Hackathon_Logo_Shiny.svg]] as basic identity for these outreach events. As you can see above, I really think we should limit the amount of visual elements we play with, and the hexagons + puzzle piece imho don't help. This is why I think Arun's design is a better starting point: the hexagons are there but almost gone and "Wikipedia" is spelled out. I would still remove the puzzle piece shape, leaving the "W". As soon as we have an SVG we can play with it. [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 21:13, 21 February 2013 (UTC)
:Hi Quim, you're calling this the '''MediaWiki''' hackathon, but that's too narrow. In the past it was always the '''Wikimedia''' developer meeting. Primary product of course MediaWiki, but not only that. [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 14:59, 23 February 2013 (UTC)
::If you look the last posts here, I'm in fact leaning towards "Wikipedia Something", since it's an out reach activity and that is the brand everybody knows. This naming is not to please you or me or anybody already in the Wikimedia / MediaWiki communities. This is about a name appealing for all the developers out there. In this context "Wikimedia" is even a worse choice than "MediaWiki" because there is even less developers that have heard this before.
::From a consistency point of view we could go for "Wikipedia DevCamp" or "Wikipedia Hackathon" and use a logo derived from [[media:DevCamp Bangalore header.png]]. [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 19:43, 23 February 2013 (UTC)
:::Can't believe I had not already done it, you can now find the original pdf of the DevCamp and associated events [http://commons.wikimedia.org/w/index.php?title=File%3AWikimedia_india_events_designs_2012.pdf&page=5 here]. Let me know if you need it in svg. [[User:Planemad|Planemad]] ([[User talk:Planemad|talk]]) 21:17, 23 February 2013 (UTC)
::::Is it possible to edit graphics in a PDF file? Are they in vector format still? I dont know enough. SVG is pretty standard for this kind of job. If you originally did it in something else please share anyway. The idea is to have the sources so someone else can do the adaptation for the Amsterdam event. Unless you volunteer... ;) [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 14:21, 26 February 2013 (UTC)
:::::I made a svg from that pdf file: [[:File:Bangalore DevCamp registration banner.svg]]. I hand-edited the code to remove cruft from the conversion and group elements together appropriately, so it should be easy to adapt this for other events. [[User:Waldir|Waldir]] ([[User talk:Waldir|talk]]) 05:44, 9 March 2013 (UTC)
:Can we reach to a conclusion? Please have a say. Or am I worrying too much / too soon about our lack of agreed name and logo? If so let me know as well, I don't want to be annoying.
:Summary of the proposal so far:
:'''<big>Wikipedia Hackathon</big>'''
:Logo derived from [[:File:DevCamp_Bangalore_header.png]]
:Reasoning:
:* Wikipedia is the brand everybody knows. It is inaccurate but "MediaWiki" and "Wikimedia" are in comparison really weak for outreach, and they are not perfect denominations either. Using this name here helps us also identifying Wikipedia as a software project, contributing to solve an identified problem.
:* Hackathon is the term used in Europe for the past years. There would be a value in consolidating a global name: "Wikipedia DevCamp". But being the event so close and having talkedso much about the "Amsterdam Hackathon" a change like this would be confusing. [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 14:43, 26 February 2013 (UTC)
::Fine with me. :) [[User:Nicole Ebber (WMDE)|Nicole Ebber (WMDE)]] ([[User talk:Nicole Ebber (WMDE)|talk]]) 21:40, 26 February 2013 (UTC)
:My 2 cents:
:* Visual identity: '''+1 for the Puzzle theme''' and ''Tech'' and everything but not the sunflower. The sunflower doesn't seem at all related to technology, especially when pulling it out of the "<nowiki>[[ bracket ]]</nowiki>" context. I'd say either use the MediaWiki logo as part of it, or not at all (but not just the sunflower). Actually, though it may have been a good logo for MediaWiki (though maybe not even that), remember that in MediaWiki the logo is for users, not developers.
:* Naming: Using the Wikipedia brand does sound better, though Wikimedia is getting more known in the public and probably good enough for this purpose. I wouldn't recommend calling it "MediaWiki Hackathon" since that's both too narrow and too unknown a term. '''+1 for Wikimedia''' as part of the name (be it ''Wikimedia Hackathon'', ''Wikimedia developer meeting'' or something else entirely).
:<div style="float: left;">[[File:Hackathon Logo Shiny.svg|206x206px]]</div>
:<div style="float: left; background: #444; width: 300px; text-align: center; margin: 20px; padding: 20px 30px 20px 20px;">[[File:Hackathon Berlin Logo Full.svg|300x300px]]</div>
:{{clear}}
:Also, if we call it "Wikimedia (something)" we might want to consider not creating a separate logo but putting it in the Wikimedia family (like chapters and various other initiatives).
:<div style="text-align: center; width: 270px;">
:[[File:Wikimedia logo text RGB.svg|270x270px]]
:Hackathon
:</div> [[User:Krinkle|Krinkle]] ([[User talk:Krinkle|talk]]) 00:31, 27 February 2013 (UTC)
::See my last proposal. There is no sunflower at [[:File:DevCamp Bangalore header.png]] so we can end that discussion.
::I really think that "Wikimedia" name and logo are no match for "Wikipedia" brand as shown at [[:File:DevCamp Bangalore header.png]].
::"Wikimedia is getting more known in the public"? Well, maybe around our circles but I can check anytime with actual developers out there and do the test: all of them know Wikipedia, almost all of them need a 3 minute explanation to understand what is "Wikimedia". [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 01:58, 27 February 2013 (UTC)
:::I'm with Krinkle on this one. Wikipedia might be a strong brand, but that doesn't mean we have to use it. [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 18:01, 2 March 2013 (UTC)
::Just thought I would upload this. I worked on this last year and never did anything with it. I've updated it for this year. It's obviously got a bit of focus on how these are done in different cities all over the world - not sure if that's a theme worth emphasizing.
::By the way - the MediaWiki logo being used is [[Media:Concept-MediaWiki-Logo-Flower.png|this one]] - it could be changed to the official logo of course.
::[[Image:Concept-Hackathon-Logo-AMS.png|500x500px]]
::[[Image:Concept-Hackathon-Logo-TXL.png|500x500px]] [[User:Trevor Parscal|Trevor Parscal]] ([[User talk:Trevor Parscal|talk]]) 00:23, 28 February 2013 (UTC)
:::I like em... good solid lines and colors suitable for tshirts. :) [[User:Brion VIBBER|brion]] ([[User talk:Brion VIBBER|talk]]) 23:20, 28 February 2013 (UTC)
:::+1 (though of course the use of wikitext in them will rapidly become anachronistic, given the VisualEditor… <tt>;-)</tt>) [[User:Jdforrester (WMF)|Jdforrester (WMF)]] ([[User talk:Jdforrester (WMF)|talk]]) 17:28, 1 March 2013 (UTC)
:After discussing with Sumana we agreed that this event is more of a Wikimedia-wide gathering than a real outreach event where we put a lot effort bringing newcomers.
:Hence this would work:
:'''Wikimedia Hackathon'''
:Please let's keep the city and year out of event name.
:Do you agree? [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 18:42, 2 March 2013 (UTC)
::<nowiki>[[Amsterdam Hackathon 2013]] = new(Wikimedia Hackathon)</nowiki> ;-)
::Fine with me. [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 18:49, 2 March 2013 (UTC)
:::This is an SVG with outlined text. I've also sent a set of SVGs and PDFs, along with original artwork and the needed font (so this can be edited in Inkscape on Linux) to Rachael (who is handling the printing).
:::[[Image:Wikimedia_Hackathon_-_Amsterdam_2013.svg|500x500px]] [[User:Trevor Parscal|Trevor Parscal]] ([[User talk:Trevor Parscal|talk]]) 20:10, 8 March 2013 (UTC)
::::Thank you very much, Trevor! [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 04:37, 9 March 2013 (UTC)

== Archived date discussion ==

{{collapse top|Discussion to determine date}}
=== Straw Poll (closed) ===
We're aiming to do the hackathon in May 2013 with a preference for the weekend of Saturday the 25th. The options are :
* Weekend of the 4th of May. This weekend is [[:en:Remembrance of the Dead|Remembrance of the Dead]] and [[:en:Liberation Day (Netherlands)|Liberation Day]] in the Netherlands so we rather not do it this weekend.
* Weekend of the 11th of May. [[:en:Ascension day|Ascension day]] will be on Thursday the 9th of May so a lot of people will go on a long weekend. Probably means more expensive ho(s)tels.
* Weekend of the 18th of May.[[:en:Pentecost|Pentecost]] will be on Monday the 20th of May so a lot of people will go on a long weekend. Probably means more expensive ho(s)tels.
* Weekend of the 25th of May. Our preferred weekend.

Please tell us when on which weekends you ''can't'' and if the preferred weekend is fine with you. If you like, let us also know ''why'' - we are especially interested in the dates for exams etc around Europe.

Use [http://www.doodle.com/3n5cg9ii7gvmcgum Doodle Time Finder] or leave a message below:

==== Specific requests to avoid a date ====
* Very difficult to do the weekends of either the 11th of May or 18th of May due to overlapping with RIPE (WMF just became a LIR and is required to attend) - https://ripe66.ripe.net/ . Weekend of the 18th is a little bit easier of the two. Cannot do the weekend of the 4th of May due to personal reasons. 25th of May weekend is fine. [[User:LeslieCarr|LeslieCarr]] ([[User talk:LeslieCarr|talk]]) 16:31, 1 December 2012 (UTC)
* The weekend of the 25th works, but I'd prefer another weekend (any other weekend is fine) --[[User:Catrope|Catrope]] ([[User talk:Catrope|talk]]) 22:09, 1 December 2012 (UTC)
* The 25th may interfere with Memorial Day weekend plans in the US (often this weekend is used for weddings, family vacations, graduations, etc). I would prefer one of the other weekends as I am fairly confident I will be invited to at least one wedding on that weekend :p [[User:Awjrichards|Arthur Richards]] ([[User talk:Awjrichards|talk]]) 23:45, 3 December 2012 (UTC)
* The 4th is a bad weekend for me (my little brother's birthday, I may try to travel home), but any other weekend works, including Memorial Day. --[[Special:Contributions/216.38.130.166|216.38.130.166]] 01:27, 4 December 2012 (UTC)
* Just in case it matters, [http://linuxtag.de/2013/ LinuxTag] is in Berlin on May 22-25.--[[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 19:18, 2 December 2012 (UTC)
* Memorial Day/25th is bad for me. My brother is getting married. [[User:LWelling|LWelling]] ([[User talk:LWelling|talk]]) 19:03, 4 December 2012 (UTC)
* Google I/O is May 15th - 17th and may make the weekend of the 18th more difficult for the mobile team. [[User:LeslieCarr|LeslieCarr]] ([[User talk:LeslieCarr|talk]]) 20:19, 4 December 2012 (UTC)
* All OK with me, but I have to teach the 17th in the afternoon. I'll be able to join in the evening and the rest of the weekend. [[User:Valhallasw|Valhallasw]] ([[User talk:Valhallasw|talk]]) 20:24, 4 December 2012 (UTC)
* Either date works fine, but mildly prefer to avoid weekend of 25th. -- [[User:RobLa-WMF|RobLa-WMF]] ([[User talk:RobLa-WMF|talk]]) 01:47, 5 December 2012 (UTC)
* Weekends of May 11, 18, 25 work. Prefer May 25 as LinuxTag is also held that week. -- [[User:Alolitas|Alolita Sharma]] 11 December 2012
* The last weekend of May is a no-go for me, for personal reasons. The other weekends are fine by me so far. --- [[User:Titopao|Titopao]] ([[User talk:Titopao|talk]]) 04:20, 12 December 2012 (UTC)
* Any weekend is equally good. If I'm not doing wikidata stuff, then doubt I can come regardless of which weekend. If I'm still in Berlin, then it doesn't matter and I can probably come, except I might like to go home for the Memorial Day (May 25) weekend. [[User:Aude|Aude]] ([[User talk:Aude|talk]]) 15:42, 7 January 2013 (UTC)

==== Specific requests to target a date ====
* Weekend of 25th is go for me. [[User:Siebrand|siebrand]] ([[User talk:Siebrand|talk]]) 20:48, 1 December 2012 (UTC)
* 25th or 18th works fine, I prefer the 18th [[User:Trevor Parscal|Trevor Parscal]] ([[User talk:Trevor Parscal|talk]]) 18:56, 3 December 2012 (UTC)
* Weekend of the 25th works for me. [[User:Steven (WMF)|Steven Walling (WMF)]] • [[User talk:Steven (WMF)|talk]] 18:30, 4 December 2012 (UTC)
* It would be great to have it back to back with MuseumNext, that way we can combine the hackathon with some WikiGLAM efforts: Amsterdam, 13 - 14 May 2013 --[[User:Martsniez|Martsniez]] ([[User talk:Martsniez|talk]]) 16:01, 6 December 2012 (UTC)
* 25th sounds good for me. --[[User:Lydia Pintscher (WMDE)|Lydia Pintscher (WMDE)]] ([[User talk:Lydia Pintscher (WMDE)|talk]]) 15:32, 7 December 2012 (UTC)
* Weekend<strike>s</strike> of the 4th <strike>and the 25th are</strike> is fine for me --[[User:Angelika Adam (WMDE)|Angelika Adam (WMDE)]] ([[User talk:Angelika Adam (WMDE)|talk]]) 16:08, 7 December 2012 (UTC)
* Weekend 25th sounds good too. Will try to make it. --[[User:Yurik|Yurik]] ([[User talk:Yurik|talk]]) 08:36, 19 December 2012 (UTC)

==== Fine with any of these dates ====
* Any week-end is fine to me. [[User:Hashar|Antoine "hashar" Musso]] ([[User talk:Hashar|talk]]) 13:09, 1 December 2012 (UTC)
* Any week-end is fine to me. --[[User:Kolossos|Kolossos]] 17:22, 1 December 2012 (UTC)
* right now any weekend is fine, but I might be in the process of selling my house in May, I won't know until later. [[User:Cmcmahon(WMF)|Cmcmahon(WMF)]] ([[User talk:Cmcmahon(WMF)|talk]]) 23:40, 3 December 2012 (UTC)
* Any of these are fine with me.--[[User:Jorm (WMF)|Jorm (WMF)]] ([[User talk:Jorm (WMF)|talk]]) 01:32, 4 December 2012 (UTC)
* Happy to attend any of the proposed dates [[User:Robmoen|Robmoen]] ([[User talk:Robmoen|talk]]) 18:37, 4 December 2012 (UTC)
* Any of the proposed dates would be great. [[User:Renklauf|Ryan Faulkner]] 18:49, 4 December 2012 (UTC)
* Any of those weekends is good for me. [[User:Anomie|Anomie]] ([[User talk:Anomie|talk]]) 19:13, 4 December 2012 (UTC)
* Any weekend in May is good for me. [[User:Zeljko.filipin(WMF)|Zeljko.filipin(WMF)]] ([[User talk:Zeljko.filipin(WMF)|talk]]) 19:48, 4 December 2012 (UTC)
* Currently any weekend is good for meeeee. [[User:Nikerabbit|Nikerabbit]] ([[User talk:Nikerabbit|talk]]) 20:06, 4 December 2012 (UTC)
* Any weekend works for me --[[User:Ori.livneh|ori]] ([[User talk:Ori.livneh|talk]]) 20:29, 4 December 2012 (UTC)
* Any of these work for me. [[User:Jdforrester (WMF)|Jdforrester (WMF)]] ([[User talk:Jdforrester (WMF)|talk]]) 20:47, 4 December 2012 (UTC)
* No preferences yet. [[User:Xqt|Xqt]] ([[User talk:Xqt|talk]]) 21:16, 4 December 2012 (UTC)
* Any date in May works for me [[User:CSteipp|CSteipp]] ([[User talk:CSteipp|talk]]) 01:20, 5 December 2012 (UTC)
* Anytime fine for me. --[[User:Brion VIBBER|brion]] ([[User talk:Brion VIBBER|talk]]) 20:26, 6 December 2012 (UTC)
* Any of these dates are fine with me. --[[User:Kai Nissen (WMDE)|Kai Nissen (WMDE)]] ([[User talk:Kai Nissen (WMDE)|talk]]) 15:58, 7 December 2012 (UTC)
* Any of these dates fine with me [[User:Johannes Kroll (WMDE)|Johannes Kroll (WMDE)]] ([[User talk:Johannes Kroll (WMDE)|talk]]) 19:06, 7 December 2012 (UTC)
* Ditto. Though, I don't usually know what I'm doing next week, never mind in 5 months ;). [[User:Reedy|Reedy]] ([[User talk:Reedy|talk]]) 19:54, 7 December 2012 (UTC)
{{collapse bottom}} [[User:Austin|Austin]] ([[User talk:Austin|talk]]) 17:01, 24 February 2013 (UTC)

== Topic suggestions ==

RFC: Topics
* ... [[User:Austin|Austin]] ([[User talk:Austin|talk]]) 17:23, 24 February 2013 (UTC)
:Topic suggestions please at [[Amsterdam Hackathon 2013/Topics]]. Thank you, [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 21:44, 25 February 2013 (UTC)

== redesign ==

Still working on it, continue tonight. [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 08:51, 26 February 2013 (UTC)
:It was left with various sections unreadable (black on grey). You may want to copy it to a sandbox next time (e.g. copy/paste to [[User:Henna/Amsterdam_Hackathon_2013]] and then copy back once done).
:I've polished it up a bit so that it is at least as readable as the old version. [[User:Krinkle|Krinkle]] ([[User talk:Krinkle|talk]]) 01:25, 27 February 2013 (UTC)
::Not only that, please also check the diff before saving, as you might be undoing legitimate edits (such as [http://www.mediawiki.org/w/index.php?title=Amsterdam_Hackathon_2013&diff=next&oldid=658642 here], where typos and spacing issues previously fixed were reintroduced) [[User:Waldir|Waldir]] ([[User talk:Waldir|talk]]) 13:36, 18 March 2013 (UTC)
:Visited links on dark gray are still unreadable, but maybe this only happens in the "See also" section? [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 15:13, 27 February 2013 (UTC)
::Trying to add [[image:Amsterdam_Zeeburg_dorm.jpeg|150x150px]] under the accomodation section but failing so far to get more then one line of text next to it :( [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 15:56, 14 March 2013 (UTC)
:::Like this? [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 18:10, 14 March 2013 (UTC)
::::yes :) I just figured it out myself (about 5 minutes aftery ou fixed it) d'oh thx [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 18:17, 14 March 2013 (UTC)

== Improving the web page ==

I just reverted the last changes in order to bring back the font color sanity. Having +5 font colors in an informative text is just not normal.

I'm not sure about the 2 column layout at the top of the page, at least in the way it is presented now. Looks ugly both in regular desktop screens (right column bigger height forces plenty of empty space in the left column) and in mobile (both content columns + the MediaWiki left navigation bar get squeezed making the page barely readable). Why not converting the content in the right column in a first horizontal block with extra design, and then have all the rest of information in mostly plain text + pictures? Or at least make the right column a proper floating box letting content around it flow and behave well in mobile (like infoboxes do).

Also, do we need to have all the past events listed at the top? Looks more of a "See also" type of content, and in fact there are past events already listed there. [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 16:11, 3 March 2013 (UTC)

== Apart from travel + accomodation, is there a conference fee? ==

I live in the Netherlands near Amsterdam, so I would not need travel or accomodation. I read the page to find either
* text specifying what the conference fee is, or
* text confirming that there is no conference fee.
I could find neither. Perhaps the page could clarify what the conference fee is, c.q. that there is no conference fee? Or, possibly, that there is a conference fee that will be announced at a later date? [[User:Sietse|Sietse]] ([[User talk:Sietse|talk]]) 11:16, 13 March 2013 (UTC)
:No fees. Free as in beer! [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 13:30, 13 March 2013 (UTC)
::Free as in beer indeed, I've updated the information on the page. [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 15:18, 13 March 2013 (UTC)

== Thursday arrival ==

The page says "€ 98,40 Euro fr-mo", but I'm sure I'm not the only person who will be arriving on Thursday. [[User:Pigsonthewing|Pigsonthewing]] ([[User talk:Pigsonthewing|talk]]) 18:39, 21 March 2013 (UTC)
:sorry I didn't see this. I've added the price thu-sun [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 11:53, 28 March 2013 (UTC)
::Thank you. [[User:Pigsonthewing|Andy Mabbett]] (Pigsonthewing); [[User talk:Pigsonthewing|Talk to Andy]]; [[Special:Contributions/Pigsonthewing|Andy's edits]] 15:47, 28 March 2013 (UTC)
::I'm concerned by [http://www.mediawiki.org/w/index.php?title=Amsterdam_Hackathon_2013&diff=666429&oldid=666414 this edit] . I shall arrive on Thursday and depart on Monday; I need a bed for ''four'' nights (Thur, Fri, Sat, Sun). [[User:Pigsonthewing|Andy Mabbett]] (Pigsonthewing); [[User talk:Pigsonthewing|Talk to Andy]]; [[Special:Contributions/Pigsonthewing|Andy's edits]] 14:23, 31 March 2013 (UTC)
:::Andy, I'm passing it on to the person who's taking care of the accomodation to doublecheck/what we can do to help [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 10:51, 1 April 2013 (UTC)
::::Hi Andy, the average costs per night is 31,50 + 1,30 Tax. Break up per night (not sure if this is final):
::::* thu 23 may 2013 1 night incl breakfast p.p. € 26,50
::::* fri 24-may-2013 1 night incl breakfast p.p. € 34,00
::::* sat 25-may-2013 1 night incl breakfast p.p. € 34,00
::::* sun 26-may-2013 1 night incl breakfast p.p. € 26,50 [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 14:32, 1 April 2013 (UTC)

== Tram & train ==

What tram do we get from Centraal to the hostel? How much is it? Can we pay cash on board, or do we have to buy tickets at the station?

Is it still €2.50 for a single ticket from Schiphol Airport to Centraal? Is there left-luggage at Centraal?

Perhaps we can arrange a meetup at Centraal at, say, 3.30 or 4pm on Thursday? [[User:Pigsonthewing|Andy Mabbett]] (Pigsonthewing); [[User talk:Pigsonthewing|Talk to Andy]]; [[Special:Contributions/Pigsonthewing|Andy's edits]] 15:51, 28 March 2013 (UTC)
:* [http://www.ns.nl/reizigers/reisinformatie/stationsvoorzieningen/stations/a/amsterdam-centraal/algemeen/bagagekluizen.html Luggage storage @ Amsterdam Central Station] costs 3.70 or 5.45 Euro per 24 hours, depending on the size of the locker. You cannot open the locker without having to pay again.
:* A one way trip from Schiphol Airport to Amsterdam Central Station costs 3.90 Euro
:IMO it's too early to arrange a time for Thursday already, but I'll try to be there... [[User:Siebrand|siebrand]] ([[User talk:Siebrand|talk]]) 18:05, 28 March 2013 (UTC)
::It probably is to early for someone who doesn't have as far to travel, but others of us have to plan ahead. [[File:Wink.png|22x22px]] ☠[[User:MarkAHershberger|MarkAHershberger]]☢([[User_talk:MarkAHershberger|talk]])☣ 18:37, 28 March 2013 (UTC)
:::We can probably find a volunteer if everybody's at Amsterdam Central together :)
:::Tickets for the tram are 2E80 and single fares can be bought on board. Alternative is to take the train to Muiderpoort. I've added the information to the page as well. [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 17:13, 29 March 2013 (UTC)
::::Let's do karaoke at http://www.theendkaraoke.nl/ some night! It's very close to Amsterdam Centraal =)
::::Who else is here on Thursday? Anyone want to do the museums? [[User:Judytuna|Judytuna]] ([[User talk:Judytuna|talk]]) 20:35, 21 May 2013 (UTC)
:::::I should be there Thursday AM. If you know your way around, I would love to have a guide. ☠[[User:MarkAHershberger|MarkAHershberger]]☢([[User_talk:MarkAHershberger|talk]])☣ 22:03, 21 May 2013 (UTC)

== Evenings ==

Are there any planned or recommended social activities for the evenings (including Thursday)? [[User:Pigsonthewing|Andy Mabbett]] (Pigsonthewing); [[User talk:Pigsonthewing|Talk to Andy]]; [[Special:Contributions/Pigsonthewing|Andy's edits]] 16:30, 28 March 2013 (UTC)
:Those of us who are getting there on Thursday should arrange to get together and tour the city. ☠[[User:MarkAHershberger|MarkAHershberger]]☢([[User_talk:MarkAHershberger|talk]])☣ 16:52, 28 March 2013 (UTC)
::we're organizing social activities on the evenings but I'm not sure about Thursday atm [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 17:12, 29 March 2013 (UTC)

== Lockers ==

Since I didn't see it anywhere here (maybe I missed it? maybe I shouldn't care?) lockers are available for €2,00 per 24 hours or €0,20 per hour. ☠[[User:MarkAHershberger|MarkAHershberger]]☢([[User_talk:MarkAHershberger|talk]])☣ 18:39, 28 March 2013 (UTC)
:[http://www.stayokay.com/en/hostel/amsterdam-zeeburg Their page] states they have luggage storage available, but it doesn't indicate at which price. [[User:Siebrand|siebrand]] ([[User talk:Siebrand|talk]]) 11:15, 1 April 2013 (UTC)
::The prices I listed were what they told me when I emailed them. ☠[[User:MarkAHershberger|MarkAHershberger]]☢([[User_talk:MarkAHershberger|talk]])☣ 13:56, 1 April 2013 (UTC)

== Suggestion to next year's organizers: publicize date poll via Facebook & wikitech-announce ==

This year we circulated info on Twitter & Identi.ca, toolserver mailing list, onwiki on various wikis, wikimedia-l, wikitech-l, and via other means to publicize the date poll. Next time, please use socialmedia{{@}}wikimedia.org and the wikitech-announce list to spread the word more widely. [[User:Sharihareswara (WMF)|Sharihareswara (WMF)]] ([[User talk:Sharihareswara (WMF)|talk]]) 20:30, 18 April 2013 (UTC)

== Does closed mean I have to cancel my train? ==

Was there a hard time limit for the registration? I paid for the train yesterday and tried to add my name here on the page. But now the registration is closed. Is there anything I can do? [[User:TMg|TMg]] 23:45, 28 April 2013 (UTC)
:Hi Tmg, we're looking into creating some extra places. We'll have a meeting tomorrow. We'll get back after the meeting. [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 21:30, 30 April 2013 (UTC)
::We had the meeting, and people that are on the waiting list without any requirements for funding, can come if they registered until futher notice.
::So you do not have to cancel your train. [[User:Siebrand|siebrand]] ([[User talk:Siebrand|talk]]) 19:19, 1 May 2013 (UTC)
:::Thank you. No, I don't need any funding. [[User:TMg|TMg]] 11:49, 2 May 2013 (UTC)
::::I don't need any accomodation since I live nearby. Is it still possible to attend the event? [[User:Edinwiki|Edinwiki]] ([[User talk:Edinwiki|talk]]) 19:03, 3 May 2013 (UTC)
:::::Please sign up for the waiting list and we'll get back to you. [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 21:00, 4 May 2013 (UTC)

== Waiting list ==

Hello :) I was just wondering how long the waiting list is?
I JUST signed up and I wanted to know my rough chances?
Is it like a 5 people in front of me kinda thing, or more like a 500 people in front of me?

Thank you! [[User:NutzTheRookie|NutzTheRookie]] ([[User talk:NutzTheRookie|talk]]) 12:52, 16 May 2013 (UTC)
:I can guarantee you it isn't 500 people. If you don't need a place to stay or can arrange your own, you may be able to just show up. But I'm not an organizer for the event, so you should verify with them. ☠[[User:MarkAHershberger|MarkAHershberger]]☢([[User_talk:MarkAHershberger|talk]])☣ 18:15, 20 May 2013 (UTC)
::Hello! Ages ago, I put my name on the list to indicate my intent to attend (#36!), and then didn't fill out the questionnaire until recently. I'm wondering if I can come too--I'd really love to!
::I don't require financial assistance. I'm also already in Amsterdam!!! Just got here from SF. Would it be possible to squash myself into one of the six-person rooms if there are any beds available starting on Thursday and going through Sundayish? ...Should I call the StayOkay Zeeburg and ask and reserve it?
::Thanks!! [[User:Judytuna|Judytuna]] ([[User talk:Judytuna|talk]]) 13:10, 21 May 2013 (UTC)
:::You should definitely call StayOkay if you want to stay there. But if you're already in Amsterdam, why do you want to stay at the hostel? ☠[[User:MarkAHershberger|MarkAHershberger]]☢([[User_talk:MarkAHershberger|talk]])☣ 14:58, 21 May 2013 (UTC)
::::I just arrived from the US and need a place to stay on those days. =) I was able to attach this to another trip! Thanks. [[User:Judytuna|Judytuna]] ([[User talk:Judytuna|talk]]) 15:07, 21 May 2013 (UTC)

== Towels ==

Some hostels do not provide towels. Does this one? Do we have to pay separately for them? [[User:Pigsonthewing|Andy Mabbett]] (Pigsonthewing); [[User talk:Pigsonthewing|Talk to Andy]]; [[Special:Contributions/Pigsonthewing|Andy's edits]] 15:12, 20 May 2013 (UTC)
:Make that most hostels (in my experience). It's the same here. You have to bring your own towels or rent them. [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 16:20, 20 May 2013 (UTC)

== 2 minute presentation proposal: Meteorjs + MediaWiki ==

I'd like to give a very brief introduction to Meteorjs and on how to build stuff related to MediaWiki with it such as [http://wikiytchanges.meteor.com/ wikifarmchanges], [http://manywiki.meteor.com/ manywiki] and even [https://github.com/guaka/mediawikixml2meteor2phonegap how to build android apps] such as [https://play.google.com/store/apps/details?id=io.cordova.hitchwiki.meteor.com this one for hitchwiki]. [[User:Guaka|Guaka]] ([[User talk:Guaka|talk]]) 20:34, 21 May 2013 (UTC)
:Hi Guaka! I'm sorry I missed this post during the hackathon but I'm also a meteorjs fan. I work for WMF on the analytics team and you can find me in IRC, #wikimedia-analytics as milimetric. Look forward to speaking with you. [[User:DAndreescu|DAndreescu]] ([[User talk:DAndreescu|talk]]) 18:23, 29 May 2013 (UTC)

== Removing MediaWiki from "getting started" session ==

The "get your development environment set up" session used to include MediaWiki setup with Vagrant. Our conference wifi couldn't handle 25 simultaneous attempts to download a 300-megabyte file, and most of the workarounds we tried were insufficient, so we've decided to instead be more thorough about getting Git set up. If you want to set up Vagrant and MediaWiki, and are having issues, and [[Mediawiki-Vagrant|the instructions on mediawiki.org]] are not helping, you should contact [[User:MarkTraceur]], who is at the event. [[User:Ori.livneh|Ori Livneh]], who is also at the hackathon, can also help you. [[User:Sharihareswara (WMF)|Sharihareswara (WMF)]] ([[User talk:Sharihareswara (WMF)|talk]]) 22:30, 24 May 2013 (UTC)
:Can we do a mirror on the local network or something? (For next time...) [[User:Brion VIBBER|brion]] ([[User talk:Brion VIBBER|talk]]) 13:29, 25 May 2013 (UTC)
::I agree - a local mirror, or something like it, would be better for next time. [[User:Sharihareswara (WMF)|Sharihareswara (WMF)]] ([[User talk:Sharihareswara (WMF)|talk]]) 10:13, 26 May 2013 (UTC)
:::Hand out CDs! :D [[User:Nemo_bis|Nemo]] 19:12, 26 May 2013 (UTC)

== Scribunto links ==

The links mentioned at the end of Scribunto's Sunday session are posted at [[Talk:Amsterdam Hackathon 2013/Workshops#Scribunto links]] [[User:Anomie|Anomie]] ([[User talk:Anomie|talk]]) 08:57, 26 May 2013 (UTC)

== Please take post-hackathon survey ==

If you attended the hackathon, please [https://wikimedia.qualtrics.com/SE/?SID=SV_8qcoZsFlZteNsBT take the quick survey!] That way we know what was good and what to fix for next year. It takes 3 minutes to fill in. [[User:Sharihareswara (WMF)|Sharihareswara (WMF)]] ([[User talk:Sharihareswara (WMF)|talk]]) 10:11, 26 May 2013 (UTC)

== Watch closing presentation on YouTube ==

https://www.youtube.com/watch?v=3YtZNzNdxqo [[User:Sharihareswara (WMF)|Sharihareswara (WMF)]] ([[User talk:Sharihareswara (WMF)|talk]]) 14:27, 26 May 2013 (UTC)

== Wiki loves monuments 2013 in Thailand ==

[//commons.wikimedia.org/wiki/Commons:Wiki_Loves_Monuments_2013_in_Thailand commons:Commons:Wiki_Loves_Monuments_2013_in_Thailand] [[User:Taweetham|Taweetham]] ([[User talk:Taweetham|talk]]) 14:50, 26 May 2013 (UTC)

== What I did in the hackathon ==

I think someone (Sumana?) asked us to post here with the stuff we did so people could follow-up, etc., so I'm kicking it off with this thread.

As I mentioned in the 30 second intro, I am interested in investigating ways to get the skin system in the path towards becoming a saner and more easily understandable component of mediawiki. I teamed with [[User:Luis Felipe Schenone|Felipe Schenone]] and [[User:Drev23|Michael Rubio]] and we collected what we hope to be the full set of documentation resources related to skins into a [https://docs.google.com/document/d/18fvyoCjPiHuJTJmmlcwZ9rBykkhZSbn_--W0z5gVbTQ shared google doc]. I proceeded to read up the pages and write summaries so the relevant ones can be then used to write a hopefully complete, coherent and readable description of the skinning system to be placed here on mw.org; this is still a work in progress, but the document is freely editable, so any comments, corrections or additions are welcome.

Felipe preferred a more hands-on approach, and worked on the changes that were eventually [https://gerrit.wikimedia.org/r/#/c/65319/ submitted to Gerrit], proposing a way to make the core skins more modular. That sparked an interesting discussion, and at the moment it seems what is needed is some general agreement within the dev community on what the goal for skins should be (e.g. making them into extensions?) so that steps to get to that could be taken towards that, with discussion focusing more on implementation details than on the merits of the changes' rationales. I will probably start a thread on the mediawiki-l to discuss this after I finish reading the documentation we collected.

Apart from that, I mostly did small changes, which I submitted to gerrit:
* [https://gerrit.wikimedia.org/r/#/c/65426/ Rename $usableSkins to $allowedSkins] (already merged)
* [https://gerrit.wikimedia.org/r/#/c/65440/ skin: Remove the id "anonlogin" and merge pt-anonlogin with pt-login]
* [https://gerrit.wikimedia.org/r/#/c/65446/ Rename Skin::getUsableSkins() to Skin::getAllowedSkins()]
* [https://gerrit.wikimedia.org/r/#/c/65554/ Rename variable _AdminPassword2 to _AdminPasswordConfirm for clarity]
* [https://gerrit.wikimedia.org/r/#/c/65613/ Several tweaks to the install.php script]

Finally, I did some [http://www.mediawiki.org/w/index.php?title=Special:Contributions/Waldir&offset=20130528000000&limit=70 edits to mediawiki.org] as I browsed around reading about the stuff I worked on. [[User:Waldir|Waldir]] ([[User talk:Waldir|talk]]) 00:01, 27 May 2013 (UTC)
:Great hackathon. I am very happy I was able to attend. I learned a lot and appreciate all of the interesting conversations. Great group of people. Special thanks to Mark for mentioning Gadgets.
:During the hackathon, I was able to repackage the extension called the [[Extension:SignWriting_MediaWiki_Plugin|SignWriting MediaWiki Plugin]] as a user script. I put a copy of the user script on the [http://en.wikipedia.org/wiki/User:Slevinski/signwriting_viewer.js English Wikipedia] and [http://incubator.wikimedia.org/wiki/User:Slevinski/signwriting_viewer.js on Incubator].
:After the hackathon, I found out that [http://incubator.wikimedia.org/wiki/User:MF-Warburg MF-Warburg] created a [http://incubator.wikimedia.org/wiki/MediaWiki:Gadget-Signwriting.js Gadget based on my script] and enabled it by default on incubator. Now everyone can see the [http://incubator.wikimedia.org/wiki/Wp/ase SignWriting script on Incubator]. This was my primary goal for the hackathon and I'm thrilled to see it become a reality. Now we can move the [http://ase.wikipedia.wmflabs.org/ ASL Wikipedia project] from Labs to [http://incubator.wikimedia.org/wiki/Wp/ase Incubator].
:My secondary goal was to enable SignWriting on the English Wikipedia. I have proposed the user script as [http://en.wikipedia.org/wiki/Wikipedia:Gadget/proposals#SignWriting_Viewer_gadget a gadget for the English Wikipedia], but regardless of the outcome, the user script can be enabled by any user who is interested using the instructions on [http://en.wikipedia.org/wiki/User:Slevinski my English Wikipedia user account page].
:Thanks to everyone involved in the hackathon. It was a wonderful event. [[User:Slevinski|Slevinski]] ([[User talk:Slevinski|talk]]) 14:05, 29 May 2013 (UTC)
:To continue on, everyone with an interest can follow my progress with my [[Extension:TranslateSvg|TranslateSvg]] extension at http://translatesvg.wmflabs.org -- and in particular, see [http://translatesvg.wmflabs.org/wiki/Special:Translate?group=Picturebook+4.svg&language=en] (select your own language in the top right). The hackathon was particularly focussed on making it play nice with "TUX", the recent overhaul of the Translate extension's visuals. [[User:Jarry1250|Jarry1250]] ([[User talk:Jarry1250|talk]]) 14:06, 29 May 2013 (UTC)
:[[User:Cmcmahon(WMF)|Chris]] wrote in detail what the two of us did at the hackathon. You can read the report at the QA list archive: http://lists.wikimedia.org/pipermail/qa/2013-May/000002.html [[User:Zeljko.filipin(WMF)|Zeljko.filipin(WMF)]] ([[User talk:Zeljko.filipin(WMF)|talk]]) 17:57, 29 May 2013 (UTC)
:Hi! Evan Rosen, David Schoonover, Andrew Otto, Jean-Frédéric Berthelot, and Dan Andreescu worked on re-platforming the user metrics API. The work is being done in a temporary repository and will be migrated to the main repository when it's ready: https://gerrit.wikimedia.org/r/gitweb?p=analytics/user-metrics-2.git;a=tree
:The focus of our effort:
:* improve reliability of the job queue using Celery
:* improve security using SQL Alchemy
:* improve accuracy and maintainability by unit testing as much as possible (using Nose)
:* improve readability by asking for code review as much as possible
:Right now, we're cleaning up some initial building blocks (the queue, database, and testing). We'll focus on the last two bullet points as soon as that's ready. [[User:DAndreescu|DAndreescu]] ([[User talk:DAndreescu|talk]]) 18:32, 29 May 2013 (UTC)
:In addition to helping with UserMetrics2 as [[User:DAndreescu|DAndreescu]] said above, I hammered out a MediaWiki extension enabling Limn graphs to be embedded via wikitext:
:https://github.com/dsc/limn-mediawiki-ext
:The embedding API is provided via a tag and a parser function which have identical functionality:
:* Tag: <code>{{#graph:<GRAPH_ID>|<OPTION_KEY>=<OPTION_VAL>|...}}</code>
:* Function: <code><nowiki><graph graph-id="<GRAPH_ID>" <OPTION_KEY>="<OPTION_VAL>" ... /></nowiki></code>
:The graph definition and data will be pulled from an existing Limn server, configured via mw-config:
:<syntaxhighlight lang='text'>require_once( "$IP/extensions/Limn/Limn.php" );
$wgLimnServerBase = 'http://dev-reportcard.wmflabs.org/';
$wgLimnServerRemoteMode = 'proxy';
</syntaxhighlight>
:This is mostly a proof of concept, but I'm pretty excited about the future -- another step toward democratizing data exploration and visualization! [[User:Dsc|Dsc]] ([[User talk:Dsc|talk]]) 19:49, 29 May 2013 (UTC)

== Photostitch of all projects ==

Some people asked me explicitly for this picture. [[File:Proposed projects for the Wikimedia Hackathon 2013.jpg|Stitch of all proposed projects]] [[User:Ter-burg|Ter-burg]] ([[User talk:Ter-burg|talk]]) 10:53, 27 May 2013 (UTC)
:This is indeed a nice picture! (and useful, the resolution is so good that it is possible to read the notes). I'll try to find an excuse to use it somewhere. Thank you! [[User:Qgil|Qgil]] ([[User talk:Qgil|talk]]) 20:08, 28 May 2013 (UTC)

Thread:Talk:Amsterdam Hackathon 2013/What I did in the hackathon/reply (5)

2013-05-29T19:49:08Z

Dsc: Reply to What I did in the hackathon

In addition to helping with UserMetrics2 as [[User:DAndreescu|DAndreescu]] said above, I hammered out a MediaWiki extension enabling Limn graphs to be embedded via wikitext:

https://github.com/dsc/limn-mediawiki-ext

The embedding API is provided via a tag and a parser function which have identical functionality:

* Tag: <code>{{#graph:<GRAPH_ID>|<OPTION_KEY>=<OPTION_VAL>|...}}</code>
* Function: <code><graph graph-id="<GRAPH_ID>" <OPTION_KEY>="<OPTION_VAL>" ... /></code>

The graph definition and data will be pulled from an existing Limn server, configured via mw-config:

<pre>
require_once( "$IP/extensions/Limn/Limn.php" );
$wgLimnServerBase = 'http://dev-reportcard.wmflabs.org/';
$wgLimnServerRemoteMode = 'proxy';
</pre>

This is mostly a proof of concept, but I'm pretty excited about the future -- another step toward democratizing data exploration and visualization!

Wikimedia Hackathon 2013/Attendees

2013-05-20T20:03:46Z

Dsc:

<noinclude>
Please indicate here your attendance (your plan to come). Don't forget to fill out the [https://docs.google.com/a/wikimedia.nl/spreadsheet/viewform?formkey=dFg2SmRRbkpxNmxCcFNFdlduVlJuTUE6MQ#gid=0 registration form]!
</noinclude>
# [[User:Brion VIBBER|brion]] ([[User talk:Brion VIBBER|talk]]) 19:08, 11 February 2013 (UTC)
# [[User:Ed Lane|Ed Lane]] ([[User talk:Ed Lane|talk]]) 21:19, 10 February 2013 (UTC)
# [[User:Petrb|Petrb]] ([[User talk:Petrb|talk]]) 09:32, 21 February 2013 (UTC)
# ☠[[User:MarkAHershberger|MarkAHershberger]]☢([[User_talk:MarkAHershberger|talk]])☣ 21:42, 10 February 2013 (UTC)
# [[User:Multichill|Multichill]] ([[User talk:Multichill|talk]]) 20:51, 10 February 2013 (UTC)
# [[User:TheDJ|TheDJ]] ([[User talk:TheDJ|talk]]) 23:29, 10 February 2013 (UTC)
# —''[[User:Ruud Koot|Ruud]]'' 10:40, 22 February 2013 (UTC)
# [[User:Siebrand|siebrand]] ([[User talk:Siebrand|talk]]) 23:32, 25 February 2013 (UTC)
# [[User:Dereckson|Dereckson]] ([[User talk:Dereckson|talk]]) 14:42, 26 February 2013 (UTC)
# [[User:Drev23|Drev23]] ([[User talk:Drev23|talk]]) 15:42, 26 February 2013 (UTC)
# [[User:Eloquence|Eloquence]] ([[User talk:Eloquence|talk]]) 21:59, 26 February 2013 (UTC)
# [[User:Henna|Henna]] ([[User talk:Henna|talk]]) 07:11, 27 February 2013 (UTC)
# [[User:Chitetskoy|Chitetskoy]] ([[User talk:Chitetskoy|talk]]) 13:36, 27 February 2013 (UTC)
# [[User:Silke WMDE|Silke WMDE]] ([[User talk:Silke WMDE|talk]]) 16:13, 1 March 2013 (UTC)
# [[User:Mutante|Mutante]] ([[User talk:Mutante|talk]]) [[User:Mutante|Mutante]] ([[User talk:Mutante|talk]]) 15:00, 2 March 2013 (UTC)
# [[User:Darkdadaah|Darkdadaah]] ([[User talk:Darkdadaah|talk]]) 13:44, 5 March 2013 (UTC)
# [[User:BJorsch (WMF)|BJorsch (WMF)]] ([[User talk:BJorsch (WMF)|talk]]) (aka [[User:Anomie|Anomie]] ([[User talk:Anomie|talk]])) 14:20, 5 March 2013 (UTC)
# [[User:Zeljko.filipin(WMF)|Zeljko.filipin(WMF)]] ([[User talk:Zeljko.filipin(WMF)|talk]]) 10:29, 6 March 2013 (UTC)
# [[User:Sharihareswara (WMF)|Sharihareswara (WMF)]] ([[User talk:Sharihareswara (WMF)|talk]]) 19:32, 6 March 2013 (UTC)
# [[User:LeslieCarr|LeslieCarr]] ([[User talk:LeslieCarr|talk]]) 22:22, 6 March 2013 (UTC)
# [[User:Jdforrester (WMF)|Jdforrester (WMF)]] ([[User talk:Jdforrester (WMF)|talk]]) 21:33, 7 March 2013 (UTC)
# [[User:JackPotte|JackPotte]] ([[User talk:JackPotte|talk]]) 21:39, 7 March 2013 (UTC)
# [[User:RobLa-WMF|RobLa-WMF]] ([[User talk:RobLa-WMF|talk]]) 19:45, 8 March 2013 (UTC)
# [[User:Catrope|Catrope]] ([[User talk:Catrope|talk]]) 20:33, 8 March 2013 (UTC)
# [[User:Ssastry|Ssastry]] ([[User talk:Ssastry|talk]]) 03:50, 9 March 2013 (UTC)
# [[User:Jane023|Jane023]] ([[User talk:Jane023|talk]]) 08:57, 9 March 2013 (UTC)
# [[User:ABaso(WMF)|ABaso(WMF)]] ([[User talk:ABaso(WMF)|talk]]) 18:56, 10 March 2013 (UTC)
# [[User:GWicke|Gabriel Wicke (GWicke)]] ([[User talk:GWicke|talk]]) 19:53, 11 March 2013 (UTC)
# [[User:Hashar|Antoine "hashar" Musso]] ([[User talk:Hashar|talk]])
# [[User:Cmcmahon(WMF)|Cmcmahon(WMF)]] ([[User talk:Cmcmahon(WMF)|talk]]) 20:06, 11 March 2013 (UTC)
# [[User:MarkTraceur|MarkTraceur]] ([[User talk:MarkTraceur|talk]]) 20:12, 11 March 2013 (UTC)
# [[User:dan-nl|dan-nl]] ([[User talk:dan-nl|talk]]) 20:14, 11 March 2013 (UTC)
# [[User:Purodha|Purodha Blissenbach]] ([[User talk:Purodha|talk]]) 20:19, 11 March 2013 (UTC)
# [[User:ESanders (WMF)|ESanders (WMF)]] ([[User talk:ESanders (WMF)|talk]]) 21:09, 11 March 2013 (UTC)
# [[User:Aude|Aude]] ([[User talk:Aude|talk]]) 19:46, 12 March 2013 (UTC)
# [[User:Judytuna|Judytuna]] ([[User talk:Judytuna|talk]]) 20:59, 12 March 2013 (UTC)
# [[User:Hoo man|Hoo man]] ([[User talk:Hoo man|talk]]) 22:40, 12 March 2013 (UTC)
# [[User:Lydia Pintscher (WMDE)|Lydia Pintscher (WMDE)]] ([[User talk:Lydia Pintscher (WMDE)|talk]]) 14:32, 13 March 2013 (UTC)
# [[User:Daniel Kinzler (WMDE)]]
# [[User:Jeroen De Dauw]]
# [[User:Kelson|Kelson]] ([[User talk:Kelson|talk]])
# [[User:AKlapper (WMF)]] 17:09, 14 March 2013 (UTC)
# [[User:cherishsantosh|Santosh Shingare]] ([[User talk:cherishsantosh|talk]]) 11:09, 17 March 2013 (UTC)
# — [[User:MPelletier (WMF)|MPelletier (WMF)]] ([[User talk:MPelletier (WMF)|talk]]) 19:54, 20 March 2013 (UTC)
# [[User:Guaka|Guaka]] ([[User talk:Guaka|talk]]) (tentative) 23:15, 20 March 2013 (UTC)
# Andy Mabbett, [[User:Pigsonthewing|Pigsonthewing]] (WikimediaUK)
# [[User:Deskana|Deskana]] ([[User talk:Deskana|talk]]) 14:27, 3 April 2013 (UTC)
# [[Wikipedia:nl:User:Spider|Spider]] ([[User talk:Ru.spider|talk]]) 15:28, 3 April 2013 (UTC)
# [[User:Reedy|Reedy]] ([[User talk:Reedy|talk]])
# [[User:Ladsgroup|Ladsgroup]] ([[User talk:Ladsgroup|talk]]) 09:08, 10 April 2013 (UTC)
# [[User:Natkabrown|Natkabrown]] ([[User talk:Natkabrown|talk]]) 10:53, 10 April 2013 (UTC)
# [[User:Schubi87|Schubi87]] ([[User talk:Schubi87|talk]]) 18:36, 10 April 2013 (UTC)
# [[User:Yug|Yug]] ([[User talk:Yug|talk]]) 12:11, 11 April 2013 (UTC)
# Edouard Lopez 12:11, 11 April 2013 (UTC)
# [[User:Edlira|Edlira]] ([[User talk:Edlira|talk]]) 22:30, 11 April 2013 (UTC)
# [[User:jpekel|jpekel]] ([[User talk:jpekel|talk]]) 12:30, 16 April 2013 (UTC)
# '''[[[User:Stwalkerster|stwalkerster]]|[[User talk:Stwalkerster|talk]]]''' 13:46, 16 April 2013 (UTC)
# -- [[User:DeltaQuad|DQ on the road]][[User_Talk:DeltaQuad| (ʞlɐʇ) ]] 23:30, 16 April 2013 (UTC)
# [[User:Luis Felipe Schenone|LFS]] ([[User talk:Luis Felipe Schenone|talk]]) 23:10, 18 April 2013 (UTC)
# [[User:Matma Rex|Matma Rex]] ([[User talk:Matma Rex|talk]]) 11:14, 19 April 2013 (UTC)
# [[User:HAndrade (WMF)|HAndrade (WMF)]] ([[User talk:HAndrade (WMF)|talk]]) 00:45, 20 April 2013 (UTC)
# [[User:Mglaser|Mglaser]] ([[User talk:Mglaser|talk]]) 13:24, 20 April 2013 (UTC)
# [[User:Jean-Frédéric|Jean-Fred]] ([[User talk:Jean-Frédéric|talk]]) 21:59, 20 April 2013 (UTC)
# [[User:Eclectiqus|Eclectiqus]] 10:46, 22 April 2013 (UTC)
# [[User:Addshore|Addshore]] ([[User talk:Addshore|talk]]) 11:45, 23 April 2013 (UTC)
# [[User:Parent5446|Parent5446]] ([[User talk:Parent5446|talk]]) 14:38, 23 April 2013 (UTC)
# [[User:Jens Ohlig|Jens Ohlig]] ([[User talk:Jens Ohlig|talk]]) 15:19, 25 April 2013 (UTC)
# [[user:nasir8891|Nasir Khan Saikat]] • [[user talk:Nasir8891|talk]] • 17:25, 25 April 2013 (UTC)
# [[User:Slevinski|Slevinski]] ([[User talk:Slevinski|talk]]) 15:33, 26 April 2013 (UTC)
# [[User:TMg|TMg]] 22:17, 28 April 2013 (UTC)
# [[User:Lokal_Profil|Lokal_Profil]] / [[User:André Costa (WMSE)|André Costa (WMSE)]] ([[User talk:André Costa (WMSE)|talk]]) 12:18, 2 May 2013 (UTC)
# [[User:Ziko|Ziko]] ([[User talk:Ziko|talk]]) 21:27, 8 May 2013 (UTC) Friday only
# [[User:CristianCantoro|CristianCantoro]] ([[User talk:CristianCantoro|talk]]) 10:45, 10 May 2013 (UTC) (on waiting list)
# [[User:Husky|Husky]] ([[User talk:Husky|talk]]) 11:32, 13 May 2013 (UTC)
# [[User:99of9|99of9]] ([[User talk:99of9|talk]]) 14:33, 13 May 2013 (UTC) (on waiting list for Saturday)
# [[User:Ocaasi|Ocaasi]] ([[User talk:Ocaasi|talk]]) 18:18, 13 May 2013 (UTC)
# [[User:Micru|Micru]] ([[User talk:Micru|talk]]) 14:04, 14 May 2013 (UTC)
# [[User:TWillemsen|TWillemsen]] ([[User talk:TWillemsen|talk]]) 19:05, 14 May 2013 (UTC)
# [[User:FatJagm|FatJagm]] ([[User talk:FatJagm|talk]]) 12:48, 15 May 2013 (UTC)
# [[User:Taweetham|Taweetham]] ([[User talk:Taweetham|talk]]) 00:05, 16 May 2013 (UTC) (Confirmed: arrival=Thursday ~8pm, departure=Monday ~1-2pm)
# Susanna Ånäs, [[User:Susannaanas|Susannaanas]] ([[User talk:Susannaanas|talk]]) 09:09, 16 May 2013 (UTC)
# [[User:OrenBochman|OrenBochman]] ''([[User Talk:OrenBochman|talk]])'' 20:03, 16 May 2013 (UTC)
# [[User:Lbenedix|Lbenedix]] ([[User talk:Lbenedix|talk]]) 11:15, 17 May 2013 (UTC)
# [[User:Erik Zachte|Erik Zachte]] ([[User talk:Erik Zachte|talk]]) 08:38, 19 May 2013 (UTC)
# [[User:1Veertje|1Veertje]] ([[User talk:1Veertje|talk]]) 18:33, 19 May 2013 (UTC)
# [[User:Nicole Ebber (WMDE)|Nicole Ebber (WMDE)]] ([[User talk:Nicole Ebber (WMDE)|talk]]) 17:03, 20 May 2013 (UTC)
# [[User:Dsc|dsc]] ([[User talk:Dsc|talk]]) 20:02, 20 May 2013 (UTC)

Analytics/Research

2013-04-29T19:13:57Z

Dsc:

[https://pinboard.in/u:wmfa Whitepapers, blog posts, etc.] of note and interest to us, the Analytics team

== Ideas ==

* Create an [https://github.com/twitter/ambrose Ambrose] action for Oozie. ([http://oozie.apache.org/docs/3.3.2/DG_CustomActionExecutor.html Docs for Custom Oozie Actions])
* Job Generator Scripts
** Job script & templates -- Instead of copy-pasting old jobs to create new jobs, we could script the use of templates, with parameters & actions for job-types and common tasks.
** Script to use Pig's <code>DESCRIBE</code> on the target of the final <code>STORE</code> command to generate a Hive <code>CREATE TABLE</code> statement.
* Aggregate and index task stats by generating code unit (bundle/coord//workflow/script), by spawning job, by user, etc
** sum counters, run stats, make graphs

Analytics/Kraken/Hadoop Tools

2013-04-29T19:07:03Z

Dsc:

This page is meant as a bucket for tips and notes on using the Hadoop toolchain.

= Pig =

[http://pig.apache.org/ Pig] is a dataflow language adept at analyzing unstructured data, converting it into a regular structure.

== Best Practices ==

* Push filters up even if you have to reprocess fields.
* Drop all unneeded fields as soon as you can.
* Syntax-check your script locally; use <code>DESCRIBE</code> to understand data shape.
* Don't explicitly set <code>PARALLEL</code> unless you have a reason. It only effects reduce operations, and Pig has heuristics based on data size for that; you are unlikely to help performance, and forcing <code>PARALLEL</code> higher than necessary uses up more slots on and increases job overhead.
* Don't Be Afraid To:
** ...Run stupid, half-broken jobs from a grunt shell just to see the data shape or test your UDF-tinkertoys. (Just pls save to your home dir.)
** ...<code>DUMP</code> after a complex statement to check output shape.
** ...Use Hive to check your data; its aggregation and filtering features are way more advanced and SQL is easy.

== Reuse & Metaprogramming ==

** Macros aren't really all that interesting as they're severely limited in scope.
** Be careful with <code>exec</code> and <code>run</code> -- they're more flexible and powerful, but very expensive, as every call spawns more MR jobs!
** Parameters are inserted via literal string substitution, which lets you do some pretty wacky metaprogramming via workflows. This means you basically have meta-macros. See [https://github.com/wikimedia/kraken/blob/master/pig/rollup.pig rollup.pig] as an example.

== Gotchas ==

* <code>MATCHES</code> RegExp must hit whole input
* Don't expect relations to stay sorted -- derived relations after a reduce-op (namely, GROUP) need re-ordering!
* Implicit coercion from bag of single-tuple to scalar ONLY works for relations! -- it does NOT work for grouped-records or other actual bags
* Escaping quotes in a parameter is impossible. I swear it. It's is worse than several layers of `ssh box -- bash -lc "eval '$WAT'"`. I gave up; no combination of backslashes and quotes made any difference.

= Oozie =

== Best Practices ==

* Test each layer, work outward; I always make a <code>$JOB-wf.properties</code> to test the workflow alone before moving on to the coord (with a <code>test-$JOB-coord.properties</code> and a <code>$JOB-coord.properties</code>).
* Everything can formally declare parameters using a <code><parameters></code> block at the beginning. DO IT! and avoid pointless defaults -- better to fail early.
* Check the <code>xmlns</code> on your root element!
** Coordinators: <code>xmlns="uri:oozie:coordinator:0.4"</code>
** Workflows: <code>xmlns="uri:oozie:workflow:0.4"</code>

== Workflows ==

* Know your <code><action></code> options: control flow, sub-workflow, fs, shell, java, streaming all have uses!
* Sub-workflows are like functions -- compose and reuse!
* <code><prepare></code> should probably be in all our jobs -- delete the output dir before starting work to ensure you don't pointlessly fail due to temporary cruft.
* <code><globals></code> allows you to set properties for all actions. All jobs should set job-tracker and namenode here.
* <code>job.xml</code>(s) -- they cascade -- will be useful once we start profiling and tuning jobs. Save those tweaks together as job-confs for similarly structured jobs to reuse!

== Coordinators ==

* <code><dataset></code> initial instances should always predate the job. This only restricts the possible valid results; it doesn't dictate anything about where the job starts.
* Always create coordinator parameters for <code>jobStart</code>, <code>jobEnd</code>, <code>jobName</code>, <code>jobQueue</code>! This lets you easily fire off the job as a backfill in a different queue, or one-of instances of the job, etc.
* <code>datasets.xml</code> lets you share <code><dataset></code> definitions. It's is worth investigating as the number of jobs grows.
* Chaining datasets between coordinators is fussy. I haven't seen it worth the energy so far.

== Gotchas ==

* Some workflow action elements are order-sensitive (!!). Ex: <code><configuration></code> must come before <code><script></code> in <code><pig-action></code>, and yes, the error message is oblique and unhelpful.

= Hadoop =

* All jobs keep performance counters and stats. These can be extremely helpful to improve job speed.
* Be familiar with the hdfs shell tool -- it's a lot more expressive than you might expect.

= Tutorials and Guides =

* [[Analytics/Kraken/coalesce_wf|The Coalesce Workflow]]: Concatenate and rename job results
* [[Analytics/Kraken/rollup_wf|The Rollup Workflow]]: Aggregate a result field into a rollup

Analytics/Kraken/Hadoop Tools

2013-04-29T19:04:24Z

Dsc: Created page with "This page is meant as a bucket for tips and notes on using the Hadoop toolchain. = Pig = [http://pig.apache.org/ Pig] is a dataflow language adept at analyzing unstructured..."

This page is meant as a bucket for tips and notes on using the Hadoop toolchain.

= Pig =

[http://pig.apache.org/ Pig] is a dataflow language adept at analyzing unstructured data, converting it into a regular structure.

== Best Practices ==

* Push filters up even if you have to reprocess fields.
* Drop all unneeded fields as soon as you can.
* Syntax-check your script locally; use <code>DESCRIBE</code> to understand data shape.
* Don't explicitly set <code>PARALLEL</code> unless you have a reason. It only effects reduce operations, and Pig has heuristics based on data size for that; you are unlikely to help performance, and forcing <code>PARALLEL</code> higher than necessary uses up more slots on and increases job overhead.
* Don't Be Afraid To:
** ...Run stupid, half-broken jobs from a grunt shell just to see the data shape or test your UDF-tinkertoys. (Just pls save to your home dir.)
** ...<code>DUMP</code> after a complex statement to check output shape.
** ...Use Hive to check your data; its aggregation and filtering features are way more advanced and SQL is easy.

== Reuse & Metaprogramming ==

** Macros aren't really all that interesting as they're severely limited in scope.
** Be careful with <code>exec</code> and <code>run</code> -- they're more flexible and powerful, but very expensive, as every call spawns more MR jobs!
** Parameters are inserted via literal string substitution, which lets you do some pretty wacky metaprogramming via workflows. This means you basically have meta-macros. See [http://github.com/wikimedia/kraken/oozie/util/rollup/rollup.pig rollup.pig] as an example.

== Gotchas ==

* <code>MATCHES</code> RegExp must hit whole input
* Don't expect relations to stay sorted -- derived relations after a reduce-op (namely, GROUP) need re-ordering!
* Implicit coercion from bag of single-tuple to scalar ONLY works for relations! -- it does NOT work for grouped-records or other actual bags
* Escaping quotes in a parameter is impossible. I swear it. It's is worse than several layers of `ssh box -- bash -lc "eval '$WAT'"`. I gave up; no combination of backslashes and quotes made any difference.

= Oozie =

== Best Practices ==

* Test each layer, work outward; I always make a <code>$JOB-wf.properties</code> to test the workflow alone before moving on to the coord (with a <code>test-$JOB-coord.properties</code> and a <code>$JOB-coord.properties</code>).
* Everything can formally declare parameters using a <code><parameters></code> block at the beginning. DO IT! and avoid pointless defaults -- better to fail early.
* Check the <code>xmlns</code> on your root element!
** Coordinators: <code>xmlns="uri:oozie:coordinator:0.4"</code>
** Workflows: <code>xmlns="uri:oozie:workflow:0.4"</code>

== Workflows ==

* Know your <code><action></code> options: control flow, sub-workflow, fs, shell, java, streaming all have uses!
* Sub-workflows are like functions -- compose and reuse!
* <code><prepare></code> should probably be in all our jobs -- delete the output dir before starting work to ensure you don't pointlessly fail due to temporary cruft.
* <code><globals></code> allows you to set properties for all actions. All jobs should set job-tracker and namenode here.
* <code>job.xml</code>(s) -- they cascade -- will be useful once we start profiling and tuning jobs. Save those tweaks together as job-confs for similarly structured jobs to reuse!

== Coordinators ==

* <code><dataset></code> initial instances should always predate the job. This only restricts the possible valid results; it doesn't dictate anything about where the job starts.
* Always create coordinator parameters for <code>jobStart</code>, <code>jobEnd</code>, <code>jobName</code>, <code>jobQueue</code>! This lets you easily fire off the job as a backfill in a different queue, or one-of instances of the job, etc.
* <code>datasets.xml</code> lets you share <code><dataset></code> definitions. It's is worth investigating as the number of jobs grows.
* Chaining datasets between coordinators is fussy. I haven't seen it worth the energy so far.

== Gotchas ==

* Some workflow action elements are order-sensitive (!!). Ex: <code><configuration></code> must come before <code><script></code> in <code><pig-action></code>, and yes, the error message is oblique and unhelpful.

= Hadoop =

* All jobs keep performance counters and stats. These can be extremely helpful to improve job speed.
* Be familiar with the hdfs shell tool -- it's a lot more expressive than you might expect.

= Tutorials and Guides =

* [[Analytics/Kraken/coalesce_wf|The Coalesce Workflow]]: Concatenate and rename job results
* [[Analytics/Kraken/rollup_wf|The Rollup Workflow]]: Aggregate a result field into a rollup

Analytics/Kraken

2013-04-29T19:03:33Z

Dsc:

{{Wikimedia engineering project information
| name = Analytics infrastructure
| description = A robust, distributed computing and data services platform.
| start = 2011-05-22
| end =
| group = Analytics
| lead =
| team = [[User:Ottomata|Andrew Otto]], [[User:Dsc|David Schoonover]], [[User:Diederik|Diederik van Liere]]
| previous =
| next =
| projectpage = Analytics/Infrastructure
| display = {{{display|}}}
}}<noinclude>

Kraken is the code-name for the robust distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

= [[{{FULLPAGENAME}}/status|Status]] =

{{Wikimedia project status line|{{FULLPAGENAME}}}}

= Rationale =

The Wiki Movement has a chronic need for analytics. We need it to understand our editors, to encourage growth, to engender diversity, to focus our resources, to improve our engineering efforts, and to measure our success. It permeates nearly all our goals, yet our current analytics capabilities are underdeveloped: we lack infrastructure to capture editor, visitor, clickstream, and device data in a way that is easily accessible; our efforts are distributed among different departments; our data is fragmented over different systems and databases; our tools are ad-hoc.

Rather than merely improve existing jobs and data pipelines, the Analytics Team aims to construct a Data Services Platform capable of mining intelligence from all datastreams of interest, providing this insight in real time, and exposing it via an API to power applications, mash up into websites, and stream to devices.

= Documentation =

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

* [[Analytics/Kraken/Access|Getting Access]]
* [[Analytics/Kraken/Overview|Overview]]
** [[Analytics/Kraken/Overview/Software|Software Overview]]
** [http://upload.wikimedia.org/wikipedia/mediawiki/1/1b/Kraken_flow_diagram-2.png Cluster Dataflow Diagram]
* '''Data'''
** [[Analytics/Product_Codes|Product Codes]]
** [[Analytics/Kraken/Data_Streams|Data Streams]]
** [[Analytics/Kraken/Data_Formats|Data Formats]]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
* [[Analytics/Kraken/Request_Logging|Request Logging]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* '''Notes'''
** [[Analytics/Kraken/Blurbs|Blurbs About Kraken]]
** [https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdEtRLVNoQWNvQzNleHBtTXR5emI3Z2c&pli=1#gid=5 Kraken Benchmarks]
** [http://etherpad.wikimedia.org/AnalyticsHadoopSetupNotes Hadoop Setup Notes]
** [http://etherpad.wikimedia.org/AnalyticsKrakenTestClusterSetup Test Cluster Setup Notes]
* '''[[Analytics/Kraken/Meetings|Meeting Notes]]'''
** [[Analytics/Kraken/Meetings/SecurityReview|Security Review Meeting]]
** [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

= Components =

== [[Analytics/Kraken/Pixel_Service|Pixel Service]] and [[Analytics/Kraken/Request_Logging|Request Logging]] ==

Public data import endpoint, and system for capturing the incoming firehose from our front-end servers.

* [[Analytics/Kraken/Request_Logging|Request Logging]]
** [[Analytics/Kraken/Request_Logging|System Recommendation]]
** [[Analytics/Kraken/Logging_Solutions_Overview|Distributed Logging Systems Research]]
** [https://docs.google.com/spreadsheet/pub?key=0AvpRkIqSY9hNdFNoYWFIX29hMnFuenV1ckRvYzEzUVE&gid=3 Feature Comparison Spreadsheet]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]

'''Tasks'''

* Work with Ori, Patrick, & ops to figure out Kafka producer situation for production [otto, dsc, ori, preilly, ops]
* MediaWiki EventLogging integration

== [[Analytics/Kraken/ETL|ETL Topology]] ==

The [http://storm-project.net Storm] topology that processes incoming data (aka, "Extract-Transform-Load" in data warehousing jargon).

* [[Analytics/Kraken/ETL|ETL Topology]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Maven + Nexus setup w/ storm pom [dsc]
* Storm Bolts:
** Consume from Kafka (by topic?)
** Serialize & canonicalize record using Avro schema
** GeoIP annotation
** Mobile carrier annotation (by IP)
** IP anonymization (where does the salt live? how often should we change it?)
** Append record to per-hour files in HDFS (path using timestamp, topic)
** Notifier Bolt to update Kafka consumer checkpoint (and/or publish event for external hooks)
* Define Avro schemas:
** WebRequest
** EventData
*** Figure out convention for EventData packets to self-identify with a more specific schema

== [[Analytics/Kraken/Core_Jobs|Core Jobs]] ==

Core data processing jobs for processing web requests and event data, maintained by the Analytics team.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Figure out request tagging -- how can we avoid mutating records?
* Reportcard Oozie workflow [DvL]
** Pig Macro for "typical request analytics"
* Figure out indexing scheme:
** Aggregated time buckets
** Referrer chains
** Event aggregations
** Funnel Analysis
*** A/B testing event data

== [[Analytics/Kraken/Data_Tools|Data Tools]] ==

Data processing library and toolkit provided & maintained by the Analytics team, esp for use with Kraken.

* [[Analytics/Infrastructure/Hadoop Tools|Tips and Notes on Hadoop Tools]]
* [[Analytics/Kraken/Oozie|Oozie Tutorial]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Setup regular sqoop imports [DvL]
* Avro De/Serialization storage format compatibility:
** WebRequest
** EventData
* Build re-usable pig/hive library, outsource this to analysts as much as possible [DvL]
* KV-Pairs parsing tools
* Conversion funnel analysis tools
** A/B testing analysis tools
* MW EventLogging extension integration
* Investigate Ori's Mediawiki extension for instrumentation, sending event/edit data

== [[Analytics/Kraken/Infrastructure|Infrastructure]] ==

Bucket for general cluster infrastructure and maintenance tasks.
* [[Analytics/Kraken/Puppet|Kraken Puppetization]]
* [[Analytics/Kraken/JMX Ports|JMX Ports]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* [[Analytics/Kraken/JMX Monitoring|JMX Monitoring Research]]

'''Meeting Notes'''

* [[Analytics/Kraken/Meetings/SecurityReview|Security Review Meeting]]
* [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

'''Tasks'''

* Fix the one busted Cisco machine (an07) [otto]
* Continue puppetization [otto]
* JMX Monitoring [otto, dsc]
* Investigate / experiment / benchmark Hadoop stack [DvL]
* Data owner services -- export, dashboard, visualizations (on Hue?) [dsc, dan]
** Hue plugin for Limn integration? [dsc]
* Get analysts experimenting with Hive/Pig [DvL]
* General purpose job-runner using Kafka+Storm
* Setup Hadoop FairScheduler [DvL]

= Research =

== Bundling ==

Investigate size of savings of structure sharing via a short-term (between 1-30sec window) queue.

* Reorder within window by timestamp? Seq number?
* Parameters of experiment:
** Buffer time: 1s, 2s, 5s, 10s
** Sharded by server
** Bundle by IP, UA, IP+UA
* Pull-up fields for structure sharing:
** Invariant: ip, ua, accept-lang, carrier
** Often invariant: method, status, referer

== Software ==
* Hbase
* Hive
* Hue plugins
* Cascading -- do we even need this as we have Oozie?

[[Category:WMF Projects]]
[[Category:WMF Projects/Analytics]]

</noinclude>

Analytics/Requirements Checklist

2013-04-04T19:29:02Z

Dsc: Created page with "= User Story & Expectations = * What: Captures the narrative as presented by the user (with little-to-no interpretation or cleanup) * Goal: Understand the user's needs, desir..."

= User Story & Expectations =

* What: Captures the narrative as presented by the user (with little-to-no interpretation or cleanup)
* Goal: Understand the user's needs, desires, and expectations.
** The raw narrative is useful even if it doesn't really make sense! -- It helps to know where we started in the process to grok their expectations.
** Honestly great to have a literal description of the deliverable in their own words
* Key Questions:
** What is their goal? Why do they want to achieve that? Gains/Pains. What are possible scale-backs?
** Who surfaced this desire (actual person) and in what context? Who do we ultimately deliver to?
** How clear was the user's vision of the feature, the deliverable and the reqs?
** What sort of expectations did they have *before* coming us about the LoE of their request and/or the timeline?
** Are there any external deadlines? Does this block anything?

= User Feature Requirements & Acceptance Criterion =

* What: Cleaned and clarified description of the deliverable, broken out into features that make sense to everyone.
** Ideally, we want both the high-level, easy-to-grok description, and the low-level, precise breakout.
** eg. "Web dashboard w/ 2 tabs -- "Mobile Site", and "Mobile Apps". A contains 6 graphs: 1. Pageviews to Mobile Site (line, logscale, step=1mo), with metrics (all pageviews / month): total across all sites, alpha site, beta site. 2. ..."
* Goal: Consensus around the concrete deliverable and especially its scope, including:
** Clear and precise enumeration capabilities/limitations
** Expectations around behavior and appearance
** Rough expectations around a timeline, and some boxing on level-of-effort/the time commitment
* Components:
** Data Requirements, specified as precisely as possible.
*** We don't necessarily need to make the user sit through us hashing this out, but (imo) we do need to get them to explicitly sign off on the result.
*** I think we should consider it a hard requirement that any feature which requires new metric work have descriptions for:
**** Definitions for all metric dimensions (with datatypes) -- this means "counts" should explicitly say what dimensions are grouped and counted
**** Data output shape -- define rows/columns with types (including the step between rows); what defines a "key", and is it row-major or col-major?; cardinality/storage estimate; drill-downs/aggregations/rollups
**** Data source(s) -- both the data necessary for the computation and the origin of said data. For example, if the expectation is that we use a stream into Kraken, it is a different feature to fall back on the sampled Squid logs, as the whole implementation toolchain is different.
**** Data output format(s) -- can't just assume tsv if we aim to deliver certain visualizations (esp geo) -- is json, xml, or avro ok? how will the data be published? does it need to be public, or can we cron files into a shared space on stat1001?
**** Minimally required frequency of recomputation -- note this is different from the timestep for rows and/or rollups; this is how stale the data can be (aka, how often the job runs)
** Presentation/Visualization Requirements, including any graphs, dashboards, domains, or other presentation features (pointing to existing examples if possible).
*** Dashboards: names, contents for each, their tabs, and the visualizations in each tab.
*** Questions for each visualization: graph type (line, bar, geo, etc); any specific formatting/appearance requests (custom labels? scale? timespan?)
*** Are any new Limn features needed?

= Solution Options w/ Tasks & LoE Estimate =

* What: List several technical solutions as options for satisfying the requirements, assessing the tasks required and the level-of-effort involved.
* Goal: Find the best way to make the user happy while conserving dev effort and advancing along the trajectory of our strategy to provide insight via a robust, automated, self-service platform.
* If necessary, Engineering Lead coordinates team expertise and presents results.
* Everyone works together to assess, revise, and prioritize (planning poker, etc).

Analytics

2013-04-04T19:25:55Z

Dsc:

[[File:WMF_Analytics_-_Ring_Logo_(thick).svg|200px|right|wmf analytics: logos mean srs bsns]] Howdy from the Wikimedia Analytics team! We're currently working on building a Data Services Platform (codenamed Kraken) to power the next-generation of intelligence and analytics, as well as a new and wondrous [http://reportcard.wmflabs.org/ Reportcard].

If you're enthusiastic about analytics, you might want to [http://blog.wikimedia.org/2012/07/25/meet-the-analytics-team/ meet the team], or subscribe to the [https://lists.wikimedia.org/mailman/listinfo/analytics Analytics Mailing List] (don't worry, it's low-traffic).

== Team ==
* [[User:DAndreescu|Dan Andreescu]]
* [[User:Ottomata|Andrew Otto]]
* [[User:Dsc|Dave Schoonover]]
* [[User:Diederik|Diederik van Liere]]
* [[User:Erik_Zachte|Erik Zachte]]
* [[User:Kparkinson|Kraig Parkinson]]
* Stefan Petrea

== Planning ==
* [[Analytics/Roadmap|Roadmap]]
** [[Roadmap#Analytics|Analytics on the Platform Roadmap]]
* [[Analytics/Roadmap/PlanningMeetings|Meeting Notes]]
* [[Analytics/Roadmap/2012-2013|2012-2013 Planning]]

== Stakeholder Corner ==
* [[Analytics/Product_Codes|Product Codes]] — required application identifier needed to get data into the cluster.
* [[Analytics/Metric_definitions|Metrics Definitions]] — canonical definition of metrics.
* [[Analytics/Dreams|Dreams]] — features, metrics/queries, and visualizations people would like to see someday. Importantly, don't worry about these things being reasonable, well-scoped, or put in the right place. Just add stuff here you think would be useful.

== Projects ==

=== [[Analytics/Kraken|Kraken]] ===
* "Kraken" is the codename for the cluster and software which powers the Data Services Platform.
* [[Analytics/Kraken|Project Info]]
* [[Analytics/Product_Codes|Product Codes]]
* [https://upload.wikimedia.org/wikipedia/mediawiki/3/38/Kraken_flow_diagram.png Cluster Dataflow Diagram]
** [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
** [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning docs]]
** [[Analytics/Distributed_Logging_Solutions|Distributed Logging]]
** [[Analytics/Kraken/Infrastructure|Kraken Server Hardware & Infrastructure]]
* Latest status: {{Wikimedia project status line|Analytics/Kraken}}

=== [[Analytics/Limn|Limn]] ===
* Limn is a drop-in GUI toolkit for building visualizations. It powers the [http://reportcard.wmflabs.org/ WMF Monthly Reportcard].
* [[Analytics/Limn|Project Info]]
* Source: https://github.com/wikimedia/limn
* Issues: https://github.com/wikimedia/limn/issues
* Latest status: {{Wikimedia project status line|Analytics/Limn}}

=== [[Analytics/Reportcard|Reportcard]] ===
* The [http://reportcard.wmflabs.org/ WMF Monthly Reportcard] is a visual dashboard of high-level metrics on the health and success of the various Wikimedia projects.
* [[Analytics/Reportcard|Project Info]]
* Site: http://reportcard.wmflabs.org/
* Latest status: {{Wikimedia project status line|Analytics/Reportcard}}

=== [[Analytics/Wikistats|Wikistats]] ===
* [[Analytics/Wikistats|Project Info]]
* Latest status: {{Wikimedia project status line|Wikistats}}

=== [[Analytics/Logging infrastructure|Logging Infrastructure]] ===
* [[Analytics/Logging infrastructure|Project Info]]
* Components:
** [[Analytics/Logging infrastructure|UDP Logging]]
** [[Analytics/UDP-filters|UDP Log Filters]]
* Resources:
** [[Analytics/Infrastructure/Stat1|<code>stat1</code> server]]
* Latest status: {{Wikimedia project status line|Analytics/Logging infrastructure}}

=== [[Analytics/Data_Releases|Data Releases]] ===
* [[Analytics/Data_Releases|Project Info]]
* [[Analytics/Pageviews|Page View Analytics]]
** [[Analytics/Pageviews/Mobile|Mobile]]
** [[Analytics/Pageviews/GLAM|GLAM]]
* Latest status: {{Wikimedia project status line|Analytics/Data Releases}}

=== See Also ===
* [[Developers/Maintainers#Analytics|Project Maintainers]]

== [[Analytics/Research|Research]] & Notes ==
* [[Analytics/Research|Research Hub]]
* [http://pinboard.in/u:wmfa Team Bookmarks on Pinboard]
* [[Analytics/Requirements_Checklist|Requirements Checklist]]
* Across WMF:
** [[OAuth]]

[[Category:WMF Projects]]

== [[Analytics/Management|Management]] ==

* Using Mingle to track improvements: https://mingle.corp.wikimedia.org/projects/analytics/wiki/Engaging_in_Continuous_Improvement

Meetings/2013-03-21

2013-03-20T22:43:25Z

Dsc:

{{nav
| title = Wikimedia Open Tech Chats
| previous = '''[[Meetings/2013-02-28|28 February]]'''
| next =
}}

{| class="wikitable" style="font-size: 120%;"
! style="width: 15%; background-color:#CC99CC" | What
| HipHop/HHVM
|-
! style="background-color:#D8BFD8" | How
| We're using a combination of Google Hangout on Air (an open YouTube stream from the 6th floor of the WMF office) + potential additional Hangout guest speakers + IRC for text-based communication. There's no attendance limit. You don't need Hangout to participate -- if you can play YouTube videos and join IRC, you're fine.
|-
! style="background-color:#CC99CC" | When
| Thursday, March 21, 2013, 19:00 [http://timeanddate.com/worldclock/fixedtime.html?msg=WMF+Open+Tech+Chat&iso=20130117T12&p1=224&ah=1 UTC] (12:00 PDT), 60 minutes
|-
! style="background-color:#D8BFD8" | Where
|
* Physical space: 6th floor SF office, "Collab" space
* IRC: {{irc|wikimedia-dev}}
* YouTube stream: https://www.youtube.com/watch?feature=player_embedded&v=isq-jid4ujQ
|- style="vertical-align: top;"
! style="background-color:#CC99CC" |Food
| Please [https://office.wikimedia.org/wiki/Lunch_2013-03-21 sign up for lunch] if you're attending in San Francisco
|-
|}

== Topics ==
Drew Paroski from Facebook will be joining us for this week's open tech chat on HipHop/HHVM

== Attendees ==

Add your name, tell your friends!

* [[User:Asher Feldman|Asher Feldman]] ([[User talk:Asher Feldman|talk]]) 22:15, 18 March 2013 (UTC)
* [[User:MaxSem|Max Semenik]] ([[User talk:MaxSem|talk]]) 22:27, 18 March 2013 (UTC)
* [[User:Brion VIBBER|brion]] ([[User talk:Brion VIBBER|talk]])
* [[User:RobLa-WMF|RobLa-WMF]] ([[User talk:RobLa-WMF|talk]])
* [[User:Ssastry|Ssastry]] ([[User talk:Ssastry|talk]]) 23:39, 18 March 2013 (UTC)
* [[User:Renklauf|Ryan Faulkner]] 2013-03-19 02:06:18 UTC
* [[User:ABaso(WMF)|ABaso(WMF)]] ([[User talk:ABaso(WMF)|talk]]) 22:51, 19 March 2013 (UTC)
* [[User:GWicke|Gabriel Wicke (GWicke)]] ([[User talk:GWicke|talk]]) 17:23, 20 March 2013 (UTC)
* [[User:Inez|Inez Korczyński]] ([[User talk:Inez|talk]]) 18:05, 20 March 2013 (UTC)
* [[User:Bsitu|Bsitu]] ([[User talk:Bsitu|talk]]) 19:15, 20 March 2013 (UTC)
* [[User:Dsc|dsc]] ([[User talk:Dsc|§]]) {{CURRENTTIME}}, {{CURRENTDAY}} {{CURRENTMONTHNAME}} {{CURRENTYEAR}} (UTC)

Analytics/Kraken

2013-03-20T19:05:55Z

Dsc:

{{Wikimedia engineering project information
| name = Kraken (Analytics Cluster)
| description = A robust, distributed computing and data services platform.
| start = 2011-05-22
| end =
| group = Analytics
| lead =
| team = [[User:Ottomata|Andrew Otto]], [[User:Dsc|David Schoonover]], [[User:Diederik|Diederik van Liere]]
| previous =
| next =
| projectpage = Analytics/Kraken
| display = {{{display|}}}
}}<noinclude>

Kraken is the code-name for the robust distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

= [[{{FULLPAGENAME}}/status|Status]] =

{{Wikimedia project status line|{{FULLPAGENAME}}}}

= Rationale =

The Wiki Movement has a chronic need for analytics. We need it to understand our editors, to encourage growth, to engender diversity, to focus our resources, to improve our engineering efforts, and to measure our success. It permeates nearly all our goals, yet our current analytics capabilities are underdeveloped: we lack infrastructure to capture editor, visitor, clickstream, and device data in a way that is easily accessible; our efforts are distributed among different departments; our data is fragmented over different systems and databases; our tools are ad-hoc.

Rather than merely improve existing jobs and data pipelines, the Analytics Team aims to construct a Data Services Platform capable of mining intelligence from all datastreams of interest, providing this insight in real time, and exposing it via an API to power applications, mash up into websites, and stream to devices.

= Documentation =

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

* [[Analytics/Kraken/Access|Getting Access]]
* [[Analytics/Kraken/Overview|Overview]]
** [[Analytics/Kraken/Overview/Software|Software Overview]]
** [http://upload.wikimedia.org/wikipedia/mediawiki/1/1b/Kraken_flow_diagram-2.png Cluster Dataflow Diagram]
* '''Data'''
** [[Analytics/Product_Codes|Product Codes]]
** [[Analytics/Kraken/Data_Streams|Data Streams]]
** [[Analytics/Kraken/Data_Formats|Data Formats]]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
* [[Analytics/Kraken/Request_Logging|Request Logging]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* '''Notes'''
** [[Analytics/Kraken/Blurbs|Blurbs About Kraken]]
** [https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdEtRLVNoQWNvQzNleHBtTXR5emI3Z2c&pli=1#gid=5 Kraken Benchmarks]
** [http://etherpad.wikimedia.org/AnalyticsHadoopSetupNotes Hadoop Setup Notes]
** [http://etherpad.wikimedia.org/AnalyticsKrakenTestClusterSetup Test Cluster Setup Notes]
* '''[[Analytics/Kraken/Meetings|Meeting Notes]]'''
** [[Analytics/Kraken/Meetings/SecurityReview|Security Review Meeting]]
** [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

= Components =

== [[Analytics/Kraken/Pixel_Service|Pixel Service]] and [[Analytics/Kraken/Request_Logging|Request Logging]] ==

Public data import endpoint, and system for capturing the incoming firehose from our front-end servers.

* [[Analytics/Kraken/Request_Logging|Request Logging]]
** [[Analytics/Kraken/Request_Logging|System Recommendation]]
** [[Analytics/Kraken/Logging_Solutions_Overview|Distributed Logging Systems Research]]
** [https://docs.google.com/spreadsheet/pub?key=0AvpRkIqSY9hNdFNoYWFIX29hMnFuenV1ckRvYzEzUVE&gid=3 Feature Comparison Spreadsheet]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]

'''Tasks'''

* Work with Ori, Patrick, & ops to figure out Kafka producer situation for production [otto, dsc, ori, preilly, ops]
* MediaWiki EventLogging integration

== [[Analytics/Kraken/ETL|ETL Topology]] ==

The [http://storm-project.net Storm] topology that processes incoming data (aka, "Extract-Transform-Load" in data warehousing jargon).

* [[Analytics/Kraken/ETL|ETL Topology]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Maven + Nexus setup w/ storm pom [dsc]
* Storm Bolts:
** Consume from Kafka (by topic?)
** Serialize & canonicalize record using Avro schema
** GeoIP annotation
** Mobile carrier annotation (by IP)
** IP anonymization (where does the salt live? how often should we change it?)
** Append record to per-hour files in HDFS (path using timestamp, topic)
** Notifier Bolt to update Kafka consumer checkpoint (and/or publish event for external hooks)
* Define Avro schemas:
** WebRequest
** EventData
*** Figure out convention for EventData packets to self-identify with a more specific schema

== [[Analytics/Kraken/Core_Jobs|Core Jobs]] ==

Core data processing jobs for processing web requests and event data, maintained by the Analytics team.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Figure out request tagging -- how can we avoid mutating records?
* Reportcard Oozie workflow [DvL]
** Pig Macro for "typical request analytics"
* Figure out indexing scheme:
** Aggregated time buckets
** Referrer chains
** Event aggregations
** Funnel Analysis
*** A/B testing event data

== [[Analytics/Kraken/Data_Tools|Data Tools]] ==

Data processing library and toolkit provided & maintained by the Analytics team, esp for use with Kraken.

* [[Analytics/Kraken/Oozie|Oozie Tutorial]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Setup regular sqoop imports [DvL]
* Avro De/Serialization storage format compatibility:
** WebRequest
** EventData
* Build re-usable pig/hive library, outsource this to analysts as much as possible [DvL]
* KV-Pairs parsing tools
* Conversion funnel analysis tools
** A/B testing analysis tools
* MW EventLogging extension integration
* Investigate Ori's Mediawiki extension for instrumentation, sending event/edit data

== [[Analytics/Kraken/Infrastructure|Infrastructure]] ==

Bucket for general cluster infrastructure and maintenance tasks.
* [[Analytics/Kraken/Puppet|Kraken Puppetization]]
* [[Analytics/Kraken/JMX Ports|JMX Ports]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* [[Analytics/Kraken/JMX Monitoring|JMX Monitoring Research]]

'''Meeting Notes'''

* [[Analytics/Kraken/Meetings/SecurityReview|Security Review Meeting]]
* [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

'''Tasks'''

* Fix the one busted Cisco machine (an07) [otto]
* Continue puppetization [otto]
* JMX Monitoring [otto, dsc]
* Investigate / experiment / benchmark Hadoop stack [DvL]
* Data owner services -- export, dashboard, visualizations (on Hue?) [dsc, dan]
** Hue plugin for Limn integration? [dsc]
* Get analysts experimenting with Hive/Pig [DvL]
* General purpose job-runner using Kafka+Storm
* Setup Hadoop FairScheduler [DvL]

= Research =

== Bundling ==

Investigate size of savings of structure sharing via a short-term (between 1-30sec window) queue.

* Reorder within window by timestamp? Seq number?
* Parameters of experiment:
** Buffer time: 1s, 2s, 5s, 10s
** Sharded by server
** Bundle by IP, UA, IP+UA
* Pull-up fields for structure sharing:
** Invariant: ip, ua, accept-lang, carrier
** Often invariant: method, status, referer

== Software ==
* Hbase
* Hive
* Hue plugins
* Cascading -- do we even need this as we have Oozie?

[[Category:WMF Projects]]
[[Category:WMF Projects/Analytics]]

</noinclude>

Analytics/Archive/Infrastructure/Oozie

2013-03-19T15:46:19Z

Dsc: fixes the awful analytics1010.wikimedia.org typo so that none-else must suffer for the greater good

Apache [http://oozie.apache.org/ Oozie] is a workflow scheduler system to manage Apache Hadoop jobs.

Oozie is a job scheduler with fancy features. Most relevantly, jobs may be scheduled based on the existence of data in HDFS. This allows jobs to be scheduled to be run not based only on a current timestamp, but for when the data needed to run a particular job is available.

= Terms =
;action
:An action generally represents a single step in a job workflow. Examples include pig scripts, failure notifications, map-reduce jobs, etc.

;workflow
:Workflows are used to chain actions together. A workflow is synonymous with a job. Workflows describe how actions should run, and how actions should flow. Actions can be chained based on success and failure conditions.

;coordinator
:Coordinators are used to schedule recurring runs of workflows. They can abstractly describe input and output datasets based on on periodicity. Coordinators will submit workflow jobs based on the existence of data.

= An Example =
For our simple example, we'll create a workflow that runs a Pig script, takes any number of files/directories as input, and saves its output to a file. A second pig script that will take the output of the first as its input, and concatenate it all together in a single file. We'll then learn to use a coordinator to hourly schedule this workflow based on the existence of input data.

== Actions ==
Our actions are just two Pig scripts. It isn't important what they do, except that they take input and output files as CLI parameters. Our example Pig scripts also take in an a regular expression, '$hour_regex' as a CLI parameter. This allows us to filter out content that doesn't match the hour for which we want to generate statistics.

This example was written while referring to [https://github.com/wikimedia/kraken/blob/master/pig/webrequest_loss_hourly.pig webrequest_loss_hourly.pig] and [https://github.com/wikimedia/kraken/blob/master/pig/concat_sort.pig concat_sort.pig].

== Workflow ==
We can define and parameterize a workflow before we start to schedule it for regular runs using a coordinator.

A workflow is just a series of parameterized actions. Parameters can be set on the Oozie CLI, in a .properties file, or by the coordinator XML file.

In workflow.xml, first we name the workflow:

<source lang="xml">
<workflow-app name="webrequest_loss_by_hour_workflow" xmlns="uri:oozie:workflow:0.4">
</source>

Then we tell the workflow which action should be run first:

<source lang="xml">
<start to="webrequest_loss_by_hour_action"/>
</source>

Now we define our actions and their flow:

<source lang="xml">
<action name="webrequest_loss_by_hour_action">
<pig>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<script>/user/dummy/pig/webrequest_loss_hourly.pig</script>
<argument>-p</argument>
<argument>input=${INPUT}</argument>
<argument>-p</argument>
<argument>output=${OUTPUT}</argument>
<argument>-p</argument>
<argument>hour_regex=${HOUR_REGEX}</argument>
</pig>
<ok to="concat"/>
<error to="kill"/>
</action>
<action name="concat">
<pig>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<script>/libs/kraken/pig/concat_sort.pig</script>
<argument>-p</argument>
<argument>input=${CONCAT_INPUT}</argument>
<argument>-p</argument>
<argument>dest=${DEST}</argument>
</pig>
<ok to="end"/>
<error to="kill"/>
</action>
</source>

Note that the if the "webrequest_loss_by_hour_action" finishes with an "ok" status, the "concat" action will be run.

And finally the end failure and success end conditions:

<source lang="xml">
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
</source>

The full file can be found [https://github.com/wikimedia/kraken/blob/master/oozie/webrequest_loss_by_hour/workflow.xml here].

This workflow can be manually submitted via the Oozie CLI. You'll need to create a job.properties file in which you set the parameterized values, as well as some properties that Oozie requires you to have to run.

<source lang="bash">
oozie.wf.application.path=${nameNode}/user/dummy/oozie/webrequest_loss_by_hour
jobTracker=analytics1010.eqiad.wmnet:8032
nameNode=hdfs://analytics1010.eqiad.wmnet:8020
oozie.use.system.libpath=true
oozie.libpath=${nameNode}/user/oozie/share/lib/,${nameNode}/libs/,${nameNode}/user/dummy/pig
INPUT=hdfs://analytics1010.eqiad.wmnet:8020/path/to/input/2013-02-25_09.45,hdfs://analytics1010.eqiad.wmnet:8020/path/to/input/2013-02-25_10.00,hdfs://analytics1010.eqiad.wmnet:8020/path/to/input/2013-02-25_10.15,hdfs://analytics1010.eqiad.wmnet:8020/path/to/input/2013-02-25_10.30,hdfs://analytics1010.eqiad.wmnet:8020/path/to/input/2013-02-25_10.45,hdfs://analytics1010.eqiad.wmnet:8020/path/to/input/2013-02-25_11.00
OUTPUT=hdfs://analytics1010.eqiad.wmnet:8020/path/to/output/2013-02-25_10.00.00
HOUR_REGEX=2013-02-25_10
CONCAT_INPUT=hdfs://analytics1010.eqiad.wmnet:8020/path/to/output/2013-02-25_10.00.00
</source>

The oozie.wf.application.path is important. It tells Oozie where to find your workflow.xml definition. You'll need to copy your workflow.xml file to this directory, and then use the above job.properties file to submit the workflow:

Note also that oozie.libpath includes /user/dummy/pig. If your Pig script imports any external Pig scripts, make sure that those external scripts are in HDFS and in the oozie.libpath.

In this example job.propeties file, we are generating statistics for the 10:00 hour on February 25th. Since our data is imported every 15 minutes, for each hour we will want to take 6 individual imports as our INPUT data. For the 10:00 hour, that will be 09:45, 10:00, 10:15, 10:30, 10:45, and 11:00.

<source lang="bash">
# upload the workflow.xml file to your oozie.wf.application.path
hadoop fs -put workflow.xml /user/dummy/oozie/webrequest_loss_by_hour/
# submit the job.properties file to oozie and start it (-run == -submit && -start)
oozie job -oozie http://analytics1027.eqiad.wmnet:11000/oozie -run -config ./job.properties
</source>
You can use the "-info" and other oozie commands to get information about the running job.

== Coordinator ==
A coordinator is submitted to Oozie in the same way as a workflow, but defined differently.

First we name our coordinator and specify how often we want it to run:

<source lang="xml">
<coordinator-app name="webrequest_loss_by_hour"
frequency="${coord:hours(1)}"
start="2013-02-01T01:00Z" end="2020-01-01T23:00Z" timezone="Universal"
xmlns="uri:oozie:coordinator:0.1">
</source>

=== Defining coordinator datasets and input events ===
Our example input data set is imported every 15 minutes. This data is saved in directories that are named based on timestamp, but the data is not bucketed in these directories by the timestamps in the data content. For example, we cannot guarantee that the data stored in a directory named 2013-02-25_10.15.00 contains only data for the 10:00 - 10:15 range. It most likely will contain some few logs from minutes before or after.

==== Datasets ====
The Oozie coordinator needs to know the path and import frequency of our input dataset, as well as when our dataset begins. Note that the initial-instance attribute is not in the same format as the input directory.

<source lang="xml">
<datasets>
<dataset name="webrequest-wikipedia-mobile" frequency="${coord:minutes(15)}"
initial-instance="2013-02-01T00:00Z" timezone="Universal">
<uri-template>hdfs://analytics1010.eqiad.wmnet:8020/path/to/input/${YEAR}-${MONTH}-${DAY}_${HOUR}.${MINUTE}.00</uri-template>
<done-flag></done-flag>
</dataset>
</source>

We also need to define our output dataset.

<source lang="xml">
<dataset name="webrequest_loss" frequency="${coord:minutes(15)}"
initial-instance="2013-02-01T00:00Z" timezone="Universal">
<uri-template>hdfs://analytics1010.eqiad.wmnet:8020/path/to/output/${YEAR}-${MONTH}-${DAY}_${HOUR}.${MINUTE}.00</uri-template>
<done-flag></done-flag>
</dataset>
</datasets>
</source>

''Note: I'm not sure that a frequency of 15 minutes is correct here. We will be running this coordinator every hour, so defining this dataset with a frequency of 15 minutes is probably incorrect.''

==== Input/Output Events ====
Now that we've described how the data on disk looks, we can now define input and output events based on ranges of this data.

The following defines the input event. We tell it that this input event is made up of the webrequest-wikipedia-mobile dataset we defined above. We also then give a range of data that belongs to to this input event. current(0) will resolve to the current hour at which we are running. For example, if the current hour is 11:00, we will want to look at the previous 6 data imports to generate statistics for the 10:00 hour. So the start-instance is current - 5 (11:00 - 5 * 15 minute intervals == 09:45), and the end-instance is the current hour.

<source lang="xml">
<input-events>
<data-in name="INPUT" dataset="webrequest-wikipedia-mobile">
<start-instance>${coord:current(-5)}</start-instance>
<end-instance>${coord:current(0)}</end-instance>
</data-in>
</input-events>
</source>

We also define an output event so that the coordinator knows what each workflow run should output. If the coordinator is scheduling a run at 11:00, then current(-4) will be 10:00, which will eventually be in the name of the output directory.

<source lang="xml">
<output-events>
<data-out name="OUTPUT" dataset="webrequest_loss">
<instance>${coord:current(-4)}</instance>
</data-out>
</output-events>
</source>

Finally, we need to tell the coordinator the workflow that it should submit for each run. By setting app-path to ${wf_application_path}, we tell Oozie to look for workflow.xml in that directory (which we specify in a .properties file later).

<source lang="xml">
<action>
<workflow>
<app-path>${wf_application_path}</app-path>
</source>

Now for the workflow configuration. The workflow works with variable parameters based on which time period is being computed, so we need to set those based on named input and output events. We also set HOUR_REGEX to the hour for which each run generates statistics to the current hour minus one our (11:00 - 1 == 10:00). (Remember that this is a peculiarity/feature of our Pig script.) CONCAT_INPUT is defined to be the whole of all output data, and DEST a single .tsv file that will contain all of the output data in a single file.

<source lang="xml">
<configuration>
<property>
<name>INPUT</name>
<value>${coord:dataIn('INPUT')}</value>
</property>
<property>
<name>OUTPUT</name>
<value>${coord:dataOut('OUTPUT')}</value>
</property>
<property>
<name>HOUR_REGEX</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -1, 'HOUR'), 'yyyy-MM-dd_HH')}</value>
</property>
<property>
<name>CONCAT_INPUT</name>
<value>hdfs://analytics1010.eqiad.wmnet:8020/path/to/output</value>
</property>
<property>
<name>DEST</name>
<value>hdfs://analytics1010.eqiad.wmnet:8020/path/to/webrequest_loss_by_hour.tsv</value>
</property>
</configuration>
</workflow>
</action>
</coordinator-app>
</source>

Now that we've got our coordinator.xml file, we can copy it (and workflow.xml) to HDFS and create a coordinator.properties file to use for submission to Oozie.

<source lang="bash">
oozie.coord.application.path=${nameNode}/user/dummy/oozie/webrequest_loss_by_hour
wf_application_path=${oozie.coord.application.path}
jobTracker=analytics1010.eqiad.wmnet:8032
nameNode=hdfs://analytics1010.eqiad.wmnet:8020
oozie.use.system.libpath=true
oozie.libpath=${nameNode}/user/oozie/share/lib/,${nameNode}/libs/,${nameNode}/user/dummy/pig
</source>

<source lang="bash">
# copy the .xml files to HDFS
hadoop fs -put coordinator.xml workflow.xml /user/dummy/oozie/webrequest_loss_by_hour/
# Submit the coordinator. Oozie will submit and run worfklows based on input events.
oozie job -oozie http://analytics1027.eqiad.wmnet:11000/oozie -submit -config ./coordinator.properties
</source>

Full versions of these .property and .xml files can be found at https://github.com/wikimedia/kraken/tree/master/oozie/webrequest_loss_by_hour.

Analytics/Archive/Infrastructure/Meetings/ArchitectureReview

2013-03-04T18:22:37Z

Dsc:

* Date: March 1, 2013 (2:30p - 3:30p)
* Present: Mark, Faidon, Tim, Ryan, Ori, Ryan, Robla, Kraig, Dan, David, Andrew, Diederik
* Slides: [https://upload.wikimedia.org/wikipedia/mediawiki/c/cb/Kraken_Arch_Review_Slides.pdf (pdf)]

== Background ==

* Project page: https://www.mediawiki.org/wiki/Analytics/Kraken
* Documentation:
** Overview: https://www.mediawiki.org/wiki/Analytics/Kraken/Overview
** Software Overview: https://www.mediawiki.org/wiki/Analytics/Kraken/Overview/Software
** Services and Ports: https://www.mediawiki.org/wiki/Analytics/Kraken/Ports
** Infrastructure / Hardware: https://www.mediawiki.org/wiki/Analytics/Kraken/Infrastructure
* Related:
** ACL Ticket: https://rt.wikimedia.org/Ticket/Display.html?id=4433
** Security Meeting Notes: https://www.mediawiki.org/wiki/Analytics/Kraken/Meetings/SecurityReview

= Meeting Notes =

== Minimum Viable Cluster ==

Our first goal is get to a "minimum viable" cluster setup.
To meet our stakeholder obligations, we'll need:

* Unsampled storage of Mobile pageview requests as txt files
* Unsampled storage of EventLogging data as JSON or txt files
* Importing MediaWiki databases
* Ability to access raw datafiles by WMF employees
* Ability to query data using Pig, Hive, MapReduce, and Hadoop streaming
* Ability to schedule jobs and workflows (Oozie)
* Automated recovery of data importers (Zookeeper)

Not part of MVC – deferring for later discussion:
* Realtime processing layer (Storm)
* Guaranteed delivery of web-request data (Kafka producers and future of udp2log)

== Dataflow Outline ==

* Mark: Asher says mediawiki import cannot be part of MVC -- he said it can be done, but I don't have all the details. Maybe something about a "binlog API"?
* Faidon: This doesn't seem like a very efficient use of hardware...
** Yeah, totally not. It's an evolutionary deployment, but this is a description of how things stand today. The end goal (and scope) looks very different. This was a way to simulate a production setup (including Kafka on the edge) without making production changes. A "production setup" is not a near-term goal any more (as we've scoped out ETL for now), so this looks a bit weird.
* Faidon: Rolling resetup/reimage means some number of machines should be detached from the current cluster for the process of reviewing and rebuilding off the mainline puppet
** Kraig: Yeah, but unfortunately the team has limited bandwidth to maintain and work on two cluster, two setups, and so on -- as well as meeting our deliverables.
** Faidon: Well, Ops should probably be contributing more time to this project.
** Otto: Also, if we're not perf testing, we can totally do most of this in labs.

== Data Analyst Access ==

* Otto: Only people who currently need access are WMF employees who have shell access. But it's a bit onerous to access all the different services needed to administer the cluster or debug jobs (as they're on many different machines with many different ports) via a bunch of SSH tunnels.
** Mark: Okay, so let's deal with that by restricting access to shell accounts, and talk about the future.
* NN user issue: use kerberos for inter-node auth, or iptables on NN to protect data comm ports. Ops will def help in making that happen ASAP.

== Privacy & Anonymization ==

* Everyone agrees this privacy & anonymization is a critical requirement, and it needs to happen as part of the Base Cluster.
* Hash(IP+Salt) is the standard; definitely need to rotate the salt regularly, not keep records, etc.
** (Note: The salt will need to live in ZK to be accessible and consistent across processing nodes, but mark the key it as volatile so it isn't persisted anywhere.)

* Tim: Currently, we keep 3 months of IPs in checkuser; if we keep IPs around for longer than that, it would be problematic.
* Mark: Performing anonymization on the edge is feasible, and probably preferable for a bunch of reasons -- especially from ESAMS. We should look into this.
** Might also be able to do it in MediaWiki, but that wouldn't touch all traffic

== Followups ==

=== Kafka Contractor ===

[http://edenhill.se/ Magnus Edenhill], who wrote the [https://github.com/edenhill/librdkafka librdkafka], the major C library for the Kafka protocol with both producer and consumer support. Patrick reached out to him; he quoted us a super, incredibly reasonable price for:

* varnishncsa logging module
* Zookeeper support
* Offered to add 0.8 protocol support pro-bono

To decide if we want to go forward with this work, we need to decide whether there's any meaningful interest in replacing udp2log, especially as Analytics doesn't have the time or bandwidth to manage this guy.

Comments:

* Cost is low, and benefit is huge
* Could use LVS to loadbalance the TCP connections
** Would be lumpy, as connections are long-lived, and it wouldn't be balancing on bytes
** Would make ZK support unnecessary

Conclusion:

* Definitely followup with the dude
* Ops (Mark) will manage/review the work

=== Minimal Viable Cluster ===

Immediate work:

* Network ACL (Ops)
* Set up iptables on the NN to only whitelist cluster nodes on the data ports (Otto)

=== Initial Base Cluster ===

At what point should the cutover happen between the MVC (current running cluster), and the newly built Base Cluster? That is: When do we cut over the data import stream?

* Puppetization of all components, Op Friendship
* Security is satisfactory
* Anonymization of IPs

Who on Ops is going to be involved in setting this up and reviewing it? Volunteers:
* Mark, Faidon, possibly others

== Conclusions & Next Steps ==

* Keep current cluster running as-is
* Continue puppet work on Initial Base Cluster
** Find machines: either pull out ~5 datanodes, or do it in labs (prefer labs)
** Start building new cluster

Analytics/Archive/Infrastructure/Meetings/ArchitectureReview

2013-03-04T18:21:57Z

Dsc: Slides.

* Date: March 1, 2013 (2:30p - 3:30p)
* Present: Mark, Faidon, Tim, Ryan, Ori, Ryan, Robla, Kraig, Dan, David, Andrew, Diederik
* Slides: [[File:Kraken_Arch_Review_Slides.pdf|(pdf)]]

== Background ==

* Project page: https://www.mediawiki.org/wiki/Analytics/Kraken
* Documentation:
** Overview: https://www.mediawiki.org/wiki/Analytics/Kraken/Overview
** Software Overview: https://www.mediawiki.org/wiki/Analytics/Kraken/Overview/Software
** Services and Ports: https://www.mediawiki.org/wiki/Analytics/Kraken/Ports
** Infrastructure / Hardware: https://www.mediawiki.org/wiki/Analytics/Kraken/Infrastructure
* Related:
** ACL Ticket: https://rt.wikimedia.org/Ticket/Display.html?id=4433
** Security Meeting Notes: https://www.mediawiki.org/wiki/Analytics/Kraken/Meetings/SecurityReview

= Meeting Notes =

== Minimum Viable Cluster ==

Our first goal is get to a "minimum viable" cluster setup.
To meet our stakeholder obligations, we'll need:

* Unsampled storage of Mobile pageview requests as txt files
* Unsampled storage of EventLogging data as JSON or txt files
* Importing MediaWiki databases
* Ability to access raw datafiles by WMF employees
* Ability to query data using Pig, Hive, MapReduce, and Hadoop streaming
* Ability to schedule jobs and workflows (Oozie)
* Automated recovery of data importers (Zookeeper)

Not part of MVC – deferring for later discussion:
* Realtime processing layer (Storm)
* Guaranteed delivery of web-request data (Kafka producers and future of udp2log)

== Dataflow Outline ==

* Mark: Asher says mediawiki import cannot be part of MVC -- he said it can be done, but I don't have all the details. Maybe something about a "binlog API"?
* Faidon: This doesn't seem like a very efficient use of hardware...
** Yeah, totally not. It's an evolutionary deployment, but this is a description of how things stand today. The end goal (and scope) looks very different. This was a way to simulate a production setup (including Kafka on the edge) without making production changes. A "production setup" is not a near-term goal any more (as we've scoped out ETL for now), so this looks a bit weird.
* Faidon: Rolling resetup/reimage means some number of machines should be detached from the current cluster for the process of reviewing and rebuilding off the mainline puppet
** Kraig: Yeah, but unfortunately the team has limited bandwidth to maintain and work on two cluster, two setups, and so on -- as well as meeting our deliverables.
** Faidon: Well, Ops should probably be contributing more time to this project.
** Otto: Also, if we're not perf testing, we can totally do most of this in labs.

== Data Analyst Access ==

* Otto: Only people who currently need access are WMF employees who have shell access. But it's a bit onerous to access all the different services needed to administer the cluster or debug jobs (as they're on many different machines with many different ports) via a bunch of SSH tunnels.
** Mark: Okay, so let's deal with that by restricting access to shell accounts, and talk about the future.
* NN user issue: use kerberos for inter-node auth, or iptables on NN to protect data comm ports. Ops will def help in making that happen ASAP.

== Privacy & Anonymization ==

* Everyone agrees this privacy & anonymization is a critical requirement, and it needs to happen as part of the Base Cluster.
* Hash(IP+Salt) is the standard; definitely need to rotate the salt regularly, not keep records, etc.
** (Note: The salt will need to live in ZK to be accessible and consistent across processing nodes, but mark the key it as volatile so it isn't persisted anywhere.)

* Tim: Currently, we keep 3 months of IPs in checkuser; if we keep IPs around for longer than that, it would be problematic.
* Mark: Performing anonymization on the edge is feasible, and probably preferable for a bunch of reasons -- especially from ESAMS. We should look into this.
** Might also be able to do it in MediaWiki, but that wouldn't touch all traffic

== Followups ==

=== Kafka Contractor ===

[http://edenhill.se/ Magnus Edenhill], who wrote the [https://github.com/edenhill/librdkafka librdkafka], the major C library for the Kafka protocol with both producer and consumer support. Patrick reached out to him; he quoted us a super, incredibly reasonable price for:

* varnishncsa logging module
* Zookeeper support
* Offered to add 0.8 protocol support pro-bono

To decide if we want to go forward with this work, we need to decide whether there's any meaningful interest in replacing udp2log, especially as Analytics doesn't have the time or bandwidth to manage this guy.

Comments:

* Cost is low, and benefit is huge
* Could use LVS to loadbalance the TCP connections
** Would be lumpy, as connections are long-lived, and it wouldn't be balancing on bytes
** Would make ZK support unnecessary

Conclusion:

* Definitely followup with the dude
* Ops (Mark) will manage/review the work

=== Minimal Viable Cluster ===

Immediate work:

* Network ACL (Ops)
* Set up iptables on the NN to only whitelist cluster nodes on the data ports (Otto)

=== Initial Base Cluster ===

At what point should the cutover happen between the MVC (current running cluster), and the newly built Base Cluster? That is: When do we cut over the data import stream?

* Puppetization of all components, Op Friendship
* Security is satisfactory
* Anonymization of IPs

Who on Ops is going to be involved in setting this up and reviewing it? Volunteers:
* Mark, Faidon, possibly others

== Conclusions & Next Steps ==

* Keep current cluster running as-is
* Continue puppet work on Initial Base Cluster
** Find machines: either pull out ~5 datanodes, or do it in labs (prefer labs)
** Start building new cluster

File:Kraken Arch Review Slides.pdf

2013-03-04T18:16:09Z

Dsc: Slides from the Kraken Arch Review meeting. == Licensing == {{PD-self|Diederik van Liere}}

Slides from the [[Analytics/Kraken/Meetings/ArchitectureReview|Kraken Arch Review meeting]].

== Licensing ==
{{PD-self|Diederik van Liere}}

Analytics/Roadmap/PlanningMeetings

2013-03-03T02:49:23Z

Dsc:

Meeting notes for Team Analytics planning.

= 2012 =

* (Sept 20) [[Analytics/Roadmap/PlanningMeetings/2012_Sept_20|Autumnal Planning]] - Data Release Process; Sept-Oct Milestone Planning
* (Dec 12) [[Analytics/Roadmap/PlanningMeetings/2012_Q2_Quarterly_Review|Q2 Quarterly Review]]

= 2013 =

* (2013) [[Analytics/Roadmap/PlanningMeetings/2013_Scrum|Daily Scrum Notes]]
* (Feb 12) [[Analytics/Kraken/Security Review Meeting|Kraken Security Review]]
* (Feb 27) [[Analytics/Roadmap/PlanningMeetings/2013_02_27_Reflection|Weekly Reflection]]
* (Mar 1) [[Analytics/Kraken/Meetings/ArchitectureReview|Kraken Architecture Review]]

Analytics/Kraken

2013-03-03T02:46:46Z

Dsc:

{{Wikimedia engineering project information
| name = Kraken (Analytics Cluster)
| description = A robust, distributed computing and data services platform.
| start = 2011-05-22
| end =
| group = Analytics
| lead =
| team = [[User:Ottomata|Andrew Otto]], [[User:Dsc|David Schoonover]], [[User:Diederik|Diederik van Liere]]
| previous =
| next =
| projectpage = Analytics/Kraken
| display = {{{display|}}}
}}<noinclude>

Kraken is the code-name for the robust distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

= [[{{FULLPAGENAME}}/status|Status]] =

{{Wikimedia project status line|{{FULLPAGENAME}}}}

= Rationale =

The Wiki Movement has a chronic need for analytics. We need it to understand our editors, to encourage growth, to engender diversity, to focus our resources, to improve our engineering efforts, and to measure our success. It permeates nearly all our goals, yet our current analytics capabilities are underdeveloped: we lack infrastructure to capture editor, visitor, clickstream, and device data in a way that is easily accessible; our efforts are distributed among different departments; our data is fragmented over different systems and databases; our tools are ad-hoc.

Rather than merely improve existing jobs and data pipelines, the Analytics Team aims to construct a Data Services Platform capable of mining intelligence from all datastreams of interest, providing this insight in real time, and exposing it via an API to power applications, mash up into websites, and stream to devices.

= Documentation =

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

* [[Analytics/Kraken/Access|Getting Access]]
* [[Analytics/Kraken/Overview|Overview]]
** [[Analytics/Kraken/Overview/Software|Software Overview]]
** [http://upload.wikimedia.org/wikipedia/mediawiki/1/1b/Kraken_flow_diagram-2.png Cluster Dataflow Diagram]
* '''Data'''
** [[Analytics/Product_Codes|Product Codes]]
** [[Analytics/Kraken/Data_Streams|Data Streams]]
** [[Analytics/Kraken/Data_Formats|Data Formats]]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
* [[Analytics/Kraken/Request_Logging|Request Logging]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* '''Notes'''
** [[Analytics/Kraken/Blurbs|Blurbs About Kraken]]
** [https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdEtRLVNoQWNvQzNleHBtTXR5emI3Z2c&pli=1#gid=5 Kraken Benchmarks]
** [http://etherpad.wikimedia.org/AnalyticsHadoopSetupNotes Hadoop Setup Notes]
** [http://etherpad.wikimedia.org/AnalyticsKrakenTestClusterSetup Test Cluster Setup Notes]
* '''[[Analytics/Kraken/Meetings|Meeting Notes]]'''
** [[Analytics/Kraken/Meetings/SecurityReview|Security Review Meeting]]
** [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

= Components =

== [[Analytics/Kraken/Pixel_Service|Pixel Service]] and [[Analytics/Kraken/Request_Logging|Request Logging]] ==

Public data import endpoint, and system for capturing the incoming firehose from our front-end servers.

* [[Analytics/Kraken/Request_Logging|Request Logging]]
** [[Analytics/Kraken/Request_Logging|System Recommendation]]
** [[Analytics/Kraken/Logging_Solutions_Overview|Distributed Logging Systems Research]]
** [https://docs.google.com/spreadsheet/pub?key=0AvpRkIqSY9hNdFNoYWFIX29hMnFuenV1ckRvYzEzUVE&gid=3 Feature Comparison Spreadsheet]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]

'''Tasks'''

* Work with Ori, Patrick, & ops to figure out Kafka producer situation for production [otto, dsc, ori, preilly, ops]
* MediaWiki EventLogging integration

== [[Analytics/Kraken/ETL|ETL Topology]] ==

The [http://storm-project.net Storm] topology that processes incoming data (aka, "Extract-Transform-Load" in data warehousing jargon).

* [[Analytics/Kraken/ETL|ETL Topology]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Maven + Nexus setup w/ storm pom [dsc]
* Storm Bolts:
** Consume from Kafka (by topic?)
** Serialize & canonicalize record using Avro schema
** GeoIP annotation
** Mobile carrier annotation (by IP)
** IP anonymization (where does the salt live? how often should we change it?)
** Append record to per-hour files in HDFS (path using timestamp, topic)
** Notifier Bolt to update Kafka consumer checkpoint (and/or publish event for external hooks)
* Define Avro schemas:
** WebRequest
** EventData
*** Figure out convention for EventData packets to self-identify with a more specific schema

== [[Analytics/Kraken/Core_Jobs|Core Jobs]] ==

Core data processing jobs for processing web requests and event data, maintained by the Analytics team.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Figure out request tagging -- how can we avoid mutating records?
* Reportcard Oozie workflow [DvL]
** Pig Macro for "typical request analytics"
* Figure out indexing scheme:
** Aggregated time buckets
** Referrer chains
** Event aggregations
** Funnel Analysis
*** A/B testing event data

== [[Analytics/Kraken/Data_Tools|Data Tools]] ==

Data processing library and toolkit provided & maintained by the Analytics team, esp for use with Kraken.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Setup regular sqoop imports [DvL]
* Avro De/Serialization storage format compatibility:
** WebRequest
** EventData
* Build re-usable pig/hive library, outsource this to analysts as much as possible [DvL]
* KV-Pairs parsing tools
* Conversion funnel analysis tools
** A/B testing analysis tools
* MW EventLogging extension integration
* Investigate Ori's Mediawiki extension for instrumentation, sending event/edit data

== [[Analytics/Kraken/Infrastructure|Infrastructure]] ==

Bucket for general cluster infrastructure and maintenance tasks.
* [[Analytics/Kraken/Puppet|Kraken Puppetization]]
* [[Analytics/Kraken/JMX Ports|JMX Ports]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* [[Analytics/Kraken/JMX Monitoring|JMX Monitoring Research]]

'''Meeting Notes'''

* [[Analytics/Kraken/Meetings/SecurityReview|Security Review Meeting]]
* [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

'''Tasks'''

* Fix the one busted Cisco machine (an07) [otto]
* Continue puppetization [otto]
* JMX Monitoring [otto, dsc]
* Investigate / experiment / benchmark Hadoop stack [DvL]
* Data owner services -- export, dashboard, visualizations (on Hue?) [dsc, dan]
** Hue plugin for Limn integration? [dsc]
* Get analysts experimenting with Hive/Pig [DvL]
* General purpose job-runner using Kafka+Storm
* Setup Hadoop FairScheduler [DvL]

= Research =

== Bundling ==

Investigate size of savings of structure sharing via a short-term (between 1-30sec window) queue.

* Reorder within window by timestamp? Seq number?
* Parameters of experiment:
** Buffer time: 1s, 2s, 5s, 10s
** Sharded by server
** Bundle by IP, UA, IP+UA
* Pull-up fields for structure sharing:
** Invariant: ip, ua, accept-lang, carrier
** Often invariant: method, status, referer

== Software ==
* Hbase
* Hive
* Hue plugins
* Cascading -- do we even need this as we have Oozie?

[[Category:WMF Projects]]
[[Category:WMF Projects/Analytics]]

</noinclude>

Analytics/Kraken

2013-03-03T02:46:16Z

Dsc:

{{Wikimedia engineering project information
| name = Kraken (Analytics Cluster)
| description = A robust, distributed computing and data services platform.
| start = 2011-05-22
| end =
| group = Analytics
| lead =
| team = [[User:Ottomata|Andrew Otto]], [[User:Dsc|David Schoonover]], [[User:Diederik|Diederik van Liere]]
| previous =
| next =
| projectpage = Analytics/Kraken
| display = {{{display|}}}
}}<noinclude>

Kraken is the code-name for the robust distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

= [[{{FULLPAGENAME}}/status|Status]] =

{{Wikimedia project status line|{{FULLPAGENAME}}}}

= Rationale =

The Wiki Movement has a chronic need for analytics. We need it to understand our editors, to encourage growth, to engender diversity, to focus our resources, to improve our engineering efforts, and to measure our success. It permeates nearly all our goals, yet our current analytics capabilities are underdeveloped: we lack infrastructure to capture editor, visitor, clickstream, and device data in a way that is easily accessible; our efforts are distributed among different departments; our data is fragmented over different systems and databases; our tools are ad-hoc.

Rather than merely improve existing jobs and data pipelines, the Analytics Team aims to construct a Data Services Platform capable of mining intelligence from all datastreams of interest, providing this insight in real time, and exposing it via an API to power applications, mash up into websites, and stream to devices.

= Documentation =

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

* [[Analytics/Kraken/Access|Getting Access]]
* [[Analytics/Kraken/Overview|Overview]]
** [[Analytics/Kraken/Overview/Software|Software Overview]]
** [http://upload.wikimedia.org/wikipedia/mediawiki/1/1b/Kraken_flow_diagram-2.png Cluster Dataflow Diagram]
* '''Data'''
** [[Analytics/Product_Codes|Product Codes]]
** [[Analytics/Kraken/Data_Streams|Data Streams]]
** [[Analytics/Kraken/Data_Formats|Data Formats]]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
* [[Analytics/Kraken/Request_Logging|Request Logging]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* '''Notes'''
** [[Analytics/Kraken/Blurbs|Blurbs About Kraken]]
** [https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdEtRLVNoQWNvQzNleHBtTXR5emI3Z2c&pli=1#gid=5 Kraken Benchmarks]
** [http://etherpad.wikimedia.org/AnalyticsHadoopSetupNotes Hadoop Setup Notes]
** [http://etherpad.wikimedia.org/AnalyticsKrakenTestClusterSetup Test Cluster Setup Notes]
* '''[[Analytics/Kraken/Meetings|Meeting Notes]]'''
** [[Analytics/Kraken/Meetings/SecurityReview|Security Review Meeting]]
** [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

= Components =

== [[Analytics/Kraken/Pixel_Service|Pixel Service]] and [[Analytics/Kraken/Request_Logging|Request Logging]] ==

Public data import endpoint, and system for capturing the incoming firehose from our front-end servers.

* [[Analytics/Kraken/Request_Logging|Request Logging]]
** [[Analytics/Kraken/Request_Logging|System Recommendation]]
** [[Analytics/Kraken/Logging_Solutions_Overview|Distributed Logging Systems Research]]
** [https://docs.google.com/spreadsheet/pub?key=0AvpRkIqSY9hNdFNoYWFIX29hMnFuenV1ckRvYzEzUVE&gid=3 Feature Comparison Spreadsheet]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]

'''Tasks'''

* Work with Ori, Patrick, & ops to figure out Kafka producer situation for production [otto, dsc, ori, preilly, ops]
* MediaWiki EventLogging integration

== [[Analytics/Kraken/ETL|ETL Topology]] ==

The [http://storm-project.net Storm] topology that processes incoming data (aka, "Extract-Transform-Load" in data warehousing jargon).

* [[Analytics/Kraken/ETL|ETL Topology]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Maven + Nexus setup w/ storm pom [dsc]
* Storm Bolts:
** Consume from Kafka (by topic?)
** Serialize & canonicalize record using Avro schema
** GeoIP annotation
** Mobile carrier annotation (by IP)
** IP anonymization (where does the salt live? how often should we change it?)
** Append record to per-hour files in HDFS (path using timestamp, topic)
** Notifier Bolt to update Kafka consumer checkpoint (and/or publish event for external hooks)
* Define Avro schemas:
** WebRequest
** EventData
*** Figure out convention for EventData packets to self-identify with a more specific schema

== [[Analytics/Kraken/Core_Jobs|Core Jobs]] ==

Core data processing jobs for processing web requests and event data, maintained by the Analytics team.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Figure out request tagging -- how can we avoid mutating records?
* Reportcard Oozie workflow [DvL]
** Pig Macro for "typical request analytics"
* Figure out indexing scheme:
** Aggregated time buckets
** Referrer chains
** Event aggregations
** Funnel Analysis
*** A/B testing event data

== [[Analytics/Kraken/Data_Tools|Data Tools]] ==

Data processing library and toolkit provided & maintained by the Analytics team, esp for use with Kraken.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Setup regular sqoop imports [DvL]
* Avro De/Serialization storage format compatibility:
** WebRequest
** EventData
* Build re-usable pig/hive library, outsource this to analysts as much as possible [DvL]
* KV-Pairs parsing tools
* Conversion funnel analysis tools
** A/B testing analysis tools
* MW EventLogging extension integration
* Investigate Ori's Mediawiki extension for instrumentation, sending event/edit data

== [[Analytics/Kraken/Infrastructure|Infrastructure]] ==

Bucket for general cluster infrastructure and maintenance tasks.
* [[Analytics/Kraken/Puppet|Kraken Puppetization]]
* [[Analytics/Kraken/JMX Ports|JMX Ports]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* [[Analytics/Kraken/JMX Monitoring|JMX Monitoring Research]]

'''Meeting Notes'''

* [[Analytics/Kraken/Meetings/Security_Review_Meeting|Security Review Meeting]]
* [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

'''Tasks'''

* Fix the one busted Cisco machine (an07) [otto]
* Continue puppetization [otto]
* JMX Monitoring [otto, dsc]
* Investigate / experiment / benchmark Hadoop stack [DvL]
* Data owner services -- export, dashboard, visualizations (on Hue?) [dsc, dan]
** Hue plugin for Limn integration? [dsc]
* Get analysts experimenting with Hive/Pig [DvL]
* General purpose job-runner using Kafka+Storm
* Setup Hadoop FairScheduler [DvL]

= Research =

== Bundling ==

Investigate size of savings of structure sharing via a short-term (between 1-30sec window) queue.

* Reorder within window by timestamp? Seq number?
* Parameters of experiment:
** Buffer time: 1s, 2s, 5s, 10s
** Sharded by server
** Bundle by IP, UA, IP+UA
* Pull-up fields for structure sharing:
** Invariant: ip, ua, accept-lang, carrier
** Often invariant: method, status, referer

== Software ==
* Hbase
* Hive
* Hue plugins
* Cascading -- do we even need this as we have Oozie?

[[Category:WMF Projects]]
[[Category:WMF Projects/Analytics]]

</noinclude>

Analytics/Archive/Infrastructure/Meetings/ArchitectureReview

2013-03-03T02:45:33Z

Dsc: Created page with " * Date: March 1, 2013 (2:30p - 3:30p) * Present: Mark, Faidon, Tim, Ryan, Ori, Ryan, Robla, Kraig, Dan, David, Andrew, Diederik * Slides: XXX == Background == * Project pa..."

* Date: March 1, 2013 (2:30p - 3:30p)
* Present: Mark, Faidon, Tim, Ryan, Ori, Ryan, Robla, Kraig, Dan, David, Andrew, Diederik
* Slides: XXX

== Background ==

* Project page: https://www.mediawiki.org/wiki/Analytics/Kraken
* Documentation:
** Overview: https://www.mediawiki.org/wiki/Analytics/Kraken/Overview
** Software Overview: https://www.mediawiki.org/wiki/Analytics/Kraken/Overview/Software
** Services and Ports: https://www.mediawiki.org/wiki/Analytics/Kraken/Ports
** Infrastructure / Hardware: https://www.mediawiki.org/wiki/Analytics/Kraken/Infrastructure
* Related:
** ACL Ticket: https://rt.wikimedia.org/Ticket/Display.html?id=4433
** Security Meeting Notes: https://www.mediawiki.org/wiki/Analytics/Kraken/Meetings/SecurityReview

= Meeting Notes =

== Minimum Viable Cluster ==

Our first goal is get to a "minimum viable" cluster setup.
To meet our stakeholder obligations, we'll need:

* Unsampled storage of Mobile pageview requests as txt files
* Unsampled storage of EventLogging data as JSON or txt files
* Importing MediaWiki databases
* Ability to access raw datafiles by WMF employees
* Ability to query data using Pig, Hive, MapReduce, and Hadoop streaming
* Ability to schedule jobs and workflows (Oozie)
* Automated recovery of data importers (Zookeeper)

Not part of MVC – deferring for later discussion:
* Realtime processing layer (Storm)
* Guaranteed delivery of web-request data (Kafka producers and future of udp2log)

== Dataflow Outline ==

* Mark: Asher says mediawiki import cannot be part of MVC -- he said it can be done, but I don't have all the details. Maybe something about a "binlog API"?
* Faidon: This doesn't seem like a very efficient use of hardware...
** Yeah, totally not. It's an evolutionary deployment, but this is a description of how things stand today. The end goal (and scope) looks very different. This was a way to simulate a production setup (including Kafka on the edge) without making production changes. A "production setup" is not a near-term goal any more (as we've scoped out ETL for now), so this looks a bit weird.
* Faidon: Rolling resetup/reimage means some number of machines should be detached from the current cluster for the process of reviewing and rebuilding off the mainline puppet
** Kraig: Yeah, but unfortunately the team has limited bandwidth to maintain and work on two cluster, two setups, and so on -- as well as meeting our deliverables.
** Faidon: Well, Ops should probably be contributing more time to this project.
** Otto: Also, if we're not perf testing, we can totally do most of this in labs.

== Data Analyst Access ==

* Otto: Only people who currently need access are WMF employees who have shell access. But it's a bit onerous to access all the different services needed to administer the cluster or debug jobs (as they're on many different machines with many different ports) via a bunch of SSH tunnels.
** Mark: Okay, so let's deal with that by restricting access to shell accounts, and talk about the future.
* NN user issue: use kerberos for inter-node auth, or iptables on NN to protect data comm ports. Ops will def help in making that happen ASAP.

== Privacy & Anonymization ==

* Everyone agrees this privacy & anonymization is a critical requirement, and it needs to happen as part of the Base Cluster.
* Hash(IP+Salt) is the standard; definitely need to rotate the salt regularly, not keep records, etc.
** (Note: The salt will need to live in ZK to be accessible and consistent across processing nodes, but mark the key it as volatile so it isn't persisted anywhere.)

* Tim: Currently, we keep 3 months of IPs in checkuser; if we keep IPs around for longer than that, it would be problematic.
* Mark: Performing anonymization on the edge is feasible, and probably preferable for a bunch of reasons -- especially from ESAMS. We should look into this.
** Might also be able to do it in MediaWiki, but that wouldn't touch all traffic

== Followups ==

=== Kafka Contractor ===

[http://edenhill.se/ Magnus Edenhill], who wrote the [https://github.com/edenhill/librdkafka librdkafka], the major C library for the Kafka protocol with both producer and consumer support. Patrick reached out to him; he quoted us a super, incredibly reasonable price for:

* varnishncsa logging module
* Zookeeper support
* Offered to add 0.8 protocol support pro-bono

To decide if we want to go forward with this work, we need to decide whether there's any meaningful interest in replacing udp2log, especially as Analytics doesn't have the time or bandwidth to manage this guy.

Comments:

* Cost is low, and benefit is huge
* Could use LVS to loadbalance the TCP connections
** Would be lumpy, as connections are long-lived, and it wouldn't be balancing on bytes
** Would make ZK support unnecessary

Conclusion:

* Definitely followup with the dude
* Ops (Mark) will manage/review the work

=== Minimal Viable Cluster ===

Immediate work:

* Network ACL (Ops)
* Set up iptables on the NN to only whitelist cluster nodes on the data ports (Otto)

=== Initial Base Cluster ===

At what point should the cutover happen between the MVC (current running cluster), and the newly built Base Cluster? That is: When do we cut over the data import stream?

* Puppetization of all components, Op Friendship
* Security is satisfactory
* Anonymization of IPs

Who on Ops is going to be involved in setting this up and reviewing it? Volunteers:
* Mark, Faidon, possibly others

== Conclusions & Next Steps ==

* Keep current cluster running as-is
* Continue puppet work on Initial Base Cluster
** Find machines: either pull out ~5 datanodes, or do it in labs (prefer labs)
** Start building new cluster

Analytics/Archive/Infrastructure/Meetings

2013-03-02T00:34:13Z

Dsc:

Meeting notes from Kraken planning.

* [[Analytics/Kraken/Meetings/SecurityReview|Security Review Meeting]]
* [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

Analytics/Infrastructure/Security Review Meeting

2013-03-02T00:33:41Z

Dsc: Dsc moved page Analytics/Kraken/Security Review Meeting to Analytics/Kraken/Meetings/SecurityReview

#REDIRECT [[Analytics/Kraken/Meetings/SecurityReview]]

Analytics/Archive/Infrastructure/Meetings/SecurityReview

2013-03-02T00:33:41Z

Dsc: Dsc moved page Analytics/Kraken/Security Review Meeting to Analytics/Kraken/Meetings/SecurityReview

= Original Suggested Agenda =

This is an outline of information and topics that might be of interest to discuss. It's not intended as comprehensive, just a starting point. Feel free to add/edit.

== Architecture Overview ==

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

There's been some discussion of swapping this meeting with the Arch Review meeting, so we can work out any questions with the components before discussing their security implications. Either way, we'll probably want a brief overview of the architecture before moving forward.

== Cluster Security ==

=== Public Surfaces ===

* ACL as outlined in the [https://rt.wikimedia.org/Ticket/Display.html?id=4433 RT ticket].
* Cross-datacenter connectivity (esams) requires a solution (public IP, or perhaps the existing bridge?)

=== Services and Access Needs ===

Many internal service dashboards and control panels:

* Hue: HDFS web access; Hadoop job scheduling (via Oozie); Hive query dashboard (Beeswax)
** Hue WebUI Login authentication uses LDAP
** Can control privileges within the dashboard to granularly restrict access to particular services,
** Limited control can be exerted on resource use
** Dashboard access needed by analysts need access for job monitoring/control, and data access
* Hadoop Admin pages
** NameNode: Provides HDFS logs and system health overview. Cluster administrator access only.
** JobTracker, DataNode: provides logs and debugging output for Hadoop jobs. Access needed by analysts to debug.
* Storm's Nimbus: storm job monitoring and scheduling. Cluster administrator access only.
* Graphite: Application and host monitoring for the cluster. Cluster administrator access only.

=== HDFS ===

* http://blog.cloudera.com/blog/2012/03/authorization-and-authentication-in-hadoop/
* <code>hdfs</code> is the superuser within the HDFS file system. Hadoop piggybacks on Unix users/groups, but uses its own protocols for communication, not the shell (ssh). This means being logged in as the <code>hdfs</code> Unix account is sufficient to take actions using the Hadoop shell tools as <code>hdfs</code>, the superuser. Hadoop provides an optional Kerberos layer to provide authentication, but there are other solutions, such as firewalling off the NameNode using iptables to ensure only whitelisted nodes from connecting to the HDFS cluster.

== Data Retention ==

=== What Potentially Private or Sensitive Data is there? ===

* IP addresses
* Browsing activity (including edits) correlated with Usernames & IPs
* Data leakage via search queries / referrers
* Low-entropy Data:
** Precise geographic information about all activity
** Login/Logout by IP across all web properties

=== Raw Logs ===

* First persistantly stored in Kafka brokers; buffer window is 7 days ([http://kafka.apache.org/configuration.html log.retention.hours]=168), and automatically deleted afterward
* ETL anonymization pipeline sees raw data, removes IPs (salted hash) after geo lookup

=== Policies ===

* Never publish raw logs or unaggregated datasets
* Raw logs only accessible by those who have signed the Data NDA.
** Can control HDFS file permissions via LDAP groups, allowing segregation of NDA/non-NDA data access, private data import, etc

== Legacy ==

This pertains only to the current, legacy system, but raw archival logs currently exist back to 2011 containing unsanitized IPs.

* '''Something must be done about this!''' GeoIP then hash the IPs, and replace them in the logs! Offsite the files if we '''really''' need backups?
* wikistats processes all of time at once, and frequently has errors that require recomputation

= Meeting Notes =

== Summary ==

Quick summary of notes / take-aways from the Analytics (Kraken) security review meeting.

* analytics1001 has been wiped and reimaged (restoring /home from backup)
* All proxies and externally-facing services have been disabled.
* Work is under way to bring everything that was puppetized under the analytics1001 puppetmaster into operations-puppet after proper review. Andrew is working closely with a number of people in ops to make this happen.
* All future deployments to the cluster will be puppetized and go through normal code review. Other than performance testing, these puppet confs will be tested in labs.
* The rest of the cluster will be wiped and reimaged out of puppet; data in HDFS will be preserved. This can be a rolling process allowing work to proceed while its under way.
* Schedule an Architectural Review meeting sometime during the SF Ops hackathon, including a look at additional services and auth methods that provide access to internal dashboards like Hue &such.
* Ensure all current "application" code (stuff written by WMF) gets reviewed:
** Cron doing HDFS import from Kafka
** Pig UDFs and other data tools used in processing
** Future: Storm ETL layer

We all agreed the overall goal is to get to an acceptable security state. During that process, the Analytics team still needs to continue to meet stakeholder needs and deliver on promises. We decided on keeping running a "minimum viable cluster" while reimaging boxes and civilizing cluster configuration:

* Wall off some portion of the boxes to continue recieving data and running jobs; all other boxes can be wiped (preserving HDFS partitions). Boxes would be incrementally removed from the "unsanitary" cluster, reimaged, and then added to the "sanitary" cluster. Stupid bathroom-related jokes to be avoided.
* Team Analytics to enumerate data processing jobs that will be running in the intermediate period; their configurations and tooling will be reviewed.
* Analytics and Ops engineers continue to have shell access. Jobs can be submitted and managed using the CLI tools; internal dashboards can be accessed via SSH tunnelling. Analysts working on the cluster will be approved for shell access on a case-by-case basis (afaik, just Evan Rosen (full-time analyst for Grantmaking & Programs), and Stefan Petrea (contractor for Analytics)).
* No public, external access of any box in either zone (including proxied, dashboards like Hue, or even static files) that hasn't gone through review.
* Analytics and Ops will work together to find a simple, acceptable mechanism for data export.

== Next Steps ==

* Analytics puppet manfiests fully reviewed and merged into master operations-puppet repository
** Andrew to come pow-wow before the SF Ops hackathon and buddy it up with ops to plow through some of this.
* Schedule Architecture Review
* Rolling reimaging of all analytics boxes (including hadoop data nodes but preserving data) implementing this "minimal viable cluster" plan.

Analytics

2013-03-02T00:32:55Z

Dsc:

[[File:WMF_Analytics_-_Ring_Logo_(thick).svg|200px|right|wmf analytics: logos mean srs bsns]] Howdy from the Wikimedia Analytics team! We're currently working on building a Data Services Platform (codenamed Kraken) to power the next-generation of intelligence and analytics, as well as a new and wondrous [http://reportcard.wmflabs.org/ Reportcard].

If you're enthusiastic about analytics, you might want to [http://blog.wikimedia.org/2012/07/25/meet-the-analytics-team/ meet the team], or subscribe to the [https://lists.wikimedia.org/mailman/listinfo/analytics Analytics Mailing List] (don't worry, it's low-traffic).

== Team ==
* [[User:DAndreescu|Dan Andreescu]]
* [[User:Ottomata|Andrew Otto]]
* [[User:Dsc|Dave Schoonover]]
* [[User:Diederik|Diederik van Liere]]
* [[User:Erik_Zachte|Erik Zachte]]
* [[User:Kparkinson|Kraig Parkinson]]

== Planning ==
* [[Analytics/Roadmap|Roadmap]]
** [[Roadmap#Analytics|Analytics on the Platform Roadmap]]
* [[Analytics/Roadmap/PlanningMeetings|Meeting Notes]]
* [[Analytics/Roadmap/2012-2013|2012-2013 Planning]]

== Stakeholder Corner ==
* [[Analytics/Product_Codes|Product Codes]] — required application identifier needed to get data into the cluster.
* [[Analytics/Metric_definitions|Metrics Definitions]] — canonical definition of metrics.
* [[Analytics/Dreams|Dreams]] — features, metrics/queries, and visualizations people would like to see someday. Importantly, don't worry about these things being reasonable, well-scoped, or put in the right place. Just add stuff here you think would be useful.

== Projects ==

=== [[Analytics/Kraken|Kraken]] ===
* "Kraken" is the codename for the cluster and software which powers the Data Services Platform.
* [[Analytics/Kraken|Project Info]]
* [[Analytics/Product_Codes|Product Codes]]
* [https://upload.wikimedia.org/wikipedia/mediawiki/3/38/Kraken_flow_diagram.png Cluster Dataflow Diagram]
** [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
** [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning docs]]
** [[Analytics/Distributed_Logging_Solutions|Distributed Logging]]
* Latest status: {{Wikimedia project status line|Analytics/Kraken}}

=== [[Analytics/Limn|Limn]] ===
* Limn is a drop-in GUI toolkit for building visualizations. It powers the [http://reportcard.wmflabs.org/ WMF Monthly Reportcard].
* [[Analytics/Limn|Project Info]]
* Source: https://github.com/wikimedia/limn
* Issues: https://github.com/wikimedia/limn/issues
* Latest status: {{Wikimedia project status line|Analytics/Limn}}

=== [[Analytics/Reportcard|Reportcard]] ===
* The [http://reportcard.wmflabs.org/ WMF Monthly Reportcard] is a visual dashboard of high-level metrics on the health and success of the various Wikimedia projects.
* [[Analytics/Reportcard|Project Info]]
* Site: http://reportcard.wmflabs.org/
* Latest status: {{Wikimedia project status line|Analytics/Reportcard}}

=== [[Analytics/Wikistats|Wikistats]] ===
* [[Analytics/Wikistats|Project Info]]
* Latest status: {{Wikimedia project status line|Wikistats}}

=== [[Analytics/Legacy_Logging|Legacy Log Collection]] ===
* [[Analytics/Legacy_Logging|Project Info]]
* Components:
** [[Analytics/udp2log_pageview_logging|UDP Logging]]
** [[Analytics/UDP-filters|UDP Log Filters]]
* Latest status: {{Wikimedia project status line|Analytics/Pageview logging}}

=== [[Analytics/Data_Releases|Data Releases]] ===
* [[Analytics/Data_Releases|Project Info]]
* [[Analytics/Pageviews|Page View Analytics]]
** [[Analytics/Pageviews/Mobile|Mobile]]
** [[Analytics/Pageviews/GLAM|GLAM]]
* Latest status: {{Wikimedia project status line|Analytics/Data Releases}}

=== [[Analytics/Infrastructure|Infrastructure]] ===
* [[Analytics/Infrastructure|Project Info]]
* Resources:
** [[Analytics/Infrastructure/Stat1|<code>stat1</code> server]]
** [[Analytics/Kraken/Infrastructure|Kraken Server Hardware & Infrastructure]]
* Latest status: {{Wikimedia project status line|Analytics/Infrastructure}}

=== See Also ===

* [[Developers/Maintainers#Analytics|Project Maintainers]]

== [[Analytics/Research|Research]] ==
* [[Analytics/Research|Research Hub]]
* [http://pinboard.in/u:wmfa Team Bookmarks on Pinboard]
* Across WMF:
** [[OAuth]]

[[Category:WMF Projects]]

== [[Analytics/Continuous Improvement|Continuous Improvement]] ==

* Using Mingle to track improvements: https://mingle.corp.wikimedia.org/projects/analytics/wiki/Engaging_in_Continuous_Improvement

Analytics/Archive/Infrastructure/Meetings

2013-03-02T00:30:55Z

Dsc: Created page with "Meeting notes from Kraken planning. * Security Review Meeting * Analytics/Kraken/Meetings/ArchitectureReview|Architect..."

Meeting notes from Kraken planning.

* [[Analytics/Kraken/Meetings/Security_Review_Meeting|Security Review Meeting]]
* [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

Analytics/Kraken

2013-03-02T00:30:01Z

Dsc:

{{Wikimedia engineering project information
| name = Kraken (Analytics Cluster)
| description = A robust, distributed computing and data services platform.
| start = 2011-05-22
| end =
| group = Analytics
| lead =
| team = [[User:Ottomata|Andrew Otto]], [[User:Dsc|David Schoonover]], [[User:Diederik|Diederik van Liere]]
| previous =
| next =
| projectpage = Analytics/Kraken
| display = {{{display|}}}
}}<noinclude>

Kraken is the code-name for the robust distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

= [[{{FULLPAGENAME}}/status|Status]] =

{{Wikimedia project status line|{{FULLPAGENAME}}}}

= Rationale =

The Wiki Movement has a chronic need for analytics. We need it to understand our editors, to encourage growth, to engender diversity, to focus our resources, to improve our engineering efforts, and to measure our success. It permeates nearly all our goals, yet our current analytics capabilities are underdeveloped: we lack infrastructure to capture editor, visitor, clickstream, and device data in a way that is easily accessible; our efforts are distributed among different departments; our data is fragmented over different systems and databases; our tools are ad-hoc.

Rather than merely improve existing jobs and data pipelines, the Analytics Team aims to construct a Data Services Platform capable of mining intelligence from all datastreams of interest, providing this insight in real time, and exposing it via an API to power applications, mash up into websites, and stream to devices.

= Documentation =

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

* [[Analytics/Kraken/Access|Getting Access]]
* [[Analytics/Kraken/Overview|Overview]]
** [[Analytics/Kraken/Overview/Software|Software Overview]]
** [http://upload.wikimedia.org/wikipedia/mediawiki/1/1b/Kraken_flow_diagram-2.png Cluster Dataflow Diagram]
* '''Data'''
** [[Analytics/Product_Codes|Product Codes]]
** [[Analytics/Kraken/Data_Streams|Data Streams]]
** [[Analytics/Kraken/Data_Formats|Data Formats]]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
* [[Analytics/Kraken/Request_Logging|Request Logging]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* '''Notes'''
** [[Analytics/Kraken/Blurbs|Blurbs About Kraken]]
** [https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdEtRLVNoQWNvQzNleHBtTXR5emI3Z2c&pli=1#gid=5 Kraken Benchmarks]
** [http://etherpad.wikimedia.org/AnalyticsHadoopSetupNotes Hadoop Setup Notes]
** [http://etherpad.wikimedia.org/AnalyticsKrakenTestClusterSetup Test Cluster Setup Notes]
* '''[[Analytics/Kraken/Meetings|Meeting Notes]]'''
** [[Analytics/Kraken/Meetings/Security_Review_Meeting|Security Review Meeting]]
** [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

= Components =

== [[Analytics/Kraken/Pixel_Service|Pixel Service]] and [[Analytics/Kraken/Request_Logging|Request Logging]] ==

Public data import endpoint, and system for capturing the incoming firehose from our front-end servers.

* [[Analytics/Kraken/Request_Logging|Request Logging]]
** [[Analytics/Kraken/Request_Logging|System Recommendation]]
** [[Analytics/Kraken/Logging_Solutions_Overview|Distributed Logging Systems Research]]
** [https://docs.google.com/spreadsheet/pub?key=0AvpRkIqSY9hNdFNoYWFIX29hMnFuenV1ckRvYzEzUVE&gid=3 Feature Comparison Spreadsheet]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]

'''Tasks'''

* Work with Ori, Patrick, & ops to figure out Kafka producer situation for production [otto, dsc, ori, preilly, ops]
* MediaWiki EventLogging integration

== [[Analytics/Kraken/ETL|ETL Topology]] ==

The [http://storm-project.net Storm] topology that processes incoming data (aka, "Extract-Transform-Load" in data warehousing jargon).

* [[Analytics/Kraken/ETL|ETL Topology]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Maven + Nexus setup w/ storm pom [dsc]
* Storm Bolts:
** Consume from Kafka (by topic?)
** Serialize & canonicalize record using Avro schema
** GeoIP annotation
** Mobile carrier annotation (by IP)
** IP anonymization (where does the salt live? how often should we change it?)
** Append record to per-hour files in HDFS (path using timestamp, topic)
** Notifier Bolt to update Kafka consumer checkpoint (and/or publish event for external hooks)
* Define Avro schemas:
** WebRequest
** EventData
*** Figure out convention for EventData packets to self-identify with a more specific schema

== [[Analytics/Kraken/Core_Jobs|Core Jobs]] ==

Core data processing jobs for processing web requests and event data, maintained by the Analytics team.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Figure out request tagging -- how can we avoid mutating records?
* Reportcard Oozie workflow [DvL]
** Pig Macro for "typical request analytics"
* Figure out indexing scheme:
** Aggregated time buckets
** Referrer chains
** Event aggregations
** Funnel Analysis
*** A/B testing event data

== [[Analytics/Kraken/Data_Tools|Data Tools]] ==

Data processing library and toolkit provided & maintained by the Analytics team, esp for use with Kraken.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Setup regular sqoop imports [DvL]
* Avro De/Serialization storage format compatibility:
** WebRequest
** EventData
* Build re-usable pig/hive library, outsource this to analysts as much as possible [DvL]
* KV-Pairs parsing tools
* Conversion funnel analysis tools
** A/B testing analysis tools
* MW EventLogging extension integration
* Investigate Ori's Mediawiki extension for instrumentation, sending event/edit data

== [[Analytics/Kraken/Infrastructure|Infrastructure]] ==

Bucket for general cluster infrastructure and maintenance tasks.
* [[Analytics/Kraken/Puppet|Kraken Puppetization]]
* [[Analytics/Kraken/JMX Ports|JMX Ports]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* [[Analytics/Kraken/JMX Monitoring|JMX Monitoring Research]]

'''Meeting Notes'''

* [[Analytics/Kraken/Meetings/Security_Review_Meeting|Security Review Meeting]]
* [[Analytics/Kraken/Meetings/ArchitectureReview|Architecture Review Meeting]]

'''Tasks'''

* Fix the one busted Cisco machine (an07) [otto]
* Continue puppetization [otto]
* JMX Monitoring [otto, dsc]
* Investigate / experiment / benchmark Hadoop stack [DvL]
* Data owner services -- export, dashboard, visualizations (on Hue?) [dsc, dan]
** Hue plugin for Limn integration? [dsc]
* Get analysts experimenting with Hive/Pig [DvL]
* General purpose job-runner using Kafka+Storm
* Setup Hadoop FairScheduler [DvL]

= Research =

== Bundling ==

Investigate size of savings of structure sharing via a short-term (between 1-30sec window) queue.

* Reorder within window by timestamp? Seq number?
* Parameters of experiment:
** Buffer time: 1s, 2s, 5s, 10s
** Sharded by server
** Bundle by IP, UA, IP+UA
* Pull-up fields for structure sharing:
** Invariant: ip, ua, accept-lang, carrier
** Often invariant: method, status, referer

== Software ==
* Hbase
* Hive
* Hue plugins
* Cascading -- do we even need this as we have Oozie?

[[Category:WMF Projects]]
[[Category:WMF Projects/Analytics]]

</noinclude>

Meetings/2013-02-28

2013-02-28T01:06:32Z

Dsc:

{{nav
| title = Wikimedia Open Tech Chats
| previous = '''[[Meetings/2012-12-13|13 December]]'''
| next = '''[[Meetings/2013-02-28|28 February]]'''
}}

{| class="wikitable" style="font-size: 120%;"
! style="width: 15%;" | What
| Selenium , Gerrit, Lua and friends
|-
! How
| We're using a combination of Google Hangout on Air (an open YouTube stream from the 6th floor of the WMF office) + potential additional Hangout guest speakers + IRC for text-based communication. There's no attendance limit. You don't need Hangout to participate -- if you can play YouTube videos and join IRC, you're fine.
|-
! When
| Thursday, February 28, 2013, 20:30 [http://timeanddate.com/worldclock/fixedtime.html?msg=WMF+Open+Tech+Chat&iso=20130117T12&p1=224&ah=1 UTC] (12:30 PST), 60 minutes
|-
! Where
|
* Physical space: Wikimedia Foundation Inc., 149 New Montgomery Street, 3rd Floor, San Francisco, CA 94105, USA, 6th floor SF office, "Collab" space
* IRC: {{irc|wikimedia-dev}}
* YouTube stream: to be announced on IRC
|- style="vertical-align: top;"
! Food
| Brown bag - bring your own
|-
! Moderator
| Erik Moeller
|}

== Topics ==

* Browser automation with Selenium and friends. (Chris/Zeljko)
* Gerrit 2.6: What's new / what's coming? (AKA: a gradual reduction in suckiness) (Chad)
* Converting templates to Lua (Brad)
* LevelUp - how you can learn, teach, & get stuff done (Sumana)

== Attendees ==

Add your name, tell your friends!

* [[User:Zeljko.filipin(WMF)|Zeljko.filipin(WMF)]] ([[User talk:Zeljko.filipin(WMF)|talk]]) 22:31, 25 February 2013 (UTC)
* [[User:Cmcmahon(WMF)|Cmcmahon(WMF)]] ([[User talk:Cmcmahon(WMF)|talk]]) 21:51, 27 February 2013 (UTC)
* [[User:Greg (WMF)|Greg (WMF)]] ([[User talk:Greg (WMF)|talk]]) 00:11, 28 February 2013 (UTC)
* [[User:Username|dsc]] ([[User talk:Dsc|talk]]) 01:06, 28 February 2013 (UTC)

Meetings/2013-02-28

2013-02-28T01:06:00Z

Dsc:

{{nav
| title = Wikimedia Open Tech Chats
| previous = '''[[Meetings/2012-12-13|13 December]]'''
| next = '''[[Meetings/2013-02-28|28 February]]'''
}}

{| class="wikitable" style="font-size: 120%;"
! style="width: 15%;" | What
| Selenium , Gerrit, Lua and friends
|-
! How
| We're using a combination of Google Hangout on Air (an open YouTube stream from the 6th floor of the WMF office) + potential additional Hangout guest speakers + IRC for text-based communication. There's no attendance limit. You don't need Hangout to participate -- if you can play YouTube videos and join IRC, you're fine.
|-
! When
| Thursday, February 28, 2013, 20:30 [http://timeanddate.com/worldclock/fixedtime.html?msg=WMF+Open+Tech+Chat&iso=20130117T12&p1=224&ah=1 UTC] (12:30 PST), 60 minutes
|-
! Where
|
* Physical space: Wikimedia Foundation Inc., 149 New Montgomery Street, 3rd Floor, San Francisco, CA 94105, USA, 6th floor SF office, "Collab" space
* IRC: {{irc|wikimedia-dev}}
* YouTube stream: to be announced on IRC
|- style="vertical-align: top;"
! Food
| Brown bag - bring your own
|-
! Moderator
| Erik Moeller
|}

== Topics ==

* Browser automation with Selenium and friends. (Chris/Zeljko)
* Gerrit 2.6: What's new / what's coming? (AKA: a gradual reduction in suckiness) (Chad)
* Converting templates to Lua (Brad)
* LevelUp - how you can learn, teach, & get stuff done (Sumana)

== Attendees ==

Add your name, tell your friends!

* [[User:Zeljko.filipin(WMF)|Zeljko.filipin(WMF)]] ([[User talk:Zeljko.filipin(WMF)|talk]]) 22:31, 25 February 2013 (UTC)
* [[User:Cmcmahon(WMF)|Cmcmahon(WMF)]] ([[User talk:Cmcmahon(WMF)|talk]]) 21:51, 27 February 2013 (UTC)
* [[User:Greg (WMF)|Greg (WMF)]] ([[User talk:Greg (WMF)|talk]]) 00:11, 28 February 2013 (UTC)
*    '''§'''    [[User:Username|dsc]] ([[User talk:Dsc|talk]]) 01:06, 28 February 2013 (UTC)

Analytics/Roadmap/PlanningMeetings/2013 02 27 Reflection

2013-02-28T00:52:11Z

Dsc:

== Reflecting on the Meeting for Reflecting on our Weekly Status ==

* dsc volunteers to help put together a stakeholder feedback survey; ez will help construct a community feedback survey as well
* ez: we should make sure our process collects feedback from the editor and research communities

== Review of [https://mingle.corp.wikimedia.org/projects/analytics/cards/list?color_by=complexity&filters%5B%5D=%5BType%5D%5Bis%5D%5BImprovement+Idea%5D&group_by%5Blane%5D=improvement+idea+status&group_by%5Brow%5D=responsible&lanes=To+Do%2CIn+Progress%2CObservation%2FReview&style=list&tab=WIP+-+Improvement+Ideas Improvement Idea Inbox] ==

Cynefin categories reminder:
* Simple: obvious to all
* Complicated: requires analysis or some other form of investigation
* Complex: can only be perceived in retrospect, but not in advance
* Chaotic: no relationship between cause and effect at systems level

=== #1: Improve business solutioning elements of analytics delivery ===
https://mingle.corp.wikimedia.org/projects/analytics/cards/1

* Many elements of delivering on this:
** visioning with an eye toward integrated solution delivery
** planning for incremental releases
** gathering concrete requirements, communicating them clearly and explicitly, and managing expections
* Vote: do
* Domain:
** Diederik: complex
** Kraig: complex
** EZachte: complicated / complex
** otto: complex
** dsc: complex
** dan: complicated

=== #3: Improve discipline of work prioritization & scheduling ===
https://mingle.corp.wikimedia.org/projects/analytics/cards/3

* dsc: Is this focused on improving the view of feature-work, or task-work? (This also relates to the management tools question.)
** K: primarily organizing/prioritizing, clarifying, and scheduling feature work (as seen from stakeholders)
* ez: re: prioritization. We have production bugs that have gone unfixed. this can't just be about new fewture development!
** K: we'll talk more about this with the work service levels card later
* Vote: do
* Domain:
** dan: complicated
** dsc: complicated
** otto: ditto
** drdee: complicated

=== #4: Introduce Lean management practices for leadership roles ===
https://mingle.corp.wikimedia.org/projects/analytics/cards/4

* Not 6 sigma oriented (which focuses on routine processes)
* drdee: let's not experiment with too many things at once.
* dsc: totally agreed. finite bandwidth -- code must be written.
* Vote: do
* Domain:
** dsc: complicated

Analytics/Roadmap/PlanningMeetings/2013 02 27 Reflection

2013-02-28T00:50:13Z

Dsc:

= Reflecting on Reflecting on our Weekly Status Meeting Meeting =

* dsc volunteers to help put together a stakeholder feedback survey; ez will help construct a community feedback survey as well
* ez: we should make sure our process collects feedback from the editor and research communities

= Review of Idea Inbox =

Cynefin categories reminder:
* Simple: obvious to all
* Complicated: requires analysis or some other form of investigation
* Complex: can only be perceived in retrospect, but not in advance
* Chaotic: no relationship between cause and effect at systems level

== #1: Improve business solutioning elements of analytics delivery ==
https://mingle.corp.wikimedia.org/projects/analytics/cards/1

* Many elements of delivering on this:
** visioning with an eye toward integrated solution delivery
** planning for incremental releases
** gathering concrete requirements, communicating them clearly and explicitly, and managing expections
* Vote: do
* Domain:
** Diederik: complex
** Kraig: complex
** EZachte: complicated / complex
** otto: complex
** dsc: complex
** dan: complicated

== #3: Improve discipline of work prioritization & scheduling ==
https://mingle.corp.wikimedia.org/projects/analytics/cards/3

* dsc: Is this focused on improving the view of feature-work, or task-work? (This also relates to the management tools question.)
** K: primarily organizing/prioritizing, clarifying, and scheduling feature work (as seen from stakeholders)
* ez: re: prioritization. We have production bugs that have gone unfixed. this can't just be about new fewture development!
** K: we'll talk more about this with the work service levels card later
* Vote: do
* Domain:
** dan: complicated
** dsc: complicated
** otto: ditto
** drdee: complicated

== #4: Introduce Lean management practices for leadership roles ==
https://mingle.corp.wikimedia.org/projects/analytics/cards/4

* Not 6 sigma oriented (which focuses on routine processes)
* drdee: let's not experiment with too many things at once.
* dsc: totally agreed. finite bandwidth -- code must be written.
* Vote: do
* Domain:
** dsc: complicated

Analytics/Roadmap/PlanningMeetings/2013 02 27 Reflection

2013-02-28T00:49:33Z

Dsc: Created page with "= Weekly Reflection Meeting - 27 Feb 2013 = == Reflecting on Reflecting on our Weekly Status Meeting Meeting == * dsc volunteers to help put together a stakeholder feedback..."

= Weekly Reflection Meeting - 27 Feb 2013 =

== Reflecting on Reflecting on our Weekly Status Meeting Meeting ==

* dsc volunteers to help put together a stakeholder feedback survey; ez will help construct a community feedback survey as well
* ez: we should make sure our process collects feedback from the editor and research communities

== Review of Idea Inbox ==

Cynefin categories reminder:
* Simple: obvious to all
* Complicated: requires analysis or some other form of investigation
* Complex: can only be perceived in retrospect, but not in advance
* Chaotic: no relationship between cause and effect at systems level

=== #1: Improve business solutioning elements of analytics delivery ===
https://mingle.corp.wikimedia.org/projects/analytics/cards/1

* Many elements of delivering on this:
** visioning with an eye toward integrated solution delivery
** planning for incremental releases
** gathering concrete requirements, communicating them clearly and explicitly, and managing expections
* Vote: do
* Domain:
** Diederik: complex
** Kraig: complex
** EZachte: complicated / complex
** otto: complex
** dsc: complex
** dan: complicated

=== #3: Improve discipline of work prioritization & scheduling ===
https://mingle.corp.wikimedia.org/projects/analytics/cards/3

* dsc: Is this focused on improving the view of feature-work, or task-work? (This also relates to the management tools question.)
** K: primarily organizing/prioritizing, clarifying, and scheduling feature work (as seen from stakeholders)
* ez: re: prioritization. We have production bugs that have gone unfixed. this can't just be about new fewture development!
** K: we'll talk more about this with the work service levels card later
* Vote: do
* Domain:
** dan: complicated
** dsc: complicated
** otto: ditto
** drdee: complicated

=== #4: Introduce Lean management practices for leadership roles ===
https://mingle.corp.wikimedia.org/projects/analytics/cards/4

* Not 6 sigma oriented (which focuses on routine processes)
* drdee: let's not experiment with too many things at once.
* dsc: totally agreed. finite bandwidth -- code must be written.
* Vote: do
* Domain:
** dsc: complicated

Analytics/Roadmap/PlanningMeetings

2013-02-28T00:48:19Z

Dsc:

Analytics/Kraken

2013-02-28T00:46:33Z

Dsc:

{{Wikimedia engineering project information
| name = Kraken (Analytics Cluster)
| description = A robust, distributed computing and data services platform.
| start = 2011-05-22
| end =
| group = Analytics
| lead =
| team = [[User:Ottomata|Andrew Otto]], [[User:Dsc|David Schoonover]], [[User:Diederik|Diederik van Liere]]
| previous =
| next =
| projectpage = Analytics/Kraken
| display = {{{display|}}}
}}<noinclude>

Kraken is the code-name for the robust distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

= [[{{FULLPAGENAME}}/status|Status]] =

{{Wikimedia project status line|{{FULLPAGENAME}}}}

= Rationale =

The Wiki Movement has a chronic need for analytics. We need it to understand our editors, to encourage growth, to engender diversity, to focus our resources, to improve our engineering efforts, and to measure our success. It permeates nearly all our goals, yet our current analytics capabilities are underdeveloped: we lack infrastructure to capture editor, visitor, clickstream, and device data in a way that is easily accessible; our efforts are distributed among different departments; our data is fragmented over different systems and databases; our tools are ad-hoc.

Rather than merely improve existing jobs and data pipelines, the Analytics Team aims to construct a Data Services Platform capable of mining intelligence from all datastreams of interest, providing this insight in real time, and exposing it via an API to power applications, mash up into websites, and stream to devices.

= Documentation =

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

* [[Analytics/Kraken/Access|Getting Access]]
* [[Analytics/Kraken/Overview|Overview]]
** [[Analytics/Kraken/Overview/Software|Software Overview]]
** [http://upload.wikimedia.org/wikipedia/mediawiki/1/1b/Kraken_flow_diagram-2.png Cluster Dataflow Diagram]
* '''Data'''
** [[Analytics/Product_Codes|Product Codes]]
** [[Analytics/Kraken/Data_Streams|Data Streams]]
** [[Analytics/Kraken/Data_Formats|Data Formats]]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
* [[Analytics/Kraken/Request_Logging|Request Logging]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* '''Notes'''
** [[Analytics/Kraken/Security_Review_Meeting|Security Review Meeting notes]]
** [http://etherpad.wikimedia.org/AnalyticsKrakenTestClusterSetup Test Cluster Setup Notes]
** [http://etherpad.wikimedia.org/AnalyticsHadoopSetupNotes Hadoop Setup Notes]
** [[Analytics/Kraken/Blurbs|Blurbs About Kraken]]
** [https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdEtRLVNoQWNvQzNleHBtTXR5emI3Z2c&pli=1#gid=5 Kraken Benchmarks]

= Components =

== [[Analytics/Kraken/Pixel_Service|Pixel Service]] and [[Analytics/Kraken/Request_Logging|Request Logging]] ==

Public data import endpoint, and system for capturing the incoming firehose from our front-end servers.

* [[Analytics/Kraken/Request_Logging|Request Logging]]
** [[Analytics/Kraken/Request_Logging|System Recommendation]]
** [[Analytics/Kraken/Logging_Solutions_Overview|Distributed Logging Systems Research]]
** [https://docs.google.com/spreadsheet/pub?key=0AvpRkIqSY9hNdFNoYWFIX29hMnFuenV1ckRvYzEzUVE&gid=3 Feature Comparison Spreadsheet]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]

'''Tasks'''

* Work with Ori, Patrick, & ops to figure out Kafka producer situation for production [otto, dsc, ori, preilly, ops]
* MediaWiki EventLogging integration

== [[Analytics/Kraken/ETL|ETL Topology]] ==

The [http://storm-project.net Storm] topology that processes incoming data (aka, "Extract-Transform-Load" in data warehousing jargon).

* [[Analytics/Kraken/ETL|ETL Topology]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Maven + Nexus setup w/ storm pom [dsc]
* Storm Bolts:
** Consume from Kafka (by topic?)
** Serialize & canonicalize record using Avro schema
** GeoIP annotation
** Mobile carrier annotation (by IP)
** IP anonymization (where does the salt live? how often should we change it?)
** Append record to per-hour files in HDFS (path using timestamp, topic)
** Notifier Bolt to update Kafka consumer checkpoint (and/or publish event for external hooks)
* Define Avro schemas:
** WebRequest
** EventData
*** Figure out convention for EventData packets to self-identify with a more specific schema

== [[Analytics/Kraken/Core_Jobs|Core Jobs]] ==

Core data processing jobs for processing web requests and event data, maintained by the Analytics team.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Figure out request tagging -- how can we avoid mutating records?
* Reportcard Oozie workflow [DvL]
** Pig Macro for "typical request analytics"
* Figure out indexing scheme:
** Aggregated time buckets
** Referrer chains
** Event aggregations
** Funnel Analysis
*** A/B testing event data

== [[Analytics/Kraken/Data_Tools|Data Tools]] ==

Data processing library and toolkit provided & maintained by the Analytics team, esp for use with Kraken.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Setup regular sqoop imports [DvL]
* Avro De/Serialization storage format compatibility:
** WebRequest
** EventData
* Build re-usable pig/hive library, outsource this to analysts as much as possible [DvL]
* KV-Pairs parsing tools
* Conversion funnel analysis tools
** A/B testing analysis tools
* MW EventLogging extension integration
* Investigate Ori's Mediawiki extension for instrumentation, sending event/edit data

== [[Analytics/Kraken/Infrastructure|Infrastructure]] ==

Bucket for general cluster infrastructure and maintenance tasks.
* [[Analytics/Kraken/Puppet|Kraken Puppetization]]
* [[Analytics/Kraken/JMX Ports|JMX Ports]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* [[Analytics/Kraken/JMX Monitoring|JMX Monitoring Research]]

'''Tasks'''

* Fix the one busted Cisco machine (an07) [otto]
* Continue puppetization [otto]
* JMX Monitoring [otto, dsc]
* Investigate / experiment / benchmark Hadoop stack [DvL]
* Data owner services -- export, dashboard, visualizations (on Hue?) [dsc, dan]
** Hue plugin for Limn integration? [dsc]
* Get analysts experimenting with Hive/Pig [DvL]
* General purpose job-runner using Kafka+Storm
* Setup Hadoop FairScheduler [DvL]

= Research =

== Bundling ==

Investigate size of savings of structure sharing via a short-term (between 1-30sec window) queue.

* Reorder within window by timestamp? Seq number?
* Parameters of experiment:
** Buffer time: 1s, 2s, 5s, 10s
** Sharded by server
** Bundle by IP, UA, IP+UA
* Pull-up fields for structure sharing:
** Invariant: ip, ua, accept-lang, carrier
** Often invariant: method, status, referer

== Software ==
* Hbase
* Hive
* Hue plugins
* Cascading -- do we even need this as we have Oozie?

[[Category:WMF Projects]]
[[Category:WMF Projects/Analytics]]

</noinclude>

Analytics/Kraken/Software

2013-02-28T00:45:25Z

Dsc:

This page is meant to be a brief overview of the software and libraries used to build [[Analytics/Kraken|Kraken]].

== Major Software and Libraries ==

* Hadoop
** HDFS: NameNode, DataNode
** YARN: NodeManager, ResourceManager
* Hadoop tools:
** Hue
** Oozie
** Pig, Hive
** HTTPfs, webhdfs
** LDAP integration?
** Kerberos integration?
* Storm
** Nimbus, Supervisors, Bolts, Spouts
** (We will be writing bolts to perform the ETL tasks, and configuring the topology that links them together)
* ETL libraries:
** MaxMind GeoIP database
** dClass: User Agent parser
** Avro: Data serialization framework
** Camus: Kafka -> HDFS library
* Kafka
* Zookeeper
** Required by Hadoop, Storm, and Kafka
* Monitoring tools:
** jmxtrans
** Ganglia
** Graphite

See the [[Analytics/Kraken/Overview|Kraken Overview]] for a glossary of terms and detail on how these pieces are connected into components.

=== Today ===

* Hadoop + tools, the Kafka brokers, and Zookeeper are all set up
* <code>udp2log</code> logs from the edge -> awk -> Kafka
* no ETL: cron import to HDFS from Kafka via MR job (this means IPs are in logs!)
* User authorization: via shell account, or group membership in labs LDAP
* Monitoring via jmxtrans into Ganglia; no Nagios

== Network Profiles ==

=== Network Usage ===

Network usage by node type:

* Kafka Brokers: 2x the full incoming stream (in from edge, out to ETL). Kafka uses its own protocol over TCP, and supports message bundling, compression, and live failover.
* Storm ETL workers: full stream sharded among nodes; Storm is smart about avoiding inter-node communication if a topology can completely fit on a single worker.
* A file in HDFS is made up of blocks; these blocks can be both local and remote to any given DataNode. HDFS replicates data 3x (configurable) for availability and durability – this means it will copy blocks between nodes to maintain the replication invariant. The NameNode acts as the coordinator for replication, and the authority for addressing files to blocks.
* HDFS is immutable (though it allows appends) – this means that "change" rewrites and replicates the entire block (our blocksize is configured at 256MB).
* A MapReduce job has two phases: the Mappers each read their section of the input from HDFS (with Hadoop ensuring the section is on local disk), and perform some work, emitting (key, value) pairs; Reducers then receive these tuples after a total ordering is applied (via the keys) and perform aggregation, each writing its result back to HDFS via the local DataNode. This result is then replicated as normal.
* At the moment, we plan regular data export <code>stat1001</code>, who acts exclusively as a public webserver.

=== Public Surfaces ===

* ACL as outlined in the [https://rt.wikimedia.org/Ticket/Display.html?id=4433 RT ticket].
* Cross-datacenter connectivity (esams) requires a solution (public IP, or perhaps the existing bridge?)

=== Services and Access Needs ===

Many internal service dashboards and control panels:

* Hue: HDFS web access; Hadoop job scheduling (via Oozie); Hive query dashboard (Beeswax)
** Hue WebUI Login authentication uses LDAP
** Can control privileges within the dashboard to granularly restrict access to particular services,
** Limited control can be exerted on resource use
** Dashboard access needed by analysts need access for job monitoring/control, and data access
* Hadoop Admin pages
** NameNode: Provides HDFS logs and system health overview. Cluster administrator access only.
** JobTracker, DataNode: provides logs and debugging output for Hadoop jobs. Access needed by analysts to debug.
* Storm's Nimbus: storm job monitoring and scheduling. Cluster administrator access only.
* Graphite: Application and host monitoring for the cluster. Cluster administrator access only.

Analytics/Kraken/Software

2013-02-28T00:40:51Z

Dsc:

= Kraken Software Overview =

== Software used in Kraken ==

* Zookeeper
* Hadoop
** HDFS: NameNode, DataNode
** YARN: NodeManager, ResourceManager
* Hadoop tools:
** Hue
** Oozie
** Pig, Hive
** HTTPfs, webhdfs
** LDAP integration?
** Kerberos integration?
* Storm
** Nimbus, Supervisors, Bolts, Spouts
** (We will be writing bolts to perform the ETL tasks, and configuring the topology that links them together)
* ETL libraries:
** MaxMind GeoIP database
** dClass: User Agent parser
** Avro: Data serialization framework
** Camus: Kafka -> HDFS library
* Kafka
* Monitoring tools:
** jmxtrans
** Ganglia
** Graphite

See the [[Analytics/Kraken/Overview|Kraken Overview]] for additional detail.

=== Today ===

* <code>udp2log</code> logs from the edge -> awk -> Kafka
* no ETL: cron import to HDFS from Kafka via MR job (this means IPs are in logs!)
* User authorization: via shell account, or group membership in labs LDAP

== Network Profiles ==

=== Network Usage ===

Network usage by node type:

* Kafka Brokers: 2x the full incoming stream (in from edge, out to ETL). Kafka uses its own protocol over TCP, and supports message bundling, compression, and live failover.
* Storm ETL workers: full stream sharded among nodes; Storm is smart about avoiding inter-node communication if a topology can completely fit on a single worker.
* A file in HDFS is made up of blocks; these blocks can be both local and remote to any given DataNode. HDFS replicates data 3x (configurable) for availability and durability – this means it will copy blocks between nodes to maintain the replication invariant. The NameNode acts as the coordinator for replication, and the authority for addressing files to blocks.
* HDFS is immutable (though it allows appends) – this means that "change" rewrites and replicates the entire block (our blocksize is configured at 256MB).
* A MapReduce job has two phases: the Mappers each read their section of the input from HDFS (with Hadoop ensuring the section is on local disk), and perform some work, emitting (key, value) pairs; Reducers then receive these tuples after a total ordering is applied (via the keys) and perform aggregation, each writing its result back to HDFS via the local DataNode. This result is then replicated as normal.
* At the moment, we plan regular data export <code>stat1001</code>, who acts exclusively as a public webserver.

=== Public Surfaces ===

* ACL as outlined in the [https://rt.wikimedia.org/Ticket/Display.html?id=4433 RT ticket].
* Cross-datacenter connectivity (esams) requires a solution (public IP, or perhaps the existing bridge?)

=== Services and Access Needs ===

Many internal service dashboards and control panels:

* Hue: HDFS web access; Hadoop job scheduling (via Oozie); Hive query dashboard (Beeswax)
** Hue WebUI Login authentication uses LDAP
** Can control privileges within the dashboard to granularly restrict access to particular services,
** Limited control can be exerted on resource use
** Dashboard access needed by analysts need access for job monitoring/control, and data access
* Hadoop Admin pages
** NameNode: Provides HDFS logs and system health overview. Cluster administrator access only.
** JobTracker, DataNode: provides logs and debugging output for Hadoop jobs. Access needed by analysts to debug.
* Storm's Nimbus: storm job monitoring and scheduling. Cluster administrator access only.
* Graphite: Application and host monitoring for the cluster. Cluster administrator access only.

Analytics/Kraken/Software

2013-02-28T00:40:30Z

Dsc: Created page with "= Kraken Software Overview = == Software used in Kraken == * Zookeeper * Hadoop ** HDFS: NameNode, DataNode ** YARN: NodeManager, ResourceManager * Hadoop tools: ** Hue ** O..."

= Kraken Software Overview =

== Software used in Kraken ==

* Zookeeper
* Hadoop
** HDFS: NameNode, DataNode
** YARN: NodeManager, ResourceManager
* Hadoop tools:
** Hue
** Oozie
** Pig, Hive
** HTTPfs, webhdfs
** LDAP integration?
** Kerberos integration?
* Storm
** Nimbus, Supervisors, Bolts, Spouts
** (We will be writing bolts to perform the ETL tasks, and configuring the topology that links them together)
* ETL libraries:
** MaxMind GeoIP database
** dClass: User Agent parser
** Avro: Data serialization framework
** Camus: Kafka -> HDFS library
* Kafka
* Monitoring tools:
** jmxtrans
** Ganglia
** Graphite

See the [[Analytics/Kraken/Overview|Overview]] for additional detail.�

=== Today ===

* <code>udp2log</code> logs from the edge -> awk -> Kafka
* no ETL: cron import to HDFS from Kafka via MR job (this means IPs are in logs!)
* User authorization: via shell account, or group membership in labs LDAP

== Network Profiles ==

=== Network Usage ===

Network usage by node type:

* Kafka Brokers: 2x the full incoming stream (in from edge, out to ETL). Kafka uses its own protocol over TCP, and supports message bundling, compression, and live failover.
* Storm ETL workers: full stream sharded among nodes; Storm is smart about avoiding inter-node communication if a topology can completely fit on a single worker.
* A file in HDFS is made up of blocks; these blocks can be both local and remote to any given DataNode. HDFS replicates data 3x (configurable) for availability and durability – this means it will copy blocks between nodes to maintain the replication invariant. The NameNode acts as the coordinator for replication, and the authority for addressing files to blocks.
* HDFS is immutable (though it allows appends) – this means that "change" rewrites and replicates the entire block (our blocksize is configured at 256MB).
* A MapReduce job has two phases: the Mappers each read their section of the input from HDFS (with Hadoop ensuring the section is on local disk), and perform some work, emitting (key, value) pairs; Reducers then receive these tuples after a total ordering is applied (via the keys) and perform aggregation, each writing its result back to HDFS via the local DataNode. This result is then replicated as normal.
* At the moment, we plan regular data export <code>stat1001</code>, who acts exclusively as a public webserver.

=== Public Surfaces ===

* ACL as outlined in the [https://rt.wikimedia.org/Ticket/Display.html?id=4433 RT ticket].
* Cross-datacenter connectivity (esams) requires a solution (public IP, or perhaps the existing bridge?)

=== Services and Access Needs ===

Many internal service dashboards and control panels:

* Hue: HDFS web access; Hadoop job scheduling (via Oozie); Hive query dashboard (Beeswax)
** Hue WebUI Login authentication uses LDAP
** Can control privileges within the dashboard to granularly restrict access to particular services,
** Limited control can be exerted on resource use
** Dashboard access needed by analysts need access for job monitoring/control, and data access
* Hadoop Admin pages
** NameNode: Provides HDFS logs and system health overview. Cluster administrator access only.
** JobTracker, DataNode: provides logs and debugging output for Hadoop jobs. Access needed by analysts to debug.
* Storm's Nimbus: storm job monitoring and scheduling. Cluster administrator access only.
* Graphite: Application and host monitoring for the cluster. Cluster administrator access only.

Analytics/Kraken

2013-02-28T00:35:13Z

Dsc:

{{Wikimedia engineering project information
| name = Kraken (Analytics Cluster)
| description = A robust, distributed computing and data services platform.
| start = 2011-05-22
| end =
| group = Analytics
| lead =
| team = [[User:Ottomata|Andrew Otto]], [[User:Dsc|David Schoonover]], [[User:Diederik|Diederik van Liere]]
| previous =
| next =
| projectpage = Analytics/Kraken
| display = {{{display|}}}
}}<noinclude>

Kraken is the code-name for the robust distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

= [[{{FULLPAGENAME}}/status|Status]] =

{{Wikimedia project status line|{{FULLPAGENAME}}}}

= Rationale =

The Wiki Movement has a chronic need for analytics. We need it to understand our editors, to encourage growth, to engender diversity, to focus our resources, to improve our engineering efforts, and to measure our success. It permeates nearly all our goals, yet our current analytics capabilities are underdeveloped: we lack infrastructure to capture editor, visitor, clickstream, and device data in a way that is easily accessible; our efforts are distributed among different departments; our data is fragmented over different systems and databases; our tools are ad-hoc.

Rather than merely improve existing jobs and data pipelines, the Analytics Team aims to construct a Data Services Platform capable of mining intelligence from all datastreams of interest, providing this insight in real time, and exposing it via an API to power applications, mash up into websites, and stream to devices.

= Documentation =

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

* [[Analytics/Kraken/Access|Getting Access]]
* [[Analytics/Kraken/Overview|Overview]]
** [[Analytics/Kraken/Overview/Software|Software Overview]]
* [http://upload.wikimedia.org/wikipedia/mediawiki/1/1b/Kraken_flow_diagram-2.png Cluster Dataflow Diagram]
* '''Data'''
** [[Analytics/Product_Codes|Product Codes]]
** [[Analytics/Kraken/Data_Streams|Data Streams]]
** [[Analytics/Kraken/Data_Formats|Data Formats]]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
* [[Analytics/Kraken/Request_Logging|Request Logging]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* '''Notes'''
** [[Analytics/Kraken/Security_Review_Meeting|Security Review Meeting notes]]
** [http://etherpad.wikimedia.org/AnalyticsKrakenTestClusterSetup Test Cluster Setup Notes]
** [http://etherpad.wikimedia.org/AnalyticsHadoopSetupNotes Hadoop Setup Notes]
** [[Analytics/Kraken/Blurbs|Blurbs About Kraken]]
** [https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdEtRLVNoQWNvQzNleHBtTXR5emI3Z2c&pli=1#gid=5 Kraken Benchmarks]

= Components =

== [[Analytics/Kraken/Pixel_Service|Pixel Service]] and [[Analytics/Kraken/Request_Logging|Request Logging]] ==

Public data import endpoint, and system for capturing the incoming firehose from our front-end servers.

* [[Analytics/Kraken/Request_Logging|Request Logging]]
** [[Analytics/Kraken/Request_Logging|System Recommendation]]
** [[Analytics/Kraken/Logging_Solutions_Overview|Distributed Logging Systems Research]]
** [https://docs.google.com/spreadsheet/pub?key=0AvpRkIqSY9hNdFNoYWFIX29hMnFuenV1ckRvYzEzUVE&gid=3 Feature Comparison Spreadsheet]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]

'''Tasks'''

* Work with Ori, Patrick, & ops to figure out Kafka producer situation for production [otto, dsc, ori, preilly, ops]
* MediaWiki EventLogging integration

== [[Analytics/Kraken/ETL|ETL Topology]] ==

The [http://storm-project.net Storm] topology that processes incoming data (aka, "Extract-Transform-Load" in data warehousing jargon).

* [[Analytics/Kraken/ETL|ETL Topology]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Maven + Nexus setup w/ storm pom [dsc]
* Storm Bolts:
** Consume from Kafka (by topic?)
** Serialize & canonicalize record using Avro schema
** GeoIP annotation
** Mobile carrier annotation (by IP)
** IP anonymization (where does the salt live? how often should we change it?)
** Append record to per-hour files in HDFS (path using timestamp, topic)
** Notifier Bolt to update Kafka consumer checkpoint (and/or publish event for external hooks)
* Define Avro schemas:
** WebRequest
** EventData
*** Figure out convention for EventData packets to self-identify with a more specific schema

== [[Analytics/Kraken/Core_Jobs|Core Jobs]] ==

Core data processing jobs for processing web requests and event data, maintained by the Analytics team.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Figure out request tagging -- how can we avoid mutating records?
* Reportcard Oozie workflow [DvL]
** Pig Macro for "typical request analytics"
* Figure out indexing scheme:
** Aggregated time buckets
** Referrer chains
** Event aggregations
** Funnel Analysis
*** A/B testing event data

== [[Analytics/Kraken/Data_Tools|Data Tools]] ==

Data processing library and toolkit provided & maintained by the Analytics team, esp for use with Kraken.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Setup regular sqoop imports [DvL]
* Avro De/Serialization storage format compatibility:
** WebRequest
** EventData
* Build re-usable pig/hive library, outsource this to analysts as much as possible [DvL]
* KV-Pairs parsing tools
* Conversion funnel analysis tools
** A/B testing analysis tools
* MW EventLogging extension integration
* Investigate Ori's Mediawiki extension for instrumentation, sending event/edit data

== [[Analytics/Kraken/Infrastructure|Infrastructure]] ==

Bucket for general cluster infrastructure and maintenance tasks.
* [[Analytics/Kraken/Puppet|Kraken Puppetization]]
* [[Analytics/Kraken/JMX Ports|JMX Ports]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* [[Analytics/Kraken/JMX Monitoring|JMX Monitoring Research]]

'''Tasks'''

* Fix the one busted Cisco machine (an07) [otto]
* Continue puppetization [otto]
* JMX Monitoring [otto, dsc]
* Investigate / experiment / benchmark Hadoop stack [DvL]
* Data owner services -- export, dashboard, visualizations (on Hue?) [dsc, dan]
** Hue plugin for Limn integration? [dsc]
* Get analysts experimenting with Hive/Pig [DvL]
* General purpose job-runner using Kafka+Storm
* Setup Hadoop FairScheduler [DvL]

= Research =

== Bundling ==

Investigate size of savings of structure sharing via a short-term (between 1-30sec window) queue.

* Reorder within window by timestamp? Seq number?
* Parameters of experiment:
** Buffer time: 1s, 2s, 5s, 10s
** Sharded by server
** Bundle by IP, UA, IP+UA
* Pull-up fields for structure sharing:
** Invariant: ip, ua, accept-lang, carrier
** Often invariant: method, status, referer

== Software ==
* Hbase
* Hive
* Hue plugins
* Cascading -- do we even need this as we have Oozie?

[[Category:WMF Projects]]
[[Category:WMF Projects/Analytics]]

</noinclude>

Analytics/Roadmap/PlanningMeetings/2013 Scrum

2013-02-27T19:39:49Z

Dsc:

Team Analytics scrum notes for 2013.

= February 2013 =

== Feb 25 ==

''Secretary: Otto''

* David
** Merged viewport stuff into develop.
** Fixed bugs caused by merge (scatterplot mode, scaling, widget for fiddling with scale type).
** Going to work on metric definition
** Need to review canvas refactoring with Dan. (Geo-json, etc.)

* Otto
** Kafka puppet module reviewed and +1 by Faidon.
** Need review of Kafka Debianization
** Spent most of Friday working on git submodule puppet module merge wrapper, going to just diff files :/

* Kraig
** Friday's meetings
** getting together an agenda / format for next week's workshop on Monday and Tuesday
** going to review this with Erik + Gang.
** Following up on experiments (Mingle, improvement work)
** Consolodate findings with stakeholder groups.

* Diederik
** Gave a lightning talk about Kraken and Limn
** Refactored Geocoding + oozie + pig (YAY|!)
** Migrated only tasks we are currently working on into Mingle.
** Worked on Architecture review document
** Want to put together an overview of use cases of Kraken

* Evan
** Messed around with Category Tree
** Trying to construct control set of articles for plagerism detection.
** Been working on Mediawiki -> git script.

* Dan
** Looked at David's changes for viewport canvas
** lots of meetings
** Going to finish tasks leftover from last sprint
** Want to realign as team to shoot for end of March to complete promises.

== Feb 26 ==

''Secretary: David''

* Otto
** Ops week! I am hanging out with Opsen!
** Wrote an Oozie tutorial as a braindump
** Wrapper script wrapping puppetmaster submodule creation and update. For submodules to be kosher, there needs to be a way to manually inspect the diff of the update. Submodules don't make that easy.
*** Bound the work at today -- not done? Weep & give up.

* Dan
** Read through David's Viewport commits and merge
** Fixed some bugs that resulted from that

* David
** Merged feature/viewport into develop
** Fixed ~11 bugs surfaced after the merge (ty to Dan for helping hunt)
** Today: Close the last bugs in in the Viewport refactor
** Add MetricDef Id to Datasource Columns

* Diederik
** Yesterday? Blocked! Going to Starve! due to the fact that this paycheck will arrive in TO only after I am in SF!
** Ported more Asana tasks to Mingle
** Spoke with Kraig about the architecture review meeting on Friday
** Started building work breakdown tree in mingle -- https://mingle.corp.wikimedia.org/projects/analytics/cards/tree?expands=81&tab=All&tree_name=Work+Breakdown+Structure

* Kraig
** Fleshed out Improvement Ideas in Mingle
** Setting up Mingle with Diederik
** Weekly Analytics Sync with Erik M, RobLa (Gayle out)
** Planned out Analytics Reboot Workshop (Mon/Tues) -- will be sharing Google Doc soon
** Continuing to plan Reboot (esp timing)
** Prep for Arch Review meeting (Diederik)

== Feb 27 ==

''Secretary: Kraig''

* Otto
** In ops meeting on event logging, ops seems to be OK with it
** Worked throughout meetings, got general approval from Faidon on deb, need Mark's review
** Trying to get kafka impl reviewed, told to use git build-package standards
** Making sure geocoding works in oozie
** Administration stuff from Ryan F
** Work out deploying Limn without running npm on target

* Dan
** (travel)

* David
** Fixed some bugs in Limn, rebooted Limn instances on Kripke
*** Permissions were wrong for /srv, causing Limn to die when trying to write browserify cache file (might have been during deploy script testing)
*** No monitoring on other instances on the same target server post-deploy -- deployer could validate, or could be part of process
** Today
*** Close out Limn Viewport refactoring task, including whatever Dan has for it
*** Start looking at creating a software Ecosystem Map for our projects & tech

* Diederik
** Hashed out more arch review meeting notes
** Want to work with otto on pros/cons in notes
** W0 Pig script

* Evan
** onboarding new 2 contractors

* Kraig
** Planning Scheduling for analytics reboot workshop
** getting into more Mingle stuff, working with views etc.
*** Good to have a visual plan when discussing plans (diagrams, visuals, workflow maps, etc.)
** Today spending time on elaborating improvement ideas.
** Today's improvement meeting.

Analytics/Roadmap/PlanningMeetings/2013 Scrum

2013-02-27T19:39:24Z

Dsc: feb 27

Team Analytics scrum notes for 2013.

= February 2013 =

== Feb 25 ==

''Secretary: Otto''

* David
** Merged viewport stuff into develop.
** Fixed bugs caused by merge (scatterplot mode, scaling, widget for fiddling with scale type).
** Going to work on metric definition
** Need to review canvas refactoring with Dan. (Geo-json, etc.)

* Otto
** Kafka puppet module reviewed and +1 by Faidon.
** Need review of Kafka Debianization
** Spent most of Friday working on git submodule puppet module merge wrapper, going to just diff files :/

* Kraig
** Friday's meetings
** getting together an agenda / format for next week's workshop on Monday and Tuesday
** going to review this with Erik + Gang.
** Following up on experiments (Mingle, improvement work)
** Consolodate findings with stakeholder groups.

* Diederik
** Gave a lightning talk about Kraken and Limn
** Refactored Geocoding + oozie + pig (YAY|!)
** Migrated only tasks we are currently working on into Mingle.
** Worked on Architecture review document
** Want to put together an overview of use cases of Kraken

* Evan
** Messed around with Category Tree
** Trying to construct control set of articles for plagerism detection.
** Been working on Mediawiki -> git script.

* Dan
** Looked at David's changes for viewport canvas
** lots of meetings
** Going to finish tasks leftover from last sprint
** Want to realign as team to shoot for end of March to complete promises.

== Feb 26 ==

''Secretary: David''

* Otto
** Ops week! I am hanging out with Opsen!
** Wrote an Oozie tutorial as a braindump
** Wrapper script wrapping puppetmaster submodule creation and update. For submodules to be kosher, there needs to be a way to manually inspect the diff of the update. Submodules don't make that easy.
*** Bound the work at today -- not done? Weep & give up.

* Dan
** Read through David's Viewport commits and merge
** Fixed some bugs that resulted from that

* David
** Merged feature/viewport into develop
** Fixed ~11 bugs surfaced after the merge (ty to Dan for helping hunt)
** Today: Close the last bugs in in the Viewport refactor
** Add MetricDef Id to Datasource Columns

* Diederik
** Yesterday? Blocked! Going to Starve! due to the fact that this paycheck will arrive in TO only after I am in SF!
** Ported more Asana tasks to Mingle
** Spoke with Kraig about the architecture review meeting on Friday
** Started building work breakdown tree in mingle -- https://mingle.corp.wikimedia.org/projects/analytics/cards/tree?expands=81&tab=All&tree_name=Work+Breakdown+Structure

* Kraig
** Fleshed out Improvement Ideas in Mingle
** Setting up Mingle with Diederik
** Weekly Analytics Sync with Erik M, RobLa (Gayle out)
** Planned out Analytics Reboot Workshop (Mon/Tues) -- will be sharing Google Doc soon
** Continuing to plan Reboot (esp timing)
** Prep for Arch Review meeting (Diederik)

== Feb 27 ==

* Otto
** In ops meeting on event logging, ops seems to be OK with it
** Worked throughout meetings, got general approval from Faidon on deb, need Mark's review
** Trying to get kafka impl reviewed, told to use git build-package standards
** Making sure geocoding works in oozie
** Administration stuff from Ryan F
** Work out deploying Limn without running npm on target

* Dan
** (travel)

* David
** Fixed some bugs in Limn, rebooted Limn instances on Kripke
*** Permissions were wrong for /srv, causing Limn to die when trying to write browserify cache file (might have been during deploy script testing)
*** No monitoring on other instances on the same target server post-deploy -- deployer could validate, or could be part of process
** Today
*** Close out Limn Viewport refactoring task, including whatever Dan has for it
*** Start looking at creating a software Ecosystem Map for our projects & tech

* Diederik
** Hashed out more arch review meeting notes
** Want to work with otto on pros/cons in notes
** W0 Pig script

* Evan
** onboarding new 2 contractors

* Kraig
** Planning Scheduling for analytics reboot workshop
** getting into more Mingle stuff, working with views etc.
*** Good to have a visual plan when discussing plans (diagrams, visuals, workflow maps, etc.)
** Today spending time on elaborating improvement ideas.
** Today's improvement meeting.

Analytics/Roadmap/PlanningMeetings/2013 Scrum

2013-02-26T18:42:29Z

Dsc:

Analytics/Roadmap/PlanningMeetings/2013 Scrum

2013-02-26T18:41:02Z

Dsc: Created page with "Team Analytics scrum notes for 2013. = February 2013 = == Feb 25 == * David ** Merged viewport stuff into develop. ** Fixed bugs caused by merge (scatterplot mode, scaling,..."

Team Analytics scrum notes for 2013.

= February 2013 =

== Feb 25 ==

* David
** Merged viewport stuff into develop.
** Fixed bugs caused by merge (scatterplot mode, scaling, widget for fiddling with scale type).
** Going to work on metric definition
** Need to review canvas refactoring with Dan. (Geo-json, etc.)

* Otto
** Kafka puppet module reviewed and +1 by Faidon.
** Need review of Kafka Debianization
** Spent most of Friday working on git submodule puppet module merge wrapper, going to just diff files :/

* Kraig
** Friday's meetings
** getting together an agenda / format for next week's workshop on Monday and Tuesday
** going to review this with Erik + Gang.
** Following up on experiments (Mingle, improvement work)
** Consolodate findings with stakeholder groups.

* Diederik
** Gave a lightning talk about Kraken and Limn
** Refactored Geocoding + oozie + pig (YAY|!)
** Migrated only tasks we are currently working on into Mingle.
** Worked on Architecture review document
** Want to put together an overview of use cases of Kraken

* Evan
** Messed around with Category Tree
** Trying to construct control set of articles for plagerism detection.
** Been working on Mediawiki -> git script.

* Dan
** Looked at David's changes for viewport canvas
** lots of meetings
** Going to finish tasks leftover from last sprint
** Want to realign as team to shoot for end of March to complete promises.

== Feb 26 ==

* Otto
** Ops week! I am hanging out with Opsen!
** Wrote an Oozie tutorial as a braindump
** Wrapper script wrapping puppetmaster submodule creation and update. For submodules to be kosher, there needs to be a way to manually inspect the diff of the update. Submodules don't make that easy.
*** Bound the work at today -- not done? Weep & give up.

* Dan
** Read through David's Viewport commits and merge
** Fixed some bugs that resulted from that

* David
** Merged feature/viewport into develop
** Fixed ~11 bugs surfaced after the merge (ty to Dan for helping hunt)
** Today: Close the last bugs in in the Viewport refactor
** Add MetricDef Id to Datasource Columns

* Diederik
** Yesterday? Blocked! Going to Starve! due to the fact that this paycheck will arrive in TO only after I am in SF!
** Ported more Asana tasks to Mingle
** Spoke with Kraig about the architecture review meeting on Friday
** Started building work breakdown tree in mingle -- https://mingle.corp.wikimedia.org/projects/analytics/cards/tree?expands=81&tab=All&tree_name=Work+Breakdown+Structure

* Kraig
** Fleshed out Improvement Ideas in Mingle
** Setting up Mingle with Diederik
** Weekly Analytics Sync with Erik M, RobLa (Gayle out)
** Planned out Analytics Reboot Workshop (Mon/Tues) -- will be sharing Google Doc soon
** Continuing to plan Reboot (esp timing)
** Prep for Arch Review meeting (Diederik)

Analytics/Roadmap/PlanningMeetings

2013-02-26T18:39:26Z

Dsc: Adds scrum notes.

Analytics/Visualization, Reporting & Applications/Ideas

2013-02-19T21:28:19Z

Dsc: Created page with " == Data Repositories == * CKAN http://ckan.org/ -- http://docs.ckan.org/en/ckan-1.8/ === Gov-Related === * FEC filings from the last few decades: http://www.fec.gov/data/ ..."

== Data Repositories ==

* CKAN http://ckan.org/ -- http://docs.ckan.org/en/ckan-1.8/

=== Gov-Related ===

* FEC filings from the last few decades: http://www.fec.gov/data/
* Bills and legislators: http://www.govtrack.us/
* Lots of data, but hard to access: http://votesmart.org/
* A nice tool/repository for extracting and sharing data: https://scraperwiki.com/

Analytics/Archive/Infrastructure/Overview

2013-02-13T22:25:25Z

Dsc:

= Glossary =

;[[Analytics/Kraken|Kraken]]
: Kraken is the code-name for the distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

;[http://hadoop.apache.org/ Hadoop]
: A collection of services for batch processing of large data. Core concepts are a distributed filesystem (HDFS) and MapReduce.

;HDFS
: Hadoop Distributed Filesystem.

;[http://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-yarn/hadoop-yarn-site/YARN.html YARN]
: Yet-Another-Resource-Negotiator. Compute resource manager and API for distributed applications. MapReduce (v2) is a YARN application. Although not technically correct, YARN is sometimes referred to as MRv2.

;[http://en.wikipedia.org/wiki/MapReduce MapReduce]
: Programming model for parallelizing batch processing of large data sets. Really good for counting things. Hadoop ships with a Java implementation of MapReduce.

;[http://cloudera.github.com/hue/ Hue]
: Hadoop User Experience. Web GUI for various Hadoop services. (HDFS, Oozie, Pig, Hive, etc.) Written in Python Django.

;[http://pig.apache.org/ Pig]
: High level language abstraction for common implementing MapReduce programs without thinking about the MapReduce model. (Feels like a mix between SQL and awk). Generates MapReduce programs that are run in Hadoop.

;[http://hive.apache.org/ Hive]
: Projects structure onto flat data (text or binary) in HDFS and allows this data to be queried in an SQL like manner. Analytics is not currently using Hive, although believe it will be useful for certain types of analysis as more clients have need.

;[http://oozie.apache.org/ Oozie]
: Scheduler for Hadoop jobs. Used for automated reporting based on data availability.

;[http://storm-project.net/ Storm]
: Distributed real-time stream processing system. Useful for transformation and computation of streaming data.

;[http://kafka.apache.org/ Kafka]
: Distributed pub/sub message queue. Useful as a big ol' reliable log buffer.

;[http://zookeeper.apache.org/ Zookeeper]
: Highly reliable dynamic configuration store. Instead of keeping configuration and simple state in flat files or a database, Zookeeper allows applications to be notified of configuration changes on the fly.

;[http://www.cloudera.com/content/cloudera/en/about.html Cloudera]
: An organization that provides Hadoop ecosystem packages and documentation. Kraken uses Cloudera's [https://ccp.cloudera.com/display/CDH4DOC/CDH4+Documentation CDH4] distribution.

;webrequest log stream (aka firehose)
: The Analytics team has been referring to the web access logs generated from all WMF frontend cache webservers as 'webrequest logs'. The existent UDP stream is often referred to as the 'firehose', or the webrequest log stream.

;event log stream
: bits varnishes are currently configured to send certain request data made to an <code>/event.gif</code> endpoint serviced by the bits Varnish caches. This is intended to be used as a way to get generic application data into Kraken for analysis; the EventLogging extension also uses this endpoint.

;[http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html jmx]
: JMX (Java Management Extensions) is a widely-adopted JVM mechanism to expose VM and application metrics, state, and management features. Nearly all the above-listed technologies provide JMX metrics; for example, Kafka brokers expose BytesIn, BytesOut, FailedFetchRequests, FailedProduceRequests, and MessagesIn aggregated across all topics (as well as individually). Kraken is using [https://github.com/jmxtrans/jmxtrans jmxtrans] to send relevant metrics to Ganglia and elsewhere.

= Architecture =

First! A summary of Kraken's current (as of Feburary 2013) setup. The end of this document will contain a wishlist of the pieces of the architecture that we'd like to change.

Kraken can be used for many other things than processing the webrequest logs but as that is our primary source of big data and this is an overview document, the focus will be on describing the architecture to import, store, and analyze this data stream.

== UDP Webrequest Log Stream ==

Frontend cache servers (varnish, squid, and nginx) are all configured to send webrequest access logs over UDP to 3 destination hosts. These are emery, locke, and oxygen. oxygen is running a socat multicast relay that is consumable by nodes in eqiad.

Analytics' spring 2013 deliverables requires running analysis on unsampled mobile webrequest logs. Node(s) in the analytics cluster run udp2log instances that filter for traffic from the 4 frontend varnish mobile hosts (cp1041-cp1044). udp2log pipes this unsampled data into Kafka shell producers.

== Log Buffering - Kafka ==

Kraken is using [https://github.com/apache/kafka/blob/0.7.2/core/src/main/scala/kafka/tools/ProducerShell.scala kafka-producer-shell.sh] to send data from udp2log into Kafka. The 2 Kafka brokers retain this data on disk for up to a week. It is available for consumption by any Kafka consumer. Kafka consumers specify a byte offset in a particular Kafka topic at which they would like to start reading, as well as a byte limit. A byte limit of -1 means to read until the end of the topic is reached.

Cron jobs running on a single analytics node are configured to ask Zookeeper for existing Kafka topics, and to consume particular topics into HDFS. This is done via [https://github.com/wikimedia-incubator/kafka-hadoop-consumer kafka-hadoop-consumer] a Hadoop MapReduce job that consumes from Kafka and writes to HDFS. kafka-hadoop-consumer saves its per-topic read state in Zookeeper, and uses this to know where it left off after the previous import.

== Batch Analysis - Hadoop ==

Raw imported data is saved in HDFS in the /wmf/raw directory. Files in /wmf/raw are not world readable. Analysis of these logs is done by various [https://github.com/wikimedia/kraken/tree/master/pig Pig scripts] and [https://github.com/wikimedia/kraken/tree/master/oozie Oozie job] scheduling, although we are (as of mid February 2013) working to clean and automate this process. Hue is used extensively to browse data in HDFS and to edit and schedule and monitor Oozie jobs as they run.

Oozie is used to schedule and chain runs of Pig scripts based on the availability of data in HDFS. Workflows are triggered once the data is available, and are configured to provide the Pig scripts with the paths of HDFS files to use for a particular run.

Oozie and Pig are generally configured to save the output of their analysis in /wmf/public directory of HDFS. This directory is world readable and is intended to be available for visualization using Limn, or for analysts to download and do further custom analysis or processing. It could also be used to trigger additional automated Oozie workflows.

= Wishlist =

Oh there are so many hacky pieces of this setup! Here's what we'd like to change:

* We'd love to run Kafka producers on the frontend cache webservers, rather than a single udp2log stream. A single stream means that every NIC must receive every webrequest log byte. Kafka is meant to be dynamically configured via Zookeeper. Scaling it horizontally and automatic failover are built in. If the frontends sent their data directly to Kafka, we could do away with the hacky and inefficient unsampled udp2log -> kafka shell producer setup we have currently.

* The existing ETL phase has got to change. The crappy kafka-hadoop-consumer spawned by a cron job to consume from Kafka topics into HDFS just sucks. Replacement contenders are LinkedIn's [https://github.com/linkedin/camus Camus] and/or Storm.

* We're importing raw webrequest logs, which contain sensitive data. We would like to use Storm topologies to sanitize this data, as well as to do other transformations before the data is stored in HDFS.

Below is the envisioned architecture in diagram form:

= Envisioned Architecture Diagram =

[[File:Kraken_flow_diagram-2.png|800px|Cluster Dataflow Diagram]]

Analytics/Archive/Infrastructure/Overview

2013-02-13T22:22:42Z

Dsc: Adds links & examples for JMX.

= Glossary =

;[[Analytics/Kraken|Kraken]]
: Kraken is the code-name for the distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

;[http://hadoop.apache.org/ Hadoop]
: A collection of services for batch processing of large data. Core concepts are a distributed filesystem (HDFS) and MapReduce.

;HDFS
: Hadoop Distributed Filesystem.

;[http://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-yarn/hadoop-yarn-site/YARN.html YARN]
: Yet-Another-Resource-Negotiator. Compute resource manager and API for distributed applications. MapReduce (v2) is a YARN application. Although not technically correct, YARN is sometimes referred to as MRv2.

;[http://en.wikipedia.org/wiki/MapReduce MapReduce]
: Programming model for parallelizing batch processing of large data sets. Really good for counting things. Hadoop ships with a Java implementation of MapReduce.

;[http://cloudera.github.com/hue/ Hue]
: Hadoop User Experience. Web GUI for various Hadoop services. (HDFS, Oozie, Pig, Hive, etc.) Written in Python Django.

;[http://pig.apache.org/ Pig]
: High level language abstraction for common implementing MapReduce programs without thinking about the MapReduce model. (Feels like a mix between SQL and awk). Generates MapReduce programs that are run in Hadoop.

;[http://hive.apache.org/ Hive]
: Projects structure onto flat data (text or binary) in HDFS and allows this data to be queried in an SQL like manner. Analytics is not currently using Hive, although believe it will be useful for certain types of analysis as more clients have need.

;[http://oozie.apache.org/ Oozie]
: Scheduler for Hadoop jobs. Used for automated reporting based on data availability.

;[http://storm-project.net/ Storm]
: Distributed stream processing system. Useful for transformation and computation of streaming data.

;[http://kafka.apache.org/ Kafka]
: Distributed pub/sub message queue. Useful as a big ol' reliable log buffer.

;[http://zookeeper.apache.org/ Zookeeper]
: Highly reliable dynamic configuration store. Instead of keeping configuration and simple state in flat files or a database, Zookeeper allows applications to be notified of configuration changes on the fly.

;[http://www.cloudera.com/content/cloudera/en/about.html Cloudera]
: An organization that provides Hadoop ecosystem packages and documentation. Kraken uses Cloudera's [https://ccp.cloudera.com/display/CDH4DOC/CDH4+Documentation CDH4] distribution.

;webrequest log stream (aka firehose)
: The Analytics team has been referring to the web access logs generated from all WMF frontend cache webservers as 'webrequest logs'. The existent UDP stream is often referred to as the 'firehose', or the webrequest log stream.

;event log stream
: bits varnishes are currently configured to send certain request data made to an <code>/event.gif</code> endpoint serviced by the bits Varnish caches. This is intended to be used as a way to get generic application data into Kraken for analysis; the EventLogging extension also uses this endpoint.

;[http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html jmx]
: JMX (Java Management Extensions) is a widely-adopted JVM mechanism to expose VM and application metrics, state, and management features. Nearly all the above-listed technologies provide JMX metrics; for example, Kafka brokers expose BytesIn, BytesOut, FailedFetchRequests, FailedProduceRequests, and MessagesIn aggregated across all topics (as well as individually). Kraken is using [https://github.com/jmxtrans/jmxtrans jmxtrans] to send relevant metrics to Ganglia and elsewhere.

= Architecture =

First! A summary of Kraken's current (as of Feburary 2013) setup. The end of this document will contain a wishlist of the pieces of the architecture that we'd like to change.

Kraken can be used for many other things than processing the webrequest logs but as that is our primary source of big data and this is an overview document, the focus will be on describing the architecture to import, store, and analyze this data stream.

== UDP Webrequest Log Stream ==

Frontend cache servers (varnish, squid, and nginx) are all configured to send webrequest access logs over UDP to 3 destination hosts. These are emery, locke, and oxygen. oxygen is running a socat multicast relay that is consumable by nodes in eqiad.

Analytics' spring 2013 deliverables requires running analysis on unsampled mobile webrequest logs. Node(s) in the analytics cluster run udp2log instances that filter for traffic from the 4 frontend varnish mobile hosts (cp1041-cp1044). udp2log pipes this unsampled data into Kafka shell producers.

== Log Buffering - Kafka ==

Kraken is using [https://github.com/apache/kafka/blob/0.7.2/core/src/main/scala/kafka/tools/ProducerShell.scala kafka-producer-shell.sh] to send data from udp2log into Kafka. The 2 Kafka brokers retain this data on disk for up to a week. It is available for consumption by any Kafka consumer. Kafka consumers specify a byte offset in a particular Kafka topic at which they would like to start reading, as well as a byte limit. A byte limit of -1 means to read until the end of the topic is reached.

Cron jobs running on a single analytics node are configured to ask Zookeeper for existing Kafka topics, and to consume particular topics into HDFS. This is done via [https://github.com/wikimedia-incubator/kafka-hadoop-consumer kafka-hadoop-consumer] a Hadoop MapReduce job that consumes from Kafka and writes to HDFS. kafka-hadoop-consumer saves its per-topic read state in Zookeeper, and uses this to know where it left off after the previous import.

== Batch Analysis - Hadoop ==

Raw imported data is saved in HDFS in the /wmf/raw directory. Files in /wmf/raw are not world readable. Analysis of these logs is done by various [https://github.com/wikimedia/kraken/tree/master/pig Pig scripts] and [https://github.com/wikimedia/kraken/tree/master/oozie Oozie job] scheduling, although we are (as of mid February 2013) working to clean and automate this process. Hue is used extensively to browse data in HDFS and to edit and schedule and monitor Oozie jobs as they run.

Oozie is used to schedule and chain runs of Pig scripts based on the availability of data in HDFS. Workflows are triggered once the data is available, and are configured to provide the Pig scripts with the paths of HDFS files to use for a particular run.

Oozie and Pig are generally configured to save the output of their analysis in /wmf/public directory of HDFS. This directory is world readable and is intended to be available for visualization using Limn, or for analysts to download and do further custom analysis or processing. It could also be used to trigger additional automated Oozie workflows.

= Wishlist =

Oh there are so many hacky pieces of this setup! Here's what we'd like to change:

* We'd love to run Kafka producers on the frontend cache webservers, rather than a single udp2log stream. A single stream means that every NIC must receive every webrequest log byte. Kafka is meant to be dynamically configured via Zookeeper. Scaling it horizontally and automatic failover are built in. If the frontends sent their data directly to Kafka, we could do away with the hacky and inefficient unsampled udp2log -> kafka shell producer setup we have currently.

* The existing ETL phase has got to change. The crappy kafka-hadoop-consumer spawned by a cron job to consume from Kafka topics into HDFS just sucks. Replacement contenders are LinkedIn's [https://github.com/linkedin/camus Camus] and/or Storm.

* We're importing raw webrequest logs, which contain sensitive data. We would like to use Storm topologies to sanitize this data, as well as to do other transformations before the data is stored in HDFS.

Below is the envisioned architecture in diagram form:

= Envisioned Architecture Diagram =

[[File:Kraken_flow_diagram-2.png|800px|Cluster Dataflow Diagram]]

Analytics/Archive/Infrastructure/Meetings/SecurityReview

2013-02-13T00:01:15Z

Dsc:

Analytics/Archive/Infrastructure/Meetings/SecurityReview

2013-02-13T00:00:43Z

Dsc: Meeting Notes

= Original Suggested Agenda =

This is an outline of information and topics that might be of interest to discuss. It's not intended as comprehensive, just a starting point. Feel free to add/edit.

== Architecture Overview ==

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

There's been some discussion of swapping this meeting with the Arch Review meeting, so we can work out any questions with the components before discussing their security implications. Either way, we'll probably want a brief overview of the architecture before moving forward.

== Cluster Security ==

=== Public Surfaces ===

* ACL as outlined in the [https://rt.wikimedia.org/Ticket/Display.html?id=4433 RT ticket].
* Cross-datacenter connectivity (esams) requires a solution (public IP, or perhaps the existing bridge?)

=== Services and Access Needs ===

Many internal service dashboards and control panels:

* Hue: HDFS web access; Hadoop job scheduling (via Oozie); Hive query dashboard (Beeswax)
** Hue WebUI Login authentication uses LDAP
** Can control privileges within the dashboard to granularly restrict access to particular services,
** Limited control can be exerted on resource use
** Dashboard access needed by analysts need access for job monitoring/control, and data access
* Hadoop Admin pages
** NameNode: Provides HDFS logs and system health overview. Cluster administrator access only.
** JobTracker, DataNode: provides logs and debugging output for Hadoop jobs. Access needed by analysts to debug.
* Storm's Nimbus: storm job monitoring and scheduling. Cluster administrator access only.
* Graphite: Application and host monitoring for the cluster. Cluster administrator access only.

=== HDFS ===

* http://blog.cloudera.com/blog/2012/03/authorization-and-authentication-in-hadoop/
* <code>hdfs</code> is the superuser within the HDFS file system. Hadoop piggybacks on Unix users/groups, but uses its own protocols for communication, not the shell (ssh). This means being logged in as the <code>hdfs</code> Unix account is sufficient to take actions using the Hadoop shell tools as <code>hdfs</code>, the superuser. Hadoop provides an optional Kerberos layer to provide authentication, but there are other solutions, such as firewalling off the NameNode using iptables to ensure only whitelisted nodes from connecting to the HDFS cluster.

== Data Retention ==

=== What Potentially Private or Sensitive Data is there? ===

* IP addresses
* Browsing activity (including edits) correlated with Usernames & IPs
* Data leakage via search queries / referrers
* Low-entropy Data:
** Precise geographic information about all activity
** Login/Logout by IP across all web properties

=== Raw Logs ===

* First persistantly stored in Kafka brokers; buffer window is 7 days ([http://kafka.apache.org/configuration.html log.retention.hours]=168), and automatically deleted afterward
* ETL anonymization pipeline sees raw data, removes IPs (salted hash) after geo lookup

=== Policies ===

* Never publish raw logs or unaggregated datasets
* Raw logs only accessible by those who have signed the Data NDA.
** Can control HDFS file permissions via LDAP groups, allowing segregation of NDA/non-NDA data access, private data import, etc

== Legacy ==

This pertains only to the current, legacy system, but raw archival logs currently exist back to 2011 containing unsanitized IPs.

* '''Something must be done about this!''' GeoIP then hash the IPs, and replace them in the logs! Offsite the files if we '''really''' need backups?
* wikistats processes all of time at once, and frequently has errors that require recomputation

= Meeting Notes =

quick summary of notes / take-aways from the Analytics (Kraken) security review meeting.

* analytics1001 has been wiped and reimaged (restoring /home from backup)
* All proxies and externally-facing services have been disabled.
* Work is under way to bring everything that was puppetized under the analytics1001 puppetmaster into operations-puppet after proper review. Andrew is working closely with a number of people in ops to make this happen.
* All future deployments to the cluster will be puppetized and go through normal code review. Other than performance testing, these puppet confs will be tested in labs.
* The rest of the cluster will be wiped and reimaged out of puppet; data in HDFS will be preserved. This can be a rolling process allowing work to proceed while its under way.
* Schedule an Architectural Review meeting sometime during the SF Ops hackathon, including a look at additional services and auth methods that provide access to internal dashboards like Hue &such.
* Ensure all current "application" code (stuff written by WMF) gets reviewed:
** Cron doing HDFS import from Kafka
** Pig UDFs and other data tools used in processing
** Future: Storm ETL layer

We all agreed the overall goal is to get to an acceptable security state. During that process, the Analytics team still needs to continue to meet stakeholder needs and deliver on promises. We decided on keeping running a "minimum viable cluster" while reimaging boxes and civilizing cluster configuration:

* Wall off some portion of the boxes to continue recieving data and running jobs; all other boxes can be wiped (preserving HDFS partitions). Boxes would be incrementally removed from the "unsanitary" cluster, reimaged, and then added to the "sanitary" cluster. Stupid bathroom-related jokes to be avoided.
* Team Analytics to enumerate data processing jobs that will be running in the intermediate period; their configurations and tooling will be reviewed.
* Analytics and Ops engineers continue to have shell access. Jobs can be submitted and managed using the CLI tools; internal dashboards can be accessed via SSH tunnelling. Analysts working on the cluster will be approved for shell access on a case-by-case basis (afaik, just Evan Rosen (full-time analyst for Grantmaking & Programs), and Stefan Petrea (contractor for Analytics)).
* No public, external access of any box in either zone (including proxied, dashboards like Hue, or even static files) that hasn't gone through review.
* Analytics and Ops will work together to find a simple, acceptable mechanism for data export.

== Next Steps ==

* Analytics puppet manfiests fully reviewed and merged into master operations-puppet repository
** Andrew to come pow-wow before the SF Ops hackathon and buddy it up with ops to plow through some of this.
* Schedule Architecture Review
* Rolling reimaging of all analytics boxes (including hadoop data nodes but preserving data) implementing this "minimal viable cluster" plan.

Analytics/Kraken

2013-02-12T19:39:24Z

Dsc:

{{Wikimedia engineering project information
| name = Kraken (Analytics Cluster)
| description = A robust, distributed computing and data services platform.
| start = 2011-05-22
| end =
| group = Analytics
| lead =
| team = [[User:Ottomata|Andrew Otto]], [[User:Dsc|David Schoonover]], [[User:Diederik|Diederik van Liere]]
| previous =
| next =
| projectpage = Analytics/Kraken
| display = {{{display|}}}
}}<noinclude>

Kraken is the code-name for the robust distributed computing and data-services platform under construction by the Wikimedia Analytics Team.

= [[{{FULLPAGENAME}}/status|Status]] =

{{Wikimedia project status line|{{FULLPAGENAME}}}}

= Rationale =

The Wiki Movement has a chronic need for analytics. We need it to understand our editors, to encourage growth, to engender diversity, to focus our resources, to improve our engineering efforts, and to measure our success. It permeates nearly all our goals, yet our current analytics capabilities are underdeveloped: we lack infrastructure to capture editor, visitor, clickstream, and device data in a way that is easily accessible; our efforts are distributed among different departments; our data is fragmented over different systems and databases; our tools are ad-hoc.

Rather than merely improve existing jobs and data pipelines, the Analytics Team aims to construct a Data Services Platform capable of mining intelligence from all datastreams of interest, providing this insight in real time, and exposing it via an API to power applications, mash up into websites, and stream to devices.

= Documentation =

[[File:Kraken_flow_diagram-2.png|320px|right|Cluster Dataflow Diagram]]

* [[Analytics/Kraken/Access|Getting Access]]
* [[Analytics/Kraken/Overview|Overview]]
* [http://upload.wikimedia.org/wikipedia/mediawiki/1/1b/Kraken_flow_diagram-2.png Cluster Dataflow Diagram]
* '''Data'''
** [[Analytics/Product_Codes|Product Codes]]
** [[Analytics/Kraken/Data_Streams|Data Streams]]
** [[Analytics/Kraken/Data_Formats|Data Formats]]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]
* [[Analytics/Kraken/Request_Logging|Request Logging]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* '''Notes'''
** [[Analytics/Kraken/Security_Review_Meeting|Security Review Meeting notes]]
** [http://etherpad.wikimedia.org/AnalyticsKrakenTestClusterSetup Test Cluster Setup Notes]
** [http://etherpad.wikimedia.org/AnalyticsHadoopSetupNotes Hadoop Setup Notes]
** [[Analytics/Kraken/Blurbs|Blurbs About Kraken]]
** [https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdEtRLVNoQWNvQzNleHBtTXR5emI3Z2c&pli=1#gid=5 Kraken Benchmarks]

= Components =

== [[Analytics/Kraken/Pixel_Service|Pixel Service]] and [[Analytics/Kraken/Request_Logging|Request Logging]] ==

Public data import endpoint, and system for capturing the incoming firehose from our front-end servers.

* [[Analytics/Kraken/Request_Logging|Request Logging]]
** [[Analytics/Kraken/Request_Logging|System Recommendation]]
** [[Analytics/Kraken/Logging_Solutions_Overview|Distributed Logging Systems Research]]
** [https://docs.google.com/spreadsheet/pub?key=0AvpRkIqSY9hNdFNoYWFIX29hMnFuenV1ckRvYzEzUVE&gid=3 Feature Comparison Spreadsheet]
* [[Analytics/Kraken/Pixel_Service|Pixel Service Endpoint]]

'''Tasks'''

* Work with Ori, Patrick, & ops to figure out Kafka producer situation for production [otto, dsc, ori, preilly, ops]
* MediaWiki EventLogging integration

== [[Analytics/Kraken/ETL|ETL Topology]] ==

The [http://storm-project.net Storm] topology that processes incoming data (aka, "Extract-Transform-Load" in data warehousing jargon).

* [[Analytics/Kraken/ETL|ETL Topology]]
* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Maven + Nexus setup w/ storm pom [dsc]
* Storm Bolts:
** Consume from Kafka (by topic?)
** Serialize & canonicalize record using Avro schema
** GeoIP annotation
** Mobile carrier annotation (by IP)
** IP anonymization (where does the salt live? how often should we change it?)
** Append record to per-hour files in HDFS (path using timestamp, topic)
** Notifier Bolt to update Kafka consumer checkpoint (and/or publish event for external hooks)
* Define Avro schemas:
** WebRequest
** EventData
*** Figure out convention for EventData packets to self-identify with a more specific schema

== [[Analytics/Kraken/Core_Jobs|Core Jobs]] ==

Core data processing jobs for processing web requests and event data, maintained by the Analytics team.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Figure out request tagging -- how can we avoid mutating records?
* Reportcard Oozie workflow [DvL]
** Pig Macro for "typical request analytics"
* Figure out indexing scheme:
** Aggregated time buckets
** Referrer chains
** Event aggregations
** Funnel Analysis
*** A/B testing event data

== [[Analytics/Kraken/Data_Tools|Data Tools]] ==

Data processing library and toolkit provided & maintained by the Analytics team, esp for use with Kraken.

* [[Analytics/Kraken/Data_Formats|Data Formats]]

'''Tasks'''

* Setup regular sqoop imports [DvL]
* Avro De/Serialization storage format compatibility:
** WebRequest
** EventData
* Build re-usable pig/hive library, outsource this to analysts as much as possible [DvL]
* KV-Pairs parsing tools
* Conversion funnel analysis tools
** A/B testing analysis tools
* MW EventLogging extension integration
* Investigate Ori's Mediawiki extension for instrumentation, sending event/edit data

== [[Analytics/Kraken/Infrastructure|Infrastructure]] ==

Bucket for general cluster infrastructure and maintenance tasks.
* [[Analytics/Kraken/Puppet|Kraken Puppetization]]
* [[Analytics/Kraken/JMX Ports|JMX Ports]]
* [[Analytics/2012-2013_Roadmap/Hardware|Hardware Planning]]
* [[Analytics/Kraken/JMX Monitoring|JMX Monitoring Research]]

'''Tasks'''

* Fix the one busted Cisco machine (an07) [otto]
* Continue puppetization [otto]
* JMX Monitoring [otto, dsc]
* Investigate / experiment / benchmark Hadoop stack [DvL]
* Data owner services -- export, dashboard, visualizations (on Hue?) [dsc, dan]
** Hue plugin for Limn integration? [dsc]
* Get analysts experimenting with Hive/Pig [DvL]
* General purpose job-runner using Kafka+Storm
* Setup Hadoop FairScheduler [DvL]

= Research =

== Bundling ==

Investigate size of savings of structure sharing via a short-term (between 1-30sec window) queue.

* Reorder within window by timestamp? Seq number?
* Parameters of experiment:
** Buffer time: 1s, 2s, 5s, 10s
** Sharded by server
** Bundle by IP, UA, IP+UA
* Pull-up fields for structure sharing:
** Invariant: ip, ua, accept-lang, carrier
** Often invariant: method, status, referer

== Software ==
* Hbase
* Hive
* Hue plugins
* Cascading -- do we even need this as we have Oozie?

[[Category:WMF Projects]]
[[Category:WMF Projects/Analytics]]

</noinclude>