WMHACK 17 – what can possibly happen in 3 days

Long story short – I am just back from the 2017 edition of Wikimedia Hackathon, and this time I got things to say. Keeping it a bit different from previous posts on hackathons, I am going to keep this short and steady so that you do not get bored too much. Cutting drama is something which I generally dont do, but this one really needs it.

More media of the event at: Wikimedia Commons

Wikimedia Hackathons are always a reminder of how little I know about the Mediawiki software, and working live with these dinosaur-mw-mentors are more than fun. You might wonder why the Foundation should fly us to a place when you can get the same done over IRC/any other medium. I can tell you why as I complete this post.

Background: We had a Google Summer of Code 2015 project to build and deploy a newsletter extension for Mediawiki – which never hit production yet. We were trying all our lucks at Wikimania hackathons, other summits etc – but we never were close enough. I do not blame this on anybody as for eg, we were almost in beta (one step before prod after WMHACK16, but Wikimania hackathon 2016 saw a huge change of codebase (shift to contenthandler), and we were back at level 0. It had always been a dream of a few community members like Quim Gil, Me and couple of others to get this thing to production – and we were trying hard with little luck.

My aim for the hackathon: Simple, but to get all security review blockers of Newsletter extension merged and get it deployable in production.

Day 0: Arrival, and other fun

Flying from berlin on an early hour flight – this went pretty smooth and I was a bit sleep deprived as well. Sadly I couldn’t get to my room until 13:00 – but I took onto working a bit of my things on my Igalia project – and killed the time. Later that night I remember even having a short dinner with Srishti Seth and spending the rest of the night mingling with various other mentors, etc. We even had some preparatory meetings – how good.

Day 1: Mentor-newcomer program

This year we had a new test of a mentoring program by the Hackathon organizers, and we had kind of like a pitching session and breakout session to induct in newcomers to projects. I had a whole list of newcomer tasks which were pitched to the newcomers as well. We created a couple of Telegram groups to co-ordinate, and I would say that the whole thing went smooth.

Day 2-3: What was that ?

I clubbed day 2 and 3 together as it went super fast and super hacky. I found out Brian Wolff who was lurking around fixing things here and there and made him sit down with the newsletter extension hacks. This turned out pretty well, as he was happy to help and that went all the way from Day 2, 10 o clock to Day 2 night 21:00. WOW. Some funny things which happened (mostly exaggerated, technically should sound fun):

  1. At one point – I was literally dictated on writing a new maintenance script by Brian (no laptop) and Florian (no laptop) together. This was equally embarrassing and informative.
  2. Srishti was there as well, and she fixed one of her first patch sets to Gerrit, which was for her embarrassment – adding an @author tag to one of the newsletter extension classes (I’m kidding here, as my first patch was a spelling mistake).
  3. Around 20:00 Nemo Bis tells me I should contribute to Malayalam translations via the Unicode web app and I get to vote on translations.
  4.  MtDu (a past GCI student) was there as well with us on the table – and he seemed to be enjoying me being dictated as well. He fixed a couple of other bugs for us – kudos for him.
  5. At around 22:00, Brian went up to take his laptop (that would mean that sleep was cancelled for the night for the table), but then at around 22:15, Florian made a statement that made a huge impact. It was something like “The party happens only tonight, and you can fix the code tomorrow too”. I remember packing back the machines, and even pulling Srishti (who apparently pulled in Andre) and what not – we had a huge team checking out the Hackathon party. I blame Florian, and almost felt like adding “Hackathon party” as a blocker for “Newsletter deployment to production”. Kidding again, but that was fun.
  6. On the way to hackathon party we rate ourselves on a 100 (work:show instead of Howard Stern 10). I am rated 60:40 for work:show while Florian on a 90:100 and Brian on a 110:-10. (more kidding, Florian had the worse ratings).

Day 3: We still have things left

We were closing down on things, and apparently Brian and me were stuck on a huge performance issue with the Newsletter tablepager listing. To make it less technical, the current query as he mentions “should be killed with fire as it can fry the database as it uses filesort and we wanted something which uses INDEXes or something cheaper”. Technically we started on this huge thing on Day 2 around 22:10, and this stayed with us all the way till EOD. Main improvements:

  1. 13:00 – we have most of the cases working, but we still havent got anywhere with our blocker patch
  2. <Missed the group photo somewhere in the middle> as we almost thought we were almost there.
  3. 15:00 – the showcase has started, and we were trying our best (mostly Brian) but we couldn’t showcase the whole thing as done.
  4. 17:00 – showcase finished, we find a better room with cool wind coming in to work on it and by around 18:45 – we have it fixed. WOW. It was such a moment to see the thing working and Brian explained his amusement with a weary ‘yaay’ (it was literally just that long).

and yay – we have all blockers resolved for the Newsletter extension security review! and a handful finished from our huge list of tiny tasks at T159081 as well.

Stats for the nerds:

  1. Gerrit changesets submitted during the hackathon for the newsletter extension.

Key Learnings

  • Hackathons work, as you get to make the mentors sit down with you patiently to get the community things done.
  • Telegram chat groups work, but mostly needs to be communicated a bit earlier. This happened during wmhack17, and happy for that.
  • Party on the second night can delay deployments – but luckily I pulled out before 1:00 am on the third morning.

Some funny chats during the hackathon:

Alright. So thats it, Vienna was hot, clean and great! Thanks to the amazing organizers again.

Note: Even though I was a signed in mentor for the mentor-newcomer task, I mostly became a newcomer from day 2 with these people. I tried to recruit in people to work on the rest of the tasks via Telegram, which brought in Srishti and MtDu to fixing things though.

Few updates after the blog was published.

“MtDu was the one who convinced me to go to the party with his SQL argument (If you LEFT JOIN the hackathon to the party, today the party has value; tomorrow it IS NULL). Although upon actually arriving I found my favourite anti-social computer nerd in a corner with her laptop complaining about lack of WiFi, and preceeded to continue to stare at a laptop screen during the party”  – bawolff 

Remove and commit a composer dependency in vendor/

Reverting a composer dependency addition directly via ‘Revert Change’ from gerrit would break composer, as we experienced after merging https://gerrit.wikimedia.org/r/#/c/188547/. The patch was to revert Plancake dependency due to compatibility issues in HHVM. I was un-aware of the fact that removing a dependency from composer.json would actually remove the dependency references from the vendor/ directory. Summing up – the correct way of producing a removal of a composer dependency would be:

  1. Remove the dependency from composer.json
  2. run composer update
  3. git add – -all
  4. git commit

A backport to an earlier branch can be produced with the following steps:

git pull --rebase
git checkout -b 'backportTo15' orgin/wmf/1.25wmf15
vi composer.json //remove dependecny
composer update
git add --all
git commt

If you screw up your backport in between – this would be an easy fix :

cd vendor
git pull --rebase
git review -d <change_id>
git checkout origin/wmf/1.25wmf15 . // would reset all files to wmf/1.25wmf15
vi composer.json // and remove the un-wanted dependency
composer update
git add --all 
git commit --amend
git review -R

Some moments from wikimedia-dev 🙂
[17:09:38] tonythomas: You’re meant to get composer to update itself. Otherwise it makes a huge mess.
[17:11:28] tonythomas: Yeah, composer and git aren’t friends. 😦

MediaWiki Unit Tests : Truncating table after each test

Writing PHP unit tests for the BounceHandler extension to prune old Bounce Records, I got into a situation where I had to truncate the ‘bounce_records’ and ‘user’ table after each single test. You would want the same when the test manipulates part of your table – and you dont want those changes to show up in your next test. Initially, I did this by manually truncating rows in both the table, but later I came across the use of

$this->tablesUsed = array( ‘table_name1’, ‘table_name2’ );

Implemented by :

class DoSomethingTest extends MediaWikiTestCase {
	protected function setUp() {
		// Your test initialization code here
		$this->tablesUsed = array( 'tableName' );

This would truncate ‘tableName’ table after each test ! Yay.
protected $tablesUsed = array(); is declared in MediaWikiTestCase.php[1] and its contents truncated after each test run !

[1] https://doc.wikimedia.org/mediawiki-core/master/php/html/MediaWikiTestCase_8php_source.html#l00038
[Ref : https://gerrit.wikimedia.org/r/#/c/176366/20/tests/PruneOldBounceRecordsTest.php ]

[MW]$wgExtensionFunctions : Run a function when an extension is evoked

Working further with making the BounceHandler extension more production friendly – we came up with a scenario in which I wanted some of the extension global variables to take values from a MediaWiki global variable. It would’ve looked like $myExtensionGlobal = $mwGlobal – which looks weird.
Thanks to Hoo( WMDE ), I got introduced to $wgExtensionFunctions[] [1]

You need to run a function once your extension gets loaded. In my case, I want to check if $wgFoo ( which defaults to null ) was set in the original myExtension.php and if not – set it to the MW global – $wgBar
in your myExtension.php

$wgFoo = null;
$wgExtensionFunctions[] = function() {
	global $wgBar;
	$wgFoo = $wgFoo ? : $wgBar;

This would introduce a nameless function which would do our required manipulations.
You can even make it a named function as in :

$wgExtensionFunctions[] = &amp;quot;myFunction&amp;quot;;
function myFunction() {
         // Code your task

Hope it help someone ! Happy Hacking 🙂
[1] https://www.mediawiki.org/wiki/Manual:$wgExtensionFunctions