Writing a Job queue to deal with load when POST-ing from exim to MediaWiki API

Last day, Tim Landscheidt from Wikimedia scribbled on my earlier post that I should use a job queue to handle load of the bounce handling API. I talked with Legoktm on this, and he said it was a great idea, as there can be a chance of multiple email bounces reaching the API simultaneously. I will jot down how we made that happen.
*Firstly, register and load the Job handler class

//Register and Load Jobs
$wgAutoloadClasses['BounceHandlerJob'] = $dir. '/includes/job/BounceHandlerJob.php';
$wgJobClasses['BounceHandlerJob'] = 'BounceHandlerJob';

* Now, create the file BounceHandlerJob.php extending class BounceHandlerJob from Job.
I wanted to get the $email, which will be passed from ApiBounceHandler::exectue();

class BounceHandlerJob extends Job {
	public function __construct( Title $title, array $params ) {
		parent::__construct( 'BounceHandlerJob', $title, $params );

	 * Queue Some more jobs
	 * @return bool
	public function run() {
		$email = $this->params[ 'email' ];

		if ( $email ) {
			// The function in the API where the header 
			// stripping and other stuff happen
			ApiBounceHandler::processEmail( $email );

		return true;

* Now, we need to make the APIBounceHandler class to receive the POST request, and create a Job queue object so that our objective is accomplished.

$email = $this->getMain()->getVal( 'email' );
$params = array ( 'email' => $email );
$title = Title::newFromText( 'BounceHandler Job' );
$job = new BounceHandlerJob( $title, $params );
JobQueueGroup::singleton()->push( $job );

Yay ! done. now the public static processEMAIL function needs to be defined with the necessary actions and you are good to go.
PS: Please add the following to LocalSettings.php to see the results.

$wgRunJobRate = 0;

To ensure things are working well, run the script

php runJobs.php

. Correct errors if any, and the perfect run will output something like.

2014-07-13 15:02:38 BounceHandlerJob BounceHandler_Job email=string(1862) STARTING
2014-07-13 15:02:38 BounceHandlerJob BounceHandler_Job email=string(1862) t=100 good

. Thanks


POST-ing bounce email to a Mediawiki API directly from exim

While moving forward with the BounceHandler extension, I was advised by my mentors to make sure that the bounce emails are directly fed to an API inside the extension, so that no manual queries are required, and its more safe.
An API, with the name foo can be accessed in the URL by http://localhost/mw-core/api.php?action=foo. A POST request with the name bar can be sent by URL http://localhost/mw-core/api.php?action=foo&bar=value
* How to add the new API
In your extension.php, register the new API, named bouncehandler firstly by:

$wgAutoloadClasses['ApiBounceHandler'] = $dir. '/ApiBounceHandler.php';
$wgAPIModules['bouncehandler'] = 'ApiBounceHandler';

* Make exim forward the emails directly to the API. Put this under the appropriate pipe transport.

command = /usr/bin/curl http://localhost/core/core/api.php -d "action=bouncehandler" --data-urlencode "email@-" -o /dev/null

or this can be done in a shell script too. You can direct the command to the shell script.

command = /path/to/script.sh

and in the script.sh

curl -d "action=bouncehandler" --data-urlencode "email@-" http://localhost/core/mw-core/api.php

* Now, on reception of a mail, it gets automatically transferred to the API, with the email as a POST param. The generated POST url will be of the form http://localhost/core/mw-core/api.php?action=bouncehandler&email=the_email
* You can recieve the email POST param in the API by

class ApiBounceHandler extends ApiBase {
	public function execute() {
             $email = $this->getMain()->getVal( 'email' );

Yay! further handling of the emails coming up in next post! 🙂 Happy Hacking!