Creating Self Hosted puppetmaster in Wikitech labs

While trying to test an exim-puppet patch ( gerrit.wikimedia.org/r/#/c/155753/ ) which adds a new router in one of my labs instance, I came across the need to create a self hosted puppetmaster. For starters ( like I was few days before ), puppet is a provisioning language, as they call it and applies pre-written configuration files to various daemons and tools coming under its realm. For example, I can set it to generate a standardized exim4.conf file when its run, this standardizing file need not be on the local machine you are working on. A self hosted puppetmaster will have all the configuration files packed inside a local directory – which is inside /var/lib/git/operations/puppet/
Where it come useful :
* It come into use when you want to play around with the default configuration files of your instance.
* When you dont want your changes not to be overwritten by the default puppet configured files.
Steps to make your Wikitech labs instance as a self hosted puppetmaster:
* make sure that all previously enabled roles were applied and puppet is running completely.

$ sudo puppet agent -tv

* go to Special:NovaInstance and select configure on your preferred instance.
* now, add tick the role role::puppet::self . This is our required self puppetmaster role
* go to your instance shell and give the command:

$ sudo service puppetmaster restart
$ sudo puppet agent -tv 

* now make sure you have the pupept repo in your local instance.

$ cd /var/lib/git/operations/puppet/ 

if the folder exists – Yay!
Now, you can apply your operations/puppet patches to this folder.

Advertisements

Using puppet realm switch to select between beta/prod ( Wikimedia clusters )

Since the BounceHandler extension is currently installed only in the beta clusters ( Official testing servers of Wikimedia- deployment.wikimedia.beta.wmflabs.org ), writing a custom router in the exim configs of operations/puppet ( configuration repo managed by puppet ) to collect in all the bounce emails and HTTP POST to the extension API seemed risky. This was due to the fact that operations/puppet is meant for production and something beta specific in it, that too continuous API requests dont look sane ( we cant estimate the traffic yet ).
Marius Hoch ( WMF ) came with the idea of an exim-realm switch, which seemed to solve the issue. You can switch variables between beta and production by:
We edited

$ sudo <editor> templates/exim/manifests/mail.pp

and added this before the inclusion of exim4.conf.SMTP_IMAP_MM.erb.

# config - labs vs. production 
case $::realm {
    'labs' : {
	$verp_post_connect_server = 'deployment.wikimedia.beta.wmflabs.org'
	$verp_bounce_post_url = 'http://deployment.wikimedia.beta.wmflabs.org/w/api.php'
    }
    'production': {
	$verp_post_connect_server = 'appservers.svc."${::mw_primary}".wmnet'
	$verp_bounce_post_url = 'http://meta.wikimedia.org/w/api.php'
    }
    default: {
	fail('unknown realm, should be labs or production')
    }
}

The internal networks selection was done to make sure the bounce gets POSTed to the right wiki. Now, this configuration can be used in exim4.conf.SMTP_IMAP_MM.erb by editing the bouncehandler router by:

command = /usr/bin/curl -H 'Host: <%= @verp_post_connect_server %>' <%= @verp_bounce_post_url %> -d "action=bouncehandler" --data-urlencode "email@-"

Hope it helps someone in beta 🙂