Few days ago, I created a simple Python script. This serves as my Python exercise as I’ve been studying Python for over a year. This script tests site’s Multi-Variant tests or A/B tests to check if the traffic split is correctly computed.
Backgrounder
Our main site is a few years old and enhancements come and go. Before an enhancement goes live to production, the marketing ladies wants to prove that the new enhancement really works and users engage more to the site.
Developers will create different versions for the site or specific page, 1 for the old version and 1 for the enhanced version. We put some analytics stuff that the marketing ladies will consolidate later on and based on the report, they will know if the new enhancement produces more sales, increases user engagement, or whatever.
On the backend, we set specific percentage when the customer will experience the new enhancement, for example a new home page will be experienced by 10% of all users or a new shiny button will be experienced by 40% of all users. It is set via cookie.
What are we trying to create
Our QA already have a script to verify that the MVT traffic split is correctly distributed based on the percentage given. They are using PHPUnit and Selenium for that purpose. For example, if a shiny button is expected to be experienced by 40% of the whole user traffic but on tests it shows to say 60% or 10%, therefore, the MVT traffic split fails.
I thought it could be done easily via cURL on PHP but I was trying to explore Python hence I created this simple script.
The script
The script simply accepts some configuration as to what is the expected traffic percentage per tests, what cookie name to use and what URL to crawl. I manage to play around with Cookie manipulation, dictionaries and HTTP. See below:
import urllib
import urllib2
import cookielib
class MVTSplitTest:
expected_split = {}
split_stat = {}
max_run = 500
test_url = None
traffic_cookie = 'mvt_traffic'
control_traffic = 'a'
def __init__(self, url, input_split, max_run=500):
self.test_url = url
self.expected_split = input_split
self.max_run = max_run
def run_test(self):
for x in range(self.max_run):
print 'Calling %s at %d of %d' % (self.test_url, x + 1, self.max_run)
t = self.fetch_url(self.test_url)
if t:
print 'Hits traffic %s with value %s' % t
else:
print 'No valid traffic hit'
def print_result(self):
print ''
for key in sorted(self.expected_split.keys()):
split = self.expected_split[key]
print 'Traffic for %s: %s' % (key, split['name'])
if key in self.split_stat:
print self.split_stat[key]
else:
print 'None'
def fetch_url(self, url):
cj = cookielib.MozillaCookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
urllib2.urlopen(url)
split = self.get_cookie(self.traffic_cookie, cj)
ret = None
if split:
traffic = self.get_split_letter(urllib.unquote(split.value))
if traffic:
if traffic == self.control_traffic:
ret = (self.get_split_name(traffic), 'Control')
if not traffic in self.split_stat:
self.split_stat[traffic] = 0
self.split_stat[traffic] = self.split_stat[traffic] + 1
else:
mvt_value = self.get_mvt_value(traffic, cj)
if mvt_value:
ret = (self.get_split_name(traffic), mvt_value)
if not traffic in self.split_stat:
self.split_stat[traffic] = {}
if not mvt_value in self.split_stat[traffic]:
self.split_stat[traffic][mvt_value] = 0
self.split_stat[traffic][mvt_value] = self.split_stat[traffic][mvt_value] + 1
return ret
def get_mvt_value(self, traffic, cookie_jar):
result = None
if traffic in self.expected_split:
mvt = self.expected_split[traffic]
if 'split' in mvt and 'cookie' in mvt:
mvt_cookie = self.get_cookie(mvt['cookie'], cookie_jar)
if mvt_cookie:
letter = self.get_split_letter(urllib.unquote(mvt_cookie.value))
if letter in mvt['split']:
result = letter
return result
def get_split_letter(self, str):
return str.split('|')[0]
def get_split_name(self, traffic):
result = None
if traffic in self.expected_split:
if 'name' in self.expected_split[traffic]:
result = self.expected_split[traffic]['name']
return result
def get_cookie(self, name, cookie_jar):
result = None
for c in cookie_jar:
if c.name == name:
result = c
break
return result
if __name__ == '__main__':
split_config = {
'a': {'name': 'Control'},
'b': {'name': 'Search Auto Suggest', 'split': ['a', 'b'], 'cookie': 'mvt_auto_suggest'},
'c': {'name': 'Search Tabs V2', 'split': ['a', 'b'], 'cookie': 'mvt_search_tabs'},
'd': {'name': 'Vehicle Selector V2', 'split': ['a', 'b'], 'cookie': 'mvt_vehicle_selector'},
'e': {'name': 'New Home Page', 'split': ['a', 'b'], 'cookie': 'mvt_new_home'},
'f': {'name': 'Checkout Force Login', 'split': ['a', 'b'], 'cookie': 'mvt_force_login'}
}
url = 'http://staging.yoursite.com/'
max_run = 500
t = MVTSplitTest(url, split_config, max_run)
t.run_test()
t.print_result()
To run the script, simply execute the script. For example the script’s file name is testsplit.py, you run:
python testsplit.py