Data Quality Assurance

Data Quality is a bit of an under estimated concept in small to mid scale projects, where it is often addressed far too late during testing or bug fixing. There are plenty of articles which elaborate why Data Quality should be a concern, not just for project managers but also for solution architects and folks in governance roles. However, it’s quite difficult finding a simple framework which you could use to assure data quality particularly for small projects. The methods typically available are meant for large programmes where you have to contract someone to build a specific solution or enterprise focused Data Quality Assurance practice, which is not what this blog is about. You'd have to reach into your wallets for that !!

In this blog, I have made an attempt at putting forward a simple and basic framework using which Data Quality assurance can be managed, monitored and measure – particularly to aid in Test Planning. This may be toned up or down to fit needs in Application Development, Solutions Architecture and Enterprise Level Architecture. 

The methodology focuses on measuring and monitoring data quality levels for key data items or entities that are used or produced by the product/domains/enterprise. We will use a hypothetical scenario where we are trying to assess data quality of a system (or collection of systems) that deal with product sales. In order to quantitatively measure data quality, the following key metrics may be used.

1. Uniqueness - achieved by ensuring there is no duplicate data within the same domain. For e.g. The sale of a product by the same vendor has not been reported different for the same time period across different reports. Typical area to invest time is ensuring each key data entity has a means of uniquely identifying the records, usually enforced using unique key constraints. The concept of 'single source of truth' comes into play here.

2. Completeness - achieved by ensuring that a complete set of data is available for each entity. For e.g. the match-rates obtained by consolidating sales reports to master product data records is of acceptable quality. This is sometimes difficult to assess, as you never know what is incomplete unless you have a truly complete set to compare with.

3. Accuracy - the correctness of the data is achieved by ensuring that the data is a true reflection of the actual circumstances. For e.g. the Tracked Sales Report produced matches the Individual Sales Report received at aggregated and granular levels from various Point of Sale. This often requires Business Users to eyeball through samples to validate that data is accurate and represents best knowledge held by the business.

4. Timeliness - The data is up-to-date and delivery of the data is timely. For e.g. the sales period used in the Master Data is the same as that used by the reporting Point of Sales. Often times, this gets complex when you have multiple systems processing same data sets i.e. high velocity data processing.

5. Consistent - The data is the same in its definition, business rules, format and value across systems at any point in time. For e.g. the Master Data provided has same expected data structure, values and data types, unless validated through a known change. Consistency also implies that the data is a true reflection of the real world scenario, hence may overall with accuracy.

The Data Quality at each completed and iterative stage of development will be monitored by aggregating percentage scores against set targets for each metric. Each data set needs to have its own DQ metric assessed. Each DQ metric has a set of test scripts / quality measures which focuses on the data sets involved and needs to be contextually relevant. For theoretical correctness, I have considered Data Integrity to be responsible for Completeness, Uniqueness & Accuracy therefore haven’t considered as a separate metric.

The DQ Assurance (artifact) can be split into each data set (product data, reference data & sales data), which have list of data quality tests that need to be performed, categorized by the metric that the test assures.

The tests under each Data Set must also have Test Phase, Data State and other relevant attributes such as Platform, Users if directly involved in Testing and Assurance otherwise can be separately addressed in a Data Definition artifact. The Test Phase correlates with the Overall Test Plan to suggest which phase of testing the test activity can be executed. Typical Test Phases can be Migration Testing, Functional, UAT/Business Acceptance and post Go-Live Change Management etc. The Data State is a more technical indicator of the state of the data in the end-to-end process the application / solution deals with. This correlates with the Business & Data process proposed & agreed for the project, to say at which stage of process the test needs to be conducted. Typical Data Stages can be: Raw Data, Ingested Data, Transformed Data and Processed Data.

Measuring Results
Each test activity will ideally have a target of 100% unless otherwise is a more appropriate measure. The target values are calculated as a percentage of how many pass the test case over the total population of the records in that data set / entity. Example:

ID	Test	Metric	Test Phase	Data State	Target	Actual	Resource
2	Does the data include products licensed in the period being reported on?	Completeness	Migration	Raw Data	100%	80% - This could mean that out of 100 products the POS has reported sales for March 2012, only 80 of them were made available in March 2012 i.e. 20 were possibly made available in 2011.	Who does the test?

The summary results of the Data Quality Assurance Tests will be compared against targets and can be visually represented to identify where the Data Quality can be improved.

The more blue you have in the radar graph, the poorer your Data Quality is. Each data set should have one of these radar graphs plotted to give an executive summary of the quality of data in that domain. The numeric results are then used either during test or during normal operation of the solution to identify issues where data quality is poor and root causes behind the gaps.

Hope this blog was useful, and would welcome hearing queries or discussion if you’re interested in this field.

Things to do in Bath, UK

I've always enjoyed exploring historic towns and cities in England. Seems a lot more sensible for short holidays than to travel to Europe. Thought I'd add to this hobby to write a blog about each mini holiday my wife and I take. This blog is about a three day mid summer trip to Bath. It's a small historic city just over 100 miles west of London as you travel towards Bristol, just under 2 hours by train. The city itself was classified as a World Heritage Site in the late 80's to mark it's contribution to history. Other heritage sites include the Great Walls of China and the Taj Mahal, so you get the picture.

Day 1

After getting off from the station, we started walking around just to get a feel of the surroundings. The landmarks worth paying a visit, or more to say unmissable are the Pulteney Bridge, Royal Circus, Royal Crescent and Queen Square.

This is a view of the Pulteney Bridge from opposite the Guildhall Market. The bridge isn't like Tower Bridge as such which you cross - the bridge is actually a collection of shops over the canal, but trust me it's an amazing site to see. 

The market is also worth a quick visit - you'll need some spare cash with you as none of the shops accept cards but have loads of tempting items to buy whether you're looking for nibbles, antiques or collectibles. 

Check out the tea pot I was almost tempted into buying, but then realized I wouldn't have the appropriate place to showcase it.

The first thing you notice about Bath is the collection of Georgian architecture - possibly the largest in UK. This is characterized by a series of buildings made out of limestone derived stone walls. All terraced with a classic elegant look. 

The photo here is of the Royal Crescent. A collection of 30 or so houses, built during the late 1700's by John Wood to house the wealthy and ill who used to visit Bath for medication and relaxation.

Each house today is worth anything around 800K to 1.5 million. 

A similar landmark is the Royal Circus - no, there aren't any animals or magicians ! Imagine a circle of Georgian buildings, with a small field in between with four roads entering in / out of - bit like a roundabout for those from the UK. While around, try to see the series of decorative artwork on each - some are quite horrific and masonic.

If you're an avid photographer unlike myself,(I just use point-n-shoot) then you'd get awesome photos in a wide angle.

After lunch, we headed off to Royal Victoria Park. A decent sized botanical garden with an impressive collection if you're interested. Not exactly my cup of tea, but great for those who like taking flower shots. I enjoyed 'bird watching' instead. Don't think of nicking any of them , they're tagged with RFID tapes.

As the day almost came to an end, we looked around for dinner and settled for Salathai. It's located about 5 - 10 minutes from the rail station and offers value for money Thai food, without compromise on quality. Highly recommend the Tom Yam soup.

If you're in Bath for just a night, there are a couple of luxury hotels where you can spend anything between £180 to £220 such as the Royal Crescent hotel or MGallery Francis. We chose to stay in a mid scale hotel called Royal Hotel or The Royal, just opposite the train station. We paid about £250 for two nights in a superior double, with breakfast and a 3 course French dinner. I've rated this hotel very highly on Tripadvisor for their comfort level, service and decor - the only issue you might have is with the noise from the road at night if you leave windows open. Other than that, much better than a lot of 4 star hotels which charge more purely because of swimming pools & gym. This hotel doesn't have any of that stuff. If you do choose this hotel, try their dessert selection from the Brunel Brasserie - 10 on 10.

Day 2

 

 After having breakfast at the hotel (both continental & cooked :) we started the day with the Bath Abbey. Very much like Westminster Abbey in London, without the foreign tourists. And a lot smaller. 

The Abbey inside is like any other chapel, stained glass windows, high ceilings, immaculate architecture etc etc.

Something worth spotting are detailed paint work depicting Jesus Christ and St Mary.

 However, the Tower Tour inside the Abbey is my highest rated guided tour in Bath. There are lots of others, but this is one which you cannot miss. It costs £6 per person, takes under an hour, and the highlight is you get to climb 200+ steps to reach the top of the Abbey. The sights of the city from there are breath taking - especially in summer. The photos below are all taken from the tower, so I'll let them do the talking.

 Overlooking the Parade Park and the Abbey Square. This place itself is great for spending 30 minutes. Always has live Italian / Latin music. 

One of the best experiences was walking inside a clock tower and seeing the inside of the Abbey clock, mechanical working of the bells etc. Finally, Shopping Malls and City Center from the top.

 Overlooking the hills and Bath stone fields. Highly suggested you take the sky line tour which lasts 40 minutes and takes you through the hilly areas. The residential houses will entice anyone to invest here - a bit pricey though, nothing under a million. History trivia - one of the residences belonged to Beau Nash. One of Bath's wealthiest individuals - when a farmer earned a few pounds a year, this guy earned a thousand. He was responsible for converting Bath into the Las Vegas of the 18th century.

After catching our breath back, we spent the rest of the afternoon at the Roman Bath. This is literally a slice out of history. The ancient Romans built this as their communal bath retreat for the wealthy. The key essence of this place is the naturally continuous spring water. We went there in the summer, hence didn't strike us as a surprise until we touched and drank the spring water at 34*C without artificial heating, in fact originally 42*c but cooled down to make it drinkable. The water contains over 27 minerals and is very rich in iron. 

The amount of historical significance this place contains has gained Bath's reputation as a World Heritage site. I could probably write a whole blog on the Roman Bath itself. The entry fee is £12.50 per person and can take more than 2 hours , so those who are in Bath for just a day may re-consider, as you need to have a lot of time in hand. The Roman Bath houses a history museum, guided tours through archaeological remains and bespoke events throughout the year.

Later took the city bus tour costs £12.50 pp and valid  for two days. Can be used for two routes, the city and skyline. Hop on / off at sites you find interesting till 7pm. Ended the day with a classic three course dinner at the Brunel Brasserie.

Day 3

The last day was spent mostly shopping and visiting the Jane Austen Center. Jane Austen has been possibly Bath's most popular resident. You can visit the houses she used to live in, the places she used when writing Hangerlane etc. The Jane Austen museum tickets cost under £10 - starts of with a story telling mode of tour, the staff give you a short history of Jane's personal life. You are then free to walk around the museum, try on costumes from the era, view rare photos etc. The highlight of the place is the Regency theme throughout. The Regency Tea room offers award winning cakes and scones, with over 15 varieties of flavored tea.

Lunch time on our last day with the weather being probably the best you could ask for. Clear skies, bright sun, mild breeze. Chose Jamies Italian at the Milsum Place. The place has a very contemporary feel - historic buildings and their terraces and modern restaurants. The view from Jamies is amazing if you can pre-book a table.

Shopping in Bath is a lot different from shopping in London - to start with, there aren't that many people. Second, it doesn't remind you of a consumerist society. The shops are 'little shops' as they are called in Britain i.e. not high street brands. There's a massive Debenhams if you're looking for familiar high street departmental stores. I personally enjoyed Jollys among the large shopping malls.

Don't let that fool you - doesn't mean they are cheap. On the contrary, some shops have one of the expensive art pieces, watches, woolen wear. The best way to describe this is - imagine Harrods quality in a North London Angel shopping environment.

Don't forget to try the Cornish pastry shops in the Guildhall Market - whether you're looking for traditional, cheese or vegetarian - pure class.

To end the day before catching the train, we had to get a slice of Bath's afternoon tea. Bath was predominantly England's most elegant city back in the days. The shops, people, culture is still very characteristic of that. We found a balcony style cafe called Boston Tea close to the center which stays open till 7pm, serving scones, clotted cream, strawberry jam and Darjeeling tea.

Hope you found the blog useful if you're planning your trip to Bath for a weekend or similar. The city like any other in UK is best planned in the summer - although if you're going to visit the Roman Bath in particular, choose winter time.

Why we will never have a revolution against corruption in Bangladesh – a naive view

Set A: Ruling Party – Reap benefits in current state when their party is in government

Set B: Government loyalists – Reap benefits all year round, working for government

Set C: Paid Political workers – Reap benefits all year round, working for or against government

Set D: Dormant – have become immune and more concerned with bringing food to the table

Set E: Revolutionary Nationalist – active protesters against any government

Set F: Opposition Party – Invest in opposing current government to reap benefits in future

From my naive understanding, in order to have revolution similar to the Arab Spring, set E i.e. the Revolutionary Nationalist has to significantly inflate. The practical options are:

a.    Employ set C i.e. political workers – this can only be done through incurring huge costs, funded by Set F, i.e. opposition party. This usually never works out, as the original agenda of Set E is lost in the process.

b.    Employ set D i.e. dormant population – although the more appropriate approach, this can be achieved only through substantial length of oppression under the same party to build enough hatred and anger with the presence of a new revolutionary leader. Trouble is, this is highly unlikely given we have a memory refresh every 5 years. Also faint chance of this happening is immediately leveraged by Set F, the opposition party, therefore again, original agenda lost in the process.

c.    Under the most unlikely event of a new party forming within set D&E, all other sets ensure it is neutralized to secure the investments they have made as incumbents – therefore the revolution becomes too costly for the dormant population to continue.

The main culprits from my point of view are the Political Workers – they have neither love for the nation, nor ethics behind the cause they fight for. Yet – they have played a pivotal role during each election in the last 20 years.

The dormant population has led itself into its own fate – we wait in traffic for 4 hours, watch the meter increase and complain on social media through our phones, instead of abandoning the rented vehicle and joining in a 30 minute march back home.