Is there really such a thing as "BAD" data?

I've been privy to see a couple of posts relating to quality and data over the last week or so.   I though the topics are wonderful and have decided to write a few words about the subjects myself.

Article #1 - Sometimes Worse Data Quality is Better

Article #2 - Is there really "Bad" Data?

Each deals a bit with quality but both don't directly deal with "Data".

The first article specifies very clearly why beta Max went by the way side in favor or VHS and how the landscape of music changed to an alternate format called MP3.  It also talks about the changes in cameras to digital and how it helped to cause the demise of Kodak.

The second was more a question posed in reference to a book called Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work.




So is there "bad" data?
That all depends on what you are using it for and what kind of data you have.

Take a bunch of sales numbers.  A few miss input values and everything goes to pot.  But, it  may not be so bad if you are just looking for some kind of trend, theme, or whether a product is doing better then another (if the values are not off by to much).  However, this would generally be a case when accuracy is paramount to obtaining usable information.

Now you may have a bunch of data in video format.  The quality of that data might not be so important.  You might not care that you can see the black head on the end of the commentators nose due to high resolution.  Then again you might want or need that level of resolution or clarity.



I deal with data on a daily basis. Phone data, address data, geodata, consumer data, and many things in between.
Some times we get files that have just what we need in them.  Say just the consumer name and address.  I call this at times "clean data".  Why?  Because there is less stuff in the file to screw up in conversion and the like.  Less to go wrong.



As you go back to the basics of databasing you find that many tables can be created and linked to a main form or table.  With the fields cut down you can do faster queries.  And since it is all linked off of a key code you can string this out over hundreds of tables.  This can also help to keep certain more sensitive table elements out of the hands of unauthorized people.
It can compartmentalize things a lot more and organize things in various ways.

When you start to mention "BAD" data you find a lot of things rely on the perspective and needs of the person that will be using the data.

Maybe a set of phone numbers is good for one person because they are testing if something is picking up, parsing or sorting in a certain way.  That same set of data may not be any good because you have no associated names, products bought, or other prudent information.  How would you use a list of phone numbers if that's all you have and want to do some product follow up.  You can't in that case because the data is "BAD" or "incomplete".

That is one of the points brought up in the discussions of "BAD" data.  There is really is not bad data.  There is incomplete data.
This could be the missing names associated to a list of phone numbers, or a portion of a video clips that has been cut off for any reason.  That data due to be partial or incomplete becomes worthless.

As we in modern society are trending everyone is collecting data and some believe you should go with "The more you have the better".  Yes, you might use some of that data down the road for some heretofore unknown application(s).  However, is that information really any good just sitting around doing nothing but take up space (main copy & backups)?



So due to the ability to acquire cheap digital storage space the trend is to save everything you can get your hands on.  This then nullifies the basic principles of data basing and has to  some extent allowed us to become lazy and less prudent in the way we store things and lay it out.

This phenomena of "BIG" data is a trend that could in the long haul allow for some really cool data analysis.  It also could become a boondoggle that only helps storage providers.

And the larger questions become - "Are you worried all this data is being collected?"

This become relevant as you have complete data or usable data.
If all that is being collected is a bunch of unconnected stuff that in the end is incomplete or "BAD" do you really care.

We have just taken the collecting of data to a whole new level.  This is nothing new to hear of a government that is collecting data.
The Domesday book from William the Conqueror's time was one of the first biggest collection of data for the purposes of taxes and associated elements.  To that point England did not really know what they had in front of them.  After that they had a better idea and could plan accordingly.  Thankfully, that data was not used for nefarious purposes.




"BIG" data does bring up some ethical questions.  Though these ethical questions are no worse now then they were say 100 years ago.  We simply did not have ways of storing so much information that could then be analyzed in such a short period of time with such accuracy.  Have you ever been frustrated with a web search that took 1 minute?  In reality Google is so efficient it can go through thousands it millions of pages in just a few seconds to give you relevant pages associated to what you want.

William the conqueror would never of dreamed of that.  Heck Einstein may never of dreamed of that...... maybe he did but, new it would be years away till anyone could do something about it.



So as we go about our daily lives we need to keep in mind how we can influence the proper use of data and how we can make sure that it stays clean and usable.....  for proper and good purposes.

Hopefully with diligence we will not end up repeating the Holocaust or some unnecessary war. For that could ultimately be what happens when we have "BAD" data!



Buaidh - NO - Bas

Deceiving to get ahead in life. Is it right or is it wrong?

Well I'd like to broach the topic of deceiving in the workplace etc to get yourself ahead in the rat race of life.

Why do I bring this up?  I was watching a documentary on David Geffen the other night on one of the local PBS stations and found out that some of his success was all based on a lie and cover up of said deceit. (About: Inventing David Geffen PBS, 2012-11-07) He basically did not have any problem lying to get ahead in the workplace.  Very sad.  We eventually had to turn it off and change the channel since our daughter was watching as well.  I'd heard a bit about this man in the past and known of some of his success.  I just did not know it was based on lies and deceit.


Also I was reading something in a trade show magazine I receive called Exhibitor.  The article - Guerrillas in the Midst is very telling and informs you about some disturbing activities in the work place.
In the commentary it referred to a couple of blogosphere postings on how to get ahead in the marketing game by not paying anything.
The mentioned examples were a person going to a trade show and plastering posters and fliers all over the place like the stalls in the restroom etc.  Essentially, taking advantage of the trade shows setup without necessarily having a booth.  Not a good method if you do not have a booth.  Your therefore acting like a leech.

And talking about a leech.  The second example was an Asian guy who basically stole a persons identity at the trade show and then went on to promote his company all the while masquerading as the other guy.  Very low in deed.  Though in both cases the posters seemed to applaud them for their guerrilla tactics.  The commentary hopes the larger marketing industry will shun this type of activity.




Also during a discussion in church I learned about the length to which person will go to cheat on their exams in law school.  The person said people would throw in the trash in the bathroom some note cards with information that they could use in the test.  In the middle of the test they go to the bathroom and retrieve the information off the cards.  Coming back they have had some fresh information giving them a leg up on the rest of the students.  So due to this they now have to be accompanied to the bathroom during tests.  Oh the length to which people will go.




We find this also in the political arena.  Look at the underhanded tactics that have been used for the presidential campaign or even the local mayoral race for Los Angeles.  Full of half truths and misinformation being spread around to confuse the voter.  Most will not find out the truth..... or if they do ... by then the damage maybe done.
Just like when a lawyer leaks something out with the jury present in a line of questioning.  How does the jury forget these things even though they have heard them.  Kind of hard to "Disregard" that piece of information though inaccurate or misleading that is now out there. It taints the information pool and that is why the lawyers love to us it.

Look at the current IRS scandal. They targeted conservative groups to hold them up and stone wall their exempt status petitions.  Why do that?  If you can't get in the game you can't be effective.  And the fall out.  Fire or force out the acting head of the IRS who was set to retire in June.  Basically, push a scape goat out one month before he is set to retire.  Will anyone be held accountable for their actions?  We may never know.  This will loss strength in our 24 hour news cycle.  Then they will quietly sweep it under the rug.  No one will really find out who authorized what and when.
The problem is more why is this appearing to happen more frequently in the last few years?
Have things changed so much in the last 100 years?

"I recall a time—and some of you here tonight will also—when the standards of most people were very similar to our standards. No longer is this true. I recently read an article in the New York Times concerning a study which took place during the summer of 2008. A distinguished Notre Dame sociologist led a research team in conducting in-depth interviews with 230 young adults across America. I believe we can safely assume that the results would be similar in most parts of the world. "
Dare to Stand Alone Thomas S. Monson President of the Church

We now have the Internet, and 24 hour news stations.  But, are things getting worse or are we really just more hypersensitive?

Though this story relates how all is not lost.

"A friend related this experience her husband had while attending medical school. “Getting into medical school is pretty competitive, and the desire to do well and be successful puts a great deal of pressure on the new incoming freshmen. My husband had worked hard on his studies and went to attend his first examination. The honor system was expected behavior at the medical school. The professor passed out the examination and left the room. Within a short time, students started to pull little cheat papers out from under their papers or from their pockets. My husband recalled his heart beginning to pound as he realized it is pretty hard to compete against cheaters. About that time a tall, lanky student stood up in the back of the room and stated: ‘I left my hometown and put my wife and three little babies in an upstairs apartment and worked very hard to get into medical school. And I’ll turn in the first one of you who cheats, and you better believe it!’ They believed it. There were many sheepish expressions, and those cheat papers started to disappear as fast as they had appeared. He set a standard for the class which eventually graduated the largest group in the school’s history.” ²

The young, lanky medical student who challenged the cheaters was J Ballard Washburn, who became a respected physician and in later years received special recognition from the Utah Medical Association for his outstanding service as a medical doctor. He also served as a General Authority and is now the president of the Las Vegas Nevada Temple."

Honesty—a Moral Compass James E. Faust

The moral fabric of society is falling apart.

What was deemed unacceptable just a few years ago is now considered not so bad by some.  Take the change in music for example.  Many complained in the past about the gyrating hips of ELVIS.  Now some dancers in shows have maneuvers that simulate sex acts or other intimate activities reserved for married couples.  Even the lyrics are getting worse.  Some however thinly veiled talk about sexual things or use language not even suitable for an adult of any age.

So what are we to do?
What should we do?


We have to hold up the standard for ourselves even if it means that we might not get ahead as some would, or will.  We will in the end be conscious that we have done the right thing.

"Brethren, none within the sound of my voice should be in any doubt concerning what is moral and what is not, nor should any be in doubt about what is expected of us ... We have been and continue to be taught God’s laws. Despite what you may see or hear elsewhere, these laws are unchanging.

As we go about living from day to day, it is almost inevitable that our faith will be challenged. We may at times find ourselves surrounded by others and yet standing in the minority or even standing alone concerning what is acceptable and what is not. Do we have the moral courage to stand firm for our beliefs, even if by so doing we must stand alone? ... it is essential that we are able to face—with courage—whatever challenges come our way. Remember the words of Tennyson: “My strength is as the strength of ten, because my heart is pure.”

Dare to Stand Alone Thomas S. Monson President of the Church

"I can share with you a simple example from my own life of what parents can do. When I was about five or six years old, I lived across the street from a small grocery store. One day two other boys invited me to go with them to the store. As we stood coveting the candy for sale there, the older boy grabbed a candy bar and slipped it into his pocket. He urged the other boy and me to do the same, and after some hesitation we did. Then we quickly left the store and ran off in separate directions. I found a hiding place at home and tore off the candy wrapper. My mother discovered me with the chocolate evidence smeared on my face and escorted me back to the grocery store. As we crossed the street, I was sure I was facing life imprisonment. With sobs and tears, I apologized to the owner and paid him for the candy bar with a dime that my mother had loaned me (which I had to earn later). My mother’s love and discipline put an abrupt and early end to my life of crime."

Moral Discipline D. Todd Christofferson Of the Quorum of the Twelve Apostles

 All is not lost.  We can start in our own homes and show the right direction by our actions.  We can train up our children to do the right.  And thus we can help to save the raising generation and the Nation in which we live.  For as the fabric of society falls apart and the family is attacked so goes the nation.  But, as we stem the tide and go against the flow of lacking morals and the like we can help to set a new standard.  Though in reality it is not a new standard.  For the standards of God do not change!

Buaidh - NO - Bas

Dysfuncional Families

You've heard the saying about someone being from a dysfunctional family.  But what does that really mean? 

As I begin to see situations in my own family I being to wonder if normal is a contrived state that psychologists have "made" up to get more business.  People really did not worry about this as much 1,000 years ago.

The best case in point is when you see the more or less "wonderful/perfect" life of those at church while at the same time not knowing what is fully occurring in their lives.   We know a couple who are now both retired and serving a mission for the church.  At meeting they appears to have everything going for them.  Nice home, minimal financial issues, reasonably good health and the like.  Few people know or knew that they have a son serving time in prison (most likely relating to drugs), and that his health is far worse then one would imagine.  And other kids are not active in the church.  On the surface they have it all.  Underneath the covers is the other half of the story.

So do we have a "standard" of perfection that in many instances is near impossible to achieve?  Take the "perfect" family with 1 boy, 1 girl a stay at home mom and dad with a wonderful job that affords them a house on a quite street with a nice white picket fence.
At least here in California  that ideal is near impossible to achieve.  To many have both spouses working just to make ends meet.  Many families are 1 car families, and due to the cost of living to many families are opting for just one child.   Sounds like the "real" normal to me.


So just because someone's family does not fit into your ideal for a family does not mean they are not normal?   I love that word NORMAL.   I can define what I want as normal and if you don't fit into it you are weird and dysfunctional.  But, the next guy a couple of houses down might not find my normal to be his normal and thus the grand old debate.

In the end sadly to say if all the husbands on a given street beat their wives daily then that would be the "norm" and the one guy that does not could be considered dysfunctional or weird.  Though, we all know that the more you become educated the more likely you will avoid these types of bad behaviors.  And I'm not talking about simply becoming educated in the University/College sense of the word.  I'm talking about an expanded mind, a less ignorant point of view, and wider horizons.  This moves bigotry and the like to the back corner.

So I'd say I have a somewhat dysfunctional family in the fact that few of us talk on a regular basis even though most of us live with a few miles of one another.  We doe get together for the main holidays like Christmas. Thanksgiving and Mother's Day.  But, in no means do we communicate like you would expect a "normal" family.
I guess the love it not there......


I'm reminded of a subway art piece we acquire recently at a store.  It has a black background and whit lettering of various styles - it is titled 'Family Rules'.   This piece is neat as it includes about 20 different things to do around the home.  Examples include - Give hugs frequently, say I'm sorry, listen to your parents, say I love you, etc and etc.

Out and about on the internet are a ton of these types of wall art.
I guess if you strive to follow what they have written on them you can have that "perfect", "normal" family that is not dysfunctional.  And you might not have to go on anti-depressants to control your feelings as you fall short of that ideal in your head.

stikmup.blogspot.com

Buaidh - NO - Bas

Politics and Religion

As the presidential field heats up we are more and more hearing about the religious affiliation of the Republican candidates.  We happen to already know it for our current president even though there are some that take it as a grain of salt and believe it to be something else.

Should the religious affiliation of the candidate really matter that much?  Does it matter that the person is a Jew, Mormon, or a Catholic?

The religious background of a person will drive certain areas of a opinion such as abortion, assisted-suicide, death penalty and the like.  But, when one considers the overall responsibilities of a candidate will these play into the day to day things they are responsible for?

Take a President of the United States.  Does he really deal with the issue of abortion on a daily basis.  More likely he's looking at some spending bill, or helping decide whether to draw down troops in a specific region of the world.  So why do we make such a stink about what religion a person affiliates with?


I've got a few ideas in this regard.  There are several major religions or faiths in the world.  Christianity, Judaism, Buddhism, Muslim etc.   Sadly, within each of these there are various churches.  There is no central head in most cases even though for the Christian we are all following the teaching of Christ.
Thus it is OK to be a christian and hold to the basic teachings - 10 commandments, love thy neighbor etc.  But, when it comes to central church based leadership people spook.   Holds true for the Catholics and the Mormons.  People are afraid that the supreme leader of the church will dictate policy.  Yes, if you hold to the faith properly you believe the Pope or the President of the LDS church is the mouth piece for God.

So are the people afraid that if Mitt Romney were President that he would always be on the phone with prophet Thomas S. Monson before he makes any decisions?  Or that a catholic candidate would be constantly in contact with the Vatican and if the they tell him to do something he would without question?  Maybe.  Some just like the idea you can have a spiritual advisor you contact but, that this person is not the final word, the definitive voice of God.  Maybe they like the ideas that you switch religions/churches like you switch your underwear.  That when one is convenient you go that way and when another is more readily at hand you go that route.  Always looking to one person or location causes a riff especial when we hold so true to the belief in separation of church and state.

Take England for example.  The queen is the head of the church there.  How would that go over here?  We have had ministers run for the presidency before, and likely at all levels of government for various positions.  But, is having a Baptist minister as a mayor of a city any different then having say an Asian?  You could claim favoritism in both cases.  Would the Mayor favor the Baptists more then another religion or the Asian his tribe more then another?  That is were things get to the ridiculous!

But, we also have to watch out for that kind of stuff.  The people in Germany in the 1930's started to let those types of things enter into the political realm.  One tribe favored over another, one element more then another.  And we know that this lead to things like the rise of the Nazi Party, persecution of not just the Jews, but other ethnic groups considered inferior like the Poles, Russians, Slavs, and the list goes on.  They so bent on this "religion" they even began exterminating people. It started slowing with the mentally and physically handicapped, and then progressed to mass deportations to the concentration camps generally ending in a trip to the gas chamber.


So could that be the fear of the people when it comes to religion?  An indirect dictatorship or theocracy?  I hope not.  Since what I want in a presidential candidate is someone that is honest, forthright, and has the capacity to make decisions that will be for the good of the country.  Basically, a morally upright person that will follow their convictions and do the right thing at the right time and not just in the hopes of being reelected.

Buaidh -  NO  - Bas