Quick and dirty test data generation

I am a noob with postgres and linux, if you are an expert why are you reading this? So I just had to generate a large ammount (>1) of rows to test against. This is a step by step dirty way to get from SELECT to INSERT (I know you can do it with sql smart ass but can you do it in under 2 minutes?)

1) Get the table with some random row of data
psql -d <my-db> -c "SELECT * FROM <table> where id = <id>" > raw_sql
gedit raw_sql&
(GEDIT? I hear you scream. NOOB! I scream back.)

2) Format the raw_sql. Depends on the data your millage will vary.
Find and replace whitespace with nothing.
Find and replace || with | null |
Find and replace || with |
Find and replace | with ,
Save as <file>.csv

3) Delete the null stuff.
Open a spreadsheet and import the csv.
Select all columns with null and delete them.
(Basically the spreadsheet allows values and attribute names to be aligned and easily deleted.)
Export to csv.

4) Create the Insert
Open the new csv file (using gedit)
Add at the top psql -d <my-db> -c "INSERT INTO <table>
Then surround the attribute list with parens
Then add below that VALUES.
Then surround the values with parens and add a final " at the end.

5) Multiply
Copy and paste as many times as necessary and change the values that you want to differ manually.

Save as <something>.sh

And run ./<something>.sh

Till next time,

Stratos out.

Good bye Steve

So many have spoken about Jobs. It doesn't matter what I say, but I have to say it.

Steve was a usability wizard and every computer person should be striving to achieve what he did.
Steve made PCs usable by everybody.
Steve helped save the music industry.
Steve brought the smartphone to the non-geek, non-business world

Steve created the mobile computing market.

In one broad stroke Steve innovated preexisting industries and became the standard overnight.

Thank you Steve. You will be missed.

Long time no C

And then it was vacation time and then vacation time was over...

Teachers and students are in back to school mode and so am I, so this is a post about going back to school work.

Updates:

My app is coming along nicely, didn't get a lot of play time with the Xperia mini from the Sony-Ericsson loaner program but found two bugs that need fixing. Still loads of bugs remain, a help function doesn't exist yet and a couple of cosmetic updates for the app to be "ready-ready," then another month for "ready-ready-ready" you see where that one is heading, right? Last week I did a soft come back which was cut short cause I had to go to a job interview on Friday.

Job hunt is now on, I still haven't heard back from two folks that said they would meet with me over coffee to talk employment. I guess people are busy/away for labor day or they give the cold shoulder here by putting your address in their spam filter.

In an effort to become a better programmer I am reading Effective Java, solving Project Euler problems in Python and listening to the Java Posse during my walks. I am also planning to attend a web developer seminar and the largest AI class ever made.

It feels like the years I spent checking registers should have been spent sending JSON objects, doing CSS transforms and designing thread safe hash maps. It's never too late though loads of programmers are productive in their 30s... they take us out back and shoot us after 45 right?


Till next time,
Stratos out.

We are anonymous! Or not?

Following Google's decision to use real names for g+ I saw the real name and anonymity debate spark up. Real name people argued that you are a coward for not using your own name while anonymity folks gently ignored them and created Biggus Dickus the third on g+.

To me the anti-anonymity crowd are either naive or evil. Evil because ultimately they want us using our own names so that the secret police and telemarketers can get to us faster. Naive because they do not realize that we all live in a totalitarian regime. Sure the secret Mountie police wont knock on my door for posting "Cartman like" jokes on my facebook to my army buddies but the Nazi judgmental recruiter who looks at my web presence evaluating me for a job might not appreciate the humor of Biggus Dickus the third. (Not only that, now I called her a Nazi. This will be an interesting interview.)

You see it is not that we are paranoid it is that we cannot be sure that every potential client, employer, CIA agent out there can appreciate irony. But this is not about dick jokes, rest assured. It is also that my loud Pastafarianism may really agitate my ultra-religious relatives, my call to arms to protest the G8 doesn't go well with my firm's CEO, my militant Islamism may land me in some cell somewhere away from toilet paper and lawyers. Anonymity protects forum trolls and flamers as much as it protects unpopular ideas and fosters revolutions.

Instead of fighting privacy I expect Google to understand the need for my SpiderMan33 persona and my Stratouklos nickname and Stratos and Mr Xakoustos they are all me, in different contexts. SpiderMan33 may admit that he spent half the day reading comics, Stratouklos shares dick jokes, Stratos posted some nice photos and Mr Xakoustos was working on a project. They all happen, they are all different and I do not need my comic buddies/mates/strangers/boss to know of each other's existence. Oh and while you are at it, Google please make my avatar change depending on who I post to so I don't have to juggle four Google accounts. Thanks!

Sincerely,

DickMan41 aka Helen from accounting.

Till next time,
Stratos out.

Fractals and design

So my first releasable app is already a month over my initial estimation. The Android learning curve is taking twists and turns, I am graphically challenged so it takes me longer to design a good icon, blah, blah, blah. Everything mentioned is true but what is interesting is that when it comes to design the challenges that need to be solved behave like fractals.

Or in other words in software design and in lesser extent electronics most problem domains are fractals. Design generally deals with fractals, for instance, to build a single engine one must build hundreds of components, that are built off hundreds of cogs and pieces, that have to have dozens of unique features. And once the engine is built it needs to go into a car/boat/plane composed of thousands of parts that they themselves represent millions of smaller problems each visible only once a designer focuses into the problem area.

Because of the fractal nature of the problem domains and the lack of serious standardization in design principles. software design is notorious for it's cost overruns, widening scope and date slips. In software a huge ten day design effort problem can be hiding behind every aspect of the project. A feature might be impossible to test without setting a ridiculous number of data by hand, a function takes too long to complete and concurrency at that stage requires redesign and so many more that volumes of books are not enough to list them all. Due to the number of unknown factors involved one could say that the model of solving a software engineering problem is chaotic and as such very difficult to correctly predict and estimate.

As software engineers we try to use our tools to accurately model and predict our future but when we actually look at our methods we generally look at the sky, lick our fingers and say hmm... it'll be OK. Only to find ourselves in a thunderstorm of epic proportions a week later. Unfortunately because business and money are also involved very often someone warns us of the coming storm but in an effort to keep our bosses, clients and investors happy we tape his/her mouth SHUT and throw them in the basement. Cause if you hope it will work out, it works out right? Ask any of sailors down at the bottom of the sea.

New methods are being developed all the time but rest assured that every tool in the world can be manipulated to predict good times when contracts full of money are at stake. I mean ask any construction company, they'll tell you: "Don't look at the sandy land we are developing on, look at the nice drawings instead and please give us money." Trust us, we won't go over budget before you are way too committed to pull out. And the world keeps on turning.

Till next time,
Stratos out.

Vertically integrated design in a 2.0 world

This is mostly inspired by the troubles faced by RIM and other companies.

Vertical integration in design used to be a good idea. The same company is in control of the architecture, the hardware, the software, the brand, the marketing, everything. Electronics produced this way rely on standardized interfaces to communicate with the outside world, everything is well contained and you as a company you control your products and subsequently your brand tightly.

It was such a good idea in the 70s and 80s but from the 90s onward the world started to accelerate. Design by comity and over-protectiveness slowed down innovation and created bottlenecks. To fight these, companies discovered outsourcing to speed up the mundane parts of design which at least allowed for features to have a quicker time to market. But the outsourcing solution is not enough in the 21st century world, time to market of 1-2 years is unacceptable in most industries if not all of them.

Mega projects are feeling the strain of trying to catch up with newer, leaner, more distributed designs and these growing pains seem to me to be the driving force behind agile methodology adoption by large corporations. If you have read some of the agile books out there the introductions sound like pitches aimed at alleviating the major problems faced by industrial mega projects. Elimination of bottlenecks via feature teams, knowledge transfer via team interactions, better control via transparency, etc, etc.

But agile has had mixed results. Change itself causes some initial slowdown in production or the fear of a slowdown. Management and employees don't easily embrace changes that undermine their job security which is a natural byproduct of some agile methods. Not to mention that often the wrong methodology is applied to the wrong project due to a silver bullet mentality.

And then came crowd-sourcing and the shit really hit the fan! The most affected industry of all is the mobile phone industry. The smartphone wars that have ensued are a testament of the changes going on in the world of electronics and software. Six companies are in the ring, one so far behind it is irrelevant, yes that's you HP, sorry. One already accepted defeat and hopes to band with competitors to survive, nice call Nokia. Microsoft, after having their crappy windows used to wipe the floor with, comes back with WP7 but is it too late? RIM is faced with mounting pressure to do something, market Blackberries to toddlers maybe? And the two kings of the ring right now, Apple and Google. Well Apple was the king to be precise before Google and their Android army started kicking their butt. But Apple, RIM, M$ and co have banded together in a last ditch effort to stop the green menace.

In the battle of Android vs everybody else it seems that Android cannot survive the constant barrage of patent trolling, leveraging the existing power in markets outside the mobile arena (WP7 and Xbox, iOS and AirPlay, MacBooks, iPad, iLife) and challenges posed by a newer and not as mature platform. In all this mess my money are still with Android. Why is that? It is not my natural tendency to side with David against Goliath, I mean Google is close to taking over the whole universe it isn't a David. It isn't even their motto "Don't be evil" that makes me like them so much.

Google will win because vertically integrated design in the 2.0 world is doomed to sink like a monolith. iPhone5s best features are already integrated into Android (smells like counter lawsuits by Google?) or are about to be topped by C2DM functionality. Simply put the practices of industrial secrecy, design by comity, content approval processes are no longer relevant in today's fast moving world. Time to market of over a month is too much at this age. Your only hope is to make your source open, attract as many individual designers as possible, give your platform to anyone who asks for it and let them change it as they please, in one word: Android.

Till next time,
Stratos out.

Lack of competence

Where I used to work "lack of competence" was the scapegoat for all the shortcomings of the organization. It annoyed the hell out of me because it reminded me George Carlin's critique of the use of euphemistic language. As the great man said: "American English is packed with euphemism because Americans have trouble dealing with reality and in order to shield themselves from it they use soft language." Well it is not just an American thing or at least it is not confined in America.

It wasn't just the descriptive nature of the phrase that bugged me it was the meaning and the way it was used. Lack of competence or incompetence was not the correct word to describe the problems that plagued our teams. If anything the competence level in that place was through the roof, however experience was lacking. Experience is the most valuable and expensive asset any employee carries and should be seen and managed by organizations as such. 

Experience is the difference between testing using the scatter-shot method (100 test cases and hopefully we'll get all the bugs) and knowing what four or five test cases will find bugs by reading the project's title. Experience creates experts/gurus/rock-star programmers, reduces design time, errors, increases quality and estimation accuracy but its cost is hefty because to create experience one has to go through the process of bad design, scatter-shot testing, blind estimations and failed projects. That is why experience needs to be actively managed, encouraged and guarded and that was why everybody blamed "lack of competence."

Hey I have been failing X amount of years so I must be experienced! Right? Wrong, just because you have been designing the same flawed solutions for X years doesn't mean you are a good programmer it means that you are an efficient, fast, proficient liability who should be promoted to management ASAP so that you stop poisoning young designers. Not everybody is capable of learning from experience and not all design configurations are conductive to building experience fast.

In the personal level experience is built with humility, drive for constant improvement and the thirst of learning of all these humility is the most difficult to hold on to once some experience is accrued yet it is what separates mediocracy from greatness because to lose humility is to lose the ability to learn.

So why was experience so lacking in my old organization were we all cocksure divas? No, it was quite the opposite actually. The culprit, for me, was the environment itself: testing was slow and cumbersome deterring experimentation, the platform system was extremely complicated and difficult to grasp, the documentation was "centrally" stored in multiple hard to reach locations and what is even worse was that a lot of the additional information (tutorials, presentations, descriptions) were not available as different individuals, teams, contractors and companies either guarded their "secrets," protected their work, or plainly were too busy to share.

This would be bad enough but adding the fear factor to the equation turned what was a bad dream into a full blown nightmare for experience building. In an economy where unemployment for youngsters reached 40% veiled termination threats were weaved into every discussion concerning human error rates which created an atmosphere of dread. In this atmosphere everybody tried to minimize their risks. There was an aversion to learning new skills, copy/paste programming was encouraged and new code was mostly shunned, estimations were based on formulas, while constant code peer reviews and scatter-shot testing strategies brought code production to a halt. Not only were we producing little to no value while working our fingers to the bone but we did not learn anything from it. Mistakes were not allowed, so they were swept under the rug.

Experience building is expensive because experience loves mistakes, but in order to learn from them we must be unafraid to assume responsibility, take ownership of error, document it, analyse it without prejudice and share the results. With the dread of error looming over our heads to take responsibility for our mistakes was an act of madness. In fact it took me accepting that "one day, I will quit this job" to allow me to spread my wings, try some crazy solutions and learn each day more than I was learning in a month under the veil of fear. How ironic that all this experience/value was being added after the decision that made it a temporary gain for the organization itself. In other words, by deciding I will leave the company I became a better employee.

All these thoughts are swimming in my head as I struggle to grasp the intricacies of building an Android app. I think they came to me because my schedule is slipping constantly, I refactor and alter the app's architecture every week, I strain to learn new concepts and I question my own competence all the time. I was trying to figure it out until it struck me, I am not incompetent I am just inexperienced. But I have no fear so I'll be fine.

Till next time,

Stratos out.