Explore Sign in Sign up
Echo Reply
Computers, Science, Technology, Xen Virtualization, Hosting, Photography, The Internet, Geekdom And More

The Engineering Panic Room

Published on Nov. 3, 2010 at midnight by XC

Have you ever been in a situation where you were forced to act knowing that the margin for error was zero? I think that most of us have, perhaps just avoiding an animal on the road without swerving into oncoming traffic while driving. In a crisis, the most appropriate response usually comes not from thinking, but simply allowing reflexes to take over.

If you are a software engineer, you will enter what I call the panic room whenever your reflexes say run like hell and doing so is not a possibility. You have three months to rescue something that has been languishing for three years and you know that there is zero room for mistakes. Every line of code you commit must be your friend next month, going back to the drawing board is not going to be an option. It is a recording with one take, every single thing you ship gets carved into stone.

What exactly is the panic room? It is a place where smart people go to observe everything and play with possibilities to try to determine what course of action is going to be the correct course to take. It is the process of removing yourself from the problem in order to more effectively solve it, while everyone else sees you wasting time and not committing anything. It is not your happy place, it is your safe place and you need to be prepared to go there and seal the door. You might wait in the panic room for a while, so you better make sure it is stocked.

Category: Programming
|
Read More

No More Favors To Internet Explorer

Published on July 2, 2010 at midnight by XC

This is a short rant. I am not going to continue to work around a browser that does not adhere to standards, no matter how many people use what I wrote. If this site looks awkward for you, get a standards compliant browser. I am also officially ignoring any bug reports pertaining to stuff that I wrote not working correctly in various versions of Internet Explorer. I’d rather just issue you a refund than add kludges that will break in the future, only to add more kludges.

Try:

Firefox
Google Chrome
Opera
Konqueror
Or even lynx might be a better experience.

We have standards, the company that wrote Internet Explorer choses to ignore them, hence I won’t support it. Note, I’m not demeaning Lynx, I use it avidly.

Category: Programming

Duplicating The Guts Of Giants

Published on July 1, 2010 at midnight by XC

We’re all impressed and fascinated with the inner workings of Stack Overflow. Its one of the few web based services that has actually impressed me. Every single button, link, label or sprite belongs exactly where it is. The tools provided are exactly adequate for users to accomplish what they want to do, no more – no less. That’s not surprising, the people who built the site have turned software usability into its own religion.

Its one of the few sites that I’ve ever seen that did not go overboard or too stringent with the use of AJAX. How do you know a great user interface? It doesn’t get in your way.

Breaking SO down into its basic useful and novel elements we have:

A voting system that catches and reverses most ‘fraud’ votes
A tagging system that scales well and is lightning fast
A revision control system that allows a convenient grace period
Algorithms to reward users for participating positively
A reputation system that lets SO trust each user up to a certain degree
A search built especially for ninjas

All of these are useful on their own outside of the context of Q&A, with the possible exception of the search. Again, I’m not dissing the search, just the results. Still, you could do very well to implement any number of these features into an application designed to let people find what they want with little effort. I’m trying to duplicate some of those building blocks, to a degree in a way that they can be re-used.

Category: Programming
|
Read More

Pygame Is Leading Me To Python

Published on April 3, 2010 at midnight by XC

I can’t help but talk about how great I think my highly modified version of Ubuntu is with my friends. I’ve got virtualization galore, a great development environment with tool chains for every architecture you would want, a nice blend of Gnome and KDE apps, good security and just the right mix of desktop effects. I like it, people who use my computer like it and I’m often asked by a friend to compile xyz for abc architecture.

Someone asked me a rather innocent question, “This looks great, what games did you write for this?” and I came up blank. None. Zero. Zilch. There I am boasting how easy it is to do practically anything with my setup and I could not produce one game, however trivial to show. I’m almost embarrassed to admit, beyond working through simple programming problems that resemble games (puzzle solvers, etc), I have never written even a simple poker game.

No problem for me, I know C! Wrong. I spent a few hours coming up with a design for a game and ended up just settling on a clone of The Bard’s Tale I (C-64 Version). I fired up my editor and started coding right away. My hands did not leave the keyboard for hours. I had the structures for the characters and inventory, various types of monsters, spells for the spell casters and then it hit me, I had to bring it to life with a fun interface. A few hours into that, I began thinking … “Isn’t this supposed to be FUN?” The thought of Windows portability crept in and I began seeing ninjas scale my building.

Category: Programming
|
Read More

Fog Creek Nailed Enterprise Mercurial Needs With Kiln

Published on March 22, 2010 at midnight by XC

Today, purely out of boredom, I created a trial Kiln account. Fog Creek is well known for pampering programmers, I wanted to see how they built an offering out of Mercurial. If you are reading this, there is a 90% chance that you found my site via Google while searching for something Mercurial related. Yes, I love HG, I’m one of the people who still, to this day, don’t quite ‘git‘ it.

Selling the idea of using Mercurial to enterprise customers is sometimes difficult, especially dealing with start-ups who have managers that only know how to court venture capitalists. These managers are often put in charge of operations and justifiably rely on, even inherit the preferences of their trusted senior developers. By the time I get called in, all titles are taken. I’m just a measly outside contractor that yaks and horks hairballs at the sight of Subversion and bleeds from all three eyes when confronted with Git. I’m not going to go into Kiln yet, I’m just going to describe what typically gets Mercurial bounced like a superball right out of the room when I bring it up.

Category: Programming
|
Read More

Breaking With Conventional Wisdom – Sometimes

Published on March 6, 2010 at midnight by XC

Despite standards, industry accepted best practices, free templates for planning networks and even common sense, not everyone can follow conventional wisdom when deploying a cloud. In fact, at least in my experience, the only times I have been able to do it ‘correctly from the start’ have been when I start completely from scratch. Most people want a relatively painless path to upgrading, not an investment in a whole new infrastructure.

This means, business nodes and storage nodes aren’t always going to have the luxury of a private interconnect. In fact, every single server in a farm may be on its own vlan, which kind of rules out things like OpenAIS, OpenMPI or anything else that needs multicast to work. Then we take the most uniquely challenging (some even say god forsaken) industry known to man, IAAS, and add that to the mix. Its very typical for successful hosts to start with just a few cheap servers completely oblivious to one another and just continue to grow in that direction.
Using tools like Xen, AoE, iSCSI, ZFS, LustreFS, Hypertable and others, you can typically reduce a host’s server footprint by 45 – 55% while raising their available capacity to approximately the same metric. As transcendent memory evolves in Xen, the metric will only increase. Additionally, with the cost of infiniband dropping, the ideal ‘poor man’s’ SAN rapidly becomes a tenable strategy even for basement based providers.

Category: Programming
|
Read More

What I Get Out Of Stack Overflow

Published on March 6, 2010 at midnight by XC

I received a very interesting e-mail today from a reader who noticed the little box on this blog that displays my participation with Stack Overflow. I’m not going to paste the entire contents of the e-mail, however the sender did bring up some interesting questions:

Isn’t it like Facebook, just programming themed?
I heard editors gang up on people like on Wikipedia
Why don’t you just join [xyz] forum or newsgroup?
What do you get out of it if you just answer lots of questions?

The questions clearly illustrate that some people spend more time listening to the rumor mill than they do actually investigating things to their own satisfaction. Why not just go look at the site and explore it? To my knowledge, there is only one serious and hidden conspiracy. Stack Overflow is nothing like Facebook, or any other social network. There is no means to ‘add friends’, send private messages, or anything else you would expect from a site that connects people with other people. Stack Overflow serves two purposes:

It connects programmers to knowledge that they seek
It connects programmers to questions that welcome their knowledge

There are trivial rewards for participation, whichever way you use the site. You gain a numerical representation of how much the site trusts you and ‘badges’ like scouts earn. Reputation has only one intended purpose, it allows the system to trust you more and give you broader access to its facilities. An interesting side effect is that it encourages participation. This is how a 100% community run site can exist without having to deal with constant vandalism. Sure, if you use the system long enough you’ll begin to see the same people answering your questions (or questions in topics you watch) and you may even get to know them a little in the process. However, if you want to chat privately, you’ll have to track them down and contact them on your own.

There are no editorial cliques on Stack Overflow. There could not be, they would obviate themselves too quickly to the moderators that the community appoints. Anyone with a suitable reputation can edit anything. Moderators serve to resolve edit wars, take on janitorial duties and (once in a blue moon) police offensive user profiles. “Offensive” on Stack Overflow is generally limited to hate speech, something illegal, something obviously deliberately obnoxious or blatant SPAM.

I don’t like forums, for the most part, if ‘forum’ to you means something running PHPBB or similar. I have nothing against PHPBB, I just don’t care for long, paginated threads where only a few posts have something useful and relevant to say. Stack Overflow lets you sort answers by votes which is extremely helpful. If I’m hunting down a cryptic error that GCC threw while speaking in tongues, I’d like to get to the helpful stuff immediately. Going through 8 pages of replies where four of them are users just agreeing with other users is far too mentally taxing to keep straight when you are already focused on something else. I also don’t like being at the whim of a single (possibly sober) moderator, especially when I make links to my own code repositories to help illustrate my answers.

I do use newsgroups, comp.lang.c, comp.lang.c.moderated and several others. At least with C questions, you’ll receive several direct answers then get CC’d on a two week long argument on which one was the most ‘correct’. Usually, after 48 hours, the thread degenerates into a completely different topic and completely different argument. Great, 100 people arguing over c-style really helps when you just need to fix something.

As far as what I get out of it, I like answering questions more than asking. I also like to spend a little time each week editing and organizing things. Why? For the same reason that so many people like working on Wikipedia. Stack Overflow is a great resource, in fact more and more, if you Google a programming problem you will likely see links to Stack Overflow in the first page results. This is for two reasons, people ask great questions and great editors make sure the titles reflect the question accurately.

I’ve heard from a few people that a reputation score of over 5000 helps when looking for a job. This may just be their experience. However, reputation is an indication of how your peers view your answers as well as your communication skills. Stack Overflow implements very stringent ‘cheating’ checks, to make sure that no organized effort robs someone of their reputation, or artificially inflates another’s. I would really frown upon a hiring manager basing even part of their decision on Stack Overflow reputation, some people have more time to use the site than others.

If Bob and Jane are equally skilled in programming and both excellent communicators, yet Bob took a year off and spent two hours a day answering questions on Stack Overflow while Jane was busy at a full time job … well … you can see the problem. Its great if you have it to add as a hobby on your resume or show on your web site, but leave it at that.

Finally, though Fog Creek Software (the people behind Stack Overflow) are very Windows centric, Linux/GNU questions are very welcome there. At least twice each week I see someone asking about writing Linux device drivers and by the time I open the question its already been answered by someone who knows what they are doing. Additionally, questions regarding gcc, flex, bison, valgrind, gdb and all of the other tools that most *nix programmers use are typically answered very quickly. I’ve also seen quite a few licensing questions answered competently, some of them by me.

Yes, they show ads, so does Sourceforge. Quite a few of the ads on Stack Overflow are actually promoting free/open source software projects. If someone could convince me that SO did harm, I’d still be able to say that they do a lot more good than harm, why should they not profit in the process? The Stack Exchange business model is not only ethical, its tenable, which is rare to find in the wild west of information brokering.

Just watch out for the search, though I heard that they are working on it. Additionally, meta is a very strange place.

I have noting to disclose in this post, I wrote it on my own accord in response to a reader’s question. I am not compensated by Stack Overflow or Fog Creek Software in any way.

Category: Programming

Stripping A File != Optimization

Published on Feb. 25, 2010 at midnight by XC

It is a common misconception that the “strip” command does some kind of magic optimization to an executable resulting in it running faster. Its also a common misconception that the smaller an executable is in size, the less memory it will consume.

Neither case is correct. I could write a program in 10 lines that allocates and leaks all available memory. Stripping all symbols from that executable (including debugging) just makes it harder for someone to figure out why that program is behaving so badly. The end result is now you have the same buggy code in a slightly smaller package, but now its much more difficult to debug.

Stripping a file is useful in two instances:

You have very limited storage space, common when discussing hand held devices. Bash’s symbol table occupies more room than any three stripped programs put together, so you would naturally want to strip them in order to make everything fit (or, compile bash differently)
You are releasing proprietary software and want to conceal what you link against, as well as the symbols you use. In this case, you often see huge statically l inked executables (bad idea) stripped to compensate for the bloat of static linkage. In this case, its just better to run the code through obfuscation prior to compiling it, then strip the symbols for good measure. You then take advantage of the same libraries already in shared memory, while hiding your already obfuscated symbols.

Category: Programming
|
Read More

Atomic Operations In User Space Using Gcc Natives

Published on Feb. 21, 2010 at midnight by XC

I have seen no less than four questions this week on Stack Overflow regarding facilities for atomic ops on Linux (and portably speaking, anything else that hosts GCC). Yes, gcc does have some facilities for this, take a look at this rather good overview.

Unfortunately, these don’t seem to work on most ARM platforms, which tend to only exaggerate just how slow the typical “fast mutex” can be on hand held devices. Because of that, I have been eschewing global counters / flags / etc as much as possible to keep time spent acquiring locks to a minimum. I was hoping to see some answers on the Stack Overflow questions that lead in another direction, especially for ARM, but no love as of yet.
Still, on almost every other platform, these are very handy and worth noting. You can use them to effect any operation you would normally do on integer types of various sizes without wrapping the operation in a mutex like a scallop in bacon
Unfortunately, you do have to wrap them in portability kludges, which is what I’m doing now. I’ll post the code in a new HG once I get it working. It will use sync_fetch_and*() if gcc is used and the platform is agreeable, otherwise resort to an ordinary arbitrated mutex of whatever type the system provides. I spent a great deal of time looking for something like this that can plug right into autoconf, but I did not find it.

If you happen to know of something that exists, please leave a link in comments to keep me from re-inventing yet another wheel.

Category: Programming

Gcc Easter Eggs – Shorter Enum Storage

Published on Nov. 9, 2009 at midnight by XC

This is the first in what will likely be a long series of posts regarding features in GCC that I find to be less than obvious. Technically, an easter egg (in a program) is some sort of functionality that accessed by undocumented and non-obvious means.

To the less than guru regarding GCC, a lot of switches seem to fall into that category, though it is one of the best documented compilers ever written. In fact, a lot of people that I know keep their own cheat sheet to document handy bits that they find for future reference. This new tag happens to be my list, hope you find it useful.

On to it – I saw an interesting answer to a question today regarding what storage type an enumerated list actually corresponds to on any given platform .. and what control (if any) one may have over it.
Typically, at least on Linux, an enum is going to boil down to a 16 bit integer. That really sucks if your list contains the values of 0, 1 and 2.

Not to worry, GCC’s -fshort-enums switch is there to help:

Allocate to an “enum” type only as many bytes as it needs for the declared range of possible values. Specifically, the “enum” type will be equivalent to the smallest integer type which has enough room.

Thanks to Michael Foukarakis for that little tidbit. This lets you cut the size of a typical enum in half (i.e. 8 bit storage) without having to revert to preprocessor abuse.

That’s really helpful, especially dealing with some kind of line protocol / device that gives 8 bit responses, so much easier to pass them around using an enumerated list.

Category: Programming