I worked as a Systems dev at Microsoft for a while.. There were many lessons to learn there - about software in general, and large systems in particular. I realized today that some of the knowledge can be applied to systems in general. This is a post on that...
Most of the software we know today involves a lot of code. Sometimes it is millions of lines of code. Needless to say, no one person can understand the entire thing. Thats natural, since our human brain can only handle a fixed set of things at a time. And that number is definitely not a million. So what is done is that a number of people work on these problems, and each person understands what he or she does and tries to enhance and modify that module. It is however important that the person modifying this piece of code has at least a basic understanding of all the ways in which whatever she is changing can affect other parts of the system. And that is hard, given the size of the problems we work on.
Now consider what happens when a new dev starts to work on such a system. They have possibly never worked with a new system their entire life (being fresh out of college). Most often they are therefore asked to fix bugs. Often, they end up making various mistakes while fixing the bug, and sometimes end up generating more bugs than what they really fix. The basic problem here is that they dont understand the system. Most bugs require change in less than 10 lines of code. However, what looks easy initially is so difficult, because one has to make sure that the changes to these 10 lines of code is not causing anything unexpected for the every other line of code in the system. (Sometimes in a well understood and modular system, your 10 lines of code will not really affect all the other lines... but the point is, your system has to be well understood for you to be able to say that there is really no effect.) This is hard to do, and most often, the effect is that new devs start introducing bugs while developing.
Microsoft (as do most other companies) has a stringent code review process in place, wherein every single change that is made to the system is monitored by someone who understands the system as well as the implications of the changes made to the code.
Now that the premise is set, it is time to come to my main argument. Earth is a very large biological system. Our knowledge of it is mostly basic, since we only understand how each part works separately, and have only just begun to understand how all the parts work together. We are just making baby steps with genetic engineering, and I think it is reasonable to say that we are new devs in that area. We definitely dont understand the entire system (having only recently sequenced the entire human genome). But despite this lack of understanding, we are tweaking the system in little ways here and there. And from my knowledge of systems, there is only one result for such kind of tweaking without full understanding - bugs. And worse sometimes - build breaks.
What I am saying is that our knowledge of genetic engineering is still too nascent for us to apply it to anything that really matters. Before a software company releases a product, they subject the product to a huge slew of tests. And that is not always random testing, there are analyzers that specifically look at every line of code that has gone into production to see if there are problems that can be caused once it is deployed. If one were to employ the same strategy for genetic engineering, there would have to be tests that checked what the effects of their modification was - long term. Considering the size of the biological system, and because effects take much longer to manifest, this would mean a very long period of testing.
However, we are already mass producing genetically modified crops and selling it to the masses, without looking at the implications it can have on the entire system. Each lab that works on these is independently modifying the system, without considering what the effects of its modification might be. There is no central authority who understands and can certify that the proposed change will not cause a problem to the system (code review). In all, we are sort of blindly modifying a very large and somewhat delicate system without understanding what we are doing. And my hunch is that this strategy is going to cause some serious problems (bugs) which are going to surface perhaps ten or twenty or perhaps even a hundred years later.
Extending the analogy:
One can extend the analogy further. A large software system usually has small parts that largely decide the most basic functions of the system. This "heart" will be what most of the rest of the system will be built on, and changing the heart is risky, because it is going to affect the entire behavior of the system. For example, the kernel in an operating system forms a kind of "heart" of the operating system. Thus making even small changes in the kernel code can cause drastic changes in the OS, resulting in effects ranging from a kernel that cannot boot to something that shuts down mysteriously. All of that happens when one does not understand what one is doing. (At Columbia, there is a course on OS which requires that we make changes to the kernel. And that is reputedly the toughest course in the CS dept. most likely because it is hard to understand the effects that changing this piece of code has on the system.
Here is the argument on genetics : Genes control everything, apparently. We understand that much. Thus genes sort of form the heart of the biological system. So essentially, with genetic engineering, what we are trying to do is modify kernel code for the biological system without understanding it. And the effects of this is going to be much larger than anything we tried earlier, because earlier, we had not started kernel hacking.
Now... does that give you a lot of confidence about eating genetically modified food stuffs?
Wednesday, June 23, 2010
An algorithm for mess allocation.
Applied algorithms for a very real world problem :
Report on new Mess Allocation Algorithm:
The two objectives of the algorithm are
• Efficiency
To ensure that maximum number of people get their desired choices
• Equality
So that over time, the maximum regret of any individual is minimized
The Algorithm
The algorithm tries to make sure that maximum people get their first choices.
However in case of tie algorithm gives preference to those who register
first.Among those who are left out (from getting their first choice), we try
to allocate the second choices to the maximum number again. The step is
repeated for the third, and fourth choices too. If any (at all) are still
left, they are randomly allocated.
How It Works
The algorithm has six rounds.
In the first round, allocation is done for a student only if his first choice
is available. For each caterer, students who opted for that caterer as their
first choice is allocated to the respective mess, as long as the number
allocated to a mess does not exceed its capacity.
If the number of students who opted for a particular mess as their first
choice exceeds the capacity of the mess, the students with higher regrets are
given more priority. Among students having equal regrets, the time of
registration is given priority.
In the second round, the same procedure is repeated. Only this time, we
concider the second choices of those students who have not been allocated in
the first round. A student gets allocated to a mess only if his second choice
is available. Students are prioritized in the decreasing order or regrets, and
based on time of registration among students having equal regrets.
Similar, third choices of those who remain unallocated are concidered in the
third round, and fourth choices of those who remain unallocated are concidered
in the fourth round.
If any student who registered for mess did not get allocated in the first four
rounds, he is randomly allocated to a mess in the fifth round. Students who
did not regsiter for mess are allocated randomly in the sixth round.
System Of Regrets ( Subject to change for next month )
Each student has a value called regret stored against him/her in the database.
This value is a measure of how much disappointed he/she would be by the
allocation.
Initially (in the first month of allocation), the regrets of all students will
be zero.
If a student is allotted to the mess of his first choice, + 0 is added to his
regret
If a student is allotted to the mess of her second choice, + 1 is added to his
regret
If a student is allotted to the mess of his third choice, + 2 is added to his
regret
If a student is allotted to the mess of his first choice, + 3 is added to his
regret
For those dining in Himalaya, if a student does not get any mess of his choice
(if he gets randomly allotted to a mess even after registering as none of his
4 choices were available), +4 is added to his regret.
The regrets of each student keep on accumulating every month. Now, higher the
regret of a student, the more he is disappointed. To ensure equality, a
student with higher regret is given priority over a student with lower regret
during allocation.
Conclusion
The new algorithm is found to be much more efficient that the one presently
being used which is solely based on first come - first serve principle.
> Maximum number of students gets their first choice. Among those remaining,
it attempts to maximize the number that gets second, third, and fourth choices
too (if any).
> The regrets of students over time are balanced in an optimal way
Regards
{ New Mess Allocation Algorithm:
Developed by Hareesh R, Institute Web Operations Core 2009 – 2010 }
Report on new Mess Allocation Algorithm:
The two objectives of the algorithm are
• Efficiency
To ensure that maximum number of people get their desired choices
• Equality
So that over time, the maximum regret of any individual is minimized
The Algorithm
The algorithm tries to make sure that maximum people get their first choices.
However in case of tie algorithm gives preference to those who register
first.Among those who are left out (from getting their first choice), we try
to allocate the second choices to the maximum number again. The step is
repeated for the third, and fourth choices too. If any (at all) are still
left, they are randomly allocated.
How It Works
The algorithm has six rounds.
In the first round, allocation is done for a student only if his first choice
is available. For each caterer, students who opted for that caterer as their
first choice is allocated to the respective mess, as long as the number
allocated to a mess does not exceed its capacity.
If the number of students who opted for a particular mess as their first
choice exceeds the capacity of the mess, the students with higher regrets are
given more priority. Among students having equal regrets, the time of
registration is given priority.
In the second round, the same procedure is repeated. Only this time, we
concider the second choices of those students who have not been allocated in
the first round. A student gets allocated to a mess only if his second choice
is available. Students are prioritized in the decreasing order or regrets, and
based on time of registration among students having equal regrets.
Similar, third choices of those who remain unallocated are concidered in the
third round, and fourth choices of those who remain unallocated are concidered
in the fourth round.
If any student who registered for mess did not get allocated in the first four
rounds, he is randomly allocated to a mess in the fifth round. Students who
did not regsiter for mess are allocated randomly in the sixth round.
System Of Regrets ( Subject to change for next month )
Each student has a value called regret stored against him/her in the database.
This value is a measure of how much disappointed he/she would be by the
allocation.
Initially (in the first month of allocation), the regrets of all students will
be zero.
If a student is allotted to the mess of his first choice, + 0 is added to his
regret
If a student is allotted to the mess of her second choice, + 1 is added to his
regret
If a student is allotted to the mess of his third choice, + 2 is added to his
regret
If a student is allotted to the mess of his first choice, + 3 is added to his
regret
For those dining in Himalaya, if a student does not get any mess of his choice
(if he gets randomly allotted to a mess even after registering as none of his
4 choices were available), +4 is added to his regret.
The regrets of each student keep on accumulating every month. Now, higher the
regret of a student, the more he is disappointed. To ensure equality, a
student with higher regret is given priority over a student with lower regret
during allocation.
Conclusion
The new algorithm is found to be much more efficient that the one presently
being used which is solely based on first come - first serve principle.
> Maximum number of students gets their first choice. Among those remaining,
it attempts to maximize the number that gets second, third, and fourth choices
too (if any).
> The regrets of students over time are balanced in an optimal way
Regards
{ New Mess Allocation Algorithm:
Developed by Hareesh R, Institute Web Operations Core 2009 – 2010 }
Tuesday, June 1, 2010
Sri Sri Ravi Shankar
Its amazing how gullible one can be at certain points in life. Its also amazing how important it is to have parental support in such times.
When I was around 20, I attended an art of living program in bangalore. I must admit that I sort of did it out of ignorance. I thought it was going to be some kind of yoga, it turned out it was sort of an opportunity for the group to identify young and gullible people and lure them into the group. By the third day of the course it became clear to me that I was never going to agree with the things they said or did - some of it involved bowing down and treating their guruji like God. In fact, they advocate that their Guruji is God.
I went home feeling confused, because it seemed like they were right about everything. It took all the strength I had and some pressure from parents to not go for subsequent meetings. The fact that one of the coordinators / organizers kept calling every other day to ask me to come to one of their satsangs did not make it any simpler. Finally, they started sending me their magazine without me even paying for it! The magazine as such was alright, although at times I felt that there was some underlying problems with what they said. I could never pinpoint what it was that felt wrong, so I kept wondering for a long time..
Then one day I met this coordinator and found out that he had flunked out his PhD program and had to transfer to a neighbboring institution. I was a little surprised, because he was one of those "Guruji will take care of everything if you do his work..." kind of people. I guess I was surprised that Guruji did not take care of his things... He continued to work for his Guruji, it looks like.
Another one was when I found these people circulating emails asking people to protest against putting flags at half mast when the Pope died. In my mind, a religious group should not have done that, given that a god should respect representatives of God, wherever they were found. But that was not what happened. To me it looked definitely like propaganda Hindutva. Worse, I heard from someone that Ravi Shankars followers were behind the attacks on Benny Hinn. So it was that I decided this was all a scam. But long after that, although I was relieved that I didnt join in, I didnt think it was per se a "cult"
Now, years later, some blogs have surfaced that show that the groups activities are not really what they claim to be. The blog is maintained by an aol teacher who left the org out of disillusionment.
I sometimes think about how close I was to believing it. How effective brain washing can be. I wonder how they were so successful at brain washing so many well educated and apparently thinking people. The most surprising thing is the number of people they have managed to lure to their courses. Although very few of them go back to them or join them, it is impressive that they have managed to make most people take at least one basic course. I guess that speaks volumes about how well trained their teachers are in catching people. I noticed that to places that contain educated people, they send their best and brightest. For example, to speak at a place like IISc, they sent someone who had similar branding. They had a few people with these brands, and effectively utilized them to impress others. A number of students from places like IISc keep going for Satsangs from then on. Perhaps the other aiding factor is that all of them have unfinished PhD thesis to complete which they are worried about...
In any case, the teachers they send manage to be fairly effective people snatchers. Once they get your email id and phone number, they keep pestering until you agree. And it is done in such a lovely way, one will not realize it was even cajoling. I wonder what keeps people there. The breathing exercises dont really do much I think... I practised it for a while and found no big difference, except that I was losing about one hour in the morning every day. I dont think it had enough benefit to warrant that.
An interesting thing I found in IIT : I went once for a yoga class and found that the lady there had exactly the same attitude. The last day of class was Palm sunday, and I told her the day before that I would have to leave early the next day. A friend of mine had agreed that she would ring my cellphone when it was time for me to leave for church. Naively, I told this lady that I was expecting this call and would leave when it came. I also told her that I was leaving to attend church, trying to impress the urgency of the situation on her. She reacted in a way that irritated me quite a bit. She told me to go into meditation, promising that she would make me leave when the call came, and said she would keep my phone. Believing that, I tried to get to meditation... I was however very conscious of what was going on around me, possibly because in the back of my mind the only thing I knew was that it was a day of obligation and that I should be getting out of there. My phone rang, and she picked it up, and told my friend I would be out soon. I kept waiting and waiting, pretending to be in meditation, but she did not come and tell me that the call had come. I was a little flustered, and tried to get up, but she insisted that I continue. By now, obviously I was not meditating at all, and just wanted to get out of there. She however wouldnt let me, and made me sit there the whole time until that exercise was over. After that when I said I wanted to leave, she insisted that we eat the food that she had prepared. I tried to stuff myself with something and get out, but she tried to delay me as much as she could, insisting that sitting there was the same as sitting in church! I was astonished that she had tried to influence me and my actions... Needless to say, that was the last time I went there. I wonder what it is in people that makes them want others to follow only their path.
When I was around 20, I attended an art of living program in bangalore. I must admit that I sort of did it out of ignorance. I thought it was going to be some kind of yoga, it turned out it was sort of an opportunity for the group to identify young and gullible people and lure them into the group. By the third day of the course it became clear to me that I was never going to agree with the things they said or did - some of it involved bowing down and treating their guruji like God. In fact, they advocate that their Guruji is God.
I went home feeling confused, because it seemed like they were right about everything. It took all the strength I had and some pressure from parents to not go for subsequent meetings. The fact that one of the coordinators / organizers kept calling every other day to ask me to come to one of their satsangs did not make it any simpler. Finally, they started sending me their magazine without me even paying for it! The magazine as such was alright, although at times I felt that there was some underlying problems with what they said. I could never pinpoint what it was that felt wrong, so I kept wondering for a long time..
Then one day I met this coordinator and found out that he had flunked out his PhD program and had to transfer to a neighbboring institution. I was a little surprised, because he was one of those "Guruji will take care of everything if you do his work..." kind of people. I guess I was surprised that Guruji did not take care of his things... He continued to work for his Guruji, it looks like.
Another one was when I found these people circulating emails asking people to protest against putting flags at half mast when the Pope died. In my mind, a religious group should not have done that, given that a god should respect representatives of God, wherever they were found. But that was not what happened. To me it looked definitely like propaganda Hindutva. Worse, I heard from someone that Ravi Shankars followers were behind the attacks on Benny Hinn. So it was that I decided this was all a scam. But long after that, although I was relieved that I didnt join in, I didnt think it was per se a "cult"
Now, years later, some blogs have surfaced that show that the groups activities are not really what they claim to be. The blog is maintained by an aol teacher who left the org out of disillusionment.
I sometimes think about how close I was to believing it. How effective brain washing can be. I wonder how they were so successful at brain washing so many well educated and apparently thinking people. The most surprising thing is the number of people they have managed to lure to their courses. Although very few of them go back to them or join them, it is impressive that they have managed to make most people take at least one basic course. I guess that speaks volumes about how well trained their teachers are in catching people. I noticed that to places that contain educated people, they send their best and brightest. For example, to speak at a place like IISc, they sent someone who had similar branding. They had a few people with these brands, and effectively utilized them to impress others. A number of students from places like IISc keep going for Satsangs from then on. Perhaps the other aiding factor is that all of them have unfinished PhD thesis to complete which they are worried about...
In any case, the teachers they send manage to be fairly effective people snatchers. Once they get your email id and phone number, they keep pestering until you agree. And it is done in such a lovely way, one will not realize it was even cajoling. I wonder what keeps people there. The breathing exercises dont really do much I think... I practised it for a while and found no big difference, except that I was losing about one hour in the morning every day. I dont think it had enough benefit to warrant that.
An interesting thing I found in IIT : I went once for a yoga class and found that the lady there had exactly the same attitude. The last day of class was Palm sunday, and I told her the day before that I would have to leave early the next day. A friend of mine had agreed that she would ring my cellphone when it was time for me to leave for church. Naively, I told this lady that I was expecting this call and would leave when it came. I also told her that I was leaving to attend church, trying to impress the urgency of the situation on her. She reacted in a way that irritated me quite a bit. She told me to go into meditation, promising that she would make me leave when the call came, and said she would keep my phone. Believing that, I tried to get to meditation... I was however very conscious of what was going on around me, possibly because in the back of my mind the only thing I knew was that it was a day of obligation and that I should be getting out of there. My phone rang, and she picked it up, and told my friend I would be out soon. I kept waiting and waiting, pretending to be in meditation, but she did not come and tell me that the call had come. I was a little flustered, and tried to get up, but she insisted that I continue. By now, obviously I was not meditating at all, and just wanted to get out of there. She however wouldnt let me, and made me sit there the whole time until that exercise was over. After that when I said I wanted to leave, she insisted that we eat the food that she had prepared. I tried to stuff myself with something and get out, but she tried to delay me as much as she could, insisting that sitting there was the same as sitting in church! I was astonished that she had tried to influence me and my actions... Needless to say, that was the last time I went there. I wonder what it is in people that makes them want others to follow only their path.
Subscribe to:
Posts (Atom)