That was the question. At least for one project I have worked on. Contrary to Hamlet I have found an answer to this question but I have also learned that finding the answer does not (always) help. For those of you who just want to know who the killer is the answer was yes – but your mileage may vary.
For those of you who understand that the journey is more important than the prize you must know that the journey was pretty difficult. The main reason for that was the one thing I don’t like in connection with technology or science: fashion. Unfortunately fashion is a key factor in science and technology and if you don’t agree just think antibiotics: when they were discovered it was a truly remarkable achievement and they started being used with great joy; if you were ill, hungry, thirsty or just feeling bored you could get some antibiotics to cheer you up. Nowadays it’s sometimes more difficult to get some antibiotics than it is to get cocaine. How did that happen? It was just another fashion that came and went.
The best part about fashion in science and technology is that it’s always sustained by a rather complicated, logical and witty technical argumentation. That’s what happened on that particular project I have worked on: someone had the idea that relational databases are obsolete. They even talked to some guys that were specialized in databases, scalability or whatever that was; anyway they were obviously very smart people unlike the database supporters. I unhappily and unwillingly was of course a database supporter on that project.
When you try to support a point of view like this you just take the weak points of the adversary and maximize them and get all your weak points and minimize them. Then you add some catch phrases preferably with famous examples such as “Twitter doesn’t scale”. It was a mere reminder for all those people still trying to recover relational database supremacy that some people much bigger than you were brought down by this unworthy database machinery. And sustaining something fashionable you will only find cheerful people to support you just because it sounds good.
But what do you replace the relational database with? Well… let’s not hurry things and jump important steps. After all it’s the journey that matters the most. First you should try to search on Google for “scalability” or “scalable” or whatever comes into your mind. You will surely find lots of projects that are designed to be scalable or “with scalability in mind”. Then you make a list of all the names of those project and can use them in your next arguments. They might not be “exactly” what you want but they will surely make you look good because this shows that you are a true specialist: you know everything about this field.
The result for that project was waste of time and money and poor performance. In the end they reluctantly agreed that I would somehow shard the database. I did it in a hurry just so we would have something that would work. We now had a “hybrid” solution that “just worked” but which would hardly scale and which had pretty bad performance. Nevertheless the middle management reported good results not because of my “temporary” solution but in spite of it. The wasted time was regarded as “research and development” time with an emphasis on “research”. If there was ever any wasted time or money at all they were surely not responsible for it.
If you understood that database sharding is a must for your project then you should probably read it all again. I’m not an advocate of sharding or of relational databases or of any particular technology at all. I’m an advocate of honesty, good design and architecture right from the start and of the right technology at the right place in the right amount. That’s what I always try to do on the projects I work on. Unfortunately this is not always understood or appreciated – the “fashion people” are just more appealing and who’s more appealing has more chance to be considered to be right by the people that lack technical expertise. I lost time, money and credibility by not compromising and by not becoming one of the “fashion people” but I just hope that I won’t change in these aspects.