Pencarian

Rss Posts

 

 

 

Berita pada kategori ‘Server, Jaringan & Keamanan’

MySQL Connection Timeouts

Apr 20, 2011

Sometimes on very busy MySQL server you will see sporadic connection timeouts, such as Can’t connect to MySQL server on ‘mydb’ (110). If you have connects timed in your
application you will see some successful connections taking well over the second. The problem may start very slow and be almost invisible for long time, for example having one out of million
of connection attempts to time out… when as the load growths it may become a lot more frequent.

If you time the connect you will often see connection times are being close to 3 and 9 seconds. These are “magic” numbers which I remember from years ago, which correspond to SYN packet being dropped during connection attempt and being resent. 3 seconds corresponds to 1 packet being dropped and 9 seconds correspond to two. If this is happening it is possible you have network issues or more likely you have listen queue overflow. You can check if it is the case by running netstat -s and finding something like:

38409 times the listen queue of a socket overflowed
38409 SYNs to LISTEN sockets dropped

This means some SYN packets have to be dropped because kernel buffer of connection requests on LISTEN socket is overflow – MySQL is not accepting connections as quickly as it needs.
There are 2 tuning places you need to consider if this is what is happening.
First – Linux kernel net.ipv4.tcp_max_syn_backlog This is size of kernel buffer for all sockets.
Default I have on my kernel is 2048 though it might vary for different versions, you might need to increase it to 8192 or so if you have intense connection. I’ll explain the math below.
Second – is MySQL parameter back_log which has default value of just 50. You may want to set it to 1000 or even higher. You may also need to increase
net.core.somaxconn kernel setting which contains the maximum depth of listen queue allowed. The kernel I’m running has it set to just 128 which would be too low for many
conditions.
Now lets look more into the problem and do some Math. First lets look into how MySQL accepts connection. There is single main thread which is accepting connections coming to LISTEN
sockets. Once there is connection coming it it needs to create a new socket for incoming connection and create a new thread or take one out of the thread cache. From this point on MySQL processes network communication in multiple threads and can benefit from multiple cores but this work done by main thread does not.
Usually main thread is able to accept connections pretty quickly, however if it stalls waiting on mutex or doing any other work such as launching new thread takes a lot of time you can have the listen queue to overflow. Lets look at the database which accepts 1000 of connects/sec in average. This is a high number but you can see ones even higher. In most cases because of “random arrivals” nature of traffic you will see some seconds where as much as 3000 connections come in. Under such conditions the default back_log of 50 is enough just for 17 milliseconds, and if main thread stalls somewhere longer than, some SYN packets may be lost.
I would suggest sizing your tcp_max_syn_backlog and back_log value to be enough for at least 2 seconds worth of connection attempts. For example If I have 100 connects/sec which means I should plan for 300 connections using 3x for “peak multiplier”. This means they should be set to at least 600.
Setting it to cover much more than 2 seconds does not make much sense because if client does not get a response within 3 seconds it will consider SYN packet is lost and will send the new one anyway.
There is something else. If you’re creating 1000 of connections a second to MySQL Server you might be pushing your luck and at very least you’re using a lot of resources setting up and tearing down connections. Consider using persistent connections or connection pool at least for applications which are responsible for most of connections being created.

Setting up replication with XtraBackup

Apr 19, 2011

I attended Vadim Tkachenko’s talk on XtraBackup during MySQL conference in Santa Clara last week. Backups are obviously very important, but the use case I had in mind is this:
Replicating a database that has Innodb tables in it, while keeping both master and slave on line if possible.
Tangent: by the way, I love the native backup utility that was once promised in MySQL 6.0, similar to SQL Server’s way of backup. It was like running “BACKUP myDb to DISK = ‘/backupDirectory/myDb.bak’” under mysql client, but I digress…
I have used mysqldump to accomplish this in the past, but I wondered how XtraBackup would fare in this task, especially after hearing Vadim’s talk and reading news on Percona’s development effort. To cut to the chase, this is my conclusion. Reproducing steps are listed immediately afterwards.
1. innobackupex provides a consisten database backup, spitting out log file and log positions in stdout, which is nice and useful for slave initiation;
2. It works with both MyISAM and innodb tables;
3. If MyISAM tables are all you have, just run innobackupex –prepare /directoryWhereBackupIs, and then move the database directory from under /directoryWhereBackupIs to under your slave’s datadir, then make the necessary group and owner change to said directory and its content files, and you are ready to run the “change master” command and start slave;
4. If the database has innodb tables, then in addition to step 3, you will also need to stop mysql on slave, move the ibdata1 file to datadir, then restart mysql, and run “change master…” and “start slave” commands. It does not matter if you are using innodb_file_per_table or not.
It will be nice if I can keep the slave up and running during this step when the database has innodb tables in it. Did I do anything wrong? Is there a better way? What if the slave has a database that has innodb tables and thus uses ibdata1 to begin with? What do you do then? Should I play with Tungsten’s replication? What are the compelling reasons to use Tungsten’s replication?
In any case, from my limited testing, I think I will use innobackupex for future replication creation tasks, if I can afford a mysqld restart. Overall, it feels a bit easier than mysqldump approach that I’ve been using in the past.
Here are the steps needed to reproduce:
1. Fire up 2 Rackspace CentOS 5.5 servers. Rackspace cloud servers beat Amazon EC2 servers hands down, in my view, for developing/sandboxing purposes;
2. Install the required mysql client, server, and XtraBackup on both servers;
3. Make /etc/my.cnf by cloning the sample cnf files under /usr/share/my-small.cnf. 3 minimum changes were necessary: log-bin=mysql-bin, server-id=a unique number, datadir=/var/lib/mysql. The first 2 are necessary for replication, the last is needed for innobackupex
Well, while you are at it, on slave, add in read-only and skip-slave-start if appropriate. That’s best practice for read only slave.
4. Add master server’s public key to authorized_keys on slave, to facilitate easy ssh connection.
5. On master, run this command:

innobackupex –databases=test –stream=tar /tmp/ –slave-info | ssh root@slave "tar xfi – -C /root"
When it finishes, you should see something like this:
110419 18:54:21 innobackupex: completed OK!
tar: Read 6656 bytes from -

Take note of 3 lines immediately above it, where it states the binlog file and log position, like this:

innobackupex: MySQL binlog position: filename 'mysql-bin.000002', position 2515

6. On slave, run this command:

innobackupex –apply-log /locationWhereBackupIs

then, assuming the database name is test, run the 2 commands below to change the group and owner to mysql:

chgrp -R mysql test
chown -R mysql test

move the directory under mysqld’s datadir:

mv test/ /mysql/datadir

If test database has innodb tables in it, stop mysql on slave, then copy ibdata1 to datadir, restart mysql.
7. On master, open up port 3306 if it is not already open, then create the replication account:

grant replication slave, replication client on *.* to repl@'50.56.121.%' identified by 'p@ssw0rd';

8. On slave, run:

change master to master_host='50.56.121.96', master_user='repl', master_password='p@ssw0rd', master_log_file='see output from innobackupex backup command on master', master_log_pos=numFrominnobackupexOutputOnMaster;

start slave;

show slave status\G

PHP on Azure and you in Vegas – Michelangelo van Dam

Feb 19, 2011

If you’re developing apps in PHP and you want a challenge that will get you some places, be sure to check out the PHP on Azure contest. Build an app with PHP, deploy it to Windows Azure and participate in a contest with a killer prize: an all-in trip to Las Vegas!

The rules are simple:

  • your app has to be written in PHP
  • preferably own development app
  • use as much as Windows Azure services
  • blog about overcoming this challenge (good/bad/ugly)

A jury will review your app and give you points on implementation, challenge and usage of Windows Azure technologies. But also on how you documented the process of deploying your app to Windows Azure.

Register before February 28, 2011. The contest itself runs from February 1, 2011 until May 15, 2011. If you register quickly, you can get attend a free Windows Azure training course given by Maarten “Mr. Azure” Balliauw on February 22, 2011.

Show the world your skills and participate. Full details of the contest can be found on http://phpazurecontest.com.

If you’re unsure how to start, I’m working on an example application to be deployed anywhere, including Windows Azure.

Roll Your Own Cloud (from linux.conf.au 2011)

Feb 04, 2011

At linux.conf.au 2011 in Brisbane, I had the honor of co-presenting a talk on Rolling Your Own Cloud with SUSE’s Tim Serong. Take a look!

Terabytes is not big data, petabytes is

Jan 07, 2011

I often wonder what’s behind the increased trend behind Hadoop and other NoSQL technologies. I realize if you’re Yahoo that such technology makes sense. I don’t get why everyone else wants to use it.
Reading Stephen O’Grady’s self-review of his predictions for 2010 for the first time gave me some insights into how such people think:

Democratization of Big Data
Consider that RedMonk, a four person analyst shop, has the technical wherewithal to attack datasets ranging from gigabytes to terabytes in size. Unless you?re making institutional money, budgets historically have not permitted this. The tools of Big Data have never been more accessible than they are today.
read more

Ubuntu Upstart for automatic MySQL start and stop

Dec 25, 2010

Here at Recorded Future we use Ubuntu (running on Amazon EC2), but so far we have not explored Ubuntu Upstart that much. During the holidays I made an effort to get acquainted with Upstart and to implement proper MySQL start and stop with it.If you do not know Upstart, this is the way you start and stop services in Ubuntu, and it serves the same purpose as the old /etc/init.d scripts, but are a bit more structured and powerful. That said, Upstart is regrettably far from complete, although the functionality is much better and Upstart has some cool features, some things do not work that well. For one thing, documentation, where it exists, is useless, at best. Secondly, there is very limited ability to test and develop Upstart scripts. And this is made worse by the fact that the documentation is so bad. Another thing is that Upstart insist on stopping services, by default, by sending a brutal kill signal. Not good for databases, mostly.In the /etc/init directory are the Upstart scripts you have. In difference to the old init.d scripts, you cannot disable a service in Upstart curenntly. If it is in /etc/init it will be started at system start. That’s it. And this is something that I am sure will be fixed, but for now, again, is something we have to live with. Upstart scripts have the suffix .conf (don’t ask me why), so the default MySQL Upstart script, for example, is called /etc/init/mysql.conf.In an Upstart script, there are Stanzas that determine what to do. Like the exec Stanza that runs a program for example. And you may then ask, when is it run? Startup? Shutdown? And the answer is startup. For shutting things down, as I said before, Upstart will by default just send a kill -9 signal.The minimal startup script you can have, and this actually works in a reasonable way, is to just have one line with an exec stanza, like this:exec /usr/bin/mydaemonWhich will start the daemon. For stopping the daemon, Upstart will send a -9 signal to the started process by default, and nothing more is needed in the Upstart script.For MySQL, we need to make things a bit more complicated. The default mysql.conf Upstart script really is not good. For one thing, it will not do a controlled shutdown of MySQL (this is possible even if Upstart will eventually send a kill -9 anyway). Secondly, this script assumes that what we use is a standard Ubunty installed MySQL distribution, so if you have installed MySQL in /usr/bin/mysql5147 or somethings like that, you are out of luck.So what I wanted to create was an Upstart script for MySQL that fullfilled these requirements:Starts MySQL automatically.Waits for MySQL to be available before exiting.Be configurable to support different MySQL install locations, data directories etc.Do a clean shutdown of MySQL when stopping the MySQL services.Before I show you what I ended up with, I want to comment on the points 2 and 4 above. With Upstart, you can define a script or command to run just before or after a services has been started or stopped, and this is what I use to wait for MySQL to become available, and to send a SIGTERM to the MySQL Server when stopping (which will do a clean MySQL shutdown).So here we go, a complete MySQL Upstart script, the way I want it to work:## MySQL Service for Recorded Future#description “MySQL Server”author “Anders Karlsson, Recorded Future”start on (net-device-up and local-filesystems and runlevel [2345])stop on runlevel [016]expect forkkill timeout 2# Set variables.env MYSQL_ETC=/etc/mysqlenv MYSQL_PIDFILE=/var/run/mysql.pidenv MYSQL_HOME=/usr/local/mysql5.5env MYSQL_INSTANCE=myumask 007exec $MYSQL_HOME/bin/mysqld_safe –defaults-file=$MYSQL_ETC/$MYSQL_INSTANCE.cnf >> /tmp/x.out &post-start script loop=600# Wait for MySQL to start. while [ $loop -gt 0 ]; do if $MYSQL_HOME/bin/mysqladmin –defaults-file=$MYSQL_ETC/$MYSQL_INSTANCE.cnf ping; then break fi loop=$(($loop – 1)) sleep 1 done exit 0end script# Send a soft SIGTERM to MySQL before Upstart will kill it.# A Sigterm to mysqld will cause a controlled shutdown.pre-stop script exec kill -SIGTERM `cat $MYSQL_PIDFILE`# Wait for MySQL to end. Flushing buffers and all. loop=600 while [ $loop -gt 0 ]; do# If the pidfile is found, then continue waiting. if [ -e $MYSQL_PIDFILE ] ; then loop=$((loop – 1)) sleep 1 continue fi break doneend scriptTo be honest, this is not what I create for all our MySQL servers. Instead I used this to create a chef template, chef is what we use for configuration management here (see http://www.opscode.com/ for more on chef), and here it is put to good ude to generate an Upstart script for MySQL. The above is just an example./Karlsson

Tuning InnoDB Configuration

Dec 21, 2010

I had earlier written a post on tuning the MySQL server configuration which was more geared towards the MyISAM storage engine. While that is not because I didn’t intend on ignoring InnoDB but because I had planned a whole post on tuning InnoDB related configuration. So this post is the post that I had planned, I have discussed the major configuration parameters in here that should help you out most of the times.

MySQL Workbench: Manage MySQL on Windows Servers the Windows way

Dec 13, 2010

The MySQL team has been continuously improving its products on the Windows platform.?Along this line, we’ve responded to a request from our users of Workbench on Windows – to provide remote access to Windows Servers using Windows management methods – as an alternative? to SSH.
Managing a MySQL server obviously requires access to the target machine, which usually requires elevated rights for certain tasks like restarting the server or manipulating the configuration file on Windows (where this file is in a protected path). For local connections this is mostly not a big deal. However for remote boxes security measures prevent easy manipulation of such essential things like server processes. In this blog post we discuss native Windows management and how it can be used in MySQL Workbench.

Remote Management
MySQL Workbench first introduced remote access via SSH (secure shell), a widely used and well known approach for secure remote access, especially in the Linux world. Microsoft Windows does not come with an in-built SSH server, hence an additional installation is due. For Windows users is SSH quite an unkown land and very often security rules, policies, company restrictions etc. do not allow to add extra software or open access via SSH. Also setting up SSH is non trivial and can be quite a challenge forless technical users. For these reason we have added support for native Windows management, which comes at no extra cost, since it is built into Windows already.
Native Windows Management
The management we will discuss here is not about how to add users or databases to a MySQL server and things like that.? It’s rather about? administering the MySQL server Instance that we’ll discuss here – for instance, DBA tasks such as manipulating the MySQL configuration file or controlling the server processes. Windows comes with a universal management layer called WMI (Windows Management Instrumentation). WMI is a very powerful means to query all kind of data from a Windows system (drivers, BIOS, motherboard, performance data etc.) and to manipulate the state of certain components (services, subsystems etc.). If you are going to manage a MySQL installation on a Windows server from a Windows machine (Workbench supports Windows 7) then WMI is the way to go. Due to security restrictions (e.g. UAC) WMI access works best in a domain setup as it already has all the necessary pieces to make the interplay work seamlessly. At the end of the article I have included additional information to help when setting up native Windows management for non-domain environments. However beware as that involves? disabling some important UACs, which is not advisable in most instances.
In a Windows domain you will need is a user login that has local administrator rights on the target machine MySQL server runs on. By default firewalls and access rules are typicallu set so that Workbench can connect without extra effort. WMI is used to query a server’s status and start or stop it as well as to get system information like CPU load and memory usage. Manipulating the MySQL configuration file is done by using Window’s normal file system functions, which means the target box must provide access to the file via a shared folder. By default Windows systems have a number of default shares that are always available (so-called administrative shares that give access to the entire hard disk, provided you know the administrator’s credentials for login). On the client side you can then use different possibilities to access the server by using:

A mapped drive, e.g. Z:\<path to file>\my.ini
An administrative share, e.g. \\<server>\C$\<path to file>\my.ini
An explicit share which gives access only to the file or its parent folder, e.g. \\<server>\<share>\my.ini.

This should be an old hat for most Windows users, though I wanted to point out the possibilities.
Setting up a Server Instance
Let’s get to the actual setup of a new server instance in MySQL Workbench. What you need is:

The target server’s name or IP address.
Name and password of a user which is a local administrator -? a member of the administors group -? on the target machine.
The name of the MySQL service to manage.
The path of the configuration file.

Workbench will try to help you with the last two points by giving you a list of MySQL servers it finds on the remote computer and their configuration file location. So it should mostly be point-and-click once you are logged in there.
Connect to the Target Machine
Open the New Server Instance Wizard from Workbench’s home screen. It comes up with the initial screen that allows you to specify the host machine of the MySQL server you want to manage thereby determining if it is a local or a remote connection. For the sake of simplicity let’s reuse a MySQL connection that has been created already in the SQL Development section. A server instance always needs such a connection. So either you reuse an already defined one or create a new one that is then added to the list of connections.

In this example I have used a Windows 2008 R2 server, which hosts the MySQL server. After selecting the connection proceed to the next page by clicking on the “Next” button. The wizard will do some initial tests and display the outcome. For a few more details (especially if something goes wrong) I recommend to open the log window using the “Show Logs” button on the wizard form.

After proceeding to the next page you will be asked to select which type of remote management you want to use. Here is where you will see the Windows Remote option.

Since we want native Windows management the option the choice is obvious. Proceeding to the next page causes the wizard to open a WMI connection to the target machine and retrieve a list of installed MySQL servers (whether running or not). Note that only the MySQL servers installed as Windows Service are recognized. In order to connect we need the credentials for a user on the target machine. As mentioned earlier, this user must be a local administrator.

Don’t get confused by the term “Service” in the login dialog. This is not the Windows service but a key that describes the given credentials. Together with the user name it defines a tupel to allow storage and retrieval of the password. Once the connection is established you can choose the server you want to manage. Selecting a server entry will automatically fill the configuration file path with a default value constructed from the server name, the administrative share C$ and the path of the file found in the service entry. Usually this is a good guess and should work most of the time.

Sometimes though things are a bit different and you can get an error when you switch to the next page.
In that case just go back to the previous page and open the file selector by clicking on the “…” button. A File Open Dialog will pop up that allows to select the file you need. This also works for remote machines, just type the machine name in the input box or use the network branch for selection. Sometimes directly accessing a share might pose problems so you may alternatively want to map a network drive in Explorer before you go on with the selection of the configuration file. For our demo the letter Z has been mapped to \\workbench.testing\C$.

After you have picked the correct file, continue to the next page which will again check if the file is accessible from MySQL Workbench. Note: only the last of the 3 tests will be performed as we already have a connection and hence don’t need to test that again. And as we are on Windows no test for any system command is required. They must be available anyway (otherwise the installation is probably broken).

Continuing, the wizard will bring up a small window asking you for a decision regarding if you just want to proceed or review what settings have been collected so far. Usually you can go on and finish the wizard after you gave your new server instance profile a proper name.

If you however want to review the details, e.g. because there is still an error at this point then click on the button “I’d like to review the settings again”. This will bring up a page that would otherwise just be skipped, which lists the collected details and allows you to opt to change them manually. This is btw. the only way to adjust the section in the configuration file that is managed by MySQL Workbench in this wizard. However, the Server Instance Manager allows to set this value as well. The default is “mysqld”.

If you checked the option “Change Parameters” you will get to a page that allows you to select the configuration file, and also let’s you test again if the file is accessible and if the given section exists in it.

Once you are satisfied with the result click “Next” to continue with the last wizard page, where you can specify a name for the new instance and finish the wizard.

Note: In the case something goes wrong you will see that the wizard does not force you to provide 100% correct details most of the time. In fact you can still continue and create the server instance even if currently no connection is possible or you do not have all relevant details. Of course without correct user credentials the wizard is not able to give you a list of installed services, so this is the minimum information, which must be correct.
What about Windows Workgroups?
In smaller and/or home networks there is typically no domain controller active, so the relationship between Windows boxes is a bit different and it is sometimes much more complicated to adjust all the necessary settings to allow this kind of access from one machine to another. Note: non-domain environments are not supported by MySQL Workbench due to various problems there.
However there are a number of web pages that discuss the use of WMI in such scenarios, which might help you to get it to work. A good overview what needs to be set can be found on the “Connecting to WMI Remotely Starting with Windows Vista” page. Most of the time you will find that these settings are already in place (except for the firewall exception), though you still get either of the most common errors:

“RPC Server not available”, which indicates the target WMI service cannot be reached at all, usually because of firewalls or wrong server name.
“Access denied” which is? the result of access-token-filtering even if an administrator is logged in.

Denied-access errors are usually caused by the UAC. Read here what Microsoft has to say about “User Account Control and WMI”. Also consider the following tips:

Add user with Administrator rights or set the proper rights for another user in “Computer Management/Services and Applications/WMI-Control -> context menu-> Security”
Firewall (“netsh firewall set service RemoteAdmin enable”)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\ForceGuest must be 0.
A user connecting from a remote computer must have the SC_MANAGER_CONNECT privilege enabled to be able to enumerate services (admins usually have).

User Account Control is the major blocking stone in a workgroup environment. You will find connections to an old Windows XP box much simpler as there is no such thing as UAC there. Even for local management (i.e. MySQL Servers on the box where MySQL Workbench runs on) is affected by UAC. Running the application as administrator solves this (or disabling UAC). However since both methods are not recommended we have changed Workbench for this special case so it uses the old “sc” command to start and stop a service, in order to minimize trouble and effort for the most common case. WMI is still used for monitoring, just not for service control.

Stage 2: http:BL with Apache2 mod_perl – Paul Gregg

Dec 02, 2010


After my earlier post
Referrer and Comment spammers are a PITA I came up with two mod_perl plugins to Apache and an “apache level” firewall.

The reason for the apache-level firewall is two-fold.? There is no direct way for the Apache user to manipulate an iptables chain (as it doesn’t run as root), and second; I was not happy with suid root access or other forms of message passing to a daemon which would manipulate the firewall for me.

Architecture is thus, in httpd.conf place the following two lines:

PerlPreConnectionHandler PGREGG::httpBLBlock
PerlLogHandler PGREGG::httpBLLog

The first tells apache to run the handler in my httpBLBlock.pm module when a connection is received (before the request has been sent by the client).? In this handler, I am simply looking for a filename matching that IP in a directory that is writable by the apache user.? The contents of the file are a SCORE:httpBL_answer:[LIST].? Based on this, the module checks the mtime of the filename is in the last SCORE days, then the firewall is in effect. If so, we simply tell apache to drop the connection.? If the file has expired, we delete the file.

The second line is more interesting, and what creates the firewall filenames. In order to not impede the general speed of request handling, processing is performed in the Logging section of the Apache process. Our module is called by apache after the response has been sent, but before the access_log entry has been written.? In our module we perform the http:BL API call and compute the above SCORE based upon the Threat* level and Age* of the API response. (* both Threat and Age are octets in the DNS lookup).? We merely discount the Threat down to zero based on the Age (0-255) where an entry 255 days old reduces the SCORE to zero.
If the SCORE is larger than our trigger level (3) then we create the firewall filename, log the entry in our own httpbl.log and return Apache2::Const::FORBIDDEN.? This causes Apache to not log the entry in the normal access_log.? Otherwise, if all is ok, we return Apache2::Const::OK and Apache logs the hit as normal.

I have a bit of code tidy up, restructure the config/firewall directory and pull some common code out to a shared module before I can release to the world.

An interesting side effect to publishing the last story out through Planet PHP and other news sources along with the Project Honey Pot image is that when browsers viewed those sources, they all asked for the image off my server. In several cases, these were known spammer, Comment spammer, and other abusers. My server then created the firewall entry blocking them before they were able to follow the links back to my server.
?
I have been reading up more on Apache Bucket Brigades in an attempt to allow the firewall filter to be placed immediately after the request has been received and allow a custom response to the browser. This may help an otherwise unsuspecting user if their machine had been trojaned. I don’t mind admitting I’m thoroughly confused right now :)

Lean Startups and Scalability

Nov 30, 2010

I wrote this as a reply to Does Lean Startup Methodology Apply to Consumer Startups?” However, due to comment length restrictions on that blog, I am posting my comment here and welcome your thoughts.”An enterprise will pilot products and iterate with a vendor: Let’s run a 6 month consulting engagement/pilot to evaluate if this new database solves the problem.”Only an enterprise where there is a major disconnect between management and engineering will opt for this path. In enterprises where needed data I/O patterns are understood, taking such path may spell disaster. The primary problem with the ‘lean startup’ methodology that I see is that it blindly preaches entrepreneurs to close their eyes, cut corners and just get the product to market without fully understanding the future scalability needs. Scalability doesn’t has to be sacrificed in order to build a lean startup, except in those cases where there is no architect on board.Many entrepreneurs think of taking route of frameworks in early days only to find out the haunting effects as growth happens.Case studies exist where massive infrastructures with low latencies have been built without blowing budgets and need for re-architecting infrastructures. The concept of lean startups is great, but incomplete (especially as it is being preached).When the right architects and team is on-board, building the right way also becomes the lean way. <update> Let’s not forget that data is the most valuable asset of an organization and every migration a great opportunity for a screwup. Do you really want to migrate it around from database to database as you sit with your fingers crossed hoping that the latest vendor will solve your problem? Or does it make more sense to invest in an experienced architect and then make decisions rather than shooting in the dark? I’ll let you decide the rest.