{"id":1567,"date":"2009-06-15T01:39:00","date_gmt":"2009-06-15T01:39:00","guid":{"rendered":"http:\/\/www.smartdatacollective.com\/index.php\/post\/19590\/"},"modified":"2009-06-15T01:39:00","modified_gmt":"2009-06-15T01:39:00","slug":"19590","status":"publish","type":"post","link":"https:\/\/www.smartdatacollective.com\/19590\/","title":{"rendered":"The problem with the RDBMS (Part 3) \u2013 Let’s Get Real"},"content":{"rendered":"

\"The<\/a>Image by ToniVC<\/a> via Flickr<\/span><\/p>\n

  • Introduction<\/a><\/a> <\/li>\n
  • The Problem with the Relational Database (Part 1 ) \u2013The Deployment Model<\/a><\/a><\/li>\n
  • The Problem with the Relational Database (Part 2) \u2013 Predictability<\/a><\/li>\n

    The two primary trends in data management that have been happening for as long as I can remember are:<\/p>\n

      \n
    1. The expectations of the volume of data we are can produce and consume is growing rapidly<\/li>\n
    2. The expected delay between data production and consumption are decreasing rapidly<\/li>\n<\/ol>\n

      We have seen \u2018typical\u2019 data volumes of databases grow from MB through GB to a point currently where TB databases are common, and PB databases are the \u201cbig guys\u201d.   But at the same time we have seen the expectations around the timeliness of response from these databases also change.  What used to be a monthly report became a weekly, then a daily and finally it is not uncommon to have near real-time expectations for databases in terms of data retrieval and analysis.  We have been on a continual path towards the point where data is consumed at the same moment in which it is created, either in raw form or in an aggregated or otherwise processed state.  <\/p>\n

      At the other end of the …<\/span>
      \n<\/p>\n

      <\/p>\n

      \"The<\/a>Image by ToniVC<\/a> via Flickr<\/span><\/p>\n

    3. Introduction<\/a><\/a>\n<\/li>\n
    4. The Problem with the Relational Database (Part 1 ) \u2013The Deployment Model<\/a><\/a><\/li>\n
    5. The Problem with the Relational Database (Part 2) \u2013 Predictability<\/a><\/li>\n

      The two primary trends in data management that have been happening for as long as I can remember are:<\/p>\n

        \n
      1. The expectations of the volume of data we are can produce and consume is growing rapidly<\/li>\n
      2. The expected delay between data production and consumption are decreasing rapidly<\/li>\n<\/ol>\n

        We have seen \u2018typical\u2019 data volumes of databases grow from MB through GB to a point currently where TB databases are common, and PB databases are the \u201cbig guys\u201d.   But at the same time we have seen the expectations around the timeliness of response from these databases also change.  What used to be a monthly report became a weekly, then a daily and finally it is not uncommon to have near real-time expectations for databases in terms of data retrieval and analysis.  We have been on a continual path towards the point where data is consumed at the same moment in which it is created, either in raw form or in an aggregated or otherwise processed state.  <\/p>\n

        At the other end of the application stack, our ability to move more data around faster has led to new styles of applications that provide users near immediate access to data as it is created.  Popular consumer web examples of such applications include Facebook, Twitter, Friend Feed etc.<\/p>\n

        But at the moment these applications aren\u2019t real time, they are near real time.  This means there is a delay of some form between data creation and consumption.  These delays may be very short or several minutes depending on the particular application and its current workload.  These delays may seem irrelevant for the above mentioned apps, but the difference between \u201cnear real-time\u201d and \u201creal-time\u201d can have a significant impact on the application functionality.  I am sure we have all been frustrated when checking in at the airport and choosing a seat, only to get the \u201csorry that seat is no longer available\u201d once you click the ok button for your selection for example.<\/p>\n<\/p>\n

        The Problem with the RDBMS<\/h2>\n

        The problem with the traditional RDBMS is that it is not a real time system.  It is poll based.  This means a query is constructed, submitted and the results are returned to the application.  This itself may happen very quickly, maybe only a few ms to execute and receive a resultset.  However the problem is of course, the data is only \u201cvalid\u201d for the exact moment when the query was executed.  From that moment onwards the data becomes stale and numerous changes could be happening on the data within the RDBMS while the extracted resultset is processed. <\/p>\n<\/p>\n

        NOTE: Yes I am aware that the disconnected approach is modern and a server side cursor approach used to be common.  We moved away from server side results processing for scalability purposes, but regardless even with server side resultset processing you weren\u2019t automatically updated with the data changed.<\/span><\/em><\/p>\n

        Using my example above, while I am deciding if I want a window or an isle or if it is better to have a middle seat at the front of the plane or an isle at the back, the underlying data set could be receiving numerous updates.  When I finally make my selection the dataset could be completely invalid requiring me to start the whole process again.<\/p>\n

        While this is a very simplistic example, the issue here is the trend towards real-time in the user experience layer is not supported by the current interfacing mechanisms to a RDBMS.  While we are seeing AJAX<\/a> etc being used to provide an interface which can update data in real time, underneath likely that data is still being collected from polled queries running intermittently.<\/p>\n

        Real time & Efficiency
        <\/h2>\n

        One solution to this problem may be simply to run our polling cycles are such a high rate that the difference between real-time and near real-time becomes indistinguishable.  This is possible but of course, it comes at a high cost in terms of impact on scalability.<\/p>\n

        Let me use a fictitious example to highlight this.  Imagine a Twitter like messaging system.  This system is to provide a real time like experience to their users so they set a 2 second polling cycle for all client update queries.<\/p>\n

        For the purpose of this example, let us assume that we have 1 million users.  Those 1 million users have a different usage profiles, for this example let us assume that:<\/p>\n