Optimizing Model loading with LazyModel

by Frank
If you are running PHP5, this is what you need. This LazyModel optimizes the loading of the models. It limits the number of loaded models to only the ones that are actually used. When using this, you can improve speed and decrease memory usage significantly. You should use LazyModel in every project you build. That said, it is easy to see this as so called ‘premature optimization’ and maybe it is, but adding this to your project will only take like 10 seconds and it will pay off most of the time.

Background

To be honest, this wasn’t my idea at all. At first the idea showed up as a bin by José Lorenzo Rodríguez (http://bin.cakephp.org/saved/39855/). Then Matt Curry picked it up and improved it (http://github.com/mcurry/lazy_loader). Next José Lorenzo Rodríguez forked it and improved it even more (http://github.com/lorenzo/lazy_loader). I think that got a bit out of hand there (no offence) and that is why I started a new project that uses a slightly different approach.

Features

  • Fully compatible with CakePHP 1.2 and 1.3 (all model core tests pass).
  • Works on PHP 5.2 and 5.3.
  • Only loads models when they are needed.
  • Works fine with Containable, OneQuery and probably every other behavior out there.

How it works

By default CakePHP will load every model that you have following the associations. This is far from efficient when you only want to use one or two models. This is an example of a few associated models, represented as a model chain:


Then take for example the next statement:

<?php
$this
->Doutzen->find('all', array(
    
'fields' => array('Doutzen.phone'),
    
'contain' => array('Marissa' => 'Heidi')
));
?>

What we want is to only load and use the Doutzen, Marissa and Heidi model. However in reality it loads all models in the model chain walking down the associations. This LazyModel actually limits the models that are loaded to the ones that are actually used, like this:


You can imagine this will speed your website up a bit and save some memory at the same time. It is basically a win-win situation here and you are a fool when you leave this awesome PHP5 feature alone (LazyModel uses __set() and __isset() magic methods which are introduced in PHP5).

Benchmarks

People want benchmarks or let's call it proof. In my opinion benchmarking this is pretty useless. If you are wondering why, I’ll explain it for you. The speed increase or memory decrease all depends on your application and the page you are calling. The more models and associations you have and the less you use them, the bigger the impact of LazyModel will be.

Here is a benchmark with a page that improved quite a bit with this model. It is the front page of a project with +/- 25 models.

Without LazyModel


Welcome to CakePHP v1.3.3 Console
---------------------------------------------------------------
App : app
Path: /var/www/hosts/test/app
---------------------------------------------------------------
-> Testing http://test.home.frankdegraaf.net

Total Requests made: 100
Total Time elapsed: 6.8480186462402 (seconds)

Requests/Second: 14.603 req/sec
Average request time: 0.068 seconds
Standard deviation of average request time: 0.002
Longest/shortest request: 0.078 sec/0.066 sec

Memory usage: 9.75MB

With LazyModel

Welcome to CakePHP v1.3.3 Console
---------------------------------------------------------------
App : app
Path: /var/www/hosts/test/app
---------------------------------------------------------------
-> Testing http://test.home.frankdegraaf.net

Total Requests made: 100
Total Time elapsed: 4.8957378864288 (seconds)

Requests/Second: 20.426 req/sec
Average request time: 0.049 seconds
Standard deviation of average request time: 0.001
Longest/shortest request: 0.056 sec/0.048 sec

Memory usage: 7.25MB

You can see how the speed of the site went up 40% (+5.823 requests/second) and the memory usage dropped 26% (-2.5MB). This is a pretty good result. However this will not guarantee anything for your site and that is why it is a useless benchmark. If you have three models, it might improve a millisecond, but that will be as good as it gets. So again, the bigger your application, the higher the impact.

P.S. I put the memory usage there myself, it wasn’t part of that particular benchmark, but the direct output of memory_get_usage(true);.

Other details

The code can be found at my GitHub repository:

http://github.com/phally/lazy_model/

There I have explained how simple it is to make it work (takes like 10 seconds) and explained some design decisions I have made. Please read the Q&A for any questions that may come up.

Phally

Report

More on Models

Advertising

Comments

  • mevdschee posted on 02/03/11 05:34:07 AM
    Hi Phally,

    Thank you very much for your plugin, it really helps us a lot and makes cake so much better!

    One remark: when assigning the same alias in 2 classes without the class pointed to being the same, a namespace collision occurs. I would vote for using "[modelname].[alias]" in the $this->map variable as key instead of just "[alias]". Just a thought.. hope you continue the good work.

    Maurits
    • Frank posted on 04/26/11 12:47:16 PM
      [quote] Hi Phally,

      Thank you very much for your plugin, it really helps us a lot and makes cake so much better!

      One remark: when assigning the same alias in 2 classes without the class pointed to being the same, a namespace collision occurs. I would vote for using "[modelname].[alias]" in the $this->map variable as key instead of just "[alias]". Just a thought.. hope you continue the good work.

      Maurits
      [end quote]
      Hi Maurits,

      Thanks for your feedback. However I don't think they can collide, because every model has its own $map. The map property is not shared across models. So Post can have an alias User in it and Comment can have an alias User in it and it still wouldn't collide. Of course it would be giving a lot of problems if it did.

      Cheers,

      Frank
  • dancastellon posted on 01/16/11 05:43:03 PM
    Hi, I just cloned this to my plugins directory and followed the instructions. For some reason I'm getting an error that says:

    Fatal error: Class 'Model' not found in C:\blabla\app\plugins\lazy_model\libs\lazy_model.php on line 28

    Any help would be appreciated. Thanks!
    • Frank posted on 02/01/11 09:16:35 AM
      Sorry for the late reply.

      [quote] Hi, I just cloned this to my plugins directory and followed the instructions. For some reason I'm getting an error that says:

      Fatal error: Class 'Model' not found in C:\blabla\app\plugins\lazy_model\libs\lazy_model.php on line 28

      Any help would be appreciated. Thanks!
      [end quote]
      Are you still having issues? It must have something to do with the setup of your application somehow, because the Model class should be loaded already. Does your AppModel extends LazyModel or are you trying something else?
  • Frank posted on 09/28/10 09:49:03 AM
    If you cloned it to your plugins directory, it just works. You didn't clone it, you just downloaded the file and put it somewhere it doesn't belong.

    Check the repository on how the folders are organised. Still, cloning takes less time and it will be easier to keep up-to-date.
  • gmreburn posted on 09/23/10 04:00:07 PM
    Something is not working with the import command suggested (App::import('Lib', 'LazyModel.LazyModel');). I placed that line in the app_model as well as placing the lazy_model.php inside my plugins folder. I noticed the github has it placed in lib and app:import is importing from a lib not a model? I'm not too good with the import command so I'm not sure why it says LazyModel.LazyModel for the name.. I tried to create a folder in plugins called LazyModel which didn't work.. I think changed import 'lib' to import 'Model' which gave me an eror:

    Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 523800 bytes) in /var/www/cakephp/cake/libs/configure.php on line 1156



    I got it to work by pasting the variables/functions of the lazy_model into my app_model which resulted in much faster and less MySQL calls.. I would like to get this working correctly as a plugin if possible!

    Thanks in advance for any help
  • Frank posted on 08/19/10 08:38:38 AM
    I pushed a bugfix which improves the performance some more. I updated the benchmarks.

    Speed went from +32% to +40% and memory went from -15% to -26%.

    Update fast.
    • zmonteca posted on 08/19/10 10:00:10 AM
      I pushed a bugfix which improves the performance some more. I updated the benchmarks.

      Speed went from +32% to +40% and memory went from -15% to -26%.

      Update fast.

      Awesome.

      Check out my issues in this test: http://bin.cakephp.org/view/907756277
      Fatal error: Call to a member function bindModel() on a non-object in ./app/tests/cases/models/address.test.php on line 32

      Is this something I'm doing wrong in constructing my models or is this a short coming of the plugin?
      • Frank posted on 08/20/10 06:55:45 AM
        I pushed a bugfix which improves the performance some more. I updated the benchmarks.

        Speed went from +32% to +40% and memory went from -15% to -26%.

        Update fast.

        Awesome.

        Check out my issues in this test: http://bin.cakephp.org/view/907756277
        Fatal error: Call to a member function bindModel() on a non-object in ./app/tests/cases/models/address.test.php on line 32

        Is this something I'm doing wrong in constructing my models or is this a short coming of the plugin?

        I think line 20 isn't giving your model, but an instance of AppModel. It can also be that the association in Address to Client isn't properly set up.

        Besides that, setting up a test case in __construct() is tricky and you should call parent::__construct(). There are the callbacks like startTest() or startCase() (which you have in yours) to do that kind of stuff. This will prevent certain forms of bleed-through. However this should be unrelated to your problem.
  • zmonteca posted on 08/17/10 02:47:14 PM
    This won't really work in the event your using model references in your application, right?

    for example:

    // assuming User belongsTo Company
    $this->Company->User->find('all'); // this will cause the an error using the lazy model
    • Frank posted on 08/18/10 08:25:56 AM
      This won't really work in the event your using model references in your application, right?

      for example:

      // assuming User belongsTo Company
      $this->Company->User->find('all'); // this will cause the an error using the lazy model

      It probably isn't LazyModel causing that error. It has been intensely tested with that, because exactly that is what LazyModel is all about.
      • zmonteca posted on 08/18/10 10:59:01 AM
        This won't really work in the event your using model references in your application, right?

        for example:

        // assuming User belongsTo Company
        $this->Company->User->find('all'); // this will cause the an error using the lazy model

        It probably isn't LazyModel causing that error. It has been intensely tested with that, because exactly that is what LazyModel is all about.

        Really? It's erring out for me. I'm going to have to look at it more closely and i'll get back to you as to the suspected cause.
        • Frank posted on 08/18/10 12:04:29 PM
          This won't really work in the event your using model references in your application, right?

          for example:

          // assuming User belongsTo Company
          $this->Company->User->find('all'); // this will cause the an error using the lazy model

          It probably isn't LazyModel causing that error. It has been intensely tested with that, because exactly that is what LazyModel is all about.

          Really? It's erring out for me. I'm going to have to look at it more closely and i'll get back to you as to the suspected cause.

          It could be a weird association of some kind. I don't know what the error is though. You could contact me on IRC when I am online, then we can talk it over.
          • nickelfault posted on 03/14/11 07:24:13 PM
            Hmmm, I'm having the same problem. I get a:

            Fatal error: Call to a member function find() on a non-object
            on $this->User->Shout->find('first');

            Seems like the associated model isn't being created. Any ideas? The relationships are set up correctly, as it works perfectly without lazyloader.
  • rynop posted on 08/13/10 02:15:23 PM
    Have you tested this with caching enabled? Anytime I muck with the inner workings of models, I worry about breaking Cake's model cache.

    I plan on giving your lib (looks sweet by the way) a try this weekend, so I'll post back what I find out with my initial tests.
    • Frank posted on 08/13/10 04:09:58 PM
      Have you tested this with caching enabled? Anytime I muck with the inner workings of models, I worry about breaking Cake's model cache.

      I plan on giving your lib (looks sweet by the way) a try this weekend, so I'll post back what I find out with my initial tests.

      Works fine with model cache. Like I said, the model test cases all pass. Besides that, I am not really messing with the Model internals, it only delays the loading process. The cache works just the same, it doesn't touch that.
      • rynop posted on 08/13/10 09:34:27 PM
        ...I'll post back what I find out with my initial tests.
        Works fine with model cache...
        Works great. Ran into a slight problem with using ClassRegistry::init('model') within a model - but it was a hack on my part in the 1st place.

        I see as much as 2x reduction in memory usage in some of my code (19MB->9MB). Thanks Frank, this is great stuff.
  • webtechnick posted on 08/12/10 01:16:49 AM
    Great plugin, helped reduce one of my large app's memory footprint in half. Much appreciated.
  • redd posted on 08/11/10 12:39:29 PM
    Really nice plugin, big thanks!
  • euromark posted on 08/11/10 04:44:12 AM
    is the benchmark console script available somewhere?
    so that we can find out the gains for our projects.
    that would be great
    • Frank posted on 08/11/10 06:51:28 AM
      is the benchmark console script available somewhere?
      so that we can find out the gains for our projects.
      that would be great

      Yes of course, it is part of DebugKit: http://github.com/cakephp/debug_kit
      However, Siege for example would show the difference too. The memory difference can be seen by adding the following code to your view/layout:

      <?php
      echo $this->Number->toReadableSize(memory_get_usage(true));
      ?>

      Of course you need to add the NumberHelper to your helpers array.
      • rynop posted on 08/13/10 08:58:14 PM
        is the benchmark console script available somewhere?
        so that we can find out the gains for our projects.
        that would be great

        Another alternative is to use cake's built in testing framework http://book.cakephp.org/view/1196/Testing. When you run tests (via /test.php) it will show you peak memory usage for a given test. Makes it very easy to compare with and without Frank's LazyModel code
  • maninderv posted on 08/11/10 01:57:42 AM
    Hi Phally,

    In case we don't define the associations in the models, but do them on the fly, then wouldn't $this->loadModel('ModelName') have the same effect ?

    I know defining them again and again is a pain, but theoretically, would it not be the same ?

    Thoughts?

    -Mandy.
    • Frank posted on 08/11/10 03:16:25 AM
      Hi Phally,

      In case we don't define the associations in the models, but do them on the fly, then wouldn't $this->loadModel('ModelName') have the same effect ?

      I know defining them again and again is a pain, but theoretically, would it not be the same ?

      Thoughts?

      -Mandy.

      Not at all. Loading a model with Controller::loadModel() will load a single model, that is true, but you won't be able to find any other data than the data from that particular model. You would at least need a couple of calls to Model::bindModel() in your app.

      So basically it is a lot easier and cleaner to bake your associations or place them in your model yourself.
login to post a comment.