Vendor Branching

by grant_cox
Vendor Branching provides a robust method of ensuring your application repository is always in a stable state with regards to third party libraries, even if these libraries have unique modifications. When your application keeps a local copy of all libraries - with the correct versions, in its own repository, you can confidently implement any revision to any system as well as expecting stable maintenance and support.
Each of your projects should be under version control, in their own repository. This repository should store the entire state of the application - it should not rely on third party libraries that are not part of the same repository. When you update your application to revision n, everything that is needed for that revision to execute correctly needs to be included.

The prime example of this is the CakePHP core, but also applies to any other third party libraries that your application relies on (PEAR, TinyMCE, Gallery2, PHPThumb, Swiftmailer, etc). Priorities of using these third party repositories in your application are:

  1. Your application's repository should store the exact version that your application was developed, and tested with.
  2. The third party repository will periodically be updated (adds/deletes/changes), and you will want to merge these updates into your application when you are ready.
  3. You may need custom modifications to these libraries, but still retain the ability to merge in additional core library changes.

The method to achieve this is called Vendor Branching. There are numerous tutorials out there on this, such as SvnBook ( http://svnbook.red-bean.com/en/1.1/ch07s05.html ), and Felix has a great screencast ( http://www.debuggable.com/posts/screencast-using-vendor-branching-and-cakephp:480f4dd6-6cac-44cb-b685-4d6bcbdd56cb ). This tutorial be a simple step by step guide for implementing Vendor Branching in your application.


Overview

At its heart, Vendor Branching is performing a merge. A merge between the old and new revisions of the third party library, against your own application. To do this, all three targets must be part of the same repository (your application). This is a limitation of Subversion - you cannot merge between repositories.

As such, this technique boils down to:
  1. Keep a clean, unmodified version of the third party library somewhere in your repository.
  2. Replace this clean version with the updated third party library, whenever you want to update.
  3. Merge the changes between these clean versions, into your own application.

As your repository will identify all of the changes between 1 and 2 (including adds, deletes, and changes) as a diff, these changes can be safely merged in 3, regardless of whether your application has unique changes to these files of its own.


Tutorial: Vendor Branch your CakePHP Core.

Step 1

You should already have a repository for your application set up, with the following folder structure

c:\YourProject\branches

c:\YourProject\tag

c:\YourProject\trunk

c:\YourProject\trunk\app

c:\YourProject\trunk\cake

c:\YourProject\trunk\docs

c:\YourProject\trunk\vendors

We'll assume you have created this application with the Cake 1.2 Beta (1.2.0.6311).


Step 2

As the CakePHP used by your application is "dirty" (you will certainly have changed files in c:\YourProject\trunk\app\config , and possibly changed files in c:\YourProject\trunk\cake), we need to get a clean version.

Create the following folders:

c:\YourProject\vendors

c:\YourProject\vendors\cakephp

c:\YourProject\vendors\cakephp\current


Step 3

Put a clean version of the Cake 1.2 Beta (1.2.0.6311) into the c:\YourProject\vendors\cakephp\current folder - this can either by from the zip file from the CakePHP site, or you can use a "svn export". Do not use "svn checkout", as this would make a working folder from the remote repository.

TortoiseSVN:
Export from https://svn.cakephp.org/repo/branches/1.2.x.x , revision 6311, into c:\YourProject\vendors\cakephp\current

CLI:
svn export -r 6311 --force https://svn.cakephp.org/repo/branches/1.2.x.x c:\YourProject\vendors\cakephp\current

You will now have the following folders, with a clean Cake core

c:\YourProject\vendors\cakephp\current\app

c:\YourProject\vendors\cakephp\current\cake

c:\YourProject\vendors\cakephp\current\vendors

c:\YourProject\vendors\cakephp\current\docs


Step 4

Commit the c:\YourProject\vendors to your application repository. Be sure to add a note with the clean CakePHP revision, otherwise it can be difficult to remember what the current revision is down the track.

TortoiseSVN:
Commit the c:\YourProject\vendors, tick the "Show unversioned files" and "Select all" checkboxes. Use the message "Added clean CakePHP 1.2 r6311"

CLI:
svn add c:\YourProject\vendors
svn commit -m "Added clean CakePHP 1.2 r6311" c:\YourProject\vendors
Now your application repository is in a good development state - you can work with your application however you need.


Step 5 Updating to a newer library.

Once a newer library is available you may want to merge it into your application. To do this you need to replace the clean "current" version in your repository with the newer library. When replacing you need to retain all of the existing ".svn" folders in the "current" folder, so that your repository stays sane.

Remove all of the existing files from the "current" folder.
Windows GUI:
Search for *.* in c:\YourProject\vendors\cakephp\current, then select all of the files (not folders) and delete

Windows CLI:
del /s /q /a:-H-R c:\YourProject\vendors\cakephp\current\*.*

MacOSX Finder:
Open the "current" folder.
Hit Cmd+F to initiate a Search.
Make sure the "current" folder is selected in the "Search:" bar.
Hold the Option key and click on the '+' to add a condition (adds a
condition group).
Select "[None] of the following are true" for the group condition.
Select "[Kind] is [Folders]" for the condition.

This will find all files - but should ignore the hidden ".svn" files. So select all of these, and delete.

Export a clean version of the latest Cake 1.2 into this same folder
TortoiseSVN:
Export the HEAD revision of https://svn.cakephp.org/repo/branches/1.2.x.x to c:\YourProject\vendors\cakephp\current

CLI:
svn export --force https://svn.cakephp.org/repo/branches/1.2.x.x c:\YourProject\vendors\cakephp\current

Make sure to make a note of the revision you checked out - at the time of writing this was r6788

Commit this updated vendor to your repository (including all adds/deletes/changes), with a note indicating the revision
TortoiseSVN:
Commit the c:\YourProject\vendors\cakephp\current, tick the "Show unversioned files" and "Select all" checkboxes. Use the message "Updated CakePHP vendor to 1.2 r6788"

Windows CLI:
#add the new files
svn add --force c:\YourProject\vendors\cakephp\current

#remove all missing files - by (tediously) manually running
svn delete FILENAMEHERE
#for each file that appears in
svn status c:\YourProject\vendors\cakephp\current
#with a ! (indicating the file has been removed from the working copy).

#commit the lot
svn commit -m "Updated CakePHP vendor to 1.2 r6788" c:\YourProject\vendors\cakephp\current

Linux CLI:
#add the new files
svn add --force /YourProject/vendors/cakephp/current
#remove all missing files
svn status | grep '\!' | awk '{print $2;}' | xargs svn rm
#commit the lot
svn commit -m "Updated CakePHP vendor to 1.2 r6788" /YourProject/vendors/cakephp/current

Step 6

Now that your repository has both clean versions of the library, you can merge these into your application.
First you need to know which revisions of your own repository your are merging between. Check the log of your c:\YourProject\vendors\cakephp\current folder to see what revisions are between the two vendor versions.

TortoiseSVN:
Log on c:\YourProject\vendors\cakephp\current, make note of the revisions of the two vendor versions

CLI:
svn log c:\YourProject\vendors\cakephp\current
#------------------------------------------------------------------------
#r50 | username | 2008-05-10 16:34:10 +1000 (Sat, 10 May 2008) | 1 line
#
#Updated CakePHP vendor to 1.2 r6788
#------------------------------------------------------------------------
#r49 | username | 2008-05-10 15:10:13 +1000 (Sat, 10 May 2008) | 1 line
#
#Added clean CakePHP 1.2 r6311
#------------------------------------------------------------------------
As for this tutorial there was no application activity in between, the two revisions to merge between are r49 and r50

Now that we have the repository revisions, merge your main project folder (trunk) against these clean revisions
TortoiseSVN:
Merge on c:\YourProject\trunk, against svn://YourRepository/YourProject/vendors/cakephp/current From: revision 49, To: revision 50

CLI:
svn merge -r 49:50 svn://YourRepository/YourProject/vendors/cakephp/current c:\YourProject\trunk

This has now updated your CakePHP core in your application working copy. As it has just merged in the diff of the core, any of your own local modifications to the CakePHP core will still be there (although watch for conflicts in this case). This will include any changes to the CakePHP /app/config/* files, and other files in the "app" folder.

You should test your application with this new core, and fix any issues that may have arisen. When you are happy with this new core, you can commit your trunk. Alternatively, you can always revert your trunk to undo this merge (leaving your application core un-updated).


Step 7: Rinse and Repeat

Repeat this from Step 5 whenever you wish to update your third party library.


Not just the CakePHP core

You should use Vendor Branching for all of your third party libraries, anything where there is a chance that the third party library will be updated (and that you might want to implement this update), and where you may possibly make local changes to those libraries. This includes any of your application vendor files (PHP libraries, Javascript frameworks etc), and even your own utility scripts (if you have common Components/Helpers etc that are used in multiple applications).

Report

More on Tutorials

Advertising

Comments

  • mincepie posted on 03/20/09 04:25:31 PM
    How can you do a branch without svn copy?

    You have to start with a clean vendor drop then svn copy it to an empty custom folder and then put your "dirty" code over it. Otherwise svn does not know the history of the vendor files underneath your code.

    As far as I can see you have two copies of cake in your repository, a dirty one and a clean one but svn does not know they are related! You might as well be merging from a different repository.

    The second and third commands on the bibles vendor branch page are svn copy http://svnbook.red-bean.com/en/1.5/svn.advanced.vendorbr.html
  • grant_cox posted on 08/07/08 05:53:14 PM
    That method sounds fine to me, you'll certainly have smaller repositories, but a little less functionality.

    I assume the svn-external to get the core in your application specifies the core revision you want (not just HEAD)? Otherwise there is still the problem that a "svn update" might change your cake core, which can break things.

    You will also be missing out on changes to the application skeleton - things like /app/config/core.php and /app/webroot/index.php. But these are comparatively few and far between, so as long as you have some testing to see if anything is missing you'll be fine.

    And finally if you find you do need to modify the cake core (if you find bugs that need to be fixed immediately), these will be applied to all your projects, not just a single one. But again, is this a negative or a positive?
  • skullshot posted on 08/07/08 07:50:26 AM
    Dont mean to be rude, as I am new to Cake and the idea of vendor branching absolutely makes sense.
    however, the example of cake seems wrong as the two apps I have done in cake i kicked off by using a different app folder than the default /app

    the whole idea of cake as i understood is to leave the cake core alone, reference it as a library, and extend/replace functions as required. plz correct me if i'm wrong!!

    ive been using svn-externals to pull /vendors/cake/current into my trunk /project/trunk/cake . i tag each cake version drop as /vendors/cake/1.2.xxx and update the /vendors/cake/current with releases using pretty much the same method outlined above (but only the cake core)

    i hope thats 'cake-safe' :D
  • grant_cox posted on 06/23/08 06:13:48 PM
    I do like the idea of having the exact foreign revision in your SVN extern props. For some vendors this would be fine.

    However, the main problem with SVN externs is that you cannot have any local modifications to this vendor branch - of which I do have a few (for the Cake core).

    Plus, if you want any application files (index.php, test.php, config/*) to be updated with the core, you need to handle these manually. These don't change often though, so not a huge issue.

    And finally, if the foreign repository is not available for some reason, you might not be able to get your required application files.
  • rapsys posted on 06/23/08 05:25:23 AM
    Il may be more usefull to setup the repository a different way...
    (this is based on rc1 of 1.2.x.x revision 7125)

    Create the repository for your project :
    $ svn mkdir https://svn.example.com/svn/project/{branches,tag,trunk}
    Checkout the repository
    $ svn co https://svn.example.com/svn/project/trunk project

    Go in project dir
    $ cd project

    Export and add cakephp app and vendors directory
    $ svn export -r 7125 https://svn.cakephp.org/repo/branches/1.2.x.x/app $ svn add app
    $ svn export -r 7125 https://svn.cakephp.org/repo/branches/1.2.x.x/vendors $ svn add vendors

    Add the cake externals propriety
    $ svn propset svn:externals 'cake -r 7125 https://svn.cakephp.org/repo/branches/1.2.x.x/cake'
    Commit the data
    $ svn ci -m 'Inital import'

    Update to get the cake external referenced files
    $ svn up

    Your repository is set, enjoy ;)

    PS: Feel free to add as externals the svn or your vendors projects like simpletest
    $ svn propset svn:externals 'simpletest -r 1728 https://simpletest.svn.sourceforge.net/svnroot/simpletest/simpletest/trunk' vendors
    $ svn up vendors
  • lauren posted on 05/18/08 05:01:46 PM
    Thank you so much for the helpful walk-through. I'm a bit stuck on the very last step. A dry-run of the merge produces the following summary:

    Conflicted:222 Skipped:5 Added:144 Deleted:11 Updated:3

    I'm a bit new to SVN but that seems like quite a lot of conflicts to have to wade through. I'm merging between 1.2 r6311 and 1.2 r6931 do you think it's an unusual amount of conflicts?
    • golovan posted on 06/09/08 08:30:40 AM
      Be sure you do not have some lines in your subversion config file in [auto-props] section. If you do not know what I am talking about, forget about this comment.. ;) Anyway you should compare files as Grant Cox told you before to see what the problem is.

      Grant Cox, thank you for so nice tutorial! Exactly what I need and working really well! 5+ from me :)
    • grant_cox posted on 05/22/08 08:11:00 AM
      The only conflicts should be with any changes you have made to the core, or core files in the app folder (ie app/config/core.php). So basically none.

      Since your summary shows only 3 non conflicted files, I expect the problem is something across the board - perhaps your IDE automatically changes all files (line endings, or SVN props). I recommend you perform the merge (you can always revert to undo this - it isn't saved until you commit), then have a look at what the actual conflicts are - it should be more obvious what the problem is then.
login to post a comment.