What is the appropriate branch-merge strategy for an audited company?

Share URL
 
philbryant's picture
philbryant asked on August 2, 2011 - 2:40am | Replies (4).

First of all, I'm brand new here. Secondly, I searched the forum for an answer to my question but failed to locate one. Apologies if this has been answered elsewhere, please let me know if so. Thirdly, my long & complicated question. (Sorry! CM is hard:)

At my company we are revisiting our branch/merge policy due to some production mishaps. I suspect we may be doing things sub-optimally, but would like your input.

Branch policy in a nutshell: branch from Trunk, using the newest code. I.e., start your project with the latest and greatest (untested code.)

Merge policy in a nutshell:

Part A: When fixing Production, or a downsteam branch, always forward integrate your change to all future branches. It is the responsibility of the production patcher to get their fix into all future projects.correctly. We typically have 6+++ active branches of future work.
Part B: Projects should merge and refresh from Trunk as often as possible, but strive to keep Trunk in a buildable state.

Release situation:
-When ready to release, branch and start testing. Fix bugs in the branch and merge fixes to trunk.
-Weekly Hotfixes
-Monthly Services Packs
-Quarterly Minor releases
-Biannual Major releases

Parallel Streams:
5 different teams, each doing multiple projects and each team doing many smaller tactical feature efforts

Number of developers:
40+ distributed globally; multiple sites / timezones

Tool:
Subversion (but evaluating Git; expensive tools not possible)

Project/Feature Flux:
-Projects are often delayed, rescheduled, reordered, scope is added/ejected late; sometimes in QA
-Certain release dates are etched in stone, but scope is morphable

Product/Business Model:
24x7 mission critical SaaS. This is a for-real mission critical application, not just some ecommerce website that will lose a little money if it's down. 11 years old, hundreds of thousands or millions of lines of code.

Specific Problems encountered:
-Production Fixes getting clobbered when projects go to prod; fixes must be reapplied in an emergency fashion.
-Project cancellation resulted in many ancillary features not being delivered to production because they were irrevocably entangled with the killed project. Many features had to be recoded because extracting them was difficult.

Audit situation:
-Need to show traceability from requirements to approval to migrate to code tested to code deployed. Need to show that only the code tested and approved was migrated. [Not sure how we pass audit since extraneous code is in every release; vestiges of future projects go in too]

My assumption of the "correct" solution:
-Branch new projects from Prod. Keep them separate until time to release. When ready to QA, merge, resolve conflicts, test/fix release. Now only the approved changes are going to Prod and we know we have built upon a stable tested baseline = current prod.

Development objections to my proposal:
-Deferring integration will cause merge conflicts that will be terrible to resolve. Better to smash things together early and find out sooner rather than later where merge conflicts exist and get them resolved.

My Objections to Development's objections to my proposal:

-Deferred integration merge conflict fear is overblown. Won't come up nearly as often as needing to pull changes.
-Deferred integration merge conflicts are easier to resolve than having to *unmerge* a bunch of half-baked projects. Furthermore, deferred integration merge conflicts are easier for QA to test properly, because test scripts are created for said projects. On the other hand, QA cannot test actively for unfinished projects. We rely on regression tests to find problems with unmerges, but this is less reliable, in my opinion, than actively testing two or more projects coming together.
-How do we even know what code we are releasing when everything is smashed together...seems to violate Audit controls...

Has anyone else out there fought this battle before? Would love to hear your perspectives :-)

Tags: 
configuration management
branch
branch-merge

4 Answers

jptownsend's picture

Phil,

Welcome to CM Crossroads!!! I will start the comments. First and foremost the problem that you are having is not branch/merge related, that is simply a symptom of a larger problem that exists accrding to your description of the situation at your workplace:

1st you stated - "Project/Feature Flux:
-Projects are often delayed, rescheduled, reordered, scope is added/ejected late; sometimes in QA
-Certain release dates are etched in stone, but scope is morphable"

This is unacceptable practice and poor planning is no excuse in any development environment, I am not aware of your dev methodology, but you can't"simply" add changes in QA, fix defects and bugs but not wholesale changes to the code base. Either your in a bad Agile environemnt or have cowboy coders.

You stated that you have:

"Parallel Streams:
5 different teams, each doing multiple projects and each team doing many smaller tactical feature efforts

Number of developers:
40+ distributed globally; multiple sites / timezones"

This leads me to believe that you have major communication issues, ie the right hand has no idea that the left hand even exists let alone what it might be doing.

You stated:

"Audit situation:
-Need to show traceability from requirements to approval to migrate to code tested to code deployed. Need to show that only the code tested and approved was migrated. [Not sure how we pass audit since extraneous code is in every release; vestiges of future projects go in too]"

Could you really have this ever given your scenario? Sounds like your audit process is weak since you later admit, some things just don't make it in, or they have to be recoded/reapplied...

Your developers are in denial because right now they are free to do as they please. Or at least thats what i gather, hence their opposition and throwing up barricades saying what you are proposing is not workable.

So in a nutshell you have much larger issues than branching and merging.

Regards,

Joe

philbryant's picture

Yes, Joe, I hear what you are saying, but I respectfully point out that I cannot change market opportunity, company strategy, marketing strategy, or sales strategy. I'm not the CEO (dangit!) I cannot exit our current business and just cannot say that we will pursue something different. Actually, not even the CEO can do that; the board would have to come together and make that call. And probably never would, given the board's objectives. Nor would I dream of trying to push them (that would be career-limiting or even career-ending.)

My job is to support the business realities. I serve at the pleasure of the business, I do not drive it.

I can adapt and communicate, but ultimately need to focus on optimizing my sphere of influence, which at the moment is branch/merge. If I throw up my hands and only say that we have bigger issues than branch/merge, that's an abdication of my responsibilities.

So, my question stands. Has anyone else encountered a similar situation and how have you dealt with it?

Thanks!

jptownsend's picture

Phil,

Fair enough. I think its obvious that the merge policty they currently have is flawed. You stated it as:

"Part A: When fixing Production, or a downsteam branch, always forward integrate your change to all future branches. It is the responsibility of the production patcher to get their fix into all future projects.correctly. We typically have 6+++ active branches of future work. Part B: Projects should merge and refresh from Trunk as often as possible, but strive to keep Trunk in a buildable state."

By the fact that their are 6+++ active branches for future work is a violation of the policy which is to merge and refresh to the trunk as often as possible but the trunk has to be buildable. Because the fact that their are 6 branches, means their is 7 code bases because the branches technically are a different code base, unless people are giving their code changes to the 6++++ other branches.

So no matter what your new policy is they can't even follow the one that is required today.

As far as your policy that you are proposing, would cause even more confusion and delays to the schedule. So the developers are right to balk at what you are suggesting. Here is why, if the developers develop in a silo only seeing how their changes affect the trunk, what happens when 6 or more different branches are trying to be merged right before QA. Now you have people stepiing all over each other. Merging is not a science its more of an art, and you start messing with the trunk that late in the game it will be a mess.

Branching and merging should never be taken lightly and your options are few given the situation you have explained.

I would be surprised if anyone could recommend a solution for this, that would be accepted by your developers. Hence my earlier post, your branching/merging is merely a symptom of a greater problem. Provided you get someone who says they have the perfect answer you I would be surprised.

I will attempt that now. So here is my solution to the original question based on my experience with merging and branching.

"Allow no more than 2 days between a branch created and it being merged to the trunk. Once we get over two branches going on a current release and if those branches are across geographical regions a developer mmeeting must be held to discuss code fixes and bug fixes beign worked on. this will at least force communication to be had. Maybe you should faciltiate the meeting."

Regards,

Joe

philbryant's picture

Ok, thanks, Joe! I admit I'm surprised to hear you echo the concern about merge conflicts and the need to synch so often. In two previous companies with I've found that largely to be a red herring. Big old product => in practice, merge conflicts only came up 1 or 2 times per year. These were with 10+ parallel streams. But the code was fairly well factored and the code base was quite large in those companies. Functions tended to be < 20 LOC, so merges just punched right through, even when two developers enhanced the same object. I recognize those shops may have been anomalies; and this is why I'm here, to gather other perspectives.

Perhaps at my new company there are monster objects and monster methods and developers can't help but touch the same code, even when working on different enhancement projects.

Now, I have seen nasty merge conflicts in other companies, just as you suggested, but those projects were New Product Development (< 1 year) and the code base small (< 100,000 LOC) giving rise to major conflicts often.

Thanks again,
Phil

CMCrossroads is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.