Multiple teams deployed on shared infrastructure

[Paul Hudson]

I'm interested in how other people have handled this.

We have multiple teams. Each team works on a (somewhat - see later)
separate solution, and produces new releases at the end of every Sprint, in
theory, but multiple releases anyway.

All the releases are deployed onto the same shared infrastucture. Multiple
(hardware) servers run a number of server (software) applications, and the
solutions use one or more of the applications, extending or customising
them as required. Solutions normally share the same schemas for the
database as these are part of the application servers.

Some of those extensions/customisations can (or should be, to avoid
multiple changes and fixes) shared between solutions.

There are several environments - Dev, test, staging and live/production.
Deployments to environments other than test are under change controll, but
are made by each project on their own schedule

So we have multiple levels of sharing. If one project wants to use a later
version of a shared server app, or change a shared customisation, then
there's a risk that the other projects won't work with it without changes,
or at least won't know this without testing.

Our first issue is on the single Dev environment. We'd like to keep things
fairly agile, so we don't want all changes to the shared parts to have to
be agreed and scheduled before they can be made. But we don't want one
project

On the others, we're not sure how best to handle the need for other
projects to test/update to be compatible with changes a specific project
wants to make to shared things.

How have other people handled this problem of multiple projects sharing
infrastructure?

Current thinking (not really thought through yet(

Create a DEV environment per project.

Define a baseline configuration for the shared components. For a sprint (or
maybe several sprints) a particular project works starting with the current
baseline configuration. Their changes may include updates to that baseline.

Eventually, all the baseline changes need to be integrated and a new
baseline configuration defined. We might do this once a month (so there's a
sort of infrastructure project that also has Sprints). At some point (it's
a bit fuzzy here). a project must update its DEV environment to the
revised baseline.

The baseline configuration of environments other than DEV is changed on a
regular schedule, following the release of a new baseline version. Project
releases to envs other than DEV must work with the current baseline
configuration version for that environment, and ideally would be released
along with the updated baseline configuration. If they don't release
before the baseline is updated again and haven't caught up with that
baseline, they cannot deploy, they'll need to update and re-release.

Paul

[Bob Paige]

We have a similar situation, with many projects sharing the same database
schema. We also have 4 different environments (from DEV to PROD) and the
need to migrate schema changes between them.

In the past, our solution was as follows:
1. all schema changes *must* be backward compatible
i.e. all new columns are nullable

2. all schema changes are tracked and migrated through environments
We have a manual process for schema changes, and whatever product that
first moves to the next environment is responsible for bringing schema
changes forward.

Shared code is updated as needed and incorporated by products as needed.
Since schema changes are always backward compatible, one product can always
be using an older version of the shared code relative to the schema.

Going forward, we are considering a different approach, more similar to
what you describe.

The key change is that the shared code would be under control of a
'framework' team with their own scrums. They will 'release' new versions of
this platform periodically. All projects are responsible to integrate with
the new platform versions. Schema/Platform changes are always backward
compatible so (in theory) we should have no issues there.

This new approach implies schema changes on the shared objects are only
updated periodically, but project-specific tables are updated as per the
original approach.

So it looks like our answer to your question of shared infrastructure is to
have it modified only periodically. What is periodicially? Some say "once
every 6 months", some say "quarterly". Not sure how well it will work (how
to handle regression bugs, etc.), especially as we near a new
infrastructure release point and a project knows their current sprint will
be released on a different infrastructure revision.
--
Bobman

Post by Paul Hudson
I'm interested in how other people have handled this.
We have multiple teams. Each team works on a (somewhat - see later)
separate solution, and produces new releases at the end of every Sprint, in
theory, but multiple releases anyway.
All the releases are deployed onto the same shared infrastucture. Multiple
(hardware) servers run a number of server (software) applications, and the
solutions use one or more of the applications, extending or customising
them as required. Solutions normally share the same schemas for the
database as these are part of the application servers.
Some of those extensions/customisations can (or should be, to avoid
multiple changes and fixes) shared between solutions.
There are several environments - Dev, test, staging and live/production.
Deployments to environments other than test are under change controll, but
are made by each project on their own schedule
So we have multiple levels of sharing. If one project wants to use a
later version of a shared server app, or change a shared customisation,
then there's a risk that the other projects won't work with it without
changes, or at least won't know this without testing.
Our first issue is on the single Dev environment. We'd like to keep things
fairly agile, so we don't want all changes to the shared parts to have to
be agreed and scheduled before they can be made. But we don't want one
project
On the others, we're not sure how best to handle the need for other
projects to test/update to be compatible with changes a specific project
wants to make to shared things.
How have other people handled this problem of multiple projects sharing
infrastructure?
Current thinking (not really thought through yet(
Create a DEV environment per project.
Define a baseline configuration for the shared components. For a sprint
(or maybe several sprints) a particular project works starting with the
current baseline configuration. Their changes may include updates to that
baseline.
Eventually, all the baseline changes need to be integrated and a new
baseline configuration defined. We might do this once a month (so there's a
sort of infrastructure project that also has Sprints). At some point (it's
a bit fuzzy here). a project must update its DEV environment to the
revised baseline.
The baseline configuration of environments other than DEV is changed on a
regular schedule, following the release of a new baseline version. Project
releases to envs other than DEV must work with the current baseline
configuration version for that environment, and ideally would be released
along with the updated baseline configuration. If they don't release
before the baseline is updated again and haven't caught up with that
baseline, they cannot deploy, they'll need to update and re-release.
Paul

[Yves Hanoulle]

what I do:

for every technical dependency for my project, we create a test. a test
that proves this situation is ok for us to use.
We do that for every project

when someone else makes a change: we run all the tests, and we quickly see
what dependency are broken
now the right people talk to eachother.

disadvantage

when we miss tests, something breaks.
Solution : we add a test and we are better of in the future

we have to create tests for a piece of software we don't own.
yes. and although developers don't like it, that is the price for being
sure no one breaks out stuff

if multiple people depend on a same setting: we might have double tests.

that is fine, it's more clear that we have a huge dependency
you could also add a comment to the test, yet, that makes it less visible

that 1000 teams have to do rework

this way of working avoids multiple environments no one wants to pay and
maintain

+ multiple environments make it pssible to keep working and it postpones to
see the integration mess you are in.
for me, this means it actually hides problems instead of making them
visible.

I'm interested in how other people have handled this.
We have multiple teams. Each team works on a (somewhat - see later)
separate solution, and produces new releases at the end of every Sprint, in
theory, but multiple releases anyway.
All the releases are deployed onto the same shared infrastucture. Multiple
(hardware) servers run a number of server (software) applications, and the
solutions use one or more of the applications, extending or customising
them as required. Solutions normally share the same schemas for the
database as these are part of the application servers.
Some of those extensions/customisations can (or should be, to avoid
multiple changes and fixes) shared between solutions.
There are several environments - Dev, test, staging and live/production.
Deployments to environments other than test are under change controll, but
are made by each project on their own schedule
So we have multiple levels of sharing. If one project wants to use a
later version of a shared server app, or change a shared customisation,
then there's a risk that the other projects won't work with it without
changes, or at least won't know this without testing.
Our first issue is on the single Dev environment. We'd like to keep things
fairly agile, so we don't want all changes to the shared parts to have to
be agreed and scheduled before they can be made. But we don't want one
project
On the others, we're not sure how best to handle the need for other
projects to test/update to be compatible with changes a specific project
wants to make to shared things.
How have other people handled this problem of multiple projects sharing
infrastructure?
Current thinking (not really thought through yet(
Create a DEV environment per project.
Define a baseline configuration for the shared components. For a sprint
(or maybe several sprints) a particular project works starting with the
current baseline configuration. Their changes may include updates to that
baseline.
Eventually, all the baseline changes need to be integrated and a new
baseline configuration defined. We might do this once a month (so there's a
sort of infrastructure project that also has Sprints). At some point (it's
a bit fuzzy here). a project must update its DEV environment to the
revised baseline.
The baseline configuration of environments other than DEV is changed on a
regular schedule, following the release of a new baseline version. Project
releases to envs other than DEV must work with the current baseline
configuration version for that environment, and ideally would be released
along with the updated baseline configuration. If they don't release
before the baseline is updated again and haven't caught up with that
baseline, they cannot deploy, they'll need to update and re-release.
Paul

[Paul Hudson]

Thanks. We do not have good (or automated) tests yet. Perhaps that's what
we should be looking at instead.

One problem we have is that the underlying server applications evolve quite
rapidly. So it may be time consuming to update to deal with the changes. On
the other hand, finding out and doing it early as you say avoids postponing
it until it becomes a really big job.

Post by Yves Hanoulle
for every technical dependency for my project, we create a test. a test
that proves this situation is ok for us to use.
We do that for every project
when someone else makes a change: we run all the tests, and we quickly see
what dependency are broken
now the right people talk to eachother.
disadvantage
when we miss tests, something breaks.
Solution : we add a test and we are better of in the future
we have to create tests for a piece of software we don't own.
yes. and although developers don't like it, that is the price for being
sure no one breaks out stuff
if multiple people depend on a same setting: we might have double tests.

that is fine, it's more clear that we have a huge dependency
you could also add a comment to the test, yet, that makes it less

visible that 1000 teams have to do rework
this way of working avoids multiple environments no one wants to pay and
maintain
+ multiple environments make it pssible to keep working and it postpones
to see the integration mess you are in.
for me, this means it actually hides problems instead of making them
visible.

I'm interested in how other people have handled this.
We have multiple teams. Each team works on a (somewhat - see later)
separate solution, and produces new releases at the end of every Sprint, in
theory, but multiple releases anyway.
All the releases are deployed onto the same shared infrastucture.
Multiple (hardware) servers run a number of server (software) applications,
and the solutions use one or more of the applications, extending or
customising them as required. Solutions normally share the same schemas for
the database as these are part of the application servers.
Some of those extensions/customisations can (or should be, to avoid
multiple changes and fixes) shared between solutions.
There are several environments - Dev, test, staging and live/production.
Deployments to environments other than test are under change controll, but
are made by each project on their own schedule
So we have multiple levels of sharing. If one project wants to use a
later version of a shared server app, or change a shared customisation,
then there's a risk that the other projects won't work with it without
changes, or at least won't know this without testing.
Our first issue is on the single Dev environment. We'd like to keep
things fairly agile, so we don't want all changes to the shared parts to
have to be agreed and scheduled before they can be made. But we don't want
one project
On the others, we're not sure how best to handle the need for other
projects to test/update to be compatible with changes a specific project
wants to make to shared things.
How have other people handled this problem of multiple projects sharing
infrastructure?
Current thinking (not really thought through yet(
Create a DEV environment per project.
Define a baseline configuration for the shared components. For a sprint
(or maybe several sprints) a particular project works starting with the
current baseline configuration. Their changes may include updates to that
baseline.
Eventually, all the baseline changes need to be integrated and a new
baseline configuration defined. We might do this once a month (so there's a
sort of infrastructure project that also has Sprints). At some point (it's
a bit fuzzy here). a project must update its DEV environment to the
revised baseline.
The baseline configuration of environments other than DEV is changed on a
regular schedule, following the release of a new baseline version. Project
releases to envs other than DEV must work with the current baseline
configuration version for that environment, and ideally would be released
along with the updated baseline configuration. If they don't release
before the baseline is updated again and haven't caught up with that
baseline, they cannot deploy, they'll need to update and re-release.
Paul

[woynam]

Fail fast.

Yes, it's a pain, but it's not like the problems are going to fix themselves. The sooner you find the issue, the quicker the fix is likely to come.

Mark

Post by Paul Hudson
Thanks. We do not have good (or automated) tests yet. Perhaps that's what
we should be looking at instead.
One problem we have is that the underlying server applications evolve quite
rapidly. So it may be time consuming to update to deal with the changes. On
the other hand, finding out and doing it early as you say avoids postponing
it until it becomes a really big job.

Post by Yves Hanoulle
for every technical dependency for my project, we create a test. a test
that proves this situation is ok for us to use.
We do that for every project
when someone else makes a change: we run all the tests, and we quickly see
what dependency are broken
now the right people talk to eachother.
disadvantage
when we miss tests, something breaks.
Solution : we add a test and we are better of in the future
we have to create tests for a piece of software we don't own.
yes. and although developers don't like it, that is the price for being
sure no one breaks out stuff
if multiple people depend on a same setting: we might have double tests.

that is fine, it's more clear that we have a huge dependency
you could also add a comment to the test, yet, that makes it less

visible that 1000 teams have to do rework
this way of working avoids multiple environments no one wants to pay and
maintain
+ multiple environments make it pssible to keep working and it postpones
to see the integration mess you are in.
for me, this means it actually hides problems instead of making them
visible.

I'm interested in how other people have handled this.
We have multiple teams. Each team works on a (somewhat - see later)
separate solution, and produces new releases at the end of every Sprint, in
theory, but multiple releases anyway.
All the releases are deployed onto the same shared infrastucture.
Multiple (hardware) servers run a number of server (software) applications,
and the solutions use one or more of the applications, extending or
customising them as required. Solutions normally share the same schemas for
the database as these are part of the application servers.
Some of those extensions/customisations can (or should be, to avoid
multiple changes and fixes) shared between solutions.
There are several environments - Dev, test, staging and live/production.
Deployments to environments other than test are under change controll, but
are made by each project on their own schedule
So we have multiple levels of sharing. If one project wants to use a
later version of a shared server app, or change a shared customisation,
then there's a risk that the other projects won't work with it without
changes, or at least won't know this without testing.
Our first issue is on the single Dev environment. We'd like to keep
things fairly agile, so we don't want all changes to the shared parts to
have to be agreed and scheduled before they can be made. But we don't want
one project
On the others, we're not sure how best to handle the need for other
projects to test/update to be compatible with changes a specific project
wants to make to shared things.
How have other people handled this problem of multiple projects sharing
infrastructure?
Current thinking (not really thought through yet(
Create a DEV environment per project.
Define a baseline configuration for the shared components. For a sprint
(or maybe several sprints) a particular project works starting with the
current baseline configuration. Their changes may include updates to that
baseline.
Eventually, all the baseline changes need to be integrated and a new
baseline configuration defined. We might do this once a month (so there's a
sort of infrastructure project that also has Sprints). At some point (it's
a bit fuzzy here). a project must update its DEV environment to the
revised baseline.
The baseline configuration of environments other than DEV is changed on a
regular schedule, following the release of a new baseline version. Project
releases to envs other than DEV must work with the current baseline
configuration version for that environment, and ideally would be released
along with the updated baseline configuration. If they don't release
before the baseline is updated again and haven't caught up with that
baseline, they cannot deploy, they'll need to update and re-release.
Paul