Re: Software configuration management tool required

From: Dave Hinz (DaveHinz_at_spamcop.net)
Date: 08/19/05 

Date: 19 Aug 2005 20:46:00 GMT

On Fri, 19 Aug 2005 13:19:35 -0700, Michael Vilain <vilain@spamcop.net> wrote:
> In article <3mmce5F17l8afU11@individual.net>,
> Dave Hinz <DaveHinz@spamcop.net> wrote:

>> Right. Which is why if you use a tool which _automates_ the
>> documentation, it's the best way to accomplish it. If it automates
>> documentation and makes the job easier (let the admin figure out "what
>> to do" and then have a "and now go do that 100 times" button) it's more
>> likely to be used and welcomed by your staff.
>
> Here's the meat of the problem we had: how do you document changes to a
> system?

Document, or document _usably_? Sadly, not a lot of overlap.

> Unless you have a snapshop before and after of every relevant
> file, software configuration (e.g. oracle layout on raw filesystems) and
> hardware configuration, how do you figure out what's changed via a
> script or program?

Right. A centralized tool that allows you to make the changes, provides
snapshots, and easy backout and automation would be the ideal. From
their claims, bladelogic is just that, and the fact that a trusted
friend who now works for them is still enthusiastic about it leads me to
believe that it's more true than "marketing fluff".

> For some of the big systems, it would take quite a
> long time to run an auditing program and a lot of storage to keep those
> records.

Well, if all changes are made by a mechanism that tracks, then you know
what changes are made, by definition. It's a different way of working,
though, and the best way to get something like that adopted is to have
using it be less work than doing it the normal way. If it's harder
_and_ a hassle, it'll get ignored.

> I was a fan of a "site logbook" for each system that required the admin
> to fill out a running dialog of what they were doing as they were doing
> it during a change (aka a "devolution"). But this model breaks down
> with a roomful of servers. We had a summer intern create an Access
> database that we all had to fill out _daily_ of what each of us did. It
> was mailed to managers and section heads daily. If there was a system
> change or outage scheduled for that night, there's better be an entry in
> the database for it the next morning unless you're still working on it.

Or unless it's monday and you completely forgot what you did friday.

> It also helped the various shifts communicate with each other so we knew
> what happened last night just by reading the email (I also scanned log
> files just to check--I caught a few problems by noticing differences in
> "that's not what it's usually like").

I wish I had the time to know my logfiles personally, but with 6 guys
and 100-ish servers, it's just not going to happen. Hell, I can't even
remember all the sites we host anymore.

> The automation method only really works if you have each system audited
> down to the serial number on each board in terms of hardware and total
> software configuration. Many of the servers were "one-offs" running a
> single application (e.g. finance or MRP or documentation or
> trouble-tickets or email & calendaring or the 100+ company-private web
> sites).

Well, to some extent, I think. Again, we don't have it in yet so I'm
somewhat speculating, but...our webserver cluster is a series of
identical enough boxes. If, for instance, I want to turn off some
encryption method on all apache instances...let's see, that's close to
100 of 'em. Too many files to edit by hand for my comfort. In that
case, the boxes don't need to be identical, and the files are _not_
identical, but on all of 'em I need to change the line which says
"blah +blurgh"
...to just say
"blurgh"

Sure, I could do some foreach server in (list) type thing, but there's
no tracking. If I use the tool for it, it's tracked, the old version of
the file is saved, and I can revert if I need to. All of these things
are, of course, scriptable, this just puts a framework and a boatload of
sample scripts to start with.

> Knowing what disks had what on them, their layout, memory, disk
> controllers, tape drives, and even the crontabs that ran nightly was all
> important and had to be tracked.

Yup.
 
>> I'm not sure what "line manager" means in your world, but here's how we
>> do it - we _have_ the root passwords in an encrypted database, but the
>> only time I use them is when I'm doing something that involves logging
>> in to single-user mode (usually patch clusters). All the day-to-day
>> work is done with sudo which is logged (individually on the servers at
>> this time, so of limited value except for investigating if something
>> went wrong).

> Line managers at this place were those that managed the people that did
> things. They didn't do things themselves accept to assign tasks,
> prioritize and go to endless meetings. They may have done the grunt
> work some years ago, but have since become a manager of grunts.

Ah. The position I'm trying to avoid, at least for now. Got it.

>> Every change? Ouch. Our policy is that if you do something that needs
>> to start at boot, you test it by running the rc?.d script that init will
>> run at boot, to start it up. Has been pretty successful, but we only
>> have 6 Unix admins to keep honest, so it's not too bad.

> Well, some developers are rather blithe about changing system parameters
> because Oracle or some vendor tells them to do so. Some of those
> parameters affect things like the SGA or the maximum open files. We had
> no problem changing them on the development systems overnight with a
> reboot to test if the change screwed up the startup of Oracle or the
> backups or other stuff. It was really a life saver.

Something to consider, anyway, yes.

>> Well, that's why making it painless and pleasant is preferable to being
>> dictatorial. But yeah, if people don't want to be accountable for what
>> they do, that's a problem.
 
> Well, the last contract _was_ rather dictatorial about such things.

Customer gets to set the rules, after all. We've got some, well, let's
just say large financial institutions whose names probably appear "in
your wallet" that we deal with, and the demands of some of them are
pretty strict. It's doubly ironic when those same companies show up on
the front page of the WSJ for data security breaches, which, if they
followed what they force us to follow, couldn't happen.

Topic drift anyone? Sorry about that.