heavy handed installer file deletion

We should investigate using the Contents file to create a list of directories to hard code in clear_partitions for explicit deletion. The fallback plan is to continue whitelisting, adding /usr/src, /var/local, and /usr/local to the list.

(09:40:53 PM) cjwatson: maybe it would be worth going through the Contents file and looking for directories in which we actually ship files, and nuking those
(09:41:07 PM) cjwatson: though the cross-compiler directories would be inconvenient
(09:44:44 PM) evand: I'm afraid of doing explicit deletion as if we ever ship a new directory and forget to upload clear_partitions, bad things happen
(09:44:52 PM) evand: err, well, can happen
(09:56:51 PM) cjwatson: bad things either way though
(09:57:07 PM) cjwatson: dunno - FHS directories except src are probably safe, at least

    Is it possible to check for the size of what is being deleted ?

    I grasp the benefit of cleaning out /lib and ....
    But pretty much anything that ends up deleting 1GB or more of files
is almost certainly wrong no matter where it is.

    Also any directory that is normally empty after a clean install -
/usr/src or /usr/local/bin, should never be cleaned.

    Can I ask what the purpose of cleaning up directories is ?

    Doing a clean install overtop of an existing install is not
fundimentally different from a "repair" install - except that the record
of installed packages will be incorrect.

    Basically I have a problem with any broad strokes cleanup before
doing an install.
    If the user wanted a completely clean system they would have
reformatted the partition.
    Since they did not I think you must err heavily on the side of
preserving what is there.

    The exception would be files you know cause problems.

Evan Dandrea wrote:
> We should investigate using the Contents file to create a list of
> directories to hard code in clear_partitions for explicit deletion. The
> fallback plan is to continue whitelisting, adding /usr/src, /var/local,
> and /usr/local to the list.
>
> (09:40:53 PM) cjwatson: maybe it would be worth going through the Contents file and looking for directories in which we actually ship files, and nuking those
> (09:41:07 PM) cjwatson: though the cross-compiler directories would be inconvenient
> (09:44:44 PM) evand: I'm afraid of doing explicit deletion as if we ever ship a new directory and forget to upload clear_partitions, bad things happen
> (09:44:52 PM) evand: err, well, can happen
> (09:56:51 PM) cjwatson: bad things either way though
> (09:57:07 PM) cjwatson: dunno - FHS directories except src are probably safe, at least
>
>

--
Dave Lynch DLA Systems
Software Development: Embedded Linux
717.627.3770 <email address hidden> http://www.dlasys.net
Cell: 1.717.587.7774
Over 25 years' experience in platforms, languages, and technologies too numerous to list.

"Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction."
Albert Einstein

The purpose of cleaning up directories that might be owned by the system is that if we do not do this then we have no chance whatsoever of being able to support the resulting conjoined-twin system. We need to clear out anything put there by the previous *operating system*, while preserving *user data*. It's a delicate balancing act and tends to involve a lot of special cases; when we miss a case we get bugs like these.

    I am an embedded systems developer.
    I grasp the reasons.

     I am extremely angry over this because I lost alot of irreplaceable
work.
    I am partly angry at myself - whie I did actually backup alot of the
trees I actually thought might get stomped on I clearly did not back up
everything I should have - of course this is an era of 300gb+ laptop
hard drives so insisting on complete backups is becoming increasingly
impossible.

    Regardless, this is not a simple issue, and I think there is a
strong argument for a very light touch.

    One of linux's appeals to me over more than a decade is the ease
with which damaged systems can be repaired - simply reinstalling overtop
of an existing system has nearly always proved equivalent to a repair.
While linux is much easier to actually repair than windows, it still can
be incredibly time consuming and often just re-installing is quicker.
Another appeal is that most anything that you think will work - that
makes sense, probably does work under Linux. I was very active in the
windows NT beta process. I got very very tired of trying to argue why
the fact that some means of solving a problem that made sense to me and
had worked in the past should continue to work, when it was clear to
them with their knowledges of the internals that there was a better way.

    And honestly I am not sure that I accept that you have to delete
anything. I grasp that you must replace numerous files, that to ensure a
working and robust system you must replace libraries that are found with
those that match this particular install - it is even useful to be able
to go backwards - re-install intrepid over karmic as an example.

    But off the top of my head I can think of extremely few instances
where if standards are being followed the mere presence of a file from a
previous install should destabalize a system. There are a few instances
but they seem to be unique to things like udev that act on any file
present in a given directory.

    Anyway with respect to principles and policy I think the best
approach is for the installer to complain and make note of the presence
of files in system directories.
    I do not care if you tell the user this is unsupported.

    Finally I would note, the only reason I had to "reinstal" was
because a system crash in the middle of an apt upgrade left a system
that I had been happy with unbootable.

    I also grasp that sometimes things like this happen in beta and
alpha stages - I support clients and develop software too.

    But there are two issues:
       The first is the bug - that has been reported and hopefully dealt
with.
       The second is that there appears to be an actual intent to get
much more aggressive about file deletion.
         I wish to make it clear that I am strongly opposed to that
choice. I understand there are benefits, but they are not outweighed by
the liabilities.

Colin Watson wrote:
> The purpose of cleaning up directories that might be owned by the system
> is that if we do not do this then we have no chance whatsoever of being
> able to support the resulting conjoined-twin system. We need to clear
> out...

On Tue, Oct 06, 2009 at 11:07:05PM -0000, dhlii wrote:
> And honestly I am not sure that I accept that you have to delete
> anything. I grasp that you must replace numerous files, that to ensure a
> working and robust system you must replace libraries that are found with
> those that match this particular install - it is even useful to be able
> to go backwards - re-install intrepid over karmic as an example.
>
> But off the top of my head I can think of extremely few instances
> where if standards are being followed the mere presence of a file from a
> previous install should destabalize a system. There are a few instances
> but they seem to be unique to things like udev that act on any file
> present in a given directory.

On the contrary, there are lots and lots of situations where we check
for the presence or absence of a file. For instance, it's standard for
init scripts to check for the presence of a binary before proceeding.
Many programs have cascading conditionals which check for one
alternative after another (gnome-wm comes quickly to mind as a fairly
dramatic example, but it's just one example). The result of not cleaning
up old system files is, in very many cases, quite likely to be a broken
system.

My greatest worry (which I think is very well-founded) about leaving old
system files around is that they will never be upgraded and yet there's
a good chance that they'll be used anyway. In this day and age where
security vulnerabilities might lurk nearly anywhere, and where people
expect that in general bug fixes will be applied automatically on
upgrade, I just don't think that's a tenable policy.

> Anyway with respect to principles and policy I think the best
> approach is for the installer to complain and make note of the presence
> of files in system directories.
> I do not care if you tell the user this is unsupported.

Frankly, I'd rather that the feature of installing over the top of an
existing system without reformatting weren't supported at all, but it
was added due to overwhelming demand. Going back and telling people that
it's unsupported (since there will be mismatching files in system
directories in nearly all cases) doesn't seem likely to help.

> The second is that there appears to be an actual intent to get
> much more aggressive about file deletion.

No more so than at present; indeed your bug suggests that we need to be
more surgical about it, and I think that's our planned response to this
bug.

Colin Watson wrote:
> On Tue, Oct 06, 2009 at 11:07:05PM -0000, dhlii wrote:
>
>> And honestly I am not sure that I accept that you have to delete
>> anything. I grasp that you must replace numerous files, that to ensure a
>> working and robust system you must replace libraries that are found with
>> those that match this particular install - it is even useful to be able
>> to go backwards - re-install intrepid over karmic as an example.
>>
>> But off the top of my head I can think of extremely few instances
>> where if standards are being followed the mere presence of a file from a
>> previous install should destabalize a system. There are a few instances
>> but they seem to be unique to things like udev that act on any file
>> present in a given directory.
>>
>
> On the contrary, there are lots and lots of situations where we check
> for the presence or absence of a file. For instance, it's standard for
> init scripts to check for the presence of a binary before proceeding.
>
    That is not directly comparable.
    I have accepted that you are going to clean out the initscripts.

    Given that you have done that, and the binary for xyz is still present,
    there is no initscript to fire it off so it will not start.
    Of course if the user/administrator wanted to start it they still could.
     So long as the only things you deleted where those that were
explicitly
    part of the bootup execution path most everything else on the system
including package configuration files could safely remain.

> Many programs have cascading conditionals which check for one
> alternative after another (gnome-wm comes quickly to mind as a fairly
> dramatic example, but it's just one example). The result of not cleaning
> up old system files is, in very many cases, quite likely to be a broken
> system.
>
    I have only dealt with ubuntu for a few years, But I have been
dealing with debian systems for about a decade,
    It is a very common occurance to do an install directly ontop of an
a broken system to repair it.
    And it virtually always works.

    It is one area at which Linux excells. It is extremely difficult to
do this with windows.

    I am not particularly familiar with gnome-wm.
    But even there why are you presuming that you will end up with
something broken.

    You already seem to deliberately preserve existing home directories
and the configurations present there.
    That means that it is highly likely the new install will still have
a user gnome configuration matching the prior install.
    If it is broken - it likely will stay broken.

    Regardless, the FUNDIMENTAL point is that the choice belongs to the
user.
    It is not the job of the installer to decide that what I have chosen
to do is stupid and silently override my decision.
    If I choose to install onto a dirty system while it is acceptable to
ask if that is really what I want to do,
    and maybe acceptable to do some deletion if I say no, it is not
acceptable to delete anything that is not explicitly going to interfere
with building the system whether it is in the /usr tree or /etc or ....

> My greatest worry (which...

This bug was fixed in the package partman-target - 64ubuntu4

---------------
partman-target (64ubuntu4) karmic; urgency=low

  * Don't preseed nonexistent passwd/user-gid. Guard against setting a
    UID of 0 when /home is a separate partition (LP: #442998).
  * Don't touch /usr/src, /var/local, and /usr/local when clearing
    partitions. Account for them existing in / (LP: #431993).

 -- Evan Dandrea <email address hidden> Fri, 16 Oct 2009 17:00:15 +0100