likewise-open fails to join Windows 2000 SP4 domain

Can you give us details about the domain? For example, are the DCs running Windows 2000, 2003, or 2008?

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
> Matt
> Sent: Tuesday, March 30, 2010 10:26 AM
> To: <email address hidden>
> Subject: [Bug 551901] [NEW] likewise-open fails to join domain (lucid)
>
> Public bug reported:
>
> Binary package hint: likewise-open
>
> Package: likewise-open
> Architecture: amd64
> Version: 5.4.0.42111-1
> uname: Linux 2.6.32-18-generic #27-Ubuntu SMP
>
> I am unable to join an AD domain. This machine was upgraded from 9.04
> to 9.10, after that update, I was able to join the domain and things
> worked fine. I upgraded to 10.04, and the likewise-open upgrade
> failed.
> I cleaned the old likewise-open install, reinstalled likewise-open and
> was unable to join the domain. I also tried using the suggestions
> offered in Bug #543963, but that resulted in the same outcome which
> follows:
>
> sudo domainjoin-cli --loglevel verbose join mydomain.com adminuser
> Joining to AD Domain: mydomain.com
> With Computer DNS Name: mycomputer.mydomain.com
>
> <email address hidden>'s password:
>
> (at this point the program pauses for 30 seconds to a minute)
>
> Error: Lsass Error [code 0x00080047]
>
> 59 (0x3B) ERROR_UNEXP_NET_ERR - Unknown error
>
> The last few syslog entries:
>
> Mar 30 10:19:07 mycomputer lwiod[17879]: GSS-API error calling
> gss_init_sec_context: 589824 (Invalid token was supplied)
> Mar 30 10:19:07 mycomputer lwiod[17879]: GSS-API error calling
> gss_init_sec_context: 100003 ()
> Mar 30 10:19:11 mycomputer lwiod[17879]: GSS-API error calling
> gss_init_sec_context: 589824 (Invalid token was supplied)
> Mar 30 10:19:11 mycomputer lwiod[17879]: GSS-API error calling
> gss_init_sec_context: 100003 ()
> Mar 30 10:19:12 mycomputer lwiod[17879]: GSS-API error calling
> gss_init_sec_context: 589824 (Invalid token was supplied)
> Mar 30 10:19:12 mycomputer lwiod[17879]: GSS-API error calling
> gss_init_sec_context: 100003 ()
> Mar 30 10:19:17 mycomputer lsassd[17901]: 0x7fee6ae8a710:Failed to run
> provider specific request (request code = 8, provider = 'lsa-
> activedirectory-provider') -> error = 59, symbol = ERROR_UNEXP_NET_ERR,
> client pid = 17933
>
> ** Affects: likewise-open (Ubuntu)
> Importance: Undecided
> Status: New
>
> --
> likewise-open fails to join domain (lucid)
> https://bugs.launchpad.net/bugs/551901
> You received this bug notification because you are a member of Likewise
> Open Developers, which is subscribed to likewise-open in ubuntu.
>
> Status in “likewise-open” package in Ubuntu: New
>
> Bug description:
> Binary package hint: likewise-open
>
> Package: likewise-open
> Architecture: amd64
> Version: 5.4.0.42111-1
> uname: Linux 2.6.32-18-generic #27-Ubuntu SMP
>
> I am unable to join an AD domain. This machine was upgraded from 9.04
> to 9.10, after that update, I was able to join the domain and things
> worked fine. I upgraded to 10.04, and the likewise-open upgrade
> failed. I cleaned the ol...

The domain controller is running Windows 2000. If it helps, lw-get-dc-name returns the correct information for the domain controller.

Matt, Is the DC runing Windows 2000 SP4 at least? I would need find a Win2000 DC VM and verify that it is still supported.

Yes, it's Windows 2000 SP4.

Can anyone confirm that they can join a Windows 2000 SP4 domain using Likewise Open 5.4? I'm totally stumped on whats wrong.

It could very likely be this commit in the likewise-open upstream.
http://git.likewiseopen.org/?p=likewise-open.git;a=commitdiff;h=e83a8e9862ed5357eb362ca617d93d8d6d133311

But that is a change to the krb package in Ubuntu so I can't do anything about it here. You could try two things.

(a) Look at a network trace and see if this is really this problem you are experiencing
(b) rebuild you krb5 libs with the patch and test it for a resolution

If you can't figure out how to do (b) easily, I can work on posting a modofied libkrb5 set of packages to to the likewise-open PPA.

Ok, I grabbed the krb5 1.8.1 sources and applied the patch you linked, built kerberos, and switched to from the 1.8.alpha1 libs (from ubuntu 10.04) to the patched 1.8.1 libs. I restarted and attempted to join the domain again using domainjoin-cli. This time I get no /var/log/syslog errors from GSS-API as I had up till now. Instead, both Likewise-Open and the Windows 2000 AD server agree that the machine has successfully joined the domain, but at the end of the process, domainjoin-cli threw the following error (from syslog):

Apr 12 15:48:13 matt-linux lsassd[1661]: 0x7f829bac7710:Failed to run provider specific request (request code = 8, provider = 'lsa-activedirectory-provider') -> error = 1225, symbol = ERROR_CONNECTION_REFUSED, client pid = 2169

I get the same error if I attempt to leave the domain, but leaving is unsuccessful. I think I'm going to clean the likewise-open install, remove the domain on the server end and try to join again from scratch.

Ok, everything is working now. I thought the connection refused error might be coming from some sort of leftover cruft from all the failed join attempts, so I followed your suggestions from Bug #543963 and was able to join the domain without errors, I'm also able to login using AD now, so everything is right with the world.

This certainly seems to have been cleared up by applying the patch you linked to Kerberos5 1.8.1 then using those libraries instead, as you suggested. So I guess it was just a Win2k SP4 specific issue.

Thanks for all your help.

I'll work on getting the krb5 patch pushed into the distro if possible. I think the patch has already been submitted upstream to the MIT devs but I'll double check.

Moving to krb5 component for requesting inclusion of the spnego patch

I don't see a upstream krb5 bug for this issue.
I would recommend against applying this patch until someone familiar
with the SPNEGO security model and the code has evaluated it.

Basically, certain versions of Windows produce bad SPNEGO tokens. It's
appropriate to ignore these in some situations spelled out in the RFC,
but creates a significant security issue in others. I suspect that this
may be OK, but I don't have the spnego state machine in my head now, nor
do I have the MIT SPNEGO code in my head now. The easiest way to get
comfortable with this patch would be for upstream krb5 to evaluate it:
they have been working on the SPNEGO code a lot lately so it would
probably require less effort for them.

As best I can tell, the behavior of the patch is explicitly forbidden by
RFC 4178 section 5; see II under clause B and C. However, I'll admit
that the behavior described in Appendix C does not seem consistent with
what I remember for Windows 2000... Perhaps that's only the Windows
behavior for krb5 but not NTLM?

OK, here's where this stands.
We've been discussing on #krbdev, the upstream krb5 IRC channel.
We agree that ignoring a MIC token that is an exact copy of the response
token is security neutral and it looks like both upstream and I are
comfortable making a change to do that even though it seems to go
against text in RFC 4178.
(I think RFC 4178 is overly conservative here).

My argument for why it is security neutral is that an attacker could
modify the token in transit and cause the same effect. So, either the
protocol is already broken, or this does no harm.

What needs to happen now is someone familiar with the MIT SPNEGO code
needs to look at the patch and confirm it actually ignores MIC tokens
only when MIC tokens are optional. In particular, we want to confirm
that if the mechanism supports integrity and a MIC token would be
required either through request-mic state or because the acceptor didn't
choose tho optimistic mechanism,that a MIC token is still required. It
may be relatively easy to argue that's the case--in particular if this
patch affects the logic before the code evaluates whether MIC is
required, then it's probably fine. I know I'm relatively busy today and
I believe the others involved in the discussion so far have been
similarly busy.

--Sam

@Sam: Thank you very much for looking into this. We'll wait for your green light before including that patch in all cases. The sooner the better, but if that comes too late in Lucid preparation, we'll fix this in a post-release StableReleaseUpdate.

@Jerry: Trying to assess the right bug importance for this. Could you confirm the impact is limited to Windows 2000 Server DCs ?

Subscribing Jerry to get his opinion on impact.

Matt, I have exactly the same errors, with a windows 2000 SP4 domain upgraded to support windows 2003 domain controllers (http://support.microsoft.com/kb/325379) (3 ubuntu PC upgrades from 9.10 to 10.04 and 1 PC installing 10.04 from scratch), but building the patched libraries does not solve them. (With 9.10 installation all PCs join the same domain without problems)
¿Can you give me a little more detail about the steps you follow?

Thanks for your help.

Now is working fine in upgraded and fresh install lucid PCs
In upgraded installation I have to rejoin the domain
(patch package following the steps in : http://www.cyberciti.biz/faq/rebuilding-ubuntu-debian-linux-binary-package/ for the package krb5_1.8.1+dfsg-2.dsc)

So, it's my understanding that we're still waiting for a confirmation
that this patch has been submitted upstream and for an upstream review
of the patch, right?

Sorry Sam, but I don't fully understand how the patch become available in ubuntu releases.
But I can confirm that last available package of krb5 in lucid repositories (krb5_1.8.1+dfsg-2) still have the bug/problem and the link giving in comment #6 correspond to and older version of krb5 (the line numbers does not match the last sources).
Hope is resolved soon !

I think Sam is wanting to know if likewise has submitted the patch to upstream MIT krb5. If that is the case, I'll check on the state of things and update the bug report.

Right, we are missing two pieces of information:

"Someone familiar with the MIT SPNEGO code needs to look at the patch and confirm it actually ignores MIC tokens only when MIC tokens are optional. In particular, we want to confirm that if the mechanism supports integrity and a MIC token would be required either through request-mic state or because the acceptor didn't choose tho optimistic mechanism,that a MIC token is still required."

--> This requires the patch to be discussed upstream, so it needs to be submitted there

"Confirm the impact is limited to Windows 2000 Server DCs"

--> Which versions of DCs are impacted, so that we can set the importance accordingly

>>>>> "Gerald" == Gerald Carter <email address hidden> writes:

    Gerald> I think Sam is wanting to know if likewise has submitted the
    Gerald> patch to upstream MIT krb5. If that is the case, I'll check
    Gerald> on the state of things and update the bug report.

That is. Early on you mentioned you thought this had happened; I looked
into it and could not find this patch upstream.

--Sam

Filed upstream as - "SPNEGO doesn't interoperate with Windows 2000" [krbdev.mit.edu #6726]

Patch has been committed upstream:

Subject: [krbdev.mit.edu #6726] SVN Commit

Apply patch from Arlene Berry to detect and ignore a duplicate
mechanism token sent in the mechListMIC field, such as sent by Windows
2000 Server.

http://src.mit.edu/fisheye/changelog/krb5/?cs=24075
Commit By: tlyu
Revision: 24075
Changed Files:
U trunk/src/lib/gssapi/spnego/spnego_mech.c

@Sam: let me know if you feel comfortable applying that patch now. Once it's fixed in sid/maverick, I'll push a SRU for lucid.

@Jerry: This is an issue specific to Windows 2000 DCs, right ?

Correct. My understanding is that we've only observed the issue on Windows 2000 DCs.

>>>>> "Thierry" == Thierry Carrez <email address hidden> writes:

    Thierry> @Sam: let me know if you feel comfortable applying that
    Thierry> patch now. Once it's fixed in sid/maverick, I'll push a SRU
    Thierry> for lucid.

Sure. I will attempt to get to it this weekend.
Anything you want me to do to make the SRU process easier for you?

--Sam

Sam: Not really, thanks for asking :) Maverick will sync with your fixed version, and I'll create a specific patched version for Lucid.

This bug was fixed in the package krb5 - 1.8.1+dfsg-5

---------------
krb5 (1.8.1+dfsg-5) unstable; urgency=low

  * Ignore duplicate token sent in mechListMIC from Windows 2000 SPNEGO
    (LP: #551901)
  * krb5-admin-server starts after krb5-kdc, Closes: #583494

krb5 (1.8.1+dfsg-4) unstable; urgency=low

  * fix prerm script (Closes: #577389), thanks Harald Dunkel
 -- Ubuntu Archive Auto-Sync <email address hidden> Fri, 28 May 2010 11:23:00 +0100

@Matt, Hernan:
I uploaded a fixed version to my PPA, please see:
https://launchpad.net/~ttx/+archive/ppa

Once it's built (should take a couple hours), could you install that version and test that it fixes the issue without bringing in new issues ?

If you confirm that this version fixes it, I'll upload it as a regular lucid update. Thanks for your help !

Thierry,
it seems to work well.
I have done the following :
-In a Lucid PC upgraded from Karmic having the "manual" patched krb5 installed : leaved the domain, installed package version 1.8.1+dfsg-2 (which has the problem with windows 2000 domains), verified it cannot join the domain, installed 1.8.1+dfsg-2ubuntu1~ppa1 version and verified it correct join the domain, log-in using a domain account and access domain network resources

-In a fresh Lucid PC : installed 1.8.1+dfsg-2ubuntu1~ppa1 version and verified it correct join the domain, log-in using a domain account, access domain network resources
When joining the domain I got the warning:
** Warning: A resumable error occurred while processing a module
Even though the configuration of 'hostname' was executed, the configuration did not fully complete. Please contact Likewise support. **
but everything worked as expected. Re-joining the domain does not issue the warning

Hope this help!

Thanks to Thierry Carrez, your krb5 release solved the problem for me.

Thanks very much for your help, I'll push this to lucid-proposed for a wider audience.

Fix uploaded to lucid-proposed.

ACK from ubuntu-sru

Accepted krb5 into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Worked OK for me !

This bug was fixed in the package krb5 - 1.8.1+dfsg-2ubuntu0.1

---------------
krb5 (1.8.1+dfsg-2ubuntu0.1) lucid-proposed; urgency=low

  * src/lib/gssapi/spnego/spnego_mech.c: Ignore duplicate token sent in
    mechListMIC from Windows 2000 SPNEGO (LP: #551901)
 -- Thierry Carrez <email address hidden> Tue, 01 Jun 2010 14:55:50 +0200