[SRU] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23)

There were no updates on the 2nd of May, the only changes were on the 30th of April, - the lucid update...

Ralf,

but the client sends the http request headers with an empty Hostname: header or did I miss something,
if Hostname header is not set it's http/1.0 ... I wonder if it's a client problem

Regards,

\sh

* Before update: no occurences
* After update: Thousands of occurences

Hmm, and this with the same set of 18.000 users.

I already checked if it's one special type of client, but that is not the case. I'm seeing this form IE, from Safari, from Windows, from Mac. It's a regression of some kind.

This site is a SquirrelMail installation; it's HTTPS only. I already tried disabling suhosin and mod_deflate, to no avail.

It's definitely a client-side issue, however before your upgrade your Apache configuration would just accept those broken requests without error. In particular, IE6/7 are knows for broken HTTP/1.1 handling.

I wonder what changes you applied to your Apache configuration. In particular, did you use to have "BrowserMatch" directives that you didn't carry through your upgrade ?

Also could you trace the Browser(s) used on those failing requests, see if we have a pattern there ?

* Thierry Carrez <email address hidden>:
> It's definitely a client-side issue, however before your upgrade your
> Apache configuration would just accept those broken requests without
> error. In particular, IE6/7 are knows for broken HTTP/1.1 handling.
>
> I wonder what changes you applied to your Apache configuration. In
> particular, did you use to have "BrowserMatch" directives that you
> didn't carry through your upgrade ?

I have now enabled these:

  # Netscape 4.x has some problems...
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

which were disabled BEFORE the upgrade.
It's quite odd. I could disable them now to see if the problem
reappears.

--
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  <email address hidden> | http://www.charite.de

* Thierry Carrez <email address hidden>:
> Also could you trace the Browser(s) used on those failing requests, see
> if we have a pattern there ?

I have no access to the clients.
--
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  <email address hidden> | http://www.charite.de

You have no access to the client, but you can trace in the server logs the UserAgent field as sent by the clients.

Any news on this subject? I'm experiencing the same issue since the upgrade on ubuntu 10.04.

I have seen "Bad Request"-error pages in Safari 4 and 5 and in the error-log I get "client sent HTTP/1.1 request without hostname" errors for the corresponding requests.

It is however not reproducible, after reloading the URLs where the "Bad Request"-error pages were shown usually the normal content displays. Sometimes several reload tries are necessary.

Any ideas where further investigations could go?

I have never set up any special BrowserMatch directives, neither before nor after the upgrade.

I suspect this is the same issue as bug #595855 and #595116: Headers are getting truncated with https. So far, I have no idea about the reason.

If you have mod_reqtimeout and/or mod_deflate enabled, you can try if disabling one or both of them makes any difference.

mod_reqtimeout could be a reason, since it has been activated according to apt-listchanges:

apache2 (2.2.15-1) unstable; urgency=low

 * This release adds and enables mod_reqtimeout, which limits the time
   Apache waits for a client to send a complete request. This helps to
   mitigate against certain denial of service attacks. In case of problems
   with slow clients, the timeout values can be adjusted in
   /etc/apache2/mods-available/reqtimeout.conf , or the module can be
   disabled with "a2dismod reqtimeout".

-- Chuck Short <email address hidden> Tue, 13 Apr 2010 09:09:34 -0400

I have now disabled it and will audit the server logs whether the error has gone.

* Stefan Fritsch <email address hidden>:
> I suspect this is the same issue as bug #595855 and #595116: Headers are
> getting truncated with https. So far, I have no idea about the reason.
>
> If you have mod_reqtimeout and/or mod_deflate enabled, you can try if
> disabling one or both of them makes any difference.

disabling mod_deflate does indeed make it go away.

This is a rather strange bug:
- It happens if I enable exactly two out of the three modules deflate, reqtimeout, dump_io. But not with only one or all three of them.
- I have also tried replacing mod_ssl.so, mod_deflate.so, and the openssl-libs with the versions from karmic and mod_ssl.so with the version from jaunty: no change

But the bug disappears if I do

   mv /lib/i686 /lib/disabled_i686

There was a bug recently in Debian related to some gcc versions creating wrong code for SSE4. Maybe that is the problem here, too.

What CPUs do you people have? Mine is a Core i7.
Does moving /lib/i686 away (or deinstalling libc6-i686) help for you too?
Does the "flags" line in /proc/cpuinfo on your machines contain sse4_1 or sse4_2?

For reference, the Debian bug was http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=583858

Actually, I have reproduced bug #595116 and not this one, but I still think it's the same.

* Stefan Fritsch <email address hidden>:

> But the bug disappears if I do
>
> mv /lib/i686 /lib/disabled_i686

I will try to reproduce this here

> There was a bug recently in Debian related to some gcc versions creating
> wrong code for SSE4. Maybe that is the problem here, too.
>
> What CPUs do you people have? Mine is a Core i7.

# cat /proc/cpuinfo
processor: 0
vendor_id: GenuineIntel
cpu family: 6
model: 15
model name: Intel(R) Xeon(R) CPU E5310 @ 1.60GHz
stepping: 11
cpu MHz: 1600.056
cache size: 4096 KB
fdiv_bug: no
hlt_bug: no
f00f_bug: no
coma_bug: no
fpu: yes
fpu_exception: yes
cpuid level: 10
wp: yes
flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss nx lm constant_tsc
arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16
hypervisor lahf_lm
bogomips: 3200.11
clflush size: 64
cache_alignment: 64
address sizes: 40 bits physical, 48 bits virtual
power management:

> Does moving /lib/i686 away (or deinstalling libc6-i686) help for you too?
> Does the "flags" line in /proc/cpuinfo on your machines contain sse4_1 or sse4_2?
>
> For reference, the Debian bug was http://bugs.debian.org/cgi-
> bin/bugreport.cgi?bug=583858

I can tell you that the error does NOT occur with a 64bit Ubuntu!
(we reinstalled a 64bit lucid from scratch and transferred the config: the error was gone!)

The problematic 32bit machine is a vmware virtual machine.

--
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  <email address hidden> | http://www.charite.de

* Stefan Fritsch <email address hidden>:
> This is a rather strange bug:
> - It happens if I enable exactly two out of the three modules deflate, reqtimeout, dump_io. But not with only one or all three of them.
> - I have also tried replacing mod_ssl.so, mod_deflate.so, and the openssl-libs with the versions from karmic and mod_ssl.so with the version from jaunty: no change
>
> But the bug disappears if I do
>
> mv /lib/i686 /lib/disabled_i686

Confirmed. I deinstalled libc6-i686 and the error is gone!

> There was a bug recently in Debian related to some gcc versions creating
> wrong code for SSE4. Maybe that is the problem here, too.

My "processor" only has sse3

# cat /proc/cpuinfo |grep -i sse
flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss nx lm constant_tsc
arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16
hypervisor lahf_lm

flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss nx lm constant_tsc
arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16
hypervisor lahf_lm

I think we ruled out an Apache2 error here...

see eglibc in lucid-proposed for a fix

Which version is that? I was using 2.11.1-0ubuntu7.2 and I'm not seeing a more recent version....

I have now installed

libc6:
  Installiert: 2.11.1-0ubuntu7.2
  Kandidat: 2.11.1-0ubuntu7.2
  Versions-Tabelle:
 *** 2.11.1-0ubuntu7.2 0
        400 http://de.archive.ubuntu.com/ubuntu/ lucid-proposed/main Packages
        100 /var/lib/dpkg/status
     2.11.1-0ubuntu7.1 0
        900 http://de.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
        990 http://security.ubuntu.com/ubuntu/ lucid-security/main Packages
     2.11.1-0ubuntu7 0
        500 http://de.archive.ubuntu.com/ubuntu/ lucid/main Packages

But I still get lots of the following errors:
[Wed Jun 30 22:02:17 2010] [error] [client X.X.X.X] request failed: error reading the headers

The only thing that helps for me is
sudo a2dismod reqtimeout

I haven't tried whether disabling mod_deflate has the same result however. But updateing the libc to the lucid-proposed version didn't fix it. (Or is a reboot required?)

Just to let all know, we are also having this issue, and am anxiously awaiting a fix :-) I'd be happy to try any fixes as well.

Time for me to chime in, we're also affected by this issue.

I'm getting (a lot of) status 400 errors in the middle of a svn checkout when using https. When using http I have no such issues.

I've tried disabling mod_deflate and mod_reqtimeout, and this doesn't complete solve the problems.

I'm using AuthType Basic in apache and can't switch to Digest because where using ldap which I just found out doesn't work with authtyp digest :-(

FWIW, I'm running i386 10.04 in a VM (VMWare ESXi) on an Intel Xeon 5530.

I'm having a similar problem, but I really don't know if it is related. I'm running lucid 10.04 on a 32bit xen machine (on ec2). We're getting the "request failed: error reading the headers" error intermittently. However, it affects all requests after an apache2 restart. There are no error messages in the log file on startup. Another apache2 restart will typically fix the error, but it may recur on a subsequent restart with perhaps 1/4 probability.

I have tried the workarounds suggested above, but so far without success (remove libc6-i686, mod_deflate). This CPU has sse4_1 in cpu flags.

Has anyone else had this version of the bug, i.e., intermittent on apache restart?

Maybe I found a solution (bug in memcpy routine) Bug #609290

I think Jiří found the source of the problem, but it is a mod_ssl bug after all. Reassigning to apache2.

This would be https://issues.apache.org/bugzilla/show_bug.cgi?id=45444

Looks like that apache/mod_ssl fix is only in the httpd trunk, doesnt seem like it's made it into the 2.2 branch even though the fix is 2 yrs old. What's the plan for putting into the Ubuntu's apache distro?

It's scheduled (apache) to backport this fix to 2.2.x branch http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/STATUS?revision=979087&view=markup
Quick fix is to uninstall optimized libc library libc6-i686.

This bug was fixed in the package apache2 - 2.2.16-1ubuntu1

---------------
apache2 (2.2.16-1ubuntu1) maverick; urgency=low

  * Merge from debian unstable. Remaining changes:
    - debian/{control, rules}: Enable PIE hardening.
    - debian/{control, rules, apache2.2-common.ufw.profile}: Add ufw profiles.
    - debian/control: Add bzr tag and point it to our tree.
    - debian/apache2-2.common.apache2.init: Add graceful restart (LP: #456381)

apache2 (2.2.16-1) unstable; urgency=medium

  * Urgency medium for security fix.
  * New upstream release:
    - CVE-2010-1452: mod_dav, mod_cache: Fix denial of service vulnerability
      due to incorrect handling of requests without a path segment.
    - mod_dir: add FallbackResource directive, to enable admin to specify
      an action to happen when a URL maps to no file, without resorting
      to ErrorDocument or mod_rewrite
  * Fix mod_ssl header line corruption because of using memcpy for overlapping
    buffers. PR 45444. LP: #609290, #589611, #595116

apache2 (2.2.15-6) unstable; urgency=low

  * Fix init script not correctly killing htcacheclean. Closes: #580971
  * Add a separate entry in README.Debian about the need to use apache2ctl
    for starting instead of calling apache2 directly. Closes: #580445
  * Fix debug info to allow gdb loading it automatically. Closes: #581514
  * Fix install target in Makefile created by apxs2 -n. Closes: #588787
  * Fix ab sending more requests than specified by the -n parameter.
    Closes: #541158
  * Add apache2 monit configuration to apache2.2-commons examples dir.
    Closes: #583127
  * Build as PIE, since gdb in squeeze now supports it.
  * Update the postrm script to also purge the version of /var/www/index.html
    introduced in 2.2.11-7.
  * Bump Standards-Version (no changes).
 -- Chuck Short <email address hidden> Mon, 26 Jul 2010 20:21:37 +0100

2.2.16-1ubuntu1 is not in lucid

Yes it should not be marked as fixed.

Just to chime in, this is affecting my 10.04 LTS nodes talking to my recently upgraded 10.04 LTS puppet master. Puppet sits behind Apache with mod_proxy_balancer and mod_ssl. I disabled reqtimeout and have yet to see this re-appear.

Fixed in development release, targeting to lucid as well.

Status of Impact: Lucid was shipped with a bug in apache that will have hundreds of "client sent HTTP/1.1 request without a hostname". This has been fixed in apache 2.2.16 and have been backported to lucid.

How to reproduce:

1. Install apache with mod_ssl.
2. Watch your logs fill up with the above.

There should be no regressions with this patch.

I have attached the debdiff for your review.

Accepted apache2 into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

apache2 and libc packages from lucid-proposed did *not* fix the issue for me.

Neither did uninstalling libc6-686.

Using 32bit Ubuntu fwiw.

Ubuntu 10.04.1 LTS 32bit - proposed package did NOT fix the problem.

2.2.14-5ubuntu8.2 doesn't contain fix for this bug. The patch is not accepted into lucid-proposed.

We patched the fix ourselves, the fix does indeed fix the problem. Now if we can just get it as part of lucid, that would be grand.

This was not committed to lucid-proposed. Current lucid-proposed is a security fix:

apache2 (2.2.14-5ubuntu8.2) lucid-security; urgency=low

  * debian/patches/211-sslinsecurerenegotiation-directive.dpatch: once
    openssl gets updated to fix CVE-2009-3555, server renegotiations with
    unpatched clients will fail. This patch adds the ability to revert to
    the previous unsafe behaviour with a new SSLInsecureRenegotiation
    directive. (LP: #616759)
  * debian/control: add specific dependency on first openssl version to get
    CVE-2009-3555 fix.

This one is next in queue.

Accepted apache2 into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Awesome, I thought I had fixed it previously but have noticed it's still happening and causing some failed puppet runs.

Confirmed the fix - we no longer see any errors saying "client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23):" - and all the web pages work as expected. I searched the internet for weeks looking for that message, and failed, until I added ubuntu to the search - and found this thread - this was a huge fix for us - thank you.

For anyone looking to fix the same issue, here are the steps I used:

echo "deb http://archive.ubuntu.com/ubuntu/ lucid-proposed restricted main multiverse universe" >> /etc/apt/sources.list

cat > /etc/apt/preferences
Package: *
Pin: release a=lucid-security
Pin-Priority: 990

Package: *
Pin: release a=lucid-updates
Pin-Priority: 900

Package: *
Pin: release a=lucid-proposed
Pin-Priority: 400

(control-D to end)

Then
aptitude install apache2/lucid-proposed

Don't pick the first solution, but instead pick the following one:

Upgrade the following packages:
apache2-mpm-worker [2.2.14-5ubuntu8.2 (lucid-updates, lucid-security, now) -> 2.2.14-5ubuntu8.3 (lucid-proposed)]
apache2-threaded-dev [2.2.14-5ubuntu8.2 (lucid-updates, lucid-security, now) -> 2.2.14-5ubuntu8.3 (lucid-proposed)]
apache2.2-bin [2.2.14-5ubuntu8.2 (lucid-updates, lucid-security, now) -> 2.2.14-5ubuntu8.3 (lucid-proposed)]
apache2.2-common [2.2.14-5ubuntu8.2 (lucid-updates, lucid-security, now) -> 2.2.14-5ubuntu8.3 (lucid-proposed)]

Score is 53

Then the install will show:

The following packages will be upgraded:
  apache2 apache2-mpm-worker apache2-threaded-dev apache2.2-bin apache2.2-common

Now be sure you have the right stuff:

root@ubuntu:/etc/apt# dpkg -l | grep apach
ii apache2 2.2.14-5ubuntu8.3 Apache HTTP Server metapackage
ii apache2-mpm-worker 2.2.14-5ubuntu8.3 Apache HTTP Server - high speed threaded model
ii apache2-threaded-dev 2.2.14-5ubuntu8.3 Apache development headers - threaded MPM
ii apache2-utils 2.2.14-5ubuntu8.2 utility programs for webservers
ii apache2.2-bin 2.2.14-5ubuntu8.3 Apache HTTP Server common binary files
ii apache2.2-common 2.2.14-5ubuntu8.3 Apache HTTP Server common files

If you see 8.3, then you have the new code... Thanks again.

This bug was fixed in the package apache2 - 2.2.14-5ubuntu8.3

---------------
apache2 (2.2.14-5ubuntu8.3) lucid-proposed; urgency=low

  * debian/apache2.2-common.postinst: Don't fail if you can load the reqtimeout module.
    (LP: #621837)
  * debian/patches/Backport fix for upstream bug PR 45444: https://issues.apache.org/bugzilla/show_bug.cgi?id=45444. (LP: #609290, #589611, #595116)
 -- Chuck Short <email address hidden> Mon, 27 Sep 2010 14:06:57 -0400