Ubuntu
eucalyptus package

Eucalyptus "403 Forbidden" when trying to run instance

Bug #430093 reported by Etienne Goyer
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
euca2ools (Ubuntu)
Fix Released
High
Thierry Carrez
Ubuntu ubuntu-9.10-beta
Karmic
Fix Released
High
Thierry Carrez
Ubuntu ubuntu-9.10-beta
eucalyptus (Ubuntu)
Invalid
High
Unassigned
Karmic
Invalid
High
Unassigned

Bug Description

On karmic, after upgrading Eucalyptus from 1.6~bzr672-0ubuntu4 to 1.6~bzr746-0ubuntu1, trying to run an instance fail with the following error:

cloudmaster@uec-frontend:~$ euca-run-instances -t c1.medium -k warthogs-key emi-073A1120
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

Sifting through /var/log/eucalyptus, the only interesting bits I found was in cloud-output.log. Here is what I think is relevant:

10:52:46 INFO erCertificateHandler | POST http://10.153.108.210:8774/axis2/services/EucalyptusGL HTTP/1.1
10:52:47 INFO ClusterUtil | ---------------------------------------------------------------
10:52:47 INFO ClusterUtil | -> [ warthogs ] Cluster certificate valid=true
10:52:47 INFO ClusterUtil | -> [ warthogs ] Node certificate valid=true
10:52:47 INFO ClusterUtil | ---------------------------------------------------------------
10:52:47 WARN PipelineRegistry | => More than one candidate pipeline. Ignoring offer by: internal-query-pipeline-Eucalyptus of type InternalQueryPipeline
10:52:47 INFO ServiceSinkHandler | cloud/ServiceSinkHandler:admin:35ebcdcd-94c4-4fd6-8bfc-def8a65646c1:MSG_RECEIVED:1253026367.1670:DescribeAvailabilityZonesType
10:52:47 INFO ServiceSinkHandler | cloud/ServiceSinkHandler:admin:35ebcdcd-94c4-4fd6-8bfc-def8a65646c1:MSG_SERVICED:1253026367.3360:178
10:52:51 WARN PipelineRegistry | => More than one candidate pipeline. Ignoring offer by: internal-query-pipeline-Eucalyptus of type InternalQueryPipeline
10:52:51 INFO ServiceSinkHandler | cloud/ServiceSinkHandler:admin:4af50fe8-2fac-4ecc-950a-74165696f604:MSG_RECEIVED:1253026371.3970:DescribeAvailabilityZonesType
10:52:51 INFO ServiceSinkHandler | cloud/ServiceSinkHandler:admin:4af50fe8-2fac-4ecc-950a-74165696f604:MSG_SERVICED:1253026371.4380:41
10:52:52 INFO erCertificateHandler | POST http://10.153.108.210:8774/axis2/services/EucalyptusGL HTTP/1.1
10:52:53 INFO ClusterUtil | ---------------------------------------------------------------
10:52:53 INFO ClusterUtil | -> [ warthogs ] Cluster certificate valid=true
10:52:53 INFO ClusterUtil | -> [ warthogs ] Node certificate valid=true
10:52:53 INFO ClusterUtil | ---------------------------------------------------------------
10:52:58 INFO erCertificateHandler | POST http://10.153.108.210:8774/axis2/services/EucalyptusGL HTTP/1.1
10:52:59 INFO ClusterUtil | ---------------------------------------------------------------
10:52:59 INFO ClusterUtil | -> [ warthogs ] Cluster certificate valid=true
10:52:59 INFO ClusterUtil | -> [ warthogs ] Node certificate valid=true
10:52:59 INFO ClusterUtil | ---------------------------------------------------------------
10:52:59 WARN PipelineRegistry | => More than one candidate pipeline. Ignoring offer by: internal-query-pipeline-Eucalyptus of type InternalQueryPipeline
com.eucalyptus.ws.AuthenticationException: User authentication failed.
        at com.eucalyptus.ws.handlers.HmacV2Handler.incomingMessage(HmacV2Handler.java:165)
        at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:115)
        at com.eucalyptus.ws.server.FilteredPipeline$StageBottomHandler.handleUpstream(FilteredPipeline.java:171)
       at com.eucalyptus.ws.server.NioServerHandler.messageReceived(NioServerHandler.java:111)
        at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:114)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:459)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:443)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:381)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:330)
        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:203)
        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
10:53:04 INFO erCertificateHandler | POST http://10.153.108.210:8774/axis2/services/EucalyptusGL HTTP/1.1
10:53:05 INFO ClusterUtil | ---------------------------------------------------------------
10:53:05 INFO ClusterUtil | -> [ warthogs ] Cluster certificate valid=true
10:53:05 INFO ClusterUtil | -> [ warthogs ] Node certificate valid=true
10:53:05 INFO ClusterUtil | ---------------------------------------------------------------

I tried redownloading the certificate bundle, in case it would help, but it did not: I still get the above 403 error when running euca-run-instances.

Tags: eucalyptus

Related branches

lp:ubuntu/karmic/euca2ools
Revision history for this message
Trevor Ellermann (trevor-ellermann) wrote :

Sorry for the double post. I posted this to bug #428010 but Etienne suggested this might be a better place for it.

I am getting a similar error to this one, here is the exact command line and error message.

$ euca-run-instances emi-67DC1321 -k mykey
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

Here is the only error message that is produced when I try to run the instance. It is from cloud-output.log

com.eucalyptus.ws.AuthenticationException: Missing required parameter: AWSAccessKeyId
        at com.eucalyptus.ws.handlers.HmacV2Handler.incomingMessage(HmacV2Handler.java:110)
        at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:115)
        at com.eucalyptus.ws.server.FilteredPipeline$StageBottomHandler.handleUpstream(FilteredPipeline.java:171)
        at com.eucalyptus.ws.server.NioServerHandler.messageReceived(NioServerHandler.java:119)
        at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:114)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:459)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:443)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:381)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:330)
        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:203)
        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)

This is off a fresh install. I have tried rebuilding the machine several times with no luck. This is using Karmic Alpha-5

Thierry Carrez (ttx)
Changed in eucalyptus (Ubuntu):
importance: Undecided → High
tags: added: eucalyptus
Revision history for this message
Soren Hansen (soren) wrote :

This happens also on fresh installs of 1.6~bzr746-0ubuntu3.

summary: - Eucalyptus "403 Forbidden" when trying to run instance after package
- upgrade
+ Eucalyptus "403 Forbidden" when trying to run instance
Changed in eucalyptus (Ubuntu):
status: New → Triaged
milestone: none → ubuntu-9.10-beta
Revision history for this message
Ben Selinger (blistovmhz) wrote :

Same error. New install. Also new re-install after much frustration :)

20:31:49 WARN PipelineRegistry | => More than one candidate pipeline. Ignoring offer by: internal-query-pipeline-Eucalyptus of type InternalQueryPipeline
com.eucalyptus.ws.AuthenticationException: User authentication failed.
.......

Revision history for this message
Thierry Carrez (ttx) wrote :

Might be a euca2ools error, parent of bug 431847. Could you try with ec2-run-instances instead ?

Revision history for this message
Thierry Carrez (ttx) wrote :

Upstream says it's a euca2ools bug, fixed in latest upstream revs.

Changed in eucalyptus (Ubuntu Karmic):
status: Triaged → Invalid
Changed in euca2ools (Ubuntu Karmic):
assignee: nobody → Thierry Carrez (ttx)
importance: Undecided → High
milestone: none → ubuntu-9.10-beta
status: New → Triaged
Changed in eucalyptus (Ubuntu Karmic):
milestone: ubuntu-9.10-beta → none
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package euca2ools - 1.0+bzr20090923-0ubuntu1

---------------
euca2ools (1.0+bzr20090923-0ubuntu1) karmic; urgency=low

  * Upstream bugfix release, fixes LP: #435126, LP: #430093.

 -- Thierry Carrez <email address hidden> Wed, 23 Sep 2009 11:56:45 +0200

Changed in euca2ools (Ubuntu Karmic):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.