确定性的创建和标记EC2实例确定性、标记、实例

2023-09-11 08:20:06 作者:逗逼队长

我创建3 EC2实例,并随后迭代和标记他们每个人。 有时标签请求失败,尽管后来情况似乎正在运行。

难道这是一个时机的问题?我应该等待几秒钟标记之前,创建实例后?有没有一种确定的方式来等待它开始?

解决方案

更新20140512

AWS已经同时在Troubleshooting API请求错误,包括部分应对Eventual一致性,基本上印证了分析,下面我最初的回答:

  

在Amazon EC2 API遵循的最终一致性模型中,由于配套的API系统的分布式特性。 这意味着API命令的结果运行影响您的Amazon EC2资源可能不会立即看到您运行所有后续命令。 [...]

     

[...]例如,[...] 如果你运行一个命令来修改或描述您刚刚创建的资源,它的ID可能不会传播到整个系统,你会得到一个错误响应,资源不存在

     

要管理最终一致性,您可以执行以下操作:

        

确认资源的状态运行命令来修改它。 运行使用指数退避相应的描述命令   算法,以确保有足够的时间为previous   命令,通过系统传播。 [...]

  

添加等待时间以后的命令之间,即使描述命令返回准确的响应。应用的指数退避   算法开始与一对夫妇的等待时间秒,并增加   逐渐高达大约五分钟的等待时间。

        

[重点煤矿] 的

请注意:大多数 AWS的SDK 的同时会自动应用这些建议,包括期权调整默认的重试的政策,甚至添加自定义的实现 - 见错误重试和指数退避的AWS 有关如何实现它自己,指导,如果需要的话。

更新20130719

在AWS API的最终一贯的设计越来越多地通过各种大规模​​的AWS的用户,谁自然需要更深入地和解决它相应的,例如参见下面的文章中遇到:

Dealing与在AWS EC2 API 最终一致性(由马丁·英格兰姆,Cloud Foundry的) 与不一致的EC2 API 处理(由亚伦斯特利,PiCloud)

初始答案

由于已经commented由@datasage,在AWS的API显然需要被普遍视为最终一致性只有 - 这当然是意外第一次碰到的时候,但实际上不是太奇怪了在事后大规模的服务,即工程RESP。操作折衷解决 CAP定理。

另请参阅我的亚历克斯Ciminian的问题Implementing幂等为AWS现货实例请求,在那里他讨论他的测试结果就类似的一致性问题:

  

有趣的问题 - [...]我在遇到的各种相似的API延误   在竹AWS插件的背景下,并得出结论认为,AWS API   需要将作为唯一横跨是最终一致性   板;例如,我甚至遇到过的情况下,我收到了资源   从创建呼叫ID,根据ID而不是可以标记资源   形容它之后的是,因为它理应不存在   (还)。

我在这里描述似乎表明,每一个单一的API操作完全是独立经营的AWS实际上(不仅是整个外部可见的服务,但即使是在一个单一的一个像EC2),从而可以从最终吃亏一致性,因此应慎重对待。 Kafka笔记 二 Kafka一些基础的概念与设计

有关提及的情况下,细节,你可能想看看Frequent对AWS API轮询导致油门额度,在这里我总结我们的分析和方法通过的 AWS的Java SDK 的 - 该解决方案几乎是理想的,但它似乎大大改善的事情暂且

在一个类似的说明,重新设计的 AWS SDK的PHP 2 推出专门的服务员的对象,让您轮询资源,直到它是在一个理想状态的解决这个问题,见的维特斯的的的快速启动的详细信息:

  

一个由SDK提供的高级抽象的是概念   的服务员。服务员有助于更容易与最终工作   一致的系统,提供一种简单的方法来等待一个资源   通过轮询进入特定状态的资源。 [...] 任何   与启动@method标签最好推迟到将利用一个服务员。

  $客户 - >最好推迟到('BucketExists',阵列('斗'=>'我斗'));
 

I am creating 3 EC2 instances, and subsequently iterating and tagging each of them. Sometimes the tag request fails, although the instance later appears to be running.

Could this be a timing issue? Should I wait a few seconds after creating the instance before tagging it? Is there a deterministic way to wait for it to start?

解决方案

Update 20140512

AWS has meanwhile added more detailed documentation on Troubleshooting API Request Errors, including a section addressing Eventual Consistency, which basically confirms the analysis in my initial answer below:

The Amazon EC2 API follows an eventual consistency model, due to the distributed nature of the system supporting the API. This means that the result of an API command you run that affects your Amazon EC2 resources might not be immediately visible to all subsequent commands you run. [...]

[...] For example, [...] if you run a command to modify or describe the resource that you just created, its ID might not have propagated throughout the system, and you will get an error responding that the resource does not exist.

To manage eventual consistency, you can do the following:

Confirm the state of the resource before you run a command to modify it. Run the appropriate Describe command using an exponential backoff algorithm to ensure that you allow enough time for the previous command to propagate through the system. [...]

Add wait time between subsequent commands, even if a Describe command returns an accurate response. Apply an exponential backoff algorithm starting with a couple of seconds of wait time, and increase gradually up to about five minutes of wait time.

[emphasis mine]

Please note: Most AWS SDKs meanwhile apply these suggestions automatically, including options to adjust the default retry policy or add a custom implementation even - see Error Retries and Exponential Backoff in AWS for guidance on how to implement it yourself, if need be.

Update 20130719

The eventually consistent design of the AWS API is increasingly encountered by various large scale AWS users, who naturally need to look deeper and work around it accordingly, see for example the following articles:

Dealing with Eventual Consistency in the AWS EC2 API (by Martin Englund, Cloud Foundry) Dealing with the Inconsistent EC2 API (by Aaron Staley, PiCloud)

Initial Answer

As already commented by @datasage, the AWS APIs apparently need to be generally treated as eventually consistent only - this is certainly unexpected when first encountered, but actually not too surprising for a large scale service in hindsight, i.e. an engineering resp. operational tradeoff to address the CAP theorem.

See also my comment on Alex Ciminian's question Implementing idempotency for AWS Spot Instance Requests, where he discusses his test results regarding similar consistency issues:

Interesting issue - [...] I've encountered various similar API delays in the context of the Bamboo AWS Plugin and concluded that the AWS API needs to the treated as being eventually consistent only across the board; e.g., I've even encountered cases where I received a resource id from a create call, could tag the resource based on its id but not describe it thereafter still, because it supposedly doesn't exist (yet).

What I describe here seems to indicate that each and every single API action is operated entirely independently by AWS in fact (and not only across the externally visible services, but even within a single one like EC2), thus can suffer from eventual consistency and must be treated accordingly.

For details on the mentioned cases you might want to look into Frequent polling of AWS API causes throttle limit, where I summarize our analysis and approach to improve the handling via the available but limited retry/backoff functionality within the AWS SDK for Java - the solution is all but ideal, but it seems to considerably improve things for the time being.

On a similar note, the redesigned AWS SDK for PHP 2 introduced dedicated "Waiter" objects that allow you to poll a resource until it is in a desired state to address the problem, see section Waiters within the Quick Start for details:

One of the high-level abstractions provided by the SDK is the concept of "waiters". Waiters help make it easier to work with eventually consistent systems by providing an easy way to wait on a resource to enter into a particular state by polling the resource. [...] Any @method tag that starts with "waitUntil" will utilize a waiter.

$client->waitUntil('BucketExists', array('Bucket' => 'my-bucket'));