Dynamodb RequestHandler的验收规范验收规范、Dynamodb、RequestHandler

2023-09-11 12:22:33 作者:晚雾

我在云中运行发电机插入时,任何帮助或线索,如何调试这样的错误发现了一个神秘的异常?

背景

在code我运行:

成功与数据插入dynamodb从我的本地机器上运行的时候,但

在云中过EMR硕士学位preduce作业运行失败时突然因身份验证。

用于认证的URL端点。

我简单地创建一个像这样的凭据:

 客户端=新DynamoDBClient(新BasicAWSCredentials(
        XXXX,
        XXXXXXXXXXX));
   client.setEndpoint(https://dynamodb.eu-west-1.amazonaws.com);
 

的异常即时得到如下:

 在线程异常主要java.lang.NoSuchFieldError:requestHandlers
        在com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.init(AWSSecurityTokenServiceClient.java:214)
        在com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.<init>(AWSSecurityTokenServiceClient.java:160)
        在com.amazonaws.auth.STSSessionCredentialsProvider&LT; INIT&GT;(STSSessionCredentialsProvider.java:73)
        在com.amazonaws.auth.SessionCredentialsProviderFactory.getSessionCredentialsProvider(SessionCredentialsProviderFactory.java:96)
        在com.amazonaws.services.dynamodb.AmazonDynamoDBClient.setEndpoint(AmazonDynamoDBClient.java:857)
        在com.amazonaws.services.dynamodb.AmazonDynamoDBClient.init(AmazonDynamoDBClient.java:262)
        在com.amazonaws.services.dynamodb.AmazonDynamoDBClient&LT; INIT&GT;(AmazonDynamoDBClient.java:181)
        在com.amazonaws.services.dynamodb.AmazonDynamoDBClient&LT; INIT&GT;(AmazonDynamoDBClient.java:142)
 

解决方案

真实的答案在这里,是,dynamodb客户不与最新的或当前版本匹配可以呈现出奇怪的反射/类加载错误时我们尝试使用他们在一个现代化的环境。

在AWS罐子在旧EMR AMI实例的类路径可以使用它调用非EMR服务的Hadoop作业正确(最新)AWS罐存在冲突(即如dynamodb,在我们的例子)。 解析DynamoDB 一个共享型数据库云服务

在我的旧AMI实例,我只是发:

  MV $ HOME / lib目录/ AWS-java的SDK-1.1.1.jar $ HOME / lib目录/ AWS-java的SDK-1.1.1.jar.old
 

要在单节点群集上解决这个问题。

此错误的根本原因是什么?是,我使用的是较旧的红宝石弹性-MA preduce客户端,从而导致创建旧的AMI版本在我的电子病历云,这对类路径过时的AWS-SDK罐子。

I have found a cryptic exception when running dynamo inserts in the cloud, any help or clues as to how to debug such an error ?

Background

The code I am running :

Succesfully inserts data into dynamodb when run from my local machines, but

Fails abruptly due to authentication when running in the cloud in a mapreduce job over EMR.

Uses a URL endpoint for authentication.

I simply create credentials like so:

   client=new DynamoDBClient(new BasicAWSCredentials(
        "XXXX",
        "XXXXXXXXXXX));
   client.setEndpoint("https://dynamodb.eu-west-1.amazonaws.com");

The exception Im getting is below:

Exception in thread "main" java.lang.NoSuchFieldError: requestHandlers
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.init(AWSSecurityTokenServiceClient.java:214)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.<init>(AWSSecurityTokenServiceClient.java:160)
        at com.amazonaws.auth.STSSessionCredentialsProvider.<init>(STSSessionCredentialsProvider.java:73)
        at com.amazonaws.auth.SessionCredentialsProviderFactory.getSessionCredentialsProvider(SessionCredentialsProviderFactory.java:96)
        at com.amazonaws.services.dynamodb.AmazonDynamoDBClient.setEndpoint(AmazonDynamoDBClient.java:857)
        at com.amazonaws.services.dynamodb.AmazonDynamoDBClient.init(AmazonDynamoDBClient.java:262)
        at com.amazonaws.services.dynamodb.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:181)
        at com.amazonaws.services.dynamodb.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:142)

解决方案

The "real" answer here, is that, dynamodb clients which don't match up with the latest or current versions can exhibit odd reflection / class loading error when we attempt to use them in a modern environment.

AWS jars exist on the class path of older EMR AMI instances can conflict with proper (latest) AWS jars used by hadoop job which invokes a non-EMR service (i.e. such as dynamodb, in our case).

On my older AMI instance, I simply issued:

mv $HOME/lib/aws-java-sdk-1.1.1.jar $HOME/lib/aws-java-sdk-1.1.1.jar.old 

To resolve the issue on a single node cluster.

The ROOT cause of this error? was that I was using an older Ruby elastic-mapreduce client, which led to creation of older AMI versions in my EMR cloud, which had obsolete aws-sdk jars on the class path.