在反序列化protobuf网的memcache提供商空类型的错误提供商、错误、类型、序列化

2023-09-03 17:42:47 作者:青沫

我使用的是最新protobuf网LIB与protobuf网memcache的供应商。我需要序列化的自定义类型MyClass的名单

  [ProtoContract]
公共MyClass类{
  [ProtoMember(1)]
  公众诠释一个{获得;组;}
  [ProtoMember(2)]
  公众诠释B {获得;组;}
}
 

所以,我需要存储/ retreive:

 名单,其中,MyClass的> myList中
 

当值通过protobuf的存储,然后从高速缓存retreived一切顺利。但是如果值存储在内存缓存(在正确的protobuf格式),即从另一个线程/应用程序,之后该值是从高速缓冲存储器的反序列化retreived由于失败类型的字段NullReferenceException异常

因此​​,当设定先行,系列化值类型存储在typeCache变量,然后从该词典retreived。但是,如果值是present中的memcache但在当前线程typeCache变种没有设置不包含任何类型和抛出NullReference的反序列化。

有什么方法来解决这个问题或者一些解决办法?

更深入的调查: 序列化/反序列化过程中 ProtoTrans coder.cs 实施enyim。它包含恢复NetTrans codeR 类有词典< ArraySegment<字节>中键入> typeCache 。 因此,当一组先行序列类型(即名单,其中,MyClass的> )存储在 typeCache var和一切顺利。如果值是present在内存缓存,但它retreived本code对当前反序列化应用程序/线程没有设置:

 键入= Type.GetType(enc.GetString(缓冲,keyOffset,LEN));
byte []的standaloneBuffer =新的字节[长度];
Buffer.BlockCopy(缓冲液,keyOffset,standaloneBuffer,0,的len);
键=新ArraySegment<字节>(standaloneBuffer,0,len个);
sync.EnterWriteLock();
尝试
   {
        //没有人打我们呢?
        键入TMP;
        如果(typeCache.TryGetValue(键,出TMP))返回TMP;
        typeCache.Add(键,输入);
        返回类型; <  - 在这里,它返回null,如果类型不是present在typeCache
   }
最后
   {
        sync.ExitWriteLock();
   }
 
如何使用protobuf做C 的序列化方案

要重现错误:

项目 创建并存储在内存缓存部分名单(已配置prototrans codeR) 重新启动当前的应用程序(或启动另一个线程) 尝试从memcache中那些其他线程获得价值者皆

下面是此错误的堆栈跟踪: [ArgumentNullException:值不能为空。 参数名:类型】

\开发\ protobuf网\ protobuf网\元\ TypeModel.cs:在C;

  ProtoBuf.Meta.TypeModel prepareDeserialize(Object类型值,类型&AMP) 592
   ProtoBuf.Meta.TypeModel.Deserialize(流源,对象的值,类型类型,SerializationContext上下文)在C:\开发\ protobuf网\ protobuf网\元\ TypeModel.cs:577
   ProtoBuf.Caching.Enyim.NetTrans coder.Enyim.Caching.Memcached.ITrans coder.Deserialize在C(CacheItem项):\ Users \用户akureniov \工作\ protobuf网-1 \ protobuf的,net.Enyim \ protobuf的-net.Enyim \ ProtoTrans coder.cs:109
   Enyim.Caching.MemcachedClient.PerformTryGet(字符串键,UINT64和放大器;中科院,对象和放大器;值)+179
   Enyim.Caching.MemcachedClient.TryGet(字符串键,对象和放大器;值)+42
   Enyim.Caching.MemcachedClient.Get(字符串键)+15
 

解决方案

通过马克Gravell 我们发现在键入错误微调在ProtoTrans coder.cs:恢复NetTrans codeR:无效WriteType(MemoryStream的MS,型型)。这code座原因的错误,因为它减少过多的:

  INT I = typeName.IndexOf(''); //先拆
如果(ⅰ> = 0){I = typeName.IndexOf(',',I + 1); } //第二次分裂
如果(ⅰ> = 0){的typeName = typeName.Substring(0,i)的;只有} //提取物型/组件
 

它运作良好的简单类型,但未能在列表,字典等。

要避免这种情况,最好使用正则表达式来削减不是信息所必要的(如文化,publickkeytoken等)。因此,这里是一个替代(需要更换上面这行),安静粗鲁,但工作在大多数情况下:

 的typeName = Regex.Replace(typeName的,@,版本= \ D + \ D + \ D + \ D +。的String.Empty);
的typeName = Regex.Replace(typeName的,@,文化= \ W +的String.Empty);
的typeName = Regex.Replace(typeName的,@,公钥= \ W +的String.Empty);
 

这正则表达式不剪式的组装,这是需要自定义类型。但对于非标准类型是的mscorlib ,它可以安全地在大多数情况下,加入另一行被删除:

 的typeName = Regex.Replace(typeName的,@,mscorlib程序的String.Empty);
 

I'm using lastest protobuf-net lib with protobuf-net memcache provider. I need to serialize list of custom type MyClass

[ProtoContract]
public class MyClass{
  [ProtoMember(1)]
  public int a {get; set;}
  [ProtoMember(2)]
  public int b {get; set;}
}

So I need to store/retreive:

List<MyClass> myList

When values are stored through protobuf and then retreived from cache all goes well. But if value is stored in memcache (in correct protobuf format) i.e. from another thread/app and after that that value is retreived from cache deserialization fails due to NullReferenceExceptions of type field.

So when set goes first, type of serialized value is stored in typeCache variable and then retreived from that dictionary. But if value is present in memcache but not set in current thread typeCache var doesn't contain that type and throws NullReference on deserialization.

Are there any ways to fix this or some workaround?

Deeper investigations: Serialization/deserialization process for enyim implemented in ProtoTranscoder.cs. It contains NetTranscoder class that has Dictionary<ArraySegment<byte>, Type> typeCache. So when set goes first serialized type (i.e. List<MyClass>) is stored in typeCache var and all goes well. If value is present in memcache, but not set in current app/thread on deserialize it retreived by this code:

type = Type.GetType(enc.GetString(buffer, keyOffset, len));
byte[] standaloneBuffer = new byte[len];
Buffer.BlockCopy(buffer, keyOffset, standaloneBuffer, 0, len);
key = new ArraySegment<byte>(standaloneBuffer, 0, len);
sync.EnterWriteLock();
try
   {
        // did somebody beat us to it?
        Type tmp;
        if (typeCache.TryGetValue(key, out tmp)) return tmp;
        typeCache.Add(key, type);
        return type;   <-- Here it returns null, if type not present in typeCache
   }
finally
   {
        sync.ExitWriteLock();
   }

To reproduce that error:

List item Create and store some List in memcache (with configured prototranscoder) Restart current app (or start another thread) Try to get value by key from memcache from those "another thread"

Here is stack trace of this error: [ArgumentNullException: Value cannot be null. Parameter name: type]

   ProtoBuf.Meta.TypeModel.PrepareDeserialize(Object value, Type& type) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:592
   ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:577
   ProtoBuf.Caching.Enyim.NetTranscoder.Enyim.Caching.Memcached.ITranscoder.Deserialize(CacheItem item) in c:\Users\akureniov\work\protobuf-net-1\protobuf-net.Enyim\protobuf-net.Enyim\ProtoTranscoder.cs:109
   Enyim.Caching.MemcachedClient.PerformTryGet(String key, UInt64& cas, Object& value) +179
   Enyim.Caching.MemcachedClient.TryGet(String key, Object& value) +42
   Enyim.Caching.MemcachedClient.Get(String key) +15

解决方案

With help of Marc Gravell we found "bug" in type trimming in ProtoTranscoder.cs: NetTranscoder: void WriteType(MemoryStream ms, Type type). This code block cause error, because it cuts too much:

int i = typeName.IndexOf(','); // first split
if (i >= 0) { i = typeName.IndexOf(',', i + 1); } // second split
if (i >= 0) { typeName = typeName.Substring(0, i); } // extract type/assembly only

It worked well for simple types, but fails on List, Dictionary, etc.

To avoid this it's better to use regexp to cut not nessesary info (like culture, publickkeytoken, etc). So here is a replacement (need to replace above rows with this), quiet rude, but working in most cases:

typeName = Regex.Replace(typeName, @", Version=\d+.\d+.\d+.\d+", string.Empty);
typeName = Regex.Replace(typeName, @", Culture=\w+", string.Empty);
typeName = Regex.Replace(typeName, @", PublicKeyToken=\w+", string.Empty);

This regexp doesn't cut type's assembly, which is needed for custom types. But for standart types it's mscorlib and it can be deleted safely in most cases by adding another line:

typeName = Regex.Replace(typeName, @", mscorlib", string.Empty);