在.NET并发BlockingCollection有内存泄漏?内存、NET、BlockingCollection

2023-09-03 16:25:46 作者:铁打的身子不锈钢的心

我使用的是生产者/消费者模式中使用 System.Collection。 Concurrent.BlockingCollection<数据表>从数据库(生产者)检索数据并创建数据Lucene索引(消费者)

I'm using a Producer/Consumer Pattern with a System.Collection.Concurrent.BlockingCollection<DataTable> to retrieve data from a database (producer) and create a Lucene Index on the data (consumer).

生产者抓住了10000条记录的时间,并增加了在设置到 BlockingCollection&LT;数据表&GT; 。消费者(这是一个稍微慢一点),然后获取这些10000,并创建一个索引。

The Producer grabs 10000 records at a time and adds the set to the BlockingCollection<DataTable>. The consumer (which is a little bit slower) then grabs those 10000 and creates an index.

阻塞收集一定到5 &LT;数据表&gt;将 10000每行。

The blocking collection is bounded to 5 <DataTable> of 10000 rows each.

起初,程序运行很大,但其获得约15万行后,我发现我的电脑内存刷爆了,它减慢至爬行。

At first the program runs great but after its gets about 150000 rows in I noticed that my computers memory is maxed out and it slows to a crawl.

看来,BlockingCollection不能设置底层数组插槽该项目是拍摄之后。

It seems that the BlockingCollection fails to set the underlying array slot to null after the item is taken.

code:

    private static LuceneIndex index;
    private static BlockingCollection<DataTable> blockingCol;

    private static void Producer()
    {
        while (true)
        {
            //...get next 10000 rows
            DataTable data = GetNextSet();
            if(data.Row.Count > 0)
                blockingCol.Add(products);
            else
                break;
        }
    }

    private static void Consumer()
    {
        while (!BlockingCol.IsCompleted || BlockingCol.Count > 0)
        {
            DataTable data = blockingCol.Take();
            index.UpdateIndex(GetLuceneDocs(data));
        }
    }


 public static void Main(System.String[] args)
 {
            index = new LuceneIndex();
            blockingCol = new BlockingCollection<DataTable>(2);
            // Create the producer and consumer tasks.
            Task Prod = new Task(Producer);
            Task Con = new Task(Consumer);
            // Start the tasks.
            Con.Start();
            Prod.Start();
            // Wait for both to finish.
            try
            {
               Task.WaitAll(Con, Prod);
            }
            catch (AggregateException exc)
            {
               Console.WriteLine(exc);
            }
            finally
            {
               Con.Dispose();
               Prod.Dispose();
               blockingCol.Dispose();
             }
}

任何人都可以证实的拒绝这种悬挂?而且没有任何变通?

Can anyone confirm of reject this suspension? And is there any work around?

推荐答案

是的,我可以证实这一点。你是不是在.NET 4.5,是吗?它应该是固定在那里(下这个回答您的评论似乎证实了这一点)。

Yes I can confirm this. You aren't on .NET 4.5, are you? It is supposed to be fixed there (and your comments under this answer seem to confirm this).

总之,写自己周围数据表的包装,并清除包装时,与该表进行。这使得它符合GC。该包装不会早GC'ed但它是很小的。

Anyway, write yourself a wrapper around a DataTable and clear that wrapper when you are done with the table. That makes it eligible for GC. The wrapper will not be GC'ed early but it is tiny.

class Wrapper<T> { public T Item; }