我有一个 pandas 数据帧df
,其列名如下
columns = ['Baillie Gifford Positive Change Fund B Accumulation',
'Stewart Investors Worldwide Select Fund Class B (accumulation) Gbp',
'Stewart Investors Worldwide Select Fund Class A (accumulation) Gbp',
'Close Ftse Techmark Fund X Acc',
'Stewart Investors Asia Pacific Leaders Fund Class B (accumulation) Gbp',
'Stewart Investors Asia Pacific Leaders Fund Class A (accumulation) Gbp',
'Stewart Investors Worldwide Sustainability Fund Class A (accumulation) Gbp',
'Stewart Investors Worldwide Sustainability Fund Class B (accumulation) Gbp',
'Mi Somerset Emerging Markets Dividend Growth A Accumulation Shares',
'Axa Framlington Biotech Fund Gbp Z Acc',
'Stewart Investors Global Emerging Markets Sustainability Fund Class B (accumulation) Gbp',
'Schroder Asian Income Fund L Accumulation Gbp',
'Fidelity Active Strategy - Fast - Asia Fund Y-acc-gbp',
'Lf Miton Uk Value Opportunities Fund B Institutional Accumulation',
'Liontrust India Fund C Acc Gbp',
'Fidelity Asian Dividend Fund W Acc',
'Stewart Investors Global Emerging Markets Sustainability Fund Class A (accumulation) Gbp',
'Quilter Investors Emerging Markets Equity Growth Fund U2 (gbp) Accumulation',
'Man Glg Continental European Growth Fund Retail Accumulation Shares (class A)',
'Quilter Investors Europe (ex Uk) Equity Growth Fund A (gbp) Accumulation']
我想要的是筛选相似的列并保留其中一列。
例如'Stewart Investors Worldwide Select Fund Class B (accumulation) Gbp'
,类似于'Stewart Investors Worldwide Select Fund Class A (accumulation) Gbp'
,
我在想,NLP中用来识别相似文本的一些相似性分数在这里可能会有所帮助。但我不知道如何申请我的情况。
预期结果应该是保存一个相似文本的列表(我将用它来过滤我的数据帧)。例如:
columns_filtered = ['Baillie Gifford Positive Change Fund B Accumulation',
'Stewart Investors Worldwide Select Fund Class B (accumulation) Gbp',
'Close Ftse Techmark Fund X Acc',
'Stewart Investors Asia Pacific Leaders Fund Class A (accumulation) Gbp',
'Stewart Investors Worldwide Sustainability Fund Class B (accumulation) Gbp',
'Mi Somerset Emerging Markets Dividend Growth A Accumulation Shares',
'Axa Framlington Biotech Fund Gbp Z Acc',
'Stewart Investors Global Emerging Markets Sustainability Fund Class B (accumulation) Gbp',
'Schroder Asian Income Fund L Accumulation Gbp',
'Fidelity Active Strategy - Fast - Asia Fund Y-acc-gbp',
'Lf Miton Uk Value Opportunities Fund B Institutional Accumulation',
'Liontrust India Fund C Acc Gbp',
'Fidelity Asian Dividend Fund W Acc',
'Stewart Investors Global Emerging Markets Sustainability Fund Class A (accumulation) Gbp',
'Quilter Investors Emerging Markets Equity Growth Fund U2 (gbp) Accumulation',
'Man Glg Continental European Growth Fund Retail Accumulation Shares (class A)',
'Quilter Investors Europe (ex Uk) Equity Growth Fund A (gbp) Accumulation']
有帮助吗?
我找到了解决方案
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
vectorizer = CountVectorizer().fit_transform(df.columns.tolist())
vector = vectorizer.toarray()
similarity_score = cosine_similarity(vector)
df_similarity = pd.DataFrame(np.asmatrix(similarity_score))
df_similarity.columns = df.columns
df_similarity.index = df.columns
df_similarity
df_similarity
是一个数据框,其中保存每个列名与其他列名的相似性索引。
请注意,我使用了NLP中使用的一个相似性分数。用户可以使用任何可能的相似性分数。