python - cannot make corr work

14,685

Well, I've encountered the same problem today. try use .astype('float64') to help make the type correct.
data['ESA Index_close_px'][5:50].astype('float64').corr(data['CCMP Index_close_px'][5:50].astype('float64'))

This works well for me. Hope it can help you as well.

Share:
14,685
a_ko
Author by

a_ko

Updated on June 18, 2022

Comments

  • a_ko
    a_ko almost 2 years

    I'm struggling with getting a simple correlation done. I've tried all that was suggested under similar questions.

    Here are the relevant parts of the code, the various attempts I've made and their results.

    import numpy as np
    import pandas as pd
    
    try01 = data[['ESA Index_close_px', 'CCMP Index_close_px' ]].corr(method='pearson')
    
    print (try01) 
    

    Out:

    Empty DataFrame
    Columns: []
    Index: []
    

    try04 = data['ESA Index_close_px'][5:50].corr(data['CCMP Index_close_px'][5:50])
    print (try04)
    

    Out:

    **AttributeError: 'float' object has no attribute 'sqrt'**
    

    using numpy

    try05 = np.corrcoef(data['ESA Index_close_px'],data['CCMP Index_close_px'])
    print (try05)
    

    Out:

    AttributeError: 'float' object has no attribute 'sqrt'
    

    converting the columns to lists

    ESA_Index_close_px_list = list()
    start_value = 1
    end_value = len (data['ESA Index_close_px']) +1
    for items in data['ESA Index_close_px']:
        ESA_Index_close_px_list.append(items)
        start_value = start_value+1    
        if start_value == end_value:
            break
        else:
            continue
    
    CCMP_Index_close_px_list = list()
    start_value = 1
    end_value = len (data['CCMP Index_close_px']) +1
    for items in data['CCMP Index_close_px']:
        CCMP_Index_close_px_list.append(items)
        start_value = start_value+1    
        if start_value == end_value:
            break
        else:
            continue
    
    try06 = np.corrcoef(['ESA_Index_close_px_list','CCMP_Index_close_px_list'])
    print (try06)
    

    Out:

    ****TypeError: cannot perform reduce with flexible type****
    

    Also tried .astype but not made any difference.

    data['ESA Index_close_px'].astype(float)
    
    data['CCMP Index_close_px'].astype(float)
    

    Using Python 3.5, pandas 0.18.1 and numpy 1.11.1

    Would really appreciate any suggestion.

    **edit1:* Data is coming from an excel spreadsheet data = pd.read_excel('C:\\Users\\Ako\\Desktop\\ako_files\\for_corr_‌​tool.xlsx') prior to the correlation attempts, there are only column renames and

    data = data.drop(data.index[0]) 
    

    to get rid of a line

    regarding the types:

    print (type (data['ESA Index_close_px']))
    
    
    
    print (type (data['ESA Index_close_px'][1]))
    

    Out:

    **edit2* parts of the data:

    print (data['ESA Index_close_px'][1:10])
    
    print (data['CCMP Index_close_px'][1:10])
    

    Out:

    2        2137
    3        2138
    4        2132
    5        2123
    6        2127
    7     2126.25
    8      2131.5
    9      2134.5
    10       2159
    Name: ESA Index_close_px, dtype: object
    2     5241.83
    3     5246.41
    4     5243.84
    5     5199.82
    6     5214.16
    7     5213.33
    8     5239.02
    9     5246.79
    10    5328.67
    Name: CCMP Index_close_px, dtype: object