Updating value in iterrow for pandas

Question

I am doing some geocoding work that I used selenium to screen scrape the x-y coordinate I need for address of a location, I imported an xls file to panda dataframe and want to use explicit loop to update the rows which do not have the x-y coordinate, like below:

for index, row in rche_df.iterrows():
    if isinstance(row.wgs1984_latitude, float):
        row = row.copy()
        target = row.address_chi        
        dict_temp = geocoding(target)
        row.wgs1984_latitude = dict_temp['lat']
        row.wgs1984_longitude = dict_temp['long']

I have read Why doesn't this function "take" after I iterrows over a pandas DataFrame? and am fully aware that iterrow only gives us a view rather than a copy for editing, but what if I really to update the value row by row? Is lambda feasible?

Solution 1

The rows you get back from iterrows are copies that are no longer connected to the original data frame, so edits don't change your dataframe. Thankfully, because each item you get back from iterrows contains the current index, you can use that to access and edit the relevant row of the dataframe:

for index, row in rche_df.iterrows():
    if isinstance(row.wgs1984_latitude, float):
        row = row.copy()
        target = row.address_chi        
        dict_temp = geocoding(target)
        rche_df.loc[index, 'wgs1984_latitude'] = dict_temp['lat']
        rche_df.loc[index, 'wgs1984_longitude'] = dict_temp['long']

In my experience, this approach seems slower than using an approach like apply or map, but as always, it's up to you to decide how to make the performance/ease of coding tradeoff.

Solution 2

Another way based on this question:

for index, row in rche_df.iterrows():
    if isinstance(row.wgs1984_latitude, float):
        row = row.copy()
        target = row.address_chi        
        dict_temp = geocoding(target)
        
        rche_df.at[index, 'wgs1984_latitude'] = dict_temp['lat']
        rche_df.at[index, 'wgs1984_longitude'] = dict_temp['long']

This link describe difference between .loc and .at. Shortly, .at faster than .loc.

Comments

[Andy Hayden]

This is not strictly true, they may not be copies. Specifically if the dtype is the same for all cols

[Peter Ehrlich]

This gave a copy warning for me. Ended up using: stackoverflow.com/questions/33518124/…

[dumbledad]

Don't you get the index back anyway? See @jpp's answer to Pandas for loop over dataframe gives too many values to unpack. The error I get from the code in this answer is ValueError: too many values to unpack (expected 2)

[Gamma032]

I found that I had to run df = df.reset_index() to get this working without an index error because I had chopped and sliced my dataframe.