Appending arrays to dataframe (python)

Question

So I ran a time series model on a small sales data set, and forecasted sales for next 12 periods. With the following code:

 mod1=ARIMA(df1, order=(2,1,1)).fit(disp=0,transparams=True)
    y_future=mod1.forecast(steps=12)[0]

where df1 contains the sales values with months being the index. Now I'm storing the predicted values in the following manner:

pred.append(y_future)

Now, I need to append the forecasted values to the original dataset df1, preferably with the same index. I'm trying to use the following code:

df1.append(pred, ignore_index=False)

But I'm getting the following error:

TypeError: cannot concatenate a non-NDFrame object

I've tried converting pred variable to list and then appending, but to no avail. Any help will be appreciated. Thanks.

Solution 1

One solution could be appending the new array to your dataFrame to the last position using df.loc

df.loc[len(df)] = your_array

But this is not efficient cause if you want to do it several times, it will have to get the length of the DataFrame for each new append.

A better solution would be to create a dictionary of the values that you need to append and append it to the dataFrame.

df = df.append(dict(zip(df.columns, your_array)), ignore_index=True)

Solution 2

You can append your results into a dictionary list and then append that dictionary list to data frame.

Let's assume that you want to append your ARIMA forecasted results to the end of actual data frame with two columns "datetime" (YYYY-MM-DD) and "value" respectively.

Steps to follow

• First find the max day in datetime column of your actual data frame and convert it to datetime. We want to assign future dates for our forecasting results.
• Create an empty dictionary list and inside a loop fill it by incrementing datetime value 1 day and place a forecasted result subsequently.
• Append that dictionary list to your dataframe. Don't forget to reassign it to itself as left hand value since append function creates a copy of appended results data frame.
• Reindex your data frame.

Code

lastDay = dfActualData[dfActualData['datetime'] == dfActualData['datetime'].max()].values[0][0]
dtLastDay = lastDay.to_pydatetime("%Y-%m-%d")

listdict = []
for i in range(len(results)):
    forecastedDate = dtLastDay + timedelta(days = i + 1)
    listdict.append({'datetime':forecastedDate , 'value':results[i]})

dfActualData= dfActualData.append(listdict, ignore_index=True)
dfActualData.reset_index(drop=True)

Comments

[IndigoChild]

hi@saloua thanks for this, but i wanted to append the forecasted values as rows to the original sales dataframe, the method you've stated would probably hold for columns I believe.