Forex historical data in Python

Question

Have been searching for hours so please be kind.

Need solutions to get historical Forex data in Python.

For stocks it is easy:

import pandas as pd
import pandas_datareader as pdr

start = dt.date.today() - dt.timedelta(days=30)
end = dt.date.today()

df = pdr.DataReader('AAPL', 'google', start, end)
print(df.head())

Have tried google, yahoo, fred and oanda. Nothing seems to work.

Please give a code example of how to request the data. (In most cases one line should be fine).

Thank you.

Solution 1

Do you just need historical currency values?

Try using the forex_python module with the datetime class ( from the datetime module ). I'm using python 3 but I doubt that matters too much.

These exchange rates are the 3pm (CET) data from the European Central Bank, since 1999.

>>> from datetime import datetime
>>> from forex_python.converter import get_rate

>>> t = datetime(2001, 10, 18)  # the 18th of October, 2001
>>> get_rate("USD", "GBP", t)
0.69233
>>> get_rate("GBP", "USD", t)
1.4444
>>> 1 / 1.4444   # check
0.6923289947382997 

>>> t = datetime(2006, 6, 26)  # June 26th, 2006
>>> get_rate("GBP", "USD", t)
1.8202

So
on 18/10/01, 1 USD == 0.69 GBP,
on 26th June, 2006, 1 GBP == 1.82 USD.

Solution 2

The retail broker feed is always skewed but I don't agree that there is no good historical feed. The industry standard for FX is EBS feed. However, it is an expensive option. The FXMarketAPI offers a feed that closely matches this. It is not affiliated to any broker. The API has a pandas endpoint which helps you pull data. Though there is a limit of 1000 request for free users. you can see an example below.

URL = "https://fxmarketapi.com/apipandas"
params = {'currency' : 'EURUSD',
'start_date' : '2018-07-02',
'end_date':'2018-12-06',
'api_key':'**************'}

response = requests.get("https://fxmarketapi.com/apipandas", params=params)
df= pd.read_json(response.text)

Solution 3

Maybe you are not looking hard enough :) A very good looking chap published this a few months ago. Admittedly this is not the best code (first open source project), however, it is currently under development and continuously improving. The next version will be much more efficient and cleaner.

fx_collect

Designed to store all of FXCM's historical data locally in Mariadb like so.

    +---------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+--------+
    | date                | bidopen   | bidhigh   | bidlow    | bidclose  | askopen   | askhigh   | asklow    | askclose  | volume |
    +---------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+--------+
    | 2017-04-27 10:01:00 | 17.294000 | 17.296000 | 17.289000 | 17.290000 | 17.340000 | 17.340000 | 17.334000 | 17.335000 |    113 |
    | 2017-04-27 10:02:00 | 17.290000 | 17.298000 | 17.285000 | 17.295000 | 17.335000 | 17.342000 | 17.330000 | 17.340000 |    114 |
    | 2017-04-27 10:03:00 | 17.295000 | 17.301000 | 17.289000 | 17.299000 | 17.340000 | 17.347000 | 17.340000 | 17.344000 |     98 |
    | 2017-04-27 10:04:00 | 17.299000 | 17.300000 | 17.286000 | 17.295000 | 17.344000 | 17.345000 | 17.330000 | 17.340000 |    124 |
    | 2017-04-27 10:05:00 | 17.295000 | 17.295000 | 17.285000 | 17.292000 | 17.340000 | 17.340000 | 17.330000 | 17.336000 |    130 |
    | 2017-04-27 10:06:00 | 17.292000 | 17.292000 | 17.279000 | 17.292000 | 17.336000 | 17.336000 | 17.328000 | 17.332000 |     65 |
    | 2017-04-27 10:07:00 | 17.292000 | 17.304000 | 17.287000 | 17.298000 | 17.332000 | 17.348000 | 17.332000 | 17.345000 |    144 |
    | 2017-04-27 10:08:00 | 17.298000 | 17.306000 | 17.297000 | 17.302000 | 17.345000 | 17.350000 | 17.343000 | 17.346000 |     96 |
    | 2017-04-27 10:09:00 | 17.302000 | 17.303000 | 17.294000 | 17.294000 | 17.346000 | 17.346000 | 17.338000 | 17.338000 |     50 |
    | 2017-04-27 10:10:00 | 17.294000 | 17.296000 | 17.281000 | 17.291000 | 17.338000 | 17.338000 | 17.328000 | 17.333000 |     50 |

or if you just want the basic tools to get you started and build your own.

python-forexconnect

A Demo or Live FXCM account is required to obtain the data. They provide free 10-year historical data, in different timeframes (fxcm).

Solution 4

There is free tick based historical data from pepperstone in monthly csv format starting from 2009 (visit: https://www.truefx.com/?page=downloads) for most popular pairs, i have wrote python code using selenium to download all csv files(the script will download all csv files into folder name forex):

import datetime, time, os
from dateutil.relativedelta import relativedelta

from selenium import webdriver

tmp_dir = os.path.join(os.getcwd(), 'forex')
if not os.path.isdir(tmp_dir): os.makedirs(tmp_dir)
options = webdriver.ChromeOptions();
options.add_argument("--window-size=1300,900")
options.add_experimental_option("prefs", {
    "download.default_directory": tmp_dir,
    "download.prompt_for_download": False,
    "download.directory_upgrade": True,
    "safebrowsing.enabled": False,
    "safebrowsing.disable_download_protection": True
})
options.add_argument("--disable-gpu")
options.add_argument("--disable-extensions")
options.add_argument('--disable-logging')
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-certificate-errors-spki-list')
options.add_argument('--no-sandbox')
browser = webdriver.Chrome(options=options)

pairs = ['AUDJPY', 'AUDNZD', 'AUDUSD', 'CADJPY', 'CHFJPY', 'EURCHF', 'EURGBP', 'EURJPY', 'EURUSD', 'GBPJPY', 'GBPUSD', 'NZDUSD', 'USDCAD', 'USDCHF', 'USDJPY']
for pair in pairs:
    curr_date = datetime.datetime(2015, 1, 1)
    while curr_date + relativedelta(months=1) < datetime.datetime.now():
        file_name = '{p}-{ym2}.zip'.format(p=pair, ym2=curr_date.strftime('%Y-%m'))
        url = 'http://www.truefx.com/dev/data/{y}/{ym1}/{p}-{ym2}.zip'.format(
            y=curr_date.strftime('%Y'),
            ym1=curr_date.strftime('%B').upper()+'-'+curr_date.strftime('%Y') if curr_date <= datetime.datetime(2017, 3, 1) else curr_date.strftime('%Y-%m'),
            p=pair,
            ym2=curr_date.strftime('%Y-%m')
        )
        file_found = False
        for root, dirs, files in os.walk(tmp_dir):
            for file in files:
                if file_name in file: file_found = True 
        if not file_found:
            time.sleep(5)
            browser.get (url)
            file_downloaded = False
            while not file_downloaded:
                time.sleep(1)
                for root, dirs, files in os.walk(tmp_dir):
                    for file in files:
                        if file_name in file and not '.crdownload' in file: file_downloaded = True
            print(file_name, 'downloaded from', url)
        curr_date = curr_date + relativedelta(months=1)
print('completed')

gist source: https://gist.github.com/mamedshahmaliyev/bca9242b7ea6a13b3f76dee7a5aa111a

Solution 5

FXCM recently released an official python wrapper for forexconnect.

There is a support forum: http://www.fxcodebase.com/code/viewforum.php?f=51&sid=e2b414c06f9714c605f117f74d689a9b

There is a code snippet from an article about getting history:

from forexconnect import fxcorepy, ForexConnect
    with ForexConnect() as fx:
        try:
            fx.login("user_id", "password", "fxcorporate.com/Hosts.jsp",
                     "Demo", session_status_callback=session_status_changed)

            history = fx.get_history("EUR/USD", "H1",
                                    datetime.datetime.strptime("MM.DD.YYYY HH:MM:SS", '%m.%d.%Y %H:%M:%S').replace(tzinfo=datetime.timezone.utc),
                                    datetime.datetime.strptime("MM.DD.YYYY HH:MM:SS", '%m.%d.%Y %H:%M:%S').replace(tzinfo=datetime.timezone.utc))

View more solutions

Comments

[Yster]

Thanks for the info. (1) So, what you are saying is that there is no daily historical data available for say EUR/USD from Google, Yahoo or Fred, right? This would have been my preference and it is a bit strange, because one can get quotes from them and historical data graphs: google.com/finance?q=EURUSD (2) How about from a broker like Oanda? I tried using the following function without success: get_oanda_currency_historical_rates() .

[user3666197]

Ad-(1) No, what I say is, that if one takes any kind of data from a Venue [A], it has no value for a trading model simulations and execution on a Venue [B] as each Venue has different conditions and thus also the quote-streams for the same currency pair, their respective elasticity & hysteresis to dynamic events, as the animation for each different Broker depicts above. There are no two Brokers having the same identical response to a flow of market events. Ad-(2) What was the Oanda official Technical Support's response to your non-functional API-claims so far?

[Yster]

For Oanda, it seems one have to be a premium member: github.com/pydata/pandas-datareader/issues/296 And it seems to be expensive: oanda.com/fx-for-business/exchange-rates-api

[Gregory Fenn]

Of course, true. I'll clarify that it's the daily (3pm) data from the European Central Bank.

[user3666197]

Absolutely worth to note that. The downvoting hysteria downthere was a bit of surprise ...

[Cray]

Those are some interesting points you bring up. However can you demonstrate that different brokers really give out different data? The GIF you linked is a table of RTT (round trip times?) it's a table of how quick brokers respond? Or what is it?