How do I use OAuth to connect to the Etrade API?

Question

E-Trade released their API recently and provided technical documentation which is somewhat useful but not complete.

Does anyone have a fully working example in C# that shows how this works?

I have been able to do the authentication using OAuth correctly, but when it comes to getting information out of my account or market data, the servers fail.

Solution 1

I was able to connect using the DevDefined OAuth Library, but i had to make some tweeks to the source to get it to work properly. I forked the repo so you can download the src i used, and build you a .dll.

Repo: GitHub

Example Class:

 public abstract class BaseOAuthRepository
{

    private static string REQUEST_URL = "https://etws.etrade.com/oauth/request_token";
    private static string AUTHORIZE_URL = "https://us.etrade.com/e/t/etws/authorize";
    private static string ACCESS_URL = "https://etws.etrade.com/oauth/access_token";

    private readonly TokenBase _tokenBase;
    private readonly string _consumerSecret;

    protected BaseOAuthRepository(TokenBase tokenBase, 
                                  string consumerSecret)
    {
        _tokenBase = tokenBase;
        _consumerSecret = consumerSecret;
    }

    public TokenBase MyTokenBase
    {
        get { return _tokenBase; }
    }

    public string MyConsumerSecret
    {
        get { return _consumerSecret; }
    }


    public OAuthSession CreateSession()
    {
        var consumerContext = new OAuthConsumerContext
        {
            ConsumerKey = MyTokenBase.ConsumerKey,
            ConsumerSecret = MyConsumerSecret,
            SignatureMethod = SignatureMethod.HmacSha1,
            UseHeaderForOAuthParameters = true,
            CallBack = "oob"
        };

        var session = new OAuthSession(consumerContext, REQUEST_URL, AUTHORIZE_URL, ACCESS_URL);    
        return session;
    }

    public IToken GetAccessToken(OAuthSession session)
    {
        IToken requestToken = session.GetRequestToken();
        string authorizationLink = session.GetUserAuthorizationUrlForToken(MyTokenBase.ConsumerKey, requestToken);
        Process.Start(authorizationLink);
        Console.Write("Please enter pin from browser: ");
        string pin = Console.ReadLine();
        IToken accessToken = session.ExchangeRequestTokenForAccessToken(requestToken, pin.ToUpper());

        return accessToken;
    }

    public string GetResponse(OAuthSession session, string url)
    {
        IToken accessToken = MyTokenBase;

        var response = session.Request(accessToken).Get().ForUrl(url).ToString();
        return response;
    }

    public XDocument GetWebResponseAsXml(HttpWebResponse response)
    {
        XmlReader xmlReader = XmlReader.Create(response.GetResponseStream());
        XDocument xdoc = XDocument.Load(xmlReader);
        xmlReader.Close();
        return xdoc;
    }

    public string GetWebResponseAsString(HttpWebResponse response)
    {
        Encoding enc = System.Text.Encoding.GetEncoding(1252);
        StreamReader loResponseStream = new
        StreamReader(response.GetResponseStream(), enc);
        return loResponseStream.ReadToEnd();
    }
}

Solution 2

Here's the code I've used to connect to the ETrade API (tested and works).

One caveat: You need to implement your own storage of user tokens. I've not included that here since the code I created is highly domain specific.

First, I added DotNetOpenAuth to the project and created an ETradeConsumer (it derives from DotNetOpenAuth's WebConsumer):

EtradeConsumer.cs

public static class ETradeConsumer
{
    public static string AccessUrl 
    { 
        get 
        { 
            return "https://etws.etrade.com/oauth/access_token"; 
        } 
    }

    public static string RequestUrl 
    { 
        get 
        { 
            return "https://etws.etrade.com/oauth/request_token"; 
        } 
    }

public static string UserAuthorizedUrl 
    {  
        get 
        { 
            return "https://us.etrade.com/e/t/etws/authorize"; 
        } 
    }
private static readonly ServiceProviderDescription ServiceProviderDescription = new ServiceProviderDescription()
{
    AccessTokenEndpoint = new MessageReceivingEndpoint(AccessUrl, HttpDeliveryMethods.PostRequest | HttpDeliveryMethods.AuthorizationHeaderRequest),
    ProtocolVersion = ProtocolVersion.V10a,
    RequestTokenEndpoint = new MessageReceivingEndpoint(RequestUrl, HttpDeliveryMethods.PostRequest | HttpDeliveryMethods.AuthorizationHeaderRequest),
    TamperProtectionElements = new ITamperProtectionChannelBindingElement[] { new HmacSha1SigningBindingElement() },
    UserAuthorizationEndpoint = new MessageReceivingEndpoint(new Uri(UserAuthorizedUrl), HttpDeliveryMethods.GetRequest | HttpDeliveryMethods.AuthorizationHeaderRequest)
};

public static DesktopConsumer CreateConsumer(IConsumerTokenManager tokenManager)
{
    return new DesktopConsumer(ServiceProviderDescription, tokenManager);
}

public static Uri PrepareRequestAuthorization(DesktopConsumer consumer, out string requestToken)
{
    if (consumer == null)
    {
        throw new ArgumentNullException("consumer");
    }

        Uri authorizationUrl = consumer.RequestUserAuthorization(null, null, out requestToken);

    authorizationUrl = new Uri(string.Format("{0}?key={1}&token={2}", ServiceProviderDescription.UserAuthorizationEndpoint.Location.AbsoluteUri, consumer.TokenManager.ConsumerKey, requestToken));
    return authorizationUrl;
}

public static AuthorizedTokenResponse CompleteAuthorization(DesktopConsumer consumer, string requestToken, string userCode)
    {
    var customServiceDescription = new ServiceProviderDescription
    {
            RequestTokenEndpoint = ServiceProviderDescription.RequestTokenEndpoint,
        UserAuthorizationEndpoint =
                new MessageReceivingEndpoint(
                string.Format("{0}?key={1}&token={2}",  ServiceProviderDescription.UserAuthorizationEndpoint.Location.AbsoluteUri,
                              consumer.TokenManager.ConsumerKey, requestToken),
                HttpDeliveryMethods.AuthorizationHeaderRequest | HttpDeliveryMethods.GetRequest),
        AccessTokenEndpoint = new MessageReceivingEndpoint(
        ServiceProviderDescription.AccessTokenEndpoint.Location.AbsoluteUri + "?oauth_verifier" + userCode + string.Empty,
                HttpDeliveryMethods.AuthorizationHeaderRequest | HttpDeliveryMethods.GetRequest),
        TamperProtectionElements = ServiceProviderDescription.TamperProtectionElements,
        ProtocolVersion = ProtocolVersion.V10a
    };

    var customConsumer = new DesktopConsumer(customServiceDescription, consumer.TokenManager);
    var response = customConsumer.ProcessUserAuthorization(requestToken, userCode);
    return response;
    }

}

Secondly, you need to create a class to manage Etrade tokens. As an example, I created the following class. It manages the tokens through an InMemoryCollection, but it really should be held somewhere else (a database, or a cookie, or something so that the user doesn't have to authenticate/authorize every single time). The ConsumerKey and ConsumerSecret tokens are things you sign up for through Etrade:

public class ETradeTokenManager : IConsumerTokenManager 
{
    private Dictionary<string, string> tokensAndSecrets = new Dictionary<string, string>();
public string ConsumerKey { get { return "YourConsumerKey"; } }
public string ConsumerSecret { get { return "YourConsumerSecret";  } }
    
public string GetTokenSecret(string token)
{
    return tokensAndSecrets[token];
}

public void StoreNewRequestToken(UnauthorizedTokenRequest request, ITokenSecretContainingMessage response)
{
    tokensAndSecrets[response.Token] = response.TokenSecret;
}

public void ExpireRequestTokenAndStoreNewAccessToken(string consumerKey, string requestToken, string accessToken, string accessTokenSecret)
{
    tokensAndSecrets.Remove(requestToken);
    tokensAndSecrets[accessToken] = accessTokenSecret;
}

public TokenType GetTokenType(string token)
{
    throw new NotImplementedException();
}
}

Finally, put the following in (I used ASP.NET MVC 3. Your framework may differ):

public ActionResult EtradeAuthorize(string returnUrl)
   {
        var consumer = ETradeConsumer.CreateConsumer(TokenManager);
    string requestToken;
    Uri popupWindow = ETradeConsumer.PrepareRequestAuthorization(consumer, out requestToken);
    var etradeViewModel = new ETradeAuthorizeViewModel(popupWindow, requestToken);
    return View(etradeViewModel);
    }

    [HttpPost]
    public ActionResult CompleteAuthorization(FormCollection formCollection)
    {
    string accessToken = "";
    var consumer = ETradeConsumer.CreateConsumer(TokenManager);
    var authorizationReponse = ETradeConsumer.CompleteAuthorization(consumer, formCollection["requestToken"], formCollection["userCode"]);
    if (authorizationReponse != null)
    {
        accessToken = authorizationReponse.AccessToken;
    }
    var etradeViewModel = new ETradeCompleteAuthorizeViewModel(formCollection["requestToken"], formCollection["userCode"], accessToken);
    return View(etradeViewModel);
    }

If you get a 400 Bad Request, take out the callbackUrl for Etrade. For some reason it throws a bad request whenever a callback URL is used. They prefer oob (Out of Band). In order to use oob, set null to the Callback URL in the Consumer.Channel.Send() method.

There are other issues. This issue: Due to a logon delay or other issue, your authentication could not be completed at this time. Please try again. is caused by the authorize portion of the call not being processed properly. Specifically, Etrade requires that the authorize URL looks as follows:

https://us.etrade.com/e/t/etws/authorize?key={yourConsumerKey}&token={requestToken}

The OAuth specification requires that the request token should be request_token={requestToken} and not token={requestToken}.

I couldn't get the Etrade API to authorize correctly with the WebConsumer but once I switched to the Desktop Consumer and manipulated the request myself, it worked correctly.

Solution 3

If you get "Due to a logon delay or other issue, your authentication could not be completed at this time. Please try again.",

then I think you put wrong keys in the authorize url.

You should follow the document by replacing the appropriate keys in this format

https://us.etrade.com/e/etws/authorize?key=&token=

Solution 4

To use the example class + GitHub's code from jejernig's answer, I used the following:

TokenBase token = new TokenBase { ConsumerKey = "oauth_consumer_key from ETRADE" }; // OAuthRepository only seems to use the consumer key
OAuthRepository rep = new OAuthRepository(token, "consumer_secret from ETRADE");
OAuthSession session = rep.CreateSession();
IToken accessToken = rep.GetAccessToken(session);

I just removed the abstract from BaseOAuthRepository, and had to fix the GitHub code because IOAuthSession.GetUserAuthorizationUrlForToken()'s parameters were reversed (I changed the rest of the code to match the interface's parameters).

I am getting the dreaded Due to a logon delay or other issue, your authentication could not be completed at this time. Please try again. message, but this may be due to an actual logon issue that I have to resolve.

View more solutions

Comments

[George Stocker]

This is not helpful because thta's not what causes the error; not to mention the OP asked for code.

[Authman Apatira]

Due to a logon delay or other issue, your authentication could not be completed at this time. Please try again. error) --- I had this issue and it took a lot of time to resolve. Look at the tokens you get back when you requested an unauthorized token. Make sure you use the entire thing! In ETrade, the token usually has encoded characters, whereas most other oauth implementations dont. so you cant use [A-za-z] regular expression to parse it. Also, you need to use urlencoded(token) in your BASESTRING when building your signature.

[George Stocker]

@AuthmanApatira I ran into the same issue, and you are correct, the token had to be URL encoded to be sent across in a querystring.

[Authman Apatira]

Glad my response could be of some help to someone =). It took me two weeks to figure that one out

[HAL9000]

I ran into this issue using RestSharp to try to connect. I think it has to do with improper Url encoding of tokens. I am going to give the devdevined etrade fork a try.

[HAL9000]

Although this sample class wasn't the best, your fork works like a charm! I started with a RestSharp implementation but just couldn't get it to work after a couple days banging on it... THANK YOU

[Jimmy]

Being a total beginner with this oauth stuff, it would be helpful to get some sample code for how to use this. My starting point is what I downloaded from the forked GitHub repo, and the etrade-supplied oath_consumer_key and consumer_secret ... BTW, in the GitHub code, the parameters for IOAuthSession.GetUserAuthorizationUrlForToken are reversed, compared to what the rest of the code seems to expect.

[Jimmy]

I figured it out, and added an answer.

[HAL9000]

Just posted first-blush C# client using the jejernig's DevDefined branch: https://github.com/bmsapp/sappworks.stocks.public

[Antony]

Are there ways to automate this so that user does not have to interact with browser at all for authentication? It would be nice if user can simply set the username and password, and be done with it.

[TinyTimZamboni]

The E*TRADE API documentation says to use us.etrade.com/e/etws/authorize but they are wrong. It's us.etrade.com/e/t/etws/authorize. Thanks

[Istiaque Ahmed]

@GeorgeStocker, I am getting in PHP SDK 'Due to a logon delay or other issue, your authentication could not be completed at this time. Please try again. ' You said : the request token should be request_token={requestToken} and not token={requestToken}. Should I change the token part in the url : us.etrade.com/e/t/etws/authorize?key={yourConsumerKey}&token‌​={requestToken} to request_token in this way : us.etrade.com/e/t/etws/authorize?key={yourConsumerKey}&reque‌​st_token={requestTok‌​en} ? That too did not work for me.

[George Stocker]

@IstiaqueAhmed It's possible they've changed the API again. This answer was relevant back in 2011, but I'm unsure of the current state of their API. It's also possible you're having another issue. I haven't read their API docs in 2 years, I hope they've improved them since then.

[George Stocker]

@IstiaqueAhmed Etrade deviates from the spec (at least they did in 2011) in that their service looked for token; that's what your URL should have for them. It wasn't clear in my answer that I was pointing out that they deviated from the spec.

[Ravi Dhoriya ツ]

hey, @IstiaqueAhmed. Did you got the solution? I'm facing same error in PHP-SDK :( please help me

[Jimmy]

Does the etrade API support refresh_token? The docs that I see at content.etrade.com/etrade/estation/pdf/… are from 2012, don't mention anything about refresh, and seem to only support oauth 1.0 (and I believe refresh_token is oauth 2), so the odds point to a negative answer, but I know of one other broker where one has to request permission to use refresh_token, so I thought I'd ask...

[George Stocker]

@Jimmy That's a good question for the etrade people; I haven't worked with this API since 2012, so I can't tell you an answer.

[Code Monkey]

Same question as above me - do you have an implementation that also does the username/password form submission via code so no human/browser intervention is needed?

[Novice Programmer]

Hi, I tried to use the above code. Can you please let me know what is ETradeAuthorizeViewModel in EtradeAuthorize(string returnUrl) action? and what is TokenManager in ETradeConsumer.CreateConsumer(TokenManager);

[Novice Programmer]

GitHub link is broken. Please share the code if you have. Thanks in Advance

[George Stocker]

@NoviceProgrammer It's possible that code was changed between the two; that TokenManager is actually ETradeTokenManager; I'm not at all sure any more, it's been years since I had access to this code; the ETradeAuthorizeViewModel IIRC is just a key value bag of the output from the Etrade calls (or input, I can never remember).

[Jimmy]

The GitHub code is now available at github.com/Investars/DevDefined.OAuth.Etrade/archive/master.‌​zip

[Jimmy]

Hmm, I am getting a nonce_used error when trying to renew a token using github.com/Investars/DevDefined.OAuth.Etrade. I just use the nonce created by the code, so I assume that is should not have been used before. The access token was received using E*TRADE's php sdk. What could I be doing wrong?