How to retrieve tweets from command line?
Solution 1
If you don't want to use the Twitter API, you could grab the RSS feed of the Twitter profile using a bash script and then proceed to format it from there.
Since Twitter API has deprecated the RSS feed, you can workaround this by generating RSS feed using the search results.
Here's the RSS feed of my tweets.
You would have to put together the necessary bash script though. From fetching the RSS feed to formatting the tweets as per your requirements.
Solution 2
This is my script made for screensaver usage
#!/bin/bash
user="$1"
user="${user:=pontifex_pl}"
last_status_url=$(lynx -dump https://twitter.com/$user \
| grep -i "$user/status/" \
| head -1 \
| awk '{print $2}')
lynx -dump "$last_status_url" \
| grep 'Twitter:' -m1 -A4 \
| tr -d '\n' \
| sed -e 's/[^"]*"//' -e 's/".*//' \
| tr -s ' '
echo
Solution 3
You can use twarc
to give an example if you want to archive Nassim Nicholas Taleb's tweets you use.
twarc timeline nntaleb --format csv> taleb.csv
You can use other formats too: --format {json,csv,csv-excel}
How to configure twarc
:
The only way to get the complete tweets is using the API
unfortunately even if the app is transparent you need to have an initial setup and configuration, you also need to apply for the API
(
As of July 2018, you must apply for a Twitter developer account and be approved before you may create new apps.
)
Once you've got your application keys you can tell twarc
what they are with the configure command.
twarc configure
This will store your credentials in a file called .twarc
in your home directory so you don't have to keep entering them in. If you would rather supply them directly you can set them in the environment (CONSUMER_KEY
, CONSUMER_SECRET
, ACCESS_TOKEN
, ACCESS_TOKEN_SECRET
) or using command line options (--consumer_key
, --consumer_secret
, --access_token
, --access_token_secret
).
Solution 4
I made a tool that should do almost exactly what you described: twitter-screen-scrape . By default it will output in JSON, with metadata, but it's trivial to pipe the output through something like underscore-cli to strip out everything you don't want.
$ twitter-screen-scrape -u slang800 | underscore pluck text --outfmt text > outputfile
Solution 5
You could go the python + tweepy route by:
- Creating your own Twitter Application (To get API Keys)
- Create Access tokens for you twitter account
- Use a script such as this together with your credentials: https://gist.github.com/yanofsky/5436496
I just tested it and it works great. There are however users that protect their tweets, so it might not let you download everything. But that's a feature of twitter.
Limit still is 3200 and you get a CSV file.
Related videos on Youtube
Strapakowsky
Updated on September 18, 2022Comments
-
Strapakowsky over 1 year
Is there a simple way to retrieve someone's tweets from the command line?
The account would be public, and the command - or script, if necessary - would retrieve all or a specified number of most recent tweets to a text file, one tweet per line, without the metadata, the newest in the first line.
Without the use of API, only bash.
-
Admin over 11 yearsuse python api, easy and sweet
-
Admin over 11 yearsnot the way I would prefer, but out of curiosity which python API, any link, or is it in the repositories?
-
Admin over 11 yearsThere is python-twitter API wrapper.
-
-
Strapakowsky over 11 yearsThanks @jokerdino. I am fine with formatting the file, if I have access to the raw file. As I ask here (webapps.stackexchange.com/questions/34066/…), there used to be an xml with all tweets, but no more. The rss feed has a limited time range and I couldn't find how to download, for example, all your tweets from the link you sent. Can you help?
-
Admin over 11 yearsLooks good, thank you @g_kaya. How can I specify, for example, all tweets, or the maximum allowed, or 400 tweets? What is the callback and the rpp options? Can you recommend a command-line JSON parser?
-
Admin over 11 yearsI've edited my answer as per your questions. Hope it helps :)
-
jokerdino over 11 yearsApparently, you can only access tweets of up to 9 days old. Anything older than that is not available through search or even through the API.
-
Admin over 11 yearsI'm glad if they're useful, you're welcome :)
-
Strapakowsky over 11 yearsAre you sure there is no way to get tweets older than 9 days? I opened a question here: webapps.stackexchange.com/questions/34070/…
-
Strapakowsky over 11 yearsGoing directly to the user's page allows you to keep scrolling down so that the javascript loads the next page without limits. Would that be exploitable to retrieve all tweets?
-
jokerdino over 11 yearsThis link says you can't fetch more than a week's old tweets using search API. And I am not sure about fetching tweets using JavaScript. From what I know, you can't read more than about 3000 tweets or so from the timeline.
-
Admin over 8 yearsNo longer working
-
Admin over 7 years"errors":[{"message":"The Twitter REST API v1 is no longer active. Please migrate to API v1.1. dev.twitter.com/docs/api/1.1/…}]}