Shell script to clone every repo of a github organization

Question

I have a normal organization repo list from curl response with command as below:

curl "https://api.github.com/orgs/[organization]/repos?access_token=[access_token]"

The output of this command is huge. I want to extract every repo's clone_url parameter, loop over those URLs and run git clone on it.

Below is the glimpse of one entry of tha JSON output curl gives me:

[
  {
    "id": 42059877,
    "name": "customrepo",
    "full_name": "SampleOrg/customrepo",
    "owner": {
      "login": "SampleOrg",
      "id": 14164701,
      "avatar_url": "https://avatars1.githubusercontent.com/u/14164701?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/SampleOrg",
      "html_url": "https://github.com/SampleOrg",
      "followers_url": "https://api.github.com/users/SampleOrg/followers",
      "following_url": "https://api.github.com/users/SampleOrg/following{/other_user}",
      "gists_url": "https://api.github.com/users/SampleOrg/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/SampleOrg/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/SampleOrg/subscriptions",
      "organizations_url": "https://api.github.com/users/SampleOrg/orgs",
      "repos_url": "https://api.github.com/users/SampleOrg/repos",
      "events_url": "https://api.github.com/users/SampleOrg/events{/privacy}",
      "received_events_url": "https://api.github.com/users/SampleOrg/received_events",
      "type": "Organization",
      "site_admin": false
    },
    "private": true,
    "html_url": "https://github.com/SampleOrg/customrepo",
    "description": null,
    "fork": false,
    "url": "https://api.github.com/repos/SampleOrg/customrepo",
    "forks_url": "https://api.github.com/repos/SampleOrg/customrepo/forks",
    "keys_url": "https://api.github.com/repos/SampleOrg/customrepo/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/SampleOrg/customrepo/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/SampleOrg/customrepo/teams",
    "hooks_url": "https://api.github.com/repos/SampleOrg/customrepo/hooks",
    "issue_events_url": "https://api.github.com/repos/SampleOrg/customrepo/issues/events{/number}",
    "events_url": "https://api.github.com/repos/SampleOrg/customrepo/events",
    "assignees_url": "https://api.github.com/repos/SampleOrg/customrepo/assignees{/user}",
    "branches_url": "https://api.github.com/repos/SampleOrg/customrepo/branches{/branch}",
    "tags_url": "https://api.github.com/repos/SampleOrg/customrepo/tags",
    "blobs_url": "https://api.github.com/repos/SampleOrg/customrepo/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/SampleOrg/customrepo/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/SampleOrg/customrepo/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/SampleOrg/customrepo/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/SampleOrg/customrepo/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/SampleOrg/customrepo/languages",
    "stargazers_url": "https://api.github.com/repos/SampleOrg/customrepo/stargazers",
    "contributors_url": "https://api.github.com/repos/SampleOrg/customrepo/contributors",
    "subscribers_url": "https://api.github.com/repos/SampleOrg/customrepo/subscribers",
    "subscription_url": "https://api.github.com/repos/SampleOrg/customrepo/subscription",
    "commits_url": "https://api.github.com/repos/SampleOrg/customrepo/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/SampleOrg/customrepo/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/SampleOrg/customrepo/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/SampleOrg/customrepo/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/SampleOrg/customrepo/contents/{+path}",
    "compare_url": "https://api.github.com/repos/SampleOrg/customrepo/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/SampleOrg/customrepo/merges",
    "archive_url": "https://api.github.com/repos/SampleOrg/customrepo/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/SampleOrg/customrepo/downloads",
    "issues_url": "https://api.github.com/repos/SampleOrg/customrepo/issues{/number}",
    "pulls_url": "https://api.github.com/repos/SampleOrg/customrepo/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/SampleOrg/customrepo/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/SampleOrg/customrepo/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/SampleOrg/customrepo/labels{/name}",
    "releases_url": "https://api.github.com/repos/SampleOrg/customrepo/releases{/id}",
    "deployments_url": "https://api.github.com/repos/SampleOrg/customrepo/deployments",
    "created_at": "2015-09-07T15:15:41Z",
    "updated_at": "2016-01-18T18:35:00Z",
    "pushed_at": "2017-08-21T18:23:26Z",
    "git_url": "git://github.com/SampleOrg/customrepo.git",
    "ssh_url": "[email protected]:SampleOrg/customrepo.git",
    "clone_url": "https://github.com/SampleOrg/customrepo.git",
    "svn_url": "https://github.com/SampleOrg/customrepo",
    "homepage": null,
    "size": 4682,
    "stargazers_count": 0,
    "watchers_count": 0,
    "language": "PHP",
    "has_issues": true,
    "has_projects": true,
    "has_downloads": true,
    "has_wiki": true,
    "has_pages": false,
    "forks_count": 0,
    "mirror_url": null,
    "archived": false,
    "open_issues_count": 0,
    "forks": 0,
    "open_issues": 0,
    "watchers": 0,
    "default_branch": "master",
    "permissions": {
      "admin": false,
      "push": true,
      "pull": true
    }
  },
...

How can I do that in a shell script with maybe an array or a for loop?

All I have tried is the following piece of code which reads the text file line by line and clones the repositories, but for creating this text file I need to do some processing priorly for filtering for clone_url param-values, but I want to skip this part entirely if possible.

while read p; do
    #echo $p
    git clone $p
done < Orgname_Repos.txt

Solution

Here you go:

#!/bin/bash
for i in $(curl "https://api.github.com/orgs/[organization]/repos?access_token=[access_token]" | grep -oP '"clone_url":\s*"\K[^"]+'); do
  echo git clone "$i"
done

This will output a list of the git clone commands to run, if you're happy just remove the echo to go for it.