Python Remove element from json if value exists

Question

I have a rather large geojson file which is converted from some National Weather Service data. I've trimmed it down to this sample here:

{
  "properties": {
    "name": "day1otlk"
  },
  "type": "FeatureCollection",
  "features": [
    {
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -122.71424459627099,
              40.229695635383166
            ],
            [
              -122.62484780364827,
              40.53410620541074
            ],
            [
              -122.71424459627099,
              40.229695635383166
            ]
          ]
        ]
      },
      "properties": {
        "Name": "General Thunder",
        "stroke-opacity": 1,
        "stroke-width": 4,
        "name": "General Thunder",
        "fill": "#c0e8c0",
        "fill-opacity": 0.75,
        "stroke": "#ffffff",
        "timeSpan": {
          "end": "2017-03-30T12:00:00Z",
          "begin": "2017-03-29T20:00:00Z"
        }
      },
      "type": "Feature"
    },
    {
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -108.65861565996833,
              32.91391108773154
            ],
            [
              -108.63932601964923,
              32.95521185698698
            ],
            [
              -108.65861565996833,
              32.91391108773154
            ]
          ]
        ]
      },
      "properties": {
        "Name": "General Thunder",
        "stroke-opacity": 1,
        "stroke-width": 4,
        "name": "General Thunder",
        "fill": "#c0e8c0",
        "fill-opacity": 0.75,
        "stroke": "#ffffff",
        "timeSpan": {
          "end": "2017-03-30T12:00:00Z",
          "begin": "2017-03-29T20:00:00Z"
        }
      },
      "type": "Feature"
    },
    {
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -92.67280213157608,
              38.47870651780003
            ],
            [
              -92.62448390998837,
              38.45534960370862
            ],
            [
              -92.59475154780039,
              38.493327413824595
            ],
            [
              -92.64308574626148,
              38.51669676139087
            ],
            [
              -92.67280213157608,
              38.47870651780003
            ]
          ]
        ]
      },
      "properties": {
        "Name": "10 %",
        "stroke-opacity": 1,
        "stroke-width": 4,
        "name": "10 %",
        "fill": "#8b4726",
        "fill-opacity": 0.89,
        "stroke": "#ffffff",
        "timeSpan": {
          "end": "2017-03-30T12:00:00Z",
          "begin": "2017-03-29T20:00:00Z"
        }
      },
      "type": "Feature"
    },
    {
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -92.67280213157608,
              38.47870651780003
            ],
            [
              -92.62448390998837,
              38.45534960370862
            ],
            [
              -92.59475154780039,
              38.493327413824595
            ],
            [
              -92.64308574626148,
              38.51669676139087
            ],
            [
              -92.67280213157608,
              38.47870651780003
            ]
          ]
        ]
      },
      "properties": {
        "Name": "10 %",
        "stroke-opacity": 1,
        "stroke-width": 4,
        "name": "20 %",
        "fill": "#8b4726",
        "fill-opacity": 0.89,
        "stroke": "#ffffff",
        "timeSpan": {
          "end": "2017-03-30T12:00:00Z",
          "begin": "2017-03-29T20:00:00Z"
        }
      },
      "type": "Feature"
    },
    {
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -97.09845994557838,
              38.43843745045377
            ],
            [
              -97.07114801649661,
              38.47751978088534
            ],
            [
              -97.09845994557838,
              38.43843745045377
            ]
          ]
        ]
      },
      "properties": {
        "Name": "5 %",
        "stroke-opacity": 1,
        "stroke-width": 4,
        "name": "5 %",
        "fill": "#b47f00",
        "fill-opacity": 0.89,
        "stroke": "#ffffff",
        "timeSpan": {
          "end": "2017-03-30T12:00:00Z",
          "begin": "2017-03-29T20:00:00Z"
        }
      },
      "type": "Feature"
    }
  ]
}

I'm looking to remove the elements where name has a % in it. I don't want those coordinates or anything included.

Here's my code:

import json

with open('test.geojson') as data_file:
    data = json.load(data_file)

for element in data["features"]:
    if '%' in element["properties"]["name"]:
        del element["type"]
        del element["properties"] # Deletes the properties
        del element["geometry"] # Deletes the  coords


with open('test_output.geojson', 'w') as data_file:
    data = json.dump(data, data_file)

This works well enough to remove the element's sub keys, but I'm left with output that looks like:

{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}

I've also tried to use

for element in data["features"]:
        if '%' in element["properties"]["name"]:
                    data["features"].remove(element)

but that seems to delete only the last element in the sample group, which is the 5 % group. It should be removing the 10 %, 20 % and the 5 % groups.

Is there a way to remove the element from data["features"] if name has a % in it all together so I'm left with clean json output? In this sample data, the only data["features"] I should have left are the General Thunder and no empty brackets.

Solution 1

The issue with using del element["type"], del element["properties"] and del element["geometry"] is that it only removes those items from properties dict of that element. Not the element itself.

For your 2nd item, when you're iterating over a list like in for element in data["features"]:, it's not good to modify a list or object while iterating over it (which is what's happening with data["features"].remove(element)). Also, list.remove() removes an item with that value. So element gets used in a value context, not as that element.

It's better to create a new list and then assign that. What you could do is:

new_features = []
for element in data["features"]:
    if '%' not in element["properties"]["name"]:    # note the 'not'
        new_features.append(element)                # new_features has the one's you want
# and then re-assign features to the list with the elements you want
data["features"] = new_features

Solution 2

Use a simple filter:

no_percent = lambda feature: '%' not in feature['properties']['name']
data['features'] = filter(no_percent, data['features'])

Or as a list comprehension:

data['features'] = [feature for feature in data['features']
                      if '%' not in feature['properties']['name']]

Comments

[aneroid]

So much more Pythonic to use filter instead.

[Patrick Haugh]

It's worth noting that in Python 3 filter returns an iterator. You would need to data['features'] = list(filter(...))

[Pat]

This is a great method too, one I'm not familiar with yet - but a great example to continue learning! Thank you

[aneroid]

FWIW, the list comprehension method in @triptych's answer is the way I would have done it, had I not been lacking coffee & sleep. It avoids assigning the unneeded new_features list, and is faster.