How to access serializer.data on ListSerializer parent class in DRF?
Solution 1
At the point in the trace where I try to access serializer.data
and get the KeyError, I note that serializer.data
only contains key/vaule pairs from the initial_data
, not the instance data (hence, I suppose, the KeyError; some model fields' keys are not present as it is a partial_update
request). However, serializer.child.data
does contain all the instance data for the last child in the list.
So, I go to the rest_framework/serializers.py
source where data
is defined:
249 @property
250 def data(self):
251 if hasattr(self, 'initial_data') and not hasattr(self, '_validated_data'):
252 msg = (
253 'When a serializer is passed a `data` keyword argument you '
254 'must call `.is_valid()` before attempting to access the '
255 'serialized `.data` representation.\n'
256 'You should either call `.is_valid()` first, '
257 'or access `.initial_data` instead.'
258 )
259 raise AssertionError(msg)
260
261 if not hasattr(self, '_data'):
262 if self.instance is not None and not getattr(self, '_errors', None):
263 self._data = self.to_representation(self.instance)
264 elif hasattr(self, '_validated_data') and not getattr(self, '_errors', None):
265 self._data = self.to_representation(self.validated_data)
266 else:
267 self._data = self.get_initial()
268 return self._data
Line 265 is problematic. I can replicate the error by calling serializer.child.to_representation({'uuid': '87956604-fbcb-4244-bda3-9e39075d510a', 'product_code': 'foobar'})
at the breakpoint.
Calling partial_update()
works fine on a single instance (because self.instance
is set, self.to_representation(self.instance)
works). However, for a bulk partial_update() implementation, self.validated_data
is missing model fields, and to_representation()
won't work, so I won't be able to access the .data
property.
One option would be to maintain some sort of self.instances
list of Product instances, and override the definition of data
on line 265:
self._data = self.to_representation(self.instances)
I'd really prefer an answer from someone more experienced in this sort of problem though, as I'm not sure if that's a sensible solution, hence I'm leaving the bounty open in the hope that someone can suggest something smarter to do.
Solution 2
As mentioned in the comment i still think the exception could be because of the user field in BulkProductSerializer class, not really anything to do with ListSerializer
There might be another minor error (but important) in the serializer DRF as mentioned in the documentation here. Here is how to specify a list_serializer_class
:
class CustomListSerializer(serializers.ListSerializer):
...
class CustomSerializer(serializers.Serializer):
...
class Meta:
list_serializer_class = CustomListSerializer
Note that it's specified inside of the Meta
class, not outside. So i think in your code, it will not understand to switch to the List Serializer with many=True
. That should cause the not-updating problem.
Update - Add example for updating nested list serializer
It seems that the question was more about a generic way to implement updating for nested List Serializer rather than the actual error. Therefore, i will try to provide a sample code.
Some notes:
- If we use ModelViewSet, the list route won't allow PUT or PATCH, so neither update nor partial_update will be called (reference). Therefore i use POST directly, this is much simpler.
- If you want to use PUT/PATCH, then see this answer here
- We can always add a query parameter like allow_update or partial directly to the Post request to differentiate between POST/PUT/PATCH
- Instead of using uuid like the question i will use the normal id, it should be very much the same
It was rather simple
For reference, models look like this:
class Product(models.Model):
name = models.CharField(max_length=200)
user = models.ForeignKey(User, null=True, blank=True)
def __unicode__(self):
return self.name
Step 1: Make sure serializer change to ListSerializer
class ProductViewSet(viewsets.ModelViewSet):
serializer_class = ProductSerializer
queryset = Product.objects.all()
def get_serializer(self, *args, **kwargs):
# checking for post only so that 'get' won't be affected
if self.request.method.lower() == 'post':
data = kwargs.get('data')
kwargs['many'] = isinstance(data, list)
return super(ProductViewSet, self).get_serializer(*args, **kwargs)
Step 2: Implement the ListSerializer by overriding create function
class ProductListSerializer(serializers.ListSerializer):
def create(self, validated_data):
new_products = [Product(**p) for p in validated_data if not p.get('id')]
updating_data = {p.get('id'): p for p in validated_data if p.get('id')}
# query old products
old_products = Product.objects.filter(id__in=updating_data.keys())
with transaction.atomic():
# create new products
all_products = Product.objects.bulk_create(new_products)
# update old products
for p in old_products:
data = updating_data.get(p.id, {})
# pop id to remove
data.pop('id')
updated_p = Product(id=p.id, **data)
updated_p.save()
all_products.append(updated_p)
return all_products
class ProductSerializer(serializers.ModelSerializer):
user = serializers.SlugRelatedField(slug_field='username', queryset=User.objects.all())
id = serializers.IntegerField(required=False)
class Meta:
model = Product
fields = '__all__'
list_serializer_class = ProductListSerializer
Related videos on Youtube
Escher
Updated on September 26, 2022Comments
-
Escher about 1 year
I'm getting an error when trying to access
serializer.data
before returning it in theResponse(serializer.data, status=something)
:Getting KeyError when attempting to get a value for field
<field>
on serializer<serializer>
.This occurs on all fields (because it turns out I'm trying to access
.data
on the parent and not the child, see below)The class definition looks like this:
class BulkProductSerializer(serializers.ModelSerializer): list_serializer_class = CustomProductListSerializer user = serializers.CharField(source='fk_user.username', read_only=False) class Meta: model = Product fields = ( 'user', 'uuid', 'product_code', ..., )
CustomProductListSerializer
is aserializers.ListSerializer
and has an overriddensave()
method that allows it to correctly handle bulk create and update.Here's an example view from the bulk Product
ViewSet
:def partial_update(self, request): serializer = self.get_serializer(data=request.data, many=isinstance(request.data, list), partial=True) if not serializer.is_valid(): return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) serializer.save() pdb.set_trace() return Response(serializer.data, status=status.HTTP_200_OK)
Trying to access
serializer.data
at the trace (or the line after, obviously) causes the error. Here's the full trace (tl;dr skip below where I diagnose with debugger):Traceback (most recent call last): File "/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner response = get_response(request) File "/lib/python3.5/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response response = self._get_response(request) File "/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response response = self.process_exception_by_middleware(e, request) File "/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view return view_func(*args, **kwargs) File "/lib/python3.5/site-packages/rest_framework/viewsets.py", line 86, in view return self.dispatch(request, *args, **kwargs) File "/lib/python3.5/site-packages/rest_framework/views.py", line 489, in dispatch response = self.handle_exception(exc) File "/lib/python3.5/site-packages/rest_framework/views.py", line 449, in handle_exception self.raise_uncaught_exception(exc) File "/lib/python3.5/site-packages/rest_framework/views.py", line 486, in dispatch response = handler(request, *args, **kwargs) File "/application/siop/views/API/product.py", line 184, in partial_update return Response(serializer.data, status=status.HTTP_200_OK) File "/lib/python3.5/site-packages/rest_framework/serializers.py", line 739, in data ret = super(ListSerializer, self).data File "/lib/python3.5/site-packages/rest_framework/serializers.py", line 265, in data self._data = self.to_representation(self.validated_data) File "/lib/python3.5/site-packages/rest_framework/serializers.py", line 657, in to_representation self.child.to_representation(item) for item in iterable File "/lib/python3.5/site-packages/rest_framework/serializers.py", line 657, in <listcomp> self.child.to_representation(item) for item in iterable File "/lib/python3.5/site-packages/rest_framework/serializers.py", line 488, in to_representation attribute = field.get_attribute(instance) File "/lib/python3.5/site-packages/rest_framework/fields.py", line 464, in get_attribute raise type(exc)(msg) KeyError: "Got KeyError when attempting to get a value for field `user` on serializer `BulkProductSerializer`.\nThe serializer field might be named incorrectly and not match any attribute or key on the `OrderedDict` instance.\nOriginal exception text was: 'fk_user'."
At the L657 of the traceback (source here) I've got:
iterable = data.all() if isinstance(data, models.Manager) else data return [ self.child.to_representation(item) for item in iterable ]
This made me wonder (digging further down in the trace) why the serializer.fields were not available. I suspected it was because the serializer was a
CustomProductListSerializer
parent, and not aBulkProductSerializer
child, and I was right. In the pdb trace just before returning theResponse(serializer.data)
:(Pdb) serializer.fields *** AttributeError: 'CustomProductListSerializer' object has no attribute 'fields' (Pdb) serializer.child.fields {'uuid': UUIDField(read_only=False, required=False, validators=[]) ...(etc)} (Pdb) 'user' in serializer.child.fields True (Pdb) serializer.data *** KeyError: "Got KeyError when attempting to get a value for field `user` on serializer `BulkProductSerializer`.\nThe serializer field might be named incorrectly and not match any attribute or key on the `OrderedDict` instance.\nOriginal exception text was: 'fk_user'." (Pdb) serializer.child.data {'uuid': '08ec13c0-ab6c-45d4-89ab-400019874c63', ...(etc)}
OK, so what's the right way to get the complete
serializer.data
and return it in the resopnse for the parent serializer class in the situation described bypartial_update
in myViewSet
?Edit:
class CustomProductListSerializer(serializers.ListSerializer): def save(self): instances = [] result = [] pdb.set_trace() for obj in self.validated_data: uuid = obj.get('uuid', None) if uuid: instance = get_object_or_404(Product, uuid=uuid) # Specify which fields to update, otherwise save() tries to SQL SET all fields. # Gotcha: remove the primary key, because update_fields will throw exception. # see https://stackoverflow.com/a/45494046 update_fields = [k for k,v in obj.items() if k != 'uuid'] for k, v in obj.items(): if k != 'uuid': setattr(instance, k, v) instance.save(update_fields=update_fields) result.append(instance) else: instances.append(Product(**obj)) if len(instances) > 0: Product.objects.bulk_create(instances) result += instances return result
-
Nathan DoThe exception you've got has nothing to do with the ListSerializer. It's about the source of the user field 'fk_user.username' which i think is wrong. Therefore, code for the models are useful. My wild guest is you need a SlugRelatedField: django-rest-framework.org/api-guide/relations/#slugrelatedfield
-
-
Escher over 6 yearsThanks for the answer. The solution I posted is actually correct, as verified with the debugger and the fact that what I proposed (overriding the data property definition in my own
ListSerializer
class. You're right that the documentation specifies thatlist_serializer_class
should be inMeta
, but the only practical consequence for me of having it as a class property (rather than a property of_meta
is to render PUT/PATCH forms in the browsable API on those other endpoints (which I find useful, as the API is equiped to handle them). -
Nathan Do over 6 yearsDon't really get it: "Practical consequence" of what ? I see no extra requirements that really needs you to override DRF to provide this functionality. It is quite capable of updating partially with a ModelSerializer alternate with ListSerializer. In fact, you customize the ListSerializer just to remove the
uuid
, the rest is the same as provided here django-rest-framework.org/api-guide/serializers/… -
Escher over 6 yearsNathan, I'm sorry but you really don't understand the point of this question at all, nor the debugger output. I invite you to try to implement your own bulk PATCH endpoint with nested relationships to understand why DRF requires you to implement your own create/update logic, as stated in the docs in the section you linked to.
-
Nathan Do over 6 yearsyup probably don't get the point of the question, it wasn't very clear. Anw, for your answer below, in the link i provided in previous comment,
instance
was used as a list in ListSerializer not a normal instance:book_mapping = {book.id: book for book in instance}
. So i don't think it will be an issue like mentioned in the answer. Of course the variable name "instance" from DRF is a bit misleading. I've done PATCH with nested relationship before, i agree on implementing it the way DRF recommended. Just that i don't think we need to overrideself._data
-
Nathan Do over 6 years@Escher: i've only got what you are trying to do now. It seems the solution should still be much simpler than what you've accepted as an answer, so i've made an update on mine. Do check it out.
-
Escher over 6 yearsThanks, well done on being able to look past the misleading error in the trace. There are some interesting ideas in your solution. I do have to implement PATCH & PUT, but it's a great template for anyone who needs to think about how they're going to do bulk operations.