Django Mptt Efficiently Serializing Relational Data With Drf

I have a Category model that is a MPTT model. It is m2m to Group and I need to serialize the tree with related counts, imagine my Category tree is this: Root (related to 1 group)

Solution 1:

You are definitely running into a N+1 query issue, which I have covered in detail in another Stack Overflow answer. I would recommend reading up on optimizing queries in Django, as this is a very common issue.

Now, Django MPTT also has a few problems that you are going to need to work around as far as N+1 queries. Both the self.get_ancestors and self.get_descendants methods create a new queryset, which in your case happens for every object that you are serializing. You may want to look into a better way to avoid these, I've described possible improvements below.

In your get_full_name method, you are calling self.get_ancestors in order to generate the chain that is being used. Considering you always have the parent when you are generating the output, you may benefit from moving this to a SerializerMethodField that reuses the parent object to generate the name. Something like the following may work:

classRecursiveField(serializers.Serializer):

    defto_native(self, value):
        return CategorySerializer(value, context={"parent": self.parent.object, "parent_serializer": self.parent})

classCategorySerializer(serializers.ModelSerializer):
    children = RecursiveField(many=True, required=False)
    full_name = SerializerMethodField("get_full_name")
    group_count = serializers.Field(source='get_group_count')

    classMeta:
        model = Category
        fields = ('id', 'name', 'children', 'full_name', 'group_count')

    defget_full_name(self, obj):
        name = obj.name

        if"parent"in self.context:
            parent = self.context["parent"]

            parent_name = self.context["parent_serializer"].get_full_name(parent)

            name = "%s - %s" % (parent_name, name, )

        return name

You may need to edit this code slightly, but the general idea is that you don't always need to get the ancestors because you will have the ancestor chain already.

This doesn't solve the Group queries, which you may not be able to optimize, but it should at least reduce your queries. Recursive queries are incredibly difficult to optimize, and they usually take a lot of planning to figure out how you can best get the required data without falling back to N+1 situations.

Solution 2:

I have found a solution for the counts. Thanks to django-mptts function get_cached_trees, you can do following:

from django.db.models import Count


classCategorySerializer(serializers.ModelSerializer):
    defget_group_count(self, obj, field=field):
        return obj.group_count

    classMeta:
        model = Category
        fields = [
            'name',
            'slug',
            'children',
            'group_count',
        ]

CategorySerializer._declared_fields['children'] = CategorySerializer(
    many=True,
    source='get_children',
)

classCategoryViewSet(ModelViewSet):
    serializer_class = CategorySerializer

    defget_queryset(self, queryset=None):
        queryset = Category.tree.annotate('group_count': Count('group')})
        queryset = queryset.get_cached_trees()
        return queryset

Where tree is mptts TreeManager, as used in django-categories, for which I have written slightly more complicated code to this PR: https://github.com/callowayproject/django-categories/pull/145/files