Consuming paginated API using periodic Celery tasks in Django

Introduction

API (Application Programming Interface) consumption is the new normal in contemporary times as many software products have shifted focus on decoupling Backend and Frontend codebases. Backend Engineers are tasked with writing consumable APIs that their Frontend counterparts consume. In some cases, even Backend Engineers utilize some other API services to accomplish their tasks.

Some services provide an enormously large dataset so making them accessible at a single API call might not be great. Pagination then comes to the rescue. Many APIs are now paginated to make available a fraction of the data. To access other fractions, you need some extra tasks.

This article demonstrates how to set up Celery background tasks to consume paginated APIs periodically. We'll explore iterative and recursive approaches for APIs paginated using page parameters and those using next URLs. The fetched data will be stored in a Django model, overwriting previous data. Note that persisting historical data is outside the scope of this article but will be addressed in a future post on building a data warehouse.

Prerequisite

Basic familiarity with Django is assumed. Refer to the Django tutorial for an introduction.

Source code Sirneij/django_excel

Exporting Django model data as excel file (.xlsx) using openpyxl library and Google Spreadsheet API python html5 bash heroku

Implementation

Step 1: Setup Django project and app with celery

Create a Django project named django excel within a virtual environment. Ensure that django and celery are installed in your environment. The shell code below is 1 line.

Create an app named core: The shell code below is 1 line.

Register your application in settings dot py. The Python code below is 14 lines, from django excel slash settings dot py.

It is time to set up our application to utilize Celery. To do this, create a file aptly named celery dot py in your project's directory and paste the following snippet: The Python code below defines debug task, 22 lines, from django excel slash celery dot py.

That was directly lifted from Celery Django documentation. Ensure you modify lines 6 and 8 to reflect your project's name. The namespace in line 14 enables you to prefix all celery-related configurations in your settings dot py file with CELERY such as CELERY BROKER URL. Note: Capitalization of Celery-Related Configurations

Because you are literarily providing constants, the celery-related configurations in your settings dot py file are capitalized. For instance, one of the configurations is beat schedule which in Django, becomes CELERY BEAT SCHEDULE.

Next, open open your project's init dot py and append the following: The Python code below is 5 lines, from django excel slash init dot py.

To conclude celery-related configurations, let's set the following in settings dot py: The Python code below is 6 lines.

We are using redis as our broker. You can opt for RabbitMQ which is supported out-of-box by celery.

In the settings above, I am linking CELERY BROKER URL to an environment variable named REDIS URL. It normally should look like redis: slash slash 127.0.0.1:6379 on a Linux system. That means I could have set my CELERY BROKER URL and CELERY RESULT BACKEND as: The Python code below is 10 lines.

Note that CELERY RESULT BACKEND is optional as well as CELERY ACCEPT CONTENT, CELERY TASK SERIALIZER, and CELERY RESULT SERIALIZER. However, not setting the last three might result in some runtime errors mostly when dealing with databases in asynchronous email broadcasting with celery.

The stage is now set, let's set up our database. We will be consuming CoinGecko's API and will be saving some data.

Step 2: Define Full Coin model

Our model will look like this: The Python code below defines Full Coin and str, 31 lines, from core slash models dot py.

These are all the fields taken directly from CoinGecko's public API for coin markets: The JSON code below is 30 lines. Note: Commission from referral

If you use Coingecko's API, when you use my code, CGSIRNEIJ, I get some commissions. That can be a good way to support me.

null equals True makes a column nullable in SQL: The SQL code below is 4 lines.

null equals False or leaving it unset makes the column non-nullable: The SQL code below is 4 lines.

blank equals True allows the field to be optional in forms and the admin page.

Talking about the admin site, let's register the model: The Python code below defines Full Coin Admin, 32 lines, from core slash admin dot py.

With this, you can migrate your database: The shell code below is 2 lines, from Terminal.

Optionally, you can create a superuser: The shell code below is 1 line, from Terminal.

Follow the prompts.

Step 3: Create and Register Periodic Tasks

Here is the juicy part: The Python code below defines build api url, fetch coins iteratively and get full coin data iteratively for page, 80 lines, from core slash tasks dot py.

The build api url helps continuously build CoinGecko API url based on the supplied page number. The BASE API URL is: The Python code below is 2 lines, from django excel slash settings dot py.

fetch coins iteratively is the core of the program. It starts with the first page and does an "infinite" loop which breaks only when there's no data returned by the API using the iterative strategy.

Its recursive alternative is: The Python code below defines fetch coins recursively, 31 lines.

Then there is the get full coin data iteratively for page which is decorated by shared task (for task autodiscovery). We supplied some parameters:

bind equals True to access task instance via self

autoretry for equals Exception, to auto-retry on exceptions

retry backoff equals True for exponential backoff

max retries equals 5 to limit retries to 5

For this task to be periodic, we must add it to the CELERY BEAT SCHEDULE in settings dot py: The Python code below is 8 lines, from django excel slash settings dot py.

It schedules this task to run every 3 minutes (star slash 3) using crontab.

These implementations were with performance in mind. However, there is still room for improvement.

Step 4: Bonus

There are APIs whose paginations are not page -based but use the next (default DRF pagination strategy). For these systems, the last bits of data have empty next. That's the breaking point: The Python code below defines get api data, 8 lines.

That's it! I hope you enjoyed it.

Outro

Enjoyed this article? I'm a Software Engineer, Technical Writer, and Technical Support Engineer actively seeking new opportunities, particularly in areas related to web security, finance, healthcare, and education. If you think my expertise aligns with your team's needs, let's chat! You can find me on LinkedIn and X. I am also an email away.

If you found this article valuable, consider sharing it with your network to help spread the knowledge!