• notice
  • Congratulations on the launch of the Sought Tech site

Trying to run a task in a separate thread on Heroku, but no new threads seem to be open

I have a Django app with a view in the admin that allows employee users to upload a csv , which is then passed to a script that builds and updates the project in the database from the data.The view runs the script in a new thread, and returns an "upload started" success message.

application/product/admin.py strong>

from threading import Thread
#...
from apps.products.scripts import update_products_from_csv

@admin.register(Product)
class ProductAdmin(admin.ModelAdmin):
    # normal ModelAdmin stuff

    def upload_csv(self, request):
        if request.method=='POST':
            csv_file=request.FILES['csv_file']
            t=Thread(target=update_products_from_csv.run, args=[csv_file])
            t.start()
            messages.success(request, 'Upload started')
            return HttpResponseRedirect(reverse('admin:products_product_changelist'))

apps/products/scripts/update_products_from_csv.py

import csv
import threading
from time import time
#...

def run(upload_file):
    # print statements just here for debugging

    print('Update script running', threading.currentThread())

    start_time=time()
    print(start_time)

    decoded_file=upload_file.read().decode('utf-8').splitlines()
    csv_data=[d for d in csv.DictReader(decoded_file)]
    print(len(csv_data))

    for i, row in enumerate(csv_data):
        if i % 500==0:
            print(i, time()-start_time)
        # code that checks if item needs to be created or updated and logs accordingly

    print('Finished', time()-start_time)

In development, this works fine."Upload started" message appears almost immediately in browser and prints it in console on Thread-3 or Thread-5 or whatever Start, then execute all other print statements. Once done, I can query the EntryLog model and confirm that it has changed.

When I push it to Heroku, I still get "Uploading" in the browser immediately Start" message, but when I look at the log, it prints Thread-1 instead of Thread-[any other number].After that I see the start_time print statement execute, but then the response starts and no other print statement runs.After a while, I queried the EntryLog model, but nothing changed.

From what I've read, it sounds like I should be able to do it locally on Heroku Using a thread, but it seems like it executes the script in the main thread and then silently kills it when the response starts.

uj5u.com enthusiastic netizens replied:

It turns out that Heroku actually opens a new thread just fine.When I call it shows Thread-1print(threading.currentThread()) is a red herring.In my development environment (Windows), the spawned thread always prints Thread-[number greater than 1], but using the one I'm sure the execution completes Further testing of simple threads always prints Thread-1 in the Heroku environment.Possible differences in library functionality on Windows and Linux? (This is my first time using a thread library, apologies if I'm missing something obvious.)

The actual problem seems to be where I am reading the csv file.A lot of typographic statements that I later narrowed down to the exact line where everything stopped working.I tried a simple quiz just reading a txt file in a new thread and got the same result. didn't throw any errors or anything, nothing after that.I moved the code that reads and decodes the archive into the main thread's view, then passed the extracted data to the new thread and everything worked fine.

Tags

Technical otaku

Sought technology together

Related Topic

1 Comments

Leave a Reply

+