Hello, I have a data collection with a large number of records (over 300k) and I need to delete all and upload new updated records. This particular collection has no relationship fields other than standard (createdby, etc.). Since it would be very time consuming to do this via UI in a table (select 100 at a time and click delete button) I went to pdadmin and deleted all records there. Everything was working fine until this morning when I tried to upload new records (via upload plugin/ excel spreadsheet) and then the entire application crashed and was inaccessible.
The support team (where I host my nocobase app) said my attempt to upload so many records exceeded my server/CPU resources and that’s what crashed the application.
Questions:
-Is deleting records via pgadmin a recommended way to remove delete larger numbers of records? If not, what is the recommended way to do this?
-I still need to upload my new records (now around 400k). Is there a recommended way to do this (via upload records Pro plugin or something else) so that I don’t exceed my resources limitations?
well it’s via sql. so in your case, pgadmin. but if not, you can use my plugin @bunnarin/plugin-block-sql-console because before update or delete query, it tries to tell you the count of affected row first as a safeguard.
and for your server limit problem. I think you can batch those create api together using my other plugin: @bunnarin/plugin-action-bulk-api. and then you can just call /api/{collectionName}:bulkCreate and pass an array of those create obj to the request body
Just an FYI for anyone else reading this thread. If you’re working on a dev setup with more limited CPU/ server resources, you can limit Async background processes (like importing or updating very large numbers of records) by adding these lines to your docker-compose.yml file:
Add these after last line under “environment:”