Based on the information you provided, could you try to set the hostname field to the service part of docker-compose.yml file? May like this:
services:
app:
image: nocobase/nocobase
hostname: <your-custom-hostname>
Because os.hostname() will try to get it, and we can not request Nodemailer to change this.