I spent an afternoon chasing "1 error" in a sync log, and the job was never actually broken.
I was re-running a sync of invoice items, twice. Both times: around 85k items from the API, all but one updated, 1 error. Same thing each run, always at exactly 10 minutes in. I was already convinced one item was missing somewhere.
It wasn't. The job had $timeout = 600 and the sync takes a bit over 10 minutes. At the 10-minute mark the worker tries to kill the job and record it in failed_jobs. That insert blew up with null value in column "id" of relation "failed_jobs", because our table wasn't created by the stock Laravel migration and the id column never got a sequence. The exception landed inside my own try/catch around a single item, got counted as "1 error", and the job happily ran to the end.
Two fixes, in this order: raise the job timeout first (keep it below queue retry_after), then give the table back what $table->id() would have created:
CREATE SEQUENCE failed_jobs_id_seq OWNED BY failed_jobs.id;
ALTER TABLE failed_jobs ALTER COLUMN id SET DEFAULT nextval('failed_jobs_id_seq');Lesson I'm keeping: an always-empty failed_jobs table is a symptom, not a success.