Two ideas, composed.
Concurrency limit. Promises start running when they are created, so the runner cannot accept promises. It accepts task functions and decides when to call each one. The simplest correct pool starts limit workers. Each worker is an async loop that claims the next unclaimed index, runs that task, stores the result at the same index, and repeats until nothing is left. Because JavaScript runs one job at a time, nextIndex++ needs no lock, and no two workers get the same task. At most limit tasks are in flight, a slot is reused as soon as a task ends, and results[index] keeps input order whatever the finish order.
Retry with backoff. A loop calls the task, returns on success, and after a failure waits baseDelayMs * 2 ** attempt (100, 200, 400, ...) before the next attempt. When the attempts are used up it throws the last error. Retrying lives inside the worker, so a task being retried keeps its slot and the limit still holds during backoff.
Failure. The runner is fail-fast like Promise.all. The first task that exhausts its retries rejects the whole call, and a failed flag stops workers from claiming new tasks. Tasks already in flight cannot be cancelled and finish on their own.
What this skips, on purpose: jitter on the delay and a maximum delay, a predicate to decide which errors are worth retrying, a per-task timeout, cancellation of in-flight tasks via AbortSignal, and settled-style results (allSettled) instead of fail-fast. Each is a small addition, but each adds options and states, so add them when a requirement asks for them.