Python Automation Scripts That Run Reliably

Most businesses run more Python than they realise: a script that pulls orders from an API every hour, another that cleans a spreadsheet and loads it into the CRM, a third that emails a report each morning. They start as quick wins on someone's laptop. Then they become part of daily operations, and their weaknesses show: a run hangs forever on a slow API, a crash halfway through loads half the data twice on the next attempt, and nobody notices failures until a customer asks why their order is missing. Python automation scripts can be just as dependable as larger systems if they are written with a few habits in mind. This guide covers timeouts and retries, efficient API use, safe reruns, logging and alerting, scheduling and configuration.

Why Python automation scripts fail in production

The problems are remarkably consistent across scripts:

• No timeouts, so one unresponsive server blocks the run indefinitely
• No retries, so a brief network error fails the whole job, or blind retries that hammer a struggling API
• Opening a new connection for every request, slowing large runs
• No record of progress, so a failure halfway forces a full rerun that duplicates work
• Output printed to a terminal nobody watches
• Credentials pasted into the code, and settings that only work on one machine

Each has a simple, well-known fix.

Always set timeouts

The requests library's quickstart is direct about this: nearly all production code should use the timeout parameter in nearly all requests, and failing to do so can cause your program to hang indefinitely. The same applies to database connections and any other network call. Choose a timeout that reflects how long a healthy response takes, plus a margin.

Retry carefully, and reuse connections

Temporary failures, such as a dropped connection or a 503 response, often succeed on a second attempt. Rather than writing retry loops by hand, configure them once. Requests can mount an adapter using urllib3's Retry class, which supports a total number of attempts, a backoff factor that increases the wait between attempts, a list of status codes to retry and the methods that are safe to retry. By default, it only retries methods considered idempotent, which notably excludes POST, so a retried request does not create a second order.

Use a Session for repeated calls to the same service. The requests documentation on session objects explains that a session uses connection pooling, so the underlying connection is reused across requests to the same host, which can make large runs noticeably faster.

Make reruns safe

Assume every script will fail halfway at some point. Design it so running it again is harmless:

• Process in batches and record progress, such as the last processed ID or timestamp, after each batch completes
• Use the destination's upsert or a unique external ID, so writing the same record twice updates rather than duplicates
• Separate reading, transforming and writing, so a failure in one stage does not leave partial results in another
• Keep a record of rejected rows with the reason, rather than skipping them silently

Our guide to ETL pipeline development covers these ideas for larger data pipelines.

Respect rate limits and volume

APIs limit how fast you can call them, and scripts that ignore those limits get blocked at the worst moment. Read the provider's documentation for its limits, request only the fields you need, use bulk or batch endpoints where they exist, and pace requests when processing large backlogs. If the API returns 429 Too Many Requests, back off rather than retrying immediately. For very large volumes, consider whether a nightly bulk export from the source is more efficient than thousands of individual calls.

Test before touching real data

Give each script a dry-run mode that reads and transforms data and reports what it would write, without writing anything. Run it against a test account or sandbox where the destination offers one. Keep a few sample input files that cover awkward cases, such as missing fields, unusual characters and duplicate records, and check the script handles each of them before every change goes live.

Log for someone who was not there

Replace print statements with Python's logging module, writing to a file or a central log service with timestamps and levels. Log the start and end of each run, counts of records read, written and rejected, and every error with enough context to find the record involved. Never log passwords, tokens or full personal records. Keep log files from growing without limit by rotating them or sending them to a service with a retention period.

Then make failures visible. A script that exits with an error code lets the scheduler detect failure; an alert by email or chat when a run fails, or when it has not run at all, means problems are noticed within hours rather than weeks.

Worked example: a reliable sync script skeleton

The simplified, untested Python sketch below fetches new orders from an API in pages, writes them to a destination using an external ID, saves progress after each page, and logs counts. The API, destination function and file paths are placeholders.

import json
import logging
import os
import sys

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

logging.basicConfig(
    filename="order_sync.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)
log = logging.getLogger("order_sync")

API_URL = os.environ["ORDERS_API_URL"]
API_TOKEN = os.environ["ORDERS_API_TOKEN"]
STATE_FILE = "order_sync_state.json"

def make_session():
    retry = Retry(total=5, backoff_factor=1,
                  status_forcelist=[429, 500, 502, 503, 504],
                  allowed_methods=["GET"])
    session = requests.Session()
    session.mount("https://", HTTPAdapter(max_retries=retry))
    session.headers["Authorization"] = f"Bearer {API_TOKEN}"
    return session

def load_cursor():
    if os.path.exists(STATE_FILE):
        with open(STATE_FILE) as f:
            return json.load(f).get("last_id", 0)
    return 0

def save_cursor(last_id):
    with open(STATE_FILE, "w") as f:
        json.dump({"last_id": last_id}, f)

def upsert_order(order):
    """Placeholder: write to the destination using order['id'] as external ID."""
    raise NotImplementedError

def main():
    session = make_session()
    last_id = load_cursor()
    written = rejected = 0
    while True:
        resp = session.get(API_URL, params={"after_id": last_id, "limit": 100},
                           timeout=(5, 30))
        resp.raise_for_status()
        orders = resp.json().get("orders", [])
        if not orders:
            break
        for order in orders:
            try:
                upsert_order(order)
                written += 1
            except ValueError as exc:
                rejected += 1
                log.warning("Rejected order %s: %s", order.get("id"), exc)
        last_id = orders[-1]["id"]
        save_cursor(last_id)
    log.info("Run complete: written=%d rejected=%d last_id=%s", written, rejected, last_id)

if __name__ == "__main__":
    try:
        main()
    except Exception:
        log.exception("Run failed")
        sys.exit(1)

If the run fails on the fifth page, the next run resumes after the last saved ID, and the external ID makes any repeated writes harmless.

Schedule and configure properly

Run scripts from a scheduler on a server or managed service, not a personal laptop. Prevent overlapping runs, either with the scheduler's settings or a simple lock, so a slow run is not joined by a second copy. Keep settings and credentials in environment variables or a secrets manager rather than in the code, and pin dependency versions so an upgrade does not change behaviour unexpectedly. Keep the code in version control with a short README explaining what it does, when it runs and who owns it.

Our guide to Python invoice parsing shows these habits applied to a document processing script.

Common mistakes in automation scripts

• Network calls without timeouts
• Retrying POST requests that create records
• New connection for every request in a large run
• No saved progress, so failures cause duplicate work
• Print statements instead of logs, and no alert on failure
• Credentials in the code and scripts running on one person's laptop

Frequently Asked Questions

How do you make Python automation scripts reliable?

Set timeouts on every network call, retry temporary failures with backoff, save progress so reruns are safe, log to a file or service, alert on failures and run from a proper scheduler.

Should every failed request be retried?

No. Retry temporary failures, such as timeouts or 503 responses, with increasing delays. Avoid retrying requests that create records unless the destination deduplicates them.

Why use a requests Session?

According to the requests documentation, a session uses connection pooling, so connections to the same host are reused, which can make many requests noticeably faster.

Where should automation scripts run?

On a server or managed scheduler with version-controlled code, environment-based configuration and monitoring, rather than on an individual's computer.

Turn fragile scripts into dependable automation

We review, harden and run Python automation so it keeps working when APIs and data misbehave. See our custom app development services, or talk to us about your scripts.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.