Two categories of tooling show up in almost every large financial services firm and are almost never discussed in engineering blogs: enterprise job schedulers and managed file transfer platforms. They are not exciting. They do not appear in conference talks. The vendors who sell them have not rebranded around AI. But if you work in financial infrastructure long enough, you will inherit one or both of them, and understanding how they actually work will save you a lot of pain.
This post covers both: what they are, the major platforms you are likely to encounter, how they fit into a modern infrastructure, and the gotchas that take people by surprise.
Enterprise job schedulers
What they are
An enterprise job scheduler is a system for defining, triggering, monitoring, and managing batch jobs across an organisation. The word "enterprise" here means something specific: these are not cron jobs. Enterprise schedulers manage dependencies between jobs, handle failures with configurable retry and escalation logic, maintain audit trails, provide centralised visibility across many systems, and integrate with the monitoring and alerting infrastructure that operations teams actually use.
In financial services, the canonical use cases are:
- End-of-day processing runs (positions, P&L calculations, reconciliations)
- Data feeds from exchanges and counterparties that arrive on schedules
- Report generation and distribution
- Database maintenance jobs (index rebuilds, archival, replication checks)
- Regulatory reporting extracts that need to complete by specific deadlines
The key characteristic is that these jobs have dependencies on each other, time constraints that are business-critical, and failure modes that need human attention. A cron job that fails sends no alert. An enterprise scheduler job that fails pages the on-call, holds downstream dependent jobs, and creates an audit record.
The major platforms
Control-M (BMC) is the dominant platform in financial services. If you work at a bank, an asset manager, or a large insurance company, there is a reasonable chance you are running Control-M somewhere. It has been around since the 1980s and has accumulated a staggering amount of functionality. The interface is dated but it works. The key concepts are jobs, job flows, resources, and conditions. Jobs have predecessors (they wait for other jobs to complete successfully before running) and conditions (they can set boolean flags that other jobs check before starting).
Autosys (Broadcom, formerly CA Technologies) is the other major incumbent in financial services. Architecturally similar to Control-M: centralised server (the Event Server), agents on the machines where jobs actually run, and a definition language (JIL, Job Information Language) for creating jobs. Autosys has a reputation for being somewhat more scriptable than Control-M, which makes it easier to manage as code.
/* Simple Autosys job definition */
insert_job: eod_reconciliation
job_type: CMD
command: /opt/scripts/run_reconciliation.sh
machine: batch-server-01
owner: batchuser
permission: gx,wx
date_conditions: y
days_of_week: mo,tu,we,th,fr
start_times: "18:00"
condition: s(market_close_confirm) & s(positions_loaded)
alarm_if_fail: y
notification: mail="ops-team@example.com"This defines a job that runs at 18:00 on weekdays, but only if two conditions are set (meaning two upstream jobs completed successfully). If it fails, it raises an alarm and sends email.
Apache Airflow deserves mention here even though it occupies a different tier. Airflow is not an enterprise scheduler in the traditional sense, but it has become the dominant tool for data pipeline orchestration in modern data engineering work. If your firm has a data team that came from a technology background rather than a traditional financial services background, they are probably running Airflow. It is Python-native, code-defined, version-controllable, and significantly more pleasant to work with than Control-M or Autosys. Its weakness is operational maturity relative to the legacy platforms.
Working with legacy schedulers
If you inherit a Control-M or Autosys environment, a few things are worth knowing.
The job definition is not usually in source control. This is the first problem. In most legacy environments, jobs are defined directly in the scheduler's database, modified through a GUI, and not tracked in version control. Getting job definitions exported and into git is one of the most valuable things you can do for operational stability. Both Control-M and Autosys have export functionality; the process is tedious but worth doing.
Dependencies are the source of most incidents. The most common failure mode is a job that is waiting for a predecessor that has itself failed or been skipped. These chains can be long and the scheduler's dependency graph visualisation is often difficult to read. Invest time in understanding the critical path through your end-of-day runs, specifically which jobs are blocking the most downstream work.
Time-based triggers are less reliable than condition-based triggers. A job triggered at 18:00 will run at 18:00 whether the upstream data it needs is ready or not. A job triggered by a condition that upstream set will only run when upstream is actually done. Wherever possible, prefer condition-based triggering over time-based triggering.
Managed file transfer
What it is
Managed File Transfer (MFT) is a category of tooling for moving files between systems in a controlled, auditable, and reliable way. The "managed" part is important: unlike a simple FTP or SCP transfer, an MFT platform tracks every transfer, handles retries and failures, enforces encryption and authentication policies, provides non-repudiation (proof that a file was sent and received), and integrates with the audit infrastructure.
In financial services, this matters enormously. Files moving between systems often contain sensitive data: trade confirmations, regulatory reports, counterparty feeds, settlement instructions. Regulators care about how these files are transmitted, whether they are encrypted, who has access to them, and whether there is an audit trail. MFT platforms exist to answer all of these questions.
The major platforms
IBM Sterling File Gateway (formerly Sterling Commerce) is the dominant MFT platform in large financial institutions. It handles an enormous range of protocols (FTP, FTPS, SFTP, AS2, HTTP/S, and several financial industry-specific protocols), supports complex routing rules that transform and redirect files based on content or metadata, and provides the audit and non-repudiation capabilities that compliance teams need. It is also complex, expensive, and notoriously difficult to administer.
GoAnywhere MFT (Fortra) is a more modern alternative that has gained significant ground in financial services over the last decade. It supports the same protocol breadth as Sterling, has a significantly better administrative interface, and is easier to automate. For firms that are running Sterling because they inherited it and not because they specifically need its edge capabilities, GoAnywhere is often a viable migration target.
SFTP-based solutions are worth mentioning even though they are not full MFT platforms. Many firms run a managed SFTP service (either self-hosted on Linux with extensive scripting around it, or a cloud-based service like AWS Transfer Family) for specific use cases where the full weight of Sterling or GoAnywhere is not needed. This is reasonable for lower-risk file flows but becomes a problem when audit requirements or protocol support requirements grow beyond what basic SFTP can provide.
The key concepts
Trading partners are the external systems you are exchanging files with. In MFT terminology, a trading partner has a defined connection profile (protocol, credentials, endpoint), a set of agreed file formats, and a relationship that is tracked in the MFT platform. When something goes wrong with a specific partner, you look at that partner's configuration and recent transfer history.
Channels or routes define how files move between trading partners and internal systems. A channel typically specifies: where files come from, any transformation applied to them, and where they end up. Complex channels can fan out to multiple destinations or route based on file content.
Non-repudiation is the audit capability that makes MFT distinct from simple file copy. The platform records sender identity, file hash, timestamp, and acknowledgement from the receiver. This creates a chain of evidence that can be used to answer "was this file actually sent?" and "was it received intact?" questions that come up frequently in reconciliation failures and regulatory enquiries.
A practical example: replacing an SFTP script with Sterling
A common scenario: you have a batch job that uses a shell script to SFTP a file to a counterparty. It works, mostly. When it fails, it fails silently unless someone checks a log file. There is no audit trail. The credentials are in a config file on the batch server. Compliance wants evidence that the file was sent and received.
The Sterling-based replacement looks like this:
- Batch job writes the file to a Sterling-monitored mailbox directory instead of calling SFTP directly
- Sterling detects the new file, applies any required transformation (PGP encryption, format conversion)
- Sterling initiates the transfer to the counterparty via the agreed protocol
- Sterling records the transfer, including the acknowledgement from the counterparty
- Sterling triggers a completion event that the job scheduler can use as a condition
The batch job itself is simpler. The operational complexity moves into Sterling, where it is visible to the operations team, auditable by compliance, and monitored centrally.
How they work together
In practice, enterprise schedulers and MFT platforms are tightly coupled in financial services batch operations.
A typical end-of-day flow might look like this:
18:00 - Positions job runs (Control-M)
-> Sets condition: positions_complete
|
18:30 - P&L calculation job runs (condition: positions_complete)
-> Sets condition: pnl_complete
|
19:00 - Regulatory extract job runs (condition: pnl_complete)
-> Writes extract file to Sterling mailbox
|
Sterling picks up file, encrypts it, sends to regulator via SFTP
Sterling sets completion flag
|
19:30 - Confirmation job runs (condition: sterling_regulatory_sent)
-> Sends internal confirmation emailThe scheduler manages job dependencies and timing. The MFT platform manages the actual file movement and provides the audit trail. Each does what it is good at.
The modernisation question
The honest answer about these platforms is that most firms would not build this way if they were starting from scratch. Event streaming, API-based integrations, and cloud-native data pipelines have replaced batch file transfers for many use cases. Airflow or Prefect have replaced enterprise schedulers for data engineering work.
But the legacy infrastructure is real and it is not going away quickly. The reconciliation runs that close the books at end of day have been running the same way for twenty years and the firm's appetite to modernise them is limited by risk tolerance and competing priorities. Counterparties that still send files over SFTP are not going to switch to API calls because you asked them to.
The useful posture is to understand the existing infrastructure well enough to operate it reliably, look for opportunities to introduce better tooling for new work rather than trying to immediately replace the legacy foundation, and build incrementally. New data feeds can go through Airflow rather than Autosys. New file transfers that are lower-risk can use AWS Transfer Family rather than Sterling. Over time the footprint of the legacy platforms shrinks, but the transition takes years and the old systems need to keep working in the meantime.
If you are walking into an infrastructure role at a financial firm and discovering Control-M and Sterling for the first time, start by finding someone who has worked with them for years and asking them to walk you through the critical path jobs and the trading partners that matter most. That knowledge is rarely documented anywhere. It lives in people, and those people are not always easy to find.
