Cross-Platform Task Scheduler

Challenge

While extending the capacities of our on premise system, we needed a scalable solution to schedule workloads across cloud and and on-premise devices. Due to core legacy applications and heavy GPU usage, we could not rely on any form of virtualization and existing scheduling solutions were not sufficient.

My Contribution

I designed and built a cross-platform task scheduler using python, AWS Lambda, SQS and SNS. With this solution we were able to schedule tasks on-premise and in the cloud and scale with ease.

Key features:

• Task scheduling and execution with configurable workers depending on machine performance
• Task status tracking
• Task monitoring
• Isolated task execution on machines with after run cleanup
• Platform independant task execution
• Deployment for scheduler on aws lambda via terraform
• Deployment for executor on-premise via terraform and codedeploy

Impact

With the new task scheduler and execution flow in place, we were able to schedule our legacy and GPU heavy workloads. We gained the ability to scale and monitor more effectively. Apart from that whilst our local machines had heavy computational capabilities, but were not used around the clock, we were able to schedule any tasks on-premise and reduce cloud costs.

pythonawsterraformfastapisqs/snslambdalinuxmongodbcodedeploy